Publication Details

Usted: Improving ASR with a Unified Speech and Text Encoder-Decoder

YUSUF Bolaji, GANDHE Ankur and SOKOLOV Alex. Usted: Improving ASR with a Unified Speech and Text Encoder-Decoder. In: ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings. Singapore: IEEE Signal Processing Society, 2022, pp. 8297-8301. ISBN 978-1-6654-0540-9. Available from: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9746554
Czech title
Zlepšení ASR pomocí unifikovaného řečového a textového enkodéru-dekodéru
Type
conference paper
Language
english
Authors
Yusuf Bolaji (DCGM FIT BUT)
Gandhe Ankur (AmazonCom)
Sokolov Alex (AmazonCom)
URL
Keywords

sequence-to-sequence, multitask, end-to-end ASR, masked language model, machine translation

Abstract

Improving end-to-end speech recognition by incorporating external text data has been a longstanding research topic. There has been a recent focus on training E2E ASR models that get the performance benefits of external text data without incurring the extra cost of evaluating an external language model at inference time. In this work, we propose training ASR model jointly with a set of text-to-text auxiliary tasks with which it shares a decoder and parts of the encoder. When we jointly train ASR and masked language model with the 960-hour Librispeech and Opensubtitles data respectively, we observe WER reductions of 16% and 20% on test-other and test-clean respectively over an ASR-only baseline without any extra cost at inference time, and reductions of 6% and 8% compared to a stronger MUTE-L baseline which trains the decoder with the same text data as our model. We achieve further improvements when we train masked language model on Librispeech data or when we use machine translation as the auxiliary task, without significantly sacrificing performance on the task itself.

Published
2022
Pages
8297-8301
Proceedings
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
Conference
2022 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), Singapore, SG
ISBN
978-1-6654-0540-9
Publisher
IEEE Signal Processing Society
Place
Singapore, SG
DOI
UT WoS
000864187908121
EID Scopus
BibTeX
@INPROCEEDINGS{FITPUB12784,
   author = "Bolaji Yusuf and Ankur Gandhe and Alex Sokolov",
   title = "Usted: Improving ASR with a Unified Speech and Text Encoder-Decoder",
   pages = "8297--8301",
   booktitle = "ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings",
   year = 2022,
   location = "Singapore, SG",
   publisher = "IEEE Signal Processing Society",
   ISBN = "978-1-6654-0540-9",
   doi = "10.1109/ICASSP43922.2022.9746554",
   language = "english",
   url = "https://www.fit.vut.cz/research/publication/12784"
}
Back to top