Faculty of Information Technology, BUT

Publication Details

Analysis of Multilingual Sequence-to-Sequence Speech Recognition Systems

KARAFIÁT Martin, BASKAR Murali K., WATANABE Shinji, HORI Takaaki, WIESNER Matthew and ČERNOCKÝ Jan. Analysis of Multilingual Sequence-to-Sequence Speech Recognition Systems. In: Proceedings of Interspeech. Graz: International Speech Communication Association, 2019, pp. 2220-2224. ISSN 1990-9772. Available from: https://www.isca-speech.org/archive/Interspeech_2019/pdfs/2355.pdf
Czech title
Analýza multilingválního systému pro rozpoznávání řeči založeného na převodu sekvencí na sekvence
Type
conference paper
Language
english
Authors
Karafiát Martin, Ing., Ph.D. (DCGM FIT BUT)
Baskar Murali K. (DCGM FIT BUT)
Watanabe Shinji, Dr. (JHU)
Hori Takaaki (MERL)
Wiesner Matthew (JHU)
Černocký Jan, doc. Dr. Ing. (DCGM FIT BUT)
URL
Keywords
multilingual ASR, sequence-to-sequence, language-transfer, multilingual bottle-neck feature
Abstract
This paper investigates the applications of various multilingual approaches developed in conventional deep neural network - hidden Markov model (DNN-HMM) systems to sequence-tosequence (seq2seq) automatic speech recognition (ASR). We employ a joint connectionist temporal classification-attention network as our base model. Our main contribution is separated into two parts. First, we investigate the effectiveness of the seq2seq model with stacked multilingual bottle-neck features obtained from a conventional DNN-HMM system on the Babel multilingual speech corpus. Second, we investigate the effectiveness of transfer learning from a pre-trained multilingual seq2seq model with and without the target language included in the original multilingual training data. In this experiment, we also explore various architectures and training strategies of the multilingual seq2seq model by making use of knowledge obtained in the DNN-HMM based transfer-learning. Although both approaches significantly improved the performance from a monolingual seq2seq baseline, interestingly, we found the multilingual bottle-neck features to be superior to multilingual models with transfer learning. This finding suggests that we can efficiently combine the benefits of the DNN-HMM system with the seq2seq system through multilingual bottle-neck feature techniques.
Published
2019
Pages
2220-2224
Journal
Proceedings of Interspeech, vol. 2019, no. 9, ISSN 1990-9772
Proceedings
Proceedings of Interspeech
Conference
INTERSPEECH 2019, INTERSPEECH 2019, AT
Publisher
International Speech Communication Association
Place
Graz, AT
DOI
BibTeX
@INPROCEEDINGS{FITPUB12088,
   author = "Martin Karafi\'{a}t and K. Murali Baskar and Shinji Watanabe and Takaaki Hori and Matthew Wiesner and Jan \v{C}ernock\'{y}",
   title = "Analysis of Multilingual Sequence-to-Sequence Speech Recognition Systems",
   pages = "2220--2224",
   booktitle = "Proceedings of Interspeech",
   journal = "Proceedings of Interspeech",
   volume = 2019,
   number = 9,
   year = 2019,
   location = "Graz, AT",
   publisher = "International Speech Communication Association",
   ISSN = "1990-9772",
   doi = "10.21437/Interspeech.2019-2355",
   language = "english",
   url = "https://www.fit.vut.cz/research/publication/12088"
}
Back to top