Semi-supervised Sequence-to-sequence ASR using Unpaired Speech and Text

Sequence-to-sequence automatic speech recognition (ASR)models require large quantities of data to attain highperformance. For this reason, there has been a recent surgein interest for unsupervised and semi-supervised training insuch models. This work builds upon recent results showingnotable improvements in semi-supervised training usingcycle-consistency and related techniques. Such techniquesderive training procedures and losses able to leverage unpairedspeech and/or text data by combining ASR with Text-to-Speech(TTS) models. In particular, this work proposes a newsemi-supervised loss combining an end-to-end differentiableASR!TTS loss with TTS!ASR loss. The method is ableto leverage both unpaired speech and text data to outperformrecently proposed related techniques in terms of %WER. Weprovide extensive results analyzing the impact of data quantityand speech and text modalities and show consistent gains acrossWSJ and Librispeech corpora. Our code is provided in ESPnetto reproduce the experiments.