Publication Details

Semi-supervised Training of Deep Neural Networks

VESELÝ Karel, HANNEMANN Mirko and BURGET Lukáš. Semi-supervised Training of Deep Neural Networks. In: Proceedings of ASRU 2013. Olomouc: IEEE Signal Processing Society, 2013, pp. 267-272. ISBN 978-1-4799-2755-5.
Czech title
Částečně kontrolované trénování hlubokých neuronových sítí
Type
conference paper
Language
english
Authors
URL
Keywords

semi-supervised training, self-training, deep network, DNN, Babel program

Abstract

Our quest in this paper is to search for an optimal dataselection strategy for the semi-supervised DNN training. We performed an analysis at all the three stages of DNN training.

Annotation

In this paper we search for an optimal strategy for semisupervised Deep Neural Network (DNN) training. We assume that a small part of the data is transcribed, while the majority of the data is untranscribed. We explore self-training strategies with data selection based on both the utterance-level and frame-level confidences. Further on, we study the interactions between semi-supervised frame-discriminative training and sequence-discriminative sMBR training. We found it beneficial to reduce the disproportion in amounts of transcribed and untranscribed data by including the transcribed data several times, as well as to do a frame-selection based on per-frame confidences derived from confusion in a lattice. For the experiments, we used the Limited language pack condition for the Surprise language task (Vietnamese) from the IARPA Babel program. The absolute Word Error Rate (WER) improvement for frame cross-entropy training is 2.2%, this corresponds to WER recovery of 36% when compared to the identical system, where the DNN is built on the fully transcribed data.

Published
2013
Pages
267-272
Proceedings
Proceedings of ASRU 2013
Conference
IEEE 2013 Workshop on Automatic Speech Recognition and Understanding, Olomouc, CZ
ISBN
978-1-4799-2755-5
Publisher
IEEE Signal Processing Society
Place
Olomouc, CZ
BibTeX
@INPROCEEDINGS{FITPUB10509,
   author = "Karel Vesel\'{y} and Mirko Hannemann and Luk\'{a}\v{s} Burget",
   title = "Semi-supervised Training of Deep Neural Networks",
   pages = "267--272",
   booktitle = "Proceedings of ASRU 2013",
   year = 2013,
   location = "Olomouc, CZ",
   publisher = "IEEE Signal Processing Society",
   ISBN = "978-1-4799-2755-5",
   language = "english",
   url = "https://www.fit.vut.cz/research/publication/10509"
}
Back to top