Result Details

Strategies for Training Large Scale Neural Network Language Models

MIKOLOV, T.; DEORAS, A.; POVEY, D.; BURGET, L.; ČERNOCKÝ, J. Strategies for Training Large Scale Neural Network Language Models. Proceedings of ASRU 2011. Hilton Waikoloa Village, Big Island, Hawaii: IEEE Signal Processing Society, 2011. p. 196-201. ISBN: 978-1-4673-0366-8.
Type
conference paper
Language
English
Authors
Mikolov Tomáš, Ing., Ph.D., DCGM (FIT)
Deoras Anoop
Povey Daniel
Burget Lukáš, doc. Ing., Ph.D., DCGM (FIT)
Černocký Jan, prof. Dr. Ing., DCGM (FIT)
Abstract

Techniques for effective training of recurrent neural network based language models are described, and new state-of-the-art results on standard speech recognition task are reported.

Keywords

recurrent neural network, language model, speech recognition, maximum entropy

URL
Annotation

We describe how to effectively train neural network based language models on large data sets. Fast convergence during training and better overall performance is observed when the training data are sorted by their relevance. We introduce hash-based implementation of a maximum entropy model, that can be trained as a part of the neural network model. This leads to significant reduction of computational complexity. We achieved around 10% relative reduction of word error rate on English Broadcast News speech recognition task, against large 4-gram model trained on 400M tokens.

Published
2011
Pages
196–201
Proceedings
Proceedings of ASRU 2011
Conference
IEEE 2011 Workshop on Automatic Speech Recognition and Understanding
ISBN
978-1-4673-0366-8
Publisher
IEEE Signal Processing Society
Place
Hilton Waikoloa Village, Big Island, Hawaii
BibTeX 
@inproceedings{BUT76453,
  author="Tomáš {Mikolov} and Anoop {Deoras} and Daniel {Povey} and Lukáš {Burget} and Jan {Černocký}",
  title="Strategies for Training Large Scale Neural Network Language Models",
  booktitle="Proceedings of ASRU 2011",
  year="2011",
  pages="196--201",
  publisher="IEEE Signal Processing Society",
  address="Hilton Waikoloa Village, Big Island, Hawaii",
  isbn="978-1-4673-0366-8",
  url="http://www.fit.vutbr.cz/research/groups/speech/publi/2011/mikolov_asru2011_00196.pdf"
}
Projects
Multilingual recognition and search in speech for electronic dictionaries, MPO, TIP, FR-TI1/034, start: 2009-09-01, end: 2013-08-31, completed
Speech Recognition under Real-World Conditions, GACR, Standardní projekty, GA102/08/0707, start: 2008-01-01, end: 2011-12-31, completed
Technologies of speech processing for efficient human-machine communication, TAČR, Program aplikovaného výzkumu a experimentálního vývoje ALFA, TA01011328, start: 2011-01-01, end: 2014-12-31, completed
Research groups
Speech Data Mining Research Group BUT Speech@FIT (RG SPEECH)
Departments
Department of Computer Graphics and Multimedia (DCGM)
Back to top