Result Details
Residual Memory Networks in Language Modeling: Improving the Reputation of Feed-Forward Networks
Baskar Murali Karthick, Ing., Ph.D., DCGM (FIT)
Burget Lukáš, doc. Ing., Ph.D., DCGM (FIT)
The paper describes the residual memory networks in language modeling: Improving the Reputation of Feed-Forward Networks.
residual memory networks, feed-forward networks,language modeling
We introduce the Residual Memory Network (RMN) architecture to language modeling. RMN is an architecture of feedforward neural networks that incorporates residual connections and time-delay connections that allow us to naturally incorporate information from a substantial time context. As this is the first time RMNs are applied for language modeling, we thoroughly investigate their behaviour on the well studied Penn Treebank corpus. We change the model slightly for the needs of language modeling, reducing both its time and memory consumption. Our results show that RMN is a suitable choice for small-sized neural language models: With test perplexity 112.7 and as few as 2.3M parameters, they out-perform both a much larger vanilla RNN (PPL 124, 8M parameters) and a similarly sized LSTM (PPL 115, 2.08M parameters), while being only by less than 3 perplexity points worse than twice as big LSTM.
@inproceedings{BUT144489,
author="Karel {Beneš} and Murali Karthick {Baskar} and Lukáš {Burget}",
title="Residual Memory Networks in Language Modeling: Improving the Reputation of Feed-Forward Networks",
booktitle="Proceedings of Interspeeech 2017",
year="2017",
journal="Proceedings of Interspeech",
volume="2017",
number="08",
pages="284--288",
publisher="International Speech Communication Association",
address="Stockholm",
doi="10.21437/Interspeech.2017-1442",
issn="1990-9772",
url="http://www.isca-speech.org/archive/Interspeech_2017/pdfs/1442.PDF"
}IT4Innovations excellence in science, MŠMT, Národní program udržitelnosti II, LQ1602, start: 2016-01-01, end: 2020-12-31, completed
Meeting Assistant (MINT), TAČR, Program aplikovaného výzkumu a experimentálního vývoje ALFA, TA04011311, start: 2014-10-01, end: 2017-12-31, completed