i-vectors in language modeling: An efficient way of domain adaptation for feed-forward models

We show an effective way of adding context information toshallow neural language models. We propose to use SubspaceMultinomial Model (SMM) for context modeling and we addthe extracted i-vectors in a computationally efficient way. Byadding this information, we shrink the gap between shallowfeed-forward network and an LSTM from 65 to 31 points of perplexityon the Wikitext-2 corpus (in the case of neural 5-grammodel). Furthermore, we show that SMM i-vectors are suitablefor domain adaptation and a very small amount of adaptationdata (e.g. endmost 5% of a Wikipedia article) brings asubstantial improvement. Our proposed changes are compatiblewith most optimization techniques used for shallow feedforwardLMs.