Residual Memory Networks: Feed-forward approach to learn long-term temporal dependencies

Training deep recurrent neural network (RNN) architectures iscomplicated due to the increased network complexity. This disruptsthe learning of higher order abstracts using deep RNN. Incase of feed-forward networks training deep structures is simpleand faster while learning long-term temporal information isnot possible. In this paper we propose a residual memory neuralnetwork (RMN) architecture to model short-time dependenciesusing deep feed-forward layers having residual and time delayedconnections. The residual connection paves way to constructdeeper networks by enabling unhindered flow of gradientsand the time delay units capture temporal information withshared weights. The number of layers in RMN signifies both thehierarchical processing depth and temporal depth. The computationalcomplexity in training RMN is significantly less whencompared to deep recurrent networks. RMN is further extendedas bi-directional RMN (BRMN) to capture both past and futureinformation. Experimental analysis is done on AMI corpus tosubstantiate the capability of RMN in learning long-term informationand hierarchical information. Recognition performanceof RMN trained with 300 hours of Switchboard corpus is comparedwith various state-of-the-art LVCSR systems. The resultsindicate that RMN and BRMN gains 6 % and 3.8 % relativeimprovement over LSTM and BLSTM networks.