Optimization of Speaker-aware Multichannel Speech Extraction with ASR Criterion

This paper addresses the problem of recognizing speech corruptedby overlapping speakers in a multichannel setting. Toextract a target speaker from the mixture, we use a neural networkbased beamformer which uses masks estimated by a neuralnetwork to compute statistically optimal spatial filters. Followingour previous work, we inform the neural network about thetarget speaker using information extracted from an adaptation utterance,enabling the network to track the target speaker. Whilein the previous work, this method was used to separately extractthe speaker and then pass such preprocessed speech to a speechrecognition system, here we explore training both systems jointlywith a common speech recognition criterion. We show that integratingthe two systems and training for the final objective improvesthe performance. In addition, the integration enables furthersharing of information between the acoustic model and thespeaker extraction system, by making use of the predicted HMMstateposteriors to refine the masks used for beamforming.