Single Channel Target Speaker Extraction and Recognition with Speaker Beam

This paper addresses the problem of single channel speech recognitionof a target speaker in a mixture of speech signals. We proposeto exploit auxiliary speaker information provided by an adaptationutterance from the target speaker to extract and recognize only thatspeaker. Using such auxiliary information, we can build a speakerextraction neural network (NN) that is independent of the numberof sources in the mixture, and that can track speakers across differentutterances, which are two challenging issues occurring withconventional approaches for speech recognition of mixtures. Wecall such an informed speaker extraction scheme "SpeakerBeam".SpeakerBeam exploits a recently developed context adaptive deepNN (CADNN) that allows tracking speech from a target speaker usinga speaker adaptation layer, whose parameters are adjusted dependingon auxiliary features representing the target speaker characteristics.SpeakerBeam was previously investigated for speaker extractionusing a microphone array. In this paper, we demonstrate thatit is also efficient for single channel speaker extraction. The speakeradaptation layer can be employed either to build a speaker adaptiveacoustic model that recognizes only the target speaker or a maskbasedspeaker extraction network that extracts the target speech fromthe speech mixture signal prior to recognition. We also show thatthe latter speaker extraction network can be optimized jointly withan acoustic model to further improve ASR performance.