Integration of Variational Autoencoder and Spatial Clustering for Adaptive Multi-Channel Neural Speech Separation

In this paper, we propose a method combining variational autoencodermodel of speech with a spatial clustering approach for multichannelspeech separation. The advantage of integrating spatial clusteringwith a spectral model was shown in several works. As thespectral model, previous works used either factorial generative modelsof the mixed speech or discriminative neural networks. In ourwork, we combine the strengths of both approaches, by building afactorial model based on a generative neural network, a variationalautoencoder. By doing so, we can exploit the modeling power ofneural networks, but at the same time, keep a structured model. Sucha model can be advantageous when adapting to new noise conditionsas only the noise part of the model needs to be modified. We showexperimentally, that our model significantly outperforms previousfactorial model based on Gaussian mixture model (DOLPHIN), performscomparably to integration of permutation invariant trainingwith spatial clustering, and enables us to easily adapt to new noiseconditions.