Speech-Based Emotion Recognition with Self-Supervised Models Using Attentive Channel-Wise Correlations and Label Smoothing

When recognizing emotions from speech, we encounter
two common problems: how to optimally capture emotion-
relevant information from the speech signal and how to best
quantify or categorize the noisy subjective emotion labels.
Self-supervised pre-trained representations can robustly capture information from speech enabling state-of-the-art results
in many downstream tasks including emotion recognition.
However, better ways of aggregating the information across
time need to be considered as the relevant emotion information is likely to appear piecewise and not uniformly across
the signal. For the labels, we need to take into account that
there is a substantial degree of noise that comes from the
subjective human annotations. In this paper, we propose a
novel approach to attentive pooling based on correlations between the representations' coefficients combined with label
smoothing, a method aiming to reduce the confidence of the
classifier on the training labels. We evaluate our proposed
approach on the benchmark dataset IEMOCAP, and demonstrate high performance surpassing that in the literature. The
code to reproduce the results is available at github.com/skakouros/s3prl_attentive_correlation.