Result Details
Leveraging Self-Supervised Learning for Speaker Diarization
Landini Federico Nicolás, Ph.D.
Rohdin Johan Andréas, M.Sc., Ph.D., FIT (FIT), DCGM (FIT)
Silnova Anna, M.Sc., Ph.D., DCGM (FIT)
Diez Sánchez Mireia, M.Sc., Ph.D., DCGM (FIT)
Burget Lukáš, doc. Ing., Ph.D., DCGM (FIT)
End-to-end neural diarization has evolved considerably over the past few years,
but data scarcity is still a major obstacle for further improvements.
Self-supervised learning methods such as WavLM have shown promising performance
on several downstream tasks, but their application on speaker diarization is
somehow limited. In this work, we explore using WavLM to alleviate the problem of
data scarcity for neural diarization training. We use the same pipeline as
Pyannote and improve the local end-to-end neural diarization with WavLM and
Conformer. Experiments on far-field AMI, AISHELL-4, and AliMeeting datasets show
that our method substantially outperforms the Pyannote baseline and achieves new
state-of-the-art results on AMI and AISHELL- 4, respectively. In addition, by
analyzing the system performance under different data quantity scenarios, we show
that WavLM representations are much more robust against data scarcity than
filterbank features, enabling less data hungry training strategies. Furthermore,
we found that simulated data, usually used to train end-to-end diarization
models, does not help when using WavLM in our experiments. Additionally, we also
evaluate our model on the recent CHiME8 NOTSOFAR-1 task where it achieves better
performance than the Pyannote baseline. Our source code is publicly available at
https://github.com/BUTSpeechFIT/DiariZen.
Speaker diarization, data scarcity, WavLM, Pyannote, far-field meeting data
@inproceedings{BUT198048,
author="Jiangyu {Han} and Federico Nicolás {Landini} and Johan Andréas {Rohdin} and Anna {Silnova} and Mireia {Diez Sánchez} and Lukáš {Burget}",
title="Leveraging Self-Supervised Learning for Speaker Diarization",
booktitle="ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings",
year="2025",
pages="1--5",
publisher="IEEE Signal Processing Society",
address="Hyderabad",
doi="10.1109/ICASSP49660.2025.10889475",
isbn="979-8-3503-6874-1",
url="https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10889475"
}Linguistics, Artificial Intelligence and Language and Speech Technologies: from Research to Applications, EU, MEZISEKTOROVÁ SPOLUPRÁCE, EH23_020/0008518, start: 2025-01-01, end: 2028-12-31, running
Practical verification of the possibility of integrating artificial intelligence for receiving emergency calls using a voice chatbot, developed within the research project BV No. VI20192022169, with technology for receiving emergency communications, MV, 1 VS OPSEC, VK01020132, start: 2023-01-06, end: 2025-10-31, completed
Robust processing of recordings for operations and security, MV, PROGRAM STRATEGICKÁ PODPORA ROZVOJE BEZPEČNOSTNÍHO VÝZKUMU ČR 2019-2025 (IMPAKT 1) PODPROGRAMU 1 SPOLEČNÉ VÝZKUMNÉ PROJEKTY (BV IMP1/1VS), VJ01010108, start: 2020-10-01, end: 2025-09-30, completed