Result Details
BUT/JHU System Description for CHiME-8 NOTSOFAR-1 Challenge
Klement Dominik, Ing., DCGM (FIT)
Han Jiangyu, DCGM (FIT)
Sedláček Šimon, Ing., DCGM (FIT)
Yusuf Bolaji, DCGM (FIT)
Maciejewski Matthew
Wiesner Matthew, PhD., FIT (FIT)
Burget Lukáš, doc. Ing., Ph.D., DCGM (FIT)
This paper presents our method for tackling the CHIME-8 chal-
lenge's NOTSOFAR-1 task, which requires participants to per-
form multi-speaker automatic speech recognition (ASR) using
audio from distant microphone arrays. We modify the Pyan-
note3 diarization pipeline, incorporating pre-trained WavLM as
local EEND to adapt effectively to new domains, and we intro-
duce two diarization-aware approaches to ASR by condition-
ing Whisper on diarization outputs for target-speaker ASR. The
first method, which we refer to as Query-Key Biasing, modi-
fies Whisper's attention mechanism and positional embeddings
with a learnable attention mask to exclude non-target speaker
segments in the audio. The second method, called Frame-
Level Diarization-Dependent Transformations, applies affine,
diarization-dependent transformations with trainable parame-
ters to the inputs of one or more transformer blocks. We also
extend both the ASR and diarization systems to a multichannel
setup by incorporating cross-channel communication into our
models. Finally, we report the performance of these approaches
on the NOTSOFAR-1 dataset.
multi-talker speech recognition, CHiME-8, NOTSOFAR-1, target-speaker
@inproceedings{BUT194002,
author="Alexander {Polok} and Dominik {Klement} and Jiangyu {Han} and Šimon {Sedláček} and Bolaji {Yusuf} and Matthew {Maciejewski} and Matthew {Wiesner} and Lukáš {Burget}",
title="BUT/JHU System Description for CHiME-8 NOTSOFAR-1 Challenge",
booktitle="Proceedings of CHiME 2024 Workshop",
year="2024",
pages="18--22",
publisher="International Speech Communication Association",
address="Kos Island",
doi="10.21437/CHiME.2024-4",
url="https://www.isca-archive.org/chime_2024/polok24_chime.pdf"
}Multilingual and Cross-cultural interactions for context-aware, and bias-controlled dialogue systems for safety-critical applications, EU, HORIZON EUROPE, start: 2024-01-01, end: 2026-12-31, running
Practical verification of the possibility of integrating artificial intelligence for receiving emergency calls using a voice chatbot, developed within the research project BV No. VI20192022169, with technology for receiving emergency communications, MV, 1 VS OPSEC, VK01020132, start: 2023-01-06, end: 2025-10-31, running
Tools To Combat Voice DeepFakes, MV, Programu bezpečnostního výzkumu ČR 2021-2026: vývoj, testování a evaluace nových bezpečnostních technologií (SECTECH) - II. veřejná soutěž, VB02000060, start: 2024-01-01, end: 2026-12-31, running