Publication Details
Evaluation Framework for Deepfake Speech Detection: A Comparative Study of State-of-the-art Deepfake Speech Detectors
Deepfake speech, Detection, Robustness, Evaluation framework, Computer security
The proliferation of deepfake speech poses a significant threat to cybersecurity,
from manipulating political speeches and impersonating public figures to spoofing
voice biometric systems. The increasing sophistication of adversaries increases
the necessity of deploying adaptive detection methods. Moreover, real-world
incidents such as fraudulent financial transactions highlight the severity of the
problem. Although numerous detectors have been developed, their evaluation
remains difficult due to different methodologies and benchmark datasets, making
direct comparisons impossible. This study presents a general and detailed
framework for evaluating and comparing deepfake speech detectors. We further
demonstrate the use of this framework to evaluate 40 state-of-the-art deepfake
speech detectors under various conditions and data samples. We objectively
compare these methods and identify the key attributes influencing performance the
most. We also stress the issue of generalisation, as current detectors struggle
to detect previously unseen deepfake speech samples or samples that have been
modified. Finally, to strengthen the defence against synthetic audio content, we
provide recommendations for improving the robustness of future detectors.
@article{BUT193261,
author="Anton {Firc} and Kamil {Malinka} and Petr {Hanáček}",
title="Evaluation Framework for Deepfake Speech Detection: A Comparative Study of State-of-the-art Deepfake Speech Detectors",
journal="Cybersecurity",
year="2025",
volume="8",
number="50",
pages="1--24",
doi="10.1186/s42400-024-00346-1",
issn="2523-3246",
url="https://cybersecurity.springeropen.com/articles/10.1186/s42400-024-00346-1"
}