Result Details
ESPnet-SpeechLM: An Open Speech Language Model Toolkit
Shi Jiatong
Chen William
Arora Siddhant
Masuyama Yoshiki
Maekaku Takashi
Wu Yihan
Peng Junyi, DCGM (FIT)
Bharadwaj Shikhar
Zhao Yiwen
Cornell Samuele
Peng Yifan
Yue Xiang
Yang Chao Han Huck
Neubig Graham
Watanabe Shinji
We present ESPnet-SpeechLM, an open toolkit designed to democratize the development of speech language models (SpeechLMs) and voice-driven agentic applications. The toolkit standardizes speech processing tasks by framing them as universal sequential modeling problems, encompassing a cohesive workflow of data preprocessing, pre-training, inference, and task evaluation. With ESPnet-SpeechLM, users can easily define task templates and configure key settings, enabling seamless and streamlined SpeechLM development. The toolkit ensures flexibility, efficiency, and scalability by offering highly configurable modules for every stage of the workflow. To illustrate its capabilities, we provide multiple use cases demonstrating how competitive SpeechLMs can be constructed with ESPnet-SpeechLM, including a 1.7B-parameter model pre-trained on both text and speech tasks, across diverse benchmarks.
Speech communication; Speech processing; Data preprocessing; Language model; Model problems; Multiple use-cases; Parameter model; Pre-training; Sequential modeling; Work-flows
@inproceedings{BUT201388,
author="{} and {} and {} and {} and {} and {} and {} and Junyi {Peng} and {} and {} and {} and {} and {} and {} and {} and {}",
title="ESPnet-SpeechLM: An Open Speech Language Model Toolkit",
booktitle="Proceedings of the 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies: Long Papers, NAACL-HLT 2025",
year="2025",
pages="116--124",
publisher="Association for Computational Linguistics (ACL)",
address="Hybrid, Albuquerque, New Mexico, USA",
doi="10.18653/v1/2025.naacl-demo.12",
isbn="9798891761919",
url="https://aclanthology.org/2025.naacl-demo.12.pdf"
}