Detail výsledku
Parameter-Efficient Tuning With Adaptive Bottlenecks For Automatic Speech Recognition
Prasad Amrutha
KHALIL, D.
Madikeri Srikanth, FIT (FIT)
DEMUYNCK, K.
Motlíček Petr, doc. Ing., Ph.D., UPGM (FIT)
Transfer learning from large multilingual pretrained models,
like XLSR, has become the new paradigm for Automatic
Speech Recognition (ASR). Considering their ever-increasing
size, fine-tuning all the weights has become impractical when
the computing budget is limited. Adapters are lightweight
trainable modules inserted between layers while the pretrained
part is kept frozen. They form a parameter-efficient
fine-tuning method, but they still require a large bottleneck
size to match standard fine-tuning performance. In this paper,
we propose ABSADAPTER, a method to further reduce the
parameter budget for equal task performance. Specifically,
ABSADAPTER uses an Adaptive Bottleneck Scheduler to
redistribute the adapter's weights to the layers that need adaptation
the most. By training only 8% of the XLSR model,
ABSADAPTER achieves close to standard fine-tuning performance
on a domain-shifted Air-Traffic Communication
(ATC) ASR task.
ASR, XLSR, Adapters, ATC
@inproceedings{BUT187932,
author="VANDERREYDT, G. and PRASAD, A. and KHALIL, D. and MADIKERI, S. and DEMUYNCK, K. and MOTLÍČEK, P.",
title="Parameter-Efficient Tuning With Adaptive Bottlenecks For Automatic Speech Recognition",
booktitle="Proceedings of IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)",
year="2023",
pages="1--7",
publisher="IEEE Signal Processing Society",
address="Taipei",
doi="10.1109/ASRU57964.2023.10389769",
isbn="979-8-3503-0689-7",
url="https://ieeexplore.ieee.org/document/10389769"
}