Result Details
Masked Self-Supervised Pre-Training for Text Recognition Transformers on Large-Scale Datasets
Self-supervised learning has emerged as a powerful approach for leveraging large-scale unlabeled data to improve model performance in various domains. In this paper, we explore masked self-supervised pre-training for text recognition transformers. Specifically, we propose two modifications to the pre-training phase: progressively increasing the masking probability, and modifying the loss function to incorporate both masked and non-masked patches. We conduct extensive experiments using a dataset of 50M unlabeled text lines for pre-training and four differently sized annotated datasets for fine-tuning. Furthermore, we compare our pre-trained models against those trained with transfer learning, demonstrating the effectiveness of the self-supervised pre-training. In particular, pre-training consistently improves the character error rate of models, in some cases up to 30 % relatively. It is also on par with transfer learning but without relying on extra annotated text lines.
Self-supervised pre-training; Transformers; OCR; HTR
@inproceedings{BUT197661,
author="Martin {Kišš} and Michal {Hradiš} and {}",
title="Masked Self-Supervised Pre-Training for Text Recognition Transformers on Large-Scale Datasets",
booktitle="Document Analysis and Recognition – ICDAR 2025 Workshops",
year="2025",
pages="53--70",
publisher="Springer Nature Switzerland",
address="Cham",
doi="10.1007/978-3-032-09368-4\{_}4",
isbn="978-3-032-09367-7",
url="https://link.springer.com/chapter/10.1007/978-3-032-09368-4_4"
}