# The THUEE System Description for the IARPA OpenASR21 Challenge

###### Abstract

This paper describes the THUEE team’s speech recognition system for the IARPA
Open Automatic Speech Recognition Challenge (OpenASR21), with further
experiment explorations. We achieve outstanding results under both the
Constrained and Constrained-plus training conditions. For the Constrained
training condition, we construct our basic ASR system based on the standard
hybrid architecture. To alleviate the Out-Of-Vocabulary (OOV) problem, we
extend the pronunciation lexicon using Grapheme-to-Phoneme (G2P) techniques
for both OOV and potential new words. Standard acoustic model structures such
as CNN-TDNN-F and CNN-TDNN-F-A are adopted. In addition, multiple data
augmentation techniques are applied. For the Constrained-plus training
condition, we use the self-supervised learning framework wav2vec2.0. We
experiment with various fine-tuning techniques with the Connectionist Temporal
Classification (CTC) criterion on top of the publicly available pre-trained
model XLSR-53. We find that the frontend feature extractor plays an important
role when applying the wav2vec2.0 pre-trained model to the encoder-decoder
based CTC/Attention ASR architecture. Extra improvements can be achieved by
using the CTC model finetuned in the target language as the frontend feature
extractor.

Index Terms: low-resource languages, OpenASR2021, speech recognition,
wav2vec2.0, XLSR-53

##  1 Introduction

The OpenASR21 Challenge111https://sat.nist.gov/openasr21 is the third open
challenge created out of the Intelligence Advanced Research Projects Activity
(IARPA) Machine Translation for English Retrieval of Information in Any
Language (MATERIAL) program222 https://www.iarpa.gov/index.php/research-
programs/material. The goal of the OpenASR21 is to assess the state of the art
of ASR technologies for low-resource languages. For most of the languages in
the world, there are no applicable ASR systems due to the lack of high-quality
annotated speech data. It is challenging to build a strong ASR system with
limited speech data, transcripts as well as lexicons. The capabilities tested
in the open challenges are expected to ultimately support the MATERIAL task of
effective triage and analysis of large volumes of data in a variety of less-
studied languages333https://www.nist.gov/document/openasr21-challenge-
evaluation-plan.

We describe our ASR systems in detail to demonstrate our solutions for
challenge in the Constrained and Constrained-plus training conditions
respectively. Post evaluation experiments for the Constrained-plus training
condition are also provided for future references.

For the Constrained training condition, we focus on hybrid acoustic models
with CNN-TDNN-F and CNN-TDNN-F-A network structures[1]. To alleviate the OOV
problem, which significantly impacts the speech recognition performance in
low-resource conditions, we extend our lexicon using G2P techniques. OOV words
from the training data and their corresponding pronunciations generated by G2P
techniques are first added to the base lexicon. The lexicon is then extended
by generating potential new words in the target language using the method
described in [2]. Recurrent neural network language models are applied by
performing lattice rescoring forward and backward successively [3]. To further
improve the robustness, we combine various data augmentation methods [4, 5, 6]
during the training process, and we train three different systems for each
language for a final system combination.

For the Constrained-plus training condition, unsupervised pre-trained models
are allowed in addition to the provided 10-hour dataset. We achieve
significant improvements with extra unlabeled speech data from the BABEL
program444https://www.iarpa.gov/index.php/research-programs/babel and
unsupervised pre-trained wav2vec2.0 [7] models compared to the Constrained
training condition. We use two finetuning methods to build our systems, and
different systems are also processed by different Speech Activity Detection
(SAD) methods. Results from those systems are fused to yield the single best
result. Besides, we also explore various ways of incorporating the pre-trained
wav2vec2.0 models in the encoder-decoder based CTC/Attention ASR system [8].

The rest of the paper is organized as follows. Sec.2 introduces the OpenASR21
challenge briefly, and Sec.3 presents our systems for the Constrained training
condition. We describe our ASR systems for the Constrained-plus training
condition in Sec.4, together with our post evaluation experiments with the
unsupervised pre-trained models. Finally, conclusions are drawn in Sec.5.

##  2 The OpenASR21 Challenge

The OpenASR21 challenge is an open challenge to evaluate the ASR performance
under low-resource language constraints. There are 15 languages in total,
three of which have additional evaluation datasets scored using Case-Sensitive
Criteria (CSS). Basic information of the 15 languages is shown in Tab. 1. For
each language, three different training conditions are set: Constrained,
Constrained-plus, and Unconstrained. The main difference among the three
conditions is the restriction on speech data and models. For the Constrained
training condition, the only permissible resource is a 10-hour dataset
provided by the National Institute of Standards and Technology (NIST) for the
language being processed. For Constrained-plus, unsupervised pre-trained
models are allowed as well. There is no limitation for Unconstrained condition
as long as the data are publicly available. Text data are not restricted for
all three conditions.

Table 1: The 15 languages in OpenASR21 challenge.  Language | Code | Dataset | Scoring
---|---|---|---
Amharic | am | BABEL | CIS
Cantonese | yue | BABEL | CIS
Farsi | fa | MATERIAL | CIS
Georgian | ka | BABEL | CIS
Guarani | gu | BABEL | CIS
Javanese | jv | BABEL | CIS
Kazakh | kk | BABEL | CIS&CSS
Kurmanji-Kurdish | ku | BABEL | CIS
Mongolian | mn | BABEL | CIS
Pashto | ps | BABEL | CIS
Somali | so | MATERIAL | CIS
Swahili | sw | BABEL | CIS&CSS
Tagalog | tl | BABEL | CIS&CSS
Tamil | ta | BABEL | CIS
Vietnamese | vi | BABEL | CIS

For Case-Insensitive Scoring (CIS) evaluation, the provided Build datasets
consist of only one data genre, conversational telephone speech, which has two
separate channels. The CSS datasets also include news broadcast and topical
broadcast. The sampling rate of speech data is 8 kHz, 44.1 kHz or 48 kHz.

##  3 The Constrained Training Condition

We build hybrid Deep Neural Networks-Hidden Markov Model (DNN-HMM) ASR systems
for the Constrained training condition with Kaldi speech recognition toolkit
[9]. The important parts of our system, including pronunciation lexicons,
acoustic model, data augmentation and pre-and-post processing, are described
one by one in the following sections.

###  3.1 Pronunciation Lexicon

It is essential to deal with OOV problem for low-resource languages. We use
the pronunciation lexicons in BABEL program for the languages from BABEL
program. In addition, the text data in Babel datasets are utilized for
training language model as shown in Tab. 2. For datasets from MATERIAL program
and datasets using CSS evaluation criterion, OOV problems are more serious.
Therefore, we adopt G2P models to generate approximate pronunciations for the
OOV words in these datasets, and then append the generated items to the
original lexicons. The G2P models are from LanguageNet [10] which provides
Phonetisaurus [11] FST G2P models for around 150 languages.

Furthermore, we perform lexicon expansion for all the 18 datasets, which
generates new items composed of word, corresponding pronunciation and
probability. In the process, we first produce 12 million phone sequences based
on a 3-gram language model trained with items in the original lexicon. After
eliminating the phone sequences already existing in the lexicon, we keep 1
million items with highest probabilities to obtain the corresponding spellings
by G2P methods. Finally, we expand the original lexicon with 1 million new
entries.

###  3.2 Acoustic Model

The CNN-TDNN-F network [12] is adopted as the neural network acoustic model,
which combines Convolution Neural Network (CNN), Factored Time Delay Neural
Network (TDNN-F) [13]. We also employ CNN-TDNN-F-A architecture [1] to build
multiple systems with diversity, which introduces self-attention mechanism
[14, 15] to CNN-TDNN-F network. For both two networks, there are 6 convolution
blocks at the beginning and 11 TDNN-F blocks in total with the hidden
dimension of $768$ and a bottleneck dimension of $160$. As for CNN-TDNN-F-A
architecture, the self-attention block is the third layer from the output end,
which is composed of an affine component, an attention nonlinearity component,
and a ReLU nonlinearity component followed by batch normalization [15]. The
acoustic models are trained with high-resolution MFCC features concatenated
with i-vector features [16] for speaker adaptation. For Cantonese, Georgian,
Tagalog and Tamil, pitch features are appended as well. The acoustic models
are trained with chain/chain2 component of Kaldi toolkit which adopts LF-MMI
criterion. We resample all speech data to 8 kHz.

Table 2: The extra available data from IARPA BABEL program.  Lang | Version | #words | Dur
---|---|---|---
am | IARPA-babel307b-v1.0b-build | 281k | 43h
yue | IARPA-babel101b-v0.4c-build | 892k | 141h
gu | IARPA-babel305b-v1.0c-build | 311k | 42h
jv | IARPA-babel402b-v1.0b-build | 309k | 45h
kk | IARPA-babel302b-v1.0a-build | 270k | 39h
ku | IARPA-babel205b-v1.0a-build | 346k | 41h
mn | IARPA-babel401b-v2.0b-build | 403k | 46h
ps | IARPA-babel104b-v0.bY-build | 888k | 78h
sw | IARPA-babel202b-v1.0d-build | 287k | 44h
tl | IARPA-babel106-v0.2g-build | 595k | 85h
ta | IARPA-babel204b-v1.1b-build | 486k | 69h
vi | IARPA-babel107b-v0.7-build | 923k | 88h

###  3.3 Data Augmentation

We apply several techniques for data augmentation at the same time to enhance
the robustness of our ASR systems, including speed and volume perturbation
[4], SpecAugment [5] and adding noise. By speed perturbation, we enlarge the
data quantity by three times with speed factors of 0.9, 1.0 and 1.1. We
quadruple the amount of data with MUSAN dataset [6] by different noise type:
babble, music, noise and reverberation. For reverberation, we add simulated
Room Impulse Responses (RIRs) of small and middle room at the same time [17].

###  3.4 Pre-and-Post Processes

For the evaluation period, Speech Activity Detection (SAD) is a necessary
operation to segment the audio appropriately so that we can decrease loss of
useful speech clips and improve the decoding efficiency and accuracy. The SAD
system follows work proposed in [18, 19]. In our SAD component, we combine a
Recurrent Neural Network (RNN) and a Convolutional RNN (CRNN) to make the
system more robust. Additionally, we also utilize an subband Order Statistic
Filters (OSFs) based VAD system [20] to increase diversity of the fused
system.

The first-pass decoding is based on Weighted Finite-State Transducers (WFST),
incorporating an N-gram language model. We also employ lattice rescoring with
NN-based Language Model (LM), composed by TDNN-LSTM networks [3]. Furthermore,
we perform backward rescoring with the same network trained on the reversed
text. For system fusion, we adopt the lattice combination method [21], which
uses minimum Bayes risk decoding to combine results from the same SAD system.
Then, we merge the combined results with the help of ROVER method [22].

###  3.5 Results on DEV

We evaluate our systems with the 10h development set. The results are shown in
Tab. 3 by Word Error Rate (WER). We show 4 single systems’ results and their
fusion results for each language and scoring criteria.

Table 3: The results (WER) of DEV under the Constrained training condition. $S1$ and $S2$ employ the CNN-TDNN-F and CNN-TDNN-F-A respectively. $S3$ performs RNNLM rescoring based on $S2$. $S4$ is the augmented with MUSAN based on $S1$ or $S2$. $Fusion$ is obtained with $S1$-$S4$ by lattice combination.  Lang | Case | S1 | S2 | S3 | S4 | Fusion
---|---|---|---|---|---|---
am | CIS | 37.3 | 37.7 | 38.5 | 40.9 | 35.3
yue | CIS | 47.3 | 47.8 | 46.6 | 48.7 | 45.0
fa | CIS | 54.4 | 54.2 | 55.0 | 58.0 | 52.6
ka | CIS | 42.8 | 42.3 | 44.3 | 47.2 | 40.8
gu | CIS | 41.0 | 41.8 | 41.6 | 44.9 | 38.9
jv | CIS | 54.0 | 54.5 | 54.3 | 57.6 | 51.9
kk | CIS | 46.4 | 47.9 | 46.4 | 52.9 | 44.2
CSS | 29.9 | 30.6 | 30.4 | 33.7 | 27.6
ku | CIS | 66.1 | 66.3 | 65.6 | 70.5 | 63.8
mn | CIS | 48.3 | 48.6 | 48.4 | 52.5 | 45.5
ps | CIS | 47.0 | 47.2 | 46.3 | 51.2 | 44.1
so | CIS | 55.9 | 56.6 | 56.4 | 59.7 | 53.8
sw | CIS | 34.4 | 34.8 | 34.5 | 37.9 | 32.2
CSS | 27.5 | 28.0 | 29.8 | 29.3 | 26.4
tl | CIS | 42.2 | 42.9 | 41.9 | 44.1 | 40.0
CSS | 33.4 | 33.3 | 30.9 | 34.1 | 28.5
ta | CIS | 62.0 | 62.9 | 62.9 | 65.4 | 60.2
vi | CIS | 47.4 | 47.0 | 46.4 | 50.3 | 44.2
Ave | – | 45.4 | 45.8 | 45.6 | 48.8 | 43.1

##  4 The Constrained-plus Training Condition

The Constrained-plus training condition follows the same training data
restrictions as the Constrained training condition, but additionally allows
publicly available speech pre-trained models. Recently the self-supervised
learning frameworks, such as wav2vec2.0 [7] and HuBERT [23], have achieved
promising progress in speech recognition. The self-supervised models are able
to transfer across languages and be applicable with small amount of labeled
data [24, 25]. We build our pipeline based on the wav2vec2.0 framework [7] for
its outstanding performances on ASR downstream task. The wav2vec2.0 model is
composed of a multi-layer convolutional feature encoder, a context network
with Transformer architecture, and a quantization module to discretize the
output of the feature encoder to a finite set of speech representations via
product quantization [7]. In our ASR systems under the Constrained-plus
training condition, we finetune the pre-trained model in two stages to take
advantage of unlabeled speech data from target language. In addition, we
further explore the application of wav2vec2.0 pre-trained model in End-to-End
(E2E) ASR systems. Details are described in the following sections
respectively.

Table 4: The results (WER) of DEV under the Constrained-plus training condition, $yue$ is shown by Char Error Rate (CER). $FT$ is to finetune the XLSR-53 model directly while $FT2$ continues training the XLSR-53 model first, which is introduced as the 2-stage finetuning method in Sec.4.1. (w/ oov) means the texts used during finetuning do not remove OOVs. $Frontend$ refers to the CTC/Attention ASR system introduced in Sec.4.2.  Lang | Case | FT | FT2 | FT2(w/ oov) | Frontend
---|---|---|---|---|---
4-gram LM | w/o LM
am | CIS | 44.0 | 40.8 | 38.6 | 44.9 | 43.4
yue | CIS | 37.0 | 37.1 | 36.6 | 36.7 | 37.1
fa | CIS | 46.3 | 45.9 | 47.4 | 49.1 | 43.5
ka | CIS | 44.7 | 45.1 | 47.3 | 50.7 | 45.5
gu | CIS | 46.2 | 42.5 | 41.3 | 46.4 | 43.5
jv | CIS | 53.6 | 51.0 | 49.8 | 56.9 | 53.0
kk | CIS | 46.5 | 43.9 | 42.1 | 48.3 | 45.6
CSS | 50.9 | 48.8 | 38.3 | – | –
ku | CIS | 62.5 | 60.1 | 59.5 | 66.9 | 64.4
mn | CIS | 47.9 | 44.7 | 43.9 | 51.4 | 46.7
ps | CIS | 45.2 | 41.8 | 37.9 | 47.1 | 44.6
so | CIS | 53.9 | 53.1 | 55.8 | 62.6 | 56.7
sw | CIS | 40.5 | 36.8 | 34.6 | 39.1 | 37.4
CSS | 48.6 | 45.4 | 38.1 | – | –
tl | CIS | 45.0 | 41.5 | 39.5 | 46.8 | 43.3
CSS | 43.6 | 42.1 | 34.3 | – | –
ta | CIS | 64.3 | 61.8 | 60.0 | 69.1 | 64.6
vi | CIS | 40.2 | 35.9 | 36.6 | 44.0 | 39.3
Average | 47.8 | 45.5 | 43.4 | 50.7 | 47.2

###  4.1 Finetuning CTC Model

For the Constrained-plus training condition, we conduct our experiments on the
fairseq toolkit555https://github.com/pytorch/fairseq. Since there are 15
different languages in the challenge, we utilize an open-source multilingual
pre-trained model [25]
XLSR-53666https://dl.fbaipublicfiles.com/fairseq/wav2vec/xlsr_53_56k.pt as
backbone, which is trained with 56k hours audio data in 53 different languages
from 3 datasets (Multilingual LibriSpeech [26], CommonVoice [27], BABEL [28]).
In the basic workflow, we finetune the pre-trained XLSR-53 model with the
labeled 10h data of the target language by adding a linear classifier on top
of the model to optimize the CTC loss [29]. The provided 10h development set
is used to validate during finetuning. For the first 10k updates only the
output classifier is trained, after which the Transformer is also updated. The
feature encoder is not trained during finetuning. For model compatibility, all
the audio data used are up or down sampled to 16kHz.

We improve the basic finetuning methods in our systems. Multilingual pre-
trained models are the experts in acquiring universal characteristics of
multiple languages. However, they may not contain enough unique information
for a specific language since the language identities are not utilized during
pre-training. Therefore, we further train the XLSR-53 model by self-supervised
learning with unlabeled speech of the target language before finetuning. At
this stage, the whole model is optimized by contrastive loss augmented by a
codebook diversity loss [7]. Next, we finetune the language-specific XLSR-53
model with the labeled 10h data as the basic finetuning method. The speech
data used for unsupervised training are from the Build dataset in BABEL
program of the corresponding language being processed, which have been shown
in Tab. 2. For the three languages not listed, Farsi, Georgian and Somali, we
utilize the same 10h data in two stages.

We put character as tokenization unit for languages with romanized spelling,
such as Tagalog and Swahili, while grapheme is utilized for languages without
romanized spelling. The token set also includes a word boundary token. For CIS
datasets, we transfer all uppercase letters into lowercase letters while
keeping the originl case for CSS datasets. Besides, we deal with the
transcript texts in two ways after removing all the speech aspects such as
mispronunciations and non-speech aspects. One is keeping all words in
transcripts while the other is filtering out OOV words with the provided
pronunciation lexicons.

For experiment settings of unsupervised training, the masking probability of a
time-step is 65%. We optimize with Adam optimizer, warming up the learning
rate for the first 32000 updates to a peak of $1$×$10^{-3}$ and then linearly
decaying it. The maximum number of updates is 100k. During finetuning, we mask
75% of the time-steps and 25% of the channels. The learning rate follows a
tri-state rate schedule where it is warmed up for the first 10% of updates,
held at $1$×$10^{-3}$ for the next 40% and then linearly decayed for the
remainder.

The results on DEV are displayed in Table 4. We decode with a 4-gram language
model, of which the weight is set to 1 with beam 5. By comparing the average
results, the proposed 2-stage finetuning method is validated to be effective,
which improves the performance by 9.2% relatively compared with basic
finetuning. Note that the XLSR-53 model has already utilized the same speech
data from BABEL of 8 languages ($yue$, $kk$, $ku$, $ps$, $sw$, $tl$, $ta$,
$vi$) out of the 15 languages during pre-training, which verifies that the
self-training stage with speech data of target language is necessary and it is
critical to apply the multilingual model to a specific language. For most
languages, the text processing method of keeping OOVs benefits the
performances. For evaluations of the challenge, we perform SAD on the datasets
with two different methods mentioned in Sec.3.4. The final result is fused
from results of different segment methods and different fine-tuning ways by
ROVER [22]. We do not present the detailed fusion results due to the limited
space.

###  4.2 Post Evaluations with wav2vec2.0 Pre-trained Model

Recently the E2E systems based on Transformer or Conformer architecture [30,
31] have achieved promising performances in speech recognition. However, they
do not benefit low-resource languages directly owing to the data-driven
property of E2E systems. Therefore, it is reasonable to combine the self-
supervised model based on large quantity of speech data and data-hungry E2E
ASR system for low-resource languages. We treat the pre-trained wav2vec2.0
model as a frontend module to take advantage of the effective representations
extracted from the pre-trained model. To further improve the ASR performance
on low-resource languages, we adopt the CTC/Attention hybrid E2E system
structure [8] enhanced by wav2vec2.0 model under the Constrained-plus training
condition, which is illustrated in Figure 1. We utilize the 2-stage finetuned
XLSR-53 model for the target language as the system frontend.

Figure 1: ASR system based on XLSR-53 pre-trained model

In CTC/Attention hybrid system, we adopt Unigram method [32] for tokenization
with a vocabulary size of 500, except for Cantonese, which is over 2,000
decided by the number of characters appeared. 500 is a suitable size for most
languages according to our ablation experiments. For system architecture, the
encoder is composed by 12 Conformer blocks, and the decoder is composed by 6
Transformer blocks. There are 8 attention heads with 512 dimensions and the
number of hidden-units in feed forward layer are 2048. The 1024-dimension
embeddings extracted from pre-trained model are first projected to
80-dimension representations before input to encoder. For optimization
objective, the weight of CTC loss is set to 0.3. The experiments are performed
with ESPnet [33].

We perform experiments on the 15 languages as shown in Table 4. The right two
columns compare the performances of 2-stage finetuned wav2vev2.0 system and
CTC/Attention E2E system with wav2vec2.0 frontend, both of which are obtained
without language model in order to eliminate the interference of different
language models. The CTC/Attention hybrid system obtains lower WER by 6.9% on
average. Therefore, it is more competitive to apply the finetuned wav2vec2.0
model to the Conformer-based E2E ASR system as frontend.

##  5 Conclusion

We describe the THUEE ASR systems of 15 low-resource languages in detail under
both the Constrained and Constrained-plus training condition in OpenASR21
challenge. We have achieved outstanding results , especially for Constrain-
plus condition. The effective combination of classical methods under
Constrained condition is a valuable case for the researchers on low-resource
languages. For Constrained-plus condition, we obtain remarkable performances
with the 2-stage finetuning methods based on wav2vec2.0 pre-trained model. In
addition, we further explore the application of pre-trained model to
CTC/Attention hybrid ASR system, which verifies the effectiveness of the
frontend utilization.

## References

  * [1] J. Zhao, G. Shi, G.-B. Wang, and W.-Q. Zhang, “Automatic speech recognition for low-resource languages: The THUEE systems for the iarpa openasr20 evaluation,” in _Proc. IEEE Workshop Autom. Speech Recognition Understanding (ASRU)_ , 2021, pp. 335–341.
  * [2] J. Trmal, G. Chen, D. Povey, S. Khudanpur, P. Ghahremani, X. Zhang, V. Manohar, C. Liu, A. Jansen, D. Klakow _et al._ , “A keyword search system using open source software,” in _2014 IEEE Spoken Language Technology Workshop (SLT)_. IEEE, 2014, pp. 530–535.
  * [3] X. Liu, Y. Wang, X. Chen, M. J. Gales, and P. C. Woodland, “Efficient lattice rescoring using recurrent neural network language models,” in _Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP)_ , 2014, pp. 4908–4912.
  * [4] T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, “Audio augmentation for speech recognition,” in _Proc. INTERSPEECH_ , 2015, pp. 3586–3589.
  * [5] D. S. Park, W. Chan, Y. Zhang _et al._ , “SpecAugment: A simple data augmentation method for automatic speech recognition,” in _Proc. INTERSPEECH_ , 2019, pp. 2613–2617.
  * [6] D. Snyder, G. Chen, and D. Povey, “MUSAN: A music, speech, and noise corpus,” _arXiv:1510.08484_ , 2015.
  * [7] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in _Proc. 34th Conf. Neural Inf. Process. Syst. ((NeurIPS)_ , 2020, pp. 1–12.
  * [8] S. Watanabe, T. Hori, S. Kim, J. R. Hershey, and T. Hayashi, “Hybrid ctc/attention architecture for end-to-end speech recognition,” _IEEE J. Sel. Top. Signal Process._ , vol. 11, no. 8, pp. 1240–1253, 2017.
  * [9] P. Daniel, G. Arnab, B. Gilles, B. Lukáš, G. Ondrej, G. Nagendra _et al._ , “The Kaldi speech recognition toolkit,” _Proc. IEEE Workshop Autom. Speech Recognition Understanding (ASRU)_ , 2011.
  * [10] M. Hasegawa-Johnson, L. Rolston, C. Goudeseune, G.-A. Levow, and K. Kirchhoff, “Grapheme-to-phoneme transduction for cross-language asr,” in _Proc. Int. Conf. Statistical Language Speech Process. (SLSP)_ , 2020, pp. 3–19.
  * [11] AdolfVonKleist. (2021) Phonetisaurus. [Online]. Available: https://github.com/AdolfVonKleist/Phonetisaurus
  * [12] A. Georgescu, H. Cucu, and C. Burileanu, “Kaldi-based DNN architectures for speech recognition in romanian,” in _Proc. Int. Conf. Speech Technol. Human-Computer Dialogue (SpeD)_ , 2019.
  * [13] D. Povey, G. Cheng, Y. Wang, K. Li, H. Xu, M. Yarmohammadi, and S. Khudanpur, “Semi-orthogonal low-rank matrix factorization for deep neural networks.” in _Proc. INTERSPEECH_ , 2018, pp. 3743–3747.
  * [14] A. Vaswani, N. Shazeer, N. Parmar _et al._ , “Attention is all you need,” in _Proc. 31st Conf. Neural Informat. Process. Syst. (NIPS)_ , 2017, pp. 5998–6008.
  * [15] D. Povey, H. Hadian, P. Ghahremani _et al._ , “A time-restricted self-attention layer for ASR,” in _Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP)_ , 2018, pp. 5874–5878.
  * [16] G. Saon, H. Soltau, D. Nahamoo, and M. Picheny, “Speaker adaptation of neural network acoustic models using i-vectors,” in _Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)_ , 2013, pp. 55–59.
  * [17] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” in _Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP)_ , 2017, pp. 5220–5224.
  * [18] G.-B. Wang and W.-Q. Zhang, “A fusion model for robust voice activity detection,” in _Proc. IEEE Internat. Symposium Signal Process. Informat. Technol. (ISSPIT)_ , 2019, pp. 1–5.
  * [19] G.-X. Shi, W.-Q. Zhang, G.-B. Wang, J. Zhao, S.-Z. Chai, and Z.-Y. Zhao, “Timestamp-aligning and keyword-biasing end-to-end ASR front-end for a KWS system,” _EURASIP J. Audio, Speech, Music Process._ , vol. 2021, no. 1, pp. 1–14, 2021.
  * [20] J. Ramírez, J. C. Segura, C. Benítez, A. De la Torre, and A. Rubio, “An effective subband osf-based vad with noise reduction for robust speech recognition,” _IEEE Trans. Speech Audio Process._ , vol. 13, no. 6, pp. 1119–1129, 2005.
  * [21] H. Xu, D. Povey, L. Mangu, and Z. Jie, “An improved consensus-like method for minimum bayes risk decoding and lattice combination,” in _Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP)_ , 2010, pp. 4938–4941.
  * [22] J. G. Fiscus, “A post-processing system to yield reduced word error rates: Recognizer output voting error reduction (rover),” in _Proc. IEEE Workshop Autom. Speech Recognition Understanding (ASRU)_ , 1997, pp. 347–354.
  * [23] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” _arXiv:2106.07447_ , 2021.
  * [24] M. Rivière, A. Joulin, P.-E. Mazaré, and E. Dupoux, “Unsupervised pretraining transfers well across languages,” in _Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP)_ , 2020, pp. 7414–7418.
  * [25] A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, “Unsupervised cross-lingual representation learning for speech recognition,” _arXiv:2006.13979_ , 2020.
  * [26] V. Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, “MLS: A largescale multilingual dataset for speech research,” in _Proc. INTERSPEECH_ , 2020, pp. 2757–2761.
  * [27] R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” in _Proc. Language Resources Evaluation Conf._ , 2020, pp. 4218–4222.
  * [28] M. J. F. Gales, K. M. Knill, A. Ragni, and S. P. Rath, “Speech recognition and keyword spotting for low-resource languages: Babel project research at CUED,” in _Proc. Workshop Spoken Language Technol. Under-resourced Languages_ , 2014, pp. 16–23.
  * [29] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in _Proc. Int. Conf. Mach. Learn. (ICML)_ , 2006, pp. 369–376.
  * [30] L. Dong, S. Xu, and B. Xu, “Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition,” in _Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP)_ , 2018, pp. 5884–5888.
  * [31] A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in _Proc. INTERSPEECH_ , 2020, pp. 5036–5040.
  * [32] T. Kudo, “Subword regularization: Improving neural network translation models with multiple subword candidates,” in _Proc. Ann. Meeting Association Comput. Linguistics (ACL)_ , 2018, pp. 66–75.
  * [33] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. Enrique Yalta Soplin, J. Heymann, M. Wiesner, N. Chen, A. Renduchintala, and T. Ochiai, “ESPnet: End-to-end speech processing toolkit,” in _Proc. INTERSPEECH_ , 2018, pp. 2207–2211.


# Efficiently Leveraging Linguistic Priors for Scene Text Spotting

Nguyen Nguyen
University of Rochester
Rochester, NY, USA
<EMAIL_ADDRESS>
Yapeng Tian
University of Texas at Dallas
Richardson, TX, USA
<EMAIL_ADDRESS>
Chenliang Xu
University of Rochester
Rochester, NY, USA
<EMAIL_ADDRESS>

###### Abstract

Incorporating linguistic knowledge can improve scene text recognition, but it
is questionable whether the same holds for scene text spotting, which
typically involves text detection and recognition. This paper proposes a
method that leverages linguistic knowledge from a large text corpus to replace
the traditional one-hot encoding used in auto-regressive scene text spotting
and recognition models. This allows the model to capture the relationship
between characters in the same word. Additionally, we introduce a technique to
generate text distributions that align well with scene text datasets, removing
the need for in-domain fine-tuning. As a result, the newly created text
distributions are more informative than pure one-hot encoding, leading to
improved spotting and recognition performance. Our method is simple and
efficient, and it can easily be integrated into existing auto-regressive-based
approaches. Experimental results show that our method not only improves
recognition accuracy but also enables more accurate localization of words. It
significantly improves both state-of-the-art scene text spotting and
recognition pipelines, achieving state-of-the-art results on several
benchmarks.

Figure 1: Comparison of detection results with ( green shaded) and without (
red shaded) language knowledge prior guidance. Language prior is not only
helpful for text recognition but also for text detection. 111This work was
supported in part by NSF under Grant 1909912, NIH under Grant R01EY034562,
DARPA under Contract HR00112220003, and the Center of Excellence in Data
Science, an Empire State Development-designated Center of Excellence. The
content of the information does not necessarily reflect the position of the
Government, and no official endorsement should be inferred.

##  1 Introduction

Text detection and recognition in natural scenes is a research area of
considerable importance, with diverse practical applications ranging from
aiding the visually impaired Jabnoun et al. (2017) to facilitating robot
navigation Liu et al. (2021) and enabling mapping, and localization Greenhalgh
and Mirmehdi (2015). However, many text instances in natural settings exhibit
inherent ambiguity stemming from aesthetic variations, environmental
deterioration, or poor illumination conditions. This ambiguity can often be
partly reduced by considering a list of lexicons or a dictionary Nguyen et al.
(2021). However, this usually increases the complexity of models.

An end-to-end text spotting system typically involves two main steps: text
detection and text recognition. Over the years, researchers have devoted
considerable effort to advancing the state of the arts in both text detection
Baek et al. (2019); Tang et al. (2022); Long et al. (2018); Zhang et al.
(2020); Zhu et al. (2021); Liao et al. (2020a); Liu et al. (2019a); Wang et
al. (2019a) and text recognition Xie et al. (2022); Bautista and Atienza
(2022); Wang et al. (2022); Zhao et al. (2022); Liu et al. (2022a); Bhunia et
al. (2021). Despite advancements, current scene text spotting models do not
match human reading capabilities, especially in recognizing distorted or
blurry characters. This remarkable feat is due to the fact that linguistic
knowledge provides a powerful prior that can help disambiguate text. Existing
autoregressive text recognition models Shi et al. (2016a); Wang and Hu (2017);
Cheng et al. (2018); Chng et al. (2019) can leverage some of the linguistic
structure from the training data in the decoder component, where the model
predicts the next character by using previous character predictions as inputs.
However, these approaches have yet to fully use linguistic knowledge for the
following reasons. Firstly, existing scene text datasets are limited, making
effective linguistic knowledge capture challenging. Secondly, current models
employ basic one-hot vector encoding, ignoring the uncertainty or relationship
between characters. Furthermore, autoregressive models predict character
distribution based on previous characters, contrasting traditional methods
that align this with one-hot, which has no connections to other characters.
Thus, better modeling of linguistic knowledge is needed in pipelines that rely
solely on autoregressive models.

During inference, text detection models often fail if the distance between two
characters of the same word is too far (detecting them as two different
words), or if the distance between two different words is too close (detecting
them as one word), or if the word length is too long (missing some
characters). Traditional methods use a one-hot label, which can encourage
better learning in detection when considering recognition as a constraint for
detection models. However, this method only looks at each character
independently, without considering the context of the entire word. This limits
the ability of the detection model to look at the entire word in a
comprehensive way. Linguistic priors allow detection models to look at the
entire word, which helps to improve detection performance. For example, we
know that words are typically made up of multiple characters, and that the
characters in a word are typically spaced close together. By taking linguistic
priors into account, detection models can better understand the context of the
entire word and make more accurate predictions.

In this paper, we propose a novel end-to-end scene text spotting method that
efficiently incorporates language priors from trained language models. Our
approach can also be viewed as a multimodal knowledge distillation technique,
which has yet to be fully utilized for scene text spotting. By directly
leveraging language model output as label embeddings to guide text spotting
learning, our approach provides more detailed guidance than traditional one-
hot encoding. We empirically tested our approach on multiple benchmark
datasets, including Total-Text Chng and Chan (2017), SCUT-CTW1500 Liu et al.
(2019b), and ICDAR2015 Karatzas et al. (2015), demonstrating its remarkable
effectiveness in transferring knowledge from language models to scene text
spotting models. Besides, to demonstrate the generalizability of our proposal,
we also implemented our method on a state-of-the-art scene text recognition
pipeline. Notably, our method allows for direct utilization of language model
knowledge by scene text models without requiring any fine-tuning in the
specific domain of scene text data.

Our contributions are threefold:

  * •

We demonstrate the efficacy of leveraging language knowledge derived from a
large language model to enhance the performance of both scene text spotting
and recognition models. Specifically, we illustrate how the incorporation of
language knowledge through the technique of knowledge distillation can benefit
both text detection and text recognition tasks.

  * •

We present a novel and intuitive approach for integrating language models into
the scene text spotting and recognition learning process without increasing
model complexity in training and testing. Our proposed approach has been shown
to be highly effective and is supported by empirical validation.

  * •

We introduce an innovative method that enables the utilization of pre-trained
language model representations without the need for domain-specific fine-
tuning on the scene text data, thus increasing the applicability and
efficiency of the scene text spotting and recognition model.

Figure 2: Traditional spotting pipeline (a) and proposed pipeline (b) on
training. In the traditional pipeline, models use the one-hot label directly
to guide the training for the scene text system. Our proposal replaces the
one-hot encoding by using soft distributions for every label character and
improving detection and recognition results. Besides, we proposed a method to
leverage knowledge from pretrained language models and construct the soft
distribution well-adapted to the scene text domain without finetuning language
models.

##  2 Related Works

Scene Text Auto-regression Model. Auto-regressive models are widely used to
tackle scene text recognition Su and Lu (2014); Shi et al. (2016a, b); Liu et
al. (2016); Wang and Hu (2017); Cheng et al. (2018); Chng et al. (2019). For
example, CRNN Shi et al. (2016b) applied a deep bidirectional LSTM to infer
the output probability distribution of the text. A CTC loss function removes
duplicate characters in the output text series. In addition to CRNN, SCATTER
Litman et al. (2020) sets up an attention layer to capture the intra-sequence
relationships. Charnet Liu et al. (2018) fuses character-level and word-level
encodings over the input to make the output more robust. Taking good practice
in past research, we adopt an auto-regressive method that decodes output
sequences at the character level. A canonical RNN can execute the auto-
regressive process with a unique direction from left to right.

Language-driven for Scene Text Recognition. The integration of language
knowledge into scene text recognition models has emerged as a popular research
direction Bautista and Atienza (2022); Fang et al. (2021); Da et al. (2022);
Wang et al. (2021a). For example, VisionLANWang et al. (2021a) proposed a
visual reasoning module that simultaneously captures visual and linguistic
information by masking the input image at the feature level. Similarly, ABINet
Fang et al. (2021) utilized a language model for iterative processing to
enhance recognition accuracy with each iteration. Additionally, LevOCR Da et
al. (2022) learned a refinement policy with insertion and deletion actions to
iteratively enhance recognition accuracy. However, these approaches face two
common challenges. Firstly, they increase the number of parameters in the
models, which can lead to computational inefficiencies or slower model
performance when iterative running is required. Secondly, these methods often
rely on limited exploitation of language knowledge from small-scale scene text
data without fully leveraging the vast knowledge from large-scale language
models. ABINet++, an extension of ABINet, incorporates linguistic knowledge
into scene text spotting models. However, like ABINet, this approach adds
complexity through iterative execution, thereby elevating the model’s
complexity. The utilization of language knowledge for scene text spotting
remains relatively limited. To the best of our knowledge, our work represents
the first attempt to leverage language knowledge from large language models
without increasing model complexity or additional computational requirements
during training and inference. This efficient approach can improve both text
spotting and detection.

Knowledge Distillation. Using knowledge transfer from a teacher to a student
model can enhance performance at low cost Hinton et al. (2015). Typically, the
teacher’s output probabilities guide the student model through a loss function
optimization. DeiT Touvron et al. (2021) distinguishes between soft and hard-
label distillation. Soft distillation directly uses the teacher’s predictions,
while hard-label distillation converts these to one-hot encoding. In this
work, we propose a cross-modal knowledge distillation to transfer knowledge
from text to image for scene text spotting.

##  3 Language-guided Scene Text Spotting

We introduce a novel approach to enhance in-the-wild scene text spotting by
integrating prior language knowledge. Scene text spotting locates text
instances and computes their feature maps from an image. Using an auto-
regressive model, we produce a probability map for each text instance,
representing character distributions. We then compute embeddings for text
labels and establish character prototypes. Using these embeddings and
character prototypes, we create a soft distribution for each character.
Ultimately, we obtain a distribution matrix for each word label, where each
column represents a soft distribution corresponding to one character in the
word. Subsequently, we employ these distributions as soft label distributions
for the training process.

In this section, we describe the pipeline for scene text spotting,
highlighting how the soft label distribution is created. Additionally, we
discuss our loss function. Section Autoregressive-based Scene Text Recognition
presents an in-depth description of the auto-regressive model for scene text
recognition. Furthermore, in section Character Embedding, we delve into the
topic of character embedding. Our approach for centroids estimation and soft
distribution generation will be introduced below.

###  3.1 Autoregressive-based Scene Text Recognition

Auto-regressive models generate outputs over time, using past predictions as
current inputs, described as $P(x_{t}|x_{t-1},...,x_{1})$. Traditional
training uses one-hot labels to represent characters, where a character $c$
might be shown as $[0,0,1,...,0]$. However, this method fails to capture
relationships between characters in natural language contexts. For example, it
does not represent the likelihood of character $x$ following $b$ being less
than $o$ after $b$. Although widespread in natural language, this knowledge
cannot be effectively represented using one-hot encoding. In other words,
during training, the model approximates a conditional distribution with a
probability distribution to represent the character, irrespective of the
context of the characters surrounding it.

The auto-regressive model used in training captures limited contextual
knowledge due to scarce scene text datasets. Despite pre-training with
synthetic datasets, the information learned is constrained by the dictionary
used for scene text image generation. Moreover, during the fine-tuning
process, the model tends to forget the limited knowledge captured from the
pre-training data. Additionally, the recognition step of the scene text
spotting pipeline is often implemented with lightweight models, restricting
their ability to generalize with language. To address these challenges, we
propose augmenting the scene text spotting model with a language model, which
can extract knowledge from vast amounts of text data and enhance the model’s
ability to learn image patterns and language in a more comprehensive way.

###  3.2 Character Embedding

Language models require converting text sequences into numeric embeddings
through tokenization, either at the word or sub-word level. For tasks like
scene text spotting, which prioritize learning character distributions, a
character-level pre-trained model is more effective. CANINE Clark et al.
(2022), a recent model, differs from traditional tokenizers such as Byte Pair
Encoding, WordPiece, and SentencePiece, as it employs a Transformer encoder
trained directly on Unicode characters. Although this increases sequence
length, CANINE addresses this issue with an efficient downsampling technique
before applying the deep Transformer encoder. The CANINE model consists of a
downsampling function, ENCODE, and an upsampling function, which operate on an
input sequence of character embeddings e $\in$ $R^{n\times d}$ with length $n$
and dimension $d$, resulting in the output sequence embedding
$\mathbf{Y}_{seq}$:

$\mathbf{Y_{seq}}\leftarrow\text{Up(ENCODE(DOWN(e)))}.$ (1)

To ensure that different characters do not have the same representation, the
model uses a generalized hashing approach that concatenates the
representations associated with various hash values. In our work, we utilize
the pre-trained embedding $\mathbf{Y}_{seq}$ to estimate the centroid for each
character, which is then considered the prototype for the embedding clusters.

###  3.3 Centroid Generation

To generate a probability distribution from a given representation, a matrix
multiplication operation is performed between the representation vector,
denoted by x, and a weight matrix, denoted by W, which produces a vector of
logit values. The logit vector is then fed through a softmax function,
resulting in a probability distribution vector, denoted by D, which represents
the probability of the representation belonging to each character in the
vocabulary. The weight matrix W is a prototype matrix, where each column
corresponds to a prototype of a character in the vocabulary. The similarity
between a representation and each character prototype in the latent space
needs to be calculated in order to determine the most appropriate label for
the representation. This similarity measurement allows for the selection of
the label that is most compatible with the embedding.

$D_{i}=\frac{\exp(W_{i}^{T}x_{j})}{\sum_{i=1}^{k}\exp(W_{i}^{T}x_{j})},$ (2)

where $W_{i}$ is $i^{th}$ column of prototype matrix, $x_{j}$ is the
representation of the character $j^{th}$ in the word, and $k$ is the size of
character vocabulary.

In our approach, selecting an optimal W in Equation (2) is crucial, where
columns serve as character prototypes. Using the W-matrix directly from the
language model might yield a too general probability distribution, causing
disparity between scene text data and text data utilized to train the language
model. Additionally, fine-tuning the language model with scene text data can
introduce biases, compromising the model’s generality. Furthermore, the
process of fine-tuning the language model is time-consuming. In this paper, we
propose using each prototype as the centroid of its character representation
cluster by averaging all character representations of the same cluster:

$W_{i}=\frac{1}{n}\sum_{j=1}^{n}x_{ij},$ (3)

where $x_{ij}$ is the $j^{th}$ representation of character $i^{th}$, which is
equivalent to $W_{i}$ in prototype matrix.

Figure 3: Centroid Estimation. Visualization of character embedding for 9
characters $a,b,c,d,e,f,g,h,i$. Each cluster equivalent with a character, and
black points in the center are the centroids generated by (3).

###  3.4 Soft Distribution Generation

Using the matrix W from equation 3, we derive the soft label distribution for
training from equation 2. Our method bridges the gap between the pretrained
language model and the scene text dataset without the need for fine-tuning.
The texts we use to generate the $W$ matrix have the trade-off between scene
text dataset and text from outside dataset. Basically, this is a sampling
process to approximate the real centroids of language model. Using only the
scene text dataset risks overfitting and losing general language knowledge.
Conversely, using many external words keeps the centroids close to the
original language model weights. In this work, we generate the centroids by a
subset of words from a generic dictionary Gupta et al. (2016), and the whole
word appeared in the scene text training dataset. We will discuss more about
our settings in the experiments part. The predicted distribution of the model
will be denoted by distribution P. We use KL-Divergence as the loss function
to measure the difference between the predicted distribution and our soft
label distribution:

$KL(D||P)\propto-\sum_{i=1}^{k}D_{i}\log(P_{i}).$ (4)

Finally, we sum the loss of all columns together as our final loss function:

$\mathcal{L}(x,y)=\sum_{i=1}^{l}KL(D^{(i)}||P^{(i)}).$ (5)

where $D^{(i)}$ and $P^{(i)}$ are $i^{th}$ predicted distribution and soft
label distribution, respectively. $l$ is the longest length over all words.

During the creation of the soft distribution label, we perform post-processing
to remove noise. First, we check whether the distributions can represent
mislabels by checking which character the highest probability belongs to.
Next, distributions are filtered using a threshold $T$; those with a label
probability greater than or equal to $T$ are retained. For those that don’t
meet the threshold, a predefined distribution is used. In positions with a
one-hot representation of 1, we set the value to $T$, and all other positions
are set to $P_{i}$:

$P_{i}=\frac{1-T}{k-1},$ (6)

where $T$ is the probability threshold we set for the label, $P_{i}$ is the
probability value of the characters that are not label, and $k$ is the size of
the character vocabulary.

The parameter $T$ greatly influences generated distributions. A low $T$ can
yield overly flat distributions with high entropy, while a high $T$ may force
shifts to non-linguistic-based ones. We determine $T$ using text labels. A
0.85 threshold effectively curbs excessive entropy while retaining language
uncertainty. With this threshold, the error rate across datasets was
satisfactory ($\leq$0.8%), and any errors were post-processed. This post-
processing is done once before training, removing the need for recalculations
later. Our approach is straightforward, with the potential for future
improvements in threshold selection.

###  3.5 Implementation Details

In our scene text spotting approach, we used the CANINE Clark et al. (2022)
model to extract character representations from the dictionary. We then
averaged embeddings of the same class from a vocabulary combining the dataset
and 30k random words from a general dictionary Gupta et al. (2016) to produce
centroids. Details about the dictionary used in scene text recognition are
shown in Scene text recognition experiment section. Soft distributions were
derived from these embeddings and centroids, which we later refined. A
threshold $T$ in Eq.(6) of 0.85 was applied to modify soft distribution
probabilities. In Mask TextSpotterv3, we only applied our method to its
spatial attention module, an auto-regressive-based model, for consistent
comparison. Adam optimizer Kingma and Ba (2014) was used for optimization.

##  4 Experiments

This section describes our method effectiveness on both scene text spotting
(TotalText Chng and Chan (2017), SCUT-CTW1500 Liu et al. (2019b), and
ICDAR2015 Karatzas et al. (2015)) and scene text recognition (IIIT5k Mishra et
al. (2009), SVT Wang et al. (2011), ICDAR2013 Karatzas et al. (2013), SVTP
Phan et al. (2013), ICDAR2015 Karatzas et al. (2015), CUTE80 Risnumawan et al.
(2014)) datasets. Besides, we will also describe experiment settings and
datasets used for both pretraining and finetuning. Figure 4 shows how our
proposed method has improved baselines.

###  4.1 Scene Text Spotting Experiments

In our approach, modifying the underlying model architecture is unnecessary,
making it non-mandatory to re-train the model with synthetic datasets. This
implies that the existing pretrained model available in the baseline
repositories can be employed. This aspect is especially important because re-
training using extensive synthetic datasets often necessitates substantial
computational resources and extended training durations. Empirical evidence
suggests that our methodology substantially enhances the baseline by directly
fine-tuning on target datasets, eliminating the need for re-train with the
synthetic dataset. Furthermore, to investigate the feasibility of integrating
linguistic insights during the pretraining phase, we re-trained ABCNetV2 using
soft linguistic labels on Curved Synthetic, TotalText Chng and Chan (2017),
ICDAR 15 Karatzas et al. (2015), ICDAR 13 Karatzas et al. (2013), MLT-2017
Nayef et al. (2017), and TextOCR Singh et al. (2021) datasets. We have termed
this variant External Dataset (ED) and reported its outcomes with the results
procured from direct fine-tuning on the provided checkpoints.

In our scene text spotting experiments, first, we removed all words present in
the test set from a generic 90k dictionary. Next, we sampled a subset of words
from this newly formed dictionary and merged it with the collection of words
found in the training set to generate centroids. This step was implemented to
ensure that the model would not unintentionally be exposed to words exclusive
to the test set that had not appeared in the training set. For each dataset,
we train a model using all the words from its training set. Subsequently, we
use a single checkpoint to evaluate the model across all dictionaries
associated with that dataset.

####  4.1.1 Experiment on Total-Text

The Total-Text dataset Chng and Chan (2017), introduced in 2017, is a
prominent benchmark for arbitrarily shaped scene text. Consisting of 1,555
photos (1,255 for training and 300 for testing), it features images with low-
contrast backgrounds and complex texts. The dataset predominantly includes
irregular text, with each photo containing at least one curved word. Text
instances are annotated with polygons, and its extended version further
annotates each instance with ten fixed points and a recognition sequence. The
dataset exclusively contains English text.

Method | None | Full
---|---|---
Charnet Xing et al. (2019) | 66.6 | -
SwinTextSpotter Huang et al. (2022) | 74.3 | 84.1
PAN++ Wang et al. (2021b) | 68.6 | 78.6
GLASS Ronen et al. (2022) | 76.6 | 83.0
TTS Kittenplon et al. (2022) | 75.6 | 84.4
CRAFTS Baek et al. (2020) | $\textbf{78.7}^{\textbf{(*)}}$ | -
SPTS Peng et al. (2022) | 74.2 | 82.4
TESTR Zhang et al. (2022) | 73.3 | 83.9
ABINet++ Fang et al. (2022) | 77.6 | 84.5
Mask TextSpotterv3 Liao et al. (2020b) | 71.2 | 78.4
Mask TextSpotterv3+L (Ours) | 73.4 | 81.2
ABCNetv2 (github checkpoint) | 71.8 | 83.4
ABCNetv2+L (Ours) | 74.5 | 84.6
ABCNetv2+L (ED) (Ours) | 76.8 | $\textbf{87.1}^{\textbf{(*)}}$

Table 1: Scene text spotting results on Total-Text. The values shown in the
table are H-mean scores for end-to-end models. None and Full represent without
and with a dictionary, respectively; the dictionary contains all testing words
in the inference phase. ED denotes for re-train with external data. Our
methods significantly improved upon the baselines, ABCNetv2 and Mask
TextSpotterv3, surpass ABINet++ when using a full dictionary with ABCNetv2+L
(directly fine-tuning from provided checkpoint) and this improvement is even
more significant when re-train with external data (ED). (*) denotes the best
score. We report scores wherever they are available on paper or GitHub.

Table 1 shows the performance of our ABCNetv2+L and Mask TextSpotterv3+L
methods in both terms of using and not using the full dictionary during the
inference process. The full dictionary includes all words appeared in test
set. The ABCNetv2 Liu et al. (2022b) results in Table 1 are the results
reported by the authors in the paper. ABCNetv2 Liu et al. (2022b) (GitHub
checkpoint) are the results of the checkpoint published by the authors on
their GitHub. We use this implementation as the baseline for our proposal.
Checkpoint results generally produce better results than the performance
reported in the paper. We compare our results with reported and checkpoint-
generated results to make a fair comparison. It can be seen that our proposal
improved both ABCNetv2 and Mask TextSpotterv3 significantly in both scenarios
with and without full dictionary. Moreover, ABCNetv2+L with dictionary and
finetuned on provided checkpoint outperformed current state-of-the-art
ABINet++ Fang et al. (2022) by 0.1. This improvement has been increased to 2.6
when doing re-train with ED using our method.

(a) ABCNetv2: AREAT
x ABCNetv2+L: GREAT

(b) ABCNetv2: ENISIAD
x ABCNetv2+L: SPANISH

(c) ABCNetv2: KARAOKS
x ABCNetv2+L: KARAOKE

(d) ABCNetv2: HONCKONG
x ABCNetv2+L: HONGKONG

Figure 4: Qualitative results on Total-Text dataset. Our approach is more
capable of recognizing scene texts than the baseline. These outputs are
directly taken from the model when the dictionary is not used in the testing
phase.

####  4.1.2 Experiment on ICDAR 15

Images from the ICDAR 15 Karatzas et al. (2015) dataset were unintentionally
taken from the real world through Google Glass. In contrast to earlier ICDAR
13 datasets Karatzas et al. (2013), which featured cleanly captured text
horizontally centered in the images, the ICDAR 15 dataset comprises 1000
training images and 500 testing photos that present significant challenges to
text recognition algorithms. The images in the dataset feature complex
backgrounds and text may appear in any orientation and at varying resolutions.
Furthermore, the dataset is exclusively comprised of English text samples
annotated at the word level.

Table 2 shows the experimental results of our proposal compared with previous
works. For ICDAR 15, we use three scenarios to measure and evaluate the model,
including strong, weak, and generic dictionary. Correspondingly, the strong
dictionary for each image is a set of 100 words taken from the test set, in
which all the words appearing in the image are in this group of 100 words. The
weak dictionary is all the words that appear in the test set, and the generic
vocabulary is the set of words that can appear, including a dictionary of 90k
words Gupta et al. (2016). Our proposal effectively enhances the performance
of the baseline models across all three types of dictionaries without any
additional complexity. Notably, our ABCNetv2+L(ED) surpassed DeepSolo on
strong and weak dictionaries by 0.3 and 0.5, and achieved state-of-the-art
results on these dictionaries. Furthermore, while directly applying of our
method to the pretrained model showed improvement, integrating it with
external datasets (ED) in line with recent state-of-the-art elevated its
performance remarkably.

Table 2: Scene text spotting results on ICDAR 15. The values shown in the table are H-mean scores for end-to-end models. S, W, and G represent Strong, Weak, and Generic dictionaries used in the inference phase, respectively. Our method improved the baselines in all settings. Incorporating our method into ABCNetv2+L with ED, we outperformed current state-of-the-art on both Strong and Weak dictionary settings. (*) denotes the best score. Method | S | W | G
---|---|---|---
SwinTextSpotter Huang et al. (2022) | 83.9 | 77.3 | 70.5
PAN++ Wang et al. (2021b) | 82.7 | 78.2 | 69.2
GLASS Ronen et al. (2022) | 84.7 | 80.1 | 76.3
MANGO Qiao et al. (2021) | 85.4 | 80.1 | 73.9
CRAFTS Baek et al. (2020) | 83.1 | 82.1 | 74.9
TTS Kittenplon et al. (2022) | 85.2 | 81.7 | 77.4
SPTS Peng et al. (2022) | 77.5 | 70.2 | 65.8
TESTR Zhang et al. (2022) | 85.2 | 79.4 | 73.6
ABINet++ Fang et al. (2022) | 86.1 | 81.9 | 77.8
DeepSolo Ye et al. (2022) | 88.1 | 83.9 | $\textbf{79.5}^{\textbf{(*)}}$
Mask Textspotterv3 Liao et al. (2020b) | 83.3 | 78.1 | 74.2
Mask TextSpotterv3+D Nguyen et al. (2021) | 85.2 | 81.9 | 75.9
Mask TextSpotterv3+L (Ours) | 85.9 | 81.9 | 77.4
ABCNetv2 (github checkpoint) | 83.7 | 78.8 | 73.2
ABCNetv2+L (Ours) | 85.1 | 80.9 | 75.4
ABCNetv2+L(ED) (Ours) | $\textbf{88.4}^{\textbf{(*)}}$ | $\textbf{84.4}^{\textbf{(*)}}$ | 78.6
Table 3: Comparison of scene text recognition accuracy on six datasets. TargetDict denotes the list of words present in training sets of IIIT5k, SVT, IC13, IC15, SVTP, and CUTE datasets. The top-2 results are highlighted.  Method |  Training Data | Regular Text | Irregular Text
---|---|---|---
|  |  IIIT |  SVT |  IC13 |  SVTP |  IC15 |  CUTE
ASTER Shi et al. (2019) |  MJ+ST |  93.4 |  89.5 |  91.8 |  78.5 |  76.1 |  79.5
DAN Wang et al. (2019b) |  MJ+ST |  94.3 |  89.2 |  93.9 |  80.0 |  74.5 |  84.4
RobustScanner Yue et al. (2020) |  MJ+ST |  95.3 |  88.1 |  94.8 |  79.5 |  77.1 |  90.3
SAR Li et al. (2018) |  ST+MJ |  91.5 |  84.5 |  91.0 |  76.4 |  69.2 |  83.3
SEED Qiao et al. (2020a) |  ST+MJ |  93.8 |  89.6 |  92.8 |  81.4 |  80.0 |  83.6
ABINet Fang et al. (2021) |  MJ+ST+Wiki |  96.2 |  93.5 |  97.4 |  89.3 |  86.0 |  89.2
S-GTR He et al. (2021) |  ST+MJ |  95.8 |  94.1 |  96.8 |  87.9 |  84.6 |  92.3
LevOCR Da et al. (2022) |  ST+MJ |  96.6 |  92.9 |  96.9 |  88.1 |  86.4 |  91.7
SIGAT Guan et al. (2022) |  ST+MJ |  96.6 |  95.1 |  96.8 |  90.5 |  83.0 |  93.1
MGP-STR Wang et al. (2022) |  ST+MJ |  96.4 |  94.7 |  97.3 |  91.0 |  87.2 |  90.3
PARSeq Bautista and Atienza (2022) |  ST+MJ |  97.0 |  93.6 |  96.2 |  82.9 |  88.9 |  92.2
CornerTransformer Xie et al. (2022) |  ST+MJ |  95.9 |  94.6 |  96.4 |  91.5 |  86.3 |  92.0
CornerTransformer+L(Ours) |  ST+MJ |  96.4 |  95.0 |  96.9 |  91.8 |  87.0 |  92.7
CornerTransformer+L(Ours) |  ST+MJ+TargetDict |  96.7 |  95.4 |  97.4 |  92.2 |  87.8 |  93.4

####  4.1.3 Experiment on SCUT-CTW1500

The SCUT-CTW1500 Liu et al. (2019b) dataset, popular for arbitrary-shaped
scene text, differs from Total-Text by including both Chinese and English.
Annotations are at the text-line level, accommodating documents with stacked
small text segments. It has 1000 training and 500 test images. For the SCUT-
CTW1500 dataset, labels include word-level annotations and groups of text
instances. Language knowledge is vital for accurate results, capturing both
intra-word and inter-word relationships in long sequences. Table 4 shows our
ABCNetv2+L notably outperforms ABCNet2, especially with a dictionary during
inference. Our method with ED surpasses all previous state-of-the-art models
for SCUT-CTW1500 using a strong dictionary, highlighting the effectiveness and
versatility of the proposed approach in addressing the challenges associated
with localizing and recognizing scene text in the long line with arbitrary
shapes.

Table 4: Scene text spotting results on SCUT-CTW1500. The values shown in the table are H-mean scores for end-to-end models. None and Strong represent without and with a strong dictionary in the inference phase, respectively. Our method improved the baselines in all settings and achieved state-of-the-art when evaluating without a dictionary for post-processing. (*) denotes the best score. Method | None | Strong
---|---|---
TextDragon Feng et al. (2019) | 39.7 | 72.4
MANGO Qiao et al. (2021) | 58.9 | 78.7
SwinTexSpotter Huang et al. (2022) | 51.8 | 77.0
Text Perceptron Qiao et al. (2020b) | 57.0 | -
ABINet++ Fang et al. (2022) | 60.2 | 80.3
TESTR Zhang et al. (2022) | 56.0 | 81.5
DeepSolo Ye et al. (2022) | $\textbf{64.2}^{\textbf{(*)}}$ | 81.4
ABCNetv2 Liu et al. (2022b) | 57.5 | 77.2
ABCNetv2+L (Ours) | 59.1 | 78.4
ABCNetv2+L(ED) (Ours) | 61.2 | $\textbf{81.8}^{\textbf{(*)}}$

####  4.1.4 Detection Results

Table 5 presents quantitative evidence of the effectiveness of our proposed
method in enhancing not only text recognition but also text detection. In
addition, Figure 1 provides a visual representation of the qualitative
improvements in text detection achieved through our approach. It is worth
noting that unlike conventional object detection tasks, text detection is
particularly sensitive since even the slightest inaccuracies in localizing
words or including extraneous details can lead to erroneous recognition
outcomes. To address this issue, we leverage language knowledge as a guiding
signal to enhance the accuracy of our text detection model. Our experimental
results, both quantitative and qualitative, clearly demonstrate the
effectiveness of this approach in improving the overall quality of our text
detection model. In summary, our simple yet effective method can significantly
enhance text detection accuracy, which, in turn, directly influences text
recognition outcomes.

Table 5: Detection H-mean score comparison between ABCNetv2 and ABCNetv2+L. Our method improves detection performance on all three datasets. | TotalText | ICDAR15 | CTW1500
---|---|---|---
ABCNetv2 | 87.2 | 88.2 | 85.0
ABCNetv2+L (Ours) | 88.5 | 88.6 | 85.4
ABCNetv2+L (ED) (Ours) | 88.9 | 89.1 | 85.8

###  4.2 Scene Text Recognition Experiments

Our approach integrates linguistic knowledge into scene text models, enabling
comprehensive generalization for text recognition. Using the CornerTransformer
Xie et al. (2022) as a baseline, we train our model on SynthText and MJSynth
datasets, following previous state-of-the-art methods. While image acquisition
and labeling incur high costs, collecting domain-specific words is simpler.
Our method capitalizes on this by generating centroids from in-domain text. We
train the model with two centroid types: one using a 90k-word dictionary for
fair comparison, and another incorporating words from training sets (excluding
test-only words) of IIIT5k, SVT, IC13, IC15, SVTP, and CUTE. We call this list
of in-domain words is target dictionary (TargetDict).

The experimental results show in Table 3 highlight the efficacy of our
proposed method in enhancing the accuracy of the baseline model. This
improvement is consistently observed across all six benchmark datasets when
trained on MJSynth and SynthText datasets, following previous approaches.
Compared with a method also incorporating linguistic knowledge into scene text
recognition, ABINet, our method helps CornerTransformer surpass ABINet on
IIIT5k and increases the existing improvement on SVT, SVTP, ICDAR 15, CUTE
datasets more significantly. This superior performance is achieved without
adding complexity to the model in training or inference.

Additionally, generating centroids with words from TargetDict increased the
improvement by a large margin. Specially, CornerTransformer+L has improved the
baseline’s accuracy on IIT, SVT, ICDAR 13, SVTP, ICDAR 15, and CUTE by 0.8%,
0.8%, 1%, 0.7%, 1.5%, and 1.4%. Our method with TargetDict achieves state-of-
the-art on SVT, ICDAR 13, SVTP, and CUTE, and secures the second-best results
on ICDAR 15 and IIIT5k. These results underscore the model’s effectiveness in
bridging the domain gap between synthetic and real data. Importantly, our
approach leverages target domain knowledge only from the text in target
domain, minimizing the need for resource-intensive image collection and
annotation processes.

##  5 Conclusion

We proposed a novel approach to transfer language knowledge to autoregressive-
based scene text spotting models. Our approach overcomes ambiguous words using
soft distributions based on character representation. We also proposed a
process to generate soft distributions suitable for dictionaries without
finetuning. Our approach was implemented on two scene text spotting backbones:
ABCNetv2 and Mask TextSpotterv3, and one scene text recognition backbone:
CornerTransformer, showing its generality. Extensive experiments on scene text
spotting and recognition datasets can validate the effectiveness of our
approach.

## References

  * Jabnoun et al. [2017] Hanen Jabnoun, Faouzi Benzarti, and Hamid Amiri.  A new method for text detection and recognition in indoor scene for assisting blind people.  In _International Conference on Machine Vision_ , 2017.
  * Liu et al. [2021] Shuhua Liu, Huixin Xu, Qi Li, Fei Zhang, and Kun Hou.  A robot object recognition method based on scene text reading in home environments.  _Sensors (Basel, Switzerland)_ , 21, 2021.
  * Greenhalgh and Mirmehdi [2015] Jack Greenhalgh and Majid Mirmehdi.  Recognizing text-based traffic signs.  _IEEE Transactions on Intelligent Transportation Systems_ , 16:1360–1369, 2015.
  * Nguyen et al. [2021] Nguyen Le Nguyen, Thua Nguyen, Vinh Quang Tran, Minh-Triet Tran, Thanh Duc Ngo, Thien Huu Nguyen, and Minh Hoai.  Dictionary-guided scene text recognition.  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7379–7388, 2021.
  * Baek et al. [2019] Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee.  Character region awareness for text detection.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9357–9366, 2019.
  * Tang et al. [2022] Jingqun Tang, Wenming Qian, Luchuan Song, Xiena Dong, Lan Li, and Xiang Bai.  Optimal boxes: Boosting end-to-end scene text recognition by adjusting annotated bounding boxes via reinforcement learning.  In _European Conference on Computer Vision_ , pages 233–248. Springer, 2022.
  * Long et al. [2018] Shangbang Long, Jiaqiang Ruan, Wenjie Zhang, Xin He, Wenhao Wu, and Cong Yao.  Textsnake: A flexible representation for detecting text of arbitrary shapes.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 20–36, 2018.
  * Zhang et al. [2020] Shi-Xue Zhang, Xiaobin Zhu, Jie-Bo Hou, Chang Liu, Chun Yang, Hongfa Wang, and Xu-Cheng Yin.  Deep relational reasoning graph network for arbitrary shape text detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9699–9708, 2020.
  * Zhu et al. [2021] Yiqin Zhu, Jianyong Chen, Lingyu Liang, Zhanghui Kuang, Lianwen Jin, and Wayne Zhang.  Fourier contour embedding for arbitrary-shaped text detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3123–3131, 2021.
  * Liao et al. [2020a] Minghui Liao, Zhaoyi Wan, Cong Yao, Kai Chen, and Xiang Bai.  Real-time scene text detection with differentiable binarization.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 11474–11481, 2020a.
  * Liu et al. [2019a] Jingchao Liu, Xuebo Liu, Jie Sheng, Ding Liang, Xin Li, and Qingjie Liu.  Pyramid mask text detector.  _arXiv preprint arXiv:1903.11800_ , 2019a.
  * Wang et al. [2019a] Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng.  Panet: Few-shot image semantic segmentation with prototype alignment.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9197–9206, 2019a.
  * Xie et al. [2022] Xudong Xie, Ling Fu, Zhifei Zhang, Zhaowen Wang, and Xiang Bai.  Toward understanding wordart: Corner-guided transformer for scene text recognition.  In _European Conference on Computer Vision_ , 2022.
  * Bautista and Atienza [2022] Darwin Bautista and Rowel Atienza.  Scene text recognition with permuted autoregressive sequence models.  _ArXiv_ , abs/2207.06966, 2022.
  * Wang et al. [2022] P. Wang, Cheng Da, and Cong Yao.  Multi-granularity prediction for scene text recognition.  In _European Conference on Computer Vision_ , 2022.
  * Zhao et al. [2022] Liang Zhao, Zhenyao Wu, Xinyi Wu, Greg Wilsbacher, and Song Wang.  Background-insensitive scene text recognition with text semantic segmentation.  In _European Conference on Computer Vision_ , 2022.
  * Liu et al. [2022a] Chang Liu, Chun Yang, and Xu-Cheng Yin.  Open-set text recognition via character-context decoupling.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4513–4522, 2022a.
  * Bhunia et al. [2021] Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Aneeshan Sain, and Yi-Zhe Song.  Towards the unseen: Iterative text recognition by distilling from errors.  _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 14930–14939, 2021.
  * Shi et al. [2016a] Baoguang Shi, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai.  Robust scene text recognition with automatic rectification.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4168–4176, 2016a.
  * Wang and Hu [2017] Jianfeng Wang and Xiaolin Hu.  Gated recurrent convolution neural network for ocr.  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Cheng et al. [2018] Zhanzhan Cheng, Yangliu Xu, Fan Bai, Yi Niu, Shiliang Pu, and Shuigeng Zhou.  Aon: Towards arbitrarily-oriented text recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 5571–5579, 2018.
  * Chng et al. [2019] Chee Kheng Chng, Yuliang Liu, Yipeng Sun, Chun Chet Ng, Canjie Luo, Zihan Ni, ChuanMing Fang, Shuaitao Zhang, Junyu Han, Errui Ding, et al.  Icdar2019 robust reading challenge on arbitrary-shaped text-rrc-art.  In _2019 International Conference on Document Analysis and Recognition (ICDAR)_ , pages 1571–1576. IEEE, 2019.
  * Chng and Chan [2017] Chee-Kheng Chng and Chee Seng Chan.  Total-text: A comprehensive dataset for scene text detection and recognition.  _2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR)_ , 01:935–942, 2017.
  * Liu et al. [2019b] Yuliang Liu, Lianwen Jin, Shuaitao Zhang, Canjie Luo, and Sheng Zhang.  Curved scene text detection via transverse and longitudinal sequence connection.  _Pattern Recognit._ , 90:337–345, 2019b.
  * Karatzas et al. [2015] Dimosthenis Karatzas, Lluís Gómez i Bigorda, Anguelos Nicolaou, Suman K. Ghosh, Andrew D. Bagdanov, M. Iwamura, Jiri Matas, Lukás Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, Faisal Shafait, Seiichi Uchida, and Ernest Valveny.  Icdar 2015 competition on robust reading.  _2015 13th International Conference on Document Analysis and Recognition (ICDAR)_ , pages 1156–1160, 2015.
  * Su and Lu [2014] Bolan Su and Shijian Lu.  Accurate scene text recognition based on recurrent neural network.  In _Asian conference on computer vision_ , pages 35–48. Springer, 2014.
  * Shi et al. [2016b] Baoguang Shi, Xiang Bai, and Cong Yao.  An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition.  _IEEE transactions on pattern analysis and machine intelligence_ , 39(11):2298–2304, 2016b.
  * Liu et al. [2016] Wei Liu, Chaofeng Chen, Kwan-Yee K Wong, Zhizhong Su, and Junyu Han.  Star-net: a spatial attention residue network for scene text recognition.  In _BMVC_ , volume 2, page 7, 2016.
  * Litman et al. [2020] Ron Litman, Oron Anschel, Shahar Tsiper, Roee Litman, Shai Mazor, and R Manmatha.  Scatter: selective context attentional scene text recognizer.  In _proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 11962–11972, 2020.
  * Liu et al. [2018] Wei Liu, Chaofeng Chen, and Kwan-Yee Wong.  Char-net: A character-aware neural network for distorted scene text recognition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Fang et al. [2021] Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang.  Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition.  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7094–7103, 2021.
  * Da et al. [2022] Cheng Da, P. Wang, and Cong Yao.  Levenshtein ocr.  _ArXiv_ , abs/2209.03594, 2022.
  * Wang et al. [2021a] Yuxin Wang, Hongtao Xie, Shancheng Fang, Jing Wang, Shenggao Zhu, and Yongdong Zhang.  From two to one: A new scene text recognizer with visual language modeling network.  _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 14174–14183, 2021a.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2(7), 2015.
  * Touvron et al. [2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv’e J’egou.  Training data-efficient image transformers & distillation through attention.  In _ICML_ , 2021.
  * Clark et al. [2022] J. Clark, Dan Garrette, Iulia Turc, and John Wieting.  Canine: Pre-training an efficient tokenization-free encoder for language representation.  _Transactions of the Association for Computational Linguistics_ , 10:73–91, 2022.
  * Gupta et al. [2016] Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman.  Synthetic data for text localisation in natural images.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2315–2324, 2016.
  * Kingma and Ba [2014] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980, 2014.
  * Mishra et al. [2009] Anand Mishra, Alahari Karteek, and C. V. Jawahar.  Scene text recognition using higher order language priors.  In _British Machine Vision Conference_ , 2009.  URL https://api.semanticscholar.org/CorpusID:9695967.
  * Wang et al. [2011] Kai Wang, Boris Babenko, and Serge J. Belongie.  End-to-end scene text recognition.  _2011 International Conference on Computer Vision_ , pages 1457–1464, 2011.  URL https://api.semanticscholar.org/CorpusID:14136313.
  * Karatzas et al. [2013] Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, M. Iwamura, Lluís Gómez i Bigorda, Sergi Robles Mestre, Joan Mas Romeu, David Fernández Mota, Jon Almazán, and Lluís-Pere de las Heras.  Icdar 2013 robust reading competition.  _2013 12th International Conference on Document Analysis and Recognition_ , pages 1484–1493, 2013.
  * Phan et al. [2013] Trung Quy Phan, Palaiahnakote Shivakumara, Shangxuan Tian, and Chew Lim Tan.  Recognizing text with perspective distortion in natural scenes.  _2013 IEEE International Conference on Computer Vision_ , pages 569–576, 2013.  URL https://api.semanticscholar.org/CorpusID:5619635.
  * Risnumawan et al. [2014] Anhar Risnumawan, Palaiahnakote Shivakumara, Chee Seng Chan, and Chew Lim Tan.  A robust arbitrary text detection system for natural scene images.  _Expert Syst. Appl._ , 41:8027–8048, 2014.  URL https://api.semanticscholar.org/CorpusID:15559857.
  * Nayef et al. [2017] Nibal Nayef, Fei Yin, Imen Bizid, Hyunsoo Choi, Yuan Feng, Dimosthenis Karatzas, Zhenbo Luo, Umapada Pal, Christophe Rigaud, Joseph Chazalon, Wafa Khlif, Muhammad Muzzamil Luqman, Jean-Christophe Burie, Cheng-Lin Liu, and Jean-Marc Ogier.  Icdar2017 robust reading challenge on multi-lingual scene text detection and script identification - rrc-mlt.  _2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR)_ , 01:1454–1459, 2017.  URL https://api.semanticscholar.org/CorpusID:4761162.
  * Singh et al. [2021] Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojciech Galuba, and Tal Hassner.  Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text.  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8798–8808, 2021.  URL https://api.semanticscholar.org/CorpusID:234469662.
  * Xing et al. [2019] Linjie Xing, Zhi Tian, Weilin Huang, and Matthew R. Scott.  Convolutional character networks.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 9125–9135, 2019.
  * Huang et al. [2022] Mingxin Huang, Yuliang Liu, Zhenghao Peng, Chongyu Liu, Dahua Lin, Shenggao Zhu, Nicholas Jing Yuan, Kai Ding, and Lianwen Jin.  Swintextspotter: Scene text spotting via better synergy between text detection and text recognition.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4583–4593, 2022.
  * Wang et al. [2021b] Wenhai Wang, Enze Xie, Xiang Li, Xuebo Liu, Ding Liang, Yang Zhibo, Tong Lu, and Chunhua Shen.  Pan++: Towards efficient and accurate end-to-end spotting of arbitrarily-shaped text.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44:5349–5367, 2021b.
  * Ronen et al. [2022] Roi Ronen, Shahar Tsiper, Oron Anschel, Inbal Lavi, Amir Markovitz, and R. Manmatha.  Glass: Global to local attention for scene-text spotting.  In _European Conference on Computer Vision_ , 2022.
  * Kittenplon et al. [2022] Yair Kittenplon, Inbal Lavi, Sharon Fogel, Yarin Bar, R Manmatha, and Pietro Perona.  Towards weakly-supervised text spotting using a multi-task transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4604–4613, 2022.
  * Baek et al. [2020] Youngmin Baek, Seung Shin, Jeonghun Baek, Sungrae Park, Junyeop Lee, Daehyun Nam, and Hwalsuk Lee.  Character region attention for text spotting.  _ArXiv_ , abs/2007.09629, 2020.
  * Peng et al. [2022] Dezhi Peng, Xinyu Wang, Yuliang Liu, Jiaxin Zhang, Mingxin Huang, Songxuan Lai, Jing Li, Shenggao Zhu, Dahua Lin, Chunhua Shen, et al.  Spts: single-point text spotting.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 4272–4281, 2022.
  * Zhang et al. [2022] Xiang Zhang, Yongwen Su, Subarna Tripathi, and Zhuowen Tu.  Text spotting transformers.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9509–9518, 2022.
  * Fang et al. [2022] Shancheng Fang, Zhendong Mao, Hongtao Xie, Yuxin Wang, Chenggang Clarence Yan, and Yongdong Zhang.  Abinet++: Autonomous, bidirectional and iterative language modeling for scene text spotting.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45:7123–7141, 2022.
  * Liao et al. [2020b] Minghui Liao, Guan Pang, Jing Huang, Tal Hassner, and Xiang Bai.  Mask textspotter v3: Segmentation proposal network for robust scene text spotting.  _ArXiv_ , abs/2007.09482, 2020b.
  * Liu et al. [2022b] Yuliang Liu, Chunhua Shen, Lianwen Jin, Tong He, Peng Chen, Chongyu Liu, and Hao Chen.  Abcnet v2: Adaptive bezier-curve network for real-time end-to-end text spotting.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44:8048–8064, 2022b.
  * Qiao et al. [2021] Liang Qiao, Ying Chen, Zhanzhan Cheng, Yunlu Xu, Yi Niu, Shiliang Pu, and Fei Wu.  Mango: A mask attention guided one-stage scene text spotter.  In _AAAI_ , 2021.
  * Ye et al. [2022] Maoyuan Ye, Jing Zhang, Shanshan Zhao, Juhua Liu, Tongliang Liu, Bo Du, and Dacheng Tao.  Deepsolo: Let transformer decoder with explicit points solo for text spotting.  _ArXiv_ , abs/2305.19957, 2022.
  * Shi et al. [2019] Baoguang Shi, Mingkun Yang, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai.  Aster: An attentional scene text recognizer with flexible rectification.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 41:2035–2048, 2019.  URL https://api.semanticscholar.org/CorpusID:206769320.
  * Wang et al. [2019b] Tianwei Wang, Yuanzhi Zhu, Lianwen Jin, Canjie Luo, Xiaoxue Chen, Y. Wu, Qianying Wang, and Mingxiang Cai.  Decoupled attention network for text recognition.  In _AAAI Conference on Artificial Intelligence_ , 2019b.  URL https://api.semanticscholar.org/CorpusID:209444482.
  * Yue et al. [2020] Xiaoyu Yue, Zhanghui Kuang, Chenhao Lin, Hongbin Sun, and Wayne Zhang.  Robustscanner: Dynamically enhancing positional clues for robust text recognition.  In _European Conference on Computer Vision_ , 2020.  URL https://api.semanticscholar.org/CorpusID:220525922.
  * Li et al. [2018] Hui Li, Peng Wang, Chunhua Shen, and Guyu Zhang.  Show, attend and read: A simple and strong baseline for irregular text recognition.  _ArXiv_ , abs/1811.00751, 2018.  URL https://api.semanticscholar.org/CorpusID:53301402.
  * Qiao et al. [2020a] Zhi Qiao, Yu Zhou, Dongbao Yang, Yucan Zhou, and Weiping Wang.  Seed: Semantics enhanced encoder-decoder framework for scene text recognition.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 13525–13534, 2020a.  URL https://api.semanticscholar.org/CorpusID:218862702.
  * He et al. [2021] Yang He, Chen Chen, Jing Zhang, Juhua Liu, Fengxiang He, Chaoyue Wang, and Bo Du.  Visual semantics allow for textual reasoning better in scene text recognition.  In _AAAI Conference on Artificial Intelligence_ , 2021.  URL https://api.semanticscholar.org/CorpusID:245502387.
  * Guan et al. [2022] Tongkun Guan, Chaochen Gu, Jingzheng Tu, Xuehang Yang, Qi Feng, Yudi Zhao, and Wei Shen.  Self-supervised implicit glyph attention for text recognition.  2022\.  URL https://api.semanticscholar.org/CorpusID:251589365.
  * Feng et al. [2019] Wei Feng, Wenhao He, Fei Yin, Xu-Yao Zhang, and Cheng-Lin Liu.  Textdragon: An end-to-end framework for arbitrary shaped text spotting.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 9075–9084, 2019.
  * Qiao et al. [2020b] Liang Qiao, Sanli Tang, Zhanzhan Cheng, Yunlu Xu, Yi Niu, Shiliang Pu, and Fei Wu.  Text perceptron: Towards end-to-end arbitrary-shaped text spotting.  In _AAAI Conference on Artificial Intelligence_ , 2020b.