an age of $10-12$ Gyr, and the typical mass of the companion MS stars is
$\lesssim 1-2\,{\rm M}_{\odot}$, depending on the cluster properties. Even for
this case, strong interactions between a low-mass MS star and the incoming BH
at the first closest approach would have higher chances of forming BBHs than
for the other cases where two BHs meet first.

At later times when all massive stars collapse to compact objects,
interactions between stars and BHs with significantly different masses would
be more probable. If the star is significantly less massive than both BHs, the
interactions would be effectively two-body with small perturbations of the BH
orbits by the star. However, if the star was disrupted by the incoming BH as
in the BBH-forming encounters, the stellar disruption would generate bright EM
flares. Furthermore, resulting momentum kicks and gas dynamical friction would
facilitate the formation of BBHs, unless the momentum kick is given to
increase the relative kinetic energy of the BHs. This process would be most
efficient when the star and the incoming BH have comparable masses.

If the encountering binary is eccentric, the binary members would spend most
of their orbital time near apocenter, indicating that the cross-section would
be enhanced by a factor of $1+e$. Unlike the increase in $e$ in our models
when initially circular binaries are considered, the final binaries can be
circularized depending on the direction of the momentum kick associated with
close interactions between the two objects at the first closest approach. We
already demonstrated in Figures 3 and 5 that the momentum kick acts to add to
or remove the momentum of the BH in the binary, depending on whether the orbit
is initially in a prograde or retrograde direction.

##  6 Summary and Conclusions

In this work we have investigated the outcomes of three-body encounters
between a $20\,{\rm M}_{\odot}$ BH – $10\,{\rm M}_{\odot}$ star circular
binary and a $10\,{\rm M}_{\odot}$ stellar-mass BH, using a suite of
hydrodynamical simulations with the moving-mesh code AREPO. We have focused on
the formation of BBHs, the conditions required for their formation, and the EM
emission from those systems. We have considered a wide range of encounter
parameters, i.e., varying the binary size ($a\simeq 34$, 68, 101$\,{\rm
R}_{\odot}$), the impact parameter ($a/4-a$), the inclination angle
($i=0^{\circ}$, 30∘, 120∘, 150∘, and 180∘), and the phase angle
($\phi=0^{\circ}$, 45∘, 90∘ and 135∘, 180∘, 225∘, 270∘, 315∘), while we have
kept fixed the masses of the star and of the BHs.

We have categorized the encounters into two classes depending on their
outcomes. This classification is primarily determined by which types of
objects meet at the first closest approach. When the star and the incoming
single BH encounter first, their close interaction imparts a momentum kick to
the BH, resulting in a dramatic decrease in the BH’s speed. The BH is
subsequently captured by the other bystander BH, forming a BBH. In this case,
the star is frequently destroyed due to its close encounter with the BH. On
the other hand, when two BHs encounter first, either the original binary’s
orbit is simply perturbed (prograde encounters), or the originally single BH
captures the star, forming a new binary (member exchange, retrograde
encounters). Although the most frequent outcomes are BH-star binaries, a
disruption of the star and BBH formation are still possible.

The most important factors that determine the outcomes are the phase angle and
the impact parameter. As explained above, the phase angle primarily demarcates
the boundary between “BBH-forming” encounters and “non-BBH-forming”
encounters. The impact parameter on the other hand affects the strength of
interactions: for $r_{\rm p}>a$, the incoming BH interacts weakly with the
binary. As a result, the binary orbit is perturbed, or the binary members are
exchanged. For $r_{\rm p}\lesssim a$, interactions can become significant,
possibly resulting in a disruption of the star when the star and the BH meet
at the first closest approach. Although our simulations do not cover the
entire parameter space for this type of encounters, the key dynamical
processes can be extrapolated within this class of encounters to other initial
parameters, and possibly also to other astrophysical systems (e.g., three-body
encounters involving a massive black hole having a stellar companion and an
isolated BH, forming extreme mass ratio inspirals).

The close correlation between BBH formation and stellar disruption in our
systems has interesting implications for the formation channel of BBHs and EM
counterparts of their merger. We confirm that three-body encounters between a
BH-star binary and a BH can produce merging BBHs. In addition, we find that
the BH that disrupts the star in the BBH-forming encounters is promptly
surrounded by an optically and geometrically thick disk with accretion flows
towards the BH exceeding the Eddington limit. If a jet is launched from the
system, the jet luminosity would likely track the accretion rate. If the disk
remains long-lived and revives at merger, EM counterparts can be produced at
the time of the BBH merger.

Our order-of-magnitude estimate for the encounter rate suggests that this type
of encounters may be rarer than other types of three-body encounters
considered in Paper 1 (i.e., between binary BHs and single stars) and Paper 2
(i.e., between stellar-binaries and single BHs). However, given the simplified
assumptions made here, more detailed estimates should be made for these
encounters, taking their specific astrophysical environments accurately into
account.

## Acknowledgements

The authors are grateful to the referee for constructive comments and
suggestions. This research project was conducted using computational resources
(and/or scientific computing services) at the Max-Planck Computing & Data
Facility. The simulations were performed on the national supercomputer Hawk at
the High Performance Computing Center Stuttgart (HLRS) under the grant number
44232. The authors gratefully acknowledge the scientific support and HPC
resources provided by the Erlangen National High Performance Computing Center
(NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) under
the NHR project b166ea10. NHR funding is provided by federal and Bavarian
state authorities. NHR@FAU hardware is partially funded by the German Research
Foundation (DFG) – 440719683. R. Perna acknowledges support by NSF award
AST-2006839.

## Data Availability

Any data used in this analysis are available on reasonable request from the
first author.

## References

  * Almeida et al. (2017) Almeida L. A., et al., 2017, A&A, 598, A84
  * Antonini & Rasio (2016) Antonini F., Rasio F. A., 2016, ApJ, 831, 187
  * Antonini et al. (2016) Antonini F., Chatterjee S., Rodriguez C. L., M orscher M., Pattabiraman B., Kalogera V., Rasio F. A., 2016, ApJ, 816, 65
  * Arca Sedda et al. (2020) Arca Sedda M., Mapelli M., Spera M., Benacquista M., Giacobbo N., 2020, ApJ, 894, 133
  * Arca Sedda et al. (2021) Arca Sedda M., Li G., Kocsis B., 2021, A&A, 650, A189
  * Armitage & Natarajan (2002) Armitage P. J., Natarajan P., 2002, ApJ, 567, L9
  * Askar et al. (2017) Askar A., Szkudlarek M., Gondek-Rosińska D., Giersz M., Bulik T., 2017, MNRAS, 464, L36
  * Askar et al. (2018) Askar A., Arca Sedda M., Giersz M., 2018, MNRAS, 478, 1844
  * Banerjee (2018) Banerjee S., 2018, MNRAS, 473, 909
  * Binder et al. (2021) Binder B. A., et al., 2021, ApJ, 910, 74
  * D’Orazio & Duffell (2021) D’Orazio D. J., Duffell P. C., 2021, ApJ, 914, L21
  * Di Carlo et al. (2019) Di Carlo U. N., Giacobbo N., Mapelli M., Pasquato M., Spera M., Wang L., Haardt F., 2019, MNRAS, 487, 2947
  * Downing et al. (2010) Downing J. M. B., Benacquista M. J., Giersz M., Spurzem R., 2010, MNRAS, 407, 1946
  * Duffell et al. (2020) Duffell P. C., D’Orazio D., Derdzinski A., Haiman Z., MacFadyen A., Rosen A. L., Zrake J., 2020, ApJ, 901, 25
  * Eggleton (1983) Eggleton P. P., 1983, ApJ, 268, 368
  * Fragione et al. (2019) Fragione G., Grishin E., Leigh N. W. C., Perets H. B., Perna R., 2019, MNRAS, 488, 47
  * Gnedin et al. (2002) Gnedin O. Y., Zhao H., Pringle J. E., Fall S. M., Livio M., Meylan G., 2002, ApJ, 568, L23
  * Harris (2010) Harris W. E., 2010, arXiv e-prints, p. arXiv:1012.3224
  * Heggie (1975) Heggie D. C., 1975, MNRAS, 173, 729
  * Herwig (2000) Herwig F., 2000, A&A, 360, 952
  * Hut et al. (1992) Hut P., et al., 1992, PASP, 104, 981
  * Jermyn et al. (2023) Jermyn A. S., et al., 2023, ApJS, 265, 15
  * Kremer et al. (2018) Kremer K., Ye C. S., Chatterjee S., Rodriguez C. L., Rasio F. A., 2018, ApJ, 855, L15
  * Langer et al. (1983) Langer N., Fricke K. J., Sugimoto D., 1983, A&A, 126, 207
  * Langer et al. (1985) Langer N., El Eid M. F., Fricke K. J., 1985, A&A, 145, 179
  * Ledoux (1947) Ledoux W. P., 1947, ApJ, 105, 305
  * Lopez et al. (2019) Lopez Martin J., Batta A., Ramirez-Ruiz E., Martinez I., Samsing J., 2019, ApJ, 877, 56
  * Mapelli et al. (2021) Mapelli M., et al., 2021, MNRAS, 505, 339
  * Miranda et al. (2017) Miranda R., Muñoz D. J., Lai D., 2017, MNRAS, 466, 1170
  * Monaghan & Lattanzio (1985) Monaghan J. J., Lattanzio J. C., 1985, A&A, 149, 135
  * Morscher et al. (2015) Morscher M., Pattabiraman B., Rodriguez C., Rasio F. A., Umbreit S., 2015, ApJ, 800, 9
  * Muñoz et al. (2019) Muñoz D. J., Miranda R., Lai D., 2019, ApJ, 871, 84
  * Pakmor et al. (2012) Pakmor R., Edelmann P., Röpke F. K., Hillebrandt W., 2012, MNRAS, 424, 2222
  * Pakmor et al. (2016) Pakmor R., Springel V., Bauer A., Mocz P., Munoz D. J., Ohlmann S. T., Schaal K., Zhu C., 2016, MNRAS, 455, 1134
  * Paxton et al. (2011) Paxton B., Bildsten L., Dotter A., Herwig F., Lesaffre P., Timmes F., 2011, ApJS, 192, 3
  * Paxton et al. (2013) Paxton B., et al., 2013, ApJS, 208, 4
  * Paxton et al. (2015) Paxton B., et al., 2015, ApJS, 220, 15
  * Paxton et al. (2018) Paxton B., et al., 2018, ApJS, 234, 34
  * Paxton et al. (2019) Paxton B., et al., 2019, ApJS, 243, 10
  * Perna et al. (2016) Perna R., Lazzati D., Giacomazzo B., 2016, ApJ, 821, L18
  * Perna et al. (2019) Perna R., Wang Y.-H., Farr W. M., Leigh N., Cantiello M., 2019, ApJ, 878, L1
  * Peters (1964) Peters P. C., 1964, Physical Review, 136, 1224
  * Portegies Zwart & McMillan (2000) Portegies Zwart S. F., McMillan S. L. W., 2000, ApJ, 528, L17
  * Renzo & Götberg (2021) Renzo M., Götberg Y., 2021, The Astrophysical Journal, 923, 277
  * Rodriguez et al. (2015) Rodriguez C. L., Morscher M., Pattabiraman B., Chatterjee S., Haster C.-J., Rasio F. A., 2015, Physical Review Letters, 115, 051101
  * Rodriguez et al. (2019) Rodriguez C. L., Zevin M., Amaro-Seoane P., Chatterjee S., Kremer K., Rasio F. A., Ye C. S., 2019, Phys. Rev. D, 100, 043027
  * Ryu et al. (2022) Ryu T., Perna R., Wang Y.-H., 2022, MNRAS, 516, 2204
  * Ryu et al. (2023) Ryu T., Perna R., Pakmor R., Ma J.-Z., Farmer R., de Mink S. E., 2023, MNRAS, 519, 5787
  * Samsing et al. (2014) Samsing J., MacLeod M., Ramirez-Ruiz E., 2014, ApJ, 784, 71
  * Samsing et al. (2019) Samsing J., Venumadhav T., Dai L., Martinez I., Batta A. ., Lopez M., Ramirez-Ruiz E., Kremer K., 2019, Phys. Rev. D, 100, 043009
  * Samsing et al. (2020) Samsing J., D’Orazio D. J., Kremer K., Rodriguez C. L., Askar A., 2020, Phys. Rev. D, 101, 123010
  * Sądowski et al. (2014) Sądowski A., Narayan R., McKinney J. C., Tchekhovskoy A., 2014, MNRAS, 439, 503
  * Springel (2010) Springel V., 2010, MNRAS, 401, 791
  * The LIGO Scientific Collaboration et al. (2021) The LIGO Scientific Collaboration et al., 2021, arXiv e-prints, p. arXiv:2111.03606
  * Timmes & Swesty (2000) Timmes F. X., Swesty F. D., 2000, ApJS, 126, 501
  * Trani et al. (2019) Trani A. A., Spera M., Leigh N. W. C., Fujii M. S., 2019, ApJ, 885, 135
  * Valtonen & Karttunen (2006) Valtonen M., Karttunen H., 2006, The Three-Body Problem
  * VandenBerg et al. (2013) VandenBerg D. A., Brogaard K., Leaman R., Casagrande L., 2013, ApJ, 775, 134
  * Vink et al. (2001) Vink J. S., de Koter A., Lamers H. J. G. L. M., 2001, A&A, 369, 574
  * Wang et al. (2021) Wang Y.-H., McKernan B., Ford S., Perna R., Leigh N. W. . C., Low M.-M. M., 2021, ApJ, 923, L23
  * Weinberger et al. (2020) Weinberger R., Springel V., Pakmor R., 2020, ApJS, 248, 32
  * Zrake et al. (2021) Zrake J., Tiede C., MacFadyen A., Haiman Z., 2021, ApJ, 909, L13


11institutetext: Intel Corporation 11email<EMAIL_ADDRESS>
# Why You Should Try the Real Data for the Scene Text Recognition

Loginov Vladimir 11 0000-0001-6952-0037

###### Abstract

Recent works in the text recognition area have pushed forward the recognition
results to the new horizons. But for a long time a lack of large human-labeled
natural text recognition datasets has been forcing researchers to use
synthetic data for training text recognition models. Even though synthetic
datasets are very large (MJSynth and SynthText, two most famous synthetic
datasets, have several million images each), their diversity could be
insufficient, compared to natural datasets like ICDAR and others. Fortunately,
the recently released text recognition annotation for OpenImages V5 dataset
has comparable with synthetic dataset number of instances and more diverse
examples. We have used this annotation with a Text Recognition head
architecture from the Yet Another Mask Text Spotter and got comparable to the
SOTA results. On some datasets we have even outperformed previous SOTA models.
In this paper we also introduce a text recognition model. The model’s code is
available. 111https://github.com/openvinotoolkit/training_extensions

###### Keywords:

Domain Shift Open Images Scene Text Recognition.

##  1 Introduction

The text recognition plays an important role in people’s lives. It can be used
for digitalizing old documents and helping blind people read. Also, the text
recognition could be utilized in a variety of automation algorithms; one of
the most common use-cases is the license plate recognition.

Since the release of the synthetic datasets (MJSynthJaderberg et al. (2014),
SynthTextGupta et al. (2016)), a common approach to the model training was the
following: use synthetic data for training and real data for evaluation. This
was caused by a lack of labeled large text recognition datasets. From the very
first deep learning text recognition architectures such as Shi et al. (2015)
to the modern works like Cai et al. (2021), Cui et al. (2021) researchers have
followed this pattern. Even though real datasets have train splits, their size
is significantly less than the size of synthetic datasets.

Due to the lack of real data for training, previous works aimed mainly at
tuning an architecture of the model rather than using different data. Baek et
al. (2019) suggested the idea of differentiating a text recognition model into
4 stages: transformation, feature extraction, sequence modelling and
prediction. Many of the existing text recognition model architectures fit into
this scheme. We follow the same approach: in our work we use TPS Jaderberg et
al. (2015) for transformation, ResNeXt Xie et al. (2016) for feature
extraction, convolutional encoder-recurrent decoder from YAMTS Krylov et al.
(2021) with 2D attention mechanism the sequence modelling and the prediction.

The rest of the paper is organized as follows:

  * •

In the Section 2 we briefly discuss other works in this field.

  * •

In the Section 3 we describe in details the suggested model architecture.

  * •

In the Section 4 we show the results of our experiments.

  * •

In the Section 5 we make conclusions and do some call to action for other
researchers.

Our main contributions are as follows:

  * •

Developing the idea of using real data together with synthetic data for
training text recognition models.

  * •

Suggesting text recognition model which performs on par with other models and
even outperforms current state-of-the-art approaches on some datasets.

##  2 Related Works

MJSynthJaderberg et al. (2014), SynthTextGupta et al. (2016) are two large
(each has several million images with text) text recognition datasets. These
two datasets contain images generated in the certain pipeline. In MJSynth,
image generation process is the following: first, certain font is picked and
text in this font is rendered; second, using the image from the previous step,
borders and shadows are colored and composed together; third, projective
distortion is applied to the image; and the last step is natural image
blending. See Jaderberg et al. (2014) for details. SynthText’s pipeline takes
into account depth of the image regions and uses segmentation network to
predict suitable positions to put the text. See the corresponding publication
Gupta et al. (2016) for details.

These two works play an important role in text recognition area, and their
contribution could hardly be overestimated. However, even though images from
MJSynth and SynthText look like real, in some cases it is clear that the
specific image is manually generated. Moreover, real world images are
sometimes so various and rich in texture, curvature, color and other visual
features that it is hard to generate them using a set of predefined rules.
Thus, there is a problem: there are datasets used for training with great
number of examples, but these examples are not diverse enough; and there are
testing datasets with more complex, real images on which the model is
evaluated. This causes a misalignment between training and testing samples.
The problem is also known as domain shift Michieli et al. (2020); Zhang et al.
(2019). Model fits to the training images and, without augmentations, shows
poorer performance on the testing ones. If a researcher decides to use
augmentation to deal with the overfitting problem, model spends its capacity
to learn augmented training samples which are still different from testing.
This results in model performance drop on the testing dataset.

Baek et al. (2021) was one of the first works which suggested the idea of
using real datasets for training text recognition models, but in unsupervised
manner, since large annotated datasets were not available then. Recently
released text spotting annotation for OpenImagesV5 Krylov et al. (2021) and
TextOCR Singh et al. (2021) made training text recognition models on real data
in a supervised manner possible.

ASTER Shi et al. (2019) is one of the works which utilizes Thin Plate Spline
transformation for automatic image rectification. Experiments from Shi et al.
(2019) show that this technique helps to boost the recognition accuracy.

ResNeXt Xie et al. (2016) and ResNet-like He et al. (2016) backbones are
widely used as feature extractors for many text recognition models (Shi et al.
(2019); Cai et al. (2021); Qiao et al. (2020); Baek et al. (2019)). We have
experimented extensively with the configuration of the backbone and found that
standard configuration without the last stage pretrained on the
ImageNetRussakovsky et al. (2014) performs the best.

Attention mechanism Bahdanau et al. (2016) was first introduced as a novel
technique for the machine translation. Recently, the attention mechanism have
become a standard for text recognition models.

Alongside with annotation for text spotting we also use the text recognition
head from Krylov et al. (2021). It performs well for text spotting task and
our experiments show that it also performs well for text recognition task.

##  3 Model Architecture

###  3.1 Overall Architecture

We follow the same approach as in Baek et al. (2019) but do not divide the
last two stages. Thus, the text recognition head consists of sequence
modelling and final prediction. We utilize TPS Jaderberg et al. (2015) as in
ASTER Shi et al. (2019) for image rectification, since this stage has become
one of the standard building blocks for modern text recognition models. We use
ResNeXt Xie et al. (2016) for feature extraction due to its good
performance/accuracy trade-off. In our work we have utilized the same text
recognition head as in Krylov et al. (2021). You can see the full model in the
Figure 1.

Figure 1: Proposed text recognition model architecture.

###  3.2 Thin Plate Spline

Thin Plate Spline (TPS) Jaderberg et al. (2015) was first proposed for image
recognition in the rectification task. Later, this module was utilized in some
text recognition models Shi et al. (2019); Qiao et al. (2020).

The spatial transformer mechanism is split into three parts. In order of
computation, first a localization network takes the image and generates
parameters for spatial transformation. Using the predicted parameters, a
sampling grid is generated, which is a set of points where the input map
should be sampled to produce the transformed output. This is done by the grid
generator. Finally, the sampler generates output image using the grid and
input image.

Thin plate spline is designed in such way that it does not require letter
level annotation. Thus, it can be utilized as is in many text recognition
architectures and its parameters can be learned in an end-to-end manner.

###  3.3 Backbone

Since the release of the ResNet He et al. (2016) this network has become
popular for many computer vision tasks due to its simplicity of learning and
capacity. During the past years, ResNet-like networks showed their
effectiveness and were used in many other text recognition models: ASTER Shi
et al. (2019), CSTR Cai et al. (2021), SEED Qiao et al. (2020), Baek et al.
(2019). The ResNeXt-101 consists of 5 stages. We use ResNeXt-101 without the
last stage. We find this configuration to have a good accuracy/performance
trade-off. In our model we utilize pretrained on ImageNet Russakovsky et al.
(2014) weights as initial for the ResNeXt-101.

###  3.4 Text Recognition Head

Our network heavily utilizes the text recognition head from Krylov et al.
(2021) with minor changes. First, we increase the number of channels to 1024
in the convolutional encoder since this is the exact number of the output
channels from the ResNext-101 (without the last stage). Convolutional encoder
of the head also produces 1024 output channels. Second, for the same reason,
we have changed the size of the feature map to $3\times 12$. For the rest of
the parameters, we use the same architecture, including the depth of the
encoder part and the type of the recurrent decoder (GRU Cho et al. (2014)).

##  4 Experiments

###  4.1 Datasets

####  4.1.1 Training Datasets

There are two large synthetic datasets:

  * •

MJSynth is a synthetically generated text recognition dataset Jaderberg et al.
(2014). It has about 9M images.

  * •

SynthText is also a synthetic dataset, but it has an annotation for text
detection and text recognition. It consists of about 800k images with
approximately 8M word instances.

Recently there were released two large natural text recognition datasets:
TextOCR Singh et al. (2021) and OpenImages V5 text spotting Krylov et al.
(2021).

  * •

TextOCR has about 900k instances of annotated words.

  * •

OpenImages V5 text spotting dataset has about 2M text instances.

In our work we have used a combination of synthetic and real data: MJSynth and
OpenImages V5 text spotting. We did not use any specific technique to balance
samples from different datasets in the batch.

####  4.1.2 Testing Datasets

In the text recognition task, datasets could be divided into two groups:
Regular and Irregular datasets. In the regular datasets, like ICDAR03,
ICDAR13, IIIT5k, SVT, text is mainly horizontally oriented without curvatures
and perspective distortions. Irregular datasets, like ICDAR15, SVT-P, CUTE,
mostly consist of curved texts, which could be with some perspective
distortions. Images from irregular datasets are usually more challenging for
text recognition models to make the prediction.

We perform evaluation of our model on the testing parts of the following
datasets:

Regular datasets:

  * •

ICDAR2003 Lucas et al. (2003) consists of 1156 images for training and 1110
images for evaluation. A common practice for text recognition models is to
ignore all images which contain non-alphanumeric symbols and are less than 3
symbols in length. This results in 867 images for evaluation.

  * •

ICDAR2013 Karatzas et al. (2013) is inherited from ICDAR2003. Its train and
test split have 848 and 1095 images, respectively. Ignoring non-alphanumeric
words results in the dataset of size 1015.

  * •

IIIT5k Mishra et al. (2012) is collected from Google image search. It has 2k
images for training and 3k for testing.

  * •

SVT Wang et al. (2011) has 257 images for training and 647 images for
evaluation. Its name stands for Street View Text and consists of outdoor
street images.

Irregular datasets:

  * •

ICDAR2015 Karatzas et al. (2015) was created for the ICDAR 2015 Robust Reading
competition and contains 4468 images for training and 2077 images for
evaluation. All evaluation subset is used for the evaluation, even though it
consists of some examples with non-alphanumeric instances. Unlike the previous
datasets, this has examples with irregular text.

  * •

SVT-P Phan et al. (2013) is also collected from Google Street View and has
many examples with perspective distortions. Dataset size is 645.

Both training and evaluation of the model were performed in case-insensitive
mode. The vocabulary of our experiments consists of 10 digits, 26 letters and
4 special symbols: start token, end token, pad token and unknown token. Pad
token is used in training phase for numeric representations of the texts to be
equal length and further concatenate them into one batch. Unknown token is
used to represent symbol that is not in the vocabulary (e.g., question mark).
Thus, the final stage of our model predicts every character among 40 classes.
We do not use any lexicon or beam search to boost the recognition accuracy.

###  4.2 Image Preprocessing

Image preprocessing and augmentation plays an important role in many deep
learning algorithms. We resize all input images to $64\times 256$ fixed
resolution. TPS preprocessing of our network produces image of fixed size
$48\times 192$. We also use color jittering (from torchvision Marcel and
Rodriguez (2010)) and Gaussian blurring (from opencv Bradski (2000)) as
augmentations. In the end of the augmentation pipeline we convert all images
to grayscale and scale images to be in [0;1] interval.

###  4.3 Loss Function

Standard Negative Loglikelihood loss function is used as cost function. We use
Adam Kingma and Ba (2017) optimizer with a constant learning rate 1e-4. Other
parameters are default for PyTorch’s Adam optimizer, including zero weight
decay. The model was trained for 15 epochs.

###  4.4 Environment

All experiments were performed on a server with 8 NVIDIA® Tesla P100 GPUs with
16 GB VRAM. We use PyTorch 1.8.1 as our training framework. Batch size was set
to 48.

###  4.5 Evaluation Results

We compare our model with current state-of-the art approaches in the Table 1.
Standard text recognition accuracy is used as a metric of quality.

Table 1: Evaluation results and comparison with other models. Results in bold are the best in every dataset. Model | IC03 | IC13 | IC15 | IIIT5K | SVT | SVT-P
---|---|---|---|---|---|---
CSTR Cai et al. (2021) | 94.8 | 93.2 | 81.6 | 90.1 | 93.7 | 85.0
ASTER Shi et al. (2019) | 94.8 | 93.2 | 81.6 | 90.1 | 93.7 | 85.0
SEED Qiao et al. (2020) | - | 92.8 | 80.0 | 93.8 | 89.6 | 81.4
RCEED Cui et al. (2021) | - | 94.7 | 82.2 | 94.9 | 91.8 | 83.6
SATRN Lee et al. (2020) | 96.7 | 94.1 | 79.0 | 92.8 | 91.3 | 86.5
Baek et al. Baek et al. (2019) | 94.4 | 92.3 | 71.8 | 87.9 | 87.9 | 79.2
Our model | 97.1 | 96.8 | 80.2 | 93.5 | 94.7 | 89.9

###  4.6 Ablation Study

####  4.6.1 Results Using Different Train Datasets

During our research, we have experimented with different datasets used in
training. You can see experiments’ results in the Table 2. All experiments
were performed under the same conditions: ResNeXt-101 backbone, YAMTS text
recognition head, case-insensitive training, case-insensitive evaluation. We
did not use TPS module for this experiment.

Table 2: Comparison of the recognition accuracy using different training datasets. Dataset | IC03 | IC13 | IC15 | IIIT5K | SVT | SVT-P
---|---|---|---|---|---|---
MJSynth | 92.7 | 88.9 | 64.2 | 84.2 | 86.2 | 78.1
MJSynth + SynthText | 95.2 | 93.5 | 69.3 | 87.5 | 88.1 | 82.9
MJSynth \+ OpenImagesV5 | 95.6 | 95.2 | 75.3 | 92 | 91.8 | 87.1
OpenImages V5 | 94.1 | 94.2 | 73.2 | 91.1 | 89.0 | 82.9

It is clear from the table, that having only synthetic data is insufficient to
train a robust text recognition model. It is also interesting that using only
real data drops recognition accuracy a bit. Possible explanation for this fact
is that since the training examples are more difficult, the model should be
trained on a longer schedule (possibly with lower learning rate) like in case
of strong augmentations.

####  4.6.2 Influence of Case Sensitive Learning

As it was stated in the previous section, our first models were trained in
case-sensitive mode while evaluation was performed in case-insensitive mode,
like in Lee et al. (2020). This was caused by the overfitting problem, and
case-sensitive training could be thought as a special augmentation type. Since
the OpenImagesV5 dataset is more challenging than synthetic dataset,
augmentation could be reduced, allowing model to spend its capacity on
learning features of the target dataset, and not augmentations. We have
compared two training setups, the only difference is case mode (and, thus,
number of output classes). The results are reported in the Table 3.

Table 3: Comparison of the recognition accuracy using different case mode. Case Mode | IC03 | IC13 | IC15 | IIIT5K | SVT | SVT-P
---|---|---|---|---|---|---
Case-Sensitive | 95.6 | 95.2 | 75.3 | 92.0 | 91.8 | 87.1
Case-Insensitive | 97.0 | 96.5 | 78.2 | 93.5 | 93.5 | 91.6

Note, these two experiments are performed without TPS module.

###  4.7 Experiments with ViTSTR

Transformer models have become very popular during the past months for many
deep learning problems, and scene text recognition is not an exception. We
have tried to train ViTSTR Atienza (2021) using the OpenImagesV5 annotation.
The results can be seen in the table 4. We have trained the models in the same
framework, but, for an honest comparison, we have disabled filtering non-
alphanumeric symbols during the test. Numbers near the name of the datasets
mean portion of the dataset in the batch. Thus, 0.5 ST + 0.5 MJ means balanced
batch containing examples from both MJSynth and SynthText. Baseline model is
ViTSTR-Tiny, both models were trained using random data augmentation from the
source framework.

Table 4: Comparison of the recognition accuracy using different training datasets for ViTSTR-Tiny+Aug. Training Dataset | IC03 | IC13 | IC15 | IIIT5K | SVT | SVT-P
---|---|---|---|---|---|---
0.5MJ + 0.5 ST (baseline) | 93.7 | 89.8 | 66.6 | 85.1 | 85.9 | 78.5
0.4MJ + 0.2 OpenImagesV5 + 0.4 ST | 95.6 | 94.2 | 77.0 | 91.7 | 90.7 | 86.7

The intention of this experiment was not to train the best model, but to show
that the model actually have higher capacity and could perform better on the
validation datasets, if trained on the real data. The more difficult is the
dataset (e.g. IC15 and SVT-P), the higher is the accuracy gain. It is clear
from the table that even though large augmentation was applied to train the
baseline model, not full capacity of the model was utilized.

##  5 Conclusion

In this paper we have shown that training on natural text recognition datasets
could be more efficient than on synthetically generated images. Our
experiments have proven the fact that training with more challenging data and
less augmentations is more efficient in terms of accuracy on the testing
dataset. We have also introduced a text recognition model which achieves
state-of-the-art results on some common text recognition benchmarks. We
encourage other researchers to try to use the real data to train their models,
because as we have shown in the Section 4, this can help to boost the
recognition accuracy, especially, for the attention-based and transformer
models.

## References

  * Atienza (2021) Atienza, R.: Vision transformer for fast and efficient scene text recognition. CoRR abs/2105.08582 (2021), URL https://arxiv.org/abs/2105.08582
  * Baek et al. (2019) Baek, J., Kim, G., Lee, J., Park, S., Han, D., Yun, S., Oh, S.J., Lee, H.: What is wrong with scene text recognition model comparisons? dataset and model analysis. In: International Conference on Computer Vision (ICCV) (2019)
  * Baek et al. (2021) Baek, J., Matsui, Y., Aizawa, K.: What if we only use real datasets for scene text recognition? toward scene text recognition with fewer labels. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * Bahdanau et al. (2016) Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate (2016)
  * Bradski (2000) Bradski, G.: The OpenCV Library. Dr. Dobb’s Journal of Software Tools (2000)
  * Cai et al. (2021) Cai, H., Sun, J., Xiong, Y.: CSTR: A classification perspective on scene text recognition. CoRR abs/2102.10884 (2021), URL https://arxiv.org/abs/2102.10884
  * Cho et al. (2014) Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder–decoder for statistical machine translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724–1734, Association for Computational Linguistics, Doha, Qatar (Oct 2014), doi:10.3115/v1/D14-1179, URL https://www.aclweb.org/anthology/D14-1179
  * Cui et al. (2021) Cui, M., Wang, W., Zhang, J., Wang, L.: Representation and correlation enhanced encoder-decoder framework for scene text recognition (2021)
  * Gupta et al. (2016) Gupta, A., Vedaldi, A., Zisserman, A.: Synthetic data for text localisation in natural images. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)
  * He et al. (2016) He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016), doi:10.1109/CVPR.2016.90
  * Jaderberg et al. (2014) Jaderberg, M., Simonyan, K., Vedaldi, A., Zisserman, A.: Synthetic data and artificial neural networks for natural scene text recognition. In: Workshop on Deep Learning, NIPS (2014)
  * Jaderberg et al. (2015) Jaderberg, M., Simonyan, K., Zisserman, A., kavukcuoglu, k.: Spatial transformer networks. In: Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 28, Curran Associates, Inc. (2015), URL https://proceedings.neurips.cc/paper/2015/file/33ceb07bf4eeb3da587e268d663aba1a-Paper.pdf
  * Karatzas et al. (2013) Karatzas, D., Gomez-Bigorda, L., Nicolaou, A., Ghosh, S., Bagdanov, A., Iwamura, M., Matas, J., Neumann, L., Chandrasekhar, V.R., Lu, S., Shafait, F., Uchida, S., Valveny, E.: ICDAR 2013 robust reading competition. pp. 1484–1493 (08 2013), doi:10.1109/ICDAR.2013.221
  * Karatzas et al. (2015) Karatzas, D., Gomez-Bigorda, L., Nicolaou, A., Ghosh, S., Bagdanov, A., Iwamura, M., Matas, J., Neumann, L., Chandrasekhar, V.R., Lu, S., Shafait, F., Uchida, S., Valveny, E.: ICDAR 2015 competition on Robust Reading. In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pp. 1156–1160 (2015), doi:10.1109/ICDAR.2015.7333942
  * Kingma and Ba (2017) Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization (2017)
  * Krylov et al. (2021) Krylov, I., Nosov, S., Sovrasov, V.: Open images v5 text annotation and yet another mask text spotter (2021)
  * Lee et al. (2020) Lee, J., Park, S., Baek, J., Oh, S.J., Kim, S., Lee, H.: On recognizing texts of arbitrary shapes with 2d self-attention. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop on Text and Documents in the Deep Learning Era WTDDLE (2020)
  * Lucas et al. (2003) Lucas, S., Panaretos, A., Sosa, L., Tang, A., Wong, S., Young, R.: ICDAR 2003 robust reading competitions. vol. 682-687, pp. 682– 687 (09 2003), ISBN 0-7695-1960-1, doi:10.1109/ICDAR.2003.1227749
  * Marcel and Rodriguez (2010) Marcel, S., Rodriguez, Y.: Torchvision the machine-vision package of torch. In: Proceedings of the 18th ACM International Conference on Multimedia, p. 1485–1488, MM ’10, Association for Computing Machinery, New York, NY, USA (2010), ISBN 9781605589336, doi:10.1145/1873951.1874254, URL https://doi.org/10.1145/1873951.1874254
  * Michieli et al. (2020) Michieli, U., Biasetton, M., Agresti, G., Zanuttigh, P.: Adversarial learning and self-teaching techniques for domain adaptation in semantic segmentation (2020)
  * Mishra et al. (2012) Mishra, A., Alahari, K., Jawahar, C.V.: Scene text recognition using higher order language priors. In: BMVC (2012)
  * Phan et al. (2013) Phan, T.Q., Shivakumara, P., Tian, S., Tan, C.L.: Recognizing text with perspective distortion in natural scenes. In: ICCV, pp. 569–576 (2013)
  * Qiao et al. (2020) Qiao, Z., Zhou, Y., Yang, D., Zhou, Y., Wang, W.: Seed: Semantics enhanced encoder-decoder framework for scene text recognition (2020)
  * Russakovsky et al. (2014) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M.S., Berg, A.C., Li, F.: ImageNet Large Scale Visual Recognition Challenge. CoRR abs/1409.0575 (2014), URL http://arxiv.org/abs/1409.0575
  * Shi et al. (2015) Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. CoRR abs/1507.05717 (2015), URL http://arxiv.org/abs/1507.05717
  * Shi et al. (2019) Shi, B., Yang, M., Wang, X., Lyu, P., Yao, C., Bai, X.: Aster: An attentional scene text recognizer with flexible rectification. IEEE Transactions on Pattern Analysis and Machine Intelligence 41(9), 2035–2048 (2019), doi:10.1109/TPAMI.2018.2848939
  * Singh et al. (2021) Singh, A., Pang, G., Toh, M., Huang, J., Galuba, W., Hassner, T.: TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. CoRR abs/2105.05486 (2021), URL https://arxiv.org/abs/2105.05486
  * Wang et al. (2011) Wang, K., Babenko, B., Belongie, S.: End-to-end scene text recognition. In: ICCV, pp. 1457–1464, IEEE (2011)
  * Xie et al. (2016) Xie, S., Girshick, R.B., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. CoRR abs/1611.05431 (2016), URL http://arxiv.org/abs/1611.05431
  * Zhang et al. (2019) Zhang, Y., Nie, S., Liu, W., Xu, X., Zhang, D., Shen, H.T.: Sequence-to-sequence domain adaptation network for robust text image recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)