# Context Perception Parallel Decoder
for Scene Text Recognition

Yongkun Du, Zhineng Chen, Caiyan Jia, Xiaoting Yin, Chenxia Li, Yuning Du, and
Yu-Gang Jiang  Yongkun Du, Zhineng Chen and Yu-Gang Jiang are with the School
of Computer Science, Fudan University, Shanghai 200433, China. E-mail:
<EMAIL_ADDRESS>{zhinchen<EMAIL_ADDRESS>Caiyan Jia is with the
School of Computer and Information Technology, Beijing Jiaotong University,
Beijing 100044, China. E-mail<EMAIL_ADDRESS>Xiaoting Yin, Chenxia Li and
Yuning Du are with Baidu Inc., Beijing 100085, China. E-mail:{yinxiaoting,
lichenxia<EMAIL_ADDRESS>Corresponding author: Zhineng Chen.

###### Abstract

Scene text recognition (STR) methods have struggled to attain high accuracy
and fast inference speed. Autoregressive (AR)-based models implement the
recognition in a character-by-character manner, showing superiority in
accuracy but with slow inference speed. Alternatively, parallel decoding
(PD)-based models infer all characters in a single decoding pass, offering
faster inference speed but generally worse accuracy. We first present an
empirical study of AR decoding in STR, and discover that the AR decoder not
only models linguistic context, but also provides guidance on visual context
perception. Consequently, we propose Context Perception Parallel Decoder
(CPPD) to predict the character sequence in a PD pass. CPPD devises a
character counting module to infer the occurrence count of each character, and
a character ordering module to deduce the content-free reading order and
placeholders. Meanwhile, the character prediction task associates the
placeholders with characters. They together build a comprehensive recognition
context. We construct a series of CPPD models and also plug the proposed
modules into existing STR decoders. Experiments on both English and Chinese
benchmarks demonstrate that the CPPD models achieve highly competitive
accuracy while running approximately 8x faster than their AR-based
counterparts. Moreover, the plugged models achieve significant accuracy
improvements. Code is at this https URL.

###### Index Terms:

Scene text recognition, parallel decoding decoder, context perception,
character counting, character ordering.

##  1 Introduction

Scene text recognition (STR) aims at reading a character sequence from the
text instance cropped from real-world scenes, e.g., text on billboards,
packaging boxes, electronic screens, etc. It has attracted attention from a
wide range of applications for its key role in precise image interpretation.
However, the task still remains challenging due to text variations such as
distortion, occlusion, blurring, multi-fonts, etc. Meanwhile, the strict time
consumption limit in many applications is another challenge posed to STR
models.

Figure 1: (a) Autoregressive (AR) decoder. (b) Parallel decoding (PD) decoder.
(c) Our CPPD. CPPD has two well-designed modules for robust and fast context
perception. C and P are character and position embeddings. Green, blue, and
yellow arrows denote visual features, character counting, and ordering
embeddings, respectively.

Among STR solutions, attention-based encoder-decoder ones are perhaps the most
popular paradigm. Typically, these methods first employ a CNN [1] or ViT [2]
as the vision encoder. During decoding, they retain all the extracted visual
features and dynamically select them for recognition, depending on the decoder
used. There are two major decoders, i.e., autoregressive (AR) and parallel
decoding (PD) decoders. AR decoders are widely adopted and among the most
accurate decoders in NLP tasks like machine translation [3, 4, 5], owing to
their superior language modeling capability. By using AR decoder for STR, as
shown in Fig. 1(a), the characters are recognized one-by-one and when decoding
the _i_ -th character, the previously identified _i-1_ characters are regarded
as priors and fed into the decoder. By doing so, linguistic knowledge is
implicitly encoded. However, AR decoders run slowly due to the use of an
iterative decoding scheme. Meanwhile, the accuracy reported by earlier AR-
based models [6, 7] is worse than recent top-ranked methods [8, 9], which is
inconsistent with the observations in NLP tasks. Recently, several efforts
[10, 11, 12] have shown that the accuracy of AR decoders could be improved and
ranked top-tier by carefully modeling the character position clue. They
suggest that language modeling is not the only clue captured by AR decoding.

In contrast, PD decoders infer all the characters in a single forward
propagation pass, as shown in Fig. 1(b). They have received much research
attention [13, 8, 14] due to the advantage in inference speed compared to AR
decoders. However, they also encounter the problem that the recognition
context could not be stably established in one forward computation. To
compensate for this, recent efforts [8, 14] have focused more on better
exploiting non-visual clues and have attained impressive recognition accuracy.
For example, ABINet [8] introduced an external language model and an iterative
correction scheme to aid the recognition. It got highly competitive accuracy
in standard benchmarks with a certain sacrifice on inference speed.

Aiming at inheriting the merits of both AR and PD decoders, i.e., accurate and
fast, we first investigate why early AR-based STR models perform inferior. Our
exploration reveals that the reason lies in the outdated visual encoder and
training strategy. Specifically, NRTR [7], a Transformer-based AR model,
employs a weak encoder. We upgrade it by using SVTR (e.g., SVTR-T) [15] and
adopting the same training strategy as in [8]. Surprisingly, such a
straightforward modification significantly improves the accuracy and makes the
generated SVTR-NRTR series top-ranked recognizers in terms of accuracy, which
motivates us to choose it as the baseline for further study. We observe that
context modeling is more complicated for AR-based STR due to the difference
between visual and text modalities. For NLP tasks, what is modeled, in
general, is a linguistic context that records the appearance probability of a
character given a preceding character sequence. However, for STR, in addition
to the linguistic association between characters, visual content such as
character appearance and positions can also be different and affect
recognition. However, how AR decoders utilize these context variables is still
less explored, hindering the design of more advanced recognizers.

To explore this further, we conduct empirical experiments based on SVTR-T-NRTR
(i.e., SVTR-T as the encoder). The results suggest that AR decoding
outperforms current PD by a relatively large accuracy margin. It is mainly
because, in addition to linguistic modeling, AR decoder can get visual context
such as the appearance and positions of previously decoded characters, which
are absent in current PD. Moreover, when adding a groundtruth-based side loss
to PD, it gets accuracy gains but is still worse than AR, implying that a
character prediction-guided loss is insufficient to fully infer the context
above. Our attempts basically indicate that properly modeling visual context
is the key to enhancing the accuracy of PD decoders.

With the exploration above, we focus on developing accurate and efficient STR
models from the angle of fast and robustly estimating visual context. To this
end, we propose a new PD decoder termed Character Perception Parallel Decoder
(CPPD), which aims at simultaneously attaining high accuracy and fast
inference speed. As shown in Fig. 1(c), we devise a character ordering (CO)
module and a character counting (CC) module. Both employ cross-attention for
targeted feature enhancement. By using dedicated side losses as guidance, CO
successfully estimates the content-free placeholders of the characters and
their reading order, while CC predicts the occurrence count of each character.
Moreover, the character prediction task also guides the learning process to
fill in the placeholders with characters. Thus, these components together form
a context that besides linguistic modeling, also describes the character
count, position, and reading order of the entire character sequence, robustly
giving what the character sequence is and where the characters appear. It is
worth noting that the context is more complete than that modeled by AR
decoding due to the inclusion of all the characters. In addition, since the
context above can be established in a single forward propagation, CPPD also
accelerates the inference significantly.

Based on the study above, we construct a series of SVTR-CPPD models and also
plug the proposed CO and CC modules into existing STR decoders. Extensive
experiments on both English and Chinese benchmarks demonstrate that the
constructed CPPD models achieve highly competitive accuracy, which is even
higher than their AR counterparts while running approximately 8x faster,
positioning them among the most accurate and fast recognizers to date.
Moreover, the plugged models also improve the accuracy by clear margins,
especially on recent large and challenging datasets. The contribution of this
paper is threefold.

  * •

We verify that AR decoder remains the most competitive STR decoder in terms of
accuracy, and empirically reveal that besides linguistic context, character
count, positions, and their reading order are essential context variables.
These variables are largely missed in existing PD models.

  * •

We devise CPPD, a novel PD decoder that incorporates CO and CC modules with
dedicated losses to perceive the desired context in a single forward pass. The
two modules and the character recognition task are complementary, and they
form a visual and linguistic involved context description.

  * •

Extensive experiments confirm the superiority of CPPD. The constructed models
achieve highly competitive accuracy, and run as fast as 400 FPS during
inference on one NVIDIA 1080Ti GPU. In addition, integrating CPPD modules into
existing STR models leads to significant accuracy improvements.

##  2 Related Work

The advance of deep learning has propelled STR into the attention-based
encoder-decoder era [16, 17]. Generally, the encoder extracts visual,
linguistic, and/or other features. The decoder integrates these features and
generates the recognized sequence. Based on how the decoder is formulated, we
can broadly classify existing models as CTC and attention-based ones, and the
latter can be further classified as two categories, i.e., AR and PD models.

CTC models implement the decoding via a two-step alignment, i.e., character-
image sub-region mapping and CTC decoding [18]. For example, CRNN [19] first
extracted the image feature by jointly utilizing CNN and RNN. It then inferred
a character or a blank for each image sub-region. The recognized sequence was
obtained by de-duplicating characters and removing blanks according to the
learned CTC rule. In [20], GTC employed a graph neural network and attention
mechanism to improve both feature learning and CTC decoding. SVTR [15] devised
a dedicated ViT to capture multi-grained character features and attained
impressive accuracy using only CTC decoding. CTC models have fast inference
speed in general. Nevertheless, they assume that the recognition is based on
character-image sub-region correspondences, restricting their ability to
accommodate variations like text curvation, etc.

Inspired by the success of attention-based models in NLP, AR models were
introduced to STR in [21, 6, 22], where gated RNN [23, 24] was employed for
decoding and showed accuracy advantages. NRTR [7] built the first Transformer-
based AR decoder [3] for STR. It achieved the top accuracy at that time.
Later, its accuracy was further improved by using ResNet as the encoder [25].
SATRN [26] was developed based on a 2D Transformer to better preserve 2D
features. To suppress attention drift during decoding, character position
embedding was leveraged and different position enhancement schemes were
proposed in [10, 11, 12]. They got improved accuracy, implying that the
character position was also a critical clue for STR. PARSeq [9] learned a
permuted AR sequence model by exhaustively utilizing the character occurrence
dependence. AR decoder was also applied to recognize artistic text in [27].
Despite the progress, AR models perform recognition in a character-by-
character manner. The inference speed is an issue and hinders their
applications in fast-response scenarios.

PD models were proposed to seek accurate and fast STR. Generally, the speed
advantage came from PD while the accuracy was mainly attributed to exploiting
non-visual clues so far. For example, SRN [13] inferred all the characters at
once, where a groundtruth-based side loss was imposed to guide the context
modeling. Later, Zhang et al. [28] got improved accuracy by recurrently
applying such a side loss. In [8], Fang et al. introduced external language
model and the iterative mechanism to its decoder. Wang et al. [14] devised an
additional LM network to endue the vision backbone with LM capability. During
inference, the LM network was discarded to ensure speedup. Qiao et al. [29]
proposed PIMNet. It used a more accurate AR decoder to distill PD decoder. The
AR decoder was discarded at inference, achieving a balance between accuracy
and efficiency.

Our CPPD also falls into the PD decoder family. It devises two dedicated side
loss-guided CC and CO modules to capture the required context. Note that CC-
like clues have been exploited in several previous studies. For example, they
were explored in the forms of predicting sequence length [30], or estimating
character count using CAN [31] and ACE [32]. However, CAN was dedicated to AR-
based mathematical expression recognition, and ACE used a probability-based
sequence-level loss. Our usage differs from those practices and its detail
will be elaborated later. Meanwhile, some works have also explored the
position information [11, 22, 12, 13, 8, 14]. Typically, these references
initialized a fixed position embedding and then fused it with other clues to
make predictions. These usages are quite different from our side loss-guided
CO learning.

Figure 2: Six decoder variants considered in our empirical studies. (a) AR
decoder trained using the left-to-right sequence (AR) and permuted sequence
(AR-P, with the dashed red box enabled), (b) AR decoder with limited
linguistic context (AR-L) and AR-L trained using permuted sequence (AR-L-P),
(c) PD decoder, (d) PD decoder with pre-decoding (PD-P), where PD block is the
gray region in (c). MHA (MHSA) denotes multi-head (self-) attention.

##  3 AR Decoder Revisit

As stated, NRTR [7], a typical Transformer-based AR model for STR, performed
worse than the top-ranked models by some margins. To investigate the reason
behind this, we first adopt the same training strategy as in [8] to train
NRTR. The obtained model, NRTR-re, gives an average accuracy of 89.62% on six
standard English benchmarks [33], surpassing the original NRTR [7] by 2.24%.
We then employ SVTR-T as the visual encoder and retrain NRTR. As seen in Tab.
VI, the resulting model, SVTR-T-NRTR, improves the averaged accuracy to
91.49%, while reducing the model size from 31.7 M to 8.6 M, and increasing the
inference FPS from 33.2 to 45.5. Meanwhile, it attains an accuracy improvement
of 1.5% compared to SVTR-T, and an even larger accuracy improvement is
observed when SVTR-B is used as the encoder (see Tab. VI), The results
highlight the benefits of AR decoding. Note that SVTR-B-NRTR is among the most
accurate recognizers, while SVTR-T-NRTR is also the most accurate among models
with similar sizes. Since NRTR uses only a standard AR architecture, the
results clearly verify the superiority of AR decoding in STR, also indicating
that previously inferior accuracy was mainly due to an outdated visual encoder
and training strategy.

Then, we investigate the context AR and PD decoders modeled. Specifically, six
decoder variants are considered on top of SVTR-T-NRTR. We group them into four
categories as shown in Fig. 2: (a) standard AR decoder, which is the same as
SVTR-T-NRTR, either trained using the standard left-to-right sequence or
permuted sequence [9] (AR-P, with the dashed red box enabled in Fig. 2(a));
(b) AR decoder with limited linguistic context (AR-L), where MHSA is removed
from the Transformer block, limiting the linguistic modeling only from the AR
decoding loop. The same as above, the permuted version is also trained (AR-
L-P); (c) Standard PD decoder; (d) PD decoder with pre-decoding (PD-P), where
a groundtruth-based side loss the same as [13] is added to guide the feature
learning. The first four AR-based variants can be uniformly written as:

$\displaystyle
P(Y|F_{v},\theta_{D})=\prod\limits_{t=1}^{T+1}p(y_{t}|C_{t},F_{v};\theta_{D})$
(1)

where $y_{t}$ is the character to be inferred at time step $t$. $T$ is the
total decoding time steps determined during the decoding process. $y_{0}$ and
$y_{T+1}$ represent the start and end tokens. $C_{t}$, $F_{v}$, and
$\theta_{D}$ denote the corresponding recognition context, visual features,
and decoder parameters, respectively. In this experiment, these variants only
differ from $C_{t}$, or saying, how the context is formulated. Specifically,
they are:

$\displaystyle C_{t}^{AR}=c_{y_{0}}:c_{y_{t-1}}$ (2) $\displaystyle
C_{t}^{AR-P}=c_{y_{k_{0}}}:c_{y_{k_{t-1}}}$ (3) $\displaystyle
C_{t}^{AR-L}=c_{y_{t-1}}$ (4) $\displaystyle C_{t}^{AR-L-P}=c_{y_{k_{t-1}}}$
(5)

where $c_{y_{i}}$ is the context of character $y_{i}$, including its category,
appearance, position, and linguistic association with other characters.
$c_{y_{0}}:c_{y_{t-1}}$ is the context of previously decoded $t-1$ characters,
and $\\{k_{1},...,k_{T}\\}=Permute\\{1,...,T\\}$ denotes a permuted sequence.

TABLE I: Results of the six decoder variants. Method | AR | AR-P | AR-L | AR-L-P | PD | PD-P
---|---|---|---|---|---|---
Avg(%) | 91.49 | 91.78 | 91.13 | 91.45 | 89.60 | 90.43
Time(ms) | 22.01 | 22.01 | 21.19 | 21.19 | 2.630 | 2.801

The average accuracy and time consumption of the six variants are listed in
Tab. I, from which the following empirical observations are inferred. We first
look into those from the AR-based ones. First, only 0.36% and 0.33% accuracy
drops are observed when comparing AR-L with AR, and AR-L-P with AR-P. The
result implies that removing the direct language modeling within each time
step, which truncates the multigram language model to a bigram one (see Eq. 2
and Eq. 4), does not trigger an obvious accuracy drop. Second, introducing
training permutation increases the accuracy by 0.29% (AR v.s. AR-P) and 0.33%
(AR-L v.s. AR-L-P), as the context has been enriched during the permutation.
Note that AR-L-P attains similar accuracy as the standard AR although it
depends on only one permuted character context, i.e., with a weaker linguistic
context. The result indicates that other context variables also have been
strengthened during the training permutation.

We then compare AR with the PD ones. First, when changing from the AR decoder
to a standard PD decoder, despite accelerating inference speed, a 1.89%
accuracy drop is observed. As seen in Fig. 2(c), the PD decoder uses a fixed
content-free position embedding as the _query_ to associate with visual
features, which may be quite different across text instances. Thus, it is
challenging to establish a robust context in such a forward pass. This is also
reflected in the attention maps in Fig. 3(c), where the _query_ could not
accurately gaze at the corresponding characters. Second, when pre-decoding is
added, PD-P gains an accuracy improvement of 0.83% compared to the standard
one. Basically, this loss imposes an additional learning objective that also
uses groundtruth characters as the aligning target. It drives the STR model to
separate context modeling and character prediction tasks to some extent, thus
enjoying more accurate character localization, as shown in Fig. 3(d). However,
it is observed that context variables like character appearance and positions
still have not been explicitly modeled. This explains why there remains a
margin between AR and PD-P decoders.

Figure 3: Attention maps of different decoder variants. (a) AR decoder, (b)
AR-L, (c) PD decoder, (d) PD-P, (e) our CPPD.

The experiments basically verify that compared to PD decoder, the success of
AR decoder mainly lies in the availability of the previously decoded
characters, whose appearance and positions serve as necessary visual context
and aid the recognition. However, AR decoder also causes a slow inference
speed. How to appropriately model those context variables in a PD decoding
pass, and achieve the goal of ”AR’s accuracy, PD’s speed” still needs
exploration.

Figure 4: An overview of CPPD. The decoder accepts visual features, character
counting and ordering embeddings. Cross-attention and corresponding side
losses are equipped to guide model learning. The right side shows details of
the character counting and ordering modules, where the red line denotes either
character counting or ordering embeddings, depending on which module is
employed.

##  4 The Proposed CPPD

###  4.1 Overall

We propose CPPD to address the above issue. Fig. 4 gives an overview of CPPD.
Given a text instance of $I\in R^{H\times W\times 3}$, we first use SVTR [15]
(with the rectification module and CTC decoder removed) as the encoder to
extract visual feature $\mathbf{F}_{v}\in
R^{\frac{H}{16}\times\frac{W}{4}\times D}$. Meanwhile, character counting (CC)
embedding $\mathbf{E}_{cc}\in R^{C\times D}$ and character ordering (CO)
embedding $\mathbf{E}_{co}\in R^{L\times D}$ are initialized with a truncated
normal distribution with mean of 0 and standard deviation of 0.2, where $C$
and $L$ are predefined parameters denoting the size of character set and the
maximum length of a character sequence, respectively. On the decoder side, a
CC module and a CO module are developed, which use either $\mathbf{E}_{cc}$ or
$\mathbf{E}_{co}$ as _query_ , $\mathbf{F}_{v}$ as _key_ and _value_ , to
calculate cross-attention. Two dedicated side losses are devised to guide
their learning. Their output features are further fused by another cross-
attention. As a result, it generates the desired context features. Then a
character recognition task is followed, which uses cross-entropy (CE) loss to
learn the decoded sequence in a single forward propagation. We detail the
structure of CPPD as follows.

###  4.2 Character counting and ordering modules

CC and CO modules are devised for feature reinforcement and context modeling.
The two modules have the same internal structure but their optimization
objectives are different. The CC module is followed by a character counting
(CC) loss, which aims at describing the occurrence count of each character as
accurately as possible. For example, given _arteta_ , the task is to tell us
that character _a_ and _t_ both appear twice, character _e_ and _r_ appear
once, while other characters do not appear, i.e., with count 0. By forcing
this, the CC embedding $\mathbf{E}_{cc}$, which is used as _query_ , can
search for the desired information in visual feature space. Since different
characters have different appearances and two characters with the same
category are likely to have similar feature representations, the _query_
feature can be strengthened from this dimension.

Let $y_{c,l}=1$ if the _c_ -th character appears _l_ times in the character
sequence and 0 otherwise, $p_{c,l}$ the probability that the _c_ -th character
be predicted to appear _l_ times in the same sequence. The CC loss
$\mathcal{L}_{cc}$ is written as:

$\mathcal{L}_{cc}=-\frac{1}{C}\sum_{c=1}^{C}\sum_{l=0}^{L}y_{c,l}\log(p_{c,l})$
(6)

Note that Eq. 6 assigns a loss term for every possible character and
occurrence frequency. Compared with the ACE loss (Eq. 7) [32] that only
considered the appeared characters, our CC loss has the merits of more
completely modeling the occurrence of characters and being decoupled from the
character sequence length _L_. We also have conducted an experiment comparing
the two (see Tab. II).

$\mathcal{L}_{ace}=-\sum_{c=1}^{C}\frac{N_{c}}{L}\log(\frac{\sum_{l=0}^{L}p_{c,l}}{L})$
(7)

Meanwhile, the CO module is followed by a character ordering (CO) loss, which
aims at inferring the locations from the first character to the last one. Also
taking _arteta_ as an example, it focuses on deducing where the first to sixth
characters appear regardless of their categories, i.e., the content-free
reading order and positions. Similar to above, the CO embedding
$\mathbf{E}_{co}$ is treated as _query_ to implement this goal. Since the
character foreground exhibits differences with the background, features
discriminating character positions could be identified and extracted from the
visual feature space, and used to reinforce the _query_ feature.

Let $p_{l}$ be the probability that the _l_ -th position is predicted to have
a character. The CO loss $\mathcal{L}_{co}$ is given by:

$\mathcal{L}_{co}=-\frac{1}{L}\sum_{l=0}^{L}{(y_{l}\log(p_{l})+(1-y_{l})\log(1-p_{l}))}$
(8)

where $y_{l}=1$ for $l\leq N$ and $y_{l}=0$ for $l>N$ always holds for a
sequence of length _N_. $\mathcal{L}_{co}$ is binary CE loss over all the _L_
positions. Note that the same label will be assigned to two different
character sequences with the same length, therefore forcing CO to capture the
content-free placeholders. Meanwhile, since different sequences may have
different lengths and they are all initialized in a left-to-right order. This
loss also implicitly encodes the reading order. Consequently, it decouples the
CO task from the final character prediction task to some extent, forming a
definite CO enhancement. It is worth mentioning that such a usage is quite
different from existing PD models [13, 8, 14].

We take the CC module as an example to illustrate the detailed implementation
of the two modules. As depicted on the right side of Fig. 4, the visual
feature $\mathbf{F}_{v}$ is first reshaped and concatenates with the CC
feature $\mathbf{E}_{cc}$, forming the _query_ feature and experienced an MHA.
Then layer norm and MLP are successively carried out. Residual connections are
also employed for feature consolidation. Next, the feature is sliced into a
new visual feature and a new CC feature with the same shape as the inputted
ones. The process is performed twice (for SVTR-T) or three times (for SVTR-B),
depending on the visual encoder employed. The generated features are termed
$\mathbf{F}_{cc}$ and $\mathbf{\hat{E}}_{cc}$ for visual and CC, respectively.
Similarly, $\mathbf{F}_{co}$ and $\mathbf{\hat{E}}_{co}$ can be obtained the
same as above from the CO module.

###  4.3 Recognition module

The two modules above generate strengthened features biased towards the
occurrence count of each character and the content-free positions. They are
different but complementary. Therefore, we fuse them using another cross-
attention.

Specifically, $\mathbf{\hat{E}}_{co}$ is recruited as _query_ , and
$\mathbf{F}_{cc}$ is treated as _value_ and _key_. A standard Transformer
block with MHA is employed to conduct the cross-attention. It generates a
combined feature that better characterizes the context. Finally, we implement
the recognition by using a simple parallel linear prediction, where cross-
entropy (CE)-based recognition loss as in Eq. 9 is imposed to learn the
correct decoding. The reason that we uses
$<\mathbf{\hat{E}}_{co},\mathbf{F}_{cc}>$ pair rather than
$<\mathbf{\hat{E}}_{cc},\mathbf{F}_{co}>$ is twofold. First,
$\mathbf{\hat{E}}_{co}$ encodes the reading positions, which is more in line
with the character prediction task. Second, the size of the character set $C$
is large for languages like Chinese, using $\mathbf{\hat{E}}_{cc}$ would
consume more computational resources.

$\mathcal{L}_{rec}=-\frac{1}{L}\sum_{l=0}^{L}\sum_{c=1}^{C}y_{l,c}\log(p_{l,c})$
(9)

where $p_{l,c}$ is the probability that the _l_ -th character in the sequence
being classified as character _c_. $y_{l,c}=1$ if character _c_ is the label
of the _l_ -th character, and 0 otherwise.

###  4.4 Training Objective

CPPD is trained in an end-to-end manner with the CC loss in Eq. 6, CO loss in
Eq. 8 and CE loss in Eq. 9. The final optimization function is formulated as:

$\mathcal{L}=\lambda_{co}\mathcal{L}_{co}+\lambda_{cc}\mathcal{L}_{cc}+\lambda_{rec}\mathcal{L}_{rec}$
(10)

where $\lambda_{co}$, $\lambda_{cc}$ and $\lambda_{rec}$ are all simply set to
1 in ours experiments.

The CE loss focuses on a position-irrelevant character prediction. Together
with the CC and CO losses, they form a mutually reinforced recognition
context, which associates the content-free placeholders with position-free
characters. Thus, the constructed context robustly delineates character
content, count, positions, and their reading order, forming a comprehensive
description of character appearance and positions. Meanwhile, Linguistic
context is also learned in the form of character appearing dependency from the
CC and the final recognition modeling. As a result, CPPD is capable of
building a visual and linguistic involved context, which is even more complete
than that of AR decoding due to the inclusion of subsequent characters.
Benefiting from their joint optimization, as shown in Fig. 4, visualization on
the CC side predicts the character counts and their positions accurately.
While on the CO side, it clearly captures the character’s reading order and
positions. Both of these explain the merit of CPPD from the loss perspective.

##  5 Experiments

###  5.1 Datasets and Implementation Details

We validate CPPD on both English and Chinese datasets. For English our models
are trained on two commonly used synthetic scene text datasets, i.e., MJSynth
(MJ) [34, 35] and SynthText (ST) [36]. Then the models are tested on: (1) six
regular and irregular text benchmarks, i.e., ICDAR 2013 (IC13) [37], Street
View Text (SVT) [38], IIIT5K-Words (IIIT) [39], ICDAR 2015 (IC15) [40], Street
View Text-Perspective (SVTP) [41] and CUTE80 (CUTE) [42]. For IC13 and IC15,
we use the versions with 857 and 1,811 images, respectively. (2) the recently
constructed Union14M-L benchmark, which consists of over 0.4 million real-
world test images with both complexity and versatility [43]. It includes
challenging text subsets such as curve, multi-oriented, artistic, etc.

For Chinese we use Chinese text recognition (CTR) benchmark [44], a public
dataset containing four challenging scenarios: Scene, Web, Document and
Writing. For each scenario, we train the model on the corresponding training
set and use the validation set to determine it, which is then assessed on the
test set.

We use AdamW optimizer [45] with a weight decay of 0.05 for training. For
English models, all text instances, unless specified, are resized to $32\times
100$ and the learning rate (LR) is set to
$\frac{5}{10^{4}}\times\frac{batchsize}{1024}$. Cosine LR scheduler [46] with
4 epochs linear warm-up is used in all the 20 epochs. Data augmentation like
rotation, perspective distortion, motion blur and gaussian noise, are randomly
performed during training. The alphabet includes all case-insensitive
alphanumerics. For Chinese models, all text instances are resized to $32\times
256$. Data augmentation is not performed and the LR is set to
$\frac{5}{10^{4}}\times\frac{batchsize}{512}$. Cosine LR scheduler with 5
epochs linear warm-up is used in all the 100 epochs. Word accuracy is used as
the evaluation metric. The size of the character set $C$ is set to 37 for
English and 6625 for Chinese [47]. The maximum prediction length $L$ is set to
25 for both. All models are trained on 4 Tesla V100 GPUs.

###  5.2 Ablation Study

To better understand CPPD, we perform controlled experiments under different
settings on SVTR-B-CPPD.

TABLE II: Ablation study on loss choice of CC and CO. Loss |  Counting or ordering label of arteta | Avg(%)
---|---|---
Ours |  Eq. 6 | [a:2,e:1,r:1,t:2,other:0] | 93.80
Eq. 8 | [1,1,1,1,1,1,0,…,0]
ACE |  Eq. 7 |  [a:$\frac{2}{L}$,e:$\frac{1}{L}$,r:$\frac{1}{L}$,t:$\frac{2}{L}$, pad:$\frac{L-6}{L}$] | 93.31
CE |  Eq. 9 |  [a,r,t,e,t,a,[E],pad,…,pad] | 93.50
TABLE III: Ablation study on CC and CO modules. CO |  | CO
---
Loss
CC |  | CC
---
Loss
IC13 | SVT | IIIT | IC15 | SVTP | CUTE | Avg(%)
|  |  |  | 97.3 | 94.4 | 96.2 | 86.4 | 89.1 | 88.5 | 92.01
✓ |  |  |  | 97.3 | 95.2 | 97.2 | 86.9 | 90.2 | 91.0 | 92.97
✓ | ✓ |  |  | 97.5 | 94.4 | 96.9 | 87.2 | 90.5 | 93.1 | 93.29
|  | ✓ |  | 97.2 | 95.1 | 97.2 | 87.0 | 89.3 | 89.2 | 92.49
|  | ✓ | ✓ | 97.4 | 95.2 | 97.1 | 87.3 | 91.0 | 92.0 | 93.35
✓ |  | ✓ |  | 97.3 | 95.4 | 97.0 | 87.3 | 90.4 | 91.0 | 93.05
✓ | ✓ | ✓ |  | 97.3 | 94.6 | 97.1 | 87.0 | 90.7 | 92.4 | 93.18
✓ |  | ✓ | ✓ | 98.0 | 94.9 | 97.4 | 87.7 | 90.7 | 92.4 | 93.52
✓ | ✓ | ✓ | ✓ | 98.2 | 95.5 | 97.6 | 87.9 | 90.9 | 92.7 | 93.80
TABLE IV: Feature representativeness analysis of different CPPD components. Features | IC13 | SVT | IIIT | IC15 | SVTP | CUTE | Avg(%)
---|---|---|---|---|---|---|---
$\mathbf{F}_{cc}$ | 96.3 | 93.7 | 95.9 | 85.9 | 88.5 | 90.3 | 91.76
$\mathbf{F}_{co}$ | 95.7 | 92.0 | 95.0 | 85.1 | 85.3 | 86.8 | 90.00
$\mathbf{F}_{v}$ | 95.2 | 88.4 | 93.4 | 81.5 | 83.3 | 86.8 | 88.10
$\mathbf{\hat{F}}_{v}$ | 94.7 | 91.5 | 94.5 | 83.3 | 84.7 | 90.3 | 89.83
TABLE V: Results on adaptability to different encoders. Decoder | Encoder | IC13 | SVT | IIIT | IC15 | SVTP | CUTE | Avg(%)
---|---|---|---|---|---|---|---|---
CTC | ResNet+En | 96.7 | 93.4 | 96.6 | 85.0 | 86.8 | 91.3 | 91.64
ABINet [48] | 97.4 | 93.5 | 96.2 | 86.0 | 89.3 | 89.2 | 91.93
CPPD | 97.0 | 94.3 | 96.8 | 87.3 | 91.2 | 94.1 | 93.43
CTC | ViT | 96.4 | 93.8 | 96.2 | 86.5 | 87.9 | 91.0 | 91.97
PARSeq [9] | 97.0 | 93.6 | 97.0 | 86.5 | 88.9 | 92.2 | 92.53
CPPD | 97.0 | 94.3 | 97.1 | 88.1 | 90.2 | 92.4 | 93.17
CTC | SVTR-Base | 97.1 | 91.5 | 96.0 | 85.2 | 89.9 | 91.7 | 91.90
NRTR | 97.7 | 95.4 | 97.4 | 88.5 | 90.7 | 91.7 | 93.54
ABINet | 97.0 | 94.4 | 97.2 | 87.5 | 90.1 | 92.0 | 93.02
PARSeq | 97.3 | 95.2 | 96.7 | 87.1 | 90.2 | 91.0 | 92.93
CPPD | 98.2 | 95.5 | 97.6 | 87.9 | 90.9 | 92.7 | 93.80
Figure 5: Attention maps of the final recognition features for different models. (a) CPPD, (b) CPPD w/o CC, (c) CPPD w/o CO, (d) NRTR. Each sub-figure depicts the 12 heads in MHSA from top to bottom, and each map corresponds to a character. Best viewed in color. TABLE VI: Results on the six English benchmarks tested against existing methods. NRTR-re denotes the re-trained NRTR by adopting the same training strategy as in [8]. FPS is obtained by averaging the inference time over 3000 English text instances on one NVIDIA 1080Ti GPU under the PaddlePaddle inference mode. The same below. Method | Encoder | Input Size | Regular | Irregular |  Average Accuracy(%) |  Parameters ($\times 10^{6}$) |  FPS
---|---|---|---|---|---|---|---
IC13 | SVT | IIIT | IC15 | SVTP | CUTE
AR |  ASTER [21] (2019) | ResNet | $32\times 100$ |  | 89.5 | 93.4 | 76.1 | 78.5 | 79.5 |  | 27.2 |
NRTR [7] (2019) | CNN+En | $32\times 100$ | 95.8 | 91.5 | 90.1 | 79.4 | 86.6 | 80.9 | 87.38 | 31.7 | 33.2
NRTR-re[7] (2019) | CNN+En | $32\times 100$ | 96.6 | 91.8 | 93.0 | 83.4 | 86.8 | 86.1 | 89.62 | 31.7 | 33.2
SAR [6] (2019) | ResNet | $48\times 160$ | 91.0 | 84.5 | 91.5 | 69.2 | 76.4 | 83.5 | 82.68 | 57.7 | 9.62
RobustScanner [11] (2020) | ResNet | $48\times 160$ | 94.8 | 88.1 | 95.3 | 77.1 | 79.5 | 90.3 | 87.52 | 48.0 | 16.4
PARSeq [9] (2022) | ViT | $32\times 128$ | 97.0 | 93.6 | 97.0 | 86.5 | 88.9 | 92.2 | 92.53 | 23.8 | 84.7
|  CDistNet [12] (2023) | ResNet+En | $32\times 128$ | 97.4 | 93.5 | 96.4 | 86.0 | 88.7 | 93.4 | 92.57 | 65.5 | 14.5
PD |  SRN [13] (2020) | ResNet+FPN | $64\times 256$ | 95.5 | 91.5 | 94.8 | 82.7 | 85.1 | 87.8 | 89.57 | 54.7 | 39.4
VisionLAN [14] (2021) | ResNet+En | $64\times 256$ | 95.7 | 91.7 | 95.8 | 83.7 | 86.0 | 88.5 | 90.23 | 32.8 | 200
ABINet [48] (2021) | ResNet+En | $32\times 128$ | 97.4 | 93.5 | 96.2 | 86.0 | 89.3 | 89.2 | 91.93 | 36.7 | 124
GTR [49] (2022) | ResNet | $64\times 256$ | 96.8 | 94.1 | 95.8 | 84.6 | 87.9 | 92.3 | 91.92 | 42.1 |
I2C2W [50] (2023) | ResNet+En | $64\times 600$ | 95.0 | 91.7 | 94.3 | 82.8 | 83.1 | 93.1 | 90.00 |  |
LPV-B [51] (2023) | SVTR-Base | $48\times 160$ | 97.6 | 94.6 | 97.3 | 87.5 | 90.9 | 94.8 | 93.78 | 35.1 | 103
CTC |  CRNN [19] (2017) | ResNet+BiLSTM | $32\times 100$ | 91.1 | 81.6 | 82.9 | 69.4 | 70.0 | 65.5 | 76.75 | 8.30 | 159
SVTR-T [15] (2022) | SVTR-Tiny | $32\times 100$ | 96.3 | 91.6 | 94.4 | 84.1 | 85.4 | 88.2 | 89.99 | 6.03 | 408
SVTR-B [15] (2022) | SVTR-Base | $48\times 160$ | 97.1 | 91.5 | 96.0 | 85.2 | 89.9 | 91.7 | 91.90 | 24.6 | 212
SVTR-L [15] (2022) | SVTR-Large | $48\times 160$ | 97.2 | 91.7 | 96.3 | 86.6 | 88.4 | 95.1 | 92.54 | 40.8 | 161
Ours | SVTR-T-NRTR | SVTR-Tiny | $32\times 100$ | 97.3 | 93.4 | 95.0 | 85.5 | 88.2 | 89.6 | 91.49 | 8.56 | 45.5
SVTR-B-NRTR | SVTR-Base | 97.7 | 95.4 | 97.4 | 88.5 | 90.7 | 91.7 | 93.54 | 32.2 | 32.5
SVTR-T-CPPD | SVTR-Tiny | 97.1 | 94.4 | 96.6 | 86.6 | 88.5 | 90.3 | 92.25 | 8.29 | 399
SVTR-B-CPPD | SVTR-Base | 98.2 | 95.5 | 97.6 | 87.9 | 90.9 | 92.7 | 93.80 | 26.8 | 252
SVTR-B-CPPD | SVTR-Base | $48\times 160$ | 97.5 | 95.5 | 97.7 | 87.7 | 92.4 | 93.7 | 94.10 | 26.8 | 206
Plug | CTC-CPPD | SVTR-Base | $32\times 100$ | 96.8 | 95.2 | 96.9 | 87.2 | 90.2 | 91.7 | 93.02 | 24.6 | 212
NRTR-CPPD | 98.6 | 96.6 | 97.6 | 88.8 | 91.9 | 93.4 | 94.50 | 32.3 | 31.1
ABINet-CPPD | 98.0 | 96.3 | 97.2 | 87.6 | 92.6 | 92.4 | 94.01 | 38.9 | 141
PARSeq-CPPD | 97.3 | 95.7 | 97.4 | 87.6 | 90.9 | 93.4 | 93.71 | 24.2 | 98.2
TABLE VII: Union14M-L performance of models trained on synthetic datasets (MJ and ST).  Method | Encoder | Input Size | Curve |  | Multi-
---
Oriented
Artistic | Contextless | Salient |  | Multi-
---
Words
General |  | Average
---
Accuracy(%)
AR |  ASTER [21] (2019) | ResNet | $32\times 100$ | 34.0 | 10.2 | 27.7 | 33.0 | 48.2 | 27.6 | 39.8 | 31.50
NRTR [7] (2019) | CNN+En | $32\times 100$ | 31.7 | 4.4 | 36.6 | 37.3 | 30.6 | 54.9 | 48.0 | 34.79
SAR [6] (2019) | ResNet | $48\times 160$ | 44.3 | 7.7 | 42.6 | 44.2 | 44.0 | 51.2 | 50.5 | 40.64
RobustScanner [11] (2020) | ResNet | $48\times 160$ | 43.6 | 7.9 | 41.2 | 42.6 | 44.9 | 46.9 | 39.5 | 38.09
PARSeq-B [9] (2022) | SVTR-Base | $32\times 100$ | 63.9 | 16.7 | 52.5 | 54.3 | 68.2 | 55.9 | 56.9 | 52.62
|  CDisNet [12] (2023) | ResNet+En | $32\times 128$ | 69.3 | 24.4 | 49.8 | 55.6 | 72.8 | 64.3 | 58.5 | 56.38
PD |  SRN [13] (2020) | ResNet+FPN | $64\times 256$ | 63.4 | 25.3 | 34.1 | 28.7 | 56.5 | 26.7 | 46.3 | 40.14
VisionLAN [14] (2021) | ResNet+En | $64\times 256$ | 57.7 | 14.2 | 47.8 | 48.0 | 64.0 | 47.9 | 52.1 | 47.39
ABINet [48] (2022) | ResNet+En | $32\times 128$ | 59.5 | 12.7 | 43.3 | 38.3 | 62.0 | 50.8 | 55.6 | 46.03
ABINet-B [48] (2022) | SVTR-Base | $32\times 100$ | 62.3 | 13.9 | 50.0 | 45.1 | 67.1 | 53.4 | 58.5 | 50.07
|  LPV-B [51] (2023) | SVTR-Base | $48\times 160$ | 68.3 | 21.0 | 59.6 | 65.1 | 76.2 | 63.6 | 62.0 | 59.40
CTC |  CRNN [19] (2017) | ResNet+BiLSTM | $32\times 100$ | 7.5 | 0.9 | 20.7 | 25.6 | 13.9 | 25.6 | 32.0 | 18.03
SVTR-T [15] (2022) | SVTR-Tiny | $32\times 100$ | 63.0 | 32.1 | 37.9 | 44.2 | 67.5 | 49.1 | 52.8 | 49.51
SVTR-B [15] (2022) | SVTR-Base | $48\times 160$ | 69.8 | 37.7 | 47.9 | 61.4 | 66.8 | 44.8 | 61.0 | 55.63
SVTR-L [15] (2022) | SVTR-Large | $48\times 160$ | 76.8 | 45.2 | 55.2 | 63.0 | 73.8 | 48.9 | 64.2 | 61.01
Ours | SVTR-T-NRTR | SVTR-Tiny | $32\times 100$ | 60.8 | 23.2 | 46.4 | 43.6 | 59.0 | 38.7 | 55.6 | 46.76
SVTR-B-NRTR | SVTR-Base | 66.2 | 19.4 | 54.0 | 58.8 | 70.5 | 61.3 | 65.3 | 56.49
SVTR-T-CPPD | SVTR-Tiny | 52.4 | 12.3 | 48.2 | 54.4 | 61.5 | 53.4 | 61.4 | 49.10
SVTR-B-CPPD | SVTR-Base | 65.5 | 18.6 | 56.0 | 61.9 | 71.0 | 57.5 | 65.8 | 56.63
SVTR-B-CPPD | SVTR-Base | $48\times 160$ | 71.9 | 22.1 | 60.5 | 67.9 | 78.3 | 63.9 | 67.1 | 61.69
Plug | CTC-CPPD | SVTR-Base | $32\times 100$ | 61.3 | 15.6 | 53.2 | 59.2 | 69.4 | 59.6 | 63.8 | 54.60
NRTR-CPPD | 69.5 | 21.0 | 59.5 | 64.2 | 74.1 | 65.5 | 67.0 | 60.10
ABINet-CPPD | 66.4 | 19.1 | 57.3 | 56.7 | 72.8 | 55.8 | 65.0 | 56.19
PARSeq-CPPD | 67.1 | 19.0 | 58.0 | 58.0 | 74.0 | 61.9 | 63.9 | 57.19

The loss choice of CC and CO. As previously stated, a binary CE loss (i.e.,
Eq. 8) is considered in the CO module. However, the CE loss described in Eq. 9
also can implement this goal. We carry out an experiment to compare the two.
As seen in Tab. II, the standard CE loss attains an accuracy of 93.50%, 0.30%
lower than using our CO loss. The CO loss only concerns whether there are
characters on the positions, while Eq. 9 aims to identify the character at
each position, making it more closely aligned with the final recognition
objective. We argue that such an arrangement introduces the follow-up learning
objective too early, and makes the feature enhancement somewhat vague. This
result in turn justifies the rationality of our design.

On the other hand, ACE loss [32] can also achieve the goal of CC. The result
in Tab. II shows that our CC loss gets a higher accuracy. As previously
mentioned, this is mainly because the CC loss includes the appearance of all
characters and is irrelevant to the character sequence length.

The effectiveness of CC and CO. As shown in Tab. III, nine experiments are
designed to enumerate the combinations of CC/CO modules or/and losses. We take
the model without both CC and CO modules as the baseline. As can be seen, when
the two side losses are not employed, adding the CC or CO module individually
also leads to an improvement of 0.48% or 0.96%, respectively, and applying
both increases the accuracy by 1.04%. This is attributed to the cross-
attention-based learning, where a counting and/or ordering-based
initialization also benefits the recognition to some extent. When the two side
losses are enabled further, the CC or CO loss solely gives 0.86% or 0.32%
additional gains, while they together report a prominent improvement of 1.79%
compared to the baseline. Besides, other enumerated combinations also report
consistent improvements. The results demonstrate that the CC and CO modules
are complementary and both take effect in CPPD. Specifically, the CC module
targets character content, and the CO module emphasizes content-free reading
order and positions. Combining them allows CC and CO to mutually reinforce and
forms a more robust context description.

We further conduct experiments to analyze whether the CC and CO modules
contribute to better feature extraction as follows. First, the complete CPPD
is trained. Then, three variants are designed by appending a CTC decoder
behind either $\mathbf{F}_{cc}$, $\mathbf{F}_{co}$ or $\mathbf{F}_{v}$, and
discarding the rest parts of CPPD. Next, the three variants are trained under
the constraint of freezing the entire network except for the CTC decoder. When
converged, their performance on the six benchmarks is evaluated, as shown in
Tab. IV. Comparing to $\mathbf{F}_{v}$, decoding based on $\mathbf{F}_{cc}$
and $\mathbf{F}_{co}$ lead to 3.66% and 1.9% average accuracy gains,
respectively.

Since a decoder would biased toward the feature closer to the character
prediction layer, we also devise another variant, termed as
$\mathbf{\hat{F}}_{v}$, for a more in-depth comparison, where the CO embedding
is used as _query_ , $\mathbf{F}_{v}$ as _key_ and _value_ , followed by the
character recognition task. Therefore $\mathbf{\hat{F}}_{v}$ is closer than
the $\mathbf{F}_{cc}$ and $\mathbf{F}_{co}$ above. However, it reports an
accuracy of 89.93%, worse than the results from the two features. The results
clearly suggest that critical recognition context is modeled by CC and CO
modules.

CC and CO visualization verification. We visualize attention maps of the final
recognition features in Fig. 5, where each of the 12 heads in MHSA is plotted
from top to bottom. As can be seen, Head 1 (H1 for short) plays the role of
character positioning and H0 mainly focuses on localizing the right-side
character. For CPPD the hotspots are more accurate and obvious compared to
those of CPPD without either CC or CO. In CPPD from H6 to H8 we see that many
maps are responsible for splitting characters, which is much less observed in
maps of the other three methods. This partially explains that CO and CC
modules jointly build a more complete context and give better recognition. On
the other hand, for AR-based NRTR, a majority of maps focus on character
localization, which is in line with its character-by-character decoding
nature.

###  5.3 Adaptability to Different Encoders

We conduct three sets of experiments to validate the adaptability of CPPD to
different encoders. For fair comparison, the experiments is carried out by
appending CPPD to the encoder of popular STR models. First, CNN-based
comparison. The same as ABINet [8], a ResNet45 and three Transformer units are
taken as the encoder. As seen in Tab. V, using CPPD receives a accuracy gain
of 1.5%. Second, ViT-based comparison. The same ViT as PARSeq [9] is employed.
CPPD gives an improvement of 0.64% in this case. Third, SVTR-B with different
decoders. It is seen that both ABINet and PARSeq get better accuracy when
SVTR-B is equipped as their encoder. Nevertheless, CPPD still gets the highest
accuracy among the competitors. The results demonstrate that CPPD is well-
suited to different encoders.

###  5.4 SVTR-CPPD Series and Plugged Models

Based on the analyses above, we construct two series of CPPD models for
further experiments. First, the SVTR-CPPD series. SVTR-T-CPPD and SVTR-B-CPPD
are constructed, which employ SVTR with different capacities as the encoder.
They show different accuracy-speed balances and thus offer flexible choices
for applications. Second, we integrate the proposed CC and CO modules into
four popular STR decoders, i.e., CTC, NRTR [7], ABINet [8], and PARSeq [9].
Specifically, we employ SVTR-B as the encoder and use these decoders to
replace the _Transformer Block with MHA_ in Fig. 4. To keep consistent with
CPPD, $\mathbf{F}_{cc}$ is recruited as _value_ and _key_ , while the _query_
is also $\mathbf{F}_{cc}$ for CTC, the character embedding for NRTR, the fixed
position embedding for ABINet and PARSeq. This series of models can validate
the plug-and-play nature of CPPD modules.

###  5.5 Comparisons on six English benchmarks

Figure 6: Visualization of attention maps. In each example, a text instance
and its recognition result obtained from SVTR-B-CPPD are given. From left to
right the three columns are visualization of CC, CO and the final recognition
features, respectively. Among them, CC maps are vertically arranged in an
alphabetical order, CO and recognition maps are arranged according to their
reading orders. Red boxes denote the erroneously localized maps. The wrongly
recognized characters are shown in red. The same below.

We compare the constructed CPPD models with previous works on the six standard
English benchmarks covering regular and irregular text instances in Tab. VI.
The previous works are grouped into three groups, i.e., AR, PD and CTC, for
better analysis. While the two series of constructed CPPD models are
categorized into ”Ours” and ”Plug”. We first look into the SVTR-CPPD ones.
There are 2.26% and 2.2% average accuracy gains from SVTR-T-CPPD v.s. SVTR-T,
and SVTR-B-CPPD v.s. SVTR-B, respectively. The results clearly verify the
effectiveness of CPPD. Note that the two SVTR-CPPD models do not incorporate
the rectification module while SVTR did, which bring additional accuracy gains
in general. Even larger accuracy gains may obtain when getting rid of the
rectification module. Moreover, incorporating CPPD only introduces 2.26 M and
2.2 M increase in model size. Since the rectification module only has 1.88 M
additional parameters, it implies that CPPD is a lightweight decoder and the
introduced model size overhead is limited. When comparing SVTR-based AR and PD
models, although SVTR-B-NRTR achieves a prominent average accuracy of 93.54%,
its CPPD counterpart, i.e., SVTR-B-CPPD, further gives 0.26% accuracy gains,
along with 7.75x inference acceleration and slightly smaller model size. Not
only reaching the new state-of-the-art accuracy, but also running the fastest
among the non-SVTR-based AR and PD models in Tab. VI. Meanwhile, the accuracy
gain observed by SVTR-T-CPPD v.s. SVTR-T-NRTR is 0.76% with an 8.77x inference
speedup. These accuracy advantages can be explained as AR decoding only models
the context of already decoded characters while CPPD models all the
characters. It thus builds a more comprehensive recognition context. In
addition, SVTR-B-CPPD shows 0.32% accuracy improvements and with smaller size
and faster inference speed when compared to the recent LPV-B [28]. The results
again demonstrate the superiority of the CPPD architecture.

We then inspect the plugged models. CTC-CPPD surpasses SVTR-B by 1.12% in
terms of accuracy, implying the effectiveness of incorporating the CC and CO
modules. The other three models also get improvement and achieve remarkable
accuracy. For example, NRTR-CPPD outperforms SVTR-B-NRTR by 0.96% and gets an
average accuracy of 94.50%, the highest so far. The result indicates that the
CPPD context somewhat differs from the context of AR decoding, and they are
still complementary to some extent. In addition, 2.08% and 1.18% accuracy
gains are observed when comparing ABINet-CPPD with ABINet, and PARSeq-CPPD
with PARSeq, respectively. By eliminating those caused by the encoder
difference, there are still 0.99% and 0.78% gaps in these two comparisons (see
Tab. V). The results indicate that the context generated by CPPD is not only
effective, but also has great generalization capability and can be used by
different decoders for further accuracy improvements.

TABLE VIII: Results on Chinese text recognition (CTR) dataset tested against existing models, which come from [52] except for the SVTR and CPPD series. Method | Scene | Web | Document | Handwriting | Average(%) |  Parameters($\times 10^{6}$) | FPS
---|---|---|---|---|---|---|---
CRNN [19] (2017) | 53.4 | 57.0 | 96.6 | 50.8 | 64.45 | 12.4 |
ASTER [21] (2019) | 61.3 | 51.7 | 96.2 | 37.0 | 61.55 | 27.2 |
MORAN [53] (2019) | 54.6 | 31.5 | 86.1 | 16.2 | 47.10 | 28.5 |
SAR [6] (2019) | 59.7 | 58.0 | 95.7 | 36.5 | 62.48 | 27.8 |
SEED [54] (2020) | 44.7 | 28.1 | 91.4 | 21.0 | 46.30 | 36.1 |
MASTER [55] (2021) | 62.8 | 52.1 | 84.4 | 26.9 | 56.55 | 62.8 |
ABINet [48] (2021) | 66.6 | 63.2 | 98.2 | 53.1 | 70.28 | 53.1 |
TransOCR [56] (2021) | 71.3 | 64.8 | 97.1 | 53.0 | 71.55 | 83.9 |
SVTR-B [15] (2022) | 71.7 | 73.8 | 98.2 | 52.2 | 73.98 | 26.3 | 175
SVTR-L [15] (2022) | 72.2 | 74.1 | 98.1 | 53.6 | 74.50 | 43.4 | 146
CCR-CLIP [52] (2023) | 71.3 | 69.2 | 98.3 | 60.3 | 74.78 | 62.0 |
SVTR-B-CPPD | 74.4 | 76.1 | 98.6 | 55.3 | 76.10 | 29.4 | 188
SVTR-B-CPPD-STN | 78.4 | 79.3 | 98.9 | 57.6 | 78.55 | 31.2 | 169

###  5.6 Comparisons on Union14M-L benchmark

We also conduct experiments on Union14M-L [43], a recent large-scale English
benchmark that more realistically reveals the recognition challenges. The
results trained based on synthetic datasets are given in Tab. VII. The
accuracy of ABINet and PARSeq-B (using SVTR-B as the encoder), which perform
well in the six standard benchmarks, is not high, revealing the challenges of
large-scale realistic dataset. Compared with the results on the six
benchmarks, different SVTR models exhibit large performance gaps, where 6.12%
and 5.38% accuracy gains are obtained from SVTR-T to SVTR-B, and further to
SVTR-L. A similar trend is also observed in the SVTR-NRTR and SVTR-CPPD
series. The results suggest that a large-capacity visual encoder is more
important on Union14M-L. The plugged models also report remarkable
improvements, especially for PARSeq. It gives a prominent 4.57% accuracy gain
when CPPD modules are equipped, again demonstrating the effectiveness of CPPD.
In addition, we observe that the resizing is a critical factor, where a large
input resolution significantly improves the accuracy. For example, SVTR-B-CPPD
improves the accuracy by 5.06% when the resolution is switched from $32\times
100$ to $48\times 160$, highlighting a high-resolution input is a priority in
challenging and accuracy-first scenarios.

###  5.7 Comparisons on Chinese benchmark

We then assess the models on the CTR benchmark. Ref. [52] gave the accuracy of
nine existing methods as listed in Tab. VIII, while the remaining SVTR and
CPPD series are computed by ourselves. Note that the model size and inference
time will increase a little for Chinese for the same method, due to involving
a larger prediction layer and a large input resolution. However, both are not
compromised by the vocabulary size due to the CC module not participating in
the inference. SVTR-B-CPPD also shows accuracy gain. It gives a 2.12% average
accuracy gain compared to SVTR-B, while running faster due to not
incorporating the rectification module. When STN is further considered, a
prominent accuracy improvement of 4.57% is obtained. This is because the CTR
dataset exhibits strong disturbances including curvature, occlusion, light-
dark contrast, etc. Merely adding an STN could contribute to an improvement of
2.45% in terms of accuracy. Note that the two CPPD models outperform the
recent CCR-CLIP, which is dedicated to CTR, by 1.48% and 3.93% in terms of
accuracy. The results convincingly validate the great cross-language
generalization ability of CPPD.

###  5.8 Visualization Analysis

Figure 7: Visualization of Attention maps for a vertical Chinese text. Figure
8: Visualization of attention maps for a long Chinese text.

We give recognition results and attention maps of different CPPD components in
Fig. 6 to verify the effectiveness of CPPD and whether its modules accomplish
their task as desired. Specifically, eight challenging text instances are
presented. The first six show the merit of CPPD. They exhibit excellent
adaptation in severely curved, highly blurred, partially occluded, artistic,
and large character space scenarios. Their attention maps show that the CC,
CO, and recognition branches basically read the characters as expected. The
first example, i.e., _shanghai_ , simultaneously exhibits text rotation,
curvation, and blur, while our uniform resizing scheme introduces additional
text deformation. Nevertheless, CPPD still correctly read the text. In the
third and fourth examples, it is observed that mistakes are observed from a
few CO or CC maps (marked by red boxes), e.g., the second _o_ is missed in the
third CC map of Fig. 6(c). Nevertheless, both cases are correctly recognized.
The two examples illustrate the complementary nature of the three kinds of
maps. Different features can complement each other and, to some extent, ward
off distractions. There are also a few bad cases. For example the seventh
example, after resizing, the image is severely deformed and character _a_
looks similar to _e_ , which causes the corresponding localization error in
the CC map and mis-recognition. On the other hand, for the eighth example,
which mixes several recognition difficulties such as severe curvation, fine
character stroke, etc, hotspots of a portion of maps are mistakenly localized.
For example, _i_ and _e_ are both counted as _i_ (and also _e_), and the
placeholders are not always presented in a left-to-right order. These factors
lead CPPD to misidentify the word. We argue that strengthening the visual
feature would be a way to correct these instances.

In Fig. 7 and Fig. 8, we also give attention maps of two challenging Chinese
text instances. The first is a blurred and vertically arranged Chinese text,
which poses a great challenge to traditional CTR models. Nevertheless, all the
maps give the right localization such that SVTR-B-CPPD correctly recognizes
it. The other instance is a long Chinese text, whose recognition easily
suffers from the attention drift [11] in AR-based decoders and PD decoders
also perform worse in general. When CPPD equipped, the hotspots in most maps
correctly identify the characters and their placeholders, thus also giving a
correct response. Both cases clearly validate how both CC and CO help the
recognition.

##  6 Conclusion

Aiming at developing accurate and fast STR models, we first analyze the
differences between AR and PD decoders, from which we have inferred that
character appearance and positions are valuable context variables for accurate
recognition. Consequently, we have presented CPPD, a novel PD decoder that
elegantly perceives these context variables using dedicated designed CC and CO
modules and losses. Extensive experiments on English and Chinese text
recognition convincingly validate our proposal. The constructed SVTR-CPPD
models show remarkable balance on accuracy and inference speed, which not only
perform top-tier in terms of accuracy, but also run approximately 8x faster
than their AR-based counterparts. Moreover, the plugged models consistently
give further accuracy improvement, especially on large and challenging
datasets. Both demonstrate the effectiveness of the proposed CPPD.

While impressive experimental results are obtained from typical benchmarks, it
is observed that the constructed context is still somewhat different and
complementary from that built by AR decoders. Therefore, one future research
objective is to further explore the missed context variables based on the CPPD
framework. In addition, we are also interested in combining CPPD with recent
large language or multi-modality models [57, 58], which also exhibit
impressive understanding ability in OCR-related tasks. Integrating them
together would be a topic worthy further study.

## Acknowledgments

This work was supported by the National Key R&D Program of China
(2022YFB3104703), and the National Natural Science Foundation of China
(62172103).

## References

  * [1] K. He, X. Zhang, S. Ren, and S., “Deep residual learning for image recognition,” in _CVPR_ , June 2016.
  * [2] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _ICLR_ , 2021.
  * [3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _NIPS_ , 2017.
  * [4] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in _NIPS_ , 2014, pp. 3104–3112.
  * [5] K. Cho, B. Merrienboer, Ç. Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” in _EMNLP_ , 2014, pp. 1724–1734.
  * [6] H. Li, P. Wang, C. Shen, and G. Zhang, “Show, attend and read: A simple and strong baseline for irregular text recognition,” in _AAAI_ , 2019, pp. 8610–8617.
  * [7] F. Sheng, Z. Chen, and B. Xu, “Nrtr: A no-recurrence sequence-to-sequence model for scene text recognition,” in _ICDAR_ , 2019, pp. 781–786.
  * [8] S. Fang, H. Xie, Y. Wang, Z. Mao, and Y. Zhang, “Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition,” in _CVPR_ , 2021.
  * [9] D. Bautista and R. Atienza, “Scene text recognition with permuted autoregressive sequence models,” in _ECCV_ , 2022, pp. 178–196.
  * [10] Z. Wan, M. He, H. Chen, X. Bai, and C. Yao, “Textscanner: Reading characters in order for robust scene text recognition,” in _AAAI_ , vol. 34, no. 07, 2020, pp. 12 120–12 127.
  * [11] X. Yue, Z. Kuang, C. Lin, H. Sun, and W. Zhang, “Robustscanner: Dynamically enhancing positional clues for robust text recognition,” in _ECCV_ , 2020, pp. 135–151.
  * [12] T. Zheng, Z. Chen, S. Fang, H. Xie, and Y.-G. Jiang, “Cdistnet: Perceiving multi-domain character distance for robust text recognition,” _International Journal of Computer Vision_ , pp. 1–19, 2023.
  * [13] D. Yu, X. Li, C. Zhang, T. Liu, J. Han, J. Liu, and E. Ding, “Towards accurate scene text recognition with semantic reasoning networks,” in _CVPR_ , 2020, pp. 12 113–12 122.
  * [14] Y. Wang, H. Xie, S. Fang, J. Wang, S. Zhu, and Y. Zhang, “From two to one: A new scene text recognizer with visual language modeling network,” in _ICCV_ , 2021, pp. 14 194–14 203.
  * [15] Y. Du, Z. Chen, C. Jia, X. Yin, T. Zheng, C. Li, Y. Du, and Y. Jiang, “Svtr: Scene text recognition with a single visual model,” in _IJCAI_ , 2022, pp. 884–890.
  * [16] X. Chen, L. Jin, Y. Zhu, C. Luo, and T. Wang, “Text recognition in the wild: A survey,” _ACM Computing Surveys (CSUR)_ , vol. 54, no. 2, pp. 1–35, 2021.
  * [17] L. Yang, D. Ergu, Y. Cai, F. Liu, and B. Ma, “A review of natural scene text detection methods,” _Procedia Computer Science_ , vol. 199, pp. 1458–1465, 2022.
  * [18] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,” in _ICML_ , 2006, p. 369–376.
  * [19] B. Shi, X. Bai, and C. Yao, “An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition,” in _TPAMI_ , 2017, pp. 2298–2304.
  * [20] W. Hu, X. Cai, J. Hou, S. Yi, and Z. Lin, “Gtc: Guided training of ctc towards efficient and accurate scene text recognition,” in _AAAI_ , 2020, pp. 11 005–11 012.
  * [21] B. Shi, M. Yang, X. Wang, P. Lyu, C. Yao, and X. Bai, “Aster: An attentional scene text recognizer with flexible rectification,” in _IEEE Trans. Pattern Anal. Mach. Intell._ , 2019, pp. 2035–2048.
  * [22] T. Wang, Y. Zhu, L. Jin, C. Luo, X. Chen, Y. Wu, Q. Wang, and M. Cai, “Decoupled attention network for text recognition,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 34, no. 07, 2020, pp. 12 216–12 224.
  * [23] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural Comput._ , pp. 1735–1780, 1997.
  * [24] J. Chung, Ç. Gülçehre, K. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” 2014.
  * [25] Z. Kuang, H. Sun, Z. Li, X. Yue, T. Lin, J. Chen, H. Wei, Y. Zhu, T. Gao, W. Zhang, K. Chen, W. Zhang, and D. Lin, “Mmocr: A comprehensive toolbox for text detection, recognition and understanding,” _arXiv preprint arXiv:2108.06543_ , 2021.
  * [26] J. Lee, S. Park, J. Baek, S. Oh, S. Kim, and H. Lee, “On recognizing texts of arbitrary shapes with 2d self-attention,” in _CVPR Workshops_ , 2020, pp. 546–547.
  * [27] X. Xie, L. Fu, Z. Zhang, Z. Wang, and X. Bai, “Toward understanding wordart: Corner-guided transformer for scene text recognition,” in _ECCV_. Springer, 2022, pp. 303–321.
  * [28] B. Zhang, H. Xie, Y. Wang, J. Xu, and Y. Zhang, “Linguistic more: Taking a further step toward efficient and accurate scene text recognition,” _arXiv preprint arXiv:2305.05140_ , 2023.
  * [29] Z. Qiao, Y. Zhou, J. Wei, W. Wang, Y. Zhang, N. Jiang, H. Wang, and W. Wang, “Pimnet: a parallel, iterative and mimicking network for scene text recognition,” in _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021, pp. 2046–2055.
  * [30] H. Jiang, Y. Xu, Z. Cheng, S. Pu, W. Niu, Y.and Ren, F. Wu, and W. Tan, “Reciprocal feature learning via explicit and implicit tasks in scene text recognition,” in _ICDAR_. Springer, 2021, pp. 287–303.
  * [31] B. Li, Y. Yuan, D. Liang, X. Liu, Z. Ji, J. Bai, W. Liu, and X. Bai, “When counting meets hmer: Counting-aware network for handwritten mathematical expression recognition,” in _ECCV_. Springer, 2022, pp. 197–214.
  * [32] Z. Xie, Y. Huang, Y. Zhu, L. Jin, Y. Liu, and L. Xie, “Aggregation cross-entropy for sequence recognition,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 6538–6547.
  * [33] J. Baek, G. Kim, J. Lee, S. Park, D. Han, S. Yun, S. J. Oh, and H. Lee, “What is wrong with scene text recognition model comparisons? dataset and model analysis,” in _ICCV_ , 2019, pp. 4714–4722.
  * [34] M. Jaderberg, K. Simonyan, A. Vedaldi, and A. Zisserman, “Synthetic data and artificial neural networks for natural scene text recognition,” in _NeurIPS Deep Learning Workshop_ , 2014.
  * [35] M. Jaderberg, K. Simonyan, A. Vedaldi, and A. Zisserman, “Reading text in the wild with convolutional neural networks,” in _IJCV_ , 2015, pp. 1–20.
  * [36] A. Gupta, A. Vedaldi, and A. Zisserman, “Synthetic data for text localisation in natural images,” in _CVPR_ , 2016, pp. 2315–2324.
  * [37] D. KaratzasAU, F. ShafaitAU, S. UchidaAU, M. IwamuraAU, L. G. i. BigordaAU, S. R. MestreAU, J. MasAU, D. F. MotaAU, J. A. AlmazànAU, and L. P. de las Heras, “Icdar 2013 robust reading competition,” in _ICDAR_ , 2013, pp. 1484–1493.
  * [38] K. Wang, B. Babenko, and S. Belongie, “End-to-end scene text recognition,” in _ICCV_ , 2011, pp. 1457–1464.
  * [39] A. Mishra, A. Karteek, and C. V. Jawahar, “Scene text recognition using higher order language priors,” in _BMVC_ , 2012.
  * [40] D. Karatzas, L. Gomez-Bigorda, A. Nicolaou, S. Ghosh, A. Bagdanov, M. Iwamura, J. Matas, L. Neumann, V. R. Chandrasekhar, S. Lu, F. Shafait, S. Uchida, and E. Valveny, “Icdar 2015 competition on robust reading,” in _ICDAR_ , 2015, pp. 1156–1160.
  * [41] T. Q. Phan, P. Shivakumara, S. Tian, and C. L. Tan, “Recognizing text with perspective distortion in natural scenes,” in _CVPR_ , 2013, pp. 569–576.
  * [42] R. Anhar, S. Palaiahnakote, C. S. Chan, and C. L. Tan, “A robust arbitrary text detection system for natural scene images,” in _Expert Systems with Applications_ , 2014, p. 8027–8048.
  * [43] Q. Jiang, J. Wang, D. Peng, C. Liu, and L. Jin, “Revisiting scene text recognition: A data perspective,” _arXiv preprint arXiv:2307.08723_ , 2023.
  * [44] J. Chen, H. Yu, J. Ma, M. Guan, X. Xu, X. Wang, S. Qu, B. Li, and X. Xue, “Benchmarking chinese text recognition: Datasets, baselines, and an empirical study,” _arXiv:2112.15093_ , 2021.
  * [45] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in _ICLR_ , 2019.
  * [46] I. Loshchilov and F. Hutter, “SGDR: stochastic gradient descent with warm restarts,” in _ICLR_ , 2017.
  * [47] C. Li, W. Liu, R. Guo, X. Yin, K. Jiang, Y. Du, Y. Du, L. Zhu, B. Lai, X. Hu, D. Yu, and Y. Ma, “Pp-ocrv3: More attempts for the improvement of ultra lightweight ocr system,” _CoRR_ , vol. abs/2206.03001, 2022.
  * [48] S. Fang, Z. Mao, H. Xie, Y. Wang, C. Yan, and Y. Zhang, “Abinet++: Autonomous, bidirectional and iterative language modeling for scene text spotting,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 45, no. 6, p. 7123–7141, 2023.
  * [49] Y. He, C. Chen, J. Zhang, J. Liu, F. He, C. Wang, and B. Du, “Visual semantics allow for textual reasoning better in scene text recognition,” in _AAAI_ , 2022, pp. 888–896.
  * [50] C. Xue, J. Huang, W. Zhang, S. Lu, C. Wang, and S. Bai, “Image-to-character-to-word transformers for accurate scene text recognition,” _IEEE Trans. Pattern Anal. Mach. Intell._ , 2023.
  * [51] B. Zhang, H. Xie, Y. Wang, J. Xu, and Y. Zhang, “Linguistic more: Taking a further step toward efficient and accurate scene text recognition.” International Joint Conferences on Artificial Intelligence Organization, 2023, pp. 1704–1712.
  * [52] H. Yu, X. Wang, B. Li, and X. Xue, “Chinese text recognition with a pre-trained clip-like model through image-ids aligning,” in _ICCV_ , 2023.
  * [53] C. Luo, L. Jin, and Z. Sun, “Moran: A multi-object rectified attention network for scene text recognition,” _Pattern Recogn._ , p. 109–118, 2019.
  * [54] Z. Qiao, Y. Zhou, D. Yang, Y. Zhou, and W. Wang, “Seed: Semantics enhanced encoder-decoder framework for scene text recognition,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 13 525–13 534.
  * [55] N. Lu, W. Yu, X. Qi, Y. Chen, P. Gong, R. Xiao, and X. Bai, “Master: Multi-aspect non-local network for scene text recognition,” _Pattern Recognition_ , p. 107980, 2021.
  * [56] J. Chen, B. Li, and X. Xue, “Scene text telescope: Text-focused scene image super-resolution,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 12 021–12 030.
  * [57] M. Awais, M. Naseer, S. Khan, R. M. Anwer, H. Cholakkal, M. Shah, M.-H. Yang, and F. S. Khan, “Foundational models defining a new era in vision: A survey and outlook,” 2023.
  * [58] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” 2021.