# Joint Visual Semantic Reasoning: Multi-Stage Decoder for Text Recognition

First Author
Institution1
Institution1 address
<EMAIL_ADDRESS>Second Author
Institution2
First line of institution2 address
<EMAIL_ADDRESS>
###### Abstract

Although text recognition has significantly evolved over the years, state-of
the-art (SOTA) models still struggle in the wild scenarios due to complex
backgrounds, varying fonts, uncontrolled illuminations, distortions and other
artifacts. This is because such models solely depend on visual information for
text recognition, thus lacking semantic reasoning capabilities. In this paper,
we argue that semantic information offers a complimentary role in addition to
visual only. More specifically, we additionally utilize semantic information
by proposing a multi-stage multi-scale attentional decoder that performs joint
visual-semantic reasoning. Our novelty lies in the intuition that for text
recognition, prediction should be refined in a stage-wise manner. Therefore
our key contribution is in designing a stage-wise unrolling attentional
decoder where non-differentiability, invoked by discretely predicted character
labels, needs to be bypassed for end-to-end training. While the first stage
predicts using visual features, subsequent stages refine on-top of it using
joint visual-semantic information. Additionally, we introduce multi-scale 2D
attention along with dense and residual connections between different stages
to deal with varying scales of character sizes, for better performance and
faster convergence during training. Experimental results show our approach to
outperform existing SOTA methods by a considerable margin.

##  1 Introduction

Text recognition has been a popular area of research [baek2019wrong,
li2019show, shi2018aster] for decades owing to its wide range of commercial
applications [long2018scene], from translation apps in mixed reality, street
signs recognition in autonomous driving to assistive technology for the
visually impaired [biten2019scene], to name a few. Significant progress in
fundamental deep learning components [jaderberg2015spatial,
bahdanau2014neural] alongside sequence-to-sequence learning frameworks
[shi2018aster, li2019show, qiao2020seed], have boosted unconstrained word
recognition accuracy (WRA) in recent times. Despite all such developments,
state-of-the-art text recognition frameworks [bai2018edit, cheng2017focusing,
litman2020scatter, wan2020vocabulary, yu2020towards] still struggle in wild
scenarios [baek2019wrong, long2018scene] due to complex backgrounds, varying
fonts, uncontrolled illuminations, distortions and other artifacts. While
machines struggle with a combination of these challenges, humans recognise
them easily via joint visual-semantic reasoning. Therefore, the question in
focus is – how to develop a visual-semantic reasoning skill for text
recognition?

Figure 1: Compared to existing attentional decoder [shi2018aster, li2019show]
architecture, we design a novel multi-scale attention decoder for text
recognition which is _unpacked in a stage-wise manner._ The problem of non-
differentiability due to discrete-character prediction is bypassed by
straight-through Gumbel-Softmax operator [jang2016categorical], such that the
later stages can learn refining strategy over the previous prediction in an
end-to-end differentiable way, with joint visual-semantic information.

State-of-the-art text recognition systems [baek2019wrong] mostly rely on
extracted visual features to recognize a word image as a machine readable
character-sequence. Follow-up efforts have been made towards improving
reasoning ability by increasing the depth of convolutional feature extractor
[cheng2017focusing] having larger receptive fields, or introducing pyramidal
pooling [wan2020vocabulary] and stacking multiple Bi-LSTM layers
[litman2020scatter]. Despite all these attempts that merely lead towards a
better context modeling [baek2019wrong], a semantic reasoning potential
[chen2018iterative] is largely missing beyond enriching the visual feature. In
wild scenarios, a word image might be blurred, distorted, partly noisy or have
artifacts, making recognition extremely difficult using visual feature alone.
In such cases, we humans first try to interpret the easily recognizable
characters using visual cues alone. A semantic reasoning skill is then applied
to decode the final text by jointly processing the visual and semantic
information from previously recognized character sequence. Motivated by this
intuition, we propose a novel _multi-stage prediction_ paradigm for text
recognition. Here the first stage predicts using visual cues, while subsequent
stages refine on top of it using joint visual-semantic information, by
iteratively [chen2018iterative, CarreiraMalikCVPR2016] building up the
estimates.

Designing this joint visual-semantic reasoning framework for text recognition
is non-trivial. One might argue that attentional decoder being a sequence-to-
sequence model, encapsulates the character dependency [shi2018aster,
li2019show, qiao2020seed] and serves the purpose of semantic reasoning.
However, due to its auto-regressive nature [bahdanau2014neural], only those
characters predicted previously, could provide semantic context at a given
step, thus making the semantic context flow unidirectional during inference.
While semantic context becomes negligible towards the initial steps, one wrong
prediction here would deal a cumulative adverse impact on the later steps
(which stays unrefined due to single stage prediction). Therefore, this single
stage attentional decoder fails to model the global semantic context, leaving
joint visual-semantic reasoning unaccomplished. To explore the entire global
semantic context, we need the completely unrolled prediction from first stage,
upon which we can build up the global semantic information. Hence as our first
contribution we propose a multi-stage attentional decoder (see Figure 1),
where we build up global semantic reasoning on the initial estimate of first
stage, which is further refined by subsequent stages.

Let us consider the word ‘aeroplane’. For a single stage attentional decoder,
if the model predicts ‘n’ instead of ‘r’, ‘aen’ would adversely affect rest of
the prediction, without any chance of refinement (being single stage). Also,
it holds almost negligible semantic context while predicting the first few
characters. Considering we unroll the prediction stage-wise, if a character is
predicted wrongly, like ‘aenoplane’, rest of the characters provide
significant context as semantic information. This helps in refining ‘n’ to ‘r’
during the later stages coupled with visual information.

Moreover, obtaining the prediction from earlier stages, needs a non-
differentiable argmax operation [jang2016categorical] as characters are
discrete tokens. This leads to an inefficient modelling of influence of a
prior stage on the next predictions. An apparent approach here might be to
adapt teacher forcing [lamb2016professor] for the later stages during
training. The later stages intend to learn _how to refine_ the initial (might
be incorrect) hypothesis towards a correct prediction. This motivation however
is defeated on feeding exact ground-truth labels as teacher forcing for
subsequent stages. Consequently, we make use of Gumbel-Softmax operation
[jang2016categorical] bypassing non-differentiability, and making the network
end-to-end trainable even across stages.

In summary our contributions are: First and foremost, we propose a multi-stage
character decoding paradigm with stage-wise unrolling. While the first stage
predicts using visual features, subsequent stages refine on-the-top of them
using joint visual-semantic information. Secondly, we employ a Gumbel-softmax
layer to make visual-to-semantic embedding layer differentiable. The model
thus learns its refining strategy from initial to final prediction in an end-
to-end manner. Thirdly, from the architectural design, we introduce multi-
scale 2D attention to deal with varying scales of character size, and
empirically found adding dense and residual connection between different
stages stabilize training for better performance leading to outperforming
other state-of-the-arts significantly on benchmark datasets.

##  2 Related Works

Text Recognition: While connectionist temporal classification (CTC) layer
[GravesICML2006] does not model dependency in the output character space
[shi2016rare], an attention based decoder [shi2018aster] encases language
modeling, weakly supervised character detection and character recognition in a
single paradigm. Following some seminal works [shi2018aster,
lee2016recursive], attention based decoder became state-of-the-art pipeline
for text recognition which includes four successive modules: i) a
rectification network [shi2018aster] to simplify irregular text image, ii)
convolutional encoder for feature extraction, iii) Bi-LSTM layer for context
modeling, and iv) an attentional decoder predicting the characters
autoregressively.

Furthermore, the motivation of recent followed-up works can broadly be
classified into following directions: (i) improve rectification network by
introducing iterative pipeline [ESIR2019] and modelling geometrical attributes
[yang2019symmetry] of text image; (ii) four directional feature encoder
[cheng2018aon] for better convolutional feature extraction; (iii) improving
attention mechanism by extending to 2-D attention [li2019show] and hard
character localized annotation [cheng2017focusing, liao2019scene], to better
guide the attention based character alignment process. (iv) Recently, stacking
multiple Bi-LSTM layers [litman2020scatter] and pyramidal pooling
[wan2020vocabulary] on convolutional feature were employed towards the goal of
_better context modeling_. These approaches however mainly focus on exploiting
visual features, via different architectural modifications [yanplugnet,
yue2020robustscanner] on top of Shi _et al_. [shi2018aster], but mostly lack
in any semantic reasoning capabilities.

Although some works claim to model semantic reasoning by stacking additional
Bi-LSTM layers [litman2020scatter, wan2020vocabulary], it only helps in
modelling better contextual information without having actual reasoning
potential. In this context, word-embeddings [qiao2020seed] from pre-trained
language model were used to initialize the hidden state of attentional
decoder, however we are skeptical towards this. For e.g. two related words
“Chair” and “Table” may lie close in word-embedding space, but their character
combination is way apart, thus questioning usage of word-embedding for text
recognition. Yu _et al_.’s [yu2020towards] architectural design in this
direction, gets severely limited on using argmax operation in visual-to-
semantic embedding layer which invokes non-differentiability, restricting
gradient flow from final prediction layer through this block; making learning
deficient (Section 4.1). To our believe, ours is the first work employing a
fully-differentiable semantic reasoning block that caters multi-stage refining
objective for discrete character sequence prediction task.

Multi-Scale Learning: This learning paradigm is widely prevalent in object
detection, recognition [kong2016hypernet, bell2016inside, liu2016ssd] and
semantic segmentation [long2015fully, hariharan2015hypercolumns]. Instead of
solely relying on low resolution, semantically strong features, multi-scale
frameworks like MSCNN [cai2016unified], DAG-CNNs [yang2015multi], and FPN
[lin2017feature] combine them with high-resolution, semantically weak features
for object detection across a diverse range of shape and sizes. We couple
multi-scale feature to generate multi-scale attention vectors for text
recognition.

Multi-Stage Frameworks: In spite of computational overhead, multi-stage
framework has gained popularity in computer vision task like pose estimation
[ramakrishna2014pose], object detection [chen2018iterative] and action
recognition [farha2019ms] for significantly improved performance.
Specifically, Convolutional Pose Machine [wei2016convolutional] is one of the
most successful and widely accepted multi-stage deep frameworks for pose-
estimation.

Joint Visual-Semantic Learning: Reasoning trails back to symbolic approaches
[newell1980physical], that used abduction [hobbs1993interpretation], and
deduction to relate abstract symbols. In computer vision, Graph Convolution
Networks [kipf2016semi] achieved success in object detection
[chen2018iterative], image-text matching [li2019visual], image captioning
[krishna2017visual], semantic navigation [yang2018visual] by generating
enhanced visual features with local and global semantic relationship. In our
work, we use transformer network [vaswani2017attention] for joint visual
semantic reasoning in subsequent stages of our multi-stage multi-scale
attentional decoder.

Figure 2:  With the extracted context-rich holistic feature ($h_{L}$) and
multi-scale feature maps ($B_{L},B_{L-1},B_{L-2}$), a multi-stage attentional
decoder predicts the character sequences in consecutive stages. Once the
previous stage’s decoder completely unrolls itself across time, the current
one begins prediction using the global-semantic information from previous
stage’s predicted character sequence, coupled with visual features refined via
joint visual-semantic reasoning. [$\oplus$ = concatenation ; $\otimes$ =
residual connection with LayerNorm]. Best viewed when zoomed.

##  3 Methodology

Overview: Given an input word image $I$, we intend to predict the character
sequence $Y=\\{y_{1},y_{2},...,y_{T}\\}$, where $T$ denotes the variable
length of text. Our framework is two-fold: _(i)_ a _visual feature extractor_
extracts context-rich holistic feature and multi-scale feature maps. _(ii)_
Following that, a _multi-stage attentional decoder_ builds up the character
sequence estimates in a stage-wise successive manner. While dealing with
irregular/curved word images [yang2019symmetry, cheng2018aon], image
rectification based approach [yang2019symmetry] often encounters difficulties
[cheng2018aon, liao2019scene]. To do away with the burden of adding a separate
sophisticated rectification network entirely, we follow a 2D attention
mechanism [li2019show] that helps to localize individual character in a
weakly-supervised manner during decoding.

###  3.1 Visual Feature Extraction

We adopt ResNet from [shi2018aster] as a backbone convolutional network to
extract visual features from input image. To deal with characters of varying
scales, we extend to multi-scale architecture for text recognition, with the
help of Feature Pyramid Networks [lin2017feature] which makes every resolution
level semantically strong using lateral connections. Let a feature-map from
particular scale be represented as $B_{l}\in\mathbb{R}^{H_{l}\times
W_{l}\times D}$; where $l=L$ denotes deepest residual block having lowest
resolution but highest level semantics. $H_{l}$ and $W_{l}$ are the height and
width of the feature map from respective scales which depend on the
accumulated strides of successive pooling layers, with all scales having $D$
channels uniformly [lin2017feature]. To balance between computational ease and
performance gain, we consider ${l=\\{L,L-1,L-2\\}}$ through empirical
validation. Visual features have two components, _(i)_ _multi-scale feature-
maps_ $\mathbf{\\{B_{L},B_{L-1},B_{L-2}\\}}$ which acts as context for 2D
attention in the later decoding process. _(ii)_ The _holistic feature_
$\mathbf{h_{L}}$, used to initialize the initial state of first stage decoder.
This $\mathbf{h_{L}}$ is recognised as the final hidden state of a 2-layer Bi-
LSTM which takes in a sequential feature ($W_{L}\times D$), obtained from
column-wise max-pooling on feature-map $B_{L}$ from deepest residual block
(ensuring height stays unity), followed by reshaping.

###  3.2 Joint Visual-Semantic Reasoning Decoder

Overview: Let the prediction from $s^{\mathrm{th}}$ stage decoder be denoted
as $Y^{s}=\\{y_{1}^{s},y_{2}^{s},...,y_{T}^{s}\\}$. Specifically, the first-
stage decoder relies only on the extracted feature. Subsequent stages
additionally use _global semantic information_ that is built on top of the
initial estimate, in a stage-wise decoding paradigm. For completeness, we
first describe basic attentional decoder in a generalized fashion (ignoring
stage notation). Later on we particularly illustrate the design for first
stage ($s=0$) vs. later stages ($s\geq 1$).

####  3.2.1 Attentional Decoder Background

Text recognition framework aims to model conditional distribution
$\mathrm{P(Y|I)}$, which can be factorized as
$\mathrm{P(Y|I)=\prod_{t=0}^{T}P(y_{t}|\mathcal{V},y_{<t})}$ where each
character output $y_{t}$ is modelled via conditional distribution over
extracted visual information $\mathcal{V}$, and the history of previously
predicted characters $y_{<t}$ till then. The basic attentional decoder
[shi2018aster] models this factored conditional distribution using an auto-
regressive Recurrent Neural Network (RNN) as follows:

$P(Y|I)=\prod_{t=0}^{T}P(y_{t}|g_{t},H_{t-1},y_{t-1})\vspace{-0.25cm}$ (1)

Every time-step prediction $y_{t}$ is conditioned on three factors: (i)
$H_{t-1}\in\mathbb{R}^{d_{rnn}}$ : the previous hidden state of RNN that
captures the history knowledge of previously predicted characters
$y_{<t}=\\{y_{0},\cdots,y_{t-1}\\}$. (ii) The apparent influence of previously
predicted character $y_{t-1}$ upon predicting $y_{t}$, following character
modelling protocol. (iii) The glimpse vector $g_{t}$, that learns to encode
the visual information by attending a smaller _specific_ part of visual
feature, which is maximally relevant to predict the character $y_{t}$.
Technically, $\mathbf{g_{t}&=\psi(\mathcal{B},{Q_{t}})}$, where $\mathcal{B}$
is a spatial feature-map, encoding visual information from previous
convolutional network, and $Q_{t}$ acts as a query to locate the attentive
regions for predicting $y_{t}$. Mathematically put,

$\displaystyle\begin{cases}J=\mathrm{tanh}(W_{\mathcal{B}}\circledast\mathcal{B}+W_{H}\otimes
Q_{t})\\\ \alpha_{i,j}=\mathrm{softmax}(W_{attn}\otimes J_{i,j})\\\
g_{t}=\sum_{i,j}\alpha_{i,j}\cdot\mathcal{B}_{i,j}\;i=[1,..H],\;j=[1,..W]\end{cases}\vspace{-0.8cm}$
(2)

Here, “$\circledast$” and “$\otimes$” denote convolution and matrix
multiplication respectively. $W_{B}$, $W_{H}$, $W_{attn}$ are the learnable
weights. Usually, $\mathbf{Q_{t}=H_{t-1}}$ containing history of prediction
information is used as a query to locate $y_{t}$. Moreover, query vector
enriched in global semantic information (e.g. as in $s\geq 1$) could also be
used instead, for better performance. While calculating the attention weight
$\alpha_{i,j}$ at every spatial position $(i,j)$, we employ a convolution
operation with $3\times 3$ kernel $W_{\mathcal{B}}$ to consider the
neighborhood information in 2D attention mechanism.

The current hidden state $H_{t}$ is updated by:
$\mathbf{H_{t}={f_{rnn}}(H_{t-1};\;[E(y_{t-1}),\;g_{t}]))}$, where $E(.)$ is
character embedding layer with embedding dimension $\mathbb{R}^{128}$, and [.]
signifies a concatenation operation. Finally, we apply a final linear
classification layer having learnable weights ($W_{c}$, $b_{c}$) and giving
logits $l_{t}=\mathrm{\mathbf{F_{cls}}(H_{t});\;l_{t}\in\mathbb{R}^{|V|}}$
where $|V|$ denotes the character vocabulary size. The current step character
is obtained as: $\mathbf{P(y_{t})=\mathrm{\texttt{softmax}}(l_{t})}$.

####  3.2.2 Decoder Stage $\mathbf{s=0}$

Henceforth, we affix notation for specific decoder stage keeping earlier
mathematical notation intact. For the first stage decoder RNN $f_{rnn}^{0}$,
the initial hidden state is initialized from holistic visual feature:
$\mathrm{\mathbf{H_{0}^{0}=\mathrm{tanh}(W_{v}\otimes h_{L}+b_{v})}}$, with
$W_{v}$, $b_{v}$ being trainable parameters. This enriches $f_{rnn}^{0}$ with
holistic visual information, while
$\mathrm{\mathbf{g_{t}^{0}=\psi^{0}(B_{L},H_{t-1}^{0})}}$ augments with
localized character specific information. At every $t$-th time step, we obtain
the distribution over the output character space as
$\mathrm{\mathbf{P(y_{t}^{0})=\texttt{softmax}(F_{cls}^{0}(H_{t}^{0}))}}$ and
$\mathrm{\mathbf{H_{t}^{0}={f_{rnn}^{0}}(H_{t-1}^{0};\;[E(y_{t-1}^{0}),\;g_{t}^{0}])}}$.
The decoding process stops once the ‘end-token’ is predicted. Sequences having
variable length in batches are handled by zero-padding.

####  3.2.3 Decoder Stage $\mathbf{s\geq 1}$

Incorrect instances might exist in the prediction of preceding stage, which is
why the later stages should work towards refining erroneous predictions while
keeping the correct ones intact. While this seems similar to Language Model
(LM) based post-processing [RozovskayaCorrection] or Error Correction Network
[RozovskayaCorrection], our proposed stage-wise decoders are all coupled in an
end-to-end trainable deep architecture. Here, gradients can backpropagate
across stages during training, thus leading to learning better data driven
refining strategy _re-utilising_ the visual feature. The later stage decoders
$\mathrm{s\geq 1}$ are modelled as follows:

$P(Y^{s}|I)=\prod_{t=0}^{T}P(y_{t}^{s}|g_{t}^{s},H_{t-1}^{s},y_{t}^{s-1},\mu_{t}^{s})\vspace{-0.25cm}$
(3)

Fundamentally, there are three differences compared to basic attentional
decoder (Eqn. 1):

(i) $y_{t}^{s}$ is conditioned on _joint visual-semantic information_
$\mathbf{\mu_{t}^{s}=[\vartheta_{t}^{s},\chi_{t}^{s}]}$, where visual-part
comes from $\mathbf{\vartheta_{t}^{s}=\phi_{t}(H^{s-1})}$ and global semantic
part comes from $\mathbf{\chi_{t}^{s}=\omega_{t}(\mathrm{E}(Y^{s-1}))}$. Here,
$\phi(\cdot)$ and $\omega(\cdot)$ are _reasoning modules_ working on previous
stage’s character aligned visual feature $H^{s-1}\in\mathbb{R}^{T\times
d_{rnn}}$ and semantic characters $Y^{s-1}\in\mathbb{R}^{T\times|V|}$ feeding
through character embedding layer $E(\cdot)$ respectively. $\phi_{t}(\cdot)$
and $\omega_{t}(\cdot)$ represent $t$-th time step output for respective
module. Once the previous stage decoder completely unrolls itself, all
characters from $Y^{s-1}=\\{y_{1}^{s-1},y_{2}^{s-1},\cdots,y_{T}^{s-1}\\}$
being concurrently present, augments a global semantic information for
reasoning. The main motive of later stages is to learn a refinement strategy.
As we already obtain _character aligned visual-semantic feature_
$H^{s-1}=\\{H_{0}^{s-1},H_{1}^{s-1},\cdots,H_{T}^{s-1}\\}$ from the previous
stage, we employ a reasoning module to capture enhanced visual reasoning over
all the character aligned visual-semantic features from the previous stage.

(ii) For $g_{t}^{s}$, we additionally use joint visual semantic information
$\mathbf{\mu_{t}^{s}}$ for query; thus
$\mathbf{Q_{t}^{s}=[\mu_{t}^{s},H_{t-1}^{s}]}$, and higher resolution feature-
map is used as $\mathcal{B}=B_{L-s}$ (e.g., $B_{L-1}$ for $s=1$) to couple
multi-scale feature learning in a multi-stage decoder. Thus glimpse vector is
$\mathbf{g_{t}^{s}=\psi^{s}(B_{L-s},[\mu_{t}^{s},H_{t-1}^{s}])}$.

(iii) While $s=0$ acts following baseline attentional decoder (Eqn. 1), the
role for $s\geq 1$ is to learn _refining strategy_ over previous predictions.
Thus instead of feeding previous time-step prediction $y_{t-1}^{s}$, we feed
prediction from previous stage corresponding to the same time-step as
$y_{t}^{s-1}$.

Differentiable Semantic Space: Obtaining _discrete character token_ from
distribution over the character vocabulary $P(y_{t})$ requires non-
differentiable argmax operation. As our motivation lies in coupling multi-
stage decoder in a end-to-end trainable framework, we employ Gumbel-softmax
re-parameterisation trick [jang2016categorical] with Straight-Through (ST)
gradient estimator such that gradient can backpropagate across stages. This
empowers the model to learn _reasoning based refining strategy_ over previous
prediction. In Gumbel-softmax, discontinuous argmax operation is replaced by a
_differentiable_ softmax function. Given the output logits
$l_{t}^{s}=F_{cls}^{s}(H_{t}^{s})$ and $l_{t}\in\mathbb{R^{|V|}}$, the output
probabilities of choosing $j$-th character token are:

$p_{t,j}^{s}=\frac{\exp(l_{t,j}^{s}+g_{t,j}^{s})/\tau}{\sum_{j=1}^{j=|V|}\exp(l_{t,j}^{s}+g_{t,j}^{s})/\tau}\vspace{-0.2cm}$
(4)

where, $g_{t,j}^{s}$ represents Gumbel-noise [jang2016categorical], and $\tau$
is a temperature parameter. During forward pass, it generates one-hot vector
$y_{t}^{s}=\\{y_{t,1}^{s},y_{t,2}^{s},\cdots,y_{t,|V|}^{s}\\}=$ Gumbel-
Softmax$(l_{t}^{s})$ where
${y}_{t,i}^{s}=\mathds{1}_{[i\;=\;\mathrm{argmax}_{j}(p_{t,j}^{s})]}$. During
backward pass, it uses the continuous $p_{t,j}^{s}$, allowing backpropagation.
At inference, largest index in $l_{t}^{s}$ is chosen.

Visual-Semantic Reasoning: The visual and semantic reasoning functions
$\phi(\cdot)$ and $\omega(\cdot)$ are employed by Transformer module
[vaswani2017attention] that uses multi-headed self-attention mechanism to
gather global context information. In brief, given key (K), query (Q) and
value (V), attention is calculated as:
$\textit{Attention}(K,Q,V)=\textit{softmax}(\frac{QK^{\intercal}}{\sqrt{dim}})V$.
At each time step output $\phi_{t}(\cdot)$ and $\omega_{t}(\cdot)$, feature
representation is enriched by information from remaining time-steps and thus
long-range dependencies are modelled carefully. Semantic reasoning module
$\omega(\cdot)$ is pre-trained separately following BERT [devlin2018bert]
language model training topology. We mask out (also purposefully replace by
erroneous instances) certain input time steps and force to predict masked
token by a linear layer. This helps the model to learn better refining
potential using text-only data in advance.

Dense and Residual Design: Glimpse vector $g_{t}$ provide character localized
visual information. For later stages $g_{t}^{s}$ is computed based on joint
visual-semantic information so that more enriched representation can be
extracted. To take advantage from multiple stages, we add a _dense connection_
[huang2017densely] between computed current $g_{t}^{s}$ and preceding
$g_{t}^{<s}=\\{g_{t}^{s-1},\cdots,g_{t}^{0}\\}$ as :
$\mathrm{\mathbf{\overline{g_{t}^{s}}=W_{g}^{s}\otimes[g_{t}^{s},g_{t}^{s-1},\cdots,g_{t}^{0}]}}$,
where, $W_{g}^{s}$ is trainable parameter and implemented through $1\times 1$
convolution.

To sum up, we get a differentiable semantic space represented by one-hot
encoding as: $Y^{s-1}=\\{y_{t}^{s-1}\\}_{t=1}^{T}$, where
$y_{t}^{s-1}=\texttt{Gumbel-Softmax}(l_{t}^{s-1})$. Next, we calculate joint
visual-semantic feature $\mu_{t}^{s}$ and successively $g_{t}^{s}$ is
computed. Glimpse vector $g_{t}^{s}$ for $s\geq 1$ is enhanced by dense
connection to give $\overline{g_{t}^{s}}$. Now we update the hidden state of
current stage decoder RNN by:
$\mathrm{\mathbf{H_{t}^{s}={f_{rnn}^{s}}(H_{t-1}^{s};\;[E(y_{t}^{s-1}),\;\overline{g_{t}^{s}},\;\mu_{t}^{s}])}}$.
Excluding the final stage, we directly apply linear classifier to get:
$\mathrm{P(y_{t}^{s})=\texttt{softmax}(F_{cls}^{s}(H_{t}^{s}))}$.

For the final stage, $s=S$, we add a residual connection [he2016deep] between
initial $H_{t}^{0}$ and final $H_{t}^{S}$using LayerNorm
[vaswani2017attention] as follows:
$\overline{H_{t}^{S}}=\texttt{LayerNorm}(H_{t}^{S}+H_{t}^{0})$. The motivation
aligns with original residual convolutional architecture [he2016deep], but
here we integrate it to train deeper model with multiple attention decoder
stages for text recognition. The final prediction is obtained as
$\mathrm{\mathbf{P(y_{t}^{S})=\texttt{softmax}(F_{cls}^{S}(\overline{H_{t}^{S}}))}}$.
See Figure 2 for more clarity.

###  3.3 Learning Objective

We accumulate cross-entropy loss from all stages of attentional decoder to
train our text-recognition model.

$L_{C}=-\sum_{s=0}^{S}\sum_{t=1}^{T}\hat{y}_{t}\cdot
logP(y_{t}^{s}|H_{t}^{s})$ (5)

where $\mathrm{\hat{Y}=\\{\hat{y_{1}},\hat{y_{2}},\cdots,\hat{y_{T}}\\}}$ is
the ground-truth label. Furthermore, we use additional auxiliary linear
classifier over character aligned individual visual and semantic features
$\vartheta_{t}^{s}$ and $\chi_{t}^{s}$ respectively, that are processed
through reasoning modules. The next two losses could be thought of as an
auxiliary loss driving towards better convergence that enrich individual
character aligned feature with better visual and semantic information. This is
given by: ${L_{V}=-\sum_{s=1}^{S}\sum_{t=1}^{T}\hat{y}_{t}\cdot
logP(y_{t}^{s}|\vartheta_{t}^{s})}$ and
${L_{S}=-\sum_{s=1}^{S}\sum_{t=1}^{T}\hat{y}_{t}\cdot
logP(y_{t}^{s}|\chi_{t}^{s})}$. The network is thus trained using :
$\mathrm{L_{Total}=\lambda_{1}\cdot L_{C}+\lambda_{2}\cdot
L_{V}+\lambda_{3}\cdot L_{S}}$, where $\lambda_{1},\lambda_{2},\lambda_{3}$
are weights decided empirically.

##  4 Experiments

Datasets: Following the similar approach described in [ESIR2019,
yang2019symmetry, baek2019wrong, cheng2018aon, shi2018aster, moran], we train
our model on synthetic datasets (without any further fine-tuning) such as
SynthText [SynthText] and Synth90k [jaderberg2014synthetic], which holds 6 and
8 million images respectively. The evaluation is performed without fine-tuning
on datasets containing real images like: Street View Text (SVT), ICDAR 2013
(IC13), ICDAR 2015 (IC15), CUTE80, SVT-Perspective (SVT-P), IIIT5K-Words.
Street View Text dataset [WangICCV2011] consists of 647 images, most of which
are blurred, noisy or have low resolution. While ICDAR 2013 [ICDAR2013] has
1015 words, ICDAR 2015 contains a total of 2077 images of which 200 images are
irregular. CUTE80 [CUTE80] offers 288 cropped high quality curved text images.
SVT-Perspective [SVT-P] presents 645 samples from side-view angle snapshots
containing perspective distortion. IIIT5K-Words [IIIT5K-Words] distinguishes
itself by presenting randomly picked 3000 cropped word images.

Implementation Details: We use ResNet architecture from [shi2018aster] with
FPN heads having 256 channels in each multi-scale feature-maps. The kernel
size of intermediate pooling layers is so adjusted that
$B_{L},B_{L-1},B_{L-2}$ have spatial size of $4\times 25$, $8\times 25$, and
$16\times 50$ respectively. The hidden state size of two-layer encoder BLSTM
and each decoder LSTM is kept at 256. Semantic $(\omega)$ and visual $(\phi)$
reasoning blocks consist of 2 stacked transformer units [vaswani2017attention]
with 4 heads and hidden state size 256. The hidden units in attention block is
of size 128. A total of 37 classes are taken including alphatbets, numbers and
end-tokens; with the maximum sequence length (N) set to 25. We use ADADELTA
optimizer [baek2019wrong] with learning rate 1.0 and batch size 32. We resize
the image to 32x100 and train our model in a 11 GB NVIDIA RTX-2080-Ti GPU
using PyTorch. We first warm-up using single stage attentional decoder for 50K
iterations, and then train our proposed three-stage ($S=\\{0,1,2\\}$)
attentional decoder (ablation on optimal stages in Sec. 4.2) framework end-to-
end, for 600K iterations with $\lambda_{1},\lambda_{2},\lambda_{3}$ set to 1,
0.1, 0.1 respectively. Please note that the first stage is fed with one-time
step shifted ground truth label to accommodate teacher forcing in sequence
modeling, however, later stages are fed with model’s prediction from previous
stage in order to learn the data driven refining strategy.

Table 1: Comparison of proposed method with different state-of-the-art
methods.

Methods | Year | IIIT-5K | SVT | IC13 | IC15 | SVT-P | CUTE80 | Remarks
---|---|---|---|---|---|---|---|---
Shi _et al_. [Shi2015] | 2015 | 81.2 | 82.7 | 89.6 | - | 66.8 | 54.9 | • End-to-end trainable CNN + RNN + CTC.
Lee _et al_. [lee2016recursive] | 2016 | 78.4 | 80.7 | - | 90.8 | - | 42.7 | • Recursive CNN + RNN + Atten. decoder.
Shi _et al_. [shi2016rare] | 2016 | 81.9 | 81.9 | 88.6 | - | - | - | • Introduce rectification network for irregular images.
Cheng _et al_. [cheng2017focusing] | 2017 | 87.4 | 85.9 | 93.3 | 70.6 | 71.5 | 63.9 | • Learning to focus on character centre, but needs char. location label.
Liu _et al_. [liu2018charnet] | 2018 | 83.6 | 84.4 | - | 60.0 | 73.5 | - | • Rectify the distortion at individual character level.
Bai _et al_. [bai2018edit] | 2018 | 88.3 | 87.5 | 94.4 | 73.9 | - | - | • Edit distance based seq. dissimilarity modeled to handle noisy characters.
Liu _et al_. [liu2018feature] | 2018 | 89.4 | 87.1 | 94.0 | - | 73.9 | 62.5 | • Leverage rendering parameters of synth. word image generation for training.
Shi _et al_. [shi2018aster] | 2018 | 93.4 | 93.6 | 91.8 | 76.1 | 78.5 | 79.5 | • Improved rectification network by Thin-Plate Spline.
Cheng _et al_. [cheng2018aon] | 2018 | 87.0 | 82.8 | - | 68.2 | 73.0 | 76.8 | • Four directional convolutional feature extraction for irregular images.
Liao _et al_. [liao2019scene] | 2019 | 91.9 | 86.4 | 91.5 | - | - | 79.9 | • Segment individual character + discrete char. recog. and word formation.
Yang _et al_. [yang2019symmetry] | 2019 | 94.4 | 88.9 | 93.9 | 78.7 | 80.8 | 87.5 | • Models geometrical attributes of text for better images rectification.
Li _et al_. [li2019show] | 2019 | 95.0 | 91.2 | 94.0 | 78.8 | 86.4 | 89.6 | • Introduce 2D-attention to deal with irregular images.
Baek _et al_. [baek2019wrong] | 2019 | 87.9 | 87.5 | 92.3 | 71.8 | 79.2 | 74.0 | • Comparative study of different methods and insightful analysis.
Zhan _et al_. [ESIR2019] | 2019 | 93.3 | 90.2 | 91.3 | 76.9 | 79.6 | 83.3 | • Iterative image rectification.
Litman _et al_. [litman2020scatter] | 2020 | 93.7 | 92.7 | 93.9 | 82.2 | 86.9 | 87.5 | • Stacking more Bi-LSTM layers + gated fusion of visual-contextual feature.
Qiao _et al_. [qiao2020seed] | 2020 | 93.8 | 89.6 | 92.8 | 80.0 | 81.4 | 83.6 | • Tries to predict the word-embedding vector to initialise the state of decoder.
Yu _et al_. [yu2020towards] | 2020 | 94.8 | 91.5 | 95.5 | 82.7 | 85.1 | 87.8 | • Faster parallel decoding + semantic reasoning block (_non-differentiable_).
Our Baseline (Stage-0) | - | 88.0 | 84.9 | 90.4 | 74.5 | 75.3 | 82.6 |  • Joint visual-semantic reasoning through multi-stage decoding using multi-scale feature maps and differential semantic space.
Our Baseline (Stage-1) | - | 92.6 | 89.5 | 93.9 | 80.3 | 81.5 | 87.2
Proposed (Stage-2) | - | 95.2 | 92.2 | 95.5 | 84.0 | 85.7 | 89.7
Our Baseline (Stage-3) | - | 95.2 | 92.1 | 95.5 | 83.6 | 85.5 | 89.6

###  4.1 Result Analysis and Discussion

Table 1 shows our proposed method to surpass SOTA methods by a reasonable
margin. Every method’s salient contributions are briefly mentioned there as
well. In this section, we first describe the limitations of the existing or
alternative (naive) designs and then illustrate (_using_ IC15) how and why all
our design components/choices contribute towards superiority over others.

Figure 3: Examples showing how joint visual-semantic information could help to
recognise through refining over stages $(s=0\rightarrow s=1\rightarrow s=2)$,
shown left to right.

_[i] Limitation of previous attentional decoders_ : Existing methods relying
on unidirectional auto-regressive attentional decoders exhibit a bottleneck,
and its drawback becomes evident from the following scenario : An easily
recognizable character present towards the end of a word would fail to provide
any contextual semantic information towards recognizing some noisy character
present earlier. We on the contrary let the first stage completely unroll
itself. Thereafter the prediction of previous stage (even if certain time-
step’s character is incorrect) could be rectified in the subsequent stages
using joint visual-semantic information. Although SCATTER [litman2020scatter]
stacks multiple BLSTM layers on the top of baseline design from ASTER
[shi2018aster], both methods lack semantic reasoning as they barely enrich
_visual_ feature encoding. Examples from our stage-wise decoder are shown in
Figure 3.

_[ii] Significance of Differentiable Semantic Space:_ Improving semantic
reasoning for better text recognition was only considered by [yu2020towards]
and [qiao2020seed] among all SOTA methods. Although Qiao _et al_.
[qiao2020seed] proposed to use word embedding, such technique relies on
semantic meaning of a word instead of the required character sequence. For
example, the word “table” and “chair”, although semantically related have
character combinations that are way-apart. Therefore, we emphasise on
modelling character sequences instead, to help recognize a noisy character
based on two-way information passing. Even though Yu _et al_. [yu2020towards]
took this direction to some extent, their non-differentiable semantic-
reasoning block imposes a significant limitation. We alleviate that with the
help of gumbel-softmax [jang2016categorical] to develop a differentiable
semantic space and allow learning of multi-stage semantic reasoning. While the
use of _teacher forcing_ for later stages by feeding ground-truth label for
training multi-stage decoder might seem an alternative, empirical evidence
suggests otherwise. The third stage decoder obtains $74.4\%$ accuracy as
compared to $74.5\%$ accuracy (on IC15) in first stage – no practical gains.
Another straight-forward way is to use _straight-through estimator_
[bengio2013estimating], which simply copies gradients from argmax output to
the next input. However, this results in significant instability where later
stage performance drops by $3.9\%$ to $80.1\%$ due to discrepancies between
forward and backward passes resulting in much higher variance than gumbel-
softmax [jang2016categorical].

_[iii] Why not directly use logits instead of gumbel-softmax for semantic
reasoning_ : Feeding logits (probability distribution over character
vocabulary prior to $argmax$) from a previous stage to the next, is a
reasonable argument that would make everything differentiable and eliminate
the need for gumbel-softmax. However, it is important to remember that
characters are _discrete_ tokens [baek2019wrong]. Using logits requires one to
replace character embedding layer $\mathrm{E(\cdot)}$ by a simple FC layer.
Unlike $\mathrm{E(\cdot)}$ that picks up a particular row of a trainable
matrix based on discrete one-hot vector, a FC layer will give varying
representations for the same character sequence based on the _confidence_ of
predictions. We confirm this hypothesis of sub-optimality empirically, as
results drop from $84.0\%$ to $82.8\%$.

_[iv] Why use top-down attentional decoder_ : While low resolution and
semantically strong features are good for classification, tasks requiring
focus in local regions, such as object detection and semantic segmentation,
benefit even further when combined with high-resolution semantically weak
features found in shallower regions of a feature extractor [cai2016unified].
Although our first stage is similar to a basic attentional decoder focusing on
feature map of the last layer to benefit from rich semantic information, that
is more invariant to distortion, later stages (refining stages) combine higher
resolution feature-map from preceding layers. This not only handles varying
character size, but also verifies prior prediction by exploiting joint
information between high resolution feature and previous predictions to guide
the _refining_ process. This hypothesis is verified by contradiction, using
high-resolution semantically weak feature $B_{L-2}$ in $s=0$ and lower
resolution semantically strong features in later stages $s>1$. We observe
performance collapses to $72.1\%$ in IC15 dataset due to inability of high
resolution semantically weak features to output the initial estimates.

_[v] Significance of self-attention based Joint Visual-Semantic Reasoning_ :
To emulate _human-like_ inference, self-attention based reasoning functions
allow two way information passing across visual and semantic spaces to obtain
a joint visual-semantic context. Its significance could be empirically
understood by removing the visual reasoning block and modifying the
architecture accordingly, which drops result by $2.9\%$. A similar drop of
$4.8\%$ was observed when the semantic reasoning block was removed. On
removing both we observe $77.1\%$ accuracy – a significant drop of $6.9\%$
from our method (Table 2).

_[vi] Do multi-scale (resolution) feature maps help?_ We empirically validate
this by excluding multi-scale feature maps and use $B_{L}$, instead of
$B_{L-s}$, to calculate $g^{s}_{t}$ at every stage $s$. Such modification
drops performance by $2.7\%$ (against ours), to $81.3\%$, which highlights the
contribution of multi-scale feature maps in our method.

_[vii] Comparison with alternative _multi-scale_ attentional decoder designs_:
In text recognition, the only other work realising importance of multi-scale
information is by Wan _et al_. [wan2020vocabulary], where pyramid pooling was
used. Here visual feature maps from different spatial resolutions were
concatenated, which eventually harmed downstream tasks owing to the large
semantic gaps between such feature maps. Consequently, we introduce lateral
connections following Feature Pyramid Networks [lin2017feature], semantically
strengthening high-resolution levels for superior performance. Simply
employing pyramid pooling for all stages $s=\\{0,1,2\\}$ however, drops
performance by $2.1\%$ (against ours) to $81.9\%$ .

_[viii] Significance of Dense and Residual Connections:_ Beside improving
visual information flow in the forward pass, the residual connection between
initial $H_{t}^{0}$ and final $H_{t}^{S}$ ensures efficient gradient flow in
visual feature networks, accelerating convergence of the whole network.
Furthermore, the dense connection is used to adaptively learn a more
discriminative glimpse vector by combining its features from preceding stages
with the current one, thus stabilising the training of multi-stage multi-scale
attentional decoder. Removing dense connection ($g_{t}$ calculation) decreases
the performance by $1.6\%$, and removing residual connection decreases it by
$1.3\%$. On removing both we get an even larger drop of $1.9\%$. Faster
training is observed while using both dense and residual connections.

_[ix] Significance of Multiple Constraints:_ We design experimental setups
(see Table 2) that reveal the following observations: (a) imposing loss
$L_{C}$ only in the last stage harms the model, resulting in $73.1\%$
accuracy. We attribute this to the poor gradient flow across stages. (b)
Adding multi-stage $L_{C}$ loss results in $77.1\%$ accuracy, performing
closer to the proposed method. (c) Adding visual-semantic constraints $L_{V}$
and $L_{S}$ finally gives the best performance of $84.0\%$. This shows multi-
stage constraint is vital for training and convergence. The intuition behind
multiple constraints sources from multi-task learning, which ensures better
convergence, thus enriching individual character aligned feature, with better
visual-semantic information.

_[ix] Varying training data size:_ Following [luo2020learn], we also vary the
training size and evaluate our proposed framework compared to single stage
baseline and Yu _et al_. [yu2020towards] in Table 2. Significant overhead at
low data regime brings the superiority to our proposed method over others.

Table 2:  (Left) Effect of multiple constraints on IC15. (Right) Varying training data size. $L_{C}^{\prime}$: Last stage only, $L_{C}$: Multi-Stage, GAP: WRA margin against final performance. | $L_{C}^{\prime}$ | $L_{C}$ | $L_{V}$ | $L_{S}$ | IC15 | GAP
---|---|---|---|---|---
✓ | - | - | - | 73.1 | 10.9
- | ✓ | - | - | 77.1 | 6.9
- | ✓ | ✓ | - | 79.2 | 4.8
- | ✓ | - | ✓ | 81.1 | 2.9
- | ✓ | ✓ | ✓ | 84.0 | -
| Method | Syn | Syn | Syn | Syn
---|---|---|---|---
10K | 50K | 100K | 1M
Yu _et al_. [luo2020learn] | 21.7 | 37.7 | 51.2 | 67.4
Luo _et al_. [yu2020towards] | 13.3 | 32.1 | 47.3 | 63.7
Baseline (s=0) | 9.9 | 27.2 | 44.9 | 62.3
Proposed | 25.3 | 41.5 | 56.4 | 73.1

###  4.2 Further Analysis and Insights

_[i] Design of Visual-Semantic Reasoning Module:_ One can capture two-way
visual semantic information using (a) Bi-LSTM (b) Transformer
[vaswani2017attention] with multi-headed self-attention mechanism. Table 3
shows Transformer to outperform LSTM by $1.3\%$. Furthermore, pre-training
global semantic reasoning module $\omega(\cdot)$ using BERT [devlin2018bert]
like training topology, scores $0.9\%$ higher accuracy than without it.

Table 3: Significance of joint visual-semantic reasoning module and comparison with Language Models (LM). Methods | IC15 | GAP
---|---|---
Our Baseline (Stage-0) + LM-shallow | 74.3 | 9.7
Our Baseline (Stage-0) + LM-deep | 75.9 | 8.1
Joint Visual-Semantic using LSTM | 81.8 | 2.2
Joint Visual-Semantic using Transformer | 83.1 | 0.9
Transformer with Pre-Training Semantic reasoning | 84.0 | -

_[ii] Weight sharing across stages:_ The stage-wise attentional decoder has
five trainable modules, $\mathrm{F_{cls}(\cdot)}$, $\mathrm{E(\cdot)}$,
$\mathrm{\Phi}(\cdot)$, $\mathrm{\omega(\cdot)}$ and $\mathrm{\psi(\cdot)}$,
whose weights can either be shared across stages or have a separate model for
each stage. Using separate weights achieves $82.5\%$ accuracy on IC15, whereas
sharing across stages results in $82.3\%$. Interestingly using a separate
$\mathrm{F_{cls},\psi}$, and shared $\mathrm{E(\cdot),\;\Phi,\;\omega}$ gives
$84.0\%$, a $1.7\%$ rise, in contrast to sharing all weights – probably
because sharing parameters which are not _stage dependent_ reduce model
complexity and has better optimization.

[iii] _Computational Analysis:_ Each stage needs to unroll itself completely,
before the next starts processing. Hence, the performance gain comes at a cost
of extra computational expenses (analysis in Table 4), which is reasonable
given the superior performance over strong baselines. Even so, we experimented
with ResNet-101 as a backbone feature extractor, having similar number of
parameters and flops to ours. This naive stacking of multiple-layers lags by
8.9%, which accredits our gain to our novel design choice.

Table 4: Computational analysis of the proposed method. Method | GFlops | Params | CPU | IC15
---|---|---|---|---
Our Baseline (Stage-0) | 15.3 | 38M | 16.38ms | 74.5
Proposed Method | 22.5 | 44M | 26.31ms | 84.0

_[iv] Comparison with SOTA Language Model:_ We compare our framework with
state-of-the-art Language Modeling (LM) based post-processing techniques based
on librispeech text-corpus. Based on [gulcehre2015using] we adopt two
techniques: (a) Shallow Fusion that results in $74.3\%$ and (b) Deep Fusion
giving $75.9\%$ accuracy on IC15 (see Table 3).

_[v] Optimum Stages:_ The optimal value for the number of stages $s$ is found
empirically on IC15. For $s=1$ we have $80.3\%$ accuracy that improves at
$s=2$ to give $84.0\%$, but saturates at $s=3$ giving $83.6\%$. Hence we
consider $s=2$ to be optimal. This performance saturation could be attributed
to vanishing gradient problem which is addressed via residual/dense
connection, but still persists to some extent. Also, for $s>2$, the joint
visual-semantic information might reach its optimum, where the result
saturates. Please refer to _supplementary material_ as well.

##  5 Conclusion

We propose a novel joint visual-semantic reasoning based multi-stage multi-
scale attentional decoding paradigm. The first stage predicts from visual
features, followed by refinement using joint visual-semantic information. We
further exploit Gumbel-softmax operation to make visual-to-semantic embedding
layer differentiable. This enables backpropagation across stages to learn the
refining strategy using joint visual-semantic information. Experimental
results indicate the superior efficiency of our model.