# Convolutional Neural Networks with Gated Recurrent Connections

Jianfeng Wang, Xiaolin Hu Jianfeng Wang is with the Computer Science
Department, University of Oxford, Oxfordshire, United Kindom. E-mail:
<EMAIL_ADDRESS>X. Hu is with the Institute for Artificial
Intelligence, the State Key Laboratory of Intelligent Technology and Systems,
Beijing National Research Center for Information Science and Technology, and
Department of Computer Science and Technology, Tsinghua University, Beijing
100084, China. E-mail<EMAIL_ADDRESS>
###### Abstract

The convolutional neural network (CNN) has become a basic model for solving
many computer vision problems. In recent years, a new class of CNNs, recurrent
convolution neural network (RCNN), inspired by abundant recurrent connections
in the visual systems of animals, was proposed. The critical element of RCNN
is the recurrent convolutional layer (RCL), which incorporates recurrent
connections between neurons in the standard convolutional layer. With
increasing number of recurrent computations, the receptive fields (RFs) of
neurons in RCL expand unboundedly, which is inconsistent with biological
facts. We propose to modulate the RFs of neurons by introducing gates to the
recurrent connections. The gates control the amount of context information
inputting to the neurons and the neurons’ RFs therefore become adaptive. The
resulting layer is called gated recurrent convolution layer (GRCL). Multiple
GRCLs constitute a deep model called gated RCNN (GRCNN). The GRCNN was
evaluated on several computer vision tasks including object recognition, scene
text recognition and object detection, and obtained much better results than
the RCNN. In addition, when combined with other adaptive RF techniques, the
GRCNN demonstrated competitive performance to the state-of-the-art models on
benchmark datasets for these tasks. The codes are released at
https://github.com/Jianf-Wang/GRCNN.

###### Index Terms:

Gated recurrent convolution neural network (GRCNN), gated recurrent
convolution layer (GRCL), object recognition, object detection, scene text
recognition

##  1 Introduction

Over the past a few years, convolutional neural networks (CNNs), invented by
LeCun et al. three decades ago [1], have achieved great success in solving
many computer vision problems. Without using handcrafted features, a CNN takes
an image in pixels as input and extracts features automatically to accomplish
a certain task, and the entire process can be trained end-to-end. Different
CNN architectures have been proposed [2, 3, 4, 5, 6, 7] and the performance is
getting better and better.

In history, CNN has got many benefits from neuroscience. For instance, the two
standard operations in CNN, convolution and pooling, can date back to Hubel
and Wiesel’s discovery of simple cells and complex cells in the primary visual
cortex (Vl) of cats [8, 9]. A predecessor of CNN is Neocognitron [10] which
also uses the two operations. Several years ago, inspired by abundant
recurrent synapses within the same visual area such as V1 [11], a recurrent
CNN (RCNN) [12, 13] was invented, and the main idea is to incorporate
recurrent connections between neurons within the same convolutional layer. The
resulting layer is called recurrent convolutional layer (RCL). Its structure
is illustrated in Fig. 1a. A neuron receives inputs from its neighboring
neurons in the same layer besides the neurons in the preceding layer. For
clarity, only recurrent connections in the same feature map are plotted, but
in practice every neuron receives recurrent inputs from all feature maps in
the current layer. Similarly, every neuron receives feedforward inputs from
all feature maps in the preceding layer. Due to the recurrent connections,
object recognition becomes a dynamic process though the input is static. Since
neurons in the same layer can exchange information, the receptive field (RF)
of every neuron, defined as the region in the input which can influence the
activity of the neuron, will become larger than that of the neuron in a
conventional convolutional layer (Fig. 1). This agrees with neuroscience
findings to some extent, as a biological neuron’s activity can be modulated,
either suppressed or facilitated, by stimuli outside the neuron’s classical RF
[14, 15, 16]. In this sense, the RF induced by the feedforward connections in
Fig. 1a, i.e., the red square projected to the input space, can be called
classical RF while the RF induced by both feedforward and recurrent
connections can be called non-classical RF. In what follows, if not specified,
the RF in a model with recurrent connections refers to the non-classical RF.

Figure 1: Illustration of the RCL and GRCL. (a) Structures of the RCL and
GRCL. The black dots denote neurons. The red and green arrows denote the
feedforward and recurrent connections, respectively. With the gates denoted by
triangles operating on the recurrent connections, the figure illustrates a
GRCL; otherwise, it illustrates an RCL. (b) Desired RFs in the GRCL in
recognizing the pear and blueberry. _Left:_ The red squares and green squares
denote the classical RF and the desired non-classical RF of a neuron. _Right:_
Desired strength of the non-classical RF. For clarity, only the x coordinate
of RF is shown. The threshold $\rho$ determines the size of the effective RF,
i.e., only regions in the RF where the strength is larger than $\rho$ are
counted.

However, there is no restriction on this RF expanding process in the RCL. With
a large number of recurrent iterations, all neurons in an RCL would “see” the
whole input image and the contents captured in different iterations are
equally fed to the neuron, which is neither consistent with biological facts
nor advantageous for visual perception. Intuitively, to recognize the pear and
the blueberry in Fig. 1b, it is better to use different RF sizes. However, in
the RCL, the RF sizes of neurons increase with recurrent iterations regardless
of the input. We need a mechanism to automatically control the sizes of RFs,
or at least the strengths at different locations inside the RFs, to better
“understand” the input.

Towards this goal, we propose to add gates onto the recurrent connections to
modulate the RFs of neurons in the RCL, denoted by the triangles in Fig. 1a.
The resulting module is called gated RCL (GRCL). The outputs of the gates are
between 0 and 1, and the value is determined by the input to the GRCL, as well
as the states of the neurons in the previous recurrent iteration. Therefore
the outputs of the gates in different iterations could be different. The
recurrent contribution to the activities of neurons is multiplied with the
corresponding outputs of the gates before adding to the feedforward
contribution. If the gates always output 1 during recurrent iterations, the
GRCL degenerates to the RCL.

It is expected that the gates could learn to modulate the RFs such that the
neurons focus on regions with appropriate sizes for different inputs. Fig. 1b
_left_ shows desired RFs (green squares) for recognizing the pear and the
blueberry. This goal can be achieved if all gates in the former case are open
while some gates in later iterations in the latter case are closed (output 0).
It can be partially achieved if all gates are open in the two cases but the
gates output smaller values in the latter case than in the former case,
because such a setting makes the recurrent contributions diminish along with
iterations faster in the latter case. Fig. 1b _right_ illustrates the desired
strengths of the RFs in the two cases. Every green rectangle denotes the
expansion of the RF in one recurrent iteration. Here, uniform feedforward
kernels and recurrent kernels are assumed. That is why the strengths are step
functions instead of smooth curve functions. If a threshold $\rho$ (dashed
blue line) is used to determine the size of the effective RF, the size of the
effective RF would be smaller in the latter case.

The GRCL is a basic building block, and multiple GRCLs can be stacked into a
deep network called gated recurrent CNN or GRCNN. This paper presents the
implementation and experiments of GRCNN.

Some preliminary results have been presented at a conference [17]. The present
work extends the previous work in several aspects. First, we integrate some
advanced techniques developed in recent years into the model to further
improve its performance. Second, to determine the states of the neurons in the
current iteration, we integrate recurrent contributions in all previous
iterations instead of only the last iteration. Third, we allow the recurrent
weights to be different in different recurrent iterations. This relaxation
allows us to use more parameters for fair comparison with the state-of-the-art
models on certain computer vision tasks. In other words, some models presented
in the paper are actually not recurrent networks. In this case, the recurrent
connections become feedforward connections. The function of the gates on them
will be elaborated in later sections. Fourth, in the previous work, we only
evaluated the model on the optical character recognition (OCR) task, but in
this work we in addition evaluated the model on object recognition and object
detection tasks. Extensive experiments showed that GRCNN obtained very good
results in these tasks.

The remaining contents of the paper are organized as follows. In Section 2,
some closely related works are introduced. In section 3, the architectures of
RCNN and GRCNN are described. Sections 4, 5 and 6 present experimental results
on object recognition, OCR and object detection, respectively. Section 7
concludes the paper.

##  2 Related Work

###  2.1 CNNs with Skip-Layer Connections

Here we briefly review CNNs with skip-layer or shortcut connections because
the proposed model can be viewed as such a model when unfolded through time.

The highway network [18] is one of the earliest CNNs with skip-layer
connections. The main idea is to add a bypassing path to the convolutional
layer such that the output of the layer has two parts: one nonlinear
transformation part (conventional convolution and nonlinear activation) and
one bypassing part (the input of the layer). The two parts are modulated by a
gate, usually implemented as the logistic sigmoid function whose output is
between 0 and 1. When the gate on the bypassing path is 1, the input is
directly added to the output of the layer without attenuation. In contrast,
the residual network (ResNet) [4] advocates bypassing paths in CNN without
gates. The argument is that without input attenuation, the convolutional layer
learns the residual between the ideal mapping and the identity mapping from
input to output. The strategy turns out to be very effective and ResNet has
been applied to numerous applications. In addition, many improved versions
have been proposed. For example, a more efficient model ResNeXt [7] is
obtained by mainly substituting the conventional convolution in ResNet with
the grouped convolution. The densely connected network (DenseNet) [5] consists
of multiple blocks and in each block every layer has direct paths from all
layers below (this is the origin of its name) besides the nonlinear
transformation path. In contrast to the highway network and ResNet which
integrate two paths by summing their outputs, DenseNet integrates multiple
paths by the concatenation operation. This modification ensures that the upper
layers in a block do not lose the information of the input to the block. The
squeeze-and-excitation network (SENet) [19] proposes to scale the feature maps
of a convolutional layer using a squeeze-and-excitation operation. The
squeeze-and-excitation path and the direct path between the original
convolutional layer are integrated by feature map-wise product. An extension
of SENet is the Gather-Excite network (GENet) [20], which uses a different
product operation for integrating the two paths.

###  2.2 CNNs with Adaptive Receptive Fields

There are a few works investigating adaptive RF in neural networks. A typical
model refers to the Deformable CNN [21]. Unlike the traditional CNN, the
Deformable CNN has a changeable grid in the sense that each point in it can be
moved by a learnable offset, and then the convolution filters operate on these
moved grid points, resulting in the deformable convolution. The objective is
to encourage the new convolutional block to learn features from the best
location in the previous layer. The Saliency Network [22] performs input
distortions to “zoom-in” salient regions, which resembles deformable RF on
input images. In the active convolution [23], the RF does not have a fixed
shape and can take diverse forms for convolutions. Inspired by the human
visual system, a novel module is proposed to be integrated into CNN, named
Receptive Field Block [24]. It utilizes multi-branch pooling with varying
kernels corresponding to RFs of different sizes, and then uses dilated
convolution to control their eccentricities, and reshapes them to generate
final representation. Self-attention is considered to be an alternative to the
convolutional layer [25]. The multi-head self-attention layer can also
implement a dynamic receptive field, since each head can attend a value at any
pixel shift and form different grid patterns. The Selective Kernel Network
(SKNet) [26] consists of several ”selective kernel” (SK) convolutions. The SK
convolution is also inspired by the adaptive RF sizes of neurons, which is
implemented by three operators, i.e., Split, Fuse and Select. Our method
implements adaptive RFs in a different way, i.e., using gates to control the
strength of different points in RFs of the RCNN [12].

###  2.3 Recurrent Neural Networks

RNNs are often used to model sequential data such as handwriting recognition
[27] and speech recognition [28]. Simple RNN has a notorious property, namely
“gradient vanishing”, which makes it difficult to train the RNN. To circumvent
this drawback, the long short term memory (LSTM) [29] and the gated recurrent
unit (GRU) [30] are proposed. Both LSTM and GRU adopt gates to control the
information flow and determine the memory that should be kept.

There are not many RNNs proposed to process images, because it is not apparent
how the build-in sequential process of an RNN could be used to process static
images. In 2015, a recurrent CNN (RCNN) was proposed to perform object
recognition [12]. The main idea is to introduce recurrent connections between
neighboring neurons in the same convolutional layer, and the resulting layer
is called RCL. Then the neurons’ activities are no longer static because the
neurons interact with others at each recurrent iteration. This is consistent
with biological neurons whose firing rates are not constants over time given a
stimulus [31]. The model was shown to be more robust to occlusions in object
recognition than feedforward models [32]. It was extended to a multi-scale
version for scene labeling [13]. It was also used in other applications
including speech recognition [33], action recognition [34] and OCR [35].

When unfolded through time, RCNN becomes a feedforward network with many skip-
layer connections. In the unfolded RCL, such skip-layer connections represent
identity mappings from the input to every layer above. Every layer integrates
the convolutional path and the identity mapping path using summation. GRCNN
differs from RCNN by introducing gates to control information flow in the
convolutional path, and it differs from the highway network [18] by not using
gates on the identity mapping path.

Another method to integrate RNN and CNN is to introduce recurrent connections
between different layers rather than within the same layer. The convolutional
deep belief network (CDBN) [36] is an earlier attempt. Every pair of adjacent
layers has both bottom-up and top-down connections. Instead of using recurrent
connections between adjacent layers, one can also use recurrent connections
from the top layer to the bottom layer of a deep model [37].

##  3 Model

Since the proposed model is based on the RCNN model [12, 13], for better
understanding, we first briefly introduce the architecture of RCNN, then
describe the proposed GRCNN in detail. As the RCNN and the GRCNN only replace
the convolutional layer in conventional CNN with the RCL and the GRCL,
respectively, we mainly describe the RCL and the GRCL. Note that the RCL
described below slightly differs from its original version [12, 13] as we
incorporate some later proposed techniques into it.

###  3.1 Recurrent Convolutional Layer and RCNN

The RCL [12, 13] is a module with recurrent connections in the convolutional
layer (Fig. 1a without the gates). The states of the neurons $x(t)$ evolve as
follows

$\displaystyle x(t)=f(w^{\text{F}}*u(t)+w^{\text{R}}*x(t-1))$ (1)

for $t\geq 0$, where “*” denotes convolution, $u(t)$ and $x(t-1)$ denote the
feed-forward input and recurrent input to the RCL, respectively,
$w^{\text{F}}$ and $w^{\text{R}}$ denote feed-forward weight and recurrent
weights, respectively, and $f(\cdot)$ denotes the activation function (usually
the ReLU function). We stress that $t$ indexes the processing steps in the
model and it does not suggest that the image is moving or in other dynamic
modes. Throughout the paper, we assume $x(t)=0$ for $t<0$. Therefore, when
$t=0$ in (1), we only have the first term in $f(\cdot)$.

Because any neuron’s activity is influenced by the neighboring neurons’
activities through recurrent weights $w^{\text{R}}$, as the recurrent
computation continues, the neuron would “see” larger and larger area in the
input. In other words, the sizes of RFs of neurons would increase unboundedly.

To learn the parameters, one needs to unfold the recurrent computation through
time in the RCL to construct a feedforward architecture where the weights
between layers are identical, i.e., all are $w^{\text{R}}$. The number of
recurrent iterations is a hyper-parameter and task-dependent. In addition, the
input $u(t)$ is present in each recurrent iteration, i.e., in each layer of
the time-unfolded architecture (Fig. 2a). Note that unfolding an RCNN with
multiple RCLs has different methods, which lead to different feedforward
architectures [13]. Here, we adopt the method in which RCLs are unfolded one
by one in the bottom-up direction. In other words, we unfold an RCL through
time and use the output in the last iteration as input to the layer above,
while the outputs in other iterations are not directly inputted to the layer
above. In this method, $u(t)$ does not change over time, and in what follows,
we will drop $t$ in the input.

Figure 2: Time-unfolded version of the RCL (a) and the GRCL (b) with $T=3$.
The diagram in (b) without the red arrows corresponds to the model presented
in the preliminary work [17].

To improve the performance of the RCNN, two modifications are made in this
work. First, we incorporate recently proposed batch normalization (BN) [38]
and pre-activation techniques [39]. The former is proposed to reduce the so-
called internal covariate shift (but a recent study suggests that its actual
effect is to smooth the landscape of the loss function [40]). The latter
advocates to apply BN [38] and ReLU before each convolution, instead of “Conv-
BN-ReLU” which is often used in prior works. Both techniques have been proved
very effective in deep neural networks. Then the states of the neurons evolve
as follows

$\displaystyle
x(t)=w^{\text{F}}*f(\text{BN}(u))+w^{\text{R}}(t-1)*f(\text{BN}(x(t-1))).$ (2)

In the original proposal [12, 13], $w^{\text{R}}(t)$ was identical for
different $t$.

Second, as explained in Introduction, we remove the weight sharing constraint
in the time-unfolded architecture. In other words, different $w^{\text{R}}(t)$
are used for different $t$. This is to use more parameters for fair comparison
with other models on certain tasks. However, no matter whether
$w^{\text{R}}(t)$ is shared or not across time steps, the statistics and the
parameters used for the affine transformation in BN are not shared throughout
this paper since $x(t-1)$ is not identical with different $t$. Note that
$w^{\text{F}}$ does not depend on $t$ and it still remains the same across
$t$. From the perspective of time-unfolded network, the RF size of neurons
always increases along the hierarchy, say, higher layer neurons have larger
RFs, no matter $w^{\text{R}}$ is shared or not. The only difference is that in
the weight sharing case, the increase can be viewed as a dynamic process of
the same set of neurons, while in the weight unsharing case we do not have
this view.

RCL (2) can be rewritten in a more general form,

$\displaystyle
x(t)=\mathcal{T}^{\text{F}}(u;w^{\text{F}})+\mathcal{T}^{\text{R}}(x(t-1);w^{\text{R}}(t-1)),$
(3)

where $\mathcal{T}^{\text{F}}$ and $\mathcal{T}^{\text{R}}$ denote the feed-
forward transformation with parameters $w^{\text{F}}$ and recurrent
transformation with parameters $w^{\text{R}}$ using the pre-activation
technique, respectively. For simplicity, BN and ReLU are omitted in equation
(3), and $\mathcal{T}^{\text{F}}$ is still implemented by a simple ”BN-ReLU-
Conv” layer throughout the paper. As for $\mathcal{T}^{\text{R}}$, it can be
implemented by different blocks, such as a simple “BN-ReLU-Conv”, a general
3-layer bottleneck block [4, 39], or a 3-layer bottleneck block with SK
convolution [26].

TABLE I: Test Error Rates of RCNN and GRCNN on CIFAR-10 (%).  Iterations | Feature maps | Tied weights |  | Untied weights
---|---|---|---|---
RCNN | Original GRCNN | Improved GRCNN |  | RCNN | Original GRCNN | Improved GRCNN
(2,2,2) | (128,128,128) |  5.81 $\pm$ 0.13 |  5.92 $\pm$ 0.10 |  5.83 $\pm$ 0.13 |  |  5.55 $\pm$ 0.11 |  5.71 $\pm$ 0.11 |  5.65 $\pm$ 0.04
(3,3,3) | (128,128,128) |  5.65 $\pm$ 0.09 |  5.80 $\pm$ 0.12 |  5.76 $\pm$ 0.07 |  |  5.39 $\pm$ 0.05 |  5.53 $\pm$ 0.12 |  5.46 $\pm$ 0.10
(4,4,4) | (128,128,128) |  5.76 $\pm$ 0.06 |  5.69 $\pm$ 0.11 |  5.65 $\pm$ 0.07 |  |  5.51 $\pm$ 0.14 |  5.30 $\pm$ 0.13 |  5.24 $\pm$ 0.07
(5,5,5) | (128,128,128) |  5.88 $\pm$ 0.11 |  5.57 $\pm$ 0.08 |  5.49 $\pm$ 0.12 |  |  5.62 $\pm$ 0.09 |  5.23 $\pm$ 0.09 |  5.16 $\pm$ 0.10

###  3.2 Original GRCL and GRCNN

In an RCL, the RF size of neurons increases unboundedly with recurrent
computation. Our primary goal is to introduce a mechanism to alleviate this
problem. The basic idea is to modulate the recurrent term by a gate before
adding it to the input term, and the resulting module is called gated RCL
(GRCL). Integrated with the pre-activation technique, the original GRCL in our
previous work [17] can be described as follows (Fig. 2b, without the red
arrows).

$\displaystyle
x(t)=\mathcal{T}^{\text{F}}(u;w^{\text{F}})+G(t)\odot\mathcal{T}^{\text{R}}(x(t-1);w^{\text{R}}(t-1))$
(4)

for $t\geq 0$, where $\odot$ denotes element-wise multiplication, and $G(t)$
is a gate whose outputs have the same dimension as
$\mathcal{T}^{\text{R}}(x(t-1);w^{\text{R}}(t-1))$ and $x(t)$,

$G(t)=\sigma(\mathcal{T}_{\text{g}}^{\text{F}}(u;w^{\text{F}}_{\text{g}})+\mathcal{T}_{\text{g}}^{\text{R}}(x(t-1);w^{\text{R}}_{\text{g}}(t-1)))$
(5)

for $t\geq 0$ with $\sigma$ being the logistic sigmoid function:
$\sigma(x)=1/(1+\exp(-x))$. In the equation,
$\mathcal{T}^{\text{F}}_{\text{g}}$ and $\mathcal{T}^{\text{R}}_{\text{g}}$
denote the feed-forward transformation and recurrent transformation using the
pre-activation technique, respectively, for determining the output of the
gate, and they have their own parameters $w^{\text{F}}_{\text{g}}$ and
$w^{\text{R}}_{\text{g}}(t)$. Note that $w^{\text{R}}_{\text{g}}(t)$ can be
shared across $t$ or not. For simplicity, $w^{\text{F}}_{\text{g}}$ and
$w^{\text{R}}_{\text{g}}(t)$ are implemented by a simple “BN-ReLU-Conv”
separately throughout the paper. The convolutional layers in
$w^{\text{F}}_{\text{g}}$ and $w^{\text{R}}_{\text{g}}(t)$ are a set of
$1\times 1$ convolutional filters in order to make the dimensions of the
transformed $u$ and $x(t-1)$ consistent with $G(t)$.

If tied weights across layers are used (as in [17]), (4) and (5) describe the
dynamics of a set of neurons. As explained before, the expansion of the RFs of
neurons is incurred by recurrent computation. By comparing (3) and (4), we can
see that the gate $G(t)$ weakens the recurrent contribution, which comes from
a larger area in the input than from the RF in the previous recurrent
iteration, because $G(t)<1$. If $G(t)\neq 0$ and a threshold is used to
delineate the effective RF, the size of the effective RF will depend on the
value of $G(t)$ (Fig. 1b). In the case of untied weights, the effect of $G(t)$
is to modulate the RFs of neurons in different layers, allowing them to be
adjusted adaptively according to the input. It should be noted that, for an
untied GRCL, the feedforward weights are still shared across the time steps.

Multiple GRCLs together with other types of layers such as pooling layers can
be stacked into a deep model. Hereafter, a CNN that contains a GRCL described
in (4) is called an original GRCNN.

###  3.3 Improved GRCL and GRCNN

However, the introduction of gates causes a new problem: the recurrent
contribution to the states of the neurons could be very small. According to
(4), only the recurrent contribution in the last iteration is explicitly added
to the feedforward contribution, and the result is propagated to the next
layer. In one extreme case, all gates in the last iteration output zeros, then
only the feedforward contribution is propagated to the next layer, which is
clearly not what we want. Even if none of the gates outputs zero, the
recurrent contributions in previous iterations are likely to decay quickly.
This is mainly due to the recursive computation of gates (4) whose outputs
fall into an interval ranging from 0 to 1.

To circumvent this deficiency, we propose to explicitly add recurrent
contributions in all previous iterations to the feedforward contribution:

$x(t)=\mathcal{T}^{\text{F}}(u;w^{\text{F}})+\sum_{n=1}^{t}{G(n)\odot\mathcal{T}^{\text{R}}(x(n-1);w^{\text{R}}(n-1))}$
(6)

for $t\geq 1$, where $G(n)$ is defined in (5). For $t=0$,
$x(t)=\mathcal{T}^{\text{F}}(u;w^{\text{F}})$. If the gates are closed at
certain time steps, the recurrent information computed at other time steps
still contributes to the state at the current time step $t$. It is easy to
verify that this modification corresponds to adding a few connections to the
structure of the original GRCL (Fig. 2b). Moreover, to save the number of
parameters, the convolutional layer in $w^{\text{F}}$,
$w^{\text{F}}_{\text{g}}$ and $w^{\text{R}}_{\text{g}}(t)$ are implemented by
the grouped convolution. This new version of GRCL is called improved GRCL.

Hereafter, a CNN that contains an improved GRCL is called an improved GRCNN.

##  4 Object Recognition

The first application of the proposed GRCNN is object recognition. We first
designed models for three small datasets and performed detailed analyses on
one of them, then designed models for a large dataset.

###  4.1 Datasets

CIFAR: We conducted experiments on two CIFAR datasets [41] which consist of
colored natural images with $32\times 32$ pixels. The images in the two
datasets are the same but the labels are different. The CIFAR-10 dataset has
10 classes and the CIFAR-100 dataset has 100 categories. The training and test
sets have 50,000 images and 10,000 images, respectively.

SVHN: The street view house numbers (SVHN) dataset [42] consists of $32\times
32$ colored digit images. There are 73257 images in the training set, 26032
images in the test set and 531131 images in the extra set. We followed the
common practice [4, 43, 5, 44] that the training set and the extra set were
combined to train the network.

ImageNet: The ILSVRC 2012 dataset [45, 46] contains around 1.2 million images
for training, and 50,000 images for validation. Those images cover 1000
different classes. We report the classification errors on the validation set.

TABLE II: Test Error Rates of Various Models on CIFAR and SVHN (%).

Model | Depth | Params | CIFAR10 | CIFAR100 | SVHN
---|---|---|---|---|---
All-CNN [47] | - | - | 7.25 | 33.71 | -
Deeply Supervised Net [44] | - | - | 7.97 | 34.57 | 1.92
Highway Network [18] | 19 | 2.3 M | 7.72 | 32.39 | -
FractalNet [48] | 21 | 38.6M | 5.22 | 23.30 | 2.01
FractalNet+Dropout+Drop-path [48] | 21 | 38.6M | 4.60 | 23.73 | 1.87
ResNet with Stochastic Depth [43] | 110 | 1.7M | 5.23 | 24.58 | 1.75
ResNet with Stochastic Depth [43] | 1202 | 10.2M | 4.91 | - | -
Wide ResNet [49] | 16 | 11.0M | 4.81 | 22.07 | -
Wide ResNet with Dropout [49] | 16 | 2.7M | - | - | 1.64
ResNet (pre-activation) [39] | 164 | 1.7M | 5.46 | 24.33 | -
ResNet (pre-activation) [39] | 1001 | 10.2M | 4.62 | 22.71 | -
DenseNet (k = 12) [5] | 100 | 7.0M | 4.10 | 20.20 | 1.67
DenseNet-BC (k = 12) [5] | 100 | 0.8M | 4.51 | 22.27 | -
DenseNet (k = 24) [5] | 100 | 27.2M | 3.74 | 19.25 | 1.67
ResNet [4] | 110 | 1.7M | 6.61 | - | -
ResNet (reported in [43]) | 110 | 1.7M | 6.41 | 27.22 | 2.01
SE-ResNet-110 [19] | 110 | 1.9M | 5.21 | 23.85 | -
SE-ResNet-110 [19] | 110 | 10.2M | 4.44 | 22.48 | 1.78
SENet-29[19] | 29 | 35.0M | 3.68 | 17.78 | -
ResNeXt-29, 16$\times$32d [7] | 29 | 25.2M | 3.87 | 18.56 | -
ResNeXt-29, 8$\times$64d [7] | 29 | 34.4M | 3.65 | 17.77 | -
SKNet-29 [26] | 29 | 27.7M | 3.47 | 17.33 | -
SKNet-110* [26] | 110 | 25.3M | 3.56$\pm$0.08 | 18.34$\pm$0.06 | 1.56$\pm$0.03
GRCNN-56 | 56 | 5.8M | 4.01$\pm$0.11 | 19.97$\pm$0.08 | 1.63$\pm$0.05
GRCNN-110 | 110 | 10.6M | 3.82$\pm$0.10 | 19.45$\pm$0.07 | 1.61$\pm$0.04
SK-GRCNN-110 | 110 | 25.7M | 3.42$\pm$0.04 | 18.18$\pm$0.09 | 1.55$\pm$0.02

* denotes our re-implementation.

TABLE III: Inference time of different models on the test set of CIFAR-10 (seconds). Model | Depth | Params | Time
---|---|---|---
ResNet [4] | 110 | 1.7M | 10.87
SE-ResNet-110 [19] | 110 | 1.9M | 11.47
SE-ResNet-110 [19] | 110 | 10.2M | 17.97
DenseNet (k = 12) [5] | 100 | 7.0M | 29.10
DenseNet-BC (k = 12) [5] | 100 | 0.8M | 24.38
GRCNN-110 | 109 | 10.6M | 22.82

###  4.2 Implementation Details on CIFAR and SVHN

Figure 3: Left: GRCNN with three GRCL blocks for object Recognition on CIFAR
and SVHN. The adjacent blocks are connected by a transition layer that is
responsible for downsampling. Right: The transition layer which is similar to
the bottleneck layer used for downsampling in [4, 39]. For clarity, batch
normalization layers and activation layers are omitted.

The overall network architecture for the three small datasets, CIFAR-10,
CIFAR-100 and SVHN, is shown in Fig. 3. The input image with size $32\times
32$ was fed into the first convolutional layer which consisted of 64
convolutional kernels with size $3\times 3$ and stride 1. It followed three
GRCLs interleaved with two “transition layers”. As Fig. 3 shows, each
transition layer is also implemented by a 3-layer bottleneck block, which is
similar to the bottleneck blocks used for downsampling in [4, 39]. The last
layer preceding the output layer was a global average pooling layer. For RCNN,
the GRCL blocks were replaced with the corresponding RCL blocks.

We trained the models using the conventional stochastic gradient descent (SGD)
with batch size 64. The training epochs were 40 and 300 on SVHN and CIFAR,
respectively. The learning rate was 0.1 initially, and it was multiplied by
0.1 at 50% and 75% of the total training epochs. For preprocessing, we
normalized the input data by using channel mean and standard deviation. On the
CIFAR datasets, we evaluated the models under the data augmentation protocol.
The augmentation techniques were mirroring and translation, which have been
broadly used [4, 43, 48, 5, 44]. As for SVHN, no data augmentation was used.
To avoid over-fitting, following previous works [43, 49] we inserted a dropout
layer in GRCNN. Similar to [5], the models saved at the end of training were
evaluated. Five independent runs were performed for every model we
investigated, and the mean error rates will be reported.

TABLE IV: GRCNN Configuration for ImageNet Classification.

Layer | Output size | GRCNN-55 | GRCNN-109 | SK-GRCNN-55 | SK-GRCNN-109
---|---|---|---|---|---
Convolution | $112\times 112$ |  $7\times 7$ conv, stride 2, feature maps: 64
Pooling | $56\times 56$ |  $3\times 3$ max pooling, stride 2
Convolution | $56\times 56$ |  $3\times 3$ conv, stride 1, feature maps: 64
GRCL #1 | $56\times 56$ | Iterations: 3 | Iterations: 3 | Iterations: 3 | Iterations: 3
[ |  $1\times 1$, 64
---
$3\times 3$, 64
$1\times 1$, 256
] [ |  $1\times 1$, 64
---
$3\times 3$, 64
$1\times 1$, 256
] [ |  $1\times 1$, 128
---
SK[M=2, G=32, r=16], 128
$1\times 1$, 256
] [ |  $1\times 1$, 128
---
SK[M=2, G=32, r=16], 128
$1\times 1$, 256
]
Transition #1 | $28\times 28$ | Transition Layer, output feature maps: 256
GRCL #2 | $28\times 28$ | Iterations: 3 | Iterations: 3 | Iterations: 3 | Iterations: 3
[ |  $1\times 1$, 128
---
$3\times 3$, 128
$1\times 1$, 512
] [ |  $1\times 1$, 128
---
$3\times 3$, 128
$1\times 1$, 512
] [ |  $1\times 1$, 256
---
SK[M=2, G=32, r=16], 256
$1\times 1$, 512
] [ |  $1\times 1$, 256
---
SK[M=2, G=32, r=16], 256
$1\times 1$, 512
]
Transition #2 | $14\times 14$ | Transition Layer, output feature maps: 512
GRCL #3 | $14\times 14$ | Iterations: 4 | Iterations: 22 | Iterations: 4 | Iterations: 22
[ |  $1\times 1$, 256
---
$3\times 3$, 256
$1\times 1$, 1024
] [ |  $1\times 1$, 256
---
$3\times 3$, 256
$1\times 1$, 1024
] [ |  $1\times 1$, 512
---
SK[M=2, G=32, r=16], 512
$1\times 1$, 1024
] [ |  $1\times 1$, 512
---
SK[M=2, G=32, r=16], 512
$1\times 1$, 1024
]
Transition #3 | $7\times 7$ | Transition Layer, output feature maps: 1024
GRCL #4 | $7\times 7$ | Iterations: 3 | Iterations: 3 | Iterations: 3 | Iterations: 3
[ |  $1\times 1$, 512
---
$3\times 3$, 512
$1\times 1$, 2048
] [ |  $1\times 1$, 512
---
$3\times 3$, 512
$1\times 1$, 2048
] [ |  $1\times 1$, 1024
---
SK[M=2, G=32, r=16], 1024
$1\times 1$, 2048
] [ |  $1\times 1$, 1024
---
SK[M=2, G=32, r=16], 1024
$1\times 1$, 2048
]
Classification | $1\times 1$ |  $7\times 7$ global average pooling
| 1000 dimensional fully-connected, softmax

Note that, as for each transition layer in SK-GRCNNs, the $3\times 3$
convolutional layer is replaced by a SKConv [26] with stride 2 for fair
comparison.

###  4.3 Model Analyses on CIFAR-10

To investigate the effects of components or hyper-parameters of the proposed
model on the classification performance on the CIFAR-10 dataset, a couple of
architectures were designed differing in the number of iterations $T$, with
and without gates, and so on (See Table I). The recurrent transformation
$\mathcal{T}^{\text{R}}$ of GRCL in each iteration is implemented by a simple
“BN-ReLU-Conv”. Therefore, in Table I, $(a,b,c)$ in the first column indicates
that the three GRCLs in Fig. 3 had $a,b$ and $c$ iterations, respectively.
$(a,b,c)$ in the second column indicates that the three GRCLs in Fig. 3 had
$a,b$ and $c$ feature maps, respectively. The group number of grouped
convolution layers used in the improved GRCNN is set to 16. For simplicity,
here the same $T$ was used in all three GRCLs or RCLs. Table I reports the
mean and standard deviation of the test error rates in percentage over five
independent runs for each architecture.

Effect of the gates. We first compared the original GRCNN described in Section
3.2 with the RCNN described in Section 3.1. Both models used tied weights,
i.e., $w^{\text{R}}(t)$ and $w^{\text{R}}_{\text{g}}(t)$ were identical across
$t$. The only difference is that the former had gates on the recurrent
connections while the latter did not. When the number of iterations $T$ in
each original GRCL increased from 2 to 5, the test error rate of the original
GRCNN decreased gradually (Table I). However, this trend was only observed in
the RCNN when $T\leq 3$. When $T>3$, the test error rate of the RCNN
increased. These results indicated that increasing RF may not always help the
RCNN, and sometimes it may even hurt the performance of the RCNN. Due to the
adaptive control of RF in GRCL, increasing $T$ always helped the original
GRCNN.

The training loss curves and test error rate curves of the RCNN and the
original GRCNN with $T=3,4,5$ are plotted in Fig. 4. With larger $T$, RCNN
converged slower and the final loss became higher. However, the original GRCNN
with different iterations had similar training curves and the converged losses
were all lower than those of RCNN. The results indicated that the original
GRCNN had better training performance than the RCNN in terms of speed and
accuracy. The test error rate curves indicated that, with the same $T$, the
original GRCNN always achieved lower test error than RCNN.

Figure 4: Loss curves (upper) and test error rate curves (lower) of the RCNN
with weight sharing and the original GRCNN with weight sharing. The horizontal
axis represents the training epoch.

Effect of the accumulated contributions over iterations. The original GRCL (4)
uses the gated recurrent contribution in the last iteration while the improved
GRCL (equation (6)) uses the gated recurrent contributions in all iterations.
With the same number of iterations in each GRCL, the improved GRCNN achieved
lower accuracies than the original GRCNN (Table I), which indicated the
advantage of the accumulated contributions over all iterations.

Effect of the number of recurrent iterations. When the number of iterations
increased from 2 to 5, the mean test error rate of the original GRCNN
decreased gradually from 5.92% to 5.57%, and the mean test error rate of the
improved GRCNN decreased gradually from 5.83% to 5.49%. The results suggested
that more iterations are more advantageous for both models.

Effect of weight sharing. We then trained and tested the RCNN, the original
GRCNN and the improved GRCNN with untied weights, i.e., $w^{\text{R}}(t)$ and
$w^{\text{R}}_{\text{g}}(t)$ were different for different $t$. The conclusions
about the effects of the gates, the accumulated contributions over iterations
and the number of iterations drew on the recurrent networks were also valid
(Table I). In addition, the architectures with untied weights always achieved
lower test errors than their counterparts with tied weights. This is
reasonable because without weight sharing, the model has much more parameters
and therefore higher capability to model the mapping from images to labels.

Behavior of the gates. Given an input image, how do these gates collaborate
with each other to give an output? We then analyzed the output values of the
gates in the GRCNN in the last row of Table I. Since there were three GRCLs
and each one had five iterations, the model had 15 gated layers. Note that
every feature map had one gate, then we had $128\times 15$ gates in total.
Fig. 5a,b show the output values of some gates given two sample input images.
It is seen that the gates in GRCL1 tended to output larger values while the
gates in GRCL3 tended to output smaller values. Statistical results over
10,000 input images verified this point, as shown in Fig. 5c. This makes sense
as neurons in lower layers had small RFs and they tended to incorporate more
context for better understanding of the context, while neurons in higher
layers already had large RFs and they tended to focus on local regions that
help the model to recognize specific objects. In addition, Fig. 5c shows that
the variances of the outputs increased from GRCL1 to GRCL3, which indicates
that the gates were more and more selective to inputs.

The stripe pattern in Fig. 5a,b indicates that the five gates in the same
GRCL, especially GRCL1 and GRCL2, at the same location (same index over 128
gates in a layer) had similar outputs. But this pattern was less prominent in
GRCL3. To provide a quantitative measure, we calculated the variance of the
outputs of the five gates with the same index in each GRCL given an input
image, then averaged over all 128 gates and 10,000 input images. The average
in GRCL3 was much larger than the average in GRCL1 and GRCL2 (Fig. 5d). The
results indicate that the gates in GRCL1 and GRCL2 in different iterations had
similar modulation effects to their inputs while the gates in GRCL3 had
different modulation effects to their inputs.

To sum up, the observations in these experiments are summarized as follows.

  * •

The improved GRCNN performed better than the original GRCNN and the RCNN.

  * •

More iterations in the improved GRCL led to better results.

  * •

The improved GRCNN performed better without weight sharing than with weight
sharing.

In order to compete with existing models in terms of accuracy, we need to
build deep GRCNNs which have many iterations. To save the number parameters,
for each GRCL, we only untie the parameters in the recurrent path, say,
$w^{R}$, but share the parameters of gates, say, $w_{\text{g}}^{R}$ in (5),
among different iterations. Hereafter, without specification, GRCNN refers to
this particular model.

Figure 5: Outputs of the gates in GRCNN given CIFAR-10 images. (a,b) The output values of the gates in different layers given two sample input images. Every layer had 128 gates but for clarity only the first 32 gates in every layer are shown. The 15 layers belong to three GRCLs (five per GRCL). Other layers in the model including intermediate layers between GRCLs are not shown. Note that the gates with the same index over 128 in different GRCLs do not have a direct relation. That is why the strips break at the boundaries of different GRCLs. (c) The mean output of the gates over 128 gates and 10,000 training images. Error bars indicate the standard deviation. (d) The mean of the variance of the gate outputs within each GRCL. TABLE V: Top-1 and Top-5 Error Rates on the ImageNet-2012 Validation with Different Popular Architectures (Single-Crop Test Error Rates) (%).  Model | GFLOPS | Param | Top-1 | Top-5
---|---|---|---|---
ResNet-50 [4] | 3.80 | 25.6M | 24.70 | 7.80
ResNet-101 [4] | 7.60 | 44.6M | 23.60 | 7.10
ResNet-152 [4] | 11.3 | 60.2M | 23.00 | 6.70
DenseNet-121 [5] | 2.90 | 8.4M | 25.02 | 7.71
DenseNet-169 [5] | 3.44 | 14.2M | 23.80 | 6.85
DenseNet-201 [5] | 4.39 | 20.0M | 22.58 | 6.34
SE-ResNet-50 [19] | 4.25 | 27.7M | 23.29 | 6.62
SE-ResNet-101 [19] | 8.00 | 49.2M | 22.38 | 6.07
SKNet-50 [26] | 4.47 | 27.5M | 20.79 | -
SKNet-100 [26] | 8.46 | 48.9M | 20.19 | -
RCNN-24 (our implem.) | 2.51 | 25.6M | 29.43 | 10.08
RCNN-55 (our implem.) | 3.36 | 25.8M | 32.92 | 13.64
GRCNN-55 | 3.73 | 24.9M | 23.12 | 6.52
GRCNN-109 | 7.80 | 45.1M | 21.95 | 5.95
SKNet-50 (our implem.) | 4.47 | 27.5M | 20.88 | 5.11
SKNet-100 (our implem.) | 8.46 | 48.9M | 20.27 | 4.89
SK-GRCNN-55 | 4.39 | 27.4M | 20.68 | 5.02
SK-GRCNN-109 | 8.45 | 50.0M | 20.09 | 4.86

TABLE VI: Compared with Two Models Having Adaptive Receptive Fields in Terms of Error Rate on the ImageNet-2012 Validation Set with Single-Crop Protocol (%).  Model | Top-1 | Top-5
---|---|---
Saliency Network ResNet-50 [22] | 23.52 | 6.92
Deformable ResNet-50 [21] | 23.40 | 6.80
GRCNN-55 | 23.12 | 6.52
Deformable GRCNN-55 | 22.91 | 6.48
Deformable v2 GRCNN-55 | 21.63 | 6.19
Saliency network ResNet-101 [22] | 22.32 | 6.19
Deformable ResNet-101 [21] | 21.60 | 5.80
GRCNN-109 | 21.95 | 5.95
Deformable GRCNN-109 | 21.35 | 5.74
Deformable v2 GRCNN-109 | 20.62 | 5.38

###  4.4 Comparison with Existing Methods on Small Datasets

In order to compare GRCNN with existing models, we experimented with three
deeper architectures as illustrated in Fig. 3:

  * •

GRCNN-56: The number of iterations (3, 5, 7), the number of feature maps (128,
160, 192).

  * •

GRCNN-110: The number of iterations (6, 9, 18), the number of feature maps
(128, 160, 192).

  * •

SK-GRCNN-110: The number of iterations (6, 9, 18), the number of feature maps
(192, 320, 448).

The number in the name of each architecture denotes the depth. Unlike the
settings in Sections 4.3, the $\mathcal{T}^{\text{R}}$ of GRCL in each
iteration was implemented by a general 3-layer bottleneck block [4, 39] or a
3-layer bottleneck block with SKConv [26], in order to build deep models. The
depth of each architecture can be calculated accordingly. For example, with
$T=3,5$ and $7$, the three GRCLs in Fig. 3 had $1+3\times 3$, $1+5\times 3$
and $1+7\times 3$ layers, respectively. There were two 3-layer transition
layers and one convolutional layer between the input and GRCLs as depicted in
Fig. 3, plus one output layer. Therefore, the depth of the first architecture
was 56. The depth of the other two architectures can be calculated in the same
way. For each architecture, the three numbers of feature maps listed above
denote the number of $3\times 3$ convolutional filters in bottleneck blocks of
each GRCL respectively. We defined the “expansion rate” of a 3-layer
bottleneck block as the ratio of the number of the third $1\times 1$
convolutional filters to the number of the first $1\times 1$ convolutional
filters. The expansion rate of the bottleneck block was set to 4 and 2 for
GRCNN and SK-GRCNN, respectively. The group number of $w^{\text{F}}$,
$w^{\text{F}}_{\text{g}}$ and $w^{\text{R}}_{\text{g}}(t)$ in GRCNN and SK-
GRCNN were all set to 16. The hyper-parameters of SKConv in SK-GRCNN-110 were
exactly the same as those used in the CIFAR experiment in [26].

The results of the three architectures on CIFAR-10, CIFAR-100 and SVHN are
presented in Table II. The mean and standard deviation of the test error rates
over five runs are reported. GRCNN-110 outperformed GRCNN-56 on CIFAR and SVHN
and SK-GRCNN-110 outperformed GRCNN-110. In addition, SK-GRCNN-110
outperformed SKNet-110 on the CIFAR datasets, indicates the usefulness of the
proposed gating mechanism. Finally, the SK-GRCNN achieved competitive results
compared with the state-of-the-art models.

Table III presents the inference time of GRCNN-110 and several popular
networks with similar depth listed in Table II on the CIFAR-10 test set
(tested on a single GeForce GTX TITAN X GPU with PyTorch [50]). The models
were run with batch size 64 and the total time on the test set was reported.
From Tables II and III, it is seen that though the GRCNN and DenseNet can
achieve lower error rates, they are not as efficient as the ResNet and its
variants. This is a shortage of both GRCNN and DenseNet.

TABLE VII: The Configuration of GRCNN in Scene Text Recognition.

Conv | GRCL | Conv | GRCL | Conv | GRCL | Conv | GRCL | Conv | GRCL | Conv
---|---|---|---|---|---|---|---|---|---|---
$3\times 3$ | $3\times 3$ | $2\times 2$ | $3\times 3$ | $2\times 2$ | $3\times 3$ | $2\times 2$ | $3\times 3$ | $2\times 2$ | $3\times 3$ | $2\times 2$
num: 32 | num: 32 | num: 32 | num: 64 | num: 64 | num: 128 | num: 128 | num: 256 | num: 256 | num: 512 | num: 512
sh:1 sw:1 | sh:1 sw:1 | sh:2 sw:2 | sh:1 sw:1 | sh:2 sw:2 | sh:1 sw:1 | sh:2 sw:1 | sh:1 sw:1 | sh:2 sw:1 | sh:1 sw:1 | sh:2 sw:1
ph:1 pw:1 | ph:1 pw:1 | ph:0 pw:0 | ph:1 pw:1 | ph:0 pw:0 | ph:1 pw:1 | ph:0 pw:1 | ph:1 pw:1 | ph:0 pw:1 | ph:1 pw:1 | ph:0 pw:1

sh and sw: the strides of the kernel along the height and width respectively;
ph and pw: the padding values of height and width respectively; num: the
number of convolutional filters.

###  4.5 Implementation Details on ImageNet

To verify the generalization of the proposed gating mechanism, we designed
GRCNN based frameworks with 4 GRCLs, including GRCNNs and SK-GRCNNs. The input
image size was $224\times 224$. The first layer was a convolutional layer that
had 64 convolutional kernels of size $7\times 7$ with stride 2 and zero-
padding 3. It was followed by a max pooling layer with size $3\times 3$ and
stride 2, and another convolutional layer that had 64 convolutional kernels of
size $3\times 3$ with stride 1 and zero-padding 1. Then four GRCLs were used
in sequence. Each of the first three GRCLs was followed by a transition layer.
In GRCNNs, $\mathcal{T}^{\text{R}}$ in each iteration and the transition layer
are implemented by the general 3-layer bottleneck block [4, 39] (See the
descriptions in sec.4.2). As for SK-GRCNNs, $\mathcal{T}^{\text{R}}$ and the
transition layer are implemented by the 3-layer bottleneck block with SK
convolution [26]. The group number of grouped convolution layers used in the
GRCNNs and SK-GRCNNs are all set to 32. See Table IV for details.

For comparison, we also evaluated two versions of RCNN with 55 and 24 layers,
respectively. The RCNN-55 was obtained by replacing GRCLs in the GRCNN-55 with
RCLs using general 3-layer bottleneck blocks. The RCNN-24 had a similar
architecture, but the iteration number of every RCL was 2, and only one “BN-
ReLU-Conv” layer with $3\times 3$ convolutional filters was used to implement
$\mathcal{T}^{\text{R}}$. The number of output feature maps for the four RCLs
were 128, 256, 512 and 1024, respectively.

During training, the input size is $224\times 224$ and we use the same data
augmentation method as [19]. The batch size was set to 256 and the models were
trained for 100 epochs. Top-1 and Top-5 error rates were reported. The
learning rate was set to 0.1 initially, and it was multiplied by 0.1 after
every 30 training epochs. Note that we cannot reproduce the results of SKNets
reported in the original paper [26]. To achieve the comparable results in the
original paper, our reproduced SKNets were trained for 120 epochs by using
cosine learning rate decay and Mixup [51], and SK-GRCNNs also follow this
setting. According to previous works [52], the weight decay and Nesterov
momentum were set to 10-4 and 0.9, respectively. The label smoothing strategy
[53] was used, and the network was initialized using the method in [54].

During testing, we performed single-crop testing, and the input size is
$224\times 224$. To be specific, for the single-crop protocol, the center crop
of a test image was used as the input.

###  4.6 Results on ImageNet

The test error rates of the GRCNNs and RCNNs are presented in Table V. First,
RCNN-55 performed worse than a shallower mode RCNN-24. This observation is
consistent with the results on CIFAR-10 (See Table I). Second, the three
GRCNNs performed much better than RCNN-55.

Table V also lists the results of some popular models including ResNet,
DenseNet, SE-ResNet and SK-Net. GRCNN outperformed other models which also
used the general 3-layer bottleneck block, i.e., ResNet and SE-ResNet. Note
that the performance gap between ResNet and GRCNN is remarkable, which
verifies the effectiveness of the proposed gating method since GRCNN is
similar to ResNet equipped with the proposed gates. When the SK convolution
was introduced, SK-GRCNN achieved a little bit higher accuracy than SK-Net as
well. Note that some models including ResNet with pre-activation [39],
Inception-V3 [53], Inception-ResNet-v1 [55], and Inception-V4 [55] used
different test settings, e.g., larger input crops than $224\times 224$ and
144-crop testing, and therefore are not compared in the table.

Finally, we compared the GRCNN with two models, the Deformable CNN [21] and
the Saliency Network [22], which also have adaptive RFs, and Table VI presents
the results. The GRCNN-55 and the GRCNN-109 obtained slightly better results
than the Saliency Network ResNet-50 and the Saliency Network ResNet-101,
respectively. The GRCNN-55 performed better than the Deformable ResNets-50
while the GRCNN-109 performed worse than the Deformable ResNets-101. The
performance of GRCNN was boosted by using the deformable convolutional layers
(Table VI). We built a deformable GRCNN by introducing deformable convolution
layers into the bottleneck of stage GRCL#4 in GRCNN. As for deformable v2
GRCNN, we followed [56] by applying deformable convolution layers at the
bottleneck of stages GRCL#2$\sim$GRCL#4 in GRCNN. The improved performance
indicates that our method and the deformable convolution are complimentary for
implementing adaptive RFs to improve the performance. This phenomenon was also
observed in [56], which introduces sigmoid layers to obtain the modulation
scalars for achieving effective RFs.

##  5 Scene Text Recognition

The second application of the proposed GRCNN is OCR. In this application,
there are multiple characters in one image to be recognized (Fig. 6).
Sometimes, the gap between characters is large and sometimes the gap is small.
It is desired that a model can automatically adjust its focus to the
characters. We expect that GRCNN suits this application because its built-in
gates enable it to do so.

Unlike object recognition, for which a backbone CNN alone suffices to obtain
good results, OCR requires additional techniques to process the character
sequences. Many sophisticated pipelines have been proposed [57, 58], where CNN
backbones play an important role. Since our aim was to examine the usefulness
of the GRCNN in this application, we replaced the original backbone CNNs with
the GRCNN in these frameworks and compared the results.

Figure 6: Two sample images from the ICDAR2003 dataset. In the first example,
for recognizing the character “h”, the classical RF (red square) and desired
non-classical RF (green) of GRCNN are shown. If the non-classical RF covers
unrelated information, such as the characters “p” and “i”, it will distract
the model. TABLE VIII: Scene Text Recognition Accuracies of Different Models.

Method | Data | SVT-50 | SVT | IIIT5K-50 | IIIT5K-1k | IIIT5K | IC03-50 | IC03-Full | IC03
---|---|---|---|---|---|---|---|---|---
ABBYY [59] | - | 35.0% | - | 24.3% | - | - | 56.0% | 55.0% | -
wang et al. [59] | - | 57.0% | - | - | - | - | 76.0% | 62.0% | -
Mishra et al. [60] | - | 73.2% | - | - | - | - | 81.8% | 67.8% | -
Novikova et al. [61] | - | 72.9% | - | 64.1% | 57.5% | - | 82.8% | - | -
wang et al. [62] | - | 70.0% | - | - | - | - | 90.0% | 84.0% | -
Bissacco et al. [63] | - | 90.4% | 78.0% | - | - | - | - | - | -
Goel et al. [64] | - | 77.3% | - | - | - | - | 89.7% | - | -
Alsharif [65] | - | 74.3% | - | - | - | - | 93.1% | 88.6% | -
Almazan et al. [66] | - | 89.2% | - | 91.2% | 82.1% | - | - | - | -
Lee et al. [67] | - | 80.0% | - | - | - | - | 88.0% | 76.0% | -
Yao et al. [68] | - | 75.9% | - | 80.2% | 69.3% | - | 88.5% | 80.3% | -
Rodriguez et al. [69] | - | 70.0% | - | 76.1% | 57.4% | - | - | - | -
Jaderberg et al. [70] | 90k | 86.1% | - | - | - | - | 96.2% | 91.5% | -
Su and Lu et al. [71] | - | 83.0% | - | - | - | - | 92.0% | 82.0% | -
Gordo [72] |  | 90.7% | - | 93.3% | 86.6% | - | - | - | -
Jaderberg et al. [73] | 90k | 93.2% | 71.1% | 95.5% | 89.6% | - | 97.8% | 97.0% | 89.6%
Baoguang et al. [57] | 90k | 96.4% | 80.8% | 97.6% | 94.4% | 78.2% | 98.7% | 97.6% | 89.4%
Chen-Yu et al. [35] | 90k | 96.3% | 80.7% | 96.8% | 94.4% | 78.4% | 97.9% | 97.0% | 88.7%
ResNet-BLSTM [17] | 90k | 96.0% | 80.2% | 97.5% | 94.9% | 79.2% | 98.1% | 97.3% | 89.9%
Original GRCNN-BLSTM [17] | 90k | 96.3% | 81.5% | 98.0% | 95.6% | 80.8% | 98.8% | 97.8% | 91.2%
GRCNN-BLSTM | 90k | 96.4% | 82.1% | 98.2% | 95.9% | 81.8% | 98.8% | 97.8% | 91.7%
FAN [74] | 90k + ST | 97.1% | 85.9% | 99.3% | 97.5% | 87.4% | 99.2% | 97.3% | 94.2%
AON [75] | 90k + ST | 96.0% | 82.8% | 99.6% | 98.1% | 87.0% | 98.5% | 97.1% | 91.5%
SAM [76] | 90k + ST | 98.6% | 90.6% | 99.4% | 98.6% | 93.9% | 98.8% | 98.0% | 95.2%
ESIR [77] | 90k + ST | 97.4% | 90.2% | 99.6% | 98.8% | 93.3% | - | - | -
SE-ASTER [78] | 90k + ST | - | 89.6% | - | - | 93.8% | - | - | -
ASTER with ResNet [58] | 90k + ST | 97.4% | 89.5% | 99.6% | 98.8% | 93.4% | 98.8% | 98.0% | 94.5%
ASTER with GRCNN | 90k + ST | 97.8% | 90.6% | 99.5% | 98.8% | 94.0% | 98.8% | 98.2% | 95.6%

”50”,”1k” and ”Full” appended to the dataset names denote the lexicon size
used for the lexicon-based recognition task. The dataset name without a
lexicon size indicates the unconstrained text recognition task.

###  5.1 Datasets

ICDAR2003: ICDAR2003 [79] contains 251 scene images and there are 860 cropped
images of the words. We performed unconstrained text recognition and
constrained text recognition on this dataset. Each image is associated with a
50-word lexicon defined by wang et al. [59]. The full lexicon is composed of
all per-image lexicons.

IIIT5K: This dataset has 3000 cropped testing word images and 2000 cropped
training images collected from the Internet [80]. Each image has a lexicon of
50 words and a lexicon of 1000 words.

Street View Text (SVT): This dataset has 647 cropped word images from Google
street view [59]. The 50-word lexicon defined by Wang et al [59] was used in
our experiment.

Synth90k: This dataset contains around 7 million training images, 800k
validation images and 900k test images [81]. All of the word images were
generated by a synthetic text engine and are highly realistic.

SynthText: This dataset was initially designed for text detection. However, it
has been widely used for scene text recognition task by cropping text image
patches according to ground-truth boxes. We followed [58] and cropped 7
million patches for training.

When evaluating the performance of our model on those benchmark datasets, we
followed the evaluation protocol in [59]. We performed recognition on the
words that contain only alphanumeric characters (A-Z and 0-9) and at least
three characters. All recognition results were case-insensitive.

###  5.2 Pipelines

We conducted experiments on two prevalent scene text recognition frameworks.
One is the Convolutional-Recurrent Neural Network (CRNN), and it consists of
three parts: feature extraction, feature map splitting and sequence modeling.
The feature extraction part can be performed by any CNN backbone to learn
visual features from the input image. The second part slices the feature maps
from left to right by column to form a feature sequence, which is then
processed by the feature modeling part. The other one is ASTER framework [58]
which is proposed to deal with the irregular scene text. It can be regarded as
introducing a text rectification module before the CRNN and attentional LSTM
decoders after the CRNN. Both frameworks are supervised by Connectionist
Temporal Classification (CTC) [82] loss during training. More details about
these two frameworks can be found in their original papers.

###  5.3 Implementation Details

The configuration of the GRCNN used in OCR is shown in Table VII. Note that
the backbones of CRNN, both ResNet and the original GRCNN, in [17] had 20
parameterized layers while the ResNet backbone in ASTER [58] had 50
parameterized layers. Therefore, in our experiment, we built GRCNNs by keeping
their depth comparable to those of backbones in these two pipelines for fairly
comparison.

To construct a GRCNN-based CRNN, the feature extraction part was performed by
the GRCNN and sequence modeling was performed by a Bidirectional LSTM with 256
units without dropout. The time steps for the GRCLs are set to 1, 3, 3, 3 and
3 separately, and only one “BN-ReLU-Conv” layer with $3\times 3$ convolutional
filters was used to implement $\mathcal{T}^{\text{R}}$ in each GRCL. The
number of feature maps is shown in Table VII. The group number of grouped
convolution layers used in GRCNN is set to 8. The input was a gray-scale image
with size 100$\times$32\. Before inputting to the network, the pixel values
were rescaled to the range (-1, 1). No further data augmentation methods were
used. The ADADELTA method [83] was used for training with the parameter
$\rho$=0.9. The batch size was set to 128 and training stopped after 300k
iterations. In order to fairly compare with our preliminary work [17] and most
other previous methods, all CRNN related models are only trained on Synth90k.

As for the ASTER pipeline [58], the time steps of the GRCLs were 1, 2, 4, 4
and 3. To construct a deep GRCNN, we used the general 3-layer bottleneck block
to implement $\mathcal{T}^{\text{R}}$ in each GRCL. Bottleneck blocks in each
GRCL contained 32, 64, 128, 256, and 512 $3\times 3$ convolutional filters
separately, and the expansion rate of those bottleneck blocks was set to 2.
The group number of grouped convolution layers used in GRCNN is set to 8. The
batch size was 64. The same ADADELTA method as described above was used for
training, but a global learning rate was initially set to 1.0, and decayed to
0.1 and 0.01 at 0.6M and 0.8M iterations, respectively. This pipeline was
trained on a combination of Synth90k and SynthText.

###  5.4 Comparison with Existing Methods

We evaluated GRCNN on several benchmark text recognition datasets in terms of
the two pipelines mentioned in sec.5.2, and we only pick the model which
performs the best on the validation set of Synth90k for testing.

Table VIII presents the results based on the CRNN pipeline. It is clear that
the GRCNN-based CRNN outperformed most previous methods. In addition, with the
comparable number of feature maps and depth, the GRCNN-based CRNN outperformed
the ResNet-based CRNN. The GRCNN-based CRNN outperformed previous state-of-
the-art methods which are only trained on Synth90k as well for both lexicon-
based and lexicon-free based recognition.

Table VIII also presents the results based on the ASTER pipeline. It is seen
that the ResNet-based model performed worse than the GRCNN-based model.
Moreover, the GRCNN-based ASTER achieved competitive results when compared
with previous state-of-the-art results.

##  6 Object Detection

The third application of the proposed model is object detection. In this
application, an image has multiple objects to be detected, and they may differ
significantly in scale. See examples in Fig. 7. The GRCNN may suit this
application because its neurons have adaptive RFs.

###  6.1 Datasets

The Microsoft common objects in context (MS COCO) [84] dataset contains 91
common object categories and 82 categories have more than 5,000 labeled
instances. The dataset consists of 2,500,000 labeled instances in 328,000
images in total. Instances are segmented individually to aid in precise object
localization. Several examples are shown in Fig. 7. Following the evaluation
protocol of MS COCO, we used the trainval35k set for training and evaluated
the results on the test-dev set.

TABLE IX: Detection Results of Anchor-based One-stage Detectors on MS COCO
Test-dev Set.

Method | Backbone network | AP | AP50 | AP75 | APS | APM | APL
---|---|---|---|---|---|---|---
YOLOv2 [85] | DarkNet-19 | 21.6 | 44.0 | 19.2 | 5.0 | 22.4 | 35.5
SSD512 [86] | VGG-16 | 28.8 | 48.5 | 30.3 | 10.9 | 31.8 | 43.5
SSD513 [87] | ResNet-101 | 31.2 | 50.4 | 33.3 | 10.2 | 34.5 | 49.8
DSSD513 [87] | ResNet-101 | 33.2 | 53.3 | 35.2 | 13.0 | 35.4 | 51.1
STDN513 [88] | DenseNet-169 | 31.8 | 51.0 | 33.6 | 14.4 | 36.1 | 43.4
FPN-Reconfig [89] | ResNet-101 | 34.6 | 54.3 | 37.3 | - | - | -
RefineDet512 [90] | ResNet-101 | 36.4 | 57.5 | 39.5 | 16.6 | 39.9 | 51.4
GHM SSD [91] | ResNeXt-101 | 41.6 | 62.8 | 44.2 | 22.3 | 45.1 | 55.3
CornerNet511 [92] | Hourglass-104 | 40.5 | 56.5 | 43.1 | 19.4 | 42.7 | 53.9
M2Det800 [93] | VGG-16 | 41.0 | 59.7 | 45.0 | 22.1 | 46.5 | 53.8
ExtremeNet [94] | Hourglass-104 | 40.2 | 55.5 | 43.2 | 20.4 | 43.2 | 53.1
CenterNet [95] | Hourglass-104 | 42.1 | 61.1 | 45.9 | 24.1 | 45.5 | 52.8
FSAF [96] | ResNeXt-101 | 42.9 | 63.8 | 46.3 | 26.6 | 46.2 | 52.7
CenterNet511 [97] | Hourglass-104 | 44.9 | 62.4 | 48.1 | 25.6 | 47.4 | 57.4
RefineDet512 [90] | ResNet-101 | 36.4 | 57.5 | 39.5 | 16.6 | 39.9 | 51.4
RefineDet512 | Deformable ResNet-101 | 37.9 | 58.1 | 40.2 | 18.5 | 40.4 | 51.6
RefineDet512 | GRCNN-109 | 37.6 | 58.7 | 40.5 | 18.0 | 41.3 | 52.1
RefineDet512 | Deformable GRCNN-109 | 38.4 | 59.4 | 41.0 | 18.5 | 42.1 | 52.3
RefineDet512 | SKNet-100 | 38.9 | 59.7 | 41.2 | 18.9 | 42.7 | 53.3
RefineDet512 | SK-GRCNN-109 | 39.5 | 59.9 | 41.9 | 19.8 | 43.4 | 53.7
RetinaNet800[98] | ResNet-101 | 39.1 | 59.1 | 42.3 | 21.8 | 42.7 | 50.2
RetinaNet800 | Deformable ResNet-101 | 40.8 | 59.6 | 42.7 | 22.7 | 43.3 | 51.0
RetinaNet800 | GRCNN-109 | 40.3 | 60.4 | 43.2 | 22.4 | 44.0 | 51.0
RetinaNet800 | Deformable GRCNN-109 | 41.2 | 60.8 | 44.3 | 23.3 | 44.3 | 51.8
RetinaNet800 | SKNet-100 | 41.6 | 61.5 | 44.7 | 23.5 | 45.1 | 52.3
RetinaNet800 | SK-GRCNN-109 | 42.3 | 61.8 | 45.4 | 23.9 | 46.0 | 52.9
FreeAnchor [99] | ResNet-101 | 43.1 | 62.2 | 46.4 | 24.5 | 46.1 | 54.8
FreeAnchor | Deformable ResNet-101 | 43.5 | 64.0 | 47.1 | 25.5 | 46.9 | 55.7
FreeAnchor | GRCNN-109 | 44.3 | 64.5 | 47.6 | 26.0 | 47.6 | 56.4
FreeAnchor | Deformable GRCNN-109 | 44.8 | 64.5 | 47.9 | 26.7 | 48.5 | 56.8
FreeAnchor | SKNet-100 | 45.2 | 64.8 | 48.4 | 27.5 | 48.3 | 56.6
FreeAnchor | SK-GRCNN-109 | 45.6 | 65.5 | 48.7 | 27.9 | 48.8 | 57.0

The AP, AP50 and AP75 indicate the average precision for IoU
$\in\\{0.50:0.05:0.95\\}$, at 0.5 and at 0.75, respectively. The APS, APM and
APL denote the average precision for small sized, medium sized and large sized
objects defined in [84] in terms of IoU $\in\\{0.50:0.05:0.95\\}$.

Figure 7: Sample images from MS COCO dataset. The objects are cluttered
together and differ significantly in scale. For example, in the first example,
the baseball bat is very small compared with the player.

###  6.2 Pipelines

To further evaluate GRCNN, we selected three popular frameworks, RefineDet
[90], RetinaNet [98], and FreeAnchor [99], which are categorized as “anchor-
based one-stage detectors”. The RefineDet consists of two inter-connected
modules named Anchor Refinement Module (ARM) and Object Detection Module
(ODM). The RetinaNet consists of a backbone network responsible for computing
feature maps over an entire input image and two task-specific networks. The
FreeAnchor introduces a detection customized likelihood function into the loss
function to supervise the training of RetinaNet. More details about these
frameworks can be found in their original papers. One common property of these
frameworks is that all of them have a backbone network (usually a pretrained
ResNet-101) for feature extraction. We were interested in whether the
performance would be boosted by substituting GRCNN based models with their
original backbone networks.

###  6.3 Implementation Details

In the publicly released codes of the three frameworks, we replaced the
backbone network ResNet-101 with the GRCNN-109 and the SK-GRCNN-109
respectively, whose network configurations are both described in Table IV. In
addition, we also evaluated Deformable GRCNN-109 on the three frameworks. All
GRCNN based models were pre-trained on the ImageNet classification task as
described in Section 4.5. Then it was fine-tuned using the SGD algorithm with
weight decay 0.0005 and momentum 0.9. The multi-scale testing strategy was not
used. The training scale range of the FreeAnchor [99] framework was set to
{640, 672, 704, 736, 768, 800}. The other training hyper-parameters were set
according to the original papers of the three frameworks [90, 99, 98].

###  6.4 Comparison with Existing Methods

Table IX presents the results of the three frameworks, as well as some other
anchor-based one-stage models which do not perform multi-scale testing. First,
we compared Deformable ResNet-101 and GRCNN-109 with ResNet-101, which were
trained as described in Section 4.6. The GRCNN and Deformable ConvNet obtained
similar performance gains on the detection task. Second, the three Deformable
GRCNN-109-based object detection frameworks consistently performed better than
the GRCNN-109-based and Deformable ResNet-101-based counterparts, indicating
that the proposed gating mechanism and Deformable convolutions are
complementary with respect to improving the detection performance. Third, by
using the SKConv, the performance of the GRCNN-109 was further improved.
Finally, the SK-GRCNN based FreeAnchor achieved the best results among the
anchor-based one-stage detectors compared in this study.

##  7 Concluding Remarks

The proposed model in this work is built on a previous model called recurrent
convolutional neural network (RCNN) [12, 13], which incorporates recurrent
connections between neurons in the same layer. Our main idea is to introduce
gates on the recurrent connections in RCNN, and the resulting model is called
gated RCNN (GRCNN). The gates control the amount of information flow from
neighboring neurons into each neuron according to the context. The unrelated
context information coming from the recurrent connections is inhibited by the
gates. Extensive experiments were carried out on some benchmark datasets for
object recognition, scene text recognition and object detection. The results
showed that GRCNN outperformed RCNN on all these datasets. The results
verified the effectiveness of the proposed gating function. In addition, on
these tasks, the proposed GRCNN often attained accuracy on par with (and
sometimes, e.g., on scene text recognition, even slightly better than) the
state-of-the-art models.

It is well-known that a deep neural network with more parameters has better
capability to model the input-output mapping. To compete with existing models,
we had to use more parameters. This was done by removing the weight sharing
constraint of recurrent networks. Then the recurrent connections became
conventional feedforward connections. Note that these connections determine
the RFs of neurons in the resulting feedforward model, and the gates operating
on these connections modulate the RFs of neurons. The effect turned out to be
very good in terms of accuracy.

If accuracy is not the main concern, one does not need to break the tied
weights, thus significantly reduces the number of parameters, which is favored
in applications where the memory of the equipment is limited. More
interestingly, one can explore to what extent the neurons in this recurrent
model resemble biological neurons by measuring their non-classical RFs (e.g.,
[14, 15, 16]). If the properties of artificial neurons and biological neurons
do not match well, one can develop new ideas based on the proposed model to
make them match better. This may lead to more brain-like and powerful computer
vision models. It may also shed light on the computational mechanism of visual
neurons in the brain.

## Acknowledgments

This work was supported in part by the National Natural Science Foundation of
China under Grant Nos. 61621136008 and 61836014.

## References

  * [1] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” _Neural Comput._ , vol. 1, no. 4, pp. 541–551, 1989.
  * [2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Proc. Neural Inf. Process. Syst._ , 2012, pp. 1097–1105.
  * [3] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _Proc. Int. Conf. Learn. Representations._ , 2015.
  * [4] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2016, pp. 770–778.
  * [5] G. Huang, Z. Liu, K. Q. Weinberger, and V. D. M. Laurens, “Densely connected convolutional networks,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2017, pp. 4700–4708.
  * [6] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2014, pp. 1–9.
  * [7] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2017, pp. 1492–1500.
  * [8] D. H. Hubel and T. N. Wiesel, “Receptive fields of single neurones in the cat’s striate cortex,” _J. of Physiol._ , vol. 148, no. 3, pp. 574–591, 1959\.
  * [9] ——, “Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex,” _J. of Physiol._ , vol. 160, no. 1, pp. 106–154, 1962.
  * [10] K. Fukushima and S. Miyake, “Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition,” in _Compet. and Coop. in Neural Nets_ , 1982, pp. 267–285.
  * [11] P. Dayan and L. F. Abbott, _Theoretical Neuroscience_. Cambridge: MIT Press, 2001.
  * [12] M. Liang and X. Hu, “Recurrent convolutional neural network for object recognition,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2015, pp. 3367–3375.
  * [13] M. Liang, X. Hu, and Z. B., “Convolutional neural networks with intra-layer recurrent connections for scene labeling,” in _Proc. Neural Inf. Process. Syst._ , 2015, pp. 937–945.
  * [14] J. Nelson and B. Frost, “Orientation-selective inhibition from beyond the classic visual receptive field,” _Brain Res._ , vol. 139, no. 2, pp. 359–365, 1978.
  * [15] H. Jones, W. Wang, and A. Sillito, “Spatial organization and magnitude of orientation contrast interactions in primate v1,” _J. Neurophysiol._ , vol. 88, no. 5, pp. 2796–2808, 2002.
  * [16] J. R. Cavanaugh, W. Bair, and J. A. Movshon, “Nature and interaction of signals from the receptive field center and surround in macaque v1 neurons,” _J. Neurophysiol._ , vol. 88, no. 5, pp. 2530–2546, 2002.
  * [17] J. Wang and X. Hu, “Gated recurrent convolution neural network for ocr,” in _Proc. Neural Inf. Process. Syst._ , vol. 30, 2017, pp. 335–344.
  * [18] R. K. Srivastava, K. Greff, and J. Schmidhuber, “Training very deep networks,” _Proc. Neural Inf. Process. Syst._ , vol. 28, pp. 2377–2385, 2015\.
  * [19] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2018, pp. 7132–7141.
  * [20] J. Hu, L. Shen, S. Albanie, G. Sun, and A. Vedaldi, “Gather-excite: Exploiting feature context in convolutional neural networks,” in _Proc. Neural Inf. Process. Syst._ , vol. 31, 2018, pp. 9401–9411.
  * [21] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei, “Deformable convolutional networks,” in _Proc. Int. Conf. Comput. Vis._ , 2017, pp. 764–773.
  * [22] A. Recasens, P. Kellnhofer, S. Stent, W. Matusik, and A. Torralba, “Learning to zoom: A saliency-based sampling layer for neural networks,” in _Proc. Eur. Conf. Comput. Vis._ , 2018, pp. 51–66.
  * [23] Y. Jeon and J. Kim, “Active convolution: Learning the shape of convolution for image classification,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2017, pp. 4201–4209.
  * [24] S. Liu, D. Huang, and W. Yunhong, “Receptive field block net for accurate and fast object detection,” in _Proc. Eur. Conf. Comput. Vis._ , 2018, pp. 385–400.
  * [25] J.-B. Cordonnier, A. Loukas, and M. Jaggi, “On the relationship between self-attention and convolutional layers,” in _Proc. Int. Conf. Learn. Representations._ , 2020.
  * [26] X. Li, W. Wang, X. Hu, and J. Yang, “Selective kernel networks,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2019, pp. 510–519.
  * [27] A. Graves, M. Liwicki, H. Bunke, and H. Bunke, “A novel connectionist system for unconstrained handwriting recognition,” _IEEE Trans. Pattern Analy. Mach. Intell._ , vol. 31, no. 5, pp. 855–868, 2009.
  * [28] A. Graves, A. R. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in _IEEE Int. Conf. on Acoust., Speech, Signal Process._ , 2013, pp. 6645–6649.
  * [29] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural Comput._ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [30] K. Cho, B. Van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” _arXiv:1406.1078_ , 2014.
  * [31] E. R. Kandel, J. H. Schwartz, and T. M. Jessell, _Principles of neural science_. McGraw-hill New York, 2000, vol. 4.
  * [32] C. J. Spoerer, P. McClure, and N. Kriegeskorte, “Recurrent convolutional neural networks: A better model of biological object recognition,” _Front. Psychol._ , vol. 8, p. 1551, 2017.
  * [33] Y. Zhao, X. Jin, and X. Hu, “Recurrent convolutional neural network for speech processing,” in _IEEE Int. Conf. on Acoust., Speech, Signal Process._ , 2017, pp. 5300–5304.
  * [34] J. Wang, W. Wang, X. Chen, R. Wang, and W. Gao, “Deep alternative neural network: Exploring contexts as early as possible for action recognition,” in _Proc. Neural Inf. Process. Syst._ , vol. 29, 2016, pp. 811–819.
  * [35] C. Y. Lee and S. Osindero, “Recursive recurrent nets with attention modeling for ocr in the wild,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2016, pp. 2231–2239.
  * [36] H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng, “Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations,” in _Proc. Int. Conf. Mach. Learn._ , 2009, pp. 609–616.
  * [37] P. H. O. Pinheiro and R. Collobert, “Recurrent convolutional neural networks for scene labeling,” in _Proc. Int. Conf. Mach. Learn._ , 2014, pp. 82–90.
  * [38] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _Proc. Int. Conf. Mach. Learn._ , 2015, pp. 448–456.
  * [39] K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” in _Proc. Eur. Conf. Comput. Vis._ , 2016, pp. 630–645.
  * [40] S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry, “How does batch normalization help optimization?” in _Proc. Neural Inf. Process. Syst._ , vol. 31, 2018, pp. 2483–2493.
  * [41] K. Alex, “Learning multiple layers of features from tiny images,” _[Online]. Available:https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf_, 2009.
  * [42] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” in _Proc. Neural Inf. Process. Syst., Workshop on Deep Learning and Unsupervised Feature Learning,_ , 2011.
  * [43] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger, “Deep networks with stochastic depth,” in _Proc. Eur. Conf. Comput. Vis._ , 2016, pp. 646–661.
  * [44] C. Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu, “Deeply-supervised nets,” in _Int. Conf. on Artif. Intell. and Stat._ , 2014, pp. 562–570.
  * [45] J. Deng, W. Dong, R. Socher, L. J. Li, K. Li, and F. F. Li, “Imagenet: A large-scale hierarchical image database,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2009, pp. 248–255.
  * [46] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein _et al._ , “Imagenet large scale visual recognition challenge,” _Int. J. Comput. Vis._ , vol. 115, no. 3, pp. 211–252, 2015.
  * [47] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, “Striving for simplicity: The all convolutional net,” _arXiv:1412.6806_ , 2014.
  * [48] G. Larsson, M. Maire, and G. Shakhnarovich, “Fractalnet: Ultra-deep neural networks without residuals,” in _Proc. Int. Conf. Learn. Representations._ , 2017.
  * [49] S. Zagoruyko and N. Komodakis, “Wide residual networks,” in _Proc. British Mach. Vis. Conf._ , 2016.
  * [50] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _Proc. Neural Inf. Process. Syst._ , 2019, pp. 8024–8035.
  * [51] H. Zhang, M. Cisse, D. N. Yann, and L.-P. David, “Mixup: Beyond empirical risk minimization,” in _Proc. Int. Conf. Learn. Representations._ , 2018.
  * [52] I. Sutskever, J. Martens, G. Dahl, and G. Hinton, “On the importance of initialization and momentum in deep learning,” in _Proc. Int. Conf. Mach. Learn._ , 2013, pp. 1139–1147.
  * [53] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2016, pp. 2818–2826.
  * [54] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2015, pp. 1026–1034.
  * [55] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in _Proc. AAAI Conf. Artif. Intell._ , 2017.
  * [56] X. Zhu, H. Hu, S. Lin, and J. Dai, “Deformable convnets v2: More deformable, better results,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2019, pp. 9308–9316.
  * [57] B. Shi, X. Bai, and C. Yao, “An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition,” _IEEE Trans. Pattern Analy. Mach. Intell._ , vol. 39, no. 11, pp. 2298–2304, 2017.
  * [58] B. Shi, M. Yang, X. Wang, P. Lyu, C. Yao, and X. Bai, “Aster: An attentional scene text recognizer with flexible rectification,” _IEEE Trans. Pattern Analy. Mach. Intell._ , vol. 41, no. 9, pp. 2035–2048, 2018.
  * [59] K. Wang, B. Babenko, and S. Belongie, “End-to-end scene text recognition,” in _Proc. Int. Conf. Comput. Vis._ , 2012, pp. 1457–1464.
  * [60] A. Mishra, K. Alahari, and C. V. Jawahar, “Scene text recognition using higher order language priors,” in _Proc. British Mach. Vis. Conf._ , 2013.
  * [61] T. Novikova, O. Barinova, V. Lempitsky, and V. Lempitsky, “Large-lexicon attribute-consistent text recognition in natural images,” in _Proc. Eur. Conf. Comput. Vis._ , 2012, pp. 752–765.
  * [62] T. Wang, D. J. Wu, A. Coates, and A. Y. Ng, “End-to-end text recognition with convolutional neural networks,” in _Int. Conf. Imag. Process._ , 2013, pp. 3304–3308.
  * [63] A. Bissacco, M. Cummins, Y. Netzer, and H. Neven, “Photoocr: Reading text in uncontrolled conditions,” in _Proc. Int. Conf. Comput. Vis._ , 2014, pp. 785–792.
  * [64] V. Goel, A. Mishra, K. Alahari, and C. V. Jawahar, “Whole is greater than sum of parts: Recognizing scene text words,” in _Int. Conf. Document Anal. Recognit._ , 2013, pp. 398–402.
  * [65] O. Alsharif and J. Pineau, “End-to-end text recognition with hybrid hmm maxout models,” _arXiv:1310.1811_ , 2013.
  * [66] J. Almazan, A. Gordo, A. Fornes, and E. Valveny, “Word spotting and recognition with embedded attributes,” _IEEE Trans. Pattern Analy. Mach. Intell._ , vol. 36, no. 12, pp. 2552–2566, 2014.
  * [67] C. Y. Lee, A. Bhardwaj, W. Di, V. Jagadeesh, and R. Piramuthu, “Region-based discriminative feature pooling for scene text recognition,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2014, pp. 4050–4057.
  * [68] C. Yao, X. Bai, B. Shi, and W. Liu, “Strokelets: A learned multi-scale representation for scene text recognition,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2014, pp. 4042–4049.
  * [69] J. A. Rodriguez-Serrano, A. Gordo, and F. Perronnin, “Label embedding: A frugal baseline for text recognition,” _Int. J. Comput. Vis._ , vol. 113, no. 3, pp. 193–207, 2015.
  * [70] M. Jaderberg, A. Vedaldi, and A. Zisserman, “Deep features for text spotting,” in _Proc. Eur. Conf. Comput. Vis._ , 2014, pp. 512–528.
  * [71] B. Su and S. Lu, “Accurate scene text recognition based on recurrent neural network,” in _Asian. Conf. on Comput. Vis._ , 2014, pp. 35–48.
  * [72] A. Gordo, “Supervised mid-level features for word image representation,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 1998, pp. 2956–2964.
  * [73] M. Jaderberg, K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep structured output learning for unconstrained text recognition,” in _Proc. Int. Conf. Learn. Representations._ , 2014.
  * [74] Z. Cheng, F. Bai, Y. Xu, G. Zheng, S. Pu, and S. Zhou, “Focusing attention: Towards accurate text recognition in natural images,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2017, pp. 5076–5084.
  * [75] Z. Cheng, Y. Xu, F. Bai, Y. Niu, S. Pu, and S. Zhou, “Aon: Towards arbitrarily-oriented text recognition,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2018, pp. 5571–5579.
  * [76] M. Liao, P. Lyu, M. He, C. Yao, W. Wu, and X. Bai, “Mask textspotter: An end-to-end trainable neural network for spotting text with arbitrary shapes,” _IEEE Trans. Pattern Analy. Mach. Intell._ , 2019.
  * [77] F. Zhan and S. Lu, “ESIR: End-to-end scene text recognition via iterative image rectification,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2019, pp. 2059–2068.
  * [78] Z. Qiao, Y. Zhou, D. Yang, Y. Zhou, and W. Wang, “SEED: Semantics enhanced encoder-decoder framework for scene text recognition,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2020, pp. 13 528–13 537.
  * [79] S. M. Lucas, A. Panaretos, L. Sosa, A. Tang, S. Wong, and R. Young, “Icdar 2003 robust reading competitions,” in _Int. Conf. Document Anal. Recognit._ , 2003, p. 682.
  * [80] C. Shi, C. Wang, B. Xiao, Y. Zhang, S. Gao, and Z. Zhang, “Scene text recognition using part-based tree-structured character detection,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2013, pp. 2961–2968.
  * [81] M. Jaderberg, K. Simonyan, A. Vedaldi, and A. Zisserman, “Synthetic data and artificial neural networks for natural scene text recognition,” _Proc. Neural Inf. Process. Syst., Workshop on Deep Learning_ , 2014.
  * [82] A. Graves and F. Gomez, “Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,” in _Proc. Int. Conf. Mach. Learn._ , 2006, pp. 369–376.
  * [83] M. D. Zeiler, “Adadelta: An adaptive learning rate method,” _arXiv:1212.5701_ , 2012.
  * [84] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in _Proc. Eur. Conf. Comput. Vis._ , 2014, pp. 740–755.
  * [85] J. Redmon and A. Farhadi, “YOLO9000: Better, faster, stronger,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2017, pp. 7263–7271.
  * [86] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “SSD: Single shot multibox detector,” in _Proc. Eur. Conf. Comput. Vis._ , 2016, pp. 21–37.
  * [87] C.-Y. Fu, W. Liu, A. Ranga, A. Tyagi, and A. C. Berg, “DSSD: Deconvolutional single shot detector,” _arXiv:1701.06659_ , 2017.
  * [88] P. Zhou, B. Ni, C. Geng, J. Hu, and Y. Xu, “Scale-transferrable object detection,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2018, pp. 528–537.
  * [89] T. Kong, F. Sun, C. Tan, H. Liu, and W. Huang, “Deep feature pyramid reconfiguration for object detection,” in _Proc. Eur. Conf. Comput. Vis._ , 2018, pp. 169–185.
  * [90] S. Zhang, L. Wen, X. Bian, Z. Lei, and S. Z. Li, “Single-shot refinement neural network for object detection,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2018, pp. 4203–4212.
  * [91] B. Li, Y. Liu, and X. Wang, “Gradient harmonized single-stage detector,” in _Proc. AAAI Conf. Artif. Intell._ , vol. 33, 2019, pp. 8577–8584.
  * [92] H. Law and J. Deng, “Cornernet: Detecting objects as paired keypoints,” in _Proc. Eur. Conf. Comput. Vis._ , 2018, pp. 734–750.
  * [93] Q. Zhao, T. Sheng, Y. Wang, Z. Tang, Y. Chen, L. Cai, and H. Ling, “M2DET: A single-shot object detector based on multi-level feature pyramid network,” in _Proc. AAAI Conf. Artif. Intell._ , vol. 33, 2019, pp. 9259–9266.
  * [94] X. Zhou, J. Zhuo, and P. Krahenbuhl, “Bottom-up object detection by grouping extreme and center points,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2019, pp. 850–859.
  * [95] X. Zhou, D. Wang, and P. Krähenbühl, “Objects as points,” _arXiv:1904.07850_ , 2019.
  * [96] C. Zhu, Y. He, and M. Savvides, “Feature selective anchor-free module for single-shot object detection,” _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , pp. 840–849, 2019.
  * [97] K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q. Tian, “Centernet: Keypoint triplets for object detection,” _Proc. Int. Conf. Comput. Vis._ , pp. 6569–6578, 2019.
  * [98] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” _IEEE Trans. Pattern Analy. Mach. Intell._ , pp. 2980–2988, 2018.
  * [99] X. Zhang, F. Wan, C. Liu, R. Ji, and Q. Ye, “FreeAnchor: Learning to match anchors for visual object detection,” in _Proc. Neural Inf. Process. Syst._ , 2019, pp. 147–155.

|  Jianfeng Wang received the B.E. and M.E. degrees from Beijing University of
Posts and Telecommunications, Beijing, China, in 2013 and 2016 respectively.
Now, he is working toward the PhD degree in the Computer Science Department
from University of Oxford. His current research interests include computer
vision, machine learning and deep learning.
---|---
|  Xiaolin Hu (S’01, M’08, SM’13) received the B.E. and M.E. degrees in
Automotive Engineering from Wuhan University of Technology, Wuhan, China, and
the Ph.D. degree in Automation and Computer-Aided Engineering from The Chinese
University of Hong Kong, Hong Kong, China, in 2001, 2004, 2007, respectively.
He is now an Associate Professor at the Department of Computer Science and
Technology, Tsinghua University, Beijing, China. His current research
interests include deep learning and computational neuroscience. He was an
Associate Editor of the IEEE Transactions on Neural Networks and Learning
Systems. Now he is an Associate Editor of IEEE Transactions on Image
Processing and an Associate Editor of Cognitive Neurodynamics.
---|---