# Conditional DETR for Fast Training Convergence

Depu Meng1 Xiaokang Chen2††footnotemark:  Zejia Fan2 Gang Zeng2
Houqiang Li1 Yuhui Yuan3 Lei Sun3 Jingdong Wang3
1University of Science and Technology of China 2Peking University 3Microsoft
Research Asia Equal Contribution. This work was done when D. Meng, X. Chen,
and Z. Fan were interns at Microsoft Research, Beijing, P.R.
ChinaCorresponding author.

###### Abstract

The recently-developed DETR approach applies the transformer encoder and
decoder architecture to object detection and achieves promising performance.
In this paper, we handle the critical issue, slow training convergence, and
present a conditional cross-attention mechanism for fast DETR training. Our
approach is motivated by that the cross-attention in DETR relies highly on the
content embeddings for localizing the four extremities and predicting the box,
which increases the need for high-quality content embeddings and thus the
training difficulty.

Our approach, named conditional DETR, learns a conditional spatial query from
the decoder embedding for decoder multi-head cross-attention. The benefit is
that through the conditional spatial query, each cross-attention head is able
to attend to a band containing a distinct region, e.g., one object extremity
or a region inside the object box. This narrows down the spatial range for
localizing the distinct regions for object classification and box regression,
thus relaxing the dependence on the content embeddings and easing the
training. Empirical results show that conditional DETR converges $6.7\times$
faster for the backbones R$50$ and R$101$ and $10\times$ faster for stronger
backbones DC$5$-R$50$ and DC$5$-R$101$. Code is available at
https://github.com/Atten4Vis/ConditionalDETR.

##  1 Introduction

The DEtection TRansformer (DETR) method [3] applies the transformer encoder
and decoder architecture to object detection and achieves good performance. It
effectively eliminates the need for many hand-crafted components, including
non-maximum suppression and anchor generation.

Figure 1: Comparison of spatial attention weight maps for our conditional
DETR-R$50$ with $50$ training epochs (the first row), the original DETR-R$50$
with $50$ training epochs (the second row), and the original DETR-R$50$ with
$500$ training epochs (the third row). The maps for our conditional DETR and
DETR trained with $500$ epochs are able to highlight the four extremity
regions satisfactorily. In contrast, the spatial attention weight maps
responsible for the left and right edges (the third and fourth images in the
second row) from DETR trained with $50$ epochs cannot highlight the
extremities satisfactorily. The green box is the ground-truth box.

The DETR approach suffers from slow convergence on training, and needs $500$
training epochs to get good performance. The very recent work, deformable DETR
[53], handles this issue by replacing the global dense attention (self-
attention and cross-attention) with deformable attention that attends to a
small set of key sampling points and using the high-resolution and multi-scale
encoder. Instead, we still use the global dense attention and propose an
improved decoder cross-attention mechanism for accelerating the training
process.

Our approach is motivated by high dependence on content embeddings and minor
contributions made by the spatial embeddings in cross-attention. The empirical
results in DETR [3] show that if removing the positional embeddings in keys
and the object queries from the second decoder layer and only using the
content embeddings in keys and queries, the detection AP drops slightly111 The
minor AP drop $1.4$ is reported on R$50$ with $300$ epochs in Table 3 from
[3]. We empirically got the consistent observation: the AP drops to $34.0$
from $34.9$ for $50$ training epochs. .

Figure 1 (the second row) shows that the spatial attention weight maps from
the cross-attention in DETR trained with $50$ epochs. One can see that two
among the four maps do not correctly highlight the bands for the corresponding
extremities, thus weak at shrinking the spatial range for the content queries
to precisely localize the extremities. The reasons are that (i) the spatial
queries, i.e., object queries, only give the general attention weight map
without exploiting the specific image information; and that (ii) due to short
training the content queries are not strong enough to match the spatial keys
well as they are also used to match the content keys. This increases the
dependence on high-quality content embeddings, thus increasing the training
difficulty.

Figure 2: Convergence curves for conditional DETR-DC5-R50 and DETR-DC5-R50 on
COCO 2017 val. The conditional DETR is trained for $50$, $75$, $108$ epochs.
Conditional DETR training is converged much faster than DETR.

We present a conditional DETR approach, which learns a conditional spatial
embedding for each query from the corresponding previous decoder output
embedding, to form a so-called conditional spatial query for decoder multi-
head cross-attention. The conditional spatial query is predicted by mapping
the information for regressing the object box to the embedding space, the same
to the space that the $2$D coordinates of the keys are also mapped to.

We empirically observe that using the spatial queries and keys, each cross-
attention head spatially attends to a band containing the object extremity or
a region inside the object box (Figure 1, the first row). This shrinks the
spatial range for the content queries to localize the effective regions for
class and box prediction. As a result, the dependence on the content
embeddings is relaxed and the training is easier. The experiments show that
conditional DETR converges $6.7\times$ faster for the backbones R$50$ and
R$101$ and $10\times$ faster for stronger backbones DC$5$-R$50$ and
DC$5$-R$101$. Figure 2 gives the convergence curves for conditional DETR and
the original DETR [3].

##  2 Related Work

Anchor-based and anchor-free detection. Most existing object detection
approaches make predictions from initial guesses that are carefully designed.
There are two main initial guesses: anchor boxes or object centers. The anchor
box-based methods inherit the ideas from the proposal-based method, Fast
R-CNN. Example methods include Faster R-CNN [9], SSD [26], YOLOv2 [31], YOLOv3
[32], YOLOv4 [1], RetinaNet [24], Cascade R-CNN [2], Libra R-CNN [29], TSD
[35] and so on.

The anchor-free detectors predict the boxes at points near the object centers.
Typical methods include YOLOv1 [30], CornerNet [21], ExtremeNet [50],
CenterNet [49, 6], FCOS [39] and others [23, 28, 52, 19, 51, 22, 15, 46, 47].

DETR and its variants. DETR successfully applies transformers to object
detection, effectively removing the need for many hand-designed components
like non-maximum suppression or initial guess generation. The high computation
complexity issue, caused by the global encoder self-attention, is handled in
adaptive clustering transformer [48] and by sparse attentions in deformable
DETR [53].

The other critical issue, slow training convergence, has been attracting a lot
of recent research attention. The TSP (transformer-based set prediction)
approach [37] eliminates the cross-attention modules and combines the FCOS and
R-CNN-like detection heads. Deformable DETR [53] adopts deformable attention,
which attends to sparse positions learned from the content embedding, to
replace decoder cross-attention.

The spatially modulated co-attention (SMCA) approach [7], which is concurrent
to our approach, is very close to our approach. It modulates the DETR multi-
head global cross-attentions with Gaussian maps around a few (shifted) centers
that are learned from the decoder embeddings, to focus more on a few regions
inside the estimated box. In contrast, the proposed conditional DETR approach
learns the conditional spatial queries from the decoder content embeddings,
and predicts the spatial attention weight maps without human-crafting the
attention attenuation, which highlight four extremities for box regression,
and distinct regions inside the object for classification.

Conditional and dynamic convolution. The proposed conditional spatial query
scheme is related to conditional convolutional kernel generation. Dynamic
filter network [16] learns the convolutional kernels from the input, which is
applied to instance segmentation in CondInst [38] and SOLOv2 [42] for learning
instance-dependent convolutional kernels. CondConv [44] and dynamic
convolution [4] mix convolutional kernels with the weights learned from the
input. SENet [14], GENet [13] abd Lite-HRNet [45] learn from the input the
channel-wise weights.

These methods learn from the input the convolutional kernel weights and then
apply the convolutions to the input. In contrast, the linear projection in our
approach is learned from the decoder embeddings for representing the
displacement and scaling information.

Transformers. The transformer [40] relies on the attention mechanism, self-
attention and cross-attention, to draw global dependencies between the input
and the output. There are several works closely related to our approach.
Gaussian transformer [11] and T-GSA (Transformer with Gaussian-weighted self-
attention) [18], followed by SMCA [7], attenuate the attention weights
according to the distance between target and context symbols with learned or
human-crafted Gaussian variance. Similar to ours, TUPE [17] computes the
attention weight also from the spatial attention weight and the content
attention weight. Instead, our approach mainly focuses on the attention
attenuation mechanism in a learnable form other than a Gaussian function, and
potentially benefits speech enhancement [18] and natural language inference
[11].

##  3 Conditional DETR

###  3.1 Overview

Pipeline. The proposed approach follows detection transformer (DETR), an end-
to-end object detector, and predicts all the objects at once without the need
for NMS or anchor generation. The architecture consists of a CNN backbone, a
transformer encoder, a transformer decoder, and object class and box position
predictors. The transformer encoder aims to improve the content embeddings
output from the CNN backbone. It is a stack of multiple encoder layers, where
each layer mainly consists of a self-attention layer and a feed-forward layer.

The transformer decoder is a stack of decoder layers. Each decoder layer,
illustrated in Figure 3, is composed of three main layers: (1) a self-
attention layer for removing duplication prediction, which performs
interactions between the embeddings, outputted from the previous decoder layer
and used for class and box prediction, (2) a cross-attention layer, which
aggregates the embeddings output from the encoder to refine the decoder
embeddings for improving class and box prediction, and (3) a feed-forward
layer.

Box regression. A candidate box is predicted from each decoder embedding as
follows,

$\displaystyle\mathbf{b}=\operatorname{sigmoid}(\operatorname{FFN}(\mathbf{f})+[{\mathbf{s}}^{\top}~{}0~{}0]^{\top}).$
(1)

Here, $\mathbf{f}$ is the decoder embedding. $\mathbf{b}$ is a four-
dimensional vector $[b_{cx}~{}b_{cy}~{}b_{w}~{}b_{h}]^{\top}$, consisting of
the box center, the box width and the box height. $\operatorname{sigmoid}()$
is used to normalize the prediction $\mathbf{b}$ to the range $[0,1]$.
$\operatorname{FFN}()$ aims to predictthe unnormalized box. ${\mathbf{s}}$ is
the unnormalized $2$D coordinate of the reference point, and is $(0,0)$ in the
original DETR. In our approach, we consider two choices: learn the reference
point $\mathbf{s}$ as a parameter for each candidate box prediction, or
generate it from the corresponding object query.

Category prediction. The classification score for each candidate box is also
predicted from the decoder embedding through an FNN,
$\mathbf{e}=\operatorname{FFN}(\mathbf{f})$.

Figure 3: Illustrating one decoder layer in conditional DETR. The main
difference from the original DETR [3] lies in the input queries and the input
keys for cross-attention. The conditional spatial query is predicted from
learnable $2$D coordinates $\mathbf{s}$ and the embeddings output from the
previous decoder layer, through the operations depicted in the gray-shaded
box. The $2$D coordinate $\mathbf{s}$ can be predicted from the object query
(the dashed box), or simply learned as model parameters The spatial query
(key) and the content query (key) are concatenated as the query (key). The
resulting cross-attention is called conditional cross-attention. Same as DETR
[3], the decoder layer is repeated $6$ times.

Main work. The cross-attention mechanism aims to _localize the distinct
regions, four extremities for box detection and regions inside the box for
object classification, and aggregates the corresponding embeddings_. We
propose a conditional cross-attention mechanism with introducing conditional
spatial queries for improving the localization capability and accelerating the
training process.

###  3.2 DETR Decoder Cross-Attention

The DETR decoder cross-attention mechanism takes three inputs: queries, keys
and values. Each key is formed by adding a content key $\mathbf{c}_{k}$ (the
content embedding output from the encoder) and a spatial key $\mathbf{p}_{k}$
(the positional embedding of the corresponding normalized $2$D coordinate).
The value is formed from the content embedding, same with the content key,
output from the encoder.

In the original DETR approach, each query is formed by adding a content query
$\mathbf{c}_{q}$ (the embedding output from the decoder self-attention), and a
spatial query $\mathbf{p}_{q}$ (i.e., the object query $\mathbf{o}_{q}$). In
our implementation, there are $N=300$ object queries, and accordingly there
are $N$ queries222For description simplicity and clearness, we drop the query,
key, and value indices., each query outputting a candidate detect result in
one decoder layer.

The attention weight is based on the dot-product between the query and the
key, used for attention weight computation,

$\displaystyle(\mathbf{c}_{q}+\mathbf{p}_{q})^{\top}(\mathbf{c}_{k}+\mathbf{p}_{k})$
$\displaystyle=~{}$
$\displaystyle\mathbf{c}_{q}^{\top}\mathbf{c}_{k}+\mathbf{c}_{q}^{\top}\mathbf{p}_{k}+\mathbf{p}_{q}^{\top}\mathbf{c}_{k}+\mathbf{p}_{q}^{\top}\mathbf{p}_{k}$
$\displaystyle=~{}$
$\displaystyle\mathbf{c}_{q}^{\top}\mathbf{c}_{k}+\mathbf{c}_{q}^{\top}\mathbf{p}_{k}+\mathbf{o}_{q}^{\top}\mathbf{c}_{k}+\mathbf{o}_{q}^{\top}\mathbf{p}_{k}.$
(2)

###  3.3 Conditional Cross-Attention

The proposed conditional cross-attention mechanism forms the query by
concatenating the content query $\mathbf{c}_{q}$, outputting from decoder
self-attention, and the spatial query $\mathbf{p}_{q}$. Accordingly, the key
is formed as the concatenation of the content key $\mathbf{c}_{k}$ and the
spatial key $\mathbf{p}_{k}$.

The cross-attention weights consist of two components, content attention
weight and spatial attention weight. The two weights are from two dot-
products, content and spatial dot-products,

$\displaystyle\mathbf{c}_{q}^{\top}\mathbf{c}_{k}+\mathbf{p}_{q}^{\top}\mathbf{p}_{k}.$
(3)

Different from the original DETR cross-attention, our mechanism separates the
roles of content and spatial queries so that spatial and content queries focus
on the spatial and content attention weights, respectively.

An additional important task is to compute the spatial query $\mathbf{p}_{q}$
from the embedding $\mathbf{f}$ of the previous decoder layer. We first
identify that the spatial information of the distinct regions are determined
by the two factors together, decoder embedding and reference point. We then
show how to map them to the embedding space, forming the query
$\mathbf{p}_{q}$, so that the spatial query lies in the same space the $2$D
coordinates of the keys are mapped to.

The decoder embedding contains the displacements of the distinct regions with
respect to the reference point. The box prediction process in Equation 1
consists of two steps: (1) predicting the box with respect to the reference
point in the unnormalized space, and (2) normalizing the predicted box to the
range $[0,1]$333The origin $(0,0)$ in the unnormalized space for the original
DETR method is mapped to $(0.5,0.5)$ (the center in the image space) in the
normalized space through the $\operatorname{sigmoid}$ function..

Step (1) means that the decoder embedding $\mathbf{f}$ contains the
displacements of the four extremities (forming the box) with respect to the
reference point $\mathbf{s}$ in the unnormalized space. This implies that both
the embedding $\mathbf{f}$ and the reference point $\mathbf{s}$ are necessary
to determine the spatial information of the distinct regions, the four
extremities as well as the region for predicting the classification score.

Conditional spatial query prediction. We predict the conditional spatial query
from the embedding $\mathbf{f}$ and the reference point $\mathbf{s}$,

$\displaystyle(\mathbf{s},\mathbf{f})\rightarrow\mathbf{p}_{q},$ (4)

so that it is aligned with the positional space which the normalized $2$D
coordinates of the keys are mapped to. The process is illustrated in the gray-
shaded box area of Figure 3.

We normalize the reference point $\mathbf{s}$ and then map it to a
$256$-dimensional sinusoidal positional embedding in the same way as the
positional embedding for keys:

$\displaystyle\mathbf{p}_{s}=\operatorname{sinusoidal}(\operatorname{sigmoid}(\mathbf{s})).$
(5)

We then map the displacement information contained in the decoder embedding
$\mathbf{f}$ to a linear projection in the same space through an FFN
consisting of learnable linear projection + ReLU + learnable linear
projection: $\mathbf{T}=\operatorname{FFN}(\mathbf{f})$.

The conditional spatial query is computed by transforming the reference point
in the embedding space: $\mathbf{p}_{q}=\mathbf{T}\mathbf{p}_{s}$. We choose
the simple and computationally-efficient projection matrix, a diagonal matrix.
The $256$ diagonal elements are denoted as a vector
$\boldsymbol{\uplambda}_{q}$. The conditional spatial query is computed by the
element-wise multiplication:

$\displaystyle\mathbf{p}_{q}=\mathbf{T}{\mathbf{p}}_{s}=\boldsymbol{\uplambda}_{q}\odot{\mathbf{p}}_{s}.$
(6)

Multi-head cross-attention. Following DETR [3], we adopt the standard multi-
head cross-attention mechanism. Object detection usually needs to implicitly
or explicitly localize the four object extremities for accurate box regression
and localize the object region for accurate object classification. The multi-
head mechanism is beneficial to disentangle the localization tasks.

We perform multi-head parallel attentions by projecting the queries, the keys,
and the values $M=8$ times with learned linear projections to low dimensions.
The spatial and content queries (keys) are separately projected to each head
with different linear projections. The projections for values are the same as
the original DETR and are only for the contents.

Figure 4:  Illustrating the spatial attention weight maps (the first row), the
content attention weight maps (the second row), and the combined attention
weight maps (the third row) computed from our conditional DETR. The attention
weight maps are from $5$ heads out of the $8$ heads and are responsible for
the four extremities and a region inside the object box. The content attention
weight maps for the four extremities highlight scattered regions inside the
box (bicycle) or similar regions in two object instances (cow), and the
corresponding combined attention weight maps highlight the extremity regions
with the help of the spatial attention weight maps. The combined attention
weight map for the region inside the object box mainly depends on the spatial
attention weight map, which implies that the representation of a region inside
the object might encode enough class information. The maps are from
conditional DETR-R$50$ trained with $50$ epochs.

###  3.4 Visualization and Analysis

Visualization. Figure 4 visualizes the attention weight maps for each head:
the spatial attention weight maps, the content attention weight maps, and the
combined attention weight maps. The maps are soft-max normalized over the
spatial dot-products $\mathbf{p}_{q}^{\top}\mathbf{p}_{k}$, the content dot-
products $\mathbf{c}_{q}^{\top}\mathbf{c}_{k}$, and the combined dot-products
$\mathbf{c}_{q}^{\top}\mathbf{c}_{k}+\mathbf{p}_{q}^{\top}\mathbf{p}_{k}$. We
show $5$ out of the $8$ maps, and other three are the duplicates,
corresponding to bottom and top extremities, and a small region inside the
object box444The duplicates might be different for models trained several
times, but the detection performance is almost the same..

We can see that the spatial attention weight map at each head is able to
localize a distinct region, a region containing one extremity or a region
inside the object box. It is interesting that each spatial attention weight
map corresponding to an extremity highlights a spatial band that overlaps with
the corresponding edge of the object box. The other spatial attention map for
the region inside the object box merely highlights a small region whose
representations might already encode enough information for object
classification.

The content attention weight maps of the four heads corresponding to the four
extremities highlight scattered regions in addition to the extremities. The
combination of the spatial and content maps filters out other highlights and
keeps extremity highlights for accurate box regression.

Comparison to DETR. Figure 1 shows the spatial attention weight maps of our
conditional DETR (the first row) and the original DETR trained with $50$
epochs (the second row). The maps of our approach are computed by soft-max
normalizing the dot-products between spatial keys and queries,
$\mathbf{p}_{q}^{\top}\mathbf{p_{k}}$. The maps for DETR are computed by soft-
max normalizing the dot-products with the spatial keys,
$(\mathbf{o}_{q}+\mathbf{c}_{q})^{\top}\mathbf{p}_{k}$.

It can be seen that our spatial attention weight maps accurately localize the
distinct regions, four extremities. In contrast, the maps from the original
DETR with $50$ epochs can not accurately localize two extremities, and $500$
training epochs (the third row) make the content queries stronger, leading to
accurate localization. This implies that it is really hard to learn the
content query $\mathbf{c}_{q}$ to serve as two roles555Strictly speaking, the
embedding output from decoder self-attention for more training epochs contains
both spatial and content information. For discussion convenience, we still
call it content query.: match the content key and the spatial key
simultaneously, and thus more training epochs are needed.

Analysis. The spatial attention weight maps shown in Figure 4 imply that the
conditional spatial query, used to form the spatial query, have at least two
effects. (i) Translate the highlight positions to the four extremities and the
position inside the object box: interestingly the highlighted positions are
spatially similarly distributed in the object box. (ii) Scale the spatial
spread for the extremity highlights: large spread for large objects and small
spread for small objects.

The two effects are realized in the spatial embedding space through applying
the transformation $\mathbf{T}$ over $\mathbf{p}_{s}$ (further disentangled
through image-independent linear projections contained in cross-attention and
distributed to each head). This indicates that the transformation $\mathbf{T}$
not only contains the displacements as discussed before, but also the object
scale.

Table 1: Comparison of conditional DETR with DETR on COCO 2017 val. Our conditional DETR approach for high-resolution backbones DC$5$-R$50$ and DC$5$-R$101$ is $10\times$ faster than the original DETR, and for low-resolution backbones R$50$ and R$101$ $6.67\times$ faster. Conditional DETR is empirically superior to other two single-scale DETR variants. ∗The results of deformable DETR are from the GitHub repository provided by the authors of deformable DETR [53].  Model | #epochs | GFLOPs | #params (M) | AP |  AP50 |  AP75 |  APS |  APM |  APL
---|---|---|---|---|---|---|---|---|---
DETR-R$50$ | $500$ | $86$ | $41$ | $42.0$ | $62.4$ | $44.2$ | $20.5$ | $45.8$ | $61.1$
DETR-R$50$ | $50$ | $86$ | $41$ | $34.9$ | $55.5$ | $36.0$ | $14.4$ | $37.2$ | $54.5$
Conditional DETR-R50 | $50$ | $90$ | $44$ | $40.9$ | $61.8$ | $43.3$ | $20.8$ | $44.6$ | $59.2$
Conditional DETR-R50 | $75$ | $90$ | $44$ | $42.1$ | $62.9$ | $44.8$ | $21.6$ | $45.4$ | $60.2$
Conditional DETR-R50 | $108$ | $90$ | $44$ | $43.0$ | $64.0$ | $45.7$ | $22.7$ | $46.7$ | $61.5$
DETR-DC5-R$50$ | $500$ | $187$ | $41$ | $43.3$ | $63.1$ | $45.9$ | $22.5$ | $47.3$ | $61.1$
DETR-DC5-R$50$ | $50$ | $187$ | $41$ | $36.7$ | $57.6$ | $38.2$ | $15.4$ | $39.8$ | $56.3$
Conditional DETR-DC5-R50 | $50$ | $195$ | $44$ | $43.8$ | $64.4$ | $46.7$ | $24.0$ | $47.6$ | $60.7$
Conditional DETR-DC5-R50 | $75$ | $195$ | $44$ | $44.5$ | $65.2$ | $47.3$ | $24.4$ | $48.1$ | $62.1$
Conditional DETR-DC5-R50 | $108$ | $195$ | $44$ | $45.1$ | $65.4$ | $48.5$ | $25.3$ | $49.0$ | $62.2$
DETR-R$101$ | $500$ | $152$ | $60$ | $43.5$ | $63.8$ | $46.4$ | $21.9$ | $48.0$ | $61.8$
DETR-R$101$ | $50$ | $152$ | $60$ | $36.9$ | $57.8$ | $38.6$ | $15.5$ | $40.6$ | $55.6$
Conditional DETR-R101 | $50$ | $156$ | $63$ | $42.8$ | $63.7$ | $46.0$ | $21.7$ | $46.6$ | $60.9$
Conditional DETR-R101 | $75$ | $156$ | $63$ | $43.7$ | $64.9$ | $46.8$ | $23.3$ | $48.0$ | $61.7$
Conditional DETR-R101 | $108$ | $156$ | $63$ | $44.5$ | $65.6$ | $47.5$ | $23.6$ | $48.4$ | $63.6$
DETR-DC5-R$101$ | $500$ | $253$ | $60$ | ${44.9}$ | ${64.7}$ | $47.7$ | $23.7$ | ${49.5}$ | ${62.3}$
DETR-DC5-R$101$ | 50 | $253$ | $60$ | $38.6$ | $59.7$ | $40.7$ | $17.2$ | $42.2$ | $57.4$
Conditional DETR-DC5-R101 | $50$ | $262$ | $63$ | $45.0$ | $65.5$ | $48.4$ | $26.1$ | $48.9$ | $62.8$
Conditional DETR-DC5-R101 | 75 | $262$ | $63$ | $45.6$ | $66.5$ | $48.8$ | $25.5$ | $49.7$ | $63.3$
Conditional DETR-DC5-R101 | 108 | $262$ | $63$ | $45.9$ | $66.8$ | $49.5$ | $27.2$ | $50.3$ | $63.3$
_Other single-scale DETR variants_ |  |  |  |  |  |  |  |  |
Deformable DETR-R50-SS∗ | $50$ | $78$ | $34$ | $39.4$ | $59.6$ | $42.3$ | $20.6$ | $43.0$ | $55.5$
UP-DETR-R$50$ [5] | $150$ | $86$ | $41$ | $40.5$ | $60.8$ | $42.6$ | $19.0$ | $44.4$ | $60.0$
UP-DETR-R$50$ [5] | $300$ | $86$ | $41$ | $42.8$ | $63.0$ | $45.3$ | $20.8$ | $47.1$ | $61.7$
Deformable DETR-DC5-R50-SS∗ | 50 | $128$ | $34$ | $41.5$ | $61.8$ | $44.9$ | $24.1$ | $45.3$ | $56.0$

###  3.5 Implementation Details

Architecture. Our architecture is almost the same with the DETR architecture
[3] and contains the CNN backbone, transformer encoder, transformer decoder,
prediction feed-forward networks (FFNs) following each decoder layer (the last
decoder layer and the $5$ internal decoder layers) with parameters shared
among the $6$ prediction FFNs. The hyper-parameters are the same as DETR.

The main architecture difference is that we introduce the conditional spatial
embeddings as the spatial queries for conditional multi-head cross-attention
and that the spatial query (key) and the content query (key) are combined
through concatenation other than addition. In the first cross-attention layer
there are no decoder content embeddings, we make simple changes based on the
DETR implementation [3]: concatenate the positional embedding predicted from
the object query (the positional embedding) into the original query (key).

Reference points. In the original DETR approach, ${\mathbf{s}}=[0~{}0]^{\top}$
is the same for all the decoder embeddings. We study two ways forming the
reference points: regard the unnormalized $2$D coordinates as learnable
parameters, and the unnormalized $2$D coordinate predicted from the object
query $\mathbf{o}_{q}$. In the latter way that is similar to deformable DETR
[53], the prediction unit is an FFN and consists of learnable linear
projection + ReLU + learnable linear projection:
$\mathbf{s}=\operatorname{FFN}(\mathbf{o}_{q})$. When used for forming the
conditional spatial query, the $2$D coordinates are normalized by the sigmoid
function.

Loss function. We follow DETR [3] to find an optimal bipartite matching [20]
between the predicted and ground-truth objects using the Hungarian algorithm,
and then form the loss function for computing and back-propagate the
gradients. We use the same way with deformable DETR [53] to formulate the
loss: the same matching cost function, the same loss function with $300$
object queries, and the same trade-off parameters; The classification loss
function is focal loss [24], and the box regression loss (including L1 and
GIoU [34] loss) is the same as DETR [3].

##  4 Experiments

###  4.1 Setting

Dataset. We perform the experiments on the COCO $2017$ [25] detection dataset.
The dataset contains about $118$K training images and $5$K validation (val)
images.

Table 2: Results for multi-scale and higher-resolution DETR variants. We do not expect that our approach performs on par as our approach (single-scale, $16\times$ resolution) does not use a strong multi-scale or $8\times$ resolution encoder. Surprisingly, the AP scores of our approach with DC$5$-R$50$ and DC$5$-R$101$ are close to the two multi-scale and higher-resolution DETR variants.  Model | #epochs | GFLOPs | #params (M) | AP |  AP50 |  AP75 |  APS |  APM |  APL
---|---|---|---|---|---|---|---|---|---
Faster RCNN-FPN-R$50$ [33] | $36$ | $180$ | $42$ | $40.2$ | $61.0$ | $43.8$ | $24.2$ | $43.5$ | $52.0$
Faster RCNN-FPN-R$50$ [33] | $108$ | $180$ | $42$ | $42.0$ | $62.1$ | $45.5$ | $26.6$ | $45.5$ | $53.4$
Deformable DETR-R$50$ [53] | $50$ | $173$ | $40$ | $43.8$ | $62.6$ | $47.7$ | $26.4$ | $47.1$ | $58.0$
TSP-FCOS-R$50$ [37] | $36$ | $189$ | $-$ | $43.1$ | $62.3$ | $47.0$ | $26.6$ | $46.8$ | $55.9$
TSP-RCNN-R$50$ [37] | $36$ | $188$ | $-$ | $43.8$ | $63.3$ | $48.3$ | $28.6$ | $46.9$ | $55.7$
TSP-RCNN-R$50$ [37] | $96$ | $188$ | $-$ | $45.0$ | $64.5$ | $49.6$ | $29.7$ | $47.7$ | $58.0$
Conditional DETR-DC5-R50 | $50$ | $195$ | $44$ | $43.8$ | $64.4$ | $46.7$ | $24.0$ | $47.6$ | $60.7$
Conditional DETR-DC5-R50 | $108$ | $195$ | $44$ | $45.1$ | $65.4$ | $48.5$ | $25.3$ | $49.0$ | $62.2$
Faster RCNN-FPN-R$101$ [33] | $36$ | $246$ | $60$ | $42.0$ | $62.5$ | $45.9$ | $25.2$ | $45.6$ | $54.6$
Faster RCNN-FPN-R$101$ [33] | $108$ | $246$ | $60$ | $44.0$ | $63.9$ | $47.8$ | $27.2$ | $48.1$ | $56.0$
TSP-FCOS-R$101$ [37] | $36$ | $255$ | $-$ | $44.4$ | $63.8$ | $48.2$ | $27.7$ | $48.6$ | $57.3$
TSP-RCNN-R$101$ [37] | $36$ | $254$ | $-$ | $44.8$ | $63.8$ | $49.2$ | $29.0$ | $47.9$ | $57.1$
TSP-RCNN-R$101$ [37] | $96$ | $254$ | $-$ | $46.5$ | $66.0$ | $51.2$ | $29.9$ | $49.7$ | $59.2$
Conditional DETR-DC5-R101 | $50$ | $262$ | $63$ | $45.0$ | $65.5$ | $48.4$ | $26.1$ | $48.9$ | $62.8$
Conditional DETR-DC5-R101 | $108$ | $262$ | $63$ | $45.9$ | $66.8$ | $49.5$ | $27.2$ | $50.3$ | $63.3$

Training. We follow the DETR training protocol [3]. The backbone is the
ImageNet-pretrained model from TORCHVISION with batchnorm layers fixed, and
the transformer parameters are initialized using the Xavier initialization
scheme [10]. The weight decay is set to be $10^{-4}$. The AdamW [27] optimizer
is used. The learning rates for the backbone and the transformer are initially
set to be $10^{-5}$ and $10^{-4}$, respectively. The dropout rate in
transformer is $0.1$. The learning rate is dropped by a factor of $10$ after
$40$ epochs for $50$ training epochs, after $60$ epochs for $75$ training
epochs, and after $80$ epochs for $108$ training epochs.

We use the augmentation scheme same as DETR [3]: resize the input image such
that the short side is at least $480$ and at most $800$ pixels and the long
size is at most $1333$ pixels; randomly crop the image such that a training
image is cropped with probability $0.5$ to a random rectangular patch.

Evaluation. We use the standard COCO evaluation. We report the average
precision (AP), and the AP scores at $0.50$, $0.75$ and for the small, medium,
and large objects.

###  4.2 Results

Comparison to DETR. We compare the proposed conditional DETR to the original
DETR [3]. We follow [3] and report the results over four backbones:
ResNet-$50$ [12], ResNet-$101$, and their $16\times$-resolution extensions
DC$5$-ResNet-$50$ and DC$5$-ResNet-$101$.

The corresponding DETR models are named as DETR-R$50$, DETR-R$101$, DETR-
DC5-R$50$, and DETR-DC5-R$101$, respectively. Our models are named as
conditional DETR-R$50$, conditional DETR-R$101$, conditional DETR-DC5-R$50$,
and conditional DETR-DC5-R$101$, respectively.

Table 1 presents the results from DETR and conditional DETR. DETR with $50$
training epochs performs much worse than $500$ training epochs. Conditional
DETR with $50$ training epochs for R$50$ and R$101$ as the backbones performs
slightly worse than DETR with $500$ training epochs. Conditional DETR with
$50$ training epochs for DC$5$-R$50$ and DC$5$-R$101$ performs similarly as
DETR with $500$ training epochs. Conditional DETR for the four backbones with
$75/108$ training epochs performs better than DETR with $500$ training epochs.
In summary, conditional DETR for high-resolution backbones DC$5$-R$50$ and
DC$5$-R$101$ is $10\times$ faster than the original DETR, and for low-
resolution backbones R$50$ and R$101$ $6.67\times$ faster. In other words,
conditional DETR performs better for stronger backbones with better
performance.

In addition, we report the results of single-scale DETR extensions: deformable
DETR-SS [53] and UP-DETR [5] in Table 1. Our results over R$50$ and
DC$5$-R$50$ are better than deformable DETR-SS: $40.9$ vs. $39.4$ and $43.8$
vs. $41.5$. The comparison might not be fully fair as for example parameter
and computation complexities are different, but it implies that the
conditional cross-attention mechanism is beneficial. Compared to UP-
DETR-R$50$, our results with fewer training epochs are obviously better.

Comparison to multi-scale and higher-resolution DETR variants. We focus on
accelerating the DETR training, without addressing the issue of high
computational complexity in the encoder. We do not expect that our approach
achieves on par with DETR variants w/ multi-scale attention and
$8\times$-resolution encoders, e.g., TSP-FCOS and TSP-RCNN [37] and deformable
DETR [53], which are able to reduce the encoder computational complexity and
improve the performance due to multi-scale and higher-resolution.

The comparisons in Table 2 surprisingly show that our approach on DC$5$-R$50$
($16\times$) performs same as deformable DETR-R$50$ (multi-scale, $8\times$).
Considering that the AP of the single-scale deformable DETR-DC$5$-R$50$-SS is
$41.5$ (lower than ours $43.8$) (Table 1), one can see that deformable DETR
benefits a lot from the multi-scale and higher-resolution encoder that
potentially benefit our approach, which is currently not our focus and left as
our future work.

The performance of our approach is also on par with TSP-FCOS and TSP-RCNN. The
two methods contain a transformer encoder over a small number of selected
positions/regions (feature of interest in TSP-FCOS and region proposals in
TSP-RCNN) without using the transformer decoder, are extensions of FCOS [39]
and Faster RCNN [33]. It should be noted that position/region selection
removes unnecessary computation in self-attention and reduces computation
complexity dramatically.

###  4.3 Ablations

Reference points. We compare three ways of forming reference points
$\mathbf{s}$: (i) $\mathbf{s}=(0,0)$, same to the original DETR, (ii) learn
$\mathbf{s}$ as model parameters and each prediction is associated with
different reference points, and (iii) predict each reference point
$\mathbf{s}$ from the corresponding object query. We conducted the experiments
with ResNet-$50$ as the backbone. The AP scores are $36.8$, $40.7$, and
$40.9$, suggesting that (ii) and (iii) perform on par and better than (i).

Table 3: Ablation study for the ways forming the conditional spatial query. CSQ = our proposed conditional spatial query scheme. Please see the first two paragraphs in Section 5.3 for the meanings of CSQ variants. Our proposed CSQ manner performs better. The backbone ResNet-$50$ is adopted. Exp. | CSQ-C | CSQ-T | CSQ-P | CSQ-I | CSQ
---|---|---|---|---|---
GFLOPs | $89.3$ | $89.5$ | $89.3$ | $89.5$ | $89.5$
AP | $37.1$ | $37.6$ | $37.8$ | $40.2$ | $40.9$

The effect of the way forming the conditional spatial query. We empirically
study how the transformation $\boldsymbol{\uplambda}_{q}$ and the positional
embedding $\mathbf{p}_{s}$ of the reference point, used to form the
conditional spatial query
$\mathbf{p}_{q}=\boldsymbol{\uplambda}_{q}\odot\mathbf{p}_{s}$, make
contributions to the detection performance.

We report the results of our conditional DETR, and the other ways forming the
spatial query with: (i) CSQ-P - only the positional embedding
$\mathbf{p}_{s}$, (ii) CSQ-T - only the transformation
$\boldsymbol{\uplambda}_{q}$, (iii) CSQ-C - the decoder content embedding
$\mathbf{f}$, and (iv) CSQ-I - the element-wise product of the transformation
predicted from the decoder self-attention output $\mathbf{c}_{q}$ and the
positional embedding $\mathbf{p}_{s}$. The studies in Table 3 imply that our
proposed way (CSQ) performs overall the best, validating our analysis about
the transformation predicted from the decoder embedding and the positional
embedding of the reference point in Section 3.3.

Focal loss and offset regression with respect to learned reference point. Our
approach follows deformable DETR [53]: use the focal loss with $300$ object
queries to form the classification loss and predict the box center by
regressing the offset with respect to the reference point. We report how the
two schemes affect the DETR performance in Table 4. One can see that
separately using the focal loss or center offset regression without learning
referecence points leads to a slight AP gain and combining them together leads
to a larger AP gain. Conditional cross-attention in our approach built on the
basis of focal loss and offset regression brings a major gain $4.0$.

The effect of linear projections $\mathbf{T}$ forming the transformation.
Predicting the conditional spatial query needs to learn the linear projection
$\mathbf{T}$ from the decoder embedding (see Equation 6). We empirically study
how the linear projection forms affect the performance. The linear projection
forms include: an _identity_ matrix that means not to learn the linear
projection, a _single_ scalar, a _block_ diagonal matrix meaning that each
head has a learned $32\times 32$ linear projection matrix, a _full_ matrix
without constraints, and a _diagonal_ matrix. Figure 5 presents the results.
It is interesting that a single-scalar helps improve the performance, maybe
due to narrowing down the spatial range to the object area. Other three forms,
_block_ diagonal, _full_ , and _diagonal_ (ours), perform on par.

Table 4: The empirical results about the focal loss (FL), offset regression (OR) for box center prediction, and our conditional spatial query (CSQ). The backbone ResNet-50 is adopted. OR |  FL |  CSQ |  GFLOPs |  AP
---|---|---|---|---
|  |  |  $85.5$ |  $34.9$
✓ |  |  |  $85.5$ |  $35.0$
|  ✓ |  |  $88.1$ |  $35.3$
✓ |  ✓ |  |  $88.1$ |  $36.9$
✓ |  ✓ |  ✓ |  $89.5$ |  $40.9$
IdentitySingleBlockFullDiagonal$37$$38$$39$$40$$41$$37.8$$39$$40.8$$40.7$$40.9$AP
Figure 5: The empirical results for different forms of linear projections that
are used to compute the spatial queries for conditional multi-head cross-
attention. Diagonal (ours), Full, and Block perform on par. The backbone
ResNet-$50$ is adopted.

##  5 Conclusion

We present a simple conditional cross-attention mechanism. The key is to learn
a spatial query from the corresponding reference point and decoder embedding.
The spatial query contains the spatial information mined for the class and box
prediction in the previous decoder layer, and leads to spatial attention
weight maps highlighting the bands containing extremities and small regions
inside the object box. This shrinks the spatial range for the content query to
localize the distinct regions, thus relaxing the dependence on the content
query and reducing the training difficulty. In the future, we will study the
proposed conditional cross-attention mechanism for human pose estimation [8,
41, 36] and line segment detection [43].

Acknowledgments. We thank the anonymous reviewers for their insightful
comments and suggestions on our manuscript.

## References

  * [1] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao.  Yolov4: Optimal speed and accuracy of object detection.  CoRR, abs/2004.10934, 2020.
  * [2] Zhaowei Cai and Nuno Vasconcelos.  Cascade R-CNN: delving into high quality object detection.  In CVPR, 2018.
  * [3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, 2020.
  * [4] Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu.  Dynamic convolution: Attention over convolution kernels.  In CVPR, 2020.
  * [5] Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen.  UP-DETR: unsupervised pre-training for object detection with transformers.  CoRR, abs/2011.09094, 2020.
  * [6] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian.  Centernet: Keypoint triplets for object detection.  In ICCV, 2019.
  * [7] Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, and Hongsheng Li.  Fast convergence of DETR with spatially modulated co-attention.  CoRR, abs/2101.07448, 2021.
  * [8] Zigang Geng, Ke Sun, Bin Xiao, Zhaoxiang Zhang, and Jingdong Wang.  Bottom-up human pose estimation via disentangled keypoint regression.  In CVPR, pages 14676–14686, June 2021.
  * [9] Ross B. Girshick.  Fast R-CNN.  In ICCV, 2015.
  * [10] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In AISTATS, 2010.
  * [11] Maosheng Guo, Yu Zhang, and Ting Liu.  Gaussian transformer: A lightweight approach for natural language inference.  In AAAI, 2019.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [13] Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Andrea Vedaldi.  Gather-excite: Exploiting feature context in convolutional neural networks.  In NeurIPS, 2018.
  * [14] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In CVPR, 2018.
  * [15] Lichao Huang, Yi Yang, Yafeng Deng, and Yinan Yu.  Densebox: Unifying landmark localization with end to end object detection.  CoRR, abs/1509.04874, 2015.
  * [16] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc Van Gool.  Dynamic filter networks.  In NeurIPS, 2016.
  * [17] Guolin Ke, Di He, and Tie-Yan Liu.  Rethinking positional encoding in language pre-training.  CoRR, abs/2006.15595, 2020.
  * [18] Jaeyoung Kim, Mostafa El-Khamy, and Jungwon Lee.  T-GSA: transformer with gaussian-weighted self-attention for speech enhancement.  In ICASSP, 2020.
  * [19] Tao Kong, Fuchun Sun, Huaping Liu, Yuning Jiang, and Jianbo Shi.  Foveabox: Beyond anchor-based object detector.  CoRR, abs/1904.03797, 2019.
  * [20] Harold W. Kuhn.  The hungarian method for the assignment problem.  Naval Research Logistics Quarterly, 1995.
  * [21] Hei Law and Jia Deng.  Cornernet: Detecting objects as paired keypoints.  In ECCV, 2018.
  * [22] Hei Law, Yun Teng, Olga Russakovsky, and Jia Deng.  Cornernet-lite: Efficient keypoint based object detection.  In BMVC. BMVA Press, 2020.
  * [23] Yanghao Li, Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang.  Scale-aware trident networks for object detection.  In ICCV, pages 6054–6063, 2019.
  * [24] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  TPAMI, 2020.
  * [25] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  In ECCV, 2014.
  * [26] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yang Fu, and Alexander C. Berg.  SSD: single shot multibox detector.  In ECCV, 2016.
  * [27] Ilya Loshchilov and Frank Hutter.  Fixing weight decay regularization in adam.  In ICLR, 2017.
  * [28] Xin Lu, Buyu Li, Yuxin Yue, Quanquan Li, and Junjie Yan.  Grid R-CNN.  In CVPR, 2019.
  * [29] Jiangmiao Pang, Kai Chen, Jianping Shi, Huajun Feng, Wanli Ouyang, and Dahua Lin.  Libra R-CNN: towards balanced learning for object detection.  In CVPR, 2019.
  * [30] Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In CVPR, 2016.
  * [31] Joseph Redmon and Ali Farhadi.  YOLO9000: better, faster, stronger.  In CVPR, 2017.
  * [32] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  CoRR, abs/1804.02767, 2018.
  * [33] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun.  Faster R-CNN: towards real-time object detection with region proposal networks.  TPAMI, 2017.
  * [34] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian D. Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In CVPR, 2019.
  * [35] Guanglu Song, Yu Liu, and Xiaogang Wang.  Revisiting the sibling head in object detector.  In CVPR, 2020.
  * [36] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang.  Deep high-resolution representation learning for human pose estimation.  In CVPR, pages 5693–5703, 2019.
  * [37] Zhiqing Sun, Shengcao Cao, Yiming Yang, and Kris Kitani.  Rethinking transformer-based set prediction for object detection.  CoRR, abs/2011.10881, 2020.
  * [38] Zhi Tian, Chunhua Shen, and Hao Chen.  Conditional convolutions for instance segmentation.  In ECCV, 2020.
  * [39] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  FCOS: fully convolutional one-stage object detection.  In ICCV, 2019.
  * [40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [41] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao.  Deep high-resolution representation learning for visual recognition.  TPAMI, 2019.
  * [42] Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen.  Solov2: Dynamic and fast instance segmentation.  In NeurIPS, 2020.
  * [43] Yifan Xu, Weijian Xu, David Cheung, and Zhuowen Tu.  Line segment detection using transformers without edges.  In CVPR, pages 4257–4266, June 2021.
  * [44] Brandon Yang, Gabriel Bender, Quoc V. Le, and Jiquan Ngiam.  Condconv: Conditionally parameterized convolutions for efficient inference.  In NeurIPS, 2019.
  * [45] Changqian Yu, Bin Xiao, Changxin Gao, Lu Yuan, Lei Zhang, Nong Sang, and Jingdong Wang.  Lite-hrnet: A lightweight high-resolution network.  In CVPR, pages 10440–10450, June 2021.
  * [46] Jiahui Yu, Yuning Jiang, Zhangyang Wang, Zhimin Cao, and Thomas S. Huang.  Unitbox: An advanced object detection network.  In MM, 2016.
  * [47] Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z. Li.  Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection.  In CVPR, 2020.
  * [48] Minghang Zheng, Peng Gao, Xiaogang Wang, Hongsheng Li, and Hao Dong.  End-to-end object detection with adaptive clustering transformer.  CoRR, abs/2011.09315, 2020.
  * [49] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  CoRR, abs/1904.07850, 2019.
  * [50] Xingyi Zhou, Jiacheng Zhuo, and Philipp Krähenbühl.  Bottom-up object detection by grouping extreme and center points.  In CVPR, 2019.
  * [51] Chenchen Zhu, Fangyi Chen, Zhiqiang Shen, and Marios Savvides.  Soft anchor-point object detection.  In ECCV, 2020.
  * [52] Chenchen Zhu, Yihui He, and Marios Savvides.  Feature selective anchor-free module for single-shot object detection.  In CVPR, 2019.
  * [53] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable DETR: deformable transformers for end-to-end object detection.  CoRR, abs/2010.04159, 2020.


# Extremely Lightweight Quantization Robust
Real-Time Single-Image Super Resolution for Mobile Devices

Mustafa Ayazoglu
Aselsan Research
Ankara  Turkey
<EMAIL_ADDRESS>
###### Abstract

Single-Image Super Resolution (SISR) is a classical computer vision problem
and it has been studied for over decades. With the recent success of deep
learning methods, recent work on SISR focuses solutions with deep learning
methodologies and achieves state-of-the-art results. However most of the
state-of-the-art SISR methods contain millions of parameters and layers, which
limits their practical applications. In this paper, we propose a hardware
(Synaptics Dolphin NPU) limitation aware, extremely lightweight quantization
robust real-time super resolution network (XLSR). The proposed model’s
building block is inspired from root modules introduced in [15] for Image
classification. We successfully applied root modules to SISR problem, further
more to make the model uint8 quantization robust we used Clipped ReLU at the
last layer of the network and achieved great balance between reconstruction
quality and runtime. Furthermore, although the proposed network contains 30x
fewer parameters than VDSR [16] its performance surpasses it on Div2K
validation set. The network proved itself by winning Mobile AI 2021 Real-Time
Single Image Super Resolution Challenge.

##  1 Introduction

Super Resolution is a classical computer vision problem and it has been
studied over decades. The aim of the problem is obtaining the high resolution
image from either single or multiple low resolution images. In both settings
the problem is ill-posed. Earlier in the literature the problem is approached
with traditional methods later with the advancement and success of deep
learning, the problem is approached with deep-learning.

(a)

(b)

(c)

(d)

Figure 1: Effect of quantization of existing methods and our proposed method.
Note the shift in colors and staircase gradient on the sky of quantized ESPCN
and FSRCNN outputs

Deep learning first applied to SISR problem by Dong et al. in [5] and achieved
state-of-the-art results compared to traditional methods with only 3 layers.
Researchers realized that upscaling the input image on the later stages of the
network reduces computational resources and proposed FSRCNN [6] further more
instead of ReLU, PReLU is utilized as the activation function, while FSRCNN
used deconvolution layer as the upscaling module. Shi et al. [24] proposed
another widely used upscaling layer so called Depth2Space for real-time SISR
problem. Later, VDSR [16] emerged and increased the number of layers and
number of parameters and showed that with the increased number of parameters
the reconstruction quality can be improved. Increasing and complicating the
network architecture trend continued later on with EDSR [20] and WDSR [28]
later on attention mechanisms applied to SISR in [22, 11] Indeed these methods
achieved superior reconstruction performance on standard datasets.

While previously mentioned methods focus on reconstruction quality, there are
also other methods based on Generative Adversarial Networks (GAN). GAN based
methods [18, 26] focuses on perceptual quality.

However, none of these methods and many more in the literature can be directly
applied and run in mobile devices either because of extremely large number of
parameters and/or severely affected performances due to uint8 quantization,
further more some specific hardware limitations may also limit their
applicability.

Method | Number of Parameters
---|---
EDSR [20] | 43M
WDSR [28] | 75M
VDSR* [16] | 668K
IMDN [11] | 500K
FSRCNN* [6] | 25K
ESPCN* [24] | 31K
XLSR (Ours) | 22K
Table 1: Example set of number of parameters from high performing deep
learning networks.
(*) For a fair comparison of parameters model is assumed to accept RGB input
and scaling is x3

To solve the real-time image super resolution problem with mobile device
deployment requirement, we proposed an extremely lightweight super resolution
(XLSR) network by investigating the limiting factors of the existing models to
run in mobile devices and modifying successfully applied mobile network
building blocks for different problems in the literature (See Figure 2). Our
method designed with reconstruction quality focus since Mobile AI 2021 Real-
time image super resolution challenge scoring formula was based on PSNR as
follows;

$\displaystyle Score(PSNR,runtime)=\frac{2^{2.{PSNR}}}{C.runtime}$ (1) Where C
is a constant

Building an extremely lightweight network with low number of parameters was
not enough by itself since the model for the challenge needs to be fully uint8
quantized. To make the model quantization robust and keep it still running
fast on the deployment platform, instead of ”Linear” activation function,
which is common at the very last layer of SISR deep learning models, we used
”Clipped ReLU” and carefully tuned training methodology. Note that this is
required since added non-linearity at the last layer, although helps with
quantization, unfortunately makes the model optimization harder due to extra
flat optimization surface. It is shown that the model trained with this
mentality is very robust to quantization and only $\sim$0.3dB PSNR drop is
observed on the quantized model with standard tensorflow post training
quantization compared to float16/32 model’s PSNR on Div2K validation dataset.

Figure 2: Our proposed network

##  2 Related Works

Successful deployment of deep learning models to mobile devices opens broader
application areas to these models and increases their usability along with
academic contribution. These motivations lead to advancements on both AI
specific hardware and on the mobile friendly models. Hardware focusing on deep
learning deployment started with the efforts of Qualcomm and Arm later on
continued with specialized AI silicon from many different vendors [14]. On the
other hand, many researchers focusing on mobile deployment, come up with many
different ideas for mobile friendly models. The examples of these ideas are;
for Object detection SqueezeNet [12] which achieves AlexNet [17] performance
on ImageNet with 50x lesser number of parameters. MobileNetV1 [9] uses
depthwise separable convolutions and achieves better performance than
SqueezeNet. ShuffleNet [31] further increases ImageNet performance with
lighter requirements by utilizing group convolutions and channel shuffling
operator. Similar to ShuffleNet, DeepRoots proposes the usage of 1x1
convolutions instead of channel shuffling. For face verification,
MobileFaceNets [3] uses depthwise separable convolutions and bottleneck layers
to build an application specific lightweight network. For image super
resolution IMDN [11] uses channel splitting to build a lighter network.
Different from previous approaches, NASNet [33] searches for the optimal
architecture and surpasses many state-of-the-art methods.

Apart from focusing on network itself, the literature also focuses on
quantization [32, 10, 4, 19] since in many cases quantization is not an option
but a hardware declared must. On the other hand, it has been shown by Hinton
et al. [8] that knowledge distillation can help a simple model to
achieve/surpass a complex model’s performance. Furthermore, Mishra et al. [21]
combined knowledge distilling with quantization. Another methodology using a
pretrained complex model while building a lighter weight model is; channel
sparsification and pruning, with this methodology a complex model can be
slimmed by removing unnecessary channels from the filters [7]

Although the most of the aforementioned methods are designed for different
areas other than SISR, they can be still very useful while building a real-
time performant SISR model. Many of these ideas are succesfully applied to the
SISR problem [30, 29] The methodologies to run/design a performant deep
learning method for mobile devices can be summarized as follows;

  * •

Hand-Designed Architectures

  * •

Efficient Building Block Design

  * •

Network Pruning / Sparsification

  * •

Network Quantization

  * •

Network Architecture Search (NAS)

  * •

Knowledge Distillation

In this paper, we followed the first and second methodologies while creating
our submission into Mobile AI 2021 Real-Time Single Image Super-Resolution
Challenge [13]. The reason and motivation for this approach was; there were
many different hardware limitations of the deployment platform (Synaptics
Dolphin NPU) that needs to be taken into account in the challenge which are
harder to incorporate into a common method. Furthermore, the challenge
required full uint8 quantization of the model (not allowing partial
quantization such as leaving the first and the last layers floating, which are
known to be severely affecting the accuracy if quantized [4]). The full uint8
quantization requirement adds extra complexity to the problem since, SISR
problem is severely affected by the quantization operation if the model at the
hand is not designed/trained/quantized properly.

The deployment hardware limiting factors can be summarized as

  * •

Elementwise operations such as (Addition, Subtraction) is not optimized

  * •

Reshaping and transpose operations are not optimized and extremely slow

  * •

Per-channel quantization is not supported

  * •

Multiple and especially long skip connections require a lot of data swaps with
slow CPU DRAM

##  3 Proposed Method

In this section, we describe the details of the proposed network and the
motivation behind the design ideas while connecting these with literature and
hardware limitations. As mentioned before, we designed our proposed
architecture by hand and adopted an efficient building block for SISR problem
inspired from [31, 15, 27].

###  3.1 Building Block Selection

Group convolutions are first used in AlexNet although the GPU hardware
limitation forced such methodology. It has been shown that when used wisely,
group convolutions can increase accuracy while decreasing computational costs.
Because of these properties, they are often used in mobile focused networks.
They are used in ResNeXt along with skip connections, in ShuffleNet with
cascaded channel shuffling and in DeepRoots with cascaded 1x1 convolutions.

From the point of the view of the SISR problem challenge and hardware
limitations, using channel shuffling is infeasible since reshape and transpose
operations are not optimized in the deployment hardware which eliminates
ShuffleNet Block. Skip connections and residual in residual type of structures
help with model converge and allow deeper architectures and are utilized in
many state-of-the-art networks. However, multiple parallel skip connections
are slow and elementwise addition is not optimized and hence ResNeXt block is
not very well optimized for the deployment hardware.

On the other hand, when channel shuffling operator is relaxed to 1x1
convolution (to still allow interchannel communication) or skip connection in
ResNext block is removed, we arrive at the building block used in our network
(See Figure 3). Note that instead of group convolutions, depthwise
convolutions can also be a candidate for building block. However, as noted by
Sheng et al. in [23] depthwise convolutions can cause large quantization error
when used without special precaution, this is what we have also observed
empirically in our experiments.

Figure 3: Building block used in the proposed network

An important aspect we have also taken into account while designing our
proposed model was elementwise operations are not optimized in the deployment
hardware. Thus, we avoided all addition and scaling operations and used
concatenation operation when necessary and input data scaling and
normalization were not used.

###  3.2 Quantization Friendly Architecture

While deploying deep learning models to mobile devices, it is very common to
use quantization especially 8-bit quantization is very important since it is
naturally supported by many mobile hardware as with the deployment hardware of
Mobile AI 2021 Real-time single image super resolution challenge. However,
applying uint8 quantization scheme to any well performing (in float32 or
float16) super resolution model and hoping to arrive at a well performing
integer quantized model is very naive and usually not working well in practice
(See Figure 4). To get over this problem and still have a well performing
model, the model should be modified and made quantization friendly. On the
other hand, this modification should be subtle and still be quickly
executable, otherwise it would contradict with real-time inference
requirement.

To make a model quantization friendly, we should first focus on the reasons
behind why quantization adversely affects the accuracy. Linear output
activation function is very common among super resolution models and it helps
with the model optimization. Although, not strictly enforced when the model
converges we are pretty sure that the output will be bounded in between 0-255
(or 0-1.0). However, if such a model is quantized, the quantized output tend
to be dull and accuracy can drop about 5-7dB compared to float16/32 model. The
reason we believe is the following; during the earlier steps of training, the
output is not guaranteed to be in between 0-1.0 and intermediate activations
can also be unbounded or might contain outliers. Later on, if we continue with
training the training data enforces boundedness in the model output
indirectly. However, the intermediate activations of the model are not acted
upon and the model may converge to a point near where its intermediate
activations are unbounded, since model usually visits these points in the
early stages of the training. These allowed intermediate unbounded activations
create outliers (very large a few numbers). The outliers in intermediate layer
activations, when uint8 quantized leads to some important information
carrying, comparably low amplitude values to be zeroed out. Hence effective
signal energy reaching to the last layer drops, which results in dull colors
and drastic PSNR drops.

With this motivation we believe if the intermediate layer activations are
forced not to visit these outlier creating points from the beginning of the
training, the model would be more quantization friendly. This is exactly what
happens when the model is trained with Clipped ReLU at the last layer. A model
trained with this mentality results in only $\sim$0.2-0.5dB loss with respect
to its floating counterpart. This is better seen in Table 2 where we trained
our proposed model without Clipped ReLU and quantized the resulting model and
noted Div2K [1] validation accuracy.

Activation | PSNR (fp32) | PSNR (uint8) | Drop
---|---|---|---
Linear | 30.11 | 24.72 | 5.39
Clipped ReLU | 30.10 | 29.82 | 0.28
Table 2: Effect of Clipped ReLU and Div2K Validation Set PSNR Results with
floating and uint8 quantized models

Note that Clipped ReLU is

$\displaystyle Clipped\\_ReLU(x)=max(0,min(x,1))$ (2)

so it is very cheap to calculate. Thus the computational burden added to the
model is minimal which is desired.

Furthermore, although placing a single Clipped ReLU was enough to regularize
the intermediate activations of our proposed network, when a model gets
deeper, regularization effect might vanish for deeper layers, to overcome this
issue we suggest to change a few of the ReLU’s with Clipped ReLU’s. We believe
that clipping only a few ReLU’s is enough to regularize the intermediate
activations though we have not experimented on this idea for this paper. Also
note that for intermediate Clipped ReLU’s clipping value does not need to be 1
and it should either be experimentally found or it should be included in the
optimization as done in [19].

One drawback of using Clipped ReLU at the output layer is unfortunately the
model is harder to optimize since it creates a lot of flat regions with
minimal or no derivative direction. Because of this reason, to converge to a
better model, a few training tricks needs to be employed which are explained
in training details.

##  4 Experiments

###  4.1 Datasets

We used the supplied Div2K Dataset [1] for the challenge and no extra data is
used. The dataset consist of 800 high quality training images and 100
validation images along with 100 test images. During the challenge the test
images were not released. Due to this, while reporting our results we only
used validation set results (which are not used either for training or
validation). For a fair comparison, on standard benchmark datasets (Set5,
Set14, BSD100, Manga109, Urban100) with the existing work, we used our
floating point model and used Y channel of our output.

###  4.2 Training Details

For training, we splitted 800 training images into two sets, we used the first
792 of training images for training while keeping last 8 images for
validation. We cropped random 32x32 LR images and for data augmentation we
used geometric transformations (8 transformations - original, rotations,
flips) with equal probability. Furthermore, to give more robustness to
illumination changes we randomly scaled the intensity of images with 1, 0.7
and 0.5 randomly. As the loss function, we used Charbonnier loss [2] with
$\epsilon=0.1$ as defined in (3) since it is the smoother version of L1 and we
empirically found out that it works better with Clipped ReLU

$\displaystyle Charbonnier(x)=\sqrt{x^{2}+\epsilon^{2}}$ (3)

As mentioned above, the critical part of the model development was to include
Clipped ReLU activation function. This is however on the other hand results in
a harder problem. So to still be able to converge to a performant model
following tricks were utilized;

  * •

We used a triangular cyclic learning rate scheduling strategy [25], which
starts with a very low value (5e-5) for the first epoch and quickly increases
to the top value(25e-4) in 50 epochs and slowly decreases to a low value till
5000 epochs (1e-4).

  * •

Trained the model with mini batch size 16 for 5000 epochs with each epoch
containing 100 mini batches. At the end of each epoch, we calculated the
validation set PSNR (last 8 images of training set) and saved the best model
for quantization

  * •

Used Adam optimizer with default beta1=0.9 and beta2=0.999 and epsilon=1e-8
values

  * •

Initialized Conv2D layers with He-Normal with 0.1 Variance Scaling [20] The
motivation behind this was to initialize kernels to closer to zero and avoid
large numbers which might in turn create outliers in activation

For quantization, we used standard Tensorflow Post-Quantization strategy and
trained the model on NVIDIA RTX 2080 Super. It takes about 2 hours to train
our proposed model in that hardware.

Our quantized model achieves the following runtimes in Samsung Galaxy A21s
using AI-Benchmark Tool 111https://ai-benchmark.com/download [14];

  * •

CPU Runtime 1130ms

  * •

NNAPI Runtime 1150ms

  * •

GPU Delegate Runtime 620ms

XLSR also achieves an exceptional target platform runtime of 45ms. Table 3
shows the results from Mobile AI 2021 Real-Time Super Resolution Challenge
[13].

Method | PSNR | SSIM | Runtime | Score
---|---|---|---|---
XLSR (ours) | 29.58 | 0.86 | 44.85ms | 51.02
2nd Method | 29.41 | 0.8537 | 38.32ms | 47.18
3rd Method | 29.52 | 0.8607 | 62.25ms | 33.82
4th Method | 28.82 | 0.8428 | 76.61ms | 10.41
5th Method | 28.92 | 0.8486 | 718.32ms | 1.28
Table 3: Mobile 2021 Real-Time Single Image Super Resolution Results and Scores on Div2K Test Dataset (not released) Dataset | Scale | Bicubic | FSRCNN | ESPCN | VDSR | IMDN | EDSR | XLSR (ours)
---|---|---|---|---|---|---|---|---
Set5 | x3 | 30.41 | 33.16 | 33.13 | 33.66 | 34.36 | 34.65 | 33.42
Set14 | x3 | 27.55 | 29.43 | 29.42 | 29.77 | 30.32 | 30.52 | 29.73
B100 | x3 | 27.22 | 28.60 | 28.50 | 28.82 | 29.09 | 29.25 | 28.55
Urban100 | x3 | 24.47 | 26.48 | 26.41 | 27.14 | 28.17 | 28.80 | 26.71
Manga109 | x3 | 26.99 | 30.98 | 30.79 | 32.01 | 33.61 | - | 31.63
Div2K(Val) | x3 | 28.22 | - | - | 30.09 | - | 31.26 | 30.10
Table 4: Comparison of our proposed method with public benchmark scores of
other methods. Note that for fair and consistent comparison with the
literature, we used our floating point model and and converted RGB output of
our method to Luminance (Y) channel only while calculating PSNR for all
dataset except Div2K

PSNR results of our proposed method, compared with some of the work in the
literature on the standard benchmark dataset and Div2K validation set can be
seen in Table 4. Note that for a fair and consistent comparison with the
literature, the result presented here are from our float32 model. Comparison
of our quantized model with quantized FSRCNN and ESPCN can be found in Table
5.

Model | FSRCNN* (dB) | ESPCN* (dB) | XLSR (dB)
---|---|---|---
float32 | 29.67 | 29.54 | 30.10
uint8 | 21.95 | 23.93 | 29.82
drop | 7.72 | 5.61 | 0.28
Table 5: Effect of quantization on the existing methods and our method on
Div2K validation (x3 scale) set. (*) Our implementation and training of the
models to convert them to RGB input and RGB output

|
---
|
---
|
---

(a)

(b)

(c)

(d)

(e)

|
---
|
---
|
---

(f)

(g)

(h)

(i)

(j)

Figure 4: Example Images from Div2K Dataset. Note false colors and staircase
gradient effect on quantized ESPCN and FSRCNN output

##  5 Conclusion

In conclusion, we proposed a real-time single image super resolution method
driven by the hardware constraints of the Mobile AI 2021 challenge, although
it is driven by the target hardware, the resulting model is very efficient in
terms of runtime and model parameters. We believe it can run in many mobile
hardware with high performance. The proposed model easily surpasses many
reported PSNR results of famous FSRCNN and ESPCN models and it even reaches
VDSR in most of the public datasets and surpasses its performance on Div2K
validation set though it has 30x fewer parameters. Furthermore, the proposed
architecture is very robust to uint8 to quantization and only 0.28dB PSNR drop
is experienced when compared with float16/32 model. This property of the model
makes it a really good candidate for many mobile devices.

## References

  * [1] Eirikur Agustsson and Radu Timofte.  Ntire 2017 challenge on single image super-resolution: Dataset and study.  In IEEE Conf. Comput. Vis. Pattern Recog. Worksh., July 2017.
  * [2] P. Charbonnier, L. Blanc-Feraud, G. Aubert, and M. Barlaud.  Two deterministic half-quadratic regularization algorithms for computed imaging.  Proceedings of 1st International Conference on Image Processing, 2:168–172 vol.2, 1994.
  * [3] Sheng Chen, Yang Liu, Xiang Gao, and Zhen Han.  Mobilefacenets: Efficient cnns for accurate real-time face verification on mobile devices.  Biometric Recognition - 13th Chinese Conference, CCBR, pages 428–438, 2018.
  * [4] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I. Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan.  Pact: Parameterized clipping activation for quantized neural networks.  Int. Conf. Learn. Represent., 2018.
  * [5] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Learning a deep convolutional network for image super-resolution.  Eur. Conf. Comput. Vis., 8692:184–199, 2014.
  * [6] Chao Dong, Chen Change Loy, and Xiaoou Tang.  Accelerating the super-resolution convolutional neural network.  Eur. Conf. Comput. Vis., 9906:391–407, 2016.
  * [7] Yihui He, Xiangyu Zhang, and Jian Sun.  Channel pruning for accelerating very deep neural networks.  In Int. Conf. Comput. Vis., pages 1398–1406, 2017.
  * [8] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean.  Distilling the knowledge in a neural network.  In Adv. Neural Inform. Process. Syst., 2015.
  * [9] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  CoRR, abs/1704.04861, 2017.
  * [10] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio.  Quantized neural networks: Training neural networks with low precision weights and activations.  J. Mach. Learn. Res., 18:187:1–187:30, 2017.
  * [11] Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang.  Lightweight image super-resolution with information multi-distillation network.  ACM Int. Conf. Multimedia, 2019.
  * [12] Forrest N. Iandola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer.  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and 1mb model size.  Int. Conf. Learn. Represent., 2017.
  * [13] Andrey Ignatov, Radu Timofte, Maurizio Denna, and Abdel Younes.  Real-time quantized image super-resolution on mobile npus, mobile ai 2021 challenge: Report.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2021.
  * [14] Andrey Ignatov, Radu Timofte, Andrei Kulik, Seungsoo Yang, Ke Wang, Felix Baum, Max Wu, Lirong Xu, and Luc Van Gool.  Ai benchmark: All about deep learning on smartphones in 2019.  pages 3617–3635, 2019.
  * [15] Yani Ioannou, Duncan P. Robertson, Roberto Cipolla, and Antonio Criminisia.  Improving cnn efficiency with hierarchical filter groups.  IEEE Conf. Comput. Vis. Pattern Recog., pages 5977–5986, 2017.
  * [16] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee.  Accurate image super-resolution using very deep convolutional networks.  IEEE Conf. Comput. Vis. Pattern Recog., pages 1646–1654, 2016.
  * [17] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  2012\.
  * [18] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi.  Photo-realistic single image super-resolution using a generative adversarial network.  IEEE Conf. Comput. Vis. Pattern Recog., pages 105–114, 2019.
  * [19] Huixia Li, Chenqian Yan, Shaohui Lin, Xiawu Zheng, Yuchao Li, Baochang Zhang, Fan Yang, and Rongrong Ji.  Pams: Quantized super-resolution via parameterized max scale.  Eur. Conf. Comput. Vis., 2020.
  * [20] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced deep residual networks for single image super-resolution.  IEEE Conf. Comput. Vis. Pattern Recog. Worksh., pages 1132–1140, 2017.
  * [21] Asit K. Mishra and Debbie Marr.  Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy.  In Int. Conf. Learn. Represent., 2018.
  * [22] Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen.  Single image super-resolution via a holistic attention network.  Int. Conf. Comput. Vis., 2020.
  * [23] Tao Sheng, Chen Feng, Shaojie Zhuo, Xiaopeng Zhang, Liang Shen, and Mickey Aleksic.  A quantization-friendly separable convolution for mobilenets.  CoRR, abs/1803.08607, 2018.
  * [24] Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  IEEE Conf. Comput. Vis. Pattern Recog., pages 1874–1883, 2016.
  * [25] L. N. Smith.  Cyclical learning rates for training neural networks.  In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 464–472, 2017.
  * [26] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy.  Esrgan: Enhanced super-resolution generative adversarial networks.  pages 63–79, 2018.
  * [27] Saining Xie, Ross B. Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 5987–5995, 2017\.
  * [28] Jiahui Yu, Yuchen Fan, Jianchao Yang, Ning Xu, Zhaowen Wang, Xinchao Wang, and Thomas S. Huang.  Wide activation for efficient and accurate image super-resolution.  IEEE Conf. Comput. Vis. Pattern Recog. Worksh., 2018.
  * [29] Kai Zhang, Martin Danelljan, Yawei Li, Radu Timofte, Jie Liu, Jie Tang, Gangshan Wu, Yu Zhu, Xiangyu He, Wenjie Xu, Chenghua Li, Cong Leng, Jian Cheng, Guangyang Wu, Wenyi Wang, Xiaohong Liu, Hengyuan Zhao, Xiangtao Kong, Jingwen He, Yu Qiao, Chao Dong, Xiaotong Luo, Liang Chen, Jiangtao Zhang, Maitreya Suin, Kuldeep Purohit, A. N. Rajagopalan, Xiaochuan Li, Zhiqiang Lang, Jiangtao Nie, Wei Wei, Lei Zhang, Abdul Muqeet, Jiwon Hwang, Subin Yang, JungHeum Kang, Sung-Ho Bae, Yongwoo Kim, Liang Chen, Jiangtao Zhang, Xiaotong Luo, Yanyun Qu, Geun-Woo Jeon, Jun-Ho Choi, Jun-Hyuk Kim, Jong-Seok Lee, Steven Marty, Eric Marty, Dongliang Xiong, Siang Chen, Lin Zha, Jiande Jiang, Xinbo Gao, Wen Lu, Haicheng Wang, Vineeth Bhaskara, Alex Levinshtein, Stavros Tsogkas, Allan Jepson, Xiangzhen Kong, Tongtong Zhao, Shanshan Zhao, Hrishikesh P S, Densen Puthussery, Jiji C V au2, Nan Nan, Shuai Liu, Jie Cai, Zibo Meng, Jiaming Ding, Chiu Man Ho, Xuehui Wang, Qiong Yan, Yuzhi Zhao, Long Chen, Jiangtao Zhang, Xiaotong Luo, Liang Chen, Yanyun Qu, Long Sun, Wenhao Wang, Zhenbing Liu, Rushi Lan, Rao Muhammad Umer, and Christian Micheloni.  Aim 2020 challenge on efficient super-resolution: Methods and results.  pages –, 2020.
  * [30] Kai Zhang, Shuhang Gu, Radu Timofte, Zheng Hui, Xiumei Wang, Xinbo Gao, Dongliang Xiong, Shuai Liu, Ruipeng Gang, Nan Nan, Chenghua Li, Xueyi Zou, Ning Kang, Zhan Wang, Hang Xu, Chaofeng Wang, Zheng Li, Linlin Wang, Jun Shi, Wenyu Sun, Zhiqiang Lang, Jiangtao Nie, Wei Wei, Lei Zhang, Yazhe Niu, Peijin Zhuo, Xiangzhen Kong, Long Sun, and Wenhao Wang.  Aim 2019 challenge on constrained super-resolution: Methods and results.  pages –, 2019.
  * [31] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun.  Shufflenet: An extremely efficient convolutional neural network for mobile devices.  IEEE Conf. Comput. Vis. Pattern Recog., 2018.
  * [32] Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou.  Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients.  CoRR, abs/1606.06160, 2016.
  * [33] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le.  Learning transferable architectures for scalable image recognition.  IEEE Conf. Comput. Vis. Pattern Recog., pages 8697–8710, 2018.