# An Attention Module for Convolutional Neural Networks

Zhu Baozhou1, Peter Hofstee12, Jinho Lee3, Zaid Al-Ars1

###### Abstract

Attention mechanism has been regarded as an advanced technique to capture
long-range feature interactions and to boost the representation capability for
convolutional neural networks. However, we found two ignored problems in
current attentional activations-based models: the approximation problem and
the insufficient capacity problem of the attention maps. To solve the two
problems together, we initially propose an attention module for convolutional
neural networks by developing an AW-convolution, where the shape of attention
maps matches that of the weights rather than the activations. Our proposed
attention module is a complementary method to previous attention-based
schemes, such as those that apply the attention mechanism to explore the
relationship between channel-wise and spatial features. Experiments on several
datasets for image classification and object detection tasks show the
effectiveness of our proposed attention module. In particular, our proposed
attention module achieves $1.00\%$ Top-1 accuracy improvement on ImageNet
classification over a ResNet101 baseline and $0.63$ COCO-style Average
Precision improvement on the COCO object detection on top of a Faster R-CNN
baseline with the backbone of ResNet101-FPN. When integrating with the
previous attentional activations-based models, our proposed attention module
can further increase their Top-1 accuracy on ImageNet classification by up to
$0.57\%$ and COCO-style Average Precision on the COCO object detection by up
to $0.45$. Code and pre-trained models will be publicly available.

## Introduction

Convolutional neural networks have demonstrated to be the gold-standard to
solving various problems in the field of computer vision, including image
classification (He et al. 2016a), object detection (Liu et al. 2016; Ren et
al. 2015), and segmentation (Chen et al. 2017). To improve their performance,
many researchers explored various aspects of CNN design and implementation (Gu
et al. 2018).

To enrich the representation power of CNNs, we can build substantially deeper
convolutional neural networks. For example, VGGNet (Simonyan and Zisserman
2015) stacks very small $3\times 3$ convolutional layers, while ResNet (He et
al. 2016a) stacks residual blocks with skip connections. GoogLeNet (Szegedy et
al. 2015) uses multi-scales of processing to capture spatial correlation. Wide
ResNet (Zagoruyko and Komodakis 2016) shows that the width increase of
residual networks can enlarge the representation capability and reuse the
features better. Xception (Chollet 2017) and ResNeXt (Xie et al. 2017) expose
new dimensions to increase cardinalities. Besides, recent literature
(Jaderberg et al. 2015; Gregor et al. 2015; Xu et al. 2015) have investigated
the attention mechanism since it can improve not only the representation power
but also the representation of interests. Convolutional neural networks can
extract informative features by blending cross-channel and spatial information
(Hu et al. 2018). Attention modules (Woo et al. 2018; Linsley et al. 2019) can
learn ”where” and ”what” to attend in channel and space axes, respectively, by
focusing on important features and suppressing unnecessary ones of the
activations. Dynamic Filter Networks (Jia et al. 2016; Li et al. 2019b)
generate the filters conditioned on the input and show the flexibility power
of such filters because of their adaptive nature, which has become popular in
prediction (Klein, Wolf, and Afek 2015) and Natural Language Processing (Wu et
al. 2019). Both Dynamic Filter Networks and attention-based models are
adaptive based on the inputs, but there are significant differences between
them. Attention-based models (Hu et al. 2018; Woo et al. 2018) produce
attention maps using the attention mechanism to operate on the activations of
convolution. On the contrary, Dynamic Filer Networks (Su et al. 2019; Liu et
al. 2019) generate input information-specific kernels, such as position-
specific kernels (Su et al. 2019), semantic label map-specific kernels (Liu et
al. 2019), and few-shot learning setting-specific kernels (Zhao et al. 2018),
which work as the weights of convolution. Our proposed attention module
leverages the attention mechanism to compute the attention maps for attending
the activations of convolution, so it is clear to categorized the models
applied with our proposed attention module as attention-based models instead
of Dynamic Filter Networks.

Based on our analysis, the approximation problem and the insufficient capacity
problem of the attention maps are ignored in current attentional activations-
based models. Motivated by solving the two problems together, we develop an
attention module and inspect the complementary relationship between our
proposed attention module and previously published attention-based models,
such as the attention augmented models and the attentional activations-based
models. Our contributions are summarized as follows.

  * •

We point out and analyze two ignored problems of the current attentional
activations-based models: the approximation problem and the insufficient
capacity problem of the attention maps. To address the two problems together,
we originally propose an attention module by developing an AW-convolution,
where the shape of the attention maps matches that of the weights instead of
the activations.

  * •

Our proposed attention module is a complementary method to previous attention
mechanism-based modules, such as Attention Augmented (AA) convolution (Bello
et al. 2019), the SE (Hu, Shen, and Sun 2018) and CBAM (Woo et al. 2018)
modules in the attentional activations-based models. Integrating with our
proposed attention module, the accuracy of AA-Net, SE-Net, and CBAM-Net will
be improved further.

  * •

We use image classification and object detection tasks to demonstrate the
effectiveness of our proposed attention module. With negligible computational
complexity increase, our proposed attention module can boost the image
classification and object detection task performance, and it can achieve
better accuracy when integrating with other attention-based models.

## Related work

In this section, we discuss the recent developments of network engineering and
attention mechanism.

### Network engineering

”Network engineering” has been one of the most active research areas since it
targets building powerful convolutional neural networks on image
classification, which are the backbones of various computer vision tasks and
ensure their remarkable performance (Kornblith, Shlens, and Le 2019).
Increasing the depth of convolutional neural networks has been regarded as an
intuitive way to boost performance, which is the philosophy of VGGNet
(Simonyan and Zisserman 2015) and ResNet (He et al. 2016a). In addition, since
the skip connection from ResNet shows a strong ability to assist the gradient
flow, WideResNet (Zagoruyko and Komodakis 2016), PyramidNet (Han, Kim, and Kim
2017), Inception-ResNet (Szegedy et al. 2017), and ResNeXt (He et al. 2016b;
Xie et al. 2017) are ResNet-based versions proposed to explore further the
influence of the width, the increase of the width, the multi-scale and the
cardinality of convolution, respectively. In terms of efficiency, DenseNet
(Huang et al. 2017) reuses the feature maps by concatenating the feature maps
from different layers. In particular, MobileNet (Howard et al. 2017, 2019) and
ShuffleNet (Ma et al. 2018) series present the advantage of depthwise
convolution and the shuffle operation between various group convolutions,
respectively. Another design approach uses automated neural architecture
search, which achieves state-of-the-art performance regarding both accuracy
and efficiency across a range of computer vision tasks (Tan et al. 2019).

### Attention mechanism

The attention mechanism plays an important role in the human vision perceptron
since it can allocate the available resources to selectively focus on
processing the salient part instead of the whole scene (Rensink 2000; Corbetta
and Shulman 2002). Multiple attention mechanisms are used to address a known
weakness in convolution (Chen et al. 2018; Szegedy et al. 2015; Ke, Maire, and
Yu 2017; Chen et al. 2019; Hu, Shen, and Sun 2018; Linsley et al. 2019), by
capturing long-range information interactions (Bello et al. 2018; Vinyals,
Fortunato, and Jaitly 2015). The Inception family of architectures (Szegedy et
al. 2015, 2017), Multigrid Neural Architectures (Ke, Maire, and Yu 2017), and
Octave Convolution (Chen et al. 2019) aggregate the scale-space information,
while Squeeze-and-Excitation Networks (Hu, Shen, and Sun 2018) and Gather-
Excite (Hu et al. 2018) adaptively recalibrate channel-wise response by
modeling interdependency between channels. GALA (Linsley et al. 2019), CBAM
(Woo et al. 2018), and BAM (Park et al. 2018) refine the feature maps
separately in the channel and spatial dimensions. Attention Modules (Wang et
al. 2017) and self-attention (Vaswani et al. 2017; Bello et al. 2019) can be
used to exploit global context information. Precisely, non-local networks
(Wang et al. 2018) deploy self-attention as a generalized global operator to
capture the relationship between all pairwise convolutional feature maps
interactions. Except for applying the attention mechanism to computer vision
tasks (Li et al. 2019a), it has been a widespread adoption to modeling
sequences in Natural Language Processing (Yang et al. 2019).

## Proposed attention module

In this section, we analyze the two ignored problems in current attentional
activations-based models: the approximation problem and the insufficient
capacity problem of the attention maps. To address the two problems together,
we develop an attention module that mainly refers to the AW-convolution, where
the shape of attention maps matches that of the weights rather than the
activations. Besides, we refine the branch of calculating the attention maps
to achieve a better trade-off between efficiency and accuracy. Last but not
least, we integrate our proposed attention module with other attention-based
models to enlarge their representational capability.

### Motivation

First, we define basic notations in a traditional convolutional layer. In a
traditional convolutional layer, the input activations, weights, and output
activations are denoted as $I$, $K$, and $O$, respectively. For the input
activations $I\in R^{N\times C_{1}\times H\times W}$, $N$, $C_{1}$, $H$, and
$W$ refer to the batch size, the number of input channels, the height, and
width of the input feature maps, respectively. For the weights $K\in
R^{C_{2}\times C_{1}\times h\times w}$, $C_{2}$, $h$ and $w$ refer to the
number of output channels, the height and width of the weights, respectively.
For the output activations $O\in R^{N\times C_{2}\times H\times W}$, it is
computed as the convolution between the input activations $I$ and the weights
$K$. In particular, every individual value of the output activations
${O_{[l,p,m,n]}}$ is calculated as follows.

$\displaystyle{O_{[l,p,m,n]}}=\text{Convolution}(I,K)$ (1)
$\displaystyle=\sum\limits_{o=1}^{{C_{1}}}{\sum\limits_{j=1}^{h-1}{\sum\limits_{k=1}^{w-1}{{I_{[l,o,m^{\prime}+j,n^{\prime}+k]}}\times{K_{[p,o,j,k]}}}}}$

where $l=0,...,N-1$, $m=0,...,H-1$, $n=0,...,W-1$, $o=0,...,C_{1}-1$,
$p=0,...,C_{2}-1$, $m^{\prime}=m-\frac{h-1}{2}$, $n^{\prime}=n-\frac{w-1}{2}$.

To apply the attention mechanism on the input activations $I$, previous
attentional activations-based models produce the channel attention maps
$A_{c}\in R^{N\times C_{1}\times 1\times 1}$ and spatial attention maps
$A_{s}\in R^{N\times 1\times H\times W}$ separately. For example, applying the
channel attention maps $A_{c}$ on the input activations $I$ is presented as
$O=\text{Convolution}((I\odot{A_{c}}),K)$, where $\odot$ refers to the
Hadamard product and broadcasting during element-wise multiplication is
omitted.

#### Approximation problem of the attention maps

Instead of directly computing the three-dimensional attention map (N is
omitted, otherwise the attention maps are of four dimensions.), all the
current attentional activations-based models produce the attention maps
separately into the channel attention maps $A_{c}$ and spatial attention maps
$A_{s}$, which leads to the approximation problem of attention maps. However,
to thoroughly attend the input activations $I$, we need to compute the
attention maps $A_{f}\in R^{N\times C_{1}\times H\times W}$ and apply it as
$O=\text{Convolution}((I\odot{A_{f}}),K)$, which requires too much
computational and parameter overhead.

Inspired by convolution, we adopt local connection and attention maps sharing
to reduce the size of the attention maps. We compute the attention maps
$A_{a}\in R^{N\times C_{1}\times h\times w}$ as follows, where $\otimes$ is a
special element-wise multiplication since it only works associated with
convolution.

$\displaystyle{O_{[l,p,m,n]}}=\text{Convolution}(I\otimes A_{a},K)$ (2)
$\displaystyle=\sum\limits_{o=1}^{{C_{1}}}{\sum\limits_{j=1}^{h-1}{\sum\limits_{k=1}^{w-1}{({{I_{[l,o,m^{\prime}+j,n^{\prime}+k]}}}\times{{A_{a}}_{[l,o,j,k]}})\times{K_{[p,o,j,k]}}}}}$

##### Insufficient capacity problem of the attention maps

To compute different channels of the output activations of the convolution,
the input activations are constrained to be recalibrated by the same attention
maps, which indicates the insufficient capacity of the attention maps. As each
channel of the feature maps is considered as a feature detector, different
channels of the output activations of the convolution expect the input
activations to be adapted by different attention maps.

Take two channels of output activations of a convolutional layer as an
example, the two channels are responsible for recognizing rectangle shape and
triangle shape, respectively. Thus, it is reasonable for the two channels to
expect that there are different attention maps for attending the input
activations of the convolution (i.e., the attention maps to compute the
channel of recognizing the rectangle shape should be different from the
attention maps to compute the channel of recognizing the triangle shape). To
meet this expectation, we need to compute the attention maps $A_{ic}\in
R^{N\times C_{2}\times C_{1}\times 1\times 1}$ and apply it on the input
activations as follows.

$\displaystyle{O_{[l,p,m,n]}}=\text{Convolution}(I\odot{{A_{ic}}_{[l,p,:,:,:]}},K)$
(3)
$\displaystyle=\sum\limits_{o=1}^{{C_{1}}}{\sum\limits_{j=1}^{h-1}{\sum\limits_{k=1}^{w-1}{({{I_{[l,o,m^{\prime}+j,n^{\prime}+k]}}}\times{{A_{ic}}_{[l,p,o,0,0]}})\times{K_{[p,o,j,k]}}}}}$

To solve the approximation problem and the insufficient capacity problem of
the attention maps together (i.e., combining the solution of Equation.2 and
the solution of Equation. 3), we introduce our proposed attention module by
developing the AW-convolution. Specifically, we propose to compute the
attention maps $A\in R^{N\times C_{2}\times C_{1}\times h\times w}$ and apply
it as follows where the attention maps ${A_{[l,:,:,:,:]}}$ has the same shape
as that of the weights instead of the input activations. In this paper,
”Attentional weights” refers to the element-wise multiplication result between
the attention maps and the weights. Similarly, ”Attentional activations”
refers to the element-wise multiplication result between the attention maps
and the activations in previous attentional activations-based models. Thus,
$I\otimes A$ and ${A_{[l,:,:,:,:]}}\odot K$ represent the attentional
activations and attentional weights, respectively. To reduces half the number
of element-wise multiplications, we calculate attentional weights instead of
attentional activations as follows.

$\displaystyle{O_{[l,p,m,n]}}=\text{Convolution}(I\otimes A,K)$ (4)
$\displaystyle=\sum\limits_{o=1}^{{C_{1}}}{\sum\limits_{j=1}^{h-1}{\sum\limits_{k=1}^{w-1}{{{I_{[l,o,m^{\prime}+j,n^{\prime}+k]}}}\times({A_{[l,p,o,j,k]}}\times{K_{[p,o,j,k]}}})}}$
$\displaystyle=\text{Convolution}(I,{A_{[l,:,:,:,:]}}\odot K)$
$\displaystyle=\text{AW-Convolution}(I,A\odot K)$

(a) The AW-convolution architecture.

(b) The architecture of calculating attention maps $A$.

Figure 1: The architecture of our proposed attention module.

### AW-convolution in proposed attention module

The AW-convolution in our proposed attention module is presented in Figure
1(a). In this figure, the attention maps $A$ has five dimensions, which is
computed from the input activations $I$ as $A=F_{1}(I)$. $F_{1}$ is a function
to calculate the attention maps $A$ given the input activations $I$. Then, the
attentional weights $AK\in R^{N\times C_{2}\times C_{1}\times h\times w}$ is
calculated as $AK=F_{2}(A,K)=K+A\odot K$. $F_{2}$ is a function to calculate
the attentional weights $AK$ given the weights $K$ and the attention maps $A$.
Finally, the output activations $O$ is calculated from the input activations
$I$ and the attentional weights $AK$ as follows.

$\displaystyle{O_{[l,p,m,n]}}={F_{3}}(I,AK)$ (5) $\displaystyle=\text{AW-
Convolution}(I,AK)$
$\displaystyle=\sum\limits_{i=1}^{{C_{1}}}{\sum\limits_{j=1}^{h-1}{\sum\limits_{k=1}^{w-1}{{I_{[l,o,m^{\prime}+j,n^{\prime}+k]}}\times
A{K_{[l,p,o,j,k]}}}}}$
$\displaystyle=\text{Convolution}(I,A{K_{[l,:,:,:,:]}})$

where $F_{3}$ is a function to calculate the output activations $O$ given the
input activations $I$ and the attentional weights $AK$. Compared with the
traditional convolution, the attentional weights $AK$ of the AW-convolution in
our proposed attention module has five dimensions rather than four dimensions,
which are different from each other for every individual sample of the input
activations batch to convolute.

It is also worth explaining the definition of the function $F_{2}$.
$AK=K+A\odot K$ instead of $AK=A\odot K$ is used to describe the function
$F_{2}$ since it can be regarded as a residual design as follows.

$\displaystyle O={F_{3}}(I,AK)$ (6) $\displaystyle=\text{AW-
Convolution}(I,{F_{2}}(A,K))$ $\displaystyle=\text{Convolution}(I,K)+\text{AW-
Convolution}(I,A\odot K)$

### Calculating the attention maps $A$

As shown in Figure 1(b), the architecture to compute the attention maps $A$
(i.e., the definition of the function $F_{1}$) is presented, which can be
expressed as follows. Avgpool2d aggregates feature responses from the whole
spatial extent and embeds them into $A_{0}$, and Pointconv1 and Pointconv2
followed by Relu redistribute the pooled information to capture the dynamic
and no-linear dependencies between channels and spatial spaces.

$\displaystyle A={F_{1}}(I)=\text{Expand}{{}_{{C_{1}}}}({A_{2}})$ (7)
$\displaystyle=\text{Expand}{{}_{{C_{1}}}}(\text{Pointconv2}({A_{1}}))$
$\displaystyle=\text{Expand}{{}_{{C_{1}}}}(\text{Pointconv2}(\text{Pointconv1}({A_{0}})))$
$\displaystyle=\text{Expand}{{}_{{C_{1}}}}(\text{Pointconv2}(\text{Pointconv1}(\text{Avgpool2d}(I))))$

where Pointconv1 and Pointconv2 are pointwise convolutions. We add Batch
Normalization and Relu layers after Pointconv1, while adding Batch
Normalization and Sigmoid layers after Pointconv2, and they are omitted here
to provide a clear expression.

In Figure 1(b), Expand function along $C_{1}$ dimension, denoted as
$\text{Expand}_{C_{1}}$, is used as an example, and Expand function can be
also executed along $N$, $C_{2}$, $h$, and $w$ dimensions in a similar way.
$\text{Expand}{{}_{C_{1}}}$ function is used to expand the tensor $A_{2}\in
R^{N\times(C_{2}C_{1}/r_{C_{1}})\times h\times w}$ into the attention maps
$A\in R^{N\times C_{2}\times C_{1}\times h\times w}$ with the reduction ratio
$r_{C_{1}}$, including necessary squeeze, reshape, and expand operations.
$\text{Expand}_{C_{1}}$ can be expressed as follows.

$\displaystyle A=\text{Expand}{{}_{{C_{1}}}}({A_{2}})$ (8)
$\displaystyle={A_{2}}.\text{reshape}(N,{C_{2}},{C_{1}}/{r_{C_{1}}},h,w)$
$\displaystyle.\text{unsqueeze(dim=3)}\text{}$
$\displaystyle\text{.expand}(N,{C_{2}},{C_{1}}/{r_{C_{1}}},{r_{C_{1}}},h,w)\text{}$
$\displaystyle\text{.reshape}(N,{C_{2}},{C_{1}},h,w)$

Calculating the five-dimension attention maps $A$ is not an easy computational
task without careful design. Thus, we analyze the additional computational
complexity of an AW-convolution compared with a traditional convolution as a
reference to refine this design. Considering the trade-off between
computational complexity and accuracy, all the experiments in the remainder of
this paper use the same settings for the architecture of calculating the
attention maps $A$ in our proposed attention module, including
$r_{C1}={C_{1}}$, $r_{C_{2}}$ = $r_{hw}$ = 1, $r$ = 16, used in all the
stages, and $AK=K+A\odot K$ as the definition for the function $F_{2}$. The
details of refining the architecture of calculating the attention maps $A$ are
in Section C of Supplementary Material.

### Integrating with other attention-based modules

(a) Integrating with SE-ResNet/CBAM-ResNet.

(b) Integrating with AA-Wide-ResNet.

Figure 2: The schema of bottlenecks and blocks when integrating with our
proposed attention module.

In this section, we show how to integrate our proposed attention module with
the previous attention-based convolutional neural networks to demonstrate the
complementary relationship between our proposed attention module and other
attention-based modules. Since applying our proposed attention module is using
the AW-convolution to replace the traditional convolution, we can easily
integrate our proposed attention module with any convolutional neural networks
consisting of traditional convolution, including all the recently developed
attention-based models (Hu, Shen, and Sun 2018; Woo et al. 2018; Park et al.
2018; Linsley et al. 2019; Bello et al. 2019).

We choose the recent attentional activations-based models, i.e., SE-Net and
CBAM-Net, as examples to show how to integrate our proposed attention module
with other attention-based models. Here we use the popular ResNet (He et al.
2016a) as the backbone to apply the attention mechanism. As shown in Figure
2(a), the left side is the structure of a primary bottleneck in ResNet. The
middle one is the structure of a bottleneck with SE/CBAM modules in SE-
ResNet/CBAM-ResNet. Integrating the central bottleneck with our proposed
attention module is completed by replacing its $3\times 3$ convolution with a
$3\times 3$ AW-convolution, and its final structure in AW-SE-ResNet/AW-CBAM-
ResNet is shown on the right side. In summary, our proposed attention module
is a general module to be integrated seamlessly with any CNNs architectures,
including previous attention-based CNNs.

In Figure 2(b), we integrate our proposed attention module with Attention
Augmented (AA) convolutional networks and describe the architecture of their
possible AW-AA-blocks. Experiments of AW-AA-Wide-ResNet (Zagoruyko and
Komodakis 2016; Bello et al. 2019) on CIFAR-100 image classification
(Krizhevsky and Hinton 2009), as shown in Table 2, suggest that integrating
our proposed attention module with attention-based models should be explored
carefully since different integration architectures achieve different
accuracy. In some cases, an improper integration architecture leads to a small
accuracy drop. With a careful design for integration, our proposed attention
module is complementary to AA convolution and improves the accuracy of AA-Net
further.

In particular, we augment Wide-ResNet-28-10 by replacing the first convolution
of all the residual blocks with Attention Augmented convolution, which is the
AA-Wide-ResNet baseline. Here we set $N_{h}=8$ heads, $k=2v=0.2$, and a
minimum of $20$ dimensions per head for the keys as in (Bello et al. 2019). In
this figure, we can develop four possible architectures to integrate our
proposed attention module with AA-Wide-ResNet. On the left side of this
figure, we can build AA-Wide-ResNet-0 by replacing the second $3\times 3$
convolution with our AW-convolution in an AA-block. On the right side, there
are three traditional convolutions in an AA convolution, including conv1,
conv2, conv3. The conv1 is parallel to the multi-head attention, while the
conv2 and conv3 are used to calculate QKV (i.e., queries, keys, and values)
and to output attention maps, respectively. On the medial side, we can replace
one traditional convolution from conv1, conv2, or conv3 with our AW-conv1, AW-
conv2, or AW-conv3 to construct AW-AA-Wide-ResNet-1, AW-AA-Wide-ResNet-2, or
AW-AA-Wide-ResNet-3, respectively. The Top-1 accuracy of AW-AA-Wide-ResNet-0
is $1.87\%$ higher than the AA-Wide-ResNet baseline, while AW-AA-Wide-ResNet-3
shows worse performance by $0.10\%$ drop.

Model | Top-1 Error | Top-5 Error | GFLOPs | Parameters (M)
---|---|---|---|---
ResNet50 Baseline (He et al. 2016a) |  $24.56\%$(+$0.00\%$) | $7.50\%$ | $3.86$ | $25.56$
AW-ResNet50 |  $23.38\%$(+$1.18\%$) | $6.79\%$ | $3.87$ | $25.72$
SE-ResNet50 (Hu, Shen, and Sun 2018) |  $23.14\%$(+$1.42\%$) | $6.70\%$ | $3.87$ | $28.09$
AW-SE-ResNet50 |  $22.72\%$(+$1.84\%$) | $6.47\%$ | $3.88$ | $28.25$
AW-CBAM-ResNet50 (MaxPool) |  $22.82\%$(+$1.74\%$) | $6.41\%$ | $3.89$ | $28.25$
AW-CBAM-ResNet50 (Spatial) |  $23.20\%$(+$1.36\%$) | $6.58\%$ | $3.90$ | $28.25$
ResNet101 Baseline (He et al. 2016a) |  $23.38\%$(+$0.00\%$) | $6.88\%$ | $7.57$ | $44.55$
AW-ResNet101 |  $22.38\%$(+$1.00\%$) | $6.21\%$ | $7.58$ | $44.95$
SE-ResNet101 (Hu, Shen, and Sun 2018) |  $22.35\%$(+$1.03\%$) | $6.19\%$ | $7.58$ | $49.33$
AW-SE-ResNet101 |  $21.78\%$(+$1.60\%$) | $5.74\%$ | $7.59$ | $49.73$
AW-CBAM-ResNet101 (MaxPool) |  $21.64\%$(+$1.74\%$) | $5.76\%$ | $7.60$ | $49.73$
AW-CBAM-ResNet101 (Spatial) |  $22.32\%$(+$1.06\%$) | $6.18\%$ | $7.61$ | $49.73$
MobileNet Baseline (Howard et al. 2017) |  $31.39\%$(+$0.00\%$) | $11.51\%$ | $0.569$ | $4.23$
SE-MobileNet (Hu, Shen, and Sun 2018) |  $29.97\%$(+$1.42\%$) | $10.63\%$ | $0.581$ | $5.07$
AW-SE-MobileNet |  $29.41\%$(+$1.98\%$) | $10.59\%$ | $0.623$ | $5.52$
CBAM-MobileNet (Woo et al. 2018) |  $29.01\%$(+$2.38\%$) | $9.99\%$ | $0.611$ | $5.07$
AW-CBAM-MobileNet (Spatial) |  $28.82\%$(+$2.57\%$) | $9.98\%$ | $0.652$ | $5.52$
Table 1: Comparisons of attention-based models on ImageNet classfication.

## Experimental results

In this section, we use extensive experiments to demonstrate the effectiveness
of our proposed attention module. We use ResNet (He et al. 2016a), MobileNet
(Howard et al. 2017), SSD300 (Liu et al. 2016), and Faster R-CNN (Ren et al.
2015) as the baseline models, and various variants of these models are
developed, including using our proposed attention module for these baseline
models and integrating our proposed attention module with their attentional
activations-based models. The datasets to train these models include CIFAR-100
classification (Krizhevsky and Hinton 2009), ImageNet classification (Deng et
al. 2009), VOC object detection datasets (Everingham et al. 2015) (The
experimental results are included in Section E of Supplementary Material.),
and COCO object detection datasets (Lin et al. 2014). To have a better
interpretation of our proposed attention module, its feature visualizations
using Grad-CAM (Selvaraju et al. 2017) can be found in Section F of
Supplementary Material. All the data augmentation and training settings can be
found in Section A of Supplementary Material.

### ImageNet image classification

To investigate the performance of our proposed attention module on high-
resolution images, we train ResNet50 and ResNet101 (He et al. 2016a) and their
attention-based variants on the ImageNet classification dataset (Deng et al.
2009). According to the results shown in Table 1, our proposed attention
module is complementary to other attentional activations-based models. AW-
ResNet50 achieves a $1.18\%$ Top-1 error reduction compared with the ResNet50
baseline. Integrating with our proposed attention module, SE-ResNet50 (Hu,
Shen, and Sun 2018) can improve further by $0.42\%$ Top-1 accuracy. The Top-1
accuracy of our AW-SE-ResNet101 is $1.60\%$ and $0.57\%$ higher than that of
ResNet101 and SE-ResNet101, respectively. To integrate with CBAM-ResNet (Woo
et al. 2018) more carefully, we define CBAM-ResNet (MaxPool) and CBAM-ResNet
(Spatial) separately. In CBAM-ResNet (MaxPool), we do not deploy the spatial
attention maps, while we do not use max-pooled features in CBAM-ResNet
(Spatial). The Top-1 accuracy of AW-CBAM-ResNet50 (MaxPool) and AW-CBAM-
ResNet50 (Spatial) are better than AW-ResNet50 by $0.56\%$ and $0.18\%$,
respectively, but worse than AW-SE-ResNet50. The number of additional
parameters for our proposed attention module is $0.16$ M, which is much
smaller than $2.83$ M (i.e., one-sixteenth) of SE and CBAM modules. Moreover,
it takes only $0.01$ GFLOPs to apply our proposed attention module on the
ResNet50 model on ImageNet classification, which is comparable with $0.01$
GFLOPs and $0.04$ to adopt the SE and CBAM modules and is negligible in terms
of FLOPs to implement the baseline model. The computational complexity
analysis introduced by the attention mechanism can be found in Section B of
Supplementary Material.

Model | Top-1 Error | Top-5 Error | GFLOPs | Parameters (M)
---|---|---|---|---
AA-Wide-ResNet Baseline (Bello et al. 2019) |  $28.01\%$($+0.00\%$) | $7.92\%$ | $3.89$ | $8.43$
AW-AA-Wide-ResNet-0 |  $26.14\%$($+1.87\%$) | $7.43\%$ | $3.90$ | $8.50$
AW-AA-Wide-ResNet-1 |  $27.17\%$($+0.84\%$) | $7.49\%$ | $3.89$ | $8.48$
AW-AA-Wide-ResNet-2 |  $27.09\%$($+0.92\%$) | $8.00\%$ | $3.89$ | $8.45$
AW-AA-Wide-ResNet-3 |  $28.11\%$($-0.10\%$) | $8.24\%$ | $3.89$ | $8.44$
ResNet50 Baseline (He et al. 2016a) |  $22.33\%$($+0.00\%$) | $5.83\%$ | $1.22$ | $23.71$
AW-ResNet50 |  $19.87\%$($+2.46\%$) | $4.76\%$ | $1.23$ | $23.87$
SE-ResNet50 (Hu, Shen, and Sun 2018) |  $20.43\%$($+1.90\%$) | $5.01\%$ | $1.23$ | $26.24$
AW-SE-ResNet50 |  $19.00\%$($+3.33\%$) | $4.51\%$ | $1.24$ | $26.40$
CBAM-ResNet50 (Woo et al. 2018) |  $19.46\%$($+2.87\%$) | $4.56\%$ | $1.24$ | $26.24$
AW-CBAM-ResNet50 |  $18.94\%$($+3.39\%$) | $4.76\%$ | $1.25$ | $26.40$
Table 2: Comparisons of attention-based models on CIFAR-100 classfication. Backbone | Detector |<EMAIL_ADDRESS>0.95] |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---|---|---
ResNet101-FPN (Lin et al. 2017) | Faster R-CNN |  $37.13$($+0.00\%$) | $58.28$ | $40.29$
ResNet101-AW-FPN | Faster R-CNN |  $37.76$($+0.63\%$) | $59.17$ | $40.91$
ResNet101-SE-FPN (Hu, Shen, and Sun 2018) | Faster R-CNN |  $38.11$($+0.98\%$) | $59.41$ | $41.33$
ResNet101-AW-SE-FPN | Faster R-CNN |  $38.45$($+1.32\%$) | $59.70$ | $41.86$
ResNet101-CBAM-FPN (Woo et al. 2018) | Faster R-CNN |  $37.74$($+0.61\%$) | $58.84$ | $40.77$
ResNet101-AW-CBAM-FPN | Faster R-CNN |  $38.19$($+1.06\%$) | $59.52$ | $41.43$
Table 3: Comparisons of attention-based Faster R-CNN on COCO.

#### Resource-constrained architecture

Driven by demand for mobile applications, many depthwise convolution-based
models are developed to take care of the trade-off between accuracy and
efficiency. To inspect the generalization of our proposed attention module in
this resource-constrained scenario, we conduct the ImageNet classification
(Deng et al. 2009) with the MobileNet architecture (Howard et al. 2017). Since
our proposed attention module is efficient in terms of both the storage and
computational complexity, integrating it into the light-weight architecture is
worth exploring. We apply our proposed attention module to pointwise
convolution instead of depthwise convolution in every two depthwise separable
convolutions. When integrating with the CBAM models (Woo et al. 2018), we
remove the max-pooled features and keep spatial attention maps. As shown in
Table 1, AW-SE-MobileNet and AW-CBAM-MobileNet achieve $0.56\%$ and $0.19\%$
Top-1 accuracy improvements compared with SE-MobileNet (Hu, Shen, and Sun
2018) and CBAM-MobileNet, respectively. It is an impressive result that the
Top-1 accuracy of AW-CBAM-MobileNet is $2.57\%$ better than that of the
MobileNet baseline. For the MobileNet model, our proposed attention module
increases the computation by $0.041$ GFLOPs, while SE and CBAM modules need
$0.012$ and $0.041$ GFLOPs, respectively. Also, the required parameters for
our proposed attention module are $0.45$ M, which is much less than $0.84$ M
for SE and CBAM modules.

### CIFAR-100 image classification

According to the results shown in Table 2, we conclude that our proposed
attention module can boost the CIFAR-100 (Krizhevsky and Hinton 2009) accuracy
of both the ResNet50 baseline model (He et al. 2016a) and their attentional
activations-based models with negligible additional computational complexity.
AW-ResNet50 achieves a $2.46\%$ Top-1 error reduction compared with the
ResNet50 baseline. Integrating with our proposed attention module, SE-ResNet50
(Hu, Shen, and Sun 2018) and CBAM-ResNet50 (Woo et al. 2018) can increase
Top-1 accuracy by $1.43\%$ and $1.52\%$, respectively. In terms of the
computational complexity, our proposed attention module requires $0.01$
GFLOPs, which is acceptable compared with $0.01$ GFLOPs for the SE module,
$0.02$ GFLOPs for the CBAM module, and $1.22$ GFLOPs for the baseline model.
Besides, we only introduce $0.16$ M parameters for our proposed attention
module, which is less than $2.53$ M parameters for the SE and CBAM modules. We
train ResNet with various depths on CIRAR-100 image classification to show
that our proposed attention module works for CNNs with different depths, which
are in Section D of Supplementary Material.

### Object Detection on COCO

To show the generalization of our proposed attention module, we apply it to
object detection tasks. We evaluate our proposed attention module further on
the COCO dataset (Lin et al. 2014), which contains $118K$ images (i.e.,
train2017) for training and $5K$ images (i.e., val2017) for validating. We use
Faster R-CNN (Ren et al. 2015) as the detection method with the ResNet101-FPN
backbone (Lin et al. 2017). Here we intend to evaluate the benefits of
applying our proposed attention module on the ResNet101-FPN backbone (Lin et
al. 2017), where all the lateral and output convolutions of the FPN adopt our
AW-convolution. The SE and CBAM modules are placed right before the lateral
and output convolutions. As shown in Table 3, applying our proposed attention
module on ResNet101-FPN boosts<EMAIL_ADDRESS>0.95] by $0.63$ for the Faster R-CNN
baseline. Integrating with attentional activations-based models, Faster R-CNNs
with the backbones of ResNet101-AW-SE-FPN and ResNet101-AW-CBAM-FPN outperform
Faster R-CNNs with the backbones of ResNet101-SE-FPN and ResNet101-CBAM-FPN by
$0.34$ and $0.45$ on COCO’s standard metric AP.

## Conclusion

In this paper, We analyze the two ignored problems in attentional activations-
based models: the approximation problem and the insufficient capacity problem
of the attention maps. To address the two problems together, we propose an
attention module by developing the AW-convolution, where the shape of the
attention maps matches that of the weights rather than the activations, and
integrate it with attention-based models as a complementary method to enlarge
their attentional capability. We have implemented extensive experiments to
demonstrate the effectiveness of our proposed attention module, both on image
classification and object detection tasks. Our proposed attention module alone
shows noticeable accuracy improvement compared with baseline models. More
importantly, integrating our proposed module with previous attention-based
models, such as AA-Net (Bello et al. 2019), SE-Net (Hu, Shen, and Sun 2018),
and CBAM-Net (Woo et al. 2018), will further boost their performance.

## References

  * Bello et al. (2018) Bello, I.; Kulkarni, S.; Jain, S.; Boutilier, C.; Chi, E. H.; Eban, E.; Luo, X.; Mackey, A.; and Meshi, O. 2018.  Seq2Slate: Re-ranking and Slate Optimization with RNNs.  _CoRR_ abs/1810.02019.  URL http://arxiv.org/abs/1810.02019.
  * Bello et al. (2019) Bello, I.; Zoph, B.; Vaswani, A.; Shlens, J.; and Le, Q. V. 2019.  Attention Augmented Convolutional Networks.  In _The IEEE International Conference on Computer Vision (ICCV)_.
  * Chen et al. (2017) Chen, L.-C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; and Yuille, A. L. 2017.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  _IEEE transactions on pattern analysis and machine intelligence_ 40(4): 834–848.
  * Chen et al. (2019) Chen, Y.; Fan, H.; Xu, B.; Yan, Z.; Kalantidis, Y.; Rohrbach, M.; Yan, S.; and Feng, J. 2019.  Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks With Octave Convolution.  In _The IEEE International Conference on Computer Vision (ICCV)_.
  * Chen et al. (2018) Chen, Y.; Kalantidis, Y.; Li, J.; Yan, S.; and Feng, J. 2018.  A^ 2-Nets: Double Attention Networks.  In _Advances in Neural Information Processing Systems_ , 352–361.
  * Chollet (2017) Chollet, F. 2017.  Xception: Deep learning with depthwise separable convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1251–1258.
  * Corbetta and Shulman (2002) Corbetta, M.; and Shulman, G. L. 2002.  Control of goal-directed and stimulus-driven attention in the brain.  _Nature reviews neuroscience_ 3(3): 201.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255. Ieee.
  * Everingham et al. (2015) Everingham, M.; Eslami, S. A.; Van Gool, L.; Williams, C. K.; Winn, J.; and Zisserman, A. 2015.  The pascal visual object classes challenge: A retrospective.  _International journal of computer vision_ 111(1): 98–136.
  * Gregor et al. (2015) Gregor, K.; Danihelka, I.; Graves, A.; Rezende, D.; and Wierstra, D. 2015.  DRAW: A Recurrent Neural Network For Image Generation.  In _International Conference on Machine Learning_ , 1462–1471.
  * Gu et al. (2018) Gu, J.; Wang, Z.; Kuen, J.; Ma, L.; Shahroudy, A.; Shuai, B.; Liu, T.; Wang, X.; Wang, G.; Cai, J.; et al. 2018.  Recent advances in convolutional neural networks.  _Pattern Recognition_ 77: 354–377.
  * Han, Kim, and Kim (2017) Han, D.; Kim, J.; and Kim, J. 2017.  Deep pyramidal residual networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 5927–5935.
  * He et al. (2016a) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016a.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * He et al. (2016b) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016b.  Identity mappings in deep residual networks.  In _European conference on computer vision_ , 630–645. Springer.
  * Howard et al. (2019) Howard, A.; Sandler, M.; Chu, G.; Chen, L.-C.; Chen, B.; Tan, M.; Wang, W.; Zhu, Y.; Pang, R.; Vasudevan, V.; Le, Q. V.; and Adam, H. 2019.  Searching for MobileNetV3.  In _The IEEE International Conference on Computer Vision (ICCV)_.
  * Howard et al. (2017) Howard, A. G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; and Adam, H. 2017.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications.  _CoRR_ abs/1704.04861.  URL http://arxiv.org/abs/1704.04861.
  * Hu et al. (2018) Hu, J.; Shen, L.; Albanie, S.; Sun, G.; and Vedaldi, A. 2018.  Gather-excite: Exploiting feature context in convolutional neural networks.  In _Advances in Neural Information Processing Systems_ , 9401–9411.
  * Hu, Shen, and Sun (2018) Hu, J.; Shen, L.; and Sun, G. 2018.  Squeeze-and-excitation networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 7132–7141.
  * Huang et al. (2017) Huang, G.; Liu, Z.; Van Der Maaten, L.; and Weinberger, K. Q. 2017.  Densely connected convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 4700–4708.
  * Jaderberg et al. (2015) Jaderberg, M.; Simonyan, K.; Zisserman, A.; et al. 2015.  Spatial transformer networks.  In _Advances in neural information processing systems_ , 2017–2025.
  * Jia et al. (2016) Jia, X.; De Brabandere, B.; Tuytelaars, T.; and Gool, L. V. 2016.  Dynamic filter networks.  In _Advances in Neural Information Processing Systems_ , 667–675.
  * Ke, Maire, and Yu (2017) Ke, T.-W.; Maire, M.; and Yu, S. X. 2017.  Multigrid neural architectures.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 6665–6673.
  * Klein, Wolf, and Afek (2015) Klein, B.; Wolf, L.; and Afek, Y. 2015.  A dynamic convolutional layer for short range weather prediction.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 4840–4848.
  * Kornblith, Shlens, and Le (2019) Kornblith, S.; Shlens, J.; and Le, Q. V. 2019.  Do better imagenet models transfer better?  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2661–2671.
  * Krizhevsky and Hinton (2009) Krizhevsky, A.; and Hinton, G. 2009.  Learning multiple layers of features from tiny images.  _Master’s thesis, Department of Computer Science, University of Toronto_ .
  * Li et al. (2019a) Li, H.; Liu, Y.; Ouyang, W.; and Wang, X. 2019a.  Zoom out-and-in network with map attention decision for region proposal and object detection.  _International Journal of Computer Vision_ 127(3): 225–238.
  * Li et al. (2019b) Li, X.; Wang, W.; Hu, X.; and Yang, J. 2019b.  Selective kernel networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 510–519.
  * Lin et al. (2017) Lin, T.-Y.; Dollár, P.; Girshick, R.; He, K.; Hariharan, B.; and Belongie, S. 2017.  Feature pyramid networks for object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2117–2125.
  * Lin et al. (2014) Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , 740–755. Springer.
  * Linsley et al. (2019) Linsley, D.; Shiebler, D.; Eberhardt, S.; and Serre, T. 2019.  Learning what and where to attend with humans in the loop.  In _International Conference on Learning Representations_.  URL https://openreview.net/forum?id=BJgLg3R9KQ.
  * Liu et al. (2016) Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.-Y.; and Berg, A. C. 2016.  Ssd: Single shot multibox detector.  In _European conference on computer vision_ , 21–37. Springer.
  * Liu et al. (2019) Liu, X.; Yin, G.; Shao, J.; Wang, X.; et al. 2019.  Learning to predict layout-to-image conditional convolutions for semantic image synthesis.  In _Advances in Neural Information Processing Systems_ , 568–578.
  * Ma et al. (2018) Ma, N.; Zhang, X.; Zheng, H.-T.; and Sun, J. 2018.  Shufflenet v2: Practical guidelines for efficient cnn architecture design.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 116–131.
  * Park et al. (2018) Park, J.; Woo, S.; Lee, J.-Y.; and Kweon, I.-S. 2018.  BAM: Bottleneck Attention Module.  In _British Machine Vision Conference (BMVC)_. British Machine Vision Association (BMVA).
  * Ren et al. (2015) Ren, S.; He, K.; Girshick, R.; and Sun, J. 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In _Advances in neural information processing systems_ , 91–99.
  * Rensink (2000) Rensink, R. A. 2000.  The dynamic representation of scenes.  _Visual cognition_ 7(1-3): 17–42.
  * Selvaraju et al. (2017) Selvaraju, R. R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; and Batra, D. 2017.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , 618–626.
  * Simonyan and Zisserman (2015) Simonyan, K.; and Zisserman, A. 2015.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  In Bengio, Y.; and LeCun, Y., eds., _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.  URL http://arxiv.org/abs/1409.1556.
  * Su et al. (2019) Su, H.; Jampani, V.; Sun, D.; Gallo, O.; Learned-Miller, E.; and Kautz, J. 2019\.  Pixel-adaptive convolutional neural networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 11166–11175.
  * Szegedy et al. (2017) Szegedy, C.; Ioffe, S.; Vanhoucke, V.; and Alemi, A. A. 2017.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In _Thirty-First AAAI Conference on Artificial Intelligence_.
  * Szegedy et al. (2015) Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; and Rabinovich, A. 2015.  Going deeper with convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1–9.
  * Tan et al. (2019) Tan, M.; Chen, B.; Pang, R.; Vasudevan, V.; Sandler, M.; Howard, A.; and Le, Q. V. 2019.  Mnasnet: Platform-aware neural architecture search for mobile.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2820–2828.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , 5998–6008.
  * Vinyals, Fortunato, and Jaitly (2015) Vinyals, O.; Fortunato, M.; and Jaitly, N. 2015.  Pointer Networks.  In Cortes, C.; Lawrence, N. D.; Lee, D. D.; Sugiyama, M.; and Garnett, R., eds., _Advances in Neural Information Processing Systems 28_ , 2692–2700. Curran Associates, Inc.  URL http://papers.nips.cc/paper/5866-pointer-networks.pdf.
  * Wang et al. (2017) Wang, F.; Jiang, M.; Qian, C.; Yang, S.; Li, C.; Zhang, H.; Wang, X.; and Tang, X. 2017.  Residual attention network for image classification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 3156–3164.
  * Wang et al. (2018) Wang, X.; Girshick, R.; Gupta, A.; and He, K. 2018.  Non-local neural networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 7794–7803.
  * Woo et al. (2018) Woo, S.; Park, J.; Lee, J.-Y.; and So Kweon, I. 2018.  Cbam: Convolutional block attention module.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 3–19.
  * Wu et al. (2019) Wu, F.; Fan, A.; Baevski, A.; Dauphin, Y.; and Auli, M. 2019.  Pay Less Attention with Lightweight and Dynamic Convolutions.  In _International Conference on Learning Representations_.  URL https://openreview.net/forum?id=SkVhlh09tX.
  * Xie et al. (2017) Xie, S.; Girshick, R.; Dollár, P.; Tu, Z.; and He, K. 2017.  Aggregated residual transformations for deep neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1492–1500.
  * Xu et al. (2015) Xu, K.; Ba, J.; Kiros, R.; Cho, K.; Courville, A.; Salakhudinov, R.; Zemel, R.; and Bengio, Y. 2015.  Show, attend and tell: Neural image caption generation with visual attention.  In _International conference on machine learning_ , 2048–2057.
  * Yang et al. (2019) Yang, B.; Wang, L.; Wong, D. F.; Chao, L. S.; and Tu, Z. 2019.  Convolutional Self-Attention Networks.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 4040–4045.
  * Zagoruyko and Komodakis (2016) Zagoruyko, S.; and Komodakis, N. 2016.  Wide Residual Networks.  In Richard C. Wilson, E. R. H.; and Smith, W. A. P., eds., _Proceedings of the British Machine Vision Conference (BMVC)_ , 87.1–87.12. BMVA Press.  ISBN 1-901725-59-6.  doi:10.5244/C.30.87.  URL https://dx.doi.org/10.5244/C.30.87.
  * Zhao et al. (2018) Zhao, F.; Zhao, J.; Yan, S.; and Feng, J. 2018.  Dynamic conditional networks for few-shot learning.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 19–35.