# Automatic Displacement and Vibration Measurement in Laboratory Experiments
with A Deep Learning Method

Yongsheng Bai, Ramzi M. Abduallah, Halil Sezen, Alper Yilmaz  Department of
Civil, Environmental and Geodetic Engineering
The Ohio State University
Columbus, OH, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This paper proposes a pipeline to automatically track and measure displacement
and vibration of structural specimens during laboratory experiments. The
latest Mask Regional Convolutional Neural Network (Mask R-CNN) can locate the
targets and monitor their movement from videos recorded by a stationary
camera. To improve precision and remove the noise, techniques such as Scale-
invariant Feature Transform (SIFT) and various filters for signal processing
are included. Experiments on three small-scale reinforced concrete beams and a
shaking table test are utilized to verify the proposed method. Results show
that the proposed deep learning method can achieve the goal to automatically
and precisely measure the motion of tested structural members during
laboratory experiments.

###### Index Terms:

Mask R-CNN, displacement measurement, vibration measurement, structural
experiments, shaking table.

##  I Introduction

For safety assessment and structural health monitoring of infrastructures such
as buildings and bridges and their components, their vibrations and
deformations need to be recorded and evaluated. Traditional structural sensors
such as Linear Variable Displacement Transducers (LVDTs), dial gauges,
accelerometers and other advanced sensors are used to measure deformations and
vibrations. However, in some cases, conventional sensors may not be a good
option to access the desirable instrumentation locations and work in a timely
and cost-efficient way. Most importantly, traditional sensors measure the
displacement or vibration at a discrete location, i.e., one sensor is used to
measure one quantity at a single point. On the other hand, a high definition
video camera can record the movement of a component or an entire structure
rather than a single point.

Recent developments on vision- and vibration-based technologies led to the
measurement applications of low-cost and non-contact sensors on deformation
and vibration monitoring of the infrastructures [1], especially because
unusual and extreme deformations or vibrations in aging bridges and buildings
may be an indication of significant serviceability or safety issues. Cameras
can collect high-quality images or high-speed videos in lab or field tests as
non-contact and non-destructive sensors. With computer vision technologies and
various deep learning techniques, the vision sensors can not only be used as
eyes for Artificial Intelligence (AI) vehicles and machines, but also provide
opportunities for scientific measurements to researchers and engineers. In a
laboratory experiment, cameras can be fixed near the testing station and
record live motion of the monitored structural members. They can be placed at
a stationary location or some distance away from an in-service bridge or
building to record its structural movements remotely in the field. The
collected visual data can be processed further to identify potential damage
and assess the motion of the observed structures precisely.

To better understand and efficiently use displacement and vibration
measurement data from cameras, we used Mask R-CNN [2] with High-resolution
network (HRNet) [3] to track a target attached on beam specimens in the
laboratory and to measure their deflections in the first study. Then the Mask
R-CNN was applied on a shaking table test to track the dynamic motion of four
targets simultaneously in the second study.

##  II Literature Review

Computer Vision (CV) and deep learning techniques are very useful to gain
high-level understanding and extraction of desired information and precise
motion measurements from images and videos.

Traditional CV techniques have been widely used by researchers for
displacement or vibration measurements by cameras. These techniques include
image processing technique [4], up-sampled cross correlation [5], adaptive
Region of Interest algorithm [6], modified Taylor approximation [7], and
contour extraction with Speeded-Up Robust Features (SURF) [8]. In addition,
Lucas-Kanade template tracking algorithm [9, 10] and Digital Image Correlation
(DIC) [11] are employed to track displacement and vibration of structural
members. Furthermore, Hu and Pai [12] utilized a camera-based 3D motion
analysis system to measure the resonant vibration of steel cables. Chen et al.
[13] described an application with a video camera-based technique to test the
vibration of an antenna tower on a tall building when a camera was placed 175
meters far from it. Hoskere et al. [14] used an Unmanned Aerial Vehicle (UAV)
to measure the modal properties and dynamic response of a full-scale
structure.

Deep learning is a relatively new research area for visual measurement
applications. Dong et al. [15] implemented a full field optical flow algorithm
named FlowNet2 to measure the displacement and vibration of structures. They
also used the Spatio-Temporal Context Learning to track targets and utilized a
Taylor approximation to gain subpixel level precision for displacement
measurement [16]. Also, Dong et al. [17] applied Visual Graph Visual Geometry
Group (VGG) to extract features of the target for monitoring and measuring
during the traffic time. These methods indicate how we can efficiently use
cameras to monitor and measure the displacement and vibration of structural
members or a structure in the laboratory or in the field.

##  III Methodology

In our previous studies[18, 19], new variants of the latest Mask R-CNNs were
successfully applied for structural damage detection with high accuracy.
Therefore, we used one of the variants, Mask R-CNN with HRNet, to track and
measure the displacement and vibration in this research. Fig. 1 shows the
framework of this Mask R-CNN.

Figure 1: The framework of Mask R-CNN for tacking and measuring displacement
of a reinforced concrete beam Figure 2: Translation of a target measured by a
bounding box or by matching keypoints. Figure 3: A flowchart of Mask R-CNN and
SIFT for automated displacement and vibration measurement in a lab experiment
with a stationary camera.

As shown in Fig. 1 and 4, a wood frame is attached near the midspan of the
tested beam so that it moves downward or upward when the beam is loaded or
unloaded. The motion of the frame represents the deflection of the point where
it is attached. Mask R-CNN is used to track the top of this wood frame (i.e.,
yellow dashed line on the left image in Fig. 1), which is marked by a bounding
box and a mask in purple on the right image. Since the tracking target is a
rigid body, its motion can be represented by any point on it or by the
bounding box. On the other hand, as shown in Fig. 2 for the image plane of a
stationary camera, translation of a target between the first frame and the
$j$th frame, $du\textsuperscript{j}$ and $dv\textsuperscript{j}$, can be
calculated as the position change of the bounding box or the average motion of
the matching keypoints with SIFT. We observed that the mask for a target may
be not exactly the same as its real shape in some cases. Therefore, SIFT is
introduced into the pipeline to eliminate this inadvertent drawback when the
bounding box is inaccurate to represent the target. Furthermore, subpixel
precision can be achieved by matching and using the average motion of these
keypoints on the target. In our pipeline (see Fig. 3), the ratio of good
matching is restricted as SIFT commonly used in practice, but the range of
coordinate change for each matching keypoint is also constrained such that the
top good matching keypoints can be secured. Therefore, the mismatching is
reduced dramatically and the accuracy of measurement is improved. Finally, the
measurement is converted from pixel to length unit (inches or millimeters),
which is also called a scalar, $s$. The horizontal and vertical displacement
$dx\textsuperscript{j}$ and $dy\textsuperscript{j}$ of the target can be
obtained as follows:

${dx\textsuperscript{j}=s\times du\textsuperscript{j}}\vspace{-2mm}$ (1)
${dy\textsuperscript{j}=s\times dv\textsuperscript{j}}\vspace{-2mm}$ (2)

As a comparison, an optical flow method called Lucas-Kanade (LK) tracker [20]
was used to track and measure the same targets in our experiments. For the LK
tracker, the relative displacement is measured between two adjacent frames in
a video. The final displacement is the sum-up of all the relative
measurements. In addition, Savitzky-Golay filter [21] and Butterworth filter
[22] were employed to handle the inconsistency and noise of the measurements.
Fast Fourier Transform (FFT) [23] was applied to extract the frequencies of
the vibrating targets.

##  IV Implementation

Two types of indoor experiments were utilized to verify our proposed methods.
The first one is a small-scale experiment involving three reinforced concrete
(RC) beams (see Fig. 4) conducted on the main campus of The Ohio State
University in Columbus, Ohio. These beams are loaded and deflected until
failure. Another test is an application on a video of a shaking table test
(see Fig. 5) [24].

Figure 4: A flexural test of a RC beam with a LVDT and a camera. Figure 5:
Original image (left) and label (right) in the shaking table video [24].

###  IV-A Deflection Measurement of RC Beams in Laboratory Tests

In this experiment, three RC beams were subjected to a monotonically
increasing point load at the midspan. A displacement sensor (a LVDT or dial
gauge) and a wood frame were used to measure the deflection near midspan of
the beams, while a camera was placed 3-feet away from the midspan (see Fig.
4). Its definition was $1600\times 1200$ and the frame rate was 15 per second.
The wood frame was clamped on the beam to represent the deflection of the
targeted point on the tested beam.

The data process for the videos of three tests is like this: since this is a
static test, which means the loading and deflection of the tested beams is
slow, images from the video at each second are selected as the visual data.
There are a total of 500 to 600 images for each test. To train the Mask R-CNN,
only 50 images are randomly selected and labeled for detecting and tracking
the top of wood frame (in purple mask) as shown in Fig. 1. The LK tracker is
used to track the same object and measure the deflection. The testing results
are shown in Fig. 8. Compared to the LK tracker, this Mask R-CNN with SIFT can
provide a measurement closer to the ground truth of beam deflections by a dial
gauge. In addition, Savitzky-Golay filter is applied to smooth the measurement
such that it becomes consistent (see Fig. 9). Table I shows MAE (mean absolute
error) for both methods. It can be inferred that SIFT and Savitzky-Golay
filter can effectively readjust the position of the bounding box predicted
from the Mask R-CNN and smooth the measurements, hence, the proposed method
can outperform the LK tracker.

TABLE I: MAE for two methods (inches) Methods | Test 1 | Test 2 | Test 3
---|---|---|---
Mask R-CNN + SIFT | 0.005 | 0.005 | 0.005
LK tracker | 0.030 | 0.012 | 0.012

###  IV-B Vibration Measurement of A Shaking Table Test

Our proposed method was also applied on a shaking table test [24] to check its
applicability of monitoring dynamic movement of objects. In this test, there
are three rectangles (masses) fixed on the shaking table at different heights
(see Fig. 5). Each rectangle, which is supported by two sticks, like a
structure has its unique resonant frequency in the horizontal direction. This
is due to differences between the lateral stiffness of each pair of sticks.
The frequencies of the applied shaking are increased from 4 Hz to 13.65 Hz to
excite these masses and cause their harmonic vibrations. From the recorded
video [24], 150 frames are randomly selected from a total of 6,674 frames and
labeled for training the Mask R-CNN. The video has an image size of
640$\times$480 and a frame rate of 30 per second. SIFT is not applied to
smooth the measurements here, since the goal of this test is to detect the
frequencies instead of accurate amplitudes of the vibration, which is in pixel
unit in this test. Thus, the motion of the bounding box represents the
translation of each object. The LK tracker is utilized to verify our method by
tracking the same vibration of the shaking table. On one hand, all the raw
data are processed by Butterworth filter, and FFT is applied to extract the
frequencies for each tracking target. The filtered vibrations of the shaking
table with the LK tracker and Mask R-CNN are shown in the left figures of Fig.
6. There are three frequencies of vibration at approximately 4 Hz, 6.35 Hz and
11.35 Hz excited by the table. Both methods capture these frequencies (yellow
captions in the right figures of Fig. 6) with a less than $2.6\%$ error. On
the other hand, the vibrations of three rectangles are measured by the Mask
R-CNN and raw data are processed like the procedures for the shaking table. As
shown in Fig. 7, their resonant frequencies are very close to the intended
frequencies (i.e., 4 Hz, 6.35 Hz and 11.35 Hz). The error rate for this
measurement is $0\%$, $0.3\%$ and $2.6\%$, respectively. This indicates that
the proposed Mask R-CNN can be used alone to track multiple objects and
capture their vibrations characteristics precisely.

Figure 6: Vibration and frequencies of the shaking table measured by Mask
R-CNN and the LK tracker. Figure 7: Vibrations of three rectangles (the
highest to the lowest one from top to bottom on the left hand side) measured
by Mask R-CNN and and the corresponding frequencies calculated by FFT (on the
right hand side). Figure 8: The deflection-time relationship measured by a
camera and a dial gauge for the three testing beams. Figure 9: The filtered
deflection-time relationship measured by a camera and a dial gauge for the
three testing beams.

##  V Conclusions

A deep learning method (i.e., Mask R-CNN with HRNet) and techniques such as
SIFT and Savitzky-Golay filter are applied to automatically track the targets
and provide the accurate measurement of their motions with a stationary
camera. In our first experiment, Mask R-CNN and SIFT were used for precise
deflection measurement of the tested RC beams, since SIFT can utilize the
keypoints on the targets to refine the measurement. Our method can be closer
to the measurement from traditional structural sensors and outperform the LK
tracker. The Mask R-CNN was also used alone to track the vibration of multiple
targets in a shaking table experiment and capture the resonant frequencies of
these targets via Butterworth filter and FFT. These preliminary tests show
that the proposed method is robust and has the potential for measuring
displacements and vibrations of structural specimens precisely and
automatically in laboratory experiments.

Our ongoing work involves application of the proposed method on actual
buildings tested in the field to confirm its applicability for outdoor
environments. Other deep learning methods are also being explored and tested.

## References

  * [1] C.-Z. Dong and F. N. Catbas, “A review of computer vision–based structural health monitoring at local and global levels,” Structural Health Monitoring, p. 1475921720935585, 2020.
  * [2] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” Proceedings of the IEEE international conference on computer vision, pp. 2961–2969, 2017.
  * [3] K. Sun, Y. Zhao, B. Jiang, T. Cheng, B. Xiao, D. Liu, Y. Mu, X. Wang, W. Liu, and J. Wang, “High-resolution representations for labeling pixels and regions,” arXiv preprint arXiv:1904.04514, 2019.
  * [4] J.-J. Lee and M. Shinozuka, “Real-time displacement measurement of a flexible bridge using digital image processing techniques,” Experimental mechanics, vol. 46, no. 1, pp. 105–114, 2006.
  * [5] D. Feng, M. Q. Feng, E. Ozer, and Y. Fukuda, “A vision-based sensor for noncontact structural displacement measurement,” Sensors, vol. 15, no. 7, pp. 16557–16575, 2015.
  * [6] J. Lee, K.-C. Lee, S. Cho, and S.-H. Sim, “Computer vision-based structural displacement measurement robust to light-induced image degradation for in-service bridges,” Sensors, vol. 17, no. 10, p. 2317, 2017.
  * [7] B. Liu, D. Zhang, and J. Guo, “Vision-based displacement measurement sensor using modified taylor approximation approach,” Optical Engineering, vol. 55, no. 11, p. 114103, 2016.
  * [8] Z. Yin, C. Wu, and G. Chen, “Concrete crack detection through full-field displacement and curvature measurements by visual mark tracking: A proof-of-concept study,” Structural Health Monitoring, vol. 13, no. 2, pp. 205–218, 2014.
  * [9] J. Guo and C. Zhu, “Dynamic displacement measurement of large-scale structures based on the lucas–kanade template tracking algorithm,” Mechanical Systems and Signal Processing, vol. 66, pp. 425–436, 2016.
  * [10] C.-Z. Dong, O. Celik, and F. N. Catbas, “Marker-free monitoring of the grandstand structures and modal identification using computer vision methods,” Structural Health Monitoring, vol. 18, no. 5-6, pp. 1491–1509, 2019.
  * [11] G. Chen, Q. Liang, W. Zhong, X. Gao, and F. Cui, “Homography-based measurement of bridge vibration using uav and dic method,” Measurement, vol. 170, p. 108683, 2021.
  * [12] J. Hu and P. F. Pai, “Experimental study of resonant vibrations of suspended steel cables using a 3d motion analysis system,” Journal of Engineering Mechanics, vol. 138, no. 6, pp. 640–661, 2012.
  * [13] J. G. Chen, A. Davis, N. Wadhwa, F. Durand, W. T. Freeman, and O. Büyüköztürk, “Video camera–based vibration measurement for civil infrastructure applications,” Journal of Infrastructure Systems, vol. 23, no. 3, p. B4016013, 2017.
  * [14] V. Hoskere, J.-W. Park, H. Yoon, and B. F. Spencer Jr, “Vision-based modal survey of civil infrastructure using unmanned aerial vehicles,” Journal of Structural Engineering, vol. 145, no. 7, p. 04019062, 2019.
  * [15] C.-Z. Dong, O. Celik, F. N. Catbas, E. J. O’Brien, and S. Taylor, “Structural displacement monitoring using deep learning-based full field optical flow methods,” Structure and Infrastructure Engineering, vol. 16, no. 1, pp. 51–71, 2020.
  * [16] C.-Z. Dong, O. Celik, F. N. Catbas, E. OBrien, and S. Taylor, “A robust vision-based method for displacement measurement under adverse environmental factors using spatio-temporal context learning and taylor approximation,” Sensors, vol. 19, no. 14, p. 3197, 2019.
  * [17] C.-Z. Dong and F. N. Catbas, “A non-target structural displacement measurement method using advanced feature matching strategy,” Advances in Structural Engineering, vol. 22, no. 16, pp. 3461–3472, 2019.
  * [18] Y. Bai, H. Sezen, and A. Yilmaz, “Detecting cracks and spalling automatically in extreme events by end-to-end deep learning frameworks,” ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. x, pp. xxx–xxx, 2021.
  * [19] Y. Bai, H. Sezen, and A. Yilmaz, “End-to-end deep learning methods for automated damage detection in extreme events at various scales,” 25th International Conference on Pattern Recognition (ICPR), pp. 6640–6647, 2021\.
  * [20] J.-Y. Bouguet, “Pyramidal implementation of the affine lucas kanade feature tracker description of the algorithm,” Intel corporation, vol. 5, no. 1-10, p. 4, 2001.
  * [21] G. Bianchi and R. Sorrentino, Electronic filter simulation & design.  McGraw Hill Professional, 2007.
  * [22] W. H. Press and S. A. Teukolsky, “Savitzky-golay smoothing filters,” Computers in Physics, vol. 4, no. 6, pp. 669–672, 1990.
  * [23] J. W. Cooley and J. W. Tukey, “An algorithm for the machine calculation of complex fourier series,” Mathematics of computation, vol. 19, no. 90, pp. 297–301, 1965.
  * [24] “Missouri university of science and technology: Sdof system resonance vibration test,” Apr 2017.


# B$\boldsymbol{A}^{2}$M: A Batch Aware Attention Module for Image
Classification

Qishang Cheng, Hongliang Li, , Qingbo Wu, , and King Ngi Ngan, Q. Cheng, H.
Li, Q. Wu and King. N. Ngan are with the School of Electronic Engineering,
University of Electronic Science and Technology of China, Chengdu 611731,
China (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>knngan@uestc.edu.cn).

###### Abstract

The attention mechanisms have been employed in Convolutional Neural Network
(CNN) to enhance the feature representation. However, existing attention
mechanisms only concentrate on refining the features inside each sample and
neglect the discrimination between different samples. In this paper, we
propose a batch aware attention module (B$\boldsymbol{A}^{2}$M) for feature
enrichment from a distinctive perspective. More specifically, we first get the
sample-wise attention representation (SAR) by fusing the channel, local
spatial and global spatial attention maps within each sample. Then, we feed
the SARs of the whole batch to a normalization function to get the weights for
each sample. The weights serve to distinguish the features’ importance between
samples in a training batch with different complexity of content. The
B$\boldsymbol{A}^{2}$M could be embedded into different parts of CNN and
optimized with the network in an end-to-end manner. The design of
B$\boldsymbol{A}^{2}$M is lightweight with few extra parameters and
calculations. We validate B$\boldsymbol{A}^{2}$M through extensive experiments
on CIFAR-100 and ImageNet-1K for the image recognition task. The results show
that B$\boldsymbol{A}^{2}$M can boost the performance of various network
architectures and outperforms many classical attention methods. Besides,
B$\boldsymbol{A}^{2}$M exceeds traditional methods of re-weighting samples
based on the loss value.

###### Index Terms:

Batch attention, Image Recognition

##  I Introduction

Due to the powerful ability of feature enhancement, the attention mechanism
has been widely used in designing various network architectures for the image
classification task. More specifically, the most popular attention modules
could be divided into three types: Channel Attention [1, 2, 3], Local Spatial
Attention [4, 5, 6, 7] and Global Spatial Attention [8, 9]. Despite their
great successes, these efforts mainly focus on exploring sample-wise
attention, whose enhancements are limited within each sample, including the
spatial and channel dimensions. However, The optimization of networks is based
on batch of images, and all the images contribute equally during the
optimization process. When the features of two samples from separated
categories are approximately similar, the existing attention mechanism cannot
effectively separate them in the feature space. As shown in Fig. 1, we show
t-SNE [10] visualization of features refined by different attention
mechanisms. The images are selected from 14 similar categories in the
validation set of ImageNet. When the features of two samples from separated
categories are approximately similar, the existing attention mechanism (SE
[1], BAM [6], CBAM [7], GC [8]) cannot effectively separate them in the
feature space and suffers from the overlapping of feature clusters.. For
example, the red rectangles in the Fig. 1 cover the feature clusters of
”Greater Swiss Mountain Dog” and ”Bernese Mountain Dog”.

Figure 1: t-SNE [10] visualization of generated features of 700 images refined
by different attention mechanisms. The feature clusters refined by
ResNet-50+B$\boldsymbol{A}^{2}$M are more discriminated than ResNet-50 [11],
SE [1], BAM [6], CBAM [7], and GE [3]. Figure 2: Illustration of the proposed
batch aware attention module (B$\boldsymbol{A}^{2}$M). The top-row illustrates
B$\boldsymbol{A}^{2}$Net. The bottom-left is a sub-module of
B$\boldsymbol{A}^{2}$M which is used to calculate sample-wise attention
representation of each image (SAR). The bottom-right shows the detailed
components of three attention modules ($\mathbf{A_{C}}$, $\mathbf{A_{LS}}$,
$\mathbf{A_{GS}}$).

To address the above issues, in this paper, we propose a batch aware attention
module (B$\boldsymbol{A}^{2}$M) to adaptively re-weight the CNN features of
different samples within a batch to improve their discrimination in the image
classification task. As shown in Fig. 1, it is seen that the feature clusters
of ResNet-50+B$\boldsymbol{A}^{2}$M are more separated than ResNet-50 [11], SE
[1], BAM [6], CBAM [7], and GE [3], and the feature clusters are more compact.
It demonstrates that B$\boldsymbol{A}^{2}$M could help CNN to obtain more
discriminative features. Hence, we first construct a sample-wise attention
representation (SAR) for each sample based on three kinds of attention
mechanisms including the channel attention (CA), local spatial attention
(LSA), and global spatial attention (GSA). The CA captures inter-channel
information of the given feature map, LSA generates local information on a
limited neighbourhood in the spatial dimension of the given feature map, and
GSA explores long-range interactions of pixels of the feature map. Then we
feed the SARs of the whole batch to softmax layer to obtain normalized weights
(Normalized-SAR) which is going to assigned to each sample. The weights serve
to increase the features’ difference between samples with different
categories.

We embed our B$\boldsymbol{A}^{2}$M into various network architectures and
test their performances in the image classification task, including ResNet
(ResNet-18/34/50/101 [11, 12], Wide-ResNet-28 [13], ResNeXt-29 [14]),
DenseNet-100 [15], ShuffleNetv2 [16], MobileNetv2 [17], and EfficientsNetB0
[18]. Extensive experiments on CIFAR-100 and ImageNet-1K show that
B$\boldsymbol{A}^{2}$M yields significant improvements with negligible
complexity increase. It achieves +2.93% Top-1 accuracy improvement on
ImageNet-1K with ResNet-50. In particular, B$\boldsymbol{A}^{2}$M
significantly outperforms many popular sample-wise attention works such as SE
[1], SRM [2], SK [4], SGE [5], GE [3], BAM [6], CBAM [7], GC [8], AA [9] and
SASA [19] in our experiments.

##  II Related work

Deep feature learning. Convolutional Neural Networks are potent tools to
extract features for computer vision. With the rise of Convolutional Neural
Networks (CNN), many sub-fields of computer vision rely on the design of new
network structures to be greatly promoted. From VGGNets [20] to Inception
models [21], ResNet [11], the powerful feature extraction capabilities of CNN
are demonstrated. On the one hand, many works constantly increased the depth
[12], width [13], and cardinality [14] of the network, and reformulate the
information flow between network [15]. On the other hand, many works are also
concerned with designing new modular components and activation functions, such
as depth-wise convolution [17], octave convolution [22], ELU [23, 24], and
PReLU [25]. These designs can further improve learning and representational
properties of CNN.

Attention for Feature Enhancement. In view of the fact that the importance of
all extracted features are different, multiple attention mechanisms are
developed to enhance the deep feature learning in image recognition task [26,
1, 3, 2, 4, 5, 6, 7, 8, 9], which focus on highlighting the semantic-aware
features within each individual sample. In 2017, Wang  et al. [26] proposed an
attention residual module to boost the ability of residual unit to generate
attention-aware features. Furthermore, Li et al. [4] designed a dynamic
selection mechanism that allows each neuron pay attention to different size of
kernels adaptively. GE [3] redistributed the gathered information from long-
range feature interactions to local spatial extent. Hu et al. [1] proposed SE-
module to focus on the channel-wise attention by re-calibrating feature maps
on channels using signals aggregated from entire feature maps. Lee et al. [2]
extract statistic information from each channels and re-weight per-channel
vial channel-independent information. [6] and CBAM [7] refine features beyond
the channel with introduced spatial dimensions. Beyond the above attention-
works, He et al. [8] explored the advantages of combination of SE [1] and Non-
Local [27] which further obtains global context information. The above
attention mechanisms focus on the intra-features enhancement and ignore the
inter-features enhancement.

##  III Approach

Figure 3: Illustration of some feature maps in the second block of ResNet-50.

###  III-A Motivation

Network optimization is carried out with batch mode. In the batch domain with
$N$ images, we define the $i_{th}$ input feature $x_{i}$ with the label
$y_{i}$ ($i\in[1,N]$). The original softmax loss for the batch could be
described as:

$\displaystyle L=\frac{1}{N}\sum^{N}_{i}w_{i}\cdot L_{i}$ (1) $\displaystyle
L_{i}=-\log\left(\frac{e^{f_{y_{i}}}}{\sum^{K}_{j}e^{f_{j}}}\right)$ (2)

where $N$ is the number of images in a batch. $w_{i}$ for the original is $1$.
$L_{i}$ is the loss for each sample $x_{i}$. where $f_{j}$ denotes the
$j_{th}$ elements of the vector of class score $\boldsymbol{f}$. $K$ is the
number of classes. $N$ is the number of images in the batch. Equ. 1 means that
all the images contribute equally during the optimization process. However,
due to the different complexity of the image content, images should have
diverse importance while calculating the loss. Focal Loss (FL) [28] and online
hard example mining (OHEM) [29] show that adaptive adjustment $w_{i}$ can
effectively improve the optimization process. OHEM ranks the loss values of
each sample in descending order and adds weight to the hard sample with high
loss. FL changes the Cross-Entropy loss function and pays more attention with
hard ones, so that the samples with large loss values are assigned larger
weights, and vice versa. However, these works do not take the image content
into account when determining the importance, which acts on the loss directly.

Combine with Equ. 2, the Equ. 1 could be modified as follows:

$\displaystyle L$ $\displaystyle=\frac{1}{N}\sum^{N}_{i}w_{i}\cdot L_{i}$ (3)
$\displaystyle=-\frac{1}{N}\sum^{N}_{i}w_{i}\cdot\log\left(\frac{e^{f_{y_{i}}}}{\sum^{K}_{j}e^{f_{j}}}\right)$
$\displaystyle=-\frac{1}{N}\sum^{N}_{i}\log\left(\frac{e^{w_{i}\cdot
f_{y_{i}}}}{(\sum^{K}_{j}e^{f_{j}})^{w_{i}}}\right)$

where $f_{y_{i}}=\boldsymbol{W}_{y_{i}}^{T}\cdot x_{i}$. $\boldsymbol{W}$ is
the weights of the last fully connected layer. $\boldsymbol{W}_{y_{i}}$ is the
$y_{i}$-th column of $\boldsymbol{W}$. The bias $\boldsymbol{b}$ is omit for
simplify analysis. $x_{i}$ is the generated feature to the classifier. Scale
the loss in Equ. 3, we could have the following relationship:

$\displaystyle L$
$\displaystyle=-\frac{1}{N}\sum^{N}_{i}\log\left(\frac{e^{w_{i}\cdot
f_{y_{i}}}}{(\sum^{K}_{j}e^{f_{j}})^{w_{i}}}\right)$ (4)
$\displaystyle\leqslant-\frac{1}{N}\sum^{N}_{i}\log\left(\frac{e^{w_{i}\cdot
f_{y_{i}}}}{\sum^{K}_{j}e^{w_{i}\cdot f_{j}}}\right)$
$\displaystyle=-\frac{1}{N}\sum^{N}_{i}\log\left(\frac{e^{\boldsymbol{W}_{y_{i}}^{T}(w_{i}\cdot\boldsymbol{x}_{i})}}{\sum^{K}_{j}e^{\boldsymbol{W}_{j}^{T}(w_{i}\cdot\boldsymbol{x}_{i})}}\right)$
$\displaystyle=L^{{}^{\prime}}$

where inequalities 32 is true if $0<w<1$. The proof of inequalities 32 is in
the supplementary material. $L^{{}^{\prime}}$ weights the features directly
instead of the loss value of each sample. Besides, $L^{{}^{\prime}}$ is the
upper bound of original softmax loss $L$. Therefore, if $L^{{}^{\prime}}$ goes
to 0, then L has to go to 0. That is, the method of feature weighting can get
better results than the direct weighting of loss value. The $w_{i}$ is also
very important, and it should be related to the content of features. The
existing attention mechanisms generate weights based on the content of
features. Thus, we try to generate $w_{i}$ based on attention mechanisms.

###  III-B Batch Aware Attention Module (B$\boldsymbol{A}^{2}$M)

The attention mechanism is to strengthens information related to optimization
goals and suppresses irrelevant information. Existing sample-wise attention
are limited to conduct feature enrichment in the spatial or channel dimensions
of the feature map. We will extend the attention to batch dimension to
generate the weight which is related to the content through the sample-wise
attention and collaboratively rectify the features of all samples.

$\displaystyle w_{i}$ $\displaystyle=B\boldsymbol{A}^{2}M(x_{i})$ (5)
$\displaystyle=Nomalization(\mathbf{A_{C}}(x_{i}),\mathbf{A_{LS}}(x_{i}),\mathbf{A_{GS}}(x_{i}))$

where $x_{i}\in\mathbb{R}^{C\times H\times W}$ is the feature map. The content
of the feature is arranged by three-dimension. Therefore, we decompose the
feature from the channel, local spatial and global spatial.
$\mathbf{A_{C}}(x_{i})$ denotes the channel attention,
$\mathbf{A_{LS}}(x_{i})$ represents the local spatial attention,
$\mathbf{A_{GS}}(x_{i})$ expresses the global spatial attention. The three
modules are calculated in parallel branches, which are shown in the bottom-
left of Fig. 2. We will introduce each module in detail in the following
sections.

####  III-B1 Channel attention module ($\mathbf{A_{C}}$).

First, we introduce the channel attention module. The channel contains rich
feature information because the feature map $x_{i}$ is arranged by $C$ feature
planes. As shown in Fig. 3, each plane has different semantic information [1].
In order to exploit the inter-channel information. We follow the practices in
SE [1] to build a channel attention module. First, we aggregate the feature
map of each channel by taking global average pooling (GAP) on the feature map
$x_{i}$ and produce a channel vector $GAP(x_{i})\in\mathbb{R}^{C\times 1\times
1}$. To estimate attention across channels from channel vector $GAP(x_{i})$,
we use two fully connected layers ($FC_{0}$ and $FC_{1}$). In order to save
parameter overhead, we set the output dimension of $FC_{0}$ as $C/R$, where
$R$ control the overhead of parameters. After $FC_{1}$, we use a batch
normalization (BN) layer [30] to regulate the output scale. In summary then,
we computed the channel attention ($\mathbf{A}_{C}\in\mathbb{R}^{C\times
1\times 1}$) as:

$\displaystyle\mathbf{A_{C}}(x_{i})$
$\displaystyle=\textbf{BN}(FC_{1}(FC_{0}(GAP(x_{i}))))$ (6)
$\displaystyle=\textbf{BN}(W_{1}(W_{0}(GAP(x_{i}))))$

where $W_{0}\in\mathbb{R}^{(C/R)\times C}$,$W_{1}\in\mathbb{R}^{C\times(C/R)}$
are trainable parameters and BN denotes a batch normalization operation.

####  III-B2 Local Spatial attention module ($\mathbf{A_{LS}}$).

Then, we introduce the local spatial attention module. The spatial dimension
usually reflect details of image content. Constructing a local spatial
attention module can highlight the area of interest effectively. We cascade
one $3\times 3$ convolution layer between two $1\times 1$ convolution layers
to perform local semantic spatial signal aggregation operations. We also
utilize a batch normalization layer to adjust the output scale. In short, we
compute local spatial attention response
($\mathbf{A_{LS}}\in\mathbb{R}^{C\times H\times W}$) as:

$\mathbf{A_{LS}}(x_{i})=\textbf{BN}(g^{1\times 1}_{2}(g^{3\times
3}_{1}(g^{1\times 1}_{0}(x_{i}))))$ (7)

where $g$ denotes a convolution operation, and the superscripts denote
convolutional filter sizes. We divided $x_{i}$ in channel for $R$ groups for
saving memory. Through this spindle-like structural design, we can effectively
aggregate local spatial information with small parameter overhead.

####  III-B3 Global Spatial attention module ($\mathbf{A_{GS}}$).

Finally, we introduce the global spatial attention module. The global spatial
emphasizes the long-range relationship between every pixel in the spatial
extent. It can be used as a supplement to local spatial attention. Many
efforts [9, 31, 8] claim that the long-range interaction can help the feature
to be more power. Inspired by the manners of extracting long-range interaction
in [9]. We simply generate global spatial attention map
($\mathbf{A_{GS}}\in\mathbb{R}^{C\times H\times W}$) as following:

$\displaystyle\mathbf{A_{GS}}(x_{i})=\mathit{softmax}(f(x_{i})\times(g(x_{i}))^{T})\times
h(x_{i})$ (8)

where $f$, $g$ and $h$ are convolutional operation with $1\times 1$ kernel.
After reshaping, $f(x_{i})\in\mathbb{R}^{WH\times C}$,
$g(x_{i})\in\mathbb{R}^{WH\times C}$ and $h(x_{i})\in\mathbb{R}^{WH\times C}$.
The reshaping operations are not shown in Eqn. 8 explicitly. However, the
specific structure can refer to the bottom-right in Fig. 2. $\times$ denotes
matrix multiplication. $X$ is processed by $R$ groups for saving memory.
Finally, $\mathbf{A_{GS}}(x_{i})$ could capture long range interactions to
compensate the locality of $\mathbf{A_{LS}}(x_{i})$.

####  III-B4 Combination and Excitation for Batch

In this section, we combine the results generated by the above sample-wised
attention modules. First, we normalize the scale of the result spectrum. Then,
we choose the result with the strongest activation. Finally, we perform
dimension reduction operation on the output of the previous step to one
dimension along the channel. In general, $\textbf{A}({x}_{i})$ is formulated
as:

$\textbf{A}({x}_{i})=mean(\max(A_{C}(x_{i}),A^{\prime}_{LS}(x_{i}),A^{\prime}_{GS}(x_{i})))$
(9)

where $A^{\prime}_{LS}(x_{i})=GAP(A_{LS}(x_{i}))$ and
$A^{\prime}_{GS}(x_{i})=GAP(A_{GS}(x_{i}))$. $A^{\prime}_{LS}(x_{i})$ and
$A^{\prime}_{GS}(x_{i})\in\mathbb{R}^{C\times 1\times 1}$. $\max$ is the
maximum operation, mean is the operation of dimension reduction along channel
(C). Thus, the $\textbf{A}({x}_{i})\in\mathbb{R}^{1\times 1\times 1}$. The
value $\textbf{A}(x_{i})$ for each $x_{i}$ is related to the complexity of
contents.

Based on the above operations, we get sample-wise attention representation
(SAR) of each sample, and then we use the SARs of the batch to rectify each
sample. The output of batch-wise representation is:

$\displaystyle\textbf{SAR}=[A_{1},A_{2},...A_{N}]\in\mathbb{R}^{N\times
1\times 1\times 1}$ (10)

where $N$ represents the number of images in the current batch. To make the
inequalities 32 be true, we normalize the vector of SAR in a batch and make
SARs more distinguishable. We use softmax function, as shown below. After
applying softmax, each value will be in the interval $(0,1)$. Then, we
multiply the SARs with the batch of sample feature maps.

$\displaystyle
w_{i}=\sigma(\textbf{SAR})_{i}=\frac{e^{A_{i}}}{\sum_{j=1}^{N}{e^{A_{i}}}}$
(11) $\displaystyle\textbf{W}=(w_{1},w_{2},...,w_{N})$ (12)

where $\textbf{W}\in\mathbb{R}^{N\times 1\times 1\times 1}$ is the vector of
weights in current training batch. The batch of refined output
$\hat{\textbf{X}}$ could be computed as:

$\hat{\textbf{X}}=\textbf{W}\otimes\textbf{X}$ (13)

where $\otimes$ denotes element-wise multiplication. It means that each sample
$x_{i}\in\mathbb{R}^{C\times H\times W}$ multiplies with a normalized batch-
wise attention representation $S_{i}$.

###  III-C Batch Aware Attention Network (B$\boldsymbol{A}^{2}$Net)

Given the feature of variations across different layers in a CNN, we try to
embed B$\boldsymbol{A}^{2}$M into different blocks in existing CNNs to build
B$\boldsymbol{A}^{2}$Net. A list of composed blocks could represent most
classical ConvNet:

$\displaystyle\textit{ConvNet}(X_{i})$
$\displaystyle=FC(F_{T}(...(F_{2}(F_{1}(X_{i})))))$ (14)

where $F_{j}$ represents basic or residual block ($j\in[1,T]$, $T$ is the
number of blocks) which is shown in Fig. 4 (a) and (b). $X_{i}$ is the $i$-th
image in the batch. $FC$ represents the classifier. Activation units (e.g.
ReLU [32]) are omit in Equ. 14. We place B$\boldsymbol{A}^{2}$M between blocks
to build B$\boldsymbol{A}^{2}$Net:

$\displaystyle H(X_{i})$
$\displaystyle=FC(w_{Ti}F_{T}(...(w_{2i}F_{2}(w_{1i}F_{1}(X_{i})))))$ (15)

where $H$ represents B$\boldsymbol{A}^{2}$Net, $F_{j}$ represents the $j_{th}$
residual block, $w_{ji}$ is the $i_{th}$ element of W in the $j_{th}$ block.
$T$ is four for ResNet-50.

Figure 4: Illustration of the proposed block in B$\boldsymbol{A}^{2}$Net,
basic block and residual block.

###  III-D Inference and Complexity analysis

In this section, we will discuss the inference and complexity of
B$\boldsymbol{A}^{2}$M. B$\boldsymbol{A}^{2}$M is an adjunct to standard
training processes. During the training stage, B$\boldsymbol{A}^{2}$M equally
scales the value of the output vector by weighting samples, thus influencing
the loss value. However, during the inference stage, the batch aware weight
only affects the absolute value of the output vector, and the rank of elements
in the vector does not change. Thus it does not change the final prediction.
To verify that B$\boldsymbol{A}^{2}$M does not affect the results in the
inference stage, we activate the B$\boldsymbol{A}^{2}$M in the ResNet50 on the
ImageNet. The size of test batch is $\\{1,2,4,8,16,32,64,128,256\\}$. All the
results are $21.63\%$. Thus, during the inference phase, we test one image a
time as the common practice. B$\boldsymbol{A}^{2}$M generates SAR value for
each test image using Eqn. 9. The feature is refined by multiplying with the
corresponding SAR. Besides, we deactivate B$\boldsymbol{A}^{2}$M in the
inference phase because we usually infer one image at a time in the common
practice where the size of the batch is 1. At this time, excitation
B$\boldsymbol{A}^{2}$M for the batch will lead to over-calculation.

The complexity of B$\boldsymbol{A}^{2}$M could be divided into three parts,
and we use FLOPs and Params to measure them, respectively. We have following
equations based on [33]:

$\displaystyle\mathbf{Params}\sim
O\Bigg{(}\begin{matrix}\underbrace{C^{2}\frac{2}{R}}\\\
\mathbf{A_{C}}\end{matrix}+\begin{matrix}\underbrace{C^{2}(\frac{9+2R}{R^{2}})}\\\
\mathbf{A_{LS}}\end{matrix}+\begin{matrix}\underbrace{\frac{3C^{2}}{R^{2}}}\\\
\mathbf{A_{GS}}\end{matrix}\Bigg{)}$ (16) $\displaystyle\textbf{FLOPs}\sim
O\Bigg{(}\begin{matrix}\underbrace{C^{2}\frac{2}{R}}\\\
\mathbf{A_{C}}\end{matrix}+\begin{matrix}\underbrace{HWC^{2}(\frac{9+2R}{R^{2}})}\\\
\mathbf{A_{LS}}\end{matrix}+$ (17)
$\displaystyle\begin{matrix}\underbrace{HWC^{2}(\frac{3}{R^{2}})+2(HW)^{2}(\frac{(2C-R)}{R})}\\\
\mathbf{A_{GS}}\end{matrix}\Bigg{)}$

where the input feature map is $\in\mathbb{R}^{N\times C\times H\times W}$ .
We can see that based on Eqn. 17 and 16, R plays an important role in
controlling the complexity of the B$\boldsymbol{A}^{2}$M. We will discuss the
influence of R in ablation experiments. The Flops of $\mathbf{A_{GS}}$ in Eqn.
17 have two parts. The first part is the time-complexity of three
convolutional operations, and the second part is related to two matrix
multiplication operations. However, there are no extra parameters in matrix
multiplication, Thus the parameter measure in $\mathbf{A_{GS}}$ is
$\frac{3C^{2}}{R^{2}}$.

##  IV Experiments

In order to evaluate the effect of proposed B$\boldsymbol{A}^{2}$M, we conduct
several experiments on two widely-used image classification datasets:
CIFAR-100 [34] and ImageNet-1k [35]. PyTorch-0.11.0 library [36] is utilized
to implement all experiments on NVIDIA TITANX GPU graphics cards. CIFAR-
related and ImageNet-related experiments are on a single card and eight cards,
respectively. For each configuration, we repeat the experiments for five times
with different random seeds and report the median result and the best (per
run) validation performance of the architectures over time. The setting of
hyper-parameters is the same as [6].

###  IV-A Experiments on CIFAR-100

CIFAR-100 [34] is a tiny natural image dataset, which contains 100 different
classes. Each image is in size of $32\times 32$. There are 500 images for
training and 100 images validation per class. We adopt some simple data
augmentation strategies, such as random crops, random horizontal flips, and
mean-variance normalization.We performed image classification with a range of
ResNet architecture and its variants: ResNet-50 [11], ResNet-101 [11],
PreResNet-110 [37], Wide-ResNet-28 (WRN-28) [13] and ResNeXt-29 [14]. We
reported classification error on validation set as well as model size (Params)
and computational complexity (Flops).

Results are shown in Table. XI. It could be concluded from the results that
B$\boldsymbol{A}^{2}$M could consistently improve the classification
performance of the network regardless of the network structure without
increasing the amount of calculation. From the above results, we can conclude
that B$\boldsymbol{A}^{2}$M is an effective method.

Architecture | Params(M) | FLOPs(G) | Error (%)
---|---|---|---
ResNet-50 [11] | 23.71 | 1.22 | 21.49
+B$\boldsymbol{A}^{2}$M (ours) | 24.21 | 1.33 | 17.60(-3.89)
ResNet-101 [11] | 42.70 | 2.44 | 20.00
+B$\boldsymbol{A}^{2}$M (ours) | 42.32 | 2.55 | 17.88(-2.12)
PreResNet-110 [37] | 1.73 | 0.25 | 22.22
+B$\boldsymbol{A}^{2}$M (ours) | 1.86 | 0.25 | 21.41(-0.81)
WRN-28(w=8) [13] | 23.40 | 3.36 | 20.40
+B$\boldsymbol{A}^{2}$M (ours) | 23.44 | 3.46 | 17.95(-2.45)
WRN-28(w=10) [13] | 36.54 | 5.24 | 18.89
+B$\boldsymbol{A}^{2}$M (ours) | 36.59 | 5.39 | 17.30(-1.59)
ResNeXt-29 [14] | 34.52 | 4.99 | 18.18
+B$\boldsymbol{A}^{2}$M (ours) | 34.78 | 5.53 | 16.10(-2.08)

TABLE I: Image classification results on CIFAR-100 [34] across a range of
ResNet architectures and its variants. The results of compared methods could
be find in corresponding papers.

###  IV-B Experiments on ImageNet-1K

In order to further validate the effectiveness of B$\boldsymbol{A}^{2}$M, in
this section, we perform image classification experiments on more challenging
1000-class ImageNet dataset [35], which contains about 1.3 million training
color images as well as 50,000 validation images. We use the same data
augmentation strategy as [11, 12] for training and a single-crop evaluation
with the size of $224\times 224$ in the testing phase. We report FLOPs and
Params for each model, as well as the top-1 and top-5 classification errors on
the validation set. We use a range of ResNet architectures and their variants:
ResNet-18 [11], ResNet-34 [11], Wide-ResNet-18 (WRN-18) [13], ResNeXt-50 [14]
and ResNeXt-101 [14].

The results are shown in Table. XII. The networks with B$\boldsymbol{A}^{2}$M
outperform the baseline network in performance, which is an excellent proof
that B$\boldsymbol{A}^{2}$M could improve the discrimination of features in
such a challenging dataset. It is worth noting that it could be negligible in
the overhead of both parameters and computation, which shows that
B$\boldsymbol{A}^{2}$M could significantly improve the model capacity
efficiently. When B$\boldsymbol{A}^{2}$M is combined with WRN, the increment
of the parameter is a little bigger. The abnormal increment is mainly due to
the wide of $WRN$ significantly increases the number of channel $C$ in the
convolutional layer.

Architecture | Params.(M) | FLOPs(G) | Top1-Error(%)
---|---|---|---
ResNet-18 [11] | 11.69 | 1.81 | 29.60
+B$\boldsymbol{A}^{2}$M (ours) | 12.28 | 1.92 | 28.62(-0.98)
ResNet-34 [11] | 21.80 | 3.66 | 26.69
+B$\boldsymbol{A}^{2}$M (ours) | 21.92 | 3.77 | 25.15(-1.54)
WRN-18(w=1.5) [13] | 25.88 | 3.87 | 26.85
+B$\boldsymbol{A}^{2}$M (ours) | 28.43 | 4.69 | 23.60(-3.25)
WRN-18(w=2) [13] | 45.62 | 6.70 | 25.63
+B$\boldsymbol{A}^{2}$M (ours) | 47.88 | 7.56 | 23.60(-2.03)
ResNeXt-50 [14] | 25.03 | 3.77 | 22.85
+B$\boldsymbol{A}^{2}$M (ours) | 27.25 | 4.47 | 20.98(-1.87)
ResNeXt-101 [14] | 44.18 | 7.51 | 21.54
+B$\boldsymbol{A}^{2}$M (ours) | 45.51 | 8.23 | 20.05(-1.49)

TABLE II: Image classification results on ImageNet-1K dataset [35] across a
range of ResNet architectures and its variants.

###  IV-C Comparison with Focal Loss and OHEM

In this section, we compare B$\boldsymbol{A}^{2}$M with Focal Loss [28] and
online hard example mining (OHEM) [29]. B$\boldsymbol{A}^{2}$M re-weights
samples in a batch by calculating the attention value of the samples, this
could also solve the problem of class imbalance in the object detection field
to some extent. Focal loss and OHEM are specifically designed to solve the
problem of class imbalance. OHEM selects a candidate ROI with a massive loss
to solve the category imbalance problem. Focal loss achieves the effect of
sample selection by reducing the weight of the easily categorized samples so
that the model more focus on difficult samples during training. We perform
experiments on the union set of PASCAL VOC 2007 trainval and PASCAL VOC 2012
trainval (VOC0712) and evaluate on the PASCAL VOC 2007 test set.

For Focal Loss, We adopt RetinaNet [28] as our detection method and ImageNet
pre-trained ResNet-50 as our baseline networks. Then we replace Focal Loss
with SoftmaxLoss and change the ResNet-50 to ResNet-50 with
B$\boldsymbol{A}^{2}$M. Results are shown in Table III, the Focal loss is
lower than B$\boldsymbol{A}^{2}$M on both Recall and mAP.

Detector | Method | Recall (%) | mAP (%)
---|---|---|---
RetinaNet | Focal Loss | 96.27 | 79.10
RetinaNet | B$\boldsymbol{A}^{2}$M(ours) | 97.12(+0.85) | 79.60(+0.50)

TABLE III: Object detection results compared with Focal Loss. Baseline
detector is RetinaNet. Backbone is ResNet-50 pre-trained on ImageNet.

For OHEM, We adopt Faster-RCNN [38] as our detection method and ImageNet pre-
trained ResNet-50 as our baseline network. From the results summarized in
Table IV. We can see that the Recall and mAP of B$\boldsymbol{A}^{2}$M are
both optimal. Compared with the traditional methods of re-weight samples based
on the loss value, the weights generated by B$\boldsymbol{A}^{2}$M are better.

Detector | Method | Recall (%) | mAP (%)
---|---|---|---
Faster R-CNN | - | 92.97 | 80.10
Faster R-CNN | OHEM [29] | 90.57 | 80.50
Faster R-CNN | B$\boldsymbol{A}^{2}$M (ours) | 93.99(+1.02) | 81.00(+0.90)

TABLE IV: Object detection results compared with OHEM. Baseline detector is
Faster R-CNN. Backbone is ResNet-50 pre-trained on ImageNet.

###  IV-D Ablation Experiments

####  IV-D1 Comparison with Sample-wise Attention

In this section, we systematically compare B$\boldsymbol{A}^{2}$M with some
attention works through image classification on ImageNet-1K [35]. We choose SE
[1] and SRM [2], SK [4], SGE [5] and GE [3]), BAM [6], CBAM [7], GC [8], AA
[9] and SASA [19] as compared methods. These works mainly focus on exploring
sample-wise attention, whose enhancements are limited within each individual
sample including the spatial and channel dimensions. We choose ResNet-50 and
ResNet-101 as baseline networks and replace attention modules at each block.
All models maintain the same parameter settings during training.

| ResNet-50 | ResNet-101
---|---|---
Params.(M) | FLOPs(G) | Top1-Error(%) | Top5-Error(%) | Params.(M) | FLOPs(G) | Top1-Error(%) | Top5-Error(%)
Baseline | 25.56 | 3.86 | 24.56 | 7.50 | 44.55 | 7.57 | 23.38 | 6.88
+SE [1] | 28.09 | 3.86 | 23.14 | 6.70 | 49.33 | 7.58 | 22.35 | 6.19
+SRM [2] | 25.62 | 3.88 | 22.87 | 6.49 | 44.68 | 7.62 | 21.53 | 5.80
+SK [4] | 26.15 | 4.19 | 22.46 | 6.30 | 45.68 | 7.98 | 21.21 | 5.73
+SGE [5] | 25.56 | 4.13 | 22.42 | 6.34 | 44.55 | 7.86 | 21.20 | 5.63
+GE [3] | 31.20 | 3.87 | 22.00 | 5.87 | 33.70 | 3.87 | 21.88 | 5.80
+BAM [6] | 25.92 | 3.94 | 24.02 | 7.18 | 44.91 | 7.65 | 22.44 | 6.29
+CBAM [7] | 28.09 | 3.86 | 22.66 | 6.31 | 49.33 | 7.58 | 21.51 | 5.69
+GC [8] | 28.08 | 3.87 | 22.30 | 6.34 | 49.36 | 7.86 | 25.36 | 7.93
+AA [9] | 25.80 | 4.15 | 22.30 | 6.20 | 45.40 | 8.05 | 21.30 | 5.60
+SASA [19] | 18.00 | 7.20 | 22.40 | - | - | - | - | -
+B$\boldsymbol{A}^{2}$M (ours) | 26.21(+0.65) | 4.32(+0.46) | 21.63(-2.93) | 5.80(-1.70) | 45.87(+1.32) | 8.05(+0.48) | 20.85(-2.53) | 5.58(-1.30)

TABLE V: Image classification results on ImageNet-1K dataset [35] with
ResNet-50 and ResNet-101 across a range of attention mechanisms:SE [1], SRM
[2], GE [3], SK [4], SGE [5], BAM [6], CBAM [7], GC [8], AA [9] and SASA [19].

The results are shown in Table. V, It could be inferred from the results that
B$\boldsymbol{A}^{2}$M could significantly improve the performance of baseline
and B$\boldsymbol{A}^{2}$M is superior to sample-wise attention mechanisms in
improving the performance of baseline with little overhead on parameters and
computation. Besides, In Table. XII and Table. V, we have performed
experiments with ResNet18/34/50/101 on ImageNet. The gains are significant
improved from 0.98% for ResNet18 to 2.53% for ResNet101 as the network
architecture becoming deeper.

####  IV-D2 Cooperation with Sample-wise Attention

In this section, we perform experiments with a combination of batch-with and
other sample-wise attention methods. We assume that the combination with other
sample-with attention methods should improve model performance even more. We
choose SE [1], CBAM [7], BAM [6] to be the instance methods. We use ResNet-50
as the baseline networks. The results are shown in Table. VI. The combination
of B$\boldsymbol{A}^{2}$M with other attention methods could improve
performance even more. Especially, the combination of BAM and
B$\boldsymbol{A}^{2}$M even gets the classification error of $17.00\%$.

Architecture | Params.(M) | FLOPs(G) | Error (%)
---|---|---|---
ResNet-50 [11] | 23.71 | 1.22 | 21.49
+SE [1] | 26.24 | 1.23 | 20.72
+CBAM [7] | 27.44 | 1.34 | 21.01
+BAM [6] | 24.07 | 1.25 | 20.00
+B$\boldsymbol{A}^{2}$M | 24.21 | 1.33 | 17.60
+SE+B$\boldsymbol{A}^{2}$M | 26.74 | 1.34 | 17.52
+CBAM+B$\boldsymbol{A}^{2}$M | 27.94 | 1.45 | 17.43
+BAM+B$\boldsymbol{A}^{2}$M | 24.57 | 1.36 | 17.00

TABLE VI: Image classification results with other attention modules on
CIFAR-100 [34].

####  IV-D3 The position of B$\boldsymbol{A}^{2}$M

The location for B$\boldsymbol{A}^{2}$M in the residual-like network has two
choices. The first is to embed B$\boldsymbol{A}^{2}$M inside the block, as
shown in Fig. 4 (c ). The other is to embed B$\boldsymbol{A}^{2}$M between
blocks, as shown in Fig. 4 (d). In this section, we conduct experiments to
determine the embed mode. The results are exhibited in Table. VII. The average
error and standard deviation of five random runs are recorded and the best
results are in bold. We found that placing B$\boldsymbol{A}^{2}$M between
blocks is better than inside blocks in improving the performance (17.60% Vs
17.71%) and stability of the model (0.15 Vs 0.40). Thus, we assign
B$\boldsymbol{A}^{2}$M between blocks to build different
B$\boldsymbol{A}^{2}$Net.

Position | Error(%) | std
---|---|---
Inside (Fig. 4(c)) | 17.71 | 0.40
Between (Fig. 4(d)) | 17.60 | 0.15
TABLE VII: The results of the position of B$\boldsymbol{A}^{2}$M.

####  IV-D4 Sensitivity of Hyper-Parameters

In this section, we empirically show the effectiveness of design choice. For
this ablation study, we use ResNet-50 as the baseline architecture and train
it on the CIFAR-100 dataset.

The size of Batch (N) . We perform experiments with
$N\in\\{4,8,16,32,64,128,256\\}$ under the memory limitation of GPU (12G). The
learning rate (lr) for $N=128$ is $0.1$. For other $N$, the lr changes
linearly. Each setting runs for five times, and we reported mean test error.
As shown in Fig. 5, classification performance continuously improves with
larger N, which indicates that B$\boldsymbol{A}^{2}$M is more efficient when
more samples are involved in re-weighting. In experiments, we set N to be 256
to balance computing efficiency and hardware overhead.

Figure 5: Test error vs The size of Batch. All models are trained on one GPU
and learning rate changed linearly.

Reduction (R).We conduct experiments to determine hyper-parameters in
B$\boldsymbol{A}^{2}$M, which is the reduction ratio (R). R is used to control
the number of channels in three modules, which enable us to control the
capacity and the overhead of B$\boldsymbol{A}^{2}$M. The minimum number of
channels in the ablation experiment is 32. In Table. VIII, we compare mean
performance for five random runs of $R\in\\{2,4,8,16,32\\}$ and corresponding
model size (Params) and computational complexity (Flops, Madds).
Interestingly, as R increases, the overhead of computation and model size
continue to decrease. However, the corresponding performance first drops and
then rises. When $R$ is 32, the corresponding performance is the best. To
balance model complexity and performance, we choose R to be 32.

R | Params(MB) | Flops(GB) | Error(%) | std
---|---|---|---|---
2 | 70.80 | 2.86 | 17.71 | 0.40
4 | 40.65 | 1.78 | 18.04 | 0.19
8 | 30.12 | 1.47 | 18.13 | 0.10
16 | 25.99 | 1.37 | 18.01 | 0.13
32 | 24.21 | 1.33 | 17.60 | 0.15

TABLE VIII: Results with different reduction in $\\{2,4,8,16,32\\}$. We train
ResNet-50 as the base network on CIFAR-100.

Design Choices of B$\boldsymbol{A}^{2}$M. We perform experiments to determine
the design choices of B$\boldsymbol{A}^{2}$M. There are three attention
modules in B$\boldsymbol{A}^{2}$M. Thus, we have seven combinations of
attention mechanisms. The results are shown in Table. IX. From the second row
to the fourth row, we can find that LSA ( $17.86\%$ ) plays the most crucial
role in B$\boldsymbol{A}^{2}$M. Besides, the combination of LSA and GSA gets
$17.78\%$ performance which is better than single LSA. The above results
demonstrate that the long-range interactions could compensate the local
spatial attention. Furthermore, the full version of B$\boldsymbol{A}^{2}$M
achieves the best performance.

CA | LSA | GSA | Error (%) | Std
---|---|---|---|---
✓ |  |  | 18.06 | 0.20
| ✓ |  | 17.86 | 0.30
|  | ✓ | 18.18 | 0.29
✓ | ✓ |  | 17.71 | 0.30
✓ |  | ✓ | 18.00 | 0.23
| ✓ | ✓ | 17.78 | 0.28
✓ | ✓ | ✓ | 17.60 | 0.15

TABLE IX: The results of different design choices of B$\boldsymbol{A}^{2}$M.
The last row is the result of B$\boldsymbol{A}^{2}$M.

##  V Conclusion

We have presented a batch-wise attention module(B$\boldsymbol{A}^{2}$M). It
provides new insight into how the attention mechanism can enhance the
discrimination power of features. B$\boldsymbol{A}^{2}$M rectifies the
features by sample-wise attention across a batch. It is a lightweight module
and can be embed between blocks of a CNN. To verify B$\boldsymbol{A}^{2}$M’s
efficacy, we conducted extensive experiments with various state-of-the-art
models. The results proofed that B$\boldsymbol{A}^{2}$M can boost the
performance of the baseline and outperform other methods.

###  -A Supplementary material

In this supplementary material, we give the proof of inequalities.4 and
present more results and conduct some object detection and instance
segmentation experiments to verify the effectiveness of
B$\boldsymbol{A}^{2}$M.

###  -B Proof of Equation

Lemma 1. For variable $x$ and $y$ hold

$\displaystyle(x+y)^{w}<x^{w}+y^{w}$ (18)

where

$\displaystyle 0<w<1,x,y>0$ (19)

proof. Let’s define a function $f(t)$ as follow:

$\displaystyle f(t)=(t+1)^{w}-t^{w}-1$ (20)

where

$\displaystyle t\geqslant 0,0<w<1$ (21)

The derivative of $f(t)$ is:

$\displaystyle f^{{}^{\prime}}(t)$ $\displaystyle=w(t+1)^{w-1}-wt^{w-1}$ (22)
$\displaystyle=w[(t+1)^{w-1}-t^{w-1}]$ $\displaystyle<0$

The last line with inequality sign of Equ. 22 is because the function
$t^{(w-1)}$ is monotonic decreasing function when $-1<w-1<0$.

Therefore, $f(t)$ is also a monotonic decreasing function. For each element
$t>0$, we have the following relation:

$\displaystyle f(t)$ $\displaystyle=(t+1)^{w}-t^{w}-1$ (23) $\displaystyle\leq
f(0)=0$

Therefore,

$\displaystyle(t+1)^{w}<t^{w}+1$ (24)

where

$\displaystyle t>0,0<w<1$ (25)

Let $t=x/y$, where $x>0$, $y>0$. We will get the following conclusion:

$\displaystyle(\frac{x}{y}+1)^{w}<(\frac{x}{y})^{w}+1$ (26)

Due to $y>0$ and $0<w<1$, thus, $y^{w}>0$. We use $y^{w}$ multiply the both
sides of Equ. 26. Therefore, we get

$\displaystyle(x+y)^{w}<x^{w}+y^{w}$ (27)

Lemma 2. For variable $x_{i}$ hold

$\displaystyle(\sum^{N}_{i}x_{i})^{w}\leqslant\sum^{N}_{i}(x_{i})^{w}$ (28)

where

$\displaystyle 0<w<1,x_{i}>0,N\in\mathbb{N}$ (29)

proof. Let’s employ mathematical induction with the number $N$.

STEP1

, When $N=1$, The Equ. 28 gets the equal sign.

STEP2

, When $N=2$, The Equ. 28 could be hold by Lemma 1.

STEP3

, if $N=K$($K\in\mathbb{N}$ and $K>2$), The Equ. 28 is true. We get

$\displaystyle(\sum^{K}_{i}x_{i})^{w}\leqslant\sum^{K}_{i}(x_{i})^{w}$ (30)

When $N=K+1$,

$\displaystyle(\sum^{K+1}_{i}x_{i})^{w}$
$\displaystyle=((\sum^{K}_{i}x_{i})+x_{(K+1)})^{w}$ (31)
$\displaystyle\leqslant\sum^{K}_{i}(x_{i})^{w}+x^{w}_{(K+1)},$
$\displaystyle=\sum^{K+1}_{i}(x_{i})^{w}$

Therefore, Equ. 28 holds when $N=K+1$.

CONCLUTION

,

The Equ. 28 is true, When $0<w<1,x_{i}>0,N\in\mathbb{N}$.

Therefore, When we apply Lemma 2. in the loss function $L$, we could got the
following relationship.

$\displaystyle L$
$\displaystyle=-\frac{1}{N}\sum^{N}_{i}\log\left(\frac{e^{w_{i}\cdot
f_{y_{i}}}}{(\sum^{K}_{j}e^{f_{j}})^{w_{i}}}\right)$ (32)
$\displaystyle\leqslant-\frac{1}{N}\sum^{N}_{i}\log\left(\frac{e^{w_{i}\cdot
f_{y_{i}}}}{\sum^{K}_{j}e^{w_{i}\cdot f_{j}}}\right)$

From Equ. 32, we could know that we could re-weight the feature map instead of
loss value.

###  -C Experiments with Lightweight Networks

We present more results with classical lightweight networks (DensNet-100 [15],
ShuffleNetv2 [16], MobileNetv2 [17] and EfficientNetB0 [18]) on CIFAR-100.
Especially, the EfficientnetB0 is obtained by Neural Architecture Search
(NAS). Results are shown in the Table. X.

Architecture | Params(M) | FLOPs(G) | Err (%)
---|---|---|---
DensNet-100 [15] | 0.76 | 0.29 | 21.95
+B$\boldsymbol{A}^{2}$M (ours) | 0.78 | 0.30 | 20.63(-1.32)
ShuffleNetv2 [16] | 1.3 | 0.05 | 26.8
+B$\boldsymbol{A}^{2}$M (ours) | 1.92 | 0.06 | 25.49(-1.31)
MobileNetv2 [17] | 2.26 | 0.07 | 30.17
+B$\boldsymbol{A}^{2}$M (ours) | 2.31 | 0.07 | 26.77(-3.40)
EfficientNetB0 [18] | 2.80 | 0.03 | 36.01
+B$\boldsymbol{A}^{2}$M (ours) | 4.18 | 0.03 | 33.98(-2.03)

TABLE X: Image classification results on CIFAR-100 with lightweight networks
and NAS network.

###  -D Experiments on CIFAR-100

We presents more results with other attention mechanisms on CIFAR-100 in
Table. XI.

Architecture | Params(M) | FLOPs(G) | Err (%)
---|---|---|---
ResNet-50 [11] | 23.71 | 1.22 | 21.49
+SE [1] | 26.24 | 1.23 | 20.72
+CBAM [7] | 27.44 | 1.34 | 21.01
+BAM [6] | 24.07 | 1.25 | 20.00
+B$\boldsymbol{A}^{2}$M (ours) | 24.21 | 1.33 | 17.60(-3.89)
ResNet-101 [11] | 42.70 | 2.44 | 20.00
+SE [1] | 45.56 | 2.54 | 20.89
+CBAM [7] | 49.84 | 2.54 | 20.30
+BAM [6] | 43.06 | 2.46 | 19.61
+B$\boldsymbol{A}^{2}$M (ours) | 42.32 | 2.55 | 17.88(-2.12)
PreResNet-110 [37] | 1.73 | 0.25 | 22.22
+BAM [6] | 1.73 | 0.25 | 21.96
+SE [1] | 1.93 | 0.25 | 21.85
+B$\boldsymbol{A}^{2}$M (ours) | 1.86 | 0.25 | 21.41(-0.81)
WRN-28(w=8) [13] | 23.40 | 3.36 | 20.40
+GE [3] | - | - | 19.74
+SE [1] | 23.58 | 3.36 | 19.85
+BAM [6] | 23.42 | 3.37 | 19.06
+B$\boldsymbol{A}^{2}$M (ours) | 23.44 | 3.46 | 17.95(-2.45)
WRN-28(w=10) [13] | 36.54 | 5.24 | 18.89
+GE [3] | 36.30 | 5.20 | 20.20
+SE [1] | 36.50 | 5.20 | 19.00
+BAM [6] | 36.57 | 5.25 | 18.56
+AA [9] | 36.20 | 5.45 | 18.40
+B$\boldsymbol{A}^{2}$M (ours) | 36.59 | 5.39 | 17.30(-1.59)
ResNeXt-29 [14] | 34.52 | 4.99 | 18.18
+BAM [6] | 34.61 | 5.00 | 16.71
+B$\boldsymbol{A}^{2}$M (ours) | 34.78 | 5.53 | 16.10(-2.08)

TABLE XI: Image classification results on CIFAR-100 [34] across a range of
ResNet architectures and its variants. The results of compared methods could
be find in corresponding papers.

###  -E Experiments on ImageNet

We presents more results with other attention mechanisms on ImageNet in Table.
XII.

Architecture | Params.(M) | FLOPs(G) | Top1Err(%) | Top5Err(%)
---|---|---|---|---
ResNet-18 [11] | 11.69 | 1.81 | 29.60 | 10.55
+SE [1] | 11.78 | 1.81 | 29.41 | 10.22
+BAM [6] | 11.71 | 1.82 | 28.88 | 10.10
+CBAM [7] | 11.78 | 1.82 | 29.27 | 10.09
+B$\boldsymbol{A}^{2}$M (ours) | 12.28 | 1.92 | 28.62(-0.98) | 10.04(-0.51)
ResNet-34 [11] | 21.80 | 3.66 | 26.69 | 8.60
+SE [1] | 21.96 | 3.66 | 26.13 | 8.35
+BAM [6] | - | - | 25.71 | 8.21
+CBAM [7] | 21.96 | 3.67 | 25.99 | 8.24
+B$\boldsymbol{A}^{2}$M (ours) | 21.92 | 3.77 | 25.15(-1.54) | 8.13(-0.47)
WRN-18(w=1.5) [13] | 25.88 | 3.87 | 26.85 | 8.88
+BAM [6] | 25.93 | 3.88 | 26.67 | 8.69
+SE [1] | 26.07 | 3.87 | 26.21 | 8.47
+CBAM [7] | 26.08 | 3.87 | 26.10 | 8.43
+B$\boldsymbol{A}^{2}$M (ours) | 28.43 | 4.69 | 23.60(-3.25) | 7.60(-1.28)
WRN-18(w=2) [13] | 45.62 | 6.70 | 25.63 | 8.20
+BAM [6] | 45.71 | 6.72 | 25.00 | 7.81
+SE [1] | 45.97 | 6.70 | 24.93 | 7.65
+CBAM [7] | 45.97 | 6.70 | 24.84 | 7.63
+B$\boldsymbol{A}^{2}$M (ours) | 47.88 | 7.56 | 23.60(-2.03) | 7.23(-0.97)
ResNeXt-50 [14] | 25.03 | 3.77 | 22.85 | 6.48
+BAM [6] | 25.39 | 3.85 | 22.56 | 6.40
+SE [1] | 27.56 | 3.77 | 21.91 | 6.04
+CBAM [7] | 27.56 | 3.77 | 21.92 | 5.91
SKNet-50 [4] | 27.50 | 4.47 | 20.79 | -
+B$\boldsymbol{A}^{2}$M (ours) | 27.25 | 4.47 | 20.98(-1.87) | 5.85(-0.63)
ResNeXt-101 [14] | 44.18 | 7.51 | 21.54 | 5.75
+SE [1] | 48.96 | 7.51 | 21.17 | 5.66
+CBAM [7] | 48.96 | 7.52 | 21.07 | 5.59
SKNet-101 [4] | 48.90 | 8.46 | 20.19 | -
+B$\boldsymbol{A}^{2}$M (ours) | 45.51 | 8.23 | 20.05(-1.49) | 5.22(-0.53)

TABLE XII: Image classification results on ImageNet-1K dataset [35] across a
range of ResNet architectures and its variants.

###  -F B$\boldsymbol{A}^{2}$M in Network

We perform experiments to determine the choice of B$\boldsymbol{A}^{2}$M
within a network. We use four blocks in the ResNet-50. Thus, we have 15
combinations. The results are shown in Table. XIII. When inserting one
B$\boldsymbol{A}^{2}$M into the ResNet-50, the performance at the second block
is the best, and the performance of the first block is the worst. Then,
embedding B$\boldsymbol{A}^{2}$M at the fourth block can put the performance
forward further. Furthermore, ResNet-50 has the highest performance
($17.60\%$) when all four blocks are embedded in B$\boldsymbol{A}^{2}$M. The
result reveals the rationality of calculating SARs at variant layers.

| No B$\boldsymbol{A}^{2}$M | One B$\boldsymbol{A}^{2}$M | Two B$\boldsymbol{A}^{2}$Ms | Three B$\boldsymbol{A}^{2}$Ms | Four B$\boldsymbol{A}^{2}$Ms
---|---|---|---|---|---
Block1 |  | ✓ |  |  |  | ✓ | ✓ | ✓ |  |  |  |  | ✓ | ✓ | ✓ | ✓
Block2 |  |  | ✓ |  |  | ✓ |  |  | ✓ | ✓ |  | ✓ |  | ✓ | ✓ | ✓
Block3 |  |  |  | ✓ |  |  | ✓ |  | ✓ |  | ✓ | ✓ | ✓ |  | ✓ | ✓
Block4 |  |  |  |  | ✓ |  |  | ✓ |  | ✓ | ✓ | ✓ | ✓ | ✓ |  | ✓
Mean Err (%) | 21.49 | 17.96 | 17.87 | 17.94 | 17.88 | 18.05 | 18.33 | 17.99 | 17.91 | 17.78 | 17.95 | 17.70 | 18.02 | 17.90 | 17.93 | 17.60
Std | 0.13 | 0.14 | 0.12 | 0.24 | 0.22 | 0.36 | 0.30 | 0.36 | 0.17 | 0.21 | 0.32 | 0.11 | 0.18 | 0.17 | 0.21 | 0.15

TABLE XIII: The result of Combination of B$\boldsymbol{A}^{2}$M within a
network. We use ResNet-50 as an instance network. The average error and
standard deviation of five random runs are reported and the best results are
in bold.

###  -G Object Detection and Segmentation

In order to further verify the generalization performance of
B$\boldsymbol{A}^{2}$M, we further carried out object detection and instance-
level segmentation on the MS COCO [39]. The dataset has 80 classes. We train
with the union of 118k train images and report ablations on the remaining 5k
val images (minival). We train all models for 24 epochs (Lr schd=2x) using
synchronized SGD with a weight decay of 0.0001 and a momentum of 0.9. The
learning rate is initialized as 0.01 and decays by a factor of 10 at the 16th
and 22th epochs. We report bounding box AP of different IoU thresholds form
0.5 to 0.95, and object size (small (S), medium (M), large (L)).We adopt
Faster-RCNN [38]and Mask R-CNN framework [40] as our detection method and
ImageNet pre-trained ResNet-50/101 as our baseline networks. As shown in Table
XIV and Table XV, we can see that the detectors embedded in
B$\boldsymbol{A}^{2}$M are better than the baseline method under different
evaluation indicators.

For the instance-level segmentation experiments, we used the classic Mask
R-CNN framework [40] as our instance-level segmentation method and ImageNet
pre-trained ResNet-50/101 as baseline networks. We report the standard COCO
metrics, including AP (averaged over IoU thresholds), $AP_{50}$, $AP_{75}$,
and $AP_{S}$, $AP_{M}$, $AP_{L}$ (AP at different scales). Unless noted, AP is
evaluating using mask IoU. The results are shown in Table XVI. We can see that
regardless of how the evaluation indicators change, the Mask R-CNN embedded in
B$\boldsymbol{A}^{2}$M is even much better than the baseline method. We can
conclude that B$\boldsymbol{A}^{2}$M can boost the performance of the detector
based on the improvement of the ability in feature extraction under the
condition of the small size of the batch.

For experiments of object detection and instance segmentation, we utilize
MMDetection [41] as the platform. All the related hyper-parameters are not
changed. Here we study the performance improvement after inserting
B$\boldsymbol{A}^{2}$M into the backbone network. Because we use the same
detection method, the performance gain can only be attributed to the feature
enhancement capabilities, given by B$\boldsymbol{A}^{2}$M.

Detector | backbone | $AP^{bb}$ | $AP^{bb}_{50}$ | $AP^{bb}_{75}$ | $AP^{bb}_{S}$ | $AP^{bb}_{M}$ | $AP^{bb}_{L}$
---|---|---|---|---|---|---|---
Faster R-CNN | ResNet-50 | 37.60 | 58.90 | 40.90 | 21.80 | 41.50 | 48.80
Faster R-CNN | ResNet-50 w B$\boldsymbol{A}^{2}$M | 38.50(+0.9) | 60.00(+1.1) | 41.90 (+1.0) | 22.20 (+0.4) | 42.10(+0.6) | 49.80(+1.0)
Faster R-CNN | ResNet-101 | 39.40 | 60.60 | 43.00 | 22.10 | 43.60 | 52.10
Faster R-CNN | ResNet-101 w B$\boldsymbol{A}^{2}$M | 40.60(+1.2) | 62.10(+1.5) | 44.40 (+1.4) | 22.70 (+0.6) | 44.30(+0.7) | 53.60(+1.5)

TABLE XIV: Object detection single-model results on COCO minival (bounding box
AP).The baseline detector is  Faster R-CNN with ResNet-50/101 .

Detector | backbone | $AP^{bb}$ | $AP^{bb}_{50}$ | $AP^{bb}_{75}$ | $AP^{bb}_{S}$ | $AP^{bb}_{M}$ | $AP^{bb}_{L}$
---|---|---|---|---|---|---|---
Mask R-CNN | ResNet-50 | 38.30 | 59.70 | 41.30 | 22.50 | 41.70 | 50.40
Mask R-CNN | ResNet-50 w B$\boldsymbol{A}^{2}$M | 39.80(+1.50) | 60.90(+1.20) | 43.70(+2.40) | 23.00(+0.50) | 43.10(+1.40) | 52.10(+1.70)
Mask R-CNN | ResNet-101 | 40.40 | 61.50 | 44.10 | 22.20 | 44.80 | 52.90
Mask R-CNN | ResNet-101 w B$\boldsymbol{A}^{2}$M | 42.20(+1.80) | 62.90(+1.40) | 46.60(+2.50) | 24.00(+1.80) | 46.70(+1.90) | 55.30(+2.40)

TABLE XV: Object detection single model results on COCO minival (bounding box
AP). The baseline is Mask R-CNN with ResNet-50/101.

Method | backbone | AP | $AP_{50}$ | $AP_{75}$ | $AP_{S}$ | $AP_{M}$ | $AP_{L}$
---|---|---|---|---|---|---|---
Mask R-CNN | ResNet-50 | 34.80 | 56.10 | 36.90 | 16.10 | 37.20 | 52.90
Mask R-CNN | ResNet-50 w B$\boldsymbol{A}^{2}$M | 36.10(+1.30) | 57.80(+1.70) | 38.80(+1.90) | 17.00(+0.90) | 38.70(+1.50) | 52.90(+0.00)
Mask R-CNN | ResNet-101 | 36.50 | 58.10 | 39.10 | 18.40 | 40.20 | 50.40
Mask R-CNN | ResNet-101 w B$\boldsymbol{A}^{2}$M | 37.70(+1.20) | 59.70(+1.60) | 40.90(+1.80) | 18.60(+0.20) | 41.00(+0.80) | 55.90(+5.50)

TABLE XVI: Instance segmentation single-model results on COCO minival (mask
AP). The baseline is Mask R-CNN with ResNet-50/101.

## References

  * [1] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _IEEE Conference on Computer Vision and Pattern Recognition CVPR_ , 2018, pp. 7132–7141.
  * [2] H. Lee, H. Kim, and H. Nam, “SRM : A style-based recalibration module for convolutional neural networks,” _CoRR_ , vol. abs/1903.10829, 2019.
  * [3] J. Hu, L. Shen, S. Albanie, G. Sun, and A. Vedaldi, “Gather-excite: Exploiting feature context in convolutional neural networks,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2018, pp. 9423–9433.
  * [4] X. Li, W. Wang, X. Hu, and J. Yang, “Selective kernel networks,” _CoRR_ , vol. abs/1903.06586, 2019.
  * [5] X. Li, X. Hu, and J. Yang, “Spatial group-wise enhance: Improving semantic feature learning in convolutional networks,” _CoRR_ , vol. abs/1905.09646, 2019.
  * [6] J. Park, S. Woo, J. Lee, and I. S. Kweon, “BAM: bottleneck attention module,” in _Proceedings of the British Machine Vision Conference (BMVC)_ , 2018, p. 147.
  * [7] S. Woo, J. Park, J. Lee, and I. S. Kweon, “CBAM: convolutional block attention module,” in _IEEE European Conference on Computer Vision (ECCV)_ , 2018, pp. 3–19.
  * [8] Y. Cao, J. Xu, S. Lin, F. Wei, and H. Hu, “Gcnet: Non-local networks meet squeeze-excitation networks and beyond,” _CoRR_ , vol. abs/1904.11492, 2019\.
  * [9] I. Bello, B. Zoph, A. Vaswani, J. Shlens, and Q. V. Le, “Attention augmented convolutional networks,” _CoRR_ , vol. abs/1904.09925, 2019.
  * [10] L. v. d. Maaten and G. Hinton, “Visualizing data using t-sne,” _Journal of machine learning research, JMLR_ , pp. 2579–2605, 2008.
  * [11] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 770–778.
  * [12] ——, “Identity mappings in deep residual networks,” in _IEEE European Conference on Computer Vision (ECCV)_ , 2016, pp. 630–645.
  * [13] S. Zagoruyko and N. Komodakis, “Wide residual networks,” in _Proceedings of the British Machine Vision Conference (BMVC)_ , 2016.
  * [14] S. Xie, R. B. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in _IEEE Conference on Computer Vision and Pattern Recognition CVPR_ , 2017, pp. 5987–5995.
  * [15] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 2261–2269.
  * [16] N. Ma, X. Zhang, H. Zheng, and J. Sun, “Shufflenet V2: practical guidelines for efficient CNN architecture design,” in _IEEE European Conference on Computer Vision (ECCV)_ , 2018, pp. 122–138.
  * [17] M. Sandler, A. G. Howard, M. Zhu, A. Zhmoginov, and L. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _IEEE Conference on Computer Vision and Pattern Recognition CVPR_ , 2018, pp. 4510–4520.
  * [18] M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _Proceedings of the 36th International Conference on Machine Learning ICML_ , 2019, pp. 6105–6114.
  * [19] N. Parmar, P. Ramachandran, A. Vaswani, I. Bello, A. Levskaya, and J. Shlens, “Stand-alone self-attention in vision models,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019, pp. 68–80.
  * [20] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _International Conference on Learning Representations, ICLR_ , 2015.
  * [21] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. E. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR_ , 2015\.
  * [22] Y. Chen, H. Fan, B. Xu, Z. Yan, Y. Kalantidis, M. Rohrbach, S. Yan, and J. Feng, “Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution,” _CoRR_ , vol. abs/1904.05049, 2019\.
  * [23] D. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and accurate deep network learning by exponential linear units (elus),” _CoRR_ , vol. abs/1511.07289, 2015.
  * [24] A. Shah, E. Kadam, H. Shah, and S. Shinde, “Deep residual networks with exponential linear unit,” _CoRR_ , vol. abs/1604.04112, 2016.
  * [25] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in _IEEE International Conference on Computer Vision (ICCV)_ , 2015, pp. 1026–1034.
  * [26] F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang, X. Wang, and X. Tang, “Residual attention network for image classification,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR_ , 2017, pp. 6450–6458.
  * [27] X. Wang, R. B. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in _IEEE Conference on Computer Vision and Pattern Recognition,CVPR_ , 2018, pp. 7794–7803.
  * [28] T. Lin, P. Goyal, R. B. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _IEEE International Conference on Computer Vision ICCV_ , 2017, pp. 2999–3007.
  * [29] A. Shrivastava, A. Gupta, and R. B. Girshick, “Training region-based object detectors with online hard example mining,” in _IEEE Conference on Computer Vision and Pattern Recognition CVPR_ , 2016, pp. 761–769.
  * [30] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _Proceedings of the 32nd International Conference on Machine Learning (ICML)_ , 2015, pp. 448–456.
  * [31] H. Zhang, I. J. Goodfellow, D. N. Metaxas, and A. Odena, “Self-attention generative adversarial networks,” in _Proceedings of the 36th International Conference on Machine Learning ICML_ , 2019, pp. 7354–7363.
  * [32] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in Neural Information Processing Systems (NIPS)_ , 2012, pp. 1106–1114.
  * [33] K. He and J. Sun, “Convolutional neural networks at constrained time cost,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR_ , 2015, pp. 5353–5360.
  * [34] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” _Master’s thesis, Department of Computer Science, University of Toronto_ , 2009.
  * [35] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and F. Li, “Imagenet large scale visual recognition challenge,” _International Journal of Computer Vision (IJCV)_ , pp. 211–252, 2015.
  * [36] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017.
  * [37] K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” in _IEEE European Conference on Computer Vision (ECCV)_ , 2016, pp. 630–645.
  * [38] S. Ren, K. He, R. B. Girshick, and J. Sun, “Faster R-CNN: towards real-time object detection with region proposal networks,” _IEEE Trans. Pattern Anal. Mach. Intell._ , pp. 1137–1149, 2017.
  * [39] T. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: common objects in context,” in _IEEE European Conference on Computer Vision (ECCV)_ , 2014, pp. 740–755.
  * [40] K. He, G. Gkioxari, P. Dollár, and R. B. Girshick, “Mask R-CNN,” in _IEEE International Conference on Computer Vision ICCV_ , 2017, pp. 2980–2988.
  * [41] K. Chen, J. Wang, J. Pang, Y. Cao, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Xu, Z. Zhang, D. Cheng, C. Zhu, T. Cheng, Q. Zhao, B. Li, X. Lu, R. Zhu, Y. Wu, J. Dai, J. Wang, J. Shi, W. Ouyang, C. C. Loy, and D. Lin, “MMDetection: Open mmlab detection toolbox and benchmark,” _CoRR_ , vol. abs/1906.07155, 2019.


Olaya, D., Vásquez, J., Maldonado, S., Miranda, J., and Verbeke, W. (2020).
Uplift modeling for preventing student dropout in higher education.
Decision support systems, 134:113320.

[37]
Pearl, J. (2009).
Causality: Models, Reasoning and Inference.
Cambridge University Press.

[38]
Peysakhovich, A. and Lada, A. (2016).
Combining observational and experimental data to find heterogeneous treatment effects.
arXiv preprint arXiv:1611.02385.

[39]
Prentice, R. L. (1989).
Surrogate endpoints in clinical trials: definition and operational criteria.
Statistics in medicine, 8(4):431–440.

[40]
Provost, F. and Fawcett, T. (2013).
Data Science for Business: What you need to know about data mining and data-analytic thinking.
" O'Reilly Media, Inc.".

[41]
Radcliffe, N. (2007).
Using control groups to target on predicted lift: Building and assessing uplift model.
Direct Marketing Analytics Journal, pages 14–21.

[42]
Radcliffe, N. J. and Surry, P. D. (2011).
Real-world uplift modelling with significance-based uplift trees.
White Paper TR-2011-1, Stochastic Solutions.

[43]
Rosenbaum, P. R. and Rubin, D. B. (1983).
The central role of the propensity score in observational studies for causal effects.
Biometrika, 70(1):41–55.

[44]
Rosenman, E., Basse, G., Owen, A., and Baiocchi, M. (2020).
Combining observational and experimental datasets using shrinkage estimators.
arXiv preprint arXiv:2002.06708.

[45]
Rubin, D. B. (1974).
Estimating causal effects of treatments in randomized and nonrandomized studies.
Journal of educational Psychology, 66(5):688.

[46]
Stitelman, O., Dalessandro, B., Perlich, C., and Provost, F. (2011).
Estimating the effect of online display advertising on browser conversion.
Data Mining and Audience Intelligence for Advertising (ADKDD 2011), 8.

[47]
Train, K. E. (2009).
Discrete choice methods with simulation.
Cambridge university press.

[48]
VanderWeele, T. J. (2013).
Surrogate measures and consistent surrogates.
Biometrics, 69(3):561–565.

[49]
Wooldridge, J. M. (2015).
Introductory econometrics: A modern approach.
Nelson Education.

[50]
Yadlowsky, S., Fleming, S., Shah, N., Brunskill, E., and Wager, S. (2021).
Evaluating treatment prioritization rules via rank-weighted average treatment effects.
arXiv preprint arXiv:2111.07966.

[51]
Yang, J., Eckles, D., Dhillon, P., and Aral, S. (2020).
Targeting for long-term outcomes.
arXiv preprint arXiv:2010.15835.

[52]
Zhang, B., Tsiatis, A. A., Davidian, M., Zhang, M., and Laber, E. (2012).
Estimating optimal treatment regimes from a classification perspective.
Stat, 1(1):103–114.

[53]
Zhao, Y., Zeng, D., Rush, A. J., and Kosorok, M. R. (2012).
Estimating individualized treatment rules using outcome weighted learning.
Journal of the American Statistical Association, 107(499):1106–1118.