# PVT v2: Improved Baselines with Pyramid Vision Transformer

Wenhai Wang1,2, Enze Xie3, Xiang Li4, Deng-Ping Fan5,
Kaitao Song4, Ding Liang6, Tong Lu2, Ping Luo3, Ling Shao5
1Shanghai AI Laboratory 2Nanjing University 3The University of Hong Kong
4Nanjing University of Science and Technology 5IIAI 6SenseTime Research
$\tt<EMAIL_ADDRESS>
###### Abstract

Transformer recently has presented encouraging progress in computer vision. In
this work, we present new baselines by improving the original Pyramid Vision
Transformer (PVT v1) by adding three designs, including (1) linear complexity
attention layer, (2) overlapping patch embedding, and (3) convolutional feed-
forward network. With these modifications, PVT v2 reduces the computational
complexity of PVT v1 to linear and achieves significant improvements on
fundamental vision tasks such as classification, detection, and segmentation.
Notably, the proposed PVT v2 achieves comparable or better performances than
recent works such as Swin Transformer. We hope this work will facilitate
state-of-the-art Transformer researches in computer vision. Code is available
at https://github.com/whai362/PVT.

##  1 Introduction

Recent studies on vision Transformer are converging on the backbone network
[8, 31, 33, 34, 23, 36, 10, 5] designed for downstream vision tasks, such as
image classification, object detection, instance and semantic segmentation. To
date, there have been some promising results. For example, Vision Transformer
(ViT) [8] first proves that a pure Transformer can archive state-of-the-art
performance in image classification. Pyramid Vision Transformer (PVT v1) [33]
shows that a pure Transformer backbone can also surpass CNN counterparts in
dense prediction tasks such as detection and segmentation tasks [22, 41].
After that, Swin Transformer [23], CoaT [36], LeViT [10], and Twins [5]
further improve the classification, detection, and segmentation performance
with Transformer backbones.

This work aims to establish stronger and more feasible baselines built on the
PVT v1 framework. We report that three design improvements, namely (1) linear
complexity attention layer, (2) overlapping patch embedding, and (3)
convolutional feed-forward network are orthogonal to the PVT v1 framework, and
when used with PVT v1, they can bring better image classification, object
detection, instance and semantic segmentation performance. The improved
framework is termed as PVT v2. Specifically, PVT v2-B5111 PVT v2 has 6
different size variants, from B0 to B5 according to the parameter number.
yields 83.8% top-1 error on ImageNet, which is better than Swin-B [23] and
Twins-SVT-L [5], while our model has fewer parameters and GFLOPs. Moreover,
GFL [19] with PVT-B2 archives 50.2 AP on COCO val2017, 2.6 AP higher than the
one with Swin-T [23], 5.7 AP higher than the one with ResNet50 [13]. We hope
these improved baselines will provide a reference for future research in
vision Transformer.

##  2 Related Work

We mainly discuss transformer backbones related to this work. ViT [8] treats
each image as a sequence of tokens (patches) with a fixed length, and then
feeds them to multiple Transformer layers to perform classification. It is the
first work to prove that a pure Transformer can also archive state-of-the-art
performance in image classification when training data is sufficient (_e.g._ ,
ImageNet-22k [7], JFT-300M). DeiT [31] further explores a data-efficient
training strategy and a distillation approach for ViT.

To improve image classification performance, recent methods make tailored
changes to ViT. T2T ViT [37] concatenates tokens within an overlapping sliding
window into one token progressively. TNT [11] utilizes inner and outer
Transformer blocks to generate pixel and patch embeddings respectively. CPVT
[6] replaces the fixed size position embedding in ViT with conditional
position encodings, making it easier to process images of arbitrary
resolution. CrossViT [2] processes image patches of different sizes via a
dual-branch Transformer. LocalViT [20] incorporates depth-wise convolution
into vision Transformers to improve the local continuity of features.

To adapt to dense prediction tasks such as object detection, instance and
semantic segmentation, there are also some methods [33, 23, 34, 36, 10, 5] to
introduce the pyramid structure in CNNs to the design of Transformer
backbones. PVT v1 is the first pyramid structure Transformer, which presents a
hierarchical Transformer with four stages, showing that a pure Transformer
backbone can be as versatile as CNN counterparts and performs better in
detection and segmentation tasks. After that, some improvements [23, 34, 36,
10, 5] are made to enhance the local continuity of features and to remove
fixed size position embedding. For example, Swin Transformer [23] replaces
fixed size position embedding with relative position biases, and restricts
self-attention within shifted windows. CvT [34], CoaT [36], and LeViT [10]
introduce convolution-like operations into vision Transformers. Twins [5]
combines local attention and global attention mechanisms to obtain stronger
feature representation.

##  3 Methodology

###  3.1 Limitations in PVT v1

There are three main limitations in PVT v1 [33] as follows: (1) Similar to ViT
[8], when processing high-resolution input (_e.g._ , shorter side being 800
pixels), the computational complexity of PVT v1 is relatively large. (2) PVT
v1 [33] treats an image as a sequence of non-overlapping patches, which loses
the local continuity of the image to a certain extent; (3) The position
encoding in PVT v1 is fixed-size, which is inflexible for process images of
arbitrary size. These problems limit the performance of PVT v1 on vision
tasks.

To address these issues, we propose PVT v2, which improves PVT v1 through
three designs, which are listed in Sec 3.2, 3.3, and 3.4.

Figure 1: Comparison of SRA in PVT v1 and linear SRA in PVT v2.

###  3.2 Linear Spatial Reduction Attention

First, to reduce the high computational cost caused by attention operations,
we propose linear spatial reduction attention (SRA) layer as illustrated in
Fig. 1. Different from SRA [33] which uses convolutions for spatial reduction,
linear SRA uses average pooling to reduce the spatial dimension (_i.e._ ,
$h\times w$) to a fixed size (_i.e._ , $P\times P$) before the attention
operation. So linear SRA enjoys linear computational and memory costs like a
convolutional layer. Specifically, given an input of size $h\times w\times c$,
the complexity of SRA and linear SRA are:

$\Omega({\rm SRA})=\frac{2h^{2}w^{2}c}{R^{2}}+hwc^{2}R^{2},$ (1) $\Omega({\rm
Linear\ SRA})=2hwP^{2}c,$ (2)

where $R$ is the spatial reduction ratio of SRA [33]. $P$ is the pooling size
of linear SRA, which is set to 7.

Figure 2: Two improvements in PVT v2. (1) Overlapping Patch Embedding. (2)
Convolutional Feed Forward Network.

###  3.3 Overlapping Patch Embedding

Second, to model the local continuity information, we utilize overlapping
patch embedding to tokenize images. As shown in Fig. 2(a), we enlarge the
patch window, making adjacent windows overlap by half of the area, and pad the
feature map with zeros to keep the resolution. In this work, we use
convolution with zero paddings to implement overlapping patch embedding.
Specifically, given an input of size $h\times w\times c$, we feed it to a
convolution with the stride of $S$, the kernel size of $2S-1$, the padding
size of $S-1$, and the kernel number of $c^{\prime}$. The output size is
$\frac{h}{S}\times\frac{w}{S}\times C^{\prime}$.

| Output Size | Layer Name | Pyramid Vision Transformer v2
---|---|---|---
| B0 | B1 | B2 | B2-Li | B3 | B4 | B5
Stage 1 |  $\frac{H}{4}\times\frac{W}{4}$ |  Overlapping Patch Embedding | $S_{1}=4$
$C_{1}=32$ | $C_{1}=64$
| Transformer
---
Encoder
$\begin{matrix}R_{1}=8\\\ N_{1}=1\\\ E_{1}=8\\\ L_{1}=2\\\ \end{matrix}$ | $\begin{matrix}R_{1}=8\\\ N_{1}=1\\\ E_{1}=8\\\ L_{1}=2\\\ \end{matrix}$ | $\begin{matrix}R_{1}=8\\\ N_{1}=1\\\ E_{1}=8\\\ L_{1}=3\\\ \end{matrix}$ | $\begin{matrix}P_{1}=7\\\ N_{1}=1\\\ E_{1}=8\\\ L_{1}=3\\\ \end{matrix}$ | $\begin{matrix}R_{1}=8\\\ N_{1}=1\\\ E_{1}=8\\\ L_{1}=3\\\ \end{matrix}$ | $\begin{matrix}R_{1}=8\\\ N_{1}=1\\\ E_{1}=8\\\ L_{1}=3\\\ \end{matrix}$ | $\begin{matrix}R_{1}=8\\\ N_{1}=1\\\ E_{1}=4\\\ L_{1}=3\\\ \end{matrix}$
Stage 2 |  $\frac{H}{8}\times\frac{W}{8}$ |  Overlapping Patch Embedding | $S_{2}=2$
$C_{2}=64$ | $C_{2}=128$
| Transformer
---
Encoder
$\begin{matrix}R_{2}=4\\\ N_{2}=2\\\ E_{2}=8\\\ L_{2}=2\\\ \end{matrix}$ | $\begin{matrix}R_{2}=4\\\ N_{2}=2\\\ E_{2}=8\\\ L_{2}=2\\\ \end{matrix}$ | $\begin{matrix}R_{2}=4\\\ N_{2}=2\\\ E_{2}=8\\\ L_{2}=3\\\ \end{matrix}$ | $\begin{matrix}P_{2}=7\\\ N_{2}=2\\\ E_{2}=8\\\ L_{2}=3\\\ \end{matrix}$ | $\begin{matrix}R_{2}=4\\\ N_{2}=2\\\ E_{2}=8\\\ L_{2}=3\\\ \end{matrix}$ | $\begin{matrix}R_{2}=4\\\ N_{2}=2\\\ E_{2}=8\\\ L_{2}=8\\\ \end{matrix}$ | $\begin{matrix}R_{2}=4\\\ N_{2}=2\\\ E_{2}=4\\\ L_{2}=6\\\ \end{matrix}$
Stage 3 |  $\frac{H}{16}\times\frac{W}{16}$ |  Overlapping Patch Embedding | $S_{3}=2$
$C_{3}=160$ | $C_{3}=320$
| Transformer
---
Encoder
$\begin{matrix}R_{3}=2\\\ N_{3}=5\\\ E_{3}=4\\\ L_{3}=2\\\ \end{matrix}$ | $\begin{matrix}R_{3}=2\\\ N_{3}=5\\\ E_{3}=4\\\ L_{3}=2\\\ \end{matrix}$ | $\begin{matrix}R_{3}=2\\\ N_{3}=5\\\ E_{3}=4\\\ L_{3}=6\\\ \end{matrix}$ | $\begin{matrix}P_{3}=7\\\ N_{3}=5\\\ E_{3}=4\\\ L_{3}=6\\\ \end{matrix}$ | $\begin{matrix}R_{3}=2\\\ N_{3}=5\\\ E_{3}=4\\\ L_{3}=18\\\ \end{matrix}$ | $\begin{matrix}R_{3}=2\\\ N_{3}=5\\\ E_{3}=4\\\ L_{3}=27\\\ \end{matrix}$ | $\begin{matrix}R_{3}=2\\\ N_{3}=5\\\ E_{3}=4\\\ L_{3}=40\\\ \end{matrix}$
Stage 4 |  $\frac{H}{32}\times\frac{W}{32}$ |  Overlapping Patch Embedding | $S_{4}=2$
$C_{4}=256$ | $C_{4}=512$
| Transformer
---
Encoder
$\begin{matrix}R_{4}=1\\\ N_{4}=8\\\ E_{4}=4\\\ L_{4}=2\\\ \end{matrix}$ | $\begin{matrix}R_{4}=1\\\ N_{4}=8\\\ E_{4}=4\\\ L_{4}=2\\\ \end{matrix}$ | $\begin{matrix}R_{4}=1\\\ N_{4}=8\\\ E_{4}=4\\\ L_{4}=3\\\ \end{matrix}$ | $\begin{matrix}P_{4}=7\\\ N_{4}=8\\\ E_{4}=4\\\ L_{4}=3\\\ \end{matrix}$ | $\begin{matrix}R_{4}=1\\\ N_{4}=8\\\ E_{4}=4\\\ L_{4}=3\\\ \end{matrix}$ | $\begin{matrix}R_{4}=1\\\ N_{4}=8\\\ E_{4}=4\\\ L_{4}=3\\\ \end{matrix}$ | $\begin{matrix}R_{4}=1\\\ N_{4}=8\\\ E_{4}=4\\\ L_{4}=3\\\ \end{matrix}$
Table 1: Detailed settings of PVT v2 series. “-Li” denotes PVT v2 with linear
SRA.

###  3.4 Convolutional Feed-Forward

Third, inspired by [17, 6, 20], we remove the fixed-size position encoding
[8], and introduce zero padding position encoding into PVT. As shown in Fig.
2(b), we add a $3\times 3$ depth-wise convolution [16] with the padding size
of 1 between the first fully-connected (FC) layer and GELU [15] in feed-
forward networks.

###  3.5 Details of PVT v2 Series

We scale up PVT v2 from B0 to B5 By changing the hyper-parameters. which are
listed as follows:

  * •

$S_{i}$: the stride of the overlapping patch embedding in Stage $i$;

  * •

$C_{i}$: the channel number of the output of Stage $i$;

  * •

$L_{i}$: the number of encoder layers in Stage $i$;

  * •

$R_{i}$: the reduction ratio of the SRA in Stage $i$;

  * •

$P_{i}$: the adaptive average pooling size of the linear SRA in Stage $i$;

  * •

$N_{i}$: the head number of the Efficient Self-Attention in Stage $i$;

  * •

$E_{i}$: the expansion ratio of the feed-forward layer [32] in Stage $i$;

Tab. 1 shows the detailed information of PVT v2 series. Our design follows the
principles of ResNet [14]. (1) the channel dimension increase while the
spatial resolution shrink with the layer goes deeper. (2) Stage 3 is assigned
to most of the computation cost.

###  3.6 Advantages of PVT v2

Combining these improvements, PVT v2 can (1) obtain more local continuity of
images and feature maps; (2) process variable-resolution input more flexibly;
(3) enjoy the same linear complexity as CNN.

Method | #Param (M) | GFLOPs | Top-1 Acc (%)
---|---|---|---
PVTv2-B0 (ours) | 3.4 | 0.6 | 70.5
ResNet18 [14] | 11.7 | 1.8 | 69.8
DeiT-Tiny/16 [31] | 5.7 | 1.3 | 72.2
PVTv1-Tiny [33] | 13.2 | 1.9 | 75.1
PVTv2-B1 (ours) | 13.1 | 2.1 | 78.7
ResNet50 [14] | 25.6 | 4.1 | 76.1
ResNeXt50-32x4d [35] | 25.0 | 4.3 | 77.6
RegNetY-4G [26] | 21.0 | 4.0 | 80.0
DeiT-Small/16 [31] | 22.1 | 4.6 | 79.9
T2T-ViTt-14 [37] | 22.0 | 6.1 | 80.7
PVTv1-Small [33] | 24.5 | 3.8 | 79.8
TNT-S [11] | 23.8 | 5.2 | 81.3
Swin-T [23] | 29.0 | 4.5 | 81.3
CvT-13 [34] | 20.0 | 4.5 | 81.6
CoaT-Lite Small [36] | 20.0 | 4.0 | 81.9
Twins-SVT-S [5] | 24.0 | 2.8 | 81.7
PVTv2-B2-Li (ours) | 22.6 | 3.9 | 82.1
PVTv2-B2 (ours) | 25.4 | 4.0 | 82.0
ResNet101 [14] | 44.7 | 7.9 | 77.4
ResNeXt101-32x4d [35] | 44.2 | 8.0 | 78.8
RegNetY-8G [26] | 39.0 | 8.0 | 81.7
T2T-ViTt-19 [37] | 39.0 | 9.8 | 81.4
PVTv1-Medium [33] | 44.2 | 6.7 | 81.2
CvT-21 [34] | 32.0 | 7.1 | 82.5
PVTv2-B3 (ours) | 45.2 | 6.9 | 83.2
ResNet152 [14] | 60.2 | 11.6 | 78.3
T2T-ViTt-24 [37] | 64.0 | 15.0 | 82.2
PVTv1-Large [33] | 61.4 | 9.8 | 81.7
TNT-B [11] | 66.0 | 14.1 | 82.8
Swin-S [23] | 50.0 | 8.7 | 83.0
Twins-SVT-B [5] | 56.0 | 8.3 | 83.2
PVTv2-B4 (ours) | 62.6 | 10.1 | 83.6
ResNeXt101-64x4d [35] | 83.5 | 15.6 | 79.6
RegNetY-16G [26] | 84.0 | 16.0 | 82.9
ViT-Base/16 [8] | 86.6 | 17.6 | 81.8
DeiT-Base/16 [31] | 86.6 | 17.6 | 81.8
Swin-B [23] | 88.0 | 15.4 | 83.3
Twins-SVT-L [5] | 99.2 | 14.8 | 83.7
PVTv2-B5 (ours) | 82.0 | 11.8 | 83.8

Table 2: Image classification performance on the ImageNet validation set.
“#Param” refers to the number of parameters. “GFLOPs” is calculated under the
input scale of $224\times 224$. “*” indicates the performance of the method
trained under the strategy of its original paper. “-Li” denotes PVT v2 with
linear SRA.

##  4 Experiment

###  4.1 Image Classification

Settings. Image classification experiments are performed on the ImageNet-1K
dataset [27], which comprises 1.28 million training images and 50K validation
images from 1,000 categories. All models are trained on the training set for
fair comparison and report the top-1 error on the validation set. We follow
DeiT [31] and apply random cropping, random horizontal flipping [29], label-
smoothing regularization [30], mixup [38], and random erasing [40] as data
augmentations. During training, we employ AdamW [25] with a momentum of 0.9, a
mini-batch size of 128, and a weight decay of $5\times 10^{-2}$ to optimize
models. The initial learning rate is set to $1\times 10^{-3}$ and decreases
following the cosine schedule [24]. All models are trained for 300 epochs from
scratch on 8 V100 GPUs. We apply a center crop on the validation set to
benchmark, where a 224$\times$ 224 patch is cropped to evaluate the
classification accuracy.

Results. In Tab. 2, we see that PVT v2 is the state-of-the-art method on
ImageNet-1K classification. Compared to PVT, PVT v2 has similar flops and
parameters, but the image classification accuracy is greatly improved. For
example, PVT v2-B1 is 3.6% higher than PVT v1-Tiny, and PVT v2-B4 is 1.9%
higher than PVT-Large.

Compared to other recent counterparts, PVT v2 series also has large advantages
in terms of accuracy and model size. For example, PVT v2-B5 achieves 83.8%
ImageNet top-1 accuracy, which is 0.5% higher than Swin Transformer [23] and
Twins [5], while our parameters and FLOPS are fewer.

###  4.2 Object Detection

Settings. Object detection experiments are conducted on the challenging COCO
benchmark [22]. All models are trained on COCO train2017 (118k images) and
evaluated on val2017 (5k images). We verify the effectiveness of PVT v2
backbones on top of mainstream detectors, including RetinaNet [21], Mask R-CNN
[12], Cascade Mask R-CNN [1], ATSS [39], GFL [19], and Sparse R-CNN [28].
Before training, we use the weights pre-trained on ImageNet to initialize the
backbone and Xavier [9] to initialize the newly added layers. We train all the
models with batch size 16 on 8 V100 GPUs, and adopt AdamW [25] with an initial
learning rate of $1\times 10^{-4}$ as optimizer. Following common practices
[21, 12, 3], we adopt 1$\times$ or 3$\times$ training schedule (_i.e._ , 12 or
36 epochs) to train all detection models. The training image is resized to
have a shorter side of 800 pixels, while the longer side does not exceed 1,333
pixels. When using the 3$\times$ training schedule, we randomly resize the
shorter side of the input image within the range of $[640,800]$. In the
testing phase, the shorter side of the input image is fixed to 800 pixels.

Results. As reported in Tab. 3, PVT v2 significantly outperforms PVT v1 on
both one-stage and two-stage object detectors with similar model size. For
example, PVT v2-B4 archive 46.1 AP on top of RetinaNet [21], and 47.5 APb on
top of Mask R-CNN [12], surpassing the models with PVT v1 by 3.5 AP and 4.6
APb, respectively. We present some qualitative object detection and instance
segmentation results on COCO val2017 [22] in Fig. 3, which also shows the good
performance of our models.

For a fair comparison between PVT v2 and Swin Transformer [23], we keep all
settings the same, including ImageNet-1K pre-training and COCO fine-tuning
strategies. We evaluate Swin Transformer and PVT v2 on four state-of-the-arts
detectors, including Cascade R-CNN [1], ATSS [39], GFL [19], and Sparse R-CNN
[28]. We see PVT v2 obtain much better AP than Swin Transformer among all the
detectors, showing its better feature representation ability. For example, on
ATSS, PVT v2 has similar parameters and flops compared to Swin-T, but PVT v2
achieves 49.9 AP, which is 2.7 higher than Swin-T. Our PVT v2-Li can largely
reduce the computation from 258 to 194 GFLOPs, while only sacrificing a little
performance.

Backbone |  RetinaNet 1$\times$ |  Mask R-CNN 1$\times$
---|---|---
#P (M) | AP |  AP50 |  AP75 |  APS |  APM |  APL | #P (M) |  APb |  AP${}_{50}^{\rm b}$ |  AP${}_{75}^{\rm b}$ |  APm |  AP${}_{50}^{\rm m}$ |  AP${}_{75}^{\rm m}$
PVTv2-B0 | 13.0 | 37.2 | 57.2 | 39.5 | 23.1 | 40.4 | 49.7 | 23.5 | 38.2 | 60.5 | 40.7 | 36.2 | 57.8 | 38.6
ResNet18 [14] | 21.3 | 31.8 | 49.6 | 33.6 | 16.3 | 34.3 | 43.2 | 31.2 | 34.0 | 54.0 | 36.7 | 31.2 | 51.0 | 32.7
PVTv1-Tiny [33] | 23.0 | 36.7 | 56.9 | 38.9 | 22.6 | 38.8 | 50.0 | 32.9 | 36.7 | 59.2 | 39.3 | 35.1 | 56.7 | 37.3
PVTv2-B1 (ours) | 23.8 | 41.2 | 61.9 | 43.9 | 25.4 | 44.5 | 54.3 | 33.7 | 41.8 | 64.3 | 45.9 | 38.8 | 61.2 | 41.6
ResNet50 [14] | 37.7 | 36.3 | 55.3 | 38.6 | 19.3 | 40.0 | 48.8 | 44.2 | 38.0 | 58.6 | 41.4 | 34.4 | 55.1 | 36.7
PVTv1-Small [33] | 34.2 | 40.4 | 61.3 | 43.0 | 25.0 | 42.9 | 55.7 | 44.1 | 40.4 | 62.9 | 43.8 | 37.8 | 60.1 | 40.3
PVTv2-B2-Li (ours) | 32.3 | 43.6 | 64.7 | 46.8 | 28.3 | 47.6 | 57.4 | 42.2 | 44.1 | 66.3 | 48.4 | 40.5 | 63.2 | 43.6
PVTv2-B2 (ours) | 35.1 | 44.6 | 65.6 | 47.6 | 27.4 | 48.8 | 58.6 | 45.0 | 45.3 | 67.1 | 49.6 | 41.2 | 64.2 | 44.4
ResNet101 [14] | 56.7 | 38.5 | 57.8 | 41.2 | 21.4 | 42.6 | 51.1 | 63.2 | 40.4 | 61.1 | 44.2 | 36.4 | 57.7 | 38.8
ResNeXt101-32x4d [35] | 56.4 | 39.9 | 59.6 | 42.7 | 22.3 | 44.2 | 52.5 | 62.8 | 41.9 | 62.5 | 45.9 | 37.5 | 59.4 | 40.2
PVTv1-Medium [33] | 53.9 | 41.9 | 63.1 | 44.3 | 25.0 | 44.9 | 57.6 | 63.9 | 42.0 | 64.4 | 45.6 | 39.0 | 61.6 | 42.1
PVTv2-B3 (ours) | 55.0 | 45.9 | 66.8 | 49.3 | 28.6 | 49.8 | 61.4 | 64.9 | 47.0 | 68.1 | 51.7 | 42.5 | 65.7 | 45.7
PVTv1-Large [33] | 71.1 | 42.6 | 63.7 | 45.4 | 25.8 | 46.0 | 58.4 | 81.0 | 42.9 | 65.0 | 46.6 | 39.5 | 61.9 | 42.5
PVTv2-B4 (ours) | 72.3 | 46.1 | 66.9 | 49.2 | 28.4 | 50.0 | 62.2 | 82.2 | 47.5 | 68.7 | 52.0 | 42.7 | 66.1 | 46.1
ResNeXt101-64x4d [35] | 95.5 | 41.0 | 60.9 | 44.0 | 23.9 | 45.2 | 54.0 | 101.9 | 42.8 | 63.8 | 47.3 | 38.4 | 60.6 | 41.3
PVTv2-B5 (ours) | 91.7 | 46.2 | 67.1 | 49.5 | 28.5 | 50.0 | 62.5 | 101.6 | 47.4 | 68.6 | 51.9 | 42.5 | 65.7 | 46.0
Table 3: Object detection and instance segmentation on COCO val2017. “#P” refers to parameter number. APb and APm denote bounding box AP and mask AP, respectively. “-Li” denotes PVT v2 with linear SRA.  Backbone | Method |  AP${}^{\text{b}}$ |  AP${}^{\text{b}}_{\text{50}}$ |  AP${}^{\text{b}}_{\text{75}}$ | #P (M) | GFLOPs
---|---|---|---|---|---|---
ResNet50 [14] |  Cascade Mask R-CNN [1] | 46.3 | 64.3 | 50.5 | 82 | 739
Swin-T [23] | 50.5 | 69.3 | 54.9 | 86 | 745
PVTv2-B2-Li (ours) | 50.9 | 69.5 | 55.2 | 80 | 725
PVTv2-B2 (ours) | 51.1 | 69.8 | 55.3 | 83 | 788
ResNet50 [14] | ATSS [39] | 43.5 | 61.9 | 47.0 | 32 | 205
Swin-T [23] | 47.2 | 66.5 | 51.3 | 36 | 215
PVTv2-B2-Li (ours) | 48.9 | 68.1 | 53.4 | 30 | 194
PVTv2-B2 (ours) | 49.9 | 69.1 | 54.1 | 33 | 258
ResNet50 [14] | GFL [19] | 44.5 | 63.0 | 48.3 | 32 | 208
Swin-T [23] | 47.6 | 66.8 | 51.7 | 36 | 215
PVTv2-B2-Li (ours) | 49.2 | 68.2 | 53.7 | 30 | 197
PVTv2-B2 (ours) | 50.2 | 69.4 | 54.7 | 33 | 261
ResNet50 [14] |  Sparse R-CNN [28] | 44.5 | 63.4 | 48.2 | 106 | 166
Swin-T [23] | 47.9 | 67.3 | 52.3 | 110 | 172
PVTv2-B2-Li (ours) | 48.9 | 68.3 | 53.4 | 104 | 151
PVTv2-B2 (ours) | 50.1 | 69.5 | 54.9 | 107 | 215
Table 4: Compare with Swin Transformer on object detection. “APb” denotes
bounding box AP. “#P” refers to parameter number. “GFLOPs” is calculated under
the input scale of $1280\times 800$. “-Li” denotes PVT v2 with linear SRA.

###  4.3 Semantic Segmentation

Backbone | Semantic FPN
---|---
#Param (M) | GFLOPs | mIoU (%)
PVTv2-B0 (ours) | 7.6 | 25.0 | 37.2
ResNet18 [14] | 15.5 | 32.2 | 32.9
PVTv1-Tiny [33] | 17.0 | 33.2 | 35.7
PVTv2-B1 (ours) | 17.8 | 34.2 | 42.5
ResNet50 [14] | 28.5 | 45.6 | 36.7
PVTv1-Small [33] | 28.2 | 44.5 | 39.8
PVTv2-B2-Li (ours) | 26.3 | 41.0 | 45.1
PVTv2-B2 (ours) | 29.1 | 45.8 | 45.2
ResNet101 [14] | 47.5 | 65.1 | 38.8
ResNeXt101-32x4d [35] | 47.1 | 64.7 | 39.7
PVTv1-Medium [33] | 48.0 | 61.0 | 41.6
PVTv2-B3 (ours) | 49.0 | 62.4 | 47.3
PVTv1-Large [33] | 65.1 | 79.6 | 42.1
PVTv2-B4 (ours) | 66.3 | 81.3 | 47.9
ResNeXt101-64x4d [35] | 86.4 | 103.9 | 40.2
PVTv2-B5 (ours) | 85.7 | 91.1 | 48.7
Table 5: Semantic segmentation performance of different backbones on the
ADE20K validation set. “GFLOPs” is calculated under the input scale of
$512\times 512$. “-Li” denotes PVT v2 with linear SRA.

Settings. Following PVT v1 [33], we choose ADE20K [41] to benchmark the
performance of semantic segmentation. For a fair comparison, we test the
performance of PVT v2 backbones by applying it to Semantic FPN [18]. In the
training phase, the backbone is initialized with the weights pre-trained on
ImageNet [7], and the newly added layers are initialized with Xavier [9]. We
optimize our models using AdamW [25] with an initial learning rate of 1e-4.
Following common practices [18, 4], we train our models for 40k iterations
with a batch size of 16 on 4 V100 GPUs. The learning rate is decayed following
the polynomial decay schedule with a power of 0.9. We randomly resize and crop
the image to $512\times 512$ for training, and rescale to have a shorter side
of 512 pixels during testing.

Figure 3: Qualitative results of object detection and instance segmentation on
COCO val2017 [22], and semantic segmentation on ADE20K [41]. The results (from
left to right) are generated by PVT v2-B2-based RetinaNet [21], Mask R-CNN
[12], and Semantic FPN [18], respectively.

Results. As shown in Tab. 5, when using Semantic FPN [18] for semantic
segmentation, PVT v2 consistently outperforms PVT v1 [33] and other
counterparts. For example, with almost the same number of parameters and
GFLOPs, PVT v2-B1/B2/B3/B4 are at least 5.3% higher than PVT
v1-Tiny/Small/Medium/Large. Moreover, although the GFLOPs of PVT-Large are 12%
lower than those of ResNeXt101-64x4d, the mIoU is still 8.5 points higher
(48.7 vs 40.2). In Fig. 3, we also visualize some qualitative semantic
segmentation results on ADE20K [41]. These results demonstrate that PVT v2
backbones can extract powerful features for semantic segmentation, benefiting
from the improved designs.

###  4.4 Ablation Study

####  4.4.1 Model Analysis

Ablation experiments of PVT v2 is reported in Tab. 6. We see that all three
designs can improve the model in terms of performance, parameter number, or
computation overhead.

Overlapping patch embedding (OPE) is important. Comparing #1 and #2 in Tab. 6,
the model with OPE obtains better top-1 accuracy (81.1% _vs_. 79.8%) on
ImageNet and better AP (42.2% _vs_. 40.4%) on COCO than the one with original
patch embedding (PE) [8]. OPE is effective because it can model the local
continuity of images and feature map via the overlapping sliding window.

Convolutional feed-forward network (CFFN) matters. Compared to original feed-
forward network (FFN) [8], our CFFN contains a zero-padding convolutional
layer. which can capture the local continuity of the input tensor. In
addition, due to the positional information introduced by zero-padding in OPE
and CFFN, we can remove the fixed-size positional embeddings used in PVT v1,
making the model flexible to handle variable resolution inputs. As reported in
#2 and #3 in Tab. 6, CFFN brings 0.9 points improvement on ImageNet (82.0%
_vs_. 81.1%) and 2.4 points improvement on COCO, which demonstrates its
effectiveness.

Linear SRA (LSRA) contributes to a better model. As reported in #3 and #4 in
Tab. 6, compared to SRA [33], our LSRA significantly reduces the computation
overhead (GFLOPs) of the model by 22%, while keeping a comparable top-1
accuracy on ImageNet (82.1% _vs_. 82.0%), and only 1 point lower AP on COCO
(43.6 _vs_. 44.6). These results show the low computational cost and good
effect of LSRA.

# | Setting |  Top-1 Acc (%) | RetinaNet 1x
---|---|---|---
#P (M) | GFLOPs | AP
1 |  PVTv1-Small [33] | 79.8 | 34.2 | 285.8 | 40.4
2 | \+ OPE | 81.1 | 34.9 | 288.6 | 42.2
3 | ++ CFFN (PVTv2-B2) | 82.0 | 35.1 | 290.7 | 44.6
4 | +++ LSRA (PVTv2-B2-Li) | 82.1 | 32.3 | 227.4 | 43.6
Table 6: Ablation experiments of PVT v2. “OPE”, “CFFN”, and “LSRA” represent
overlapping patch embedding, convolutional feed-forward network, and linear
SRA, respectively.

####  4.4.2 Computation Overhead Analysis

As shown in Figure 4, with increasing input scale, the GFLOPs growth rate of
the proposed PVT v2-B2-Li is much lower than that of PVT v1-Small [33], and is
similar to that of ResNet-50 [13]. This result proves that our PVT v2-Li
successfully addresses the high computational overhead problem caused by the
attention layer.

Figure 4: Models’ GFLOPs under different input scales. The growth rate of
GFLOPs: ViT-Small/16 [8]$>$ViT-Small/32 [8]$>$PVT v1-Small [33]$>$ResNet50
[14]$>$PVT v2-B2-Li (ours).

##  5 Conclusion

We study the limitations of Pyramid Vision Transformer (PVT v1) and improve it
with three designs, which are overlapping patch embedding, convolutional feed-
forward network, and linear spatial reduction attention layer. Extensive
experiments on different tasks, such as image classification, object
detection, and semantic segmentation demonstrate that the proposed PVT v2 is
stronger than its predecessor PVT v1 and other state-of-the-art transformer-
based backbones, under comparable numbers of parameters. We hope these
improved baselines will provide a reference for future research in vision
Transformer.

## References

  * [1] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: Delving into high quality object detection.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2018.
  * [2] Chun-Fu Chen, Quanfu Fan, and Rameswar Panda.  Crossvit: Cross-attention multi-scale vision transformer for image classification.  arXiv preprint arXiv:2103.14899, 2021.
  * [3] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al.  Mmdetection: Open mmlab detection toolbox and benchmark.  arXiv preprint arXiv:1906.07155, 2019.
  * [4] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE Trans. Pattern Anal. Mach. Intell., 2017.
  * [5] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen.  Twins: Revisiting the design of spatial attention in vision transformers.  arXiv preprint arXiv:2104.13840, 2021.
  * [6] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen.  Conditional positional encodings for vision transformers.  arXiv preprint arXiv:2102.10882, 2021.
  * [7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2009.
  * [8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  Proc. Int. Conf. Learn. Representations, 2021.
  * [9] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In Proc. Int. Conf. Artificial Intell. & Stat., 2010.
  * [10] Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze.  Levit: a vision transformer in convnet’s clothing for faster inference.  In Proc. IEEE Int. Conf. Comp. Vis., 2021.
  * [11] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang.  Transformer in transformer.  arXiv preprint arXiv:2103.00112, 2021.
  * [12] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proc. IEEE Int. Conf. Comp. Vis., 2017.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In Proc. IEEE Int. Conf. Comp. Vis., 2015.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2016.
  * [15] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  arXiv preprint arXiv:1606.08415, 2016.
  * [16] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [17] Md. Amirul Islam, Sen Jia, and Neil D. B. Bruce.  How much position information do convolutional neural networks encode?  In Proc. Int. Conf. Learn. Representations, 2020.
  * [18] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár.  Panoptic feature pyramid networks.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2019.
  * [19] Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang.  Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection.  In Proc. Advances in Neural Inf. Process. Syst., 2020.
  * [20] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool.  Localvit: Bringing locality to vision transformers.  arXiv preprint arXiv:2104.05707, 2021.
  * [21] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proc. IEEE Int. Conf. Comp. Vis., 2017.
  * [22] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Proc. Eur. Conf. Comp. Vis., 2014.
  * [23] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  arXiv preprint arXiv:2103.14030, 2021.
  * [24] Ilya Loshchilov and Frank Hutter.  SGDR: stochastic gradient descent with warm restarts.  In Proc. Int. Conf. Learn. Representations, 2017.
  * [25] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In Proc. Int. Conf. Learn. Representations, 2019.
  * [26] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár.  Designing network design spaces.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2020.
  * [27] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  Int. J. Comput. Vision, 2015.
  * [28] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al.  Sparse r-cnn: End-to-end object detection with learnable proposals.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2021.
  * [29] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2015.
  * [30] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2016.
  * [31] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In Proc. Int. Conf. Mach. Learn., 2021.
  * [32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Proc. Advances in Neural Inf. Process. Syst., 2017.
  * [33] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  arXiv preprint arXiv:2102.12122, 2021.
  * [34] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang.  Cvt: Introducing convolutions to vision transformers.  arXiv preprint arXiv:2103.15808, 2021.
  * [35] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2017.
  * [36] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu.  Co-scale conv-attentional image transformers.  arXiv preprint arXiv:2104.06399, 2021.
  * [37] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan.  Tokens-to-token vit: Training vision transformers from scratch on imagenet.  arXiv preprint arXiv:2101.11986, 2021.
  * [38] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In Proc. Int. Conf. Learn. Representations, 2018.
  * [39] Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z Li.  Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2020.
  * [40] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.  Random erasing data augmentation.  In Proc. AAAI Conf. Artificial Intell., 2020.
  * [41] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2017.


# Better Sampling, towards Better End-to-end Small Object Detection

Zile Huang &Chong Zhang &Mingyu Jin &Fangyu Wu &Chengzhi Liu &Xiaobo Jin111
Xi’an Jiaotong-Liverpool University
Zile.Huang21, Chong.zhang19<EMAIL_ADDRESS>
Fangyu.Wu02<EMAIL_ADDRESS>
###### Abstract

While deep learning-based general object detection has made significant
strides in recent years, the effectiveness and efficiency of small object
detection remain unsatisfactory. This is primarily attributed not only to the
limited characteristics of such small targets but also to the high density and
mutual overlap among these targets. The existing transformer-based small
object detectors do not leverage the gap between accuracy and inference speed.
To address challenges, we propose methods enhancing sampling within an end-to-
end framework. Sample Points Refinement (SPR) constrains localization and
attention, preserving meaningful interactions in the region of interest and
filtering out misleading information. Scale-aligned Target (ST) integrates
scale information into target confidence, improving classification for small
object detection. A task-decoupled Sample Reweighting (SR) mechanism guides
attention toward challenging positive examples, utilizing a weight generator
module to assess the difficulty and adjust classification loss based on
decoder layer outcomes. Comprehensive experiments across various benchmarks
reveal that our proposed detector excels in detecting small objects. Our model
demonstrates a significant enhancement, achieving a 2.9% increase in average
precision (AP) over the state-of-the-art (SOTA) on the VisDrone dataset and a
1.7% improvement on the SODA-D dataset.

##  1 Introduction

Since the emergence of deep convolutional neural networks (CNN), the
performance of object detection methods has improved rapidly. There are two
main approaches: two-stage proposal-based models with accuracy advantages [1]
and single-stage proposal-free models with speed advantages [2]. However,
despite recent tremendous advances in object detection, detecting objects
under certain conditions is still difficult, such as being small, occluded, or
truncated.

Figure 1: Visualization of our methods within challenging scenes featuring
crowds and overlapping objects. Our visual illustration emphasizes the
effectiveness of scale-aligned target, sample reweighting, and sample point
refinement within intricate scenes, allowing for a comparative analysis with a
standard or baseline method.

Small target detection problems are mainly divided into four categories:
multi-scale representation, contextual information, super-resolution and
region proposal. Due to the tiny size and low resolution of small objects,
location details are gradually lost in high-level feature maps. Multi-scale
representation [3] combines the location information of low-level features and
the semantic information of high-level features. Contextual information, which
exploits the relationship between an object in an image and its surrounding
environment, is another novel way to improve small object detection accuracy.
It is very necessary to extract more additional contextual information [4] as
a supplement to the original ROI (Region of Interest) features because there
are too few ROI features extracted from small objects. As mentioned above,
fine details are crucial for object instance localization, which means that
more details for small objects should be obtained. Super-resolution technology
[5] attempts to restore or reconstruct the original low-resolution image to a
higher resolution, which means more details of small objects can be obtained.
Region-proposal [6] is a strategy aimed at designing more suitable anchor
points for small objects. The anchors of the current leading detectors mainly
focus on generic objects, and the size, shape and number of anchors used in
general detectors do not match small objects well.

Through the Transformer architecture, DETR [7] achieves competitive end-to-end
detection performance in a one-to-one scheme, abandoning the manual components
of NMS and eliminating the need for post-processing. However, the limited
number and low quality of positive samples may lead to poor performance on
objects of extremely limited size. Deformable DETR [8] uses deformable
sampling points to enhance its feature decoding capabilities, but its
performance in densely populated and overlapping scenes is unsatisfactory, and
the ongoing challenge of effectively distinguishing positive and negative
samples remains solve.

To address the inherent challenges associated with small object detection and
the limitations observed in current end-to-end object detectors, we introduce
three different strategies aimed at mitigating these issues from different
perspectives. In this work, we focus on improving the DETR method [7] for
detecting small target objects. First, to mitigate potential clutter caused by
background noise and surrounding elements, our approach follows an iterative
refinement paradigm similar to [8], which focuses on refining the attention
mechanism by merging updated box predictions from the previous layer, and
effectively reduce confusion with background elements and noise. Second,
sample point refinement (SPR) constrains the distribution and attention of
sample points by setting learning goals for the deformable attention module in
the decoder, instead of directly discarding reference points located outside
the bounding box. This intentional inhibition can serve as a strategic
stimulus for the model, resulting in a more object-centered deformable
attentional sampling capability without losing surrounding information.
Finally, a decoupled sample reweighting (SR) mechanism integrates a scaling
factor into the target confidence, making the model inherently scale-aligned.
The decoupled reweighting technique helps the model refocus its focus to
learning from challenging positive examples at the feature level instead of
the object level.

In summary, the main contributions of this article are as follows:

  * •

We proposed Sample Points Refinement (SPR), which imposes constraints on the
localization and attention of sample points within deformable attention. This
ensures the preservation of meaningful interactions within the region of
interest while filtering out potentially misleading information.

  * •

We incorporate scale information from samples into the target confidence,
namely Scale-aligned Target (ST), thereby establishing a more appropriate
learning objective for classification. This adjustment proves highly
advantageous, particularly in the realm of small object detection.

  * •

We present a task-decoupled Sample Reweighting (SR) mechanism aimed at guiding
the model to redirect its attention toward learning from challenging positive
examples. Our approach includes a weight generator module that gauges
difficulty from decoder layer outcomes, facilitating the adjustment of the
classification loss accordingly.

  * •

We conduct comprehensive experiments and ablations on VisDrone [9] and SODA-D
[10] to verify the effectiveness of the proposed method and the importance of
correct sampling in small object detection.

##  2 Related work

#### Multi-scale Feature Learning

The basic idea of the multi-scale feature learning method is to learn targets
at different scales separately, mainly to solve the problem that small targets
themselves have few discriminative features, including based on feature
pyramids and based on receptive fields. The main idea of the feature pyramid-
based method [11] is to integrate low-level spatial information and high-level
semantic information to enhance target features. Receptive field-based
methods, such as the Trident network [12], use dilated convolutions with
different expansion rates to form three branches with different receptive
fields, responsible for detecting objects of different scales.

#### GAN-based Methods

Different from multi-scale feature learning methods, GAN methods solve the
problem of small targets with few discriminative features by generating high-
resolution images or high-resolution features. SOD-MTGAN [13] uses a trained
detector (such as Faster R-CNN) to obtain a subgraph containing objects, and
then uses a generator to generate corresponding high-definition images, where
the detector also acts as a discriminator. After generating high-resolution
features, GaN network [14] reduce the input image by 2 times, then extracts
the features as low-resolution features, and the features extracted from the
original image as the corresponding high-resolution features. The generator
generates “fake" high-resolution features based on low-resolution features,
and the discriminator is responsible for distinguishing “fake" high-resolution
features from “real" high-resolution features.

#### Context-based Methods

Context-based methods assist the detection of small objects based on the
environmental information where the small objects are located or their
relationship with other easily detected objects. Inside-Outside Net [15]
extracts the global context information of each object in the four directions
of up, down, left, and right through RNN. RelationNetwork [16] uses the
relationship between objects to assist in the detection of small objects. The
relationship between two objects is implicitly modeled by a Transformer, and
this relationship is used to enhance the characteristics of each object.

#### Loss Reweighting Methods

This type of method makes the network pay more attention to the training of
small objects by increasing the weight in the small object loss. The feedback-
driven loss method [17] increases the weight of small object loss when the
proportion of small object loss is low, allowing the network to treat objects
of different scales more equally.

#### Special Design for Small Object

This type of method designs target detection algorithms based on the
characteristics of small targets. S3FD [18] directly reduces the IoU threshold
of small object positive samples, thereby increasing the number of anchors for
small object matching. If there are still few anchors matching small objects,
the top $N$ anchors are selected from all anchors that meet the threshold as
matching anchors. Xu et al. [19] proposed a new metric to alleviate the
situation where a slight shift in the prediction box of a small object causes
a huge change in the IoU index. Wang et al. [20] introduced the normalized
Wasserstein distance to optimize the position metric for small object
detectors. However, this method is not sensitive to box offset, and a small
amount of offset will not cause a sharp drop in the indicator.

##  3 Background on Deformable DETR-based Detection

###  3.1 Deformable DETR-based Models

Following the Deformable DETR-based models [8], our model consists of three
main components: a CNN-backbone, an encoder-decoder transformer, and a
prediction head.

The backbone network flattens the raw features of the input image into a
sequence of tokens $X=\\{x_{1},x_{2},\cdots,x_{l}\\}$. The transformer then
extracts information from $X$ using a set of learnable queries
$Z=\\{z_{1},z_{2},\cdots,z_{n}\\}$.

We can compute the attention representation for the ith query point

$\displaystyle\textrm{Attn}(z_{i},p_{i},X)$ $\displaystyle=$
$\displaystyle\sum_{m}W_{m}[\sum_{l,k}A_{milk}W^{\prime}_{m}x_{l}(\phi_{l}(p_{i})+\Delta
p_{milk})],$

where the set of subscripts $\\{m,i,l,k\\}$ represents the $k$-th sampling
point of the $l$-th sampling layer of the $m$-th attention head corresponding
to the $i$-th query and the reference point of the $i$-th query is $p_{i}$.
The matrices $W$ and $W^{\prime}$ correspond to two linear transformations
respectively. Here, $\Delta p$ and $A$ denote the tensor matrices of sampling
offsets and attention weights, respectively, where the weight matrix $A$
satisfies the constraints $\sum_{l,k}A_{milk}=1$. It is worth noting that
$p_{i}$ is a normalized two-dimensional coordinate located at $[0,1]^{2}$, and
the function $\phi$ will convert it to the coordinates in the original image.

Subsequently, for each query $z_{i}$, the decoder will output two heads: the
head $H^{\textrm{reg}}(z_{i})$ predicts the location of the object box in the
image, and another head $H^{\textrm{cls}}(z_{i})$ predicts the category of the
object

$s_{i}=H^{\textrm{cls}}(z_{i}),\quad b_{i}=H^{\textrm{reg}}(z_{i})$ (2)

where $s_{i}$ represents the object category corresponding to the $i$-th
query, and $b_{i}$ is the box predicted by multiple decoders, including the
upper left corner position and the length and width of the box.

###  3.2 Varifocal Loss

We adopt Varifocal Loss (VFL) [21] as our baseline method for classification
loss. Varifocal Loss is an extension of Focal Loss [22], incorporating a
trade-off parameter $\alpha$ that adjusts the loss based on the difficulty of
each sample:

$\hat{\mathcal{L}}_{\textrm{cls}}=-\sum_{i=1}^{N^{+}}q_{i}\textrm{BCE}(p_{i},q_{i})-\alpha\sum_{i=1}^{N^{-}}p_{i}^{\gamma}\textrm{BCE}(p_{i},0),$
(3)

where $p_{i}$ is the predicted IoU-aware classification score (IACS) and
$q_{i}$ is the target score. For a foreground point, $q_{i}$ is set to the IoU
between the generated bounding box and its ground-truth value, otherwise it is
$0$. For background points, the target $q_{i}$ is 0 for all classes. The
variables $N^{+}$ and $N^{-}$ respectively represent the number of bounding
boxes containing the object.

We perform bounding box regression using a combination of norm loss
$\mathcal{L}_{1}$ and Intersection over Union (IoU) loss
$\mathcal{L}^{\textrm{IoU}}$, where $L1$ loss measures the absolute difference
in width, height, center $x$, and center $y$ between the predicted bounding
box and the ground truth, a measure of localization accuracy, while IoU
computes the intersection area between the predicted and ground-truth bounding
boxes divided by their union as a similarity measure and penalizes differences
in spatial overlap

$\hat{\mathcal{L}}^{\textrm{reg}}=\sum_{i=1}^{N^{+}}[\mathcal{L}_{1}(b_{i},\hat{b}_{i})+\mathcal{L}^{\textrm{IoU}}(b_{i},\hat{b}_{i})],$
(4)

where $b_{i}$ and $\hat{b}_{i}$ represent the predicted bounding boxes and the
ground truth bounding box respectively. The total loss will be presented in
the following section.

##  4 Our Method

Figure 2: Framework overall. We proposed a series of transferable methods to
support end-to-end models to perform better in small object detection. The
final loss could be represented as
$\mathcal{L}=\mathcal{L}^{\textrm{cls}}+\mathcal{L}^{\textrm{reg}}+\mathcal{L}^{\textrm{offset}}+\mathcal{L}^{\textrm{atten}}$.

Our methodology involves multi-step training supervision aimed at refining
sampling strategies, addressing challenges posed by box edge regression and
ambiguous classification in small object detection, often caused by limited
information availability. Notably, our approach maintains the original
inference speed as we solely modify the training objective of the model.

###  4.1 Refinement of Sample Points Outside Box

We employ learnable offsets in deformable attention to make the convolution
kernel more flexible to focus on objects with significant differences in size
and shape. In Deformable DETR [8], attention is focused on the x-coordinate
and width of the left or right border of the object and the y-coordinate and
height of the top or bottom border. However, when the model is not constrained
in learning deformation offsets, the perceptual field has a tendency to extend
beyond the target, especially in the context of small objects. As in Fig 2,
the red square within the deformable attention region signifies a sample point
necessitating refinement due to its substantial distance from the object,
resulting in diminished informativeness.

Suppose that when we use the $i$-th query to sample in the $l$-th layer of the
$m$-th attention head, the set of sampling points contained within the
bounding box is $\mathcal{I}_{ilm}$, and the set of sampling points located
outside the bounding box is $\mathcal{O}_{ilm}$, then in order to make the
points outside the bounding box as close as possible to the bounding box, we
define the following loss for sampling points outside the bounding box

$\mathcal{L}^{\textrm{offset}}=\sum_{i=1}^{N^{+}}\left[\sum_{m,l,k\in\mathcal{O}_{ilm}}\left\|\Delta
p_{milk}-\eta\begin{bmatrix}w_{i}\\\
h_{i}\end{bmatrix}\right\|_{1}^{2}\right],$ (5)

where $w_{i}$ and $h_{i}$ represent the width and height of the previous layer
on the query $z_{i}$ in the decoder, and $\eta>1$ is an expansion parameter
indicating the expansion degree of the bounding box of the previous layer.
Note that since $\eta>1$, the sample point is still outside the bounding box.
The orange dotted line box in Fig 2 functions as an expanded buffer,
preserving both local and global attention cues.

At the same time, we hope that the weight $A$ of the sampling points outside
the box should be smaller than the weight of all sampling points within the
box, so we define the attention loss for sampling points outside the bounding
box as follows

$\displaystyle\mathcal{L}^{\textrm{atten}}$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{N^{+}}\sum_{m,l,k\in\mathcal{O}_{ilm}}$
$\displaystyle\max(A_{milk}-\max_{k\in\mathcal{I}_{mil}}A_{milk},0).$

###  4.2 Estimation of Target Confidence

$c$ $c$ Figure 3: The IoU values on the two target detection tasks are equal,
but the right side has a larger area ratio (predicted box area and true box
area).

As shown in Eqn. (3), although Varifocal loss increases the penalty for
negative class samples, it does not consider the interaction between target
detection and target classification. To this end, we introduce the results of
small target detection and redefine the target confidence $s$. For small
target detection, detection performance is not only related to the
intersection area between the predicted box and the grounding-truth box, but
also related to the size of the predicted box.

Assume that the annotations of the predicted box and the grounding-box box are
$B=(x,y,w,h)$ and $\hat{B}=(\hat{x},\hat{y},\hat{w},\hat{h})$, respectively,
then we introduce two variables to measure the performance of the detection
algorithm

$\displaystyle u$ $\displaystyle=$
$\displaystyle\frac{\mathcal{A}(B\cap\hat{B})}{\mathcal{A}(B\cup\hat{B})}=\textrm{IoU}(B,\hat{B}),$
(7) $\displaystyle\rho$ $\displaystyle=$
$\displaystyle\frac{\mathcal{A}(B)}{\mathcal{A}(\hat{B})},$ (8)

where $\mathcal{A}(\cdot)$ represents the area of the bounding box. If we
define $x=\mathcal{A}(B)$, $y=\mathcal{A}(\hat{B})$ and
$z=\mathcal{A}(B\cap\hat{B})$, then we have

$\displaystyle u$ $\displaystyle=$ $\displaystyle\frac{z}{x+y-z},$ (9)
$\displaystyle\rho$ $\displaystyle=$ $\displaystyle\frac{x}{y}.$ (10)

Furthermore, we get the relationship between $u$ and $\rho$

$\frac{z/y}{\rho+1}=\frac{u}{1+u}.$ (11)

When IoU ($u$) is constant, $z/y\propto(\rho+1)$. But obviously, we prefer
larger $\rho$ or $z/y$, which indicates better detection results, since the
area of the predicted box is closer to the area of the grounding-truth box.

Based on the above analysis, we define a function with respect to IoU $u$ and
area ratio $\rho$ to re-represent the target confidence score

$\displaystyle r$ $\displaystyle=$ $\displaystyle\sqrt{\rho},$ (12)
$\displaystyle v$ $\displaystyle=$ $\displaystyle e^{-\theta(r-1)^{2}},$ (13)
$\displaystyle c$ $\displaystyle=$ $\displaystyle u^{\beta}\cdot v^{1-\beta},$
(14) $\displaystyle t$ $\displaystyle=$ $\displaystyle c\cdot s,$ (15)

where $\beta$ and $\theta$ are the hyper-parameters. Note that our goal is
that the area of the predicted box is as close as possible to the area of the
true box, that is, $\rho\rightarrow 1$. Therefore, we interpret the meaning of
the above formula as follows: the square root operation in Eqn. (12) is to
make the values near $1$ more concentrated; the exponential operation in Eqn.
(13) is to convert the distance loss into a bounded interval $[0,1]$; Eqn.
(14) combines the indicators IoU ($u$) and $\rho$ through geometric mean as
the proportional coefficient of $s$, where the geometric mean rather than the
algorithmic mean is used to measure measurements on two different scales.

###  4.3 Reweighting of Positive Samples

Figure 4: Illustration for the weight generator in the reweighting module. The
kernel size of Conv1, Conv2, and Conv3 are $C\times 2C\times 1\times 1$,
$C\times C\times 1\times 1$, and $1\times C\times 1\times 1$, respectively.

Below we describe in detail our reweighting module for positive samples (boxes
containing targets).

First, we generate a shared attention feature between the classification and
the regression tasks through the hidden layer representation of the two tasks:
the inputs $H^{\textrm{cls}}_{i}$ and $H^{\textrm{reg}}_{i}$ respectively come
from the hidden layer features decoded by the classification head and the
regression head that decode the query $z_{i}$. Then, we directly concatenate
the two inputs and go through the following operations, as shown in the dotted
box in Fig 4.

$A_{i}=\sigma(\mathcal{B}(\textrm{cov}(\textrm{cat}(H^{\textrm{reg}}_{i},H^{\textrm{cls}}_{i})))),$
(16)

where $\mathcal{B}$ represents ReLU and batch normalization operations and
$\sigma$ is the sigmoid function.

Based on the shared attention weight $A_{i}$, We implement channel compression
through convolution operation and BN/ReLU operation to obtain the weights
$w^{\textrm{cls}}_{i}$ and $w^{\textrm{reg}}_{i}$ corresponding to query
$z_{i}$

$\displaystyle w^{\textrm{cls}}_{i}$ $\displaystyle=$
$\displaystyle\sigma(\mathcal{B}(\textrm{cov}(H^{\textrm{cls}}_{i}\otimes\textrm{cov}(A_{i}))),$
(17) $\displaystyle w^{\textrm{reg}}_{i}$ $\displaystyle=$
$\displaystyle\sigma(\mathcal{B}(\textrm{cov}(H^{\textrm{reg}}_{i}\otimes\textrm{cov}(A_{i}))),$
(18)

where $\otimes$ is the element-wise multiplication of matrices.

However, in the initial stage of training, when the model has not fully
learned accurate feature representations, we discount the weights, where the
weights are a decreasing function of the difference between target confidence
and prediction confidence

$r=w^{1-|t-s|},$ (19)

where $w$ can be $w^{\textrm{cls}}_{i}$ or $w^{\textrm{reg}}_{i}$ related to
query $z_{i}$, $t$ and $s$ are respectively the target confidence and
prediction confidence corresponding to $z_{i}$. Note that when our algorithm
converges, then $t=s$ and $r=w$. Finally, we get the classification weight
$r^{\textrm{cls}}_{i}$ and regression weight $r^{\textrm{reg}}_{i}$ of each
bounding box containing the object.

Based on the above discussion, the reweighted loss for both classification (3)
and regression (4) is derived by substituting the suppression factor on
positive samples with $r^{\textrm{cls}}_{i}$, while retaining the suppression
on negative samples.

$\displaystyle\mathcal{L}^{\textrm{cls}}$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{N^{+}}r^{\textrm{cls}}_{i}\textrm{BCE}(s_{i},t_{i})$
(20)
$\displaystyle+\alpha\sum_{i=1}^{N^{-}}p_{i}^{\gamma}\textrm{BCE}(s_{i},0),$
$\displaystyle\mathcal{L}^{\textrm{reg}}$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{N^{+}}r^{\textrm{reg}}_{i}[\mathcal{L}_{1}(b_{i},\hat{b}_{i})+\mathcal{L}^{\textrm{IoU}}(b_{i},\hat{b}_{i})].$

Finally, we train our object detection architecture using the following loss
function

$\mathcal{L}=\mathcal{L}^{\textrm{cls}}+\mathcal{L}^{\textrm{reg}}+\mathcal{L}^{\textrm{offset}}+\mathcal{L}^{\textrm{atten}}.$
(22)

Table 1: Comparison with state-of-the-art end-to-end detectors detection
approaches on the Visdrone2019 test-set: The subscripts S, M, and L
respectively indicate the size of the object as small, medium, and large.

Model | Publication | #epochs | AP | $\mathrm{AP}_{50}$ | $\mathrm{AP}_{75}$ | $\mathrm{AP}_{S}$ | $\mathrm{AP}_{M}$ | $\mathrm{AP}_{L}$ | Params |
---|---|---|---|---|---|---|---|---|---|---
Non-Deformable-DETR-based |  |  |  |  |  |  |  |  |  |
YOLOX-R50 [23] | CVPR’21 | 70 | 24.1 | 52.6 | 25.1 | 13.7 | 39.5 | 53.4 | $43\mathrm{M}$ |
Anchor-DETR-R50 [24] | AAAI’21 | 70 | 27.7 | 54.7 | 26.3 | 14.1 | 40.8 | 56.7 | $39\mathrm{M}$ |
Sparse RCNN-R50 [25] | CVPR‘21 | 12 | 27.2 | 56.9 | 25.7 | 17.4 | 46.2 | 57.1 | $40\mathrm{M}$ |
ViTDet-ViT-B [26] | ECCV‘22 | 50 | 28.9 | 57.1 | 25.3 | 16.5 | 47.3 | 55.4 | $47\mathrm{M}$ |
Deformable-DETR-based |  |  |  |  |  |  |  |  |  |
Deformable-DETR-R50 [8] | ICLR’21 | 50 | 21.7 | 48.4 | 23.5 | 11.7 | 34.8 | 49.2 | $43\mathrm{M}$ |
Conditional-DETR-R50 [27] | ICCV’21 | 50 | 26.4 | 53.7 | 27.4 | 16.2 | 46.2 | 53.6 | $44\mathrm{M}$ |
DAB-DETR-R50 [28] | ICLR’22 | 50 | 27.8 | 56.8 | 26.9 | 16.4 | 45.5 | 54.8 | $47\mathrm{M}$ |
DINO-R50 [29] | ICLR’23 | 36 | 26.8 | 58.2 | 28.9 | 17.5 | 47.3 | 58.3 | $47\mathrm{M}$ |
Co-DINO-R50 [30] | ICCV‘23 | 36 | 29.4 | 58.7 | 29.5 | 18.9 | 46.8 | 61.3 | $55\mathrm{M}$ |
Baseline-R50 [31] | - | 72 | 28.9 | 57.3 | 31.1 | 19.3 | 46.0 | 57.3 | $42\mathrm{M}$ |
Ours-R50 | - | 72 | 32.3 | 66.4 | 36.7 | 23.7 | 50.0 | 60.2 | $43\mathrm{M}$ |

Table 2: Comparison with state-of-the-art end-to-end detection approaches on
the SODA-D test-set, where the subscripts ES, RS and GS represent extremely
small, relatively small, generally small and normal respectively.

Model | Publication | #epochs | AP | $\mathrm{AP}_{50}$ | $\mathrm{AP}_{75}$ | $\mathrm{AP}_{ES}$ | $\mathrm{AP}_{RS}$ | $\mathrm{AP}_{GS}$ | $\mathrm{AP}_{N}$ | Params
---|---|---|---|---|---|---|---|---|---|---
Non-Deformable-DETR-based |  |  |  |  |  |  |  |  |  |
YOLOX-R50[23] | CVPR’21 | 70 | 26.7 | 53.4 | 23.0 | 13.6 | 25.1 | 30.9 | 30.4 | $43\mathrm{M}$
Anchor-DETR-R50[24] | AAAI’21 | 70 | 27.3 | 55.9 | 22.7 | 12.2 | 24.2 | 32.8 | 41.7 | $39\mathrm{M}$
Sparse RCNN-R50[25] | CVPR‘21 | 12 | 24.2 | 50.3 | 20.3 | 8.8 | 20.4 | 30.2 | 39.4 | $40\mathrm{M}$
ViTDet-ViT-B[26] | ECCV‘22 | 50 | 28.1 | 55.7 | 24.8 | 10.2 | 23.9 | 35.2 | 45.4 | $47\mathrm{M}$
Deformable-DETR-based |  |  |  |  |  |  |  |  |  |
Deformable-DETR-R50[8] | ICLR’21 | 50 | 19.2 | 44.8 | 13.7 | 6.3 | 15.4 | 24.9 | 34.2 | $42\mathrm{M}$
Conditional-DETR-R50[27] | ICCV’21 | 50 | 25.7 | 52.8 | 15.0 | 7.9 | 20.3 | 28.0 | 36.5 | $44\mathrm{M}$
DAB-DETR-R50[28] | ICLR’22 | 50 | 27.2 | 55.1 | 20.6 | 10.3 | 22.5 | 31.9 | 37.2 | $47\mathrm{M}$
DINO-R50[29] | ICLR’23 | 36 | 28.9 | 59.4 | 22.4 | 12.5 | 22.7 | 34.7 | 42.8 | $47\mathrm{M}$
Co-DINO-R50[30] | ICCV‘23 | 36 | 29.7 | 61.3 | 23.9 | 13.6 | 25.3 | 36.1 | 45.1 | $55\mathrm{M}$
Baseline-R50 [31] | - | 72 | 29.3 | 60.2 | 25.2 | 13.2 | 26.9 | 35.4 | 44.6 | $42\mathrm{M}$
Ours-R50 | - | 72 | 31.4 | 63.7 | 28.1 | 15.6 | 29.7 | 38.4 | 46.4 | $43\mathrm{M}$

##  5 Experiment

###  5.1 Dataset & Metrics

We perform quantitative analysis on small object datasets in driving scenes
and aerial photography scenes. SODA-D [10] and VisDrone [9]. SODA-D contains
24,828 carefully selected high-quality images associated with 278,433
instances in nine categories and annotated with horizontal bounding boxes.
These images exhibit significant diversity in time periods, geographical
locations, weather conditions, camera perspectives, etc., making them a
significant strength of this dataset. VisDrone is a data set dedicated to the
detection of drone-captured images, mainly targeting small-sized objects. It
contains various environmental settings (urban and rural), scenes with
different population densities, and a variety of objects, including
pedestrians, vehicles, and bicycles. The dataset contains over 2.6 million
manually annotated bounding boxes for common target objects.

The average precision (AP) is used to verify the performance in our
experiments. The average AP across multiple IoU thresholds is between 0.5 and
0.95 with an interval of 0.05. Furthermore, AP50 and AP75 are calculated under
a single IoU threshold of 0.5 and 0.75, respectively. The definition of AP
across scales differs in Visdrone and SODA-D. In Visdrone, the scale is
divided into small (S), medium (M) and large (L), including objects in the
range of $(0,32^{2}]$, $(32^{2},96^{2}]$, respectively. is $(96^{2},\infty]$.
In SODA-D, the scales are divided into extremely small (ES), relatively small
(RS), generally small (GS) and normal (N), including those located at
$(0,144]$, $(144,400]$, $(400,1024]$ and $(1024,2000]$.

###  5.2 Setup

We adopt RT-DETR [31] as our baseline method and leverage its codebase. RT-
DETR exhibits superior accuracy and speed compared to traditional CNN-based
methods. During training, we use the AdamW optimizer with a base learning rate
of 0.0002, weight decay configured to 0.0001, and a linear warm-up step
lasting 2000 iterations. The learning rate of the backbone network will be
adjusted accordingly. Additionally, we also used an exponential moving average
(EMA) with an EMA decay rate of 0.9999 for stability. Our basic data
augmentation strategy includes stochastic operations such as color distortion,
flipping, resizing, expansion, and cropping.

###  5.3 Main Results

In this study, we conduct a thorough comparison of state-of-the-art end-to-end
object detection methods. To ensure a fair comparison, we equip all baseline
models with ResNet-50 as the underlying backbone architecture.

Our evaluation results clearly position our proposed method as a superior
performer on both datasets considered. Notably, our method shows superior
performance on the VisDrone dataset, achieving a significant overall average
precision (AP) score of 32.3%. Impressively, this performance outperforms the
baseline model by a significant 2.9%.

Similarly, on the SODA-D dataset, our method once again demonstrates its
strength, achieving state-of-the-art performance with an overall AP of 31.3%.
Compared with the baseline model, our method shows significant superiority,
significantly outperforming the baseline model by 1.7%.

These results highlight the effectiveness and dominance of our proposed method
in advancing object detection, especially in scenarios involving small
objects. The state-of-the-art performance achieved on two datasets validates
the superiority of our approach.

Table 3: Ablation study on three strategies including scale-aligned target
(ST), sample reweighting (SR), and sample point refinement (SPR)

ST | SR | SPR | AP | $\mathrm{AP}_{50}$ | $\mathrm{AP}_{S}$ | $\mathrm{AP}_{M}$ | $\mathrm{AP}_{L}$
---|---|---|---|---|---|---|---
$\checkmark$ |  |  | 29.9 | 58.1 | 20.5 | 46.9 | 58.2
| $\checkmark$ |  | 30.5 | 58.7 | 21.1 | 48.2 | 58.6
|  | $\checkmark$ | 31.2 | 61.0 | 21.5 | 48.3 | 58.5
$\checkmark$ | $\checkmark$ |  | 31.8 | 62.1 | 22.8 | 49.5 | 59.8
$\checkmark$ | $\checkmark$ | $\checkmark$ | 32.3 | 66.4 | 23.7 | 50.0 | 60.2

Table 4: Ablation study on the effectiveness of different choices of $\beta$

$\beta$ | 0.3 | 0.5 | 0.73 | 0.9
---|---|---|---|---
AP | 30.4 | 31.4 | 32.3 | 31.9

Table 5: Ablation study on the effectiveness of different choices of $\theta$

$\theta$ | 1.5 | 3 | 6 | 9
---|---|---|---|---
$\mathrm{AP}$ | 30.6 | 30.9 | 32.3 | 31.7

Table 6: Ablation study on scaling factor $\alpha$ and suppression degree $\gamma$ $\gamma$ | 1.5 | 1.75
---|---|---
$\alpha$ |  0.25 |  0.5 |  0.75 |  0.25 |  0.5 |  0.75
AP |  32.1 |  32.3 |  31.5 |  30.8 |  30.6 |  30.2
Table 7: Ablation study on the effectiveness of different choices of expansion
rate $\eta$

$\eta$ | $\mathrm{AP}$ | $\mathrm{AP}_{50}$ | $\mathrm{AP}_{S}$ | $\mathrm{AP}_{M}$ | $\mathrm{AP}_{L}$
---|---|---|---|---|---
(1.5, 1.4, 1.3, 1.2, 1.1, 1.0) | 32.2 | 64.8 | 23.4 | 50.6 | 60.1
(1.5, 1.3, 1.2, 1.1, 1.05, 1.0) | 32.3 | 66.4 | 23.7 | 50.4 | 60.2
(2.0, 1.8, 1.6, 1.4, 1.2, 1.0) | 31.9 | 65.1 | 21.6 | 50.9 | 60.9
(2.0, 1.6, 1.4, 1.2, 1.1, 1.0) | 32.0 | 66.0 | 22.3 | 49.7 | 59.6

Table 8: Generalizability of proposed scale-aligned target (ST) and sample
reweighting (SR) across different state-of-the-art label assignment methods.

Method | $\mathrm{AP}$ | $\mathrm{AP}_{50}$ | $\mathrm{AP}_{S}$ | $\mathrm{AP}_{M}$ | $\mathrm{AP}_{L}$
---|---|---|---|---|---
GFL[32] | 29.7 | 60.7 | 23.7 | 47.7 | 52.8
+ST | 30.8 | 62.1 | 24.1 | 48.6 | 53.2
+SR | 31.6 | 64.5 | 23.7 | 49.8 | 60.2
TOOD[33] | 29.5 | 59.4 | 21.7 | 46.6 | 55.9
+ST | 30.3 | 61.1 | 22.5 | 46.8 | 55.2
+SR | 31.5 | 64.2 | 23.1 | 48.0 | 57.4

###  5.4 Ablation Study

Individual Component Contribution We perform a series of ablations on key
components, namely task-decoupled reweighting and object-oriented losses. The
experimental results are listed in Table 3. All components significantly
contribute to the overall performance, improving objects of different sizes,
with the most significant impact observed in the case of small objects.

Hyperparameters The parameter $\beta$ plays a crucial role in balancing the
contribution of IoU score and scale score in the confidence objective. In a
series of experiments detailed in Table 4, we observe that when $\beta$ varies
from 0.3 to 0.9, the overall performance initially increases to 32.3% and then
drops to 31.9%. Therefore, we choose $\beta=0.73$ as the best choice for our
method. This suggests that scale information plays a crucial role in small
object detection.

Similarly, $\theta$ acts as a scalar controlling factor, from area ratio to
scale fraction, affecting the shape of the projection. In the experiments
summarized in Table 5, we found that when $\theta$ is between 1.5 and 9, the
overall performance shows a trend of first increasing and then decreasing.
Therefore, we choose $\theta=6$ as the best choice for our method.

We also conduct experiments on suppression factors $\gamma$ and $\alpha$ to
determine the optimal balance between positive and negative samples shown in
Table 6. The results indicate substantial variations with $\gamma$, and the
combination of $\gamma=1.5$ and $\alpha=0.5$ yields the best performance.

Figure 5: Comparison of heatmap visualizations between the baseline and our
method.

In our final experiment, we investigated the impact of various settings on the
expansion rate $\eta$ in the six layers of the deformable attention module, as
detailed in Table 7. The settings involved six elements, each controlling the
expansion rate. The first row represents a uniform reduction approach, while
the second row represents an accelerated reduction method. The third and
fourth rows involve a straightforward doubling of the expansion rate $\eta$.
The setting of $(1.5,1.3,1.2,1.1,1.05,1.0)$ shows the best performance of
32.3%. One potential explanation for this behavior is that deformable
attention may initially explore and identify the object in the early layers
and subsequently refine the bounding box in the later layers.

Generalization To further verify the generalizability of our target design and
target reweighting method, we apply our method to two other state-of-the-art
label assignment methods GFL [32] and TOOD [33]. The experiments are based on
our framework, as shown in Table 8. The results demonstrate that our Scale-
aligned Target proves to be a superior choice for small object detection, and
the reweighting method facilitates the model in adaptively learning from data
with class and size biases.

###  5.5 Visualization

Fig. 5 shows the visualization results of various methods applied to images in
the VisDrone dataset. It is worth noting that our method shows quite superior
performance compared with other methods. Initial numbers indicate that the
baseline approach shows insufficient emphasis on extremely small targets. In
contrast, our sampling method significantly enhances attention distribution,
demonstrating enhanced accuracy and the ability to detect very small and faint
targets effectively.

##  6 Conclusion

This paper presents sample point refinement, a novel approach to confine the
localization and attention of sample points within deformable attention. This
constraint ensures meaningful interactions within the region of interest while
filtering out misleading information. Additionally, we introduce Scale-aligned
Targets, incorporating scale information into target confidence, proving
beneficial for small object detection. Furthermore, a task-decoupled sample
reweighting mechanism guides the model to focus on challenging positive
examples, adjusting classification loss based on difficulty assessed by a
weight generator module. Extensive experiments on public datasets consistently
demonstrate our approach outperforming state-of-the-art methods.

## References

  * [1] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun.  R-fcn: object detection via region-based fully convolutional networks.  In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, page 379–387, 2016.
  * [2] Cheng-Yang Fu, Wei Liu, Ananth Ranga, Ambrish Tyagi, and Alexander C. Berg.  DSSD : Deconvolutional single shot detector.  CoRR, abs/1701.06659, 2017.
  * [3] Lisha Cui, Rui Ma, Pei Lv, Xiaoheng Jiang, Zhimin Gao, Bing Zhou, and Mingliang Xu.  Mdssd: Multi-scale deconvolutional single shot detector for small objects, 2020.
  * [4] Yazhou Liu, Sen Cao, Pongsak Lasang, and Shengmei Shen.  Modular lightweight network for road object detection using a feature fusion approach.  IEEE Transactions on Systems, Man, and Cybernetics: Systems, 51(8):4716–4728, 2021.
  * [5] Jiale Cao, Yanwei Pang, and Xuelong Li.  Learning multilayer channel features for pedestrian detection.  IEEE Transactions on Image Processing, 26(7):3210–3220, 2017.
  * [6] Zuge Chen, Kehe Wu, Yuanbo Li, Minjian Wang, and Wei Li.  Ssd-msn: An improved multi-scale object detection network based on ssd.  IEEE Access, 7:80622–80632, 2019.
  * [7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, pages 213–229. Springer, 2020.
  * [8] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  arXiv preprint arXiv:2010.04159, 2020.
  * [9] Dawei Du, Pengfei Zhu, Longyin Wen, Xiao Bian, Haibin Lin, Qinghua Hu, Tao Peng, Jiayu Zheng, Xinyao Wang, Yue Zhang, et al.  Visdrone-det2019: The vision meets drone object detection in image challenge results.  In CVPRW, pages 0–0, 2019.
  * [10] Gong Cheng, Xiang Yuan, Xiwen Yao, Ke Yan, Qinghua Zeng, and Junwei Han.  Towards large-scale small object detection: Survey and benchmarks.  TPAMI, 45:13467–13488, 2022.
  * [11] Chaoxu Guo, Bin Fan, Qian Zhang, Shiming Xiang, and Chunhong Pan.  Augfpn: Improving multi-scale feature learning for object detection.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12592–12601, 2020.
  * [12] Y. Li, Y. Chen, N. Wang, and Z. Zhang.  Scale-aware trident networks for object detection.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6053–6062, 2019.
  * [13] Yancheng Bai, Yongqiang Zhang, Mingli Ding, and Bernard Ghanem.  Sod-mtgan: Small object detection via multi-task generative adversarial network.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [14] Junhyug Noh, Wonho Bae, Wonhee Lee, Jinhwan Seo, and Gunhee Kim.  Better to follow, follow to be better: Towards precise supervision of feature super-resolution for small object detection.  2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9724–9733, 2019.
  * [15] S. Bell, C. Zitnick, K. Bala, and R. Girshick.  Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks.  In CVPR 2016, pages 2874–2883, jun 2016.
  * [16] Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei.  Relation networks for object detection.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3588–3597, 2018.
  * [17] Gen Liu, Jin Han, and Wenzhong Rong.  Feedback-driven loss function for small object detection.  Image and Vision Computing, 111:104197, 2021.
  * [18] Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, and Stan Z Li.  S3fd: Single shot scale-invariant face detector.  In Proceedings of the IEEE international conference on computer vision, pages 192–201, 2017.
  * [19] Chang Xu, Jinwang Wang, Wen Yang, and Lei Yu.  Dot distance for tiny object detection in aerial images.  In 2021 CVPRW, pages 1192–1201, 2021.
  * [20] Jinwang Wang, Chang Xu, Wen Yang, and Lei Yu.  A normalized gaussian wasserstein distance for tiny object detection, 2022.
  * [21] Haoyang Zhang, Ying Wang, Feras Dayoub, and Niko Sunderhauf.  Varifocalnet: An iou-aware dense object detector.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8514–8523, 2021.
  * [22] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [23] Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun.  Yolox: Exceeding yolo series in 2021.  arXiv preprint arXiv:2107.08430, 2021.
  * [24] Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun.  Anchor detr: Query design for transformer-based object detection.  arXiv preprint arXiv:2109.07107, 3(6), 2021.
  * [25] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al.  Sparse r-cnn: End-to-end object detection with learnable proposals.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14454–14463, 2021.
  * [26] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He.  Exploring plain vision transformer backbones for object detection.  In European Conference on Computer Vision, pages 280–296. Springer, 2022.
  * [27] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang.  Conditional detr for fast training convergence.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
  * [28] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang.  DAB-DETR: Dynamic anchor boxes are better queries for DETR.  In International Conference on Learning Representations, 2022.
  * [29] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung-Yeung Shum.  Dino: Detr with improved denoising anchor boxes for end-to-end object detection.  In The Eleventh International Conference on Learning Representations, 2022.
  * [30] Zhuofan Zong, Guanglu Song, and Yu Liu.  Detrs with collaborative hybrid assignments training, 2022.
  * [31] Wenyu Lv, Shangliang Xu, Yian Zhao, Guanzhong Wang, Jinman Wei, Cheng Cui, Yuning Du, Qingqing Dang, and Yi Liu.  Detrs beat yolos on real-time object detection.  ArXiv, abs/2304.08069, 2023.
  * [32] Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang.  Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection.  Advances in Neural Information Processing Systems, 33:21002–21012, 2020.
  * [33] Chengjian Feng, Yujie Zhong, Yu Gao, Matthew R Scott, and Weilin Huang.  Tood: Task-aligned one-stage object detection.  In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3490–3499. IEEE Computer Society, 2021.