#  Feedback RoI Features Improve Aerial Object Detection

Botao Ren1 Botian Xu1 Tengyu Liu2 Jingyi Wang1 Zhidong Deng1 Corresponding
author. Zhidong Deng is with Beijing National Research Center for Information
Science and Technology (BNRist), Institute for Artificial Intelligence at
Tsinghua University (THUAI), Department of Computer Science, State Key
Laboratory of Intelligent Technology and Systems, Tsinghua University, Beijing
100084, China.
1Tsinghua University
2National Key Laboratory of General Artificial Intelligence, BIGAI

###### Abstract

Neuroscience studies have shown that the human visual system utilizes high-
level feedback information to guide lower-level perception, enabling
adaptation to signals of different characteristics. In light of this, we
propose Feedback multi-Level feature Extractor (Flex) to incorporate a similar
mechanism for object detection. Flex refines feature selection based on image-
wise and instance-level feedback information in response to image quality
variation and classification uncertainty. Experimental results show that Flex
offers consistent improvement to a range of existing SOTA methods on the
challenging aerial object detection datasets including DOTA-v1.0, DOTA-v1.5,
and HRSC2016. Although the design originates in aerial image detection,
further experiments on MS COCO also reveal our module’s efficacy in general
detection models. Quantitative and qualitative analyses indicate that the
improvements are closely related to image qualities, which match our
motivation.

##  1 Introduction

Despite impressive advances in object detection [30, 29, 24, 31], detecting
objects in aerial images remains a challenging problem for two reasons: (a)
the objects vary drastically in orientation, scale, aspect ratios, and
density; (b) the qualities of images vary a lot due to difference in various
factor including sensor condition, lighting, and air quality. Most existing
methods address the former challenge with advanced neural network
architectures [11, 35, 37] and objectives [40, 39, 41], and delegate the
latter challenge with data augmentation.

However, research in neuroscience [32, 7, 6] has established that the human
visual system has a special mechanism called neural adaptation to handle low-
quality visual signals efficiently. Inspired by this previously unexplored
mechanism, we propose a plug-and-play feedback-refine module that adaptively
fuses multi-level features.

Our module is compatible with all two-stage detectors, in which the first
stage generates a multi-level feature map with an Feature Pyramid Network
(FPN) [21] and a set of Region of Interest (RoI) [8] proposals, and the second
stage extracts features from them for bounding box and class prediction.
Specifically, after the first pass of detection, for each detection, our
module predicts a set of feature fusion weights based on the image-level
features, the classification results, and the area of the RoI. The first two
features reflect the ambiguities in the image, and the last provides an anchor
for the fusion weights. We then fuse the FPN features from the first stage
with the new weights for the final classification.

Through extensive experiments, we show that including our plug-and-play module
significantly improves recent two-stage object detection models on challenging
aerial object detection datasets DOTA-v1.0, DOTA-v1.5[34], and HRSC2016[25].
To verify that our module is generally applicable and can bring consistent
improvements, we also conducted experiments on MS COCO[20]. We further perform
detailed analyses to unveil that the improvements indeed come from adapting to
images of different qualities. The contribution can be summarized as three-
fold:

  * •

We propose a simple yet effective module to leverage feedback information to
improve detection in aerial images. It adaptively fuses multi-level features
to address the challenges of varying object characteristics and image quality
in aerial images.

  * •

With Flex we verify and validate how the challenges affect detection
performance, and how image- and object-level information could be useful.

  * •

Experimental results demonstrate that our feedback-based module can be plugged
into existing SOTA methods and provides consistent improvements on challenging
aerial object detection datasets including DOTA-v1.0, DOTA-v1.5, and HRSC2016.
It also offers notable performance gains on MS COCO when images become
blurred.

##  2 Related Works

###  2.1 Object Detection in Aerial Images

Object detection in aerial imagery presents unique challenges due to the
scale, orientation, and aspect ratio variations of objects, as well as dense
clusters of small objects and variances in image quality, including lighting
and air quality. Recent advances, regardless of single-stage and two-stage
methods, have focused on addressing the former challenge. Representative two-
stage methods for aerial images include ReDet[11], Oriented RCNN[35],
SCRDet[37], SASM[14], Gliding vertex[36], RoI Transformer[4], etc., while
single-stage methods include R3Det[38], S2ANet[10], DAL[26], etc. ReDet[11]
and ARC[28] modified the convolution layers to explicitly cope with rotation;
Oriented RCNN[35] designed a better representation of oriented bounding boxes;
to deal with objects with large aspect ratios, Hou et al. designed AFPN[13]
where kernels can have asymmetric shapes. S2ANet[10] aligns features between
axis-aligned convolutional features and oriented objects. Novel loss functions
such as GWD[39], KLD[40], and KFIoU[41] were proposed to locate and regress
oriented bounding boxes more accurately. In addition, multi-scale training and
testing strategies are applied to detect small and dense objects. Our method
stands as the first attempt in addressing the challenge of varying image
qualities.

###  2.2 Multi-level Feature Fusion

Multi-level feature fusion has been widely used in object detection to detect
objects of various scales. Bell et al. [1] proposed ION, which connects RoI
features of different granularity to form new features, and then classifies
and regresses RoI. RON[17] uses a reverse connection to pass the coarse-
grained information of the latter layer to the lower layer. Lin et al.[21]
introduced Feature Pyramid Network (FPN), which builds a hierarchy of feature
maps via top-down, bottom-up, and lateral connections, enabling the network to
better leverage information from different scales. To further improve the
effectiveness of FPN for object detection, several recent works have proposed
approaches that adaptively mix multi-level features. Gong et al.[9] proposed
to adjust the fusion factor in FPN to make it more suitable for detecting tiny
objects. ASFF[23] fuses feature maps from different layers with learnable
weights, showing advantages over element-wise sum or concatenation. SENet[15]
and CBAM[33] adaptively modulate channel-wise information. Zhao et al.
proposed M2Det[44], which generates and extracts multi-scale features and then
synthesizes a multi-level feature pyramid using channel attention. Closest to
our work, Zhen et al.[45] propose to use adaptive multi-layer feature fusion
in aerial images at the channel level.

Our work is motivated by the observation that how we fuse multi-level features
should depend on the image quality and the proposed RoI. Although adaptive in
some sense, prior works may fail to capture the interplay between image-level
information (e.g., lighting and quality) and object-level semantics, which we
address by providing such information as feedback to readjust the fusion
process.

###  2.3 Adaptive Feedback in Neural Networks

In recent years, there has been growing interest in incorporating feedback
mechanisms in neural networks for various tasks, including object detection.
These mechanisms are inspired by the feedback processes in the human visual
system, where higher-level visual areas send signals back to lower-level areas
to refine feature representations. Cao et al.[2] proposed a gating strategy to
change the activation state of hidden layers based on the optimization goals
of the network. Zamir et al.[42] proposed Feedback Networks where the
representation is formed iteratively with feedback received from the previous
iteration’s output. Li et al.[19] used high-level information to refine low-
level representations for image super-resolution. Feng et al.[5] designed an
attentive feedback module for Boundary-Aware Salient Object Detection (BSD),
where the predictions from the previous stage are fed to the latter stage to
implement corrections. Huang et al.[16] proposed CNN-F (CNN with Feedback),
which introduces generative feedback with latent variables to improve
performance.

Inspired by this mechanism, our method uses image-level and RoI features to
condition the feature fusion process in the second stage, effectively refining
the representation obtained.

##  3 Methodology

Figure 1: Overview of the Flex. In the pre-classification stage, Flex obtains
RoI features $f_{\text{pre}}$ by fusing different pyramid levels with weights
given by a Gaussian kernel and then computes the bounding box and class
prediction. The feedback refine stage then computes (1) image content feedback
$\Phi^{\text{img}}$ from the feature pyramid and (2) classification feedback
$\Phi^{\text{cls}}$ for each RoI from the pre-classification result. Then we
use $\Phi^{\text{img}}$, $\Phi^{\text{cls}}$ and RoI area to compute refined
feature weights with which the final feature $f_{\text{refine}}$ is obtained
and used to give the final result with a second detection module. The whole
model is jointly trained with standard loss functions for object detection.

Motivated by the neural adaptation mechanism, we designed Flex, Feedback
Multi-Level RoI Feature Extractor, a feedback refine module that plugs into
any two-stage detection models. We name the original two-stage detection as
the pre-classification stage and our design as the feedback refine stage. Flex
takes the image feature from the FPN, the classification logits from the pre-
classification stage, and the area of an RoI to predict a set of fusion
weights. We then use these weights to aggregate FPN features and classify the
RoI in the feedback refine stage. We outline Flex in Fig. 1.

###  3.1 The Pre-Classification Stage

Flex accepts any two-stage object detection method with a feature pyramid as
its pre-classification stage, with one slight modification on feature
extraction.

The original way of extracting the feature for an RoI from an $N$-level
feature pyramid $f=\\{f_{k}\\}_{k=1}^{N}$ is to directly take the feature from
the $i$-th level, where

$i=\max(\min(\lfloor 1+\log_{2}{\sqrt{wh}/\delta}\rfloor,N),1)$ (1)

is determined by the RoI area, $w$ and $h$ are the RoI’s height and width.
$\delta$ is a scale factor which we set to $56$ following the original
publication.

To better leverage information from multiple levels, we do not restrict
feature extraction to be from a single level. Instead, we use a Gaussian
kernel centered at $i$ to obtain a set of weights for the pre-classification
stage by first dropping the floor operation:

$i=\max(\min(1+\log_{2}{\sqrt{wh}/\delta},N),1),$ (2)

and then compute

$W_{k}=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{(k-i)^{2}}{2\sigma^{2}}},k\in[1,N]$
(3)

where $W_{k}$ is the weight of the $k$-th level, and $\sigma$ is a
hyperparameter.

We then compute a weighted sum of $f$ as the RoI feature in the pre-
classification stage:

$f_{\text{pre}}=\frac{\sum_{k=1}^{N}W_{k}f_{k}}{\sum_{j=1}^{N}W_{j}}.$ (4)

###  3.2 Computing Feedbacks from Pre-Classification

We then compute the image and classification feedback from the results of the
pre-classification stage.

We obtain the image feedback $\Phi^{\text{img}}\in\mathbb{R}^{N}$ from the
$N$-level FPN features $f$, whose $i$-th level feature $f_{i}$ has size
$[C_{0},H_{0}/2^{i},W_{0}/2^{i}]$, $C_{0}$, $H_{0}$ and $W_{0}$ being the
channel size, height, and width of the lowest level feature map. We project
each feature level to the same height and width
$[C_{i},H_{0}/2^{N},W_{0}/2^{N}]$ by a convolutional network, where

$C_{i}=C_{0}/2^{N+1-\min{(2,i)}}$ (5)

Then, we concatenate the features channel-wise and feed them through another
convolutional network to obtain $\Phi^{\text{img}}$. Notice the shape of
$\Phi^{\text{img}}$ since we need one value for each feature level.

We further obtain the classification feedback
$\Phi^{\text{cls}}\in\mathbb{R}^{2\lfloor N/2\rfloor+1}$ for each RoI by
applying a fully connected network to the classification output of the RoI in
the pre-classification stage. We require an odd number of values to perform
the interpolation in the feedback refine stage.

###  3.3 Feedback Refine: Multi-Level Feature Extraction

Figure 2: RoI area-aware weight calculation from classification feedback
$\Phi^{\text{cls}}$. We interpret $\Phi^{\text{cls}}\in R^{2\lfloor
N/2\rfloor+1}$ as a piece-wise linear kernel centered at the target level $i$
which is determined by RoI area according to Eq. 2. $W_{k}^{\text{cls}}$ is
then evaluated at $k$ with linear interpolation.

In the feedback refinement stage, we compute the feature weight as

$W_{k}^{FB}=\Phi_{k}^{\text{img}}W_{k}^{\text{cls}},$ (6)

where $\Phi_{k}^{\text{img}}$ is the $k$-th value of $\Phi^{\text{img}}$. To
obtain $W_{k}^{\text{cls}}\in\mathbb{R}^{N}$, we interpret
$\Phi_{k}^{\text{cls}}$ as a piece-wise linear kernel centered at the target
level $i$ such that

$W_{k}^{\text{cls}}=\Phi_{\lfloor j\rfloor}^{\text{cls}}+(j-\lfloor
j\rfloor)(\Phi_{\lfloor j\rfloor+1}^{\text{cls}}-\Phi_{\lfloor
j\rfloor}^{\text{cls}})$ (7)

for $i-\left\lfloor\frac{N}{2}\right\rfloor\leq k\leq
i+\left\lfloor\frac{N}{2}\right\rfloor$ and $0$ otherwise. Here
$j=k-i+\lfloor\frac{N}{2}\rfloor$, and $\Phi^{\text{cls}}_{k}$ is the $k$-th
value of $\Phi^{\text{cls}}$. We discard feature levels that are too far from
$i$, i.e., not in the range of $[i-\lfloor N/2\rfloor,i+\lfloor N/2\rfloor]$.
We visually illustrate the calculation in Fig. 2. We remark that directly
learning a weight vector $\Phi^{\text{cls}}$ of dimension $N$ out of the class
prediction is also viable but fails to leverage the area information.

We again normalize $W^{FB}$ to obtain the feedback-refined RoI feature:

$f_{\text{refine}}=\frac{\sum_{i=1}^{N}W_{i}^{FB}f_{i}}{\sum_{j=1}^{N}W_{j}^{FB}}.$
(8)

and $f_{\text{refine}}$ is used for the final classification for each RoI.

###  3.4 Losses and Hyperparameters

The feedback refine stage is trained with the pre-classification stage in an
end-to-end manner following the overall loss function:

$L=L_{\text{reg}}^{\text{pre-cls}}+\gamma L_{\text{cls}}^{\text{pre-
cls}}+L_{\text{cls}}^{\text{refine}}.$ (9)

We use the standard bounding box regression loss $L_{\text{reg}}$ and cross-
entropy classification loss $L_{\text{cls}}$ for both stages. $\gamma$ is a
hyperparameter.

##  4 Experiment

###  4.1 Experimental Setup

####  4.1.1 Dataset

DOTA[34] is a large-scale dataset for object detection in aerial images.

DOTA-v1.0 contains 2,806 images and 188,282 instances and is labeled with 15
categories, annotated as: Plane (PL), Baseball Diamond (BD), Bridge (BR),
Ground Track Field (GTF), Small Vehicle (SV), Large Vehicle (LV), Ship (SH),
Tennis Court (TC), Basketball Court (BC), Storage Tank (ST), Soccer Ball Field
(SBF), Roundabout (RA), Harbor (HA), Swimming Pool (SP), and Helicopter (HC).
The images range in size from 800×800 to 4000×4000 and contain targets with
various scales, orientations, and shapes. Following the common practice [11],
we use both the training set and validation set for training. We crop the
original images into patches of size $\text{1024}\times\text{1024}$ with the
stride of 824.

DOTA-v1.5 has the same images as DOTA-v1.0 but with different labels and more
small targets (e.g., less than 10 pixels), and an additional category
container crane. It has a total of 403,318 instances. Container Crane (CC)

For the multi-scale task, the images were rescaled with factors of {0.5, 1.0,
1.5} and cropped into patches of size $\text{1024}\times\text{1024}$ with a
stride of 524.

HRSC2016[25] is an aerial image dataset for detecting ships labeled with
oriented bounding boxes. It has 1,061 images with sizes ranging from 300×300
to 1500×900, divided into 436, 181, and 444 images for the training set,
validation set, and testing set. We use both training set and validation set
for training, and testing set for testing. We resized the images to
$\text{800}\times\text{800}$.

####  4.1.2 Implementation Details

We implemented our method based on the widely-adopted MMRotate[46] library. In
the pre-classification stage, we set $\sigma$ as $\frac{\sqrt{2}}{2}$. In the
loss function, we set $\gamma$ as 0.5. Different baseline methods used
different optimizers, which are detailed in the supplementary material. For
each plug-and-play experiment, we trained with the settings reported in the
original paper of the base model. We refer the readers to the Supplementary
Materials for further details.

Table 1: Results of each object class on the DOTA-v1.5 dataset. MS means
multi-scale training and testing, and RR means random rotation augmentation.
We highlight Flex improvements in blue.

Method MS RR PL BD BR GTF SV LV SH TC BC ST SBF RA HA SP HC CC mAP
RetinaNet-O[22] 71.43 77.64 42.12 64.65 44.53 56.79 73.31 90.84 76.02 59.96
46.95 69.24 59.65 64.52 48.06 0.83 59.16 Mask R-CNN[12] 76.84 73.51 49.90
57.80 51.31 71.34 79.75 90.46 74.21 66.07 46.21 70.61 63.07 64.46 57.81 9.42
62.67 HTC [3] 77.80 73.67 51.40 63.99 51.54 73.31 80.31 90.48 75.21 67.34
48.51 70.63 64.84 64.48 55.87 5.15 63.40 OWSR [18] $\checkmark$ $\checkmark$ -
- - - - - - - - - - - - - - - 74.90 RF. R-CNN [31] 72.20 76.43 47.58 69.91
51.99 70.52 80.27 90.87 79.16 68.63 59.57 72.34 66.44 66.07 55.29 6.87 64.63
RF. R-CNN + Flex 72.15 82.50 48.82 70.98 51.81 70.78 80.23 90.86 78.69 67.71
59.23 73.14 66.31 66.37 55.63 9.16 65.27 RoI Trans. [4] 72.27 81.95 54.47
70.02 52.49 76.31 81.03 90.90 84.19 69.12 62.85 72.73 68.67 65.89 57.09 7.12
66.69 RoI Trans. + Flex 72.33 82.83 55.18 71.55 52.41 76.54 81.06 90.86 78.85
69.10 62.91 72.43 75.60 66.14 59.94 8.27 67.25 ReDet [11] 79.20 82.81 51.92
71.41 52.38 75.73 80.92 90.83 75.81 68.64 49.29 72.03 73.36 70.55 63.33 11.53
66.86 ReDet + Flex 80.26 81.65 54.15 73.43 52.80 76.15 88.18 90.87 85.28 74.10
63.05 74.57 76.47 72.23 64.48 16.88 70.28 ReDet + Flex $\checkmark$ 80.62
84.78 55.08 72.49 57.95 80.86 88.56 90.88 85.73 73.80 63.73 73.55 76.56 71.91
68.78 20.59 71.62 ReDet [11] $\checkmark$ $\checkmark$ 88.51 86.45 61.23 81.20
67.60 83.65 90.00 90.86 84.30 75.33 71.49 72.06 78.32 74.73 76.10 46.98 76.80
ReDet + Flex $\checkmark$ $\checkmark$ 86.52 85.03 61.27 81.73 67.96 83.70
89.81 90.88 86.88 84.10 71.43 76.90 78.44 74.18 73.36 49.20 77.59

Table 2: Results on the HRSC2016 dataset. We highlight the best result in blue. Method | mAP07 | mAP12
---|---|---
R2CNN | 73.07 | 79.73
Rotated RPN[43] | 79.08 | 85.64
DRN[27] | - | 92.70
RoI Trans. | 86.20 | -
Gliding Vertex[36] | 88.20 | -
R3Det[38] | 89.26 | 96.01
S2A-Net[10] | 90.17 | 95.01
ReDet | 90.46 | 97.63
Oriented RCNN[35] | 90.50 | 97.60
ReDet + Flex | 90.70 | 98.62

###  4.2 Improvements to Existing Methods

We plugged Flex into ReDet [11] to achieve SOTA on DOTA-v1.5 and HRSC2016. We
show in Tab. 1 that our method achieves 77.59 mAP with multi-scale training
and testing on the DOTA-v1.5 dataset, surpassing all previous methods by a
clear margin. We also observe significant improvement when plugging Flex into
other methods.

We show in Tab. 2 that our method achieves 90.70 mAP under VOC2007 metric and
98.62 mAP under VOC2012 metric on the HRSC2016 dataset, surpassing all other
existing methods.

To further validate that Flex is generally applicable, we add Flex on top of
different two-stage methods and report the detection results of experiments on
DOTA-v1.0 in Tab. 3 and on HRSC2016 in Tab. 4. We find that our method
consistently improves the performances of various two-stage methods.
Interestingly, we observe consistently larger improvements over average
precisions (APs) at higher IoU thresholds. Since our method does not refine
the regression head, this biased improvement suggests that our method is
making better classifications on the proposals that were previously accurately
located but misclassified.

Table 3: Results in COCO style on the DOTA-v1.0 dataset. * means multi-scale training and testing with random rotation augmentation. For ReDet*, it didn’t report AP75 and mAP in paper. We highlight Flex improvements in blue. Method |  AP50 |  AP75 | mAP
---|---|---|---
RF. R-CNN | 73.40 | 39.61 | 40.76
with Flex |  73.95(+0.55) | 40.77(+1.16) | 41.66(+0.90)
ReDet | 76.25 | 50.86 | 47.11
with Flex | 77.09(+0.84) | 51.90(+1.04) | 48.70(+1.59)
ReDet* | 80.10 | - | -
with Flex | 80.61(+0.51) | 58.87 | 53.85
Table 4: Results in COCO style on the HRSC2016 dataset. ‡means using random rotation augmentation. For Gliding Vertex and RoI Transformer, they didn’t report AP75 and mAP in paper. We highlight Flex improvements in blue. Method |  AP50 |  AP75 | mAP
---|---|---|---
RF. R-CNN | 84.97 | 35.89 | 41.80
with Flex | 86.85(+1.88) | 47.75(+11.86) | 46.16(+4.36)
Gliding V. | 88.20 | - | -
with Flex | 89.19(+0.99) | 67.85 | 58.11
RoI Trans. | 86.20 | - | -
with Flex | 90.04(+3.84) | 78.96 | 64.50
ReDet | 90.46 | 89.46 | 70.41
with Flex | 90.58(+0.12) | 89.99(+0.46) | 74.23(+3.82)
with Flex ‡ | 90.70(+0.24) | 90.03(+0.57) | 76.43(+6.02)

###  4.3 Experimental result on MS COCO

To show that our method is generally applicable beyond the aerial image
domain, we also conduct experiments on MS COCO. Using Faster R-CNN and Mask
R-CNN as baselines, we find that Flex provides consistent improvement without
any further parameter tuning, as shown in Tab. 5. However, since MS COCO
includes images of notably better quality and clarity compared to aerial
images, the improvements afforded by Flex are less significant than those on
aerial object detection datasets like DOTA.

We further validate the efficacy of Flex by purposely blurring the images
using a mean kernel. In this case, Flex yields a more substantial improvement,
which aligns with our motivation that the feedback information is helpful in
response to image quality.

Table 5: Results on MS COCO dataset. FR. means Faster R-CNN[31], and MR. means Mask R-CNN[12]. "Blur X" means a $\text{X}\times\text{X}$ kernel is used to blur the images. We highlight Flex improvements in blue. Method | mAP
---|---
Origin | Blur 9 | Blur 15
FR. [31] | 37.7 | 29.8 | 24.6
with Flex | 38.0 (+0.3) | 30.7 (+0.9) | 25.7 (+1.1)
MR. [12] | 38.5 | 30.6 | 25.5
with Flex | 38.8 (+0.3) | 31.6 (+1.0) | 26.5 (+1.0)

###  4.4 Recurrent Feedback using cascade Flex

Table 6: Results on DOTA-v1.5 of Cascading Flex using ReDet as baseline. # Refine Layers | 0 | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---|---
mAP | 66.86 | 70.28 | 70.59 | 70.78 | 70.34 | 70.13

The feedback mechanism in biological neural systems is believed to function in
an iterative manner. Analogously, since Flex is a plug-and-play module, we
further explored plugging Flex into itself to form a cascaded feedback
structure. Specifically, we utilized the classification result output from the
previous refinement layer as classification feedback for the subsequent
refinement layer. The image feedback, like the original Flex method, was
obtained through the shared feature pyramid. Additionally, we observed that
adding one layer only increases training time by approximately 5%.

As shown in Tab. 6, additional Flex layers can further improve performance by
smaller margins, and the improvement wears out at 3 layers on DOTA-v1.5.

The performance first increases and then decreases as the number of layers
increases. This may be attributed to the classification feedback requiring
high classification accuracy from the preceding layer to function optimally.
When the number of layers increases, the variation in classification output
from the initial layers during the early training epochs can hinder the
training of subsequent layers. Only when the early layers stabilize can the
later layers learn more meaningful information. Thus training cascade Flex may
require more than 12 epochs.

###  4.5 Ablation Studies

To analyze and understand each component of our method, we use ReDet[11] as
the base method to conduct ablation studies on DOTA v1.5.

We first examine the effectiveness of each component in the feedback refine
stage in Tab. 8. The column "multi-level" refers to fusing features with
weights produced by a fixed Gaussian kernel as in the pre-classification
stage. We observe that each component improves the final mAP, while the multi-
level fusion contributes the most.

We further examine the form of classification feedback in Tab. 8. The baseline
method refers to the original method where features are extracted from a
single feature level according to RoI area. Direct refers to generating
$W^{\text{cls}}$ directly from RoI features. Gaussian is similar to
Interpolation, our full model, except the generated values are a Gaussian
kernel’s mean and standard deviation.

Both ablation studies show that fusing features from multiple levels bring the
largest improvement and that incorporating feedback from image and
classification further improves the performance. Furthermore, Tab. 8 shows the
importance of the linear interpolation we used to compute the fusion weights.

Table 7: Ablation study of each component in the feedback refine stage.

Multi-level | Class feedback | Image feedback | mAP
---|---|---|---
|  |  | 66.86
$\checkmark$ |  |  | 69.13
$\checkmark$ | $\checkmark$ |  | 69.88
$\checkmark$ | $\checkmark$ | $\checkmark$ | 70.28

Table 8: Ablation study of different learning methods of pre-classification
feedback. For the Input column, it regards what we use for computing the
weights, and A means we use the RoI area, and C means we use the
classification feedback.

Learning Method | Input | Parameterization | mAP
---|---|---|---
Baseline | A | Fixed | 66.86
Direct | C | Learned | 69.78
Gaussian | C+A | Gaussian | 69.60
Interpolation | C+A | Learned | 70.28

###  4.6 Further Analyses

####  4.6.1 Parameterization of the Classification Feedback

The most straightforward way to obtain the classification feedback
$W^{\text{cls}}$ is to directly compute the weights for the N levels with a
fully connected network. Although being the most flexible, this approach does
not directly leverage the RoI area information. In contrast, a second option
is to generate the parameters of a Gaussian kernel to compute the weights,
similar to Eq. 3. However, this approach may fall too restrictive as the
optimal weight distribution may not be Gaussian. We argue that the RoI area is
a valuable prior since the object size inherently implies its scale. Our
proposed interpolation approach captures this information while allowing the
weights to be more flexible. As shown in Tab. 8, Interpolation performs best.
Comparing Interpolation and Direct, we find that incorporating RoI area can
improve mAP by a clear margin, which is in line with our intuition. Moreover,
we empirically observe that the learned classification feedback weights
$W_{k}^{\text{cls}}$ roughly have a bell shape, which implies the Gaussian
kernel is also a good prior.

####  4.6.2 Image Feedback on Blurry Images

Table 9: Average image quality feedback on clean and blurred images from DOTA-v1.5. Blur X means applying $\text{X}\times\text{X}$ mean kernel. Input | First layer | Last layer
---|---|---
Origin | 0.8289 | 1.1806
Blur 5 | 0.8175 | 1.1855
Blur 9 | 0.7970 | 1.2050
Blur 21 | 0.7768 | 1.2223
Table 10: Average image quality feedback on the top 10% clean images and their blurred versions from DOTA-v1.5. Blur X means applying $\text{X}\times\text{X}$ mean kernel. The last column shows the average of $W_{N}$ over the dataset.  Input | First layer | Last layer | Last layer avg.
---|---|---|---
Origin | 1.1477 | 0.9897 | 1.1806
Blur 5 | 1.0955 | 1.0166 | 1.1855
Blur 9 | 0.9910 | 1.0768 | 1.2050
Blur 21 | 0.8490 | 1.1689 | 1.2223

The intuition that motivated this work is that the human brain adapts to
blurry images, and so should neural networks. To verify if our method does
adapt to images of low quality, we intentionally blurred clean images and
observed how our method reacted to them.

Specifically, we applied mean kernels with various sizes to images from
DOTA-v1.5 and examined how the image feedback $\Phi^{\text{img}}$ responded to
the manually imposed blurs in Tab. 9. We observe that the first feature level,
which contains more image details, receives higher weights for cleaner images;
and that the last level, which contains higher-level information, is more
useful when dealing with blurrier images. This observation matches our
intuition.

We then selected the 10% images with the largest $\Phi^{\text{img}}_{1}$
values from DOTA-v1.5 and applied mean kernels of different sizes. We show the
observed changes in $\Phi^{\text{img}}$ in Tab. 10. We find that these images
show more significant changes in $\Phi^{\text{img}}_{1}$ and
$\Phi^{\text{img}}_{N}$ than in Tab. 9. This also matches our expectation
because, in Tab. 9, some images are already of low quality when all images are
considered. Adding blur to low-quality images is less significant than
applying the same blur to clean images.

Figure 3: The correlation between image quality and the feature weight of the
last level. $\Phi_{1}^{\text{img}}$ is generally large for images with better
quality, e.g., better lighting, higher contrast, and clearer boundaries.
Figure 4: Information gain attained through the refinement stage. The $x$-axis
is the entropy of the pre-classification result in bits. Higher entropy
indicates uncertain class prediction, which is expected to be resolved with
the feedback refinement process.

Both analyses suggest that the image feedback learned to adapt to the blurs in
the images to improve detection in low-quality images, which confirms our
intuition.

We further show image patches with different values of $\Phi_{1}^{\text{img}}$
in Fig. 3. We can observe a clear tendency that the image quality improves as
values of $\Phi_{1}^{\text{img}}$ increase.

Figure 5: Examples of detection results of ReDet and ReDet+Flex.

####  4.6.3 Information Gain on Classification Result Feedback

To inspect how the feedback benefits detection more closely, we examine the
information gain attained with the refined features by computing the entropy
difference between the classifications from the pre-classification
$Y_{\text{pre}}$ and from the feedback refinement stages $Y_{\text{refine}}$.
Specifically, it is calculated as:

$IG=H(Y_{\text{pre}})-H(Y_{\text{refine}})$ (10)

where $H(X)=-\sum P(x)\log P(x)$ is the information entropy.

The correlation between the information gain and the entropy in the pre-
classification stage is depicted in Fig. 4 (in bits). We observe that for pre-
classification results with larger entropy, i.e., higher uncertainty, the
information gain is more substantial. More interestingly, we observe two
sudden jumps at around $H(Y_{\text{pre}})=1.0$ and $H(Y_{\text{pre}})=1.6$.
These two values coincide with $\log_{2}{2}$ and $\log_{2}{3}$, which account
for the case where a classifier cannot identify an RoI between 2 and 3 similar
classes, respectively. However, when the pre-classification entropy is low,
the information gain is less significant, indicating that the refinement stage
has less impact on more distinct classes that can already be predicted fairly
well without feedback information.

###  4.7 Limitations

Although our model has shown significant improvement in average precision at
higher IoUs, we did not refine the bounding box regression head for
performance issues. We suspect the reason is that compared to the
classification head that is sensitive to image qualities, the regression head
performs fairly well in low-quality images. Therefore, refining the regression
head does not bring much improvement. Another limitation of our submission is
that we only experimented with refining only once, while it is entirely
possible to refine for multiple iterations and attain larger improvements.

##  5 Conclusion

We proposed a feedback-based RoI feature extraction method that can be easily
plugged into any two-stage detectors that extract features from multi-level
feature maps. Specifically, this method divides the extraction of RoI into a
pre-classification stage and a feedback refine stage. The refinement stage
considers feedback from both the image feature and the RoI classification from
the pre-classification stage and adaptively adjusts the weights of the
features fused from different layers. Extensive experiments on DOTA and
HRSC2016 show that our method achieves consistent improvements over various
baseline methods, including the SOTA, and further analyses show that the
improvements are closely related to image quality, which aligns with our
motivation.

## References

  * Bell et al. [2016] Sean Bell, C Lawrence Zitnick, Kavita Bala, and Ross Girshick.  Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2874–2883, 2016.
  * Cao et al. [2015] Chunshui Cao, Xianming Liu, Yi Yang, Yinan Yu, Jiang Wang, Zilei Wang, Yongzhen Huang, Liang Wang, Chang Huang, Wei Xu, et al.  Look and think twice: Capturing top-down visual attention with feedback convolutional neural networks.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2956–2964, 2015.
  * Chen et al. [2019] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al.  Hybrid task cascade for instance segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4974–4983, 2019.
  * Ding et al. [2019] Jian Ding, Nan Xue, Yang Long, Gui-Song Xia, and Qikai Lu.  Learning roi transformer for oriented object detection in aerial images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2849–2858, 2019.
  * Feng et al. [2019] Mengyang Feng, Huchuan Lu, and Errui Ding.  Attentive feedback network for boundary-aware salient object detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1623–1632, 2019.
  * Friston [2010] Karl Friston.  The free-energy principle: a unified brain theory?  _Nature reviews neuroscience_ , 11(2):127–138, 2010.
  * Gilbert and Sigman [2007] Charles D Gilbert and Mariano Sigman.  Brain states: top-down influences in sensory processing.  _Neuron_ , 54(5):677–696, 2007.
  * Girshick et al. [2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 580–587, 2014.
  * Gong et al. [2021] Yuqi Gong, Xuehui Yu, Yao Ding, Xiaoke Peng, Jian Zhao, and Zhenjun Han.  Effective fusion factor in fpn for tiny object detection.  In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , pages 1160–1168, 2021.
  * Han et al. [2021a] Jiaming Han, Jian Ding, Jie Li, and Gui-Song Xia.  Align deep features for oriented object detection.  _IEEE Transactions on Geoscience and Remote Sensing_ , 60:1–11, 2021a.
  * Han et al. [2021b] Jiaming Han, Jian Ding, Nan Xue, and Gui-Song Xia.  Redet: A rotation-equivariant detector for aerial object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2786–2795, 2021b.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2961–2969, 2017.
  * Hou et al. [2022a] Liping Hou, Ke Lu, and Jian Xue.  Refined one-stage oriented object detection method for remote sensing images.  _IEEE Transactions on Image Processing_ , 31:1545–1558, 2022a.
  * Hou et al. [2022b] Liping Hou, Ke Lu, Jian Xue, and Yuqiu Li.  Shape-adaptive selection and measurement for oriented object detection.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 923–932, 2022b.
  * Hu et al. [2018] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7132–7141, 2018.
  * Huang et al. [2020] Yujia Huang, James Gornet, Sihui Dai, Zhiding Yu, Tan Nguyen, Doris Tsao, and Anima Anandkumar.  Neural networks with recurrent generative feedback.  _Advances in Neural Information Processing Systems_ , 33:535–545, 2020.
  * Kong et al. [2017] Tao Kong, Fuchun Sun, Anbang Yao, Huaping Liu, Ming Lu, and Yurong Chen.  Ron: Reverse connection with objectness prior networks for object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5936–5944, 2017.
  * Li et al. [2019a] Chengzheng Li, Chunyan Xu, Zhen Cui, Dan Wang, Zequn Jie, Tong Zhang, and Jian Yang.  Learning object-wise semantic representation for detection in remote sensing imagery.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 20–27, 2019a.
  * Li et al. [2019b] Zhen Li, Jinglei Yang, Zheng Liu, Xiaomin Yang, Gwanggil Jeon, and Wei Wu.  Feedback network for image super-resolution.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3867–3876, 2019b.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Lin et al. [2017a] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2117–2125, 2017a.
  * Lin et al. [2017b] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2980–2988, 2017b.
  * Liu et al. [2019] Songtao Liu, Di Huang, and Yunhong Wang.  Learning spatial fusion for single-shot object detection.  _arXiv preprint arXiv:1911.09516_ , 2019.
  * Liu et al. [2016] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg.  SSD: Single shot MultiBox detector.  In _Computer Vision – ECCV 2016_ , pages 21–37. Springer International Publishing, 2016.
  * Liu et al. [2017] Zikun Liu, Liu Yuan, Lubin Weng, and Yiping Yang.  A high resolution optical satellite image dataset for ship recognition and some new baselines.  In _ICPRAM_ , pages 324–331, 2017.
  * Ming et al. [2021] Qi Ming, Zhiqiang Zhou, Lingjuan Miao, Hongwei Zhang, and Linhao Li.  Dynamic anchor learning for arbitrary-oriented object detection.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 2355–2363, 2021.
  * Pan et al. [2020] Xingjia Pan, Yuqiang Ren, Kekai Sheng, Weiming Dong, Haolei Yuan, Xiaowei Guo, Chongyang Ma, and Changsheng Xu.  Dynamic refinement network for oriented and densely packed object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11207–11216, 2020.
  * Pu et al. [2023] Yifan Pu, Yiru Wang, Zhuofan Xia, Yizeng Han, Yulin Wang, Weihao Gan, Zidong Wang, Shiji Song, and Gao Huang.  Adaptive rotated convolution for rotated object detection.  _arXiv preprint arXiv:2303.07820_ , 2023.
  * Redmon and Farhadi [2018] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement, 2018.
  * Redmon et al. [2016] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 779–788, 2016.
  * Ren et al. [2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ , 28, 2015.
  * Webster et al. [2002] Michael A Webster, Mark A Georgeson, and Shernaaz M Webster.  Neural adjustments to image blur.  _Nature neuroscience_ , 5(9):839–840, 2002.
  * Woo et al. [2018] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.  Cbam: Convolutional block attention module.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 3–19, 2018.
  * Xia et al. [2018] Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang.  Dota: A large-scale dataset for object detection in aerial images.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3974–3983, 2018.
  * Xie et al. [2021] Xingxing Xie, Gong Cheng, Jiabao Wang, Xiwen Yao, and Junwei Han.  Oriented r-cnn for object detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3520–3529, 2021.
  * Xu et al. [2020] Yongchao Xu, Mingtao Fu, Qimeng Wang, Yukang Wang, Kai Chen, Gui-Song Xia, and Xiang Bai.  Gliding vertex on the horizontal bounding box for multi-oriented object detection.  _IEEE transactions on pattern analysis and machine intelligence_ , 43(4):1452–1459, 2020.
  * Yang et al. [2019] Xue Yang, Jirui Yang, Junchi Yan, Yue Zhang, Tengfei Zhang, Zhi Guo, Xian Sun, and Kun Fu.  Scrdet: Towards more robust detection for small, cluttered and rotated objects.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8232–8241, 2019.
  * Yang et al. [2021a] Xue Yang, Junchi Yan, Ziming Feng, and Tao He.  R3det: Refined single-stage detector with feature refinement for rotating object.  In _Proceedings of the AAAI conference on artificial intelligence_ , pages 3163–3171, 2021a.
  * Yang et al. [2021b] Xue Yang, Junchi Yan, Qi Ming, Wentao Wang, Xiaopeng Zhang, and Qi Tian.  Rethinking rotated object detection with gaussian wasserstein distance loss.  In _International Conference on Machine Learning_ , pages 11830–11841. PMLR, 2021b.
  * Yang et al. [2021c] Xue Yang, Xiaojiang Yang, Jirui Yang, Qi Ming, Wentao Wang, Qi Tian, and Junchi Yan.  Learning high-precision bounding box for rotated object detection via kullback-leibler divergence.  _Advances in Neural Information Processing Systems_ , 34:18381–18394, 2021c.
  * Yang et al. [2022] Xue Yang, Yue Zhou, Gefan Zhang, Jirui Yang, Wentao Wang, Junchi Yan, Xiaopeng Zhang, and Qi Tian.  The kfiou loss for rotated object detection.  _arXiv preprint arXiv:2201.12558_ , 2022.
  * Zamir et al. [2017] Amir R Zamir, Te-Lin Wu, Lin Sun, William B Shen, Bertram E Shi, Jitendra Malik, and Silvio Savarese.  Feedback networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1308–1317, 2017.
  * Zhang et al. [2018] Zenghui Zhang, Weiwei Guo, Shengnan Zhu, and Wenxian Yu.  Toward arbitrary-oriented ship detection with rotated region proposal and discrimination networks.  _IEEE Geoscience and Remote Sensing Letters_ , 15(11):1745–1749, 2018.
  * Zhao et al. [2019] Qijie Zhao, Tao Sheng, Yongtao Wang, Zhi Tang, Ying Chen, Ling Cai, and Haibin Ling.  M2det: A single-shot object detector based on multi-level feature pyramid network.  In _Proceedings of the AAAI conference on artificial intelligence_ , pages 9259–9266, 2019.
  * Zhen et al. [2023] Peining Zhen, Shuqi Wang, Suming Zhang, Xiaotao Yan, Wei Wang, Zhigang Ji, and Hai-Bao Chen.  Towards accurate oriented object detection in aerial images with adaptive multi-level feature fusion.  _ACM Transactions on Multimedia Computing, Communications and Applications_ , 19(1):1–22, 2023.
  * Zhou et al. [2022] Yue Zhou, Xue Yang, Gefan Zhang, Jiabao Wang, Yanyi Liu, Liping Hou, Xue Jiang, Xingzhao Liu, Junchi Yan, Chengqi Lyu, et al.  Mmrotate: A rotated object detection benchmark using pytorch.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 7331–7334, 2022.

  *[FPN]: Feature Pyramid Network
  *[RoI]: Region of Interest