11institutetext: Korea University22institutetext: KAIST33institutetext:
AITRICS
33email<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Object Detection in Aerial Images
with Uncertainty-Aware Graph Network

Jongha Kim 11 Jinheon Baek 22 Sung Ju Hwang 2233

###### Abstract

In this work, we propose a novel uncertainty-aware object detection framework
with a structured-graph, where nodes and edges are denoted by objects and
their spatial-semantic similarities, respectively. Specifically, we aim to
consider relationships among objects for effectively contextualizing them. To
achieve this, we first detect objects and then measure their semantic and
spatial distances to construct an object graph, which is then represented by a
graph neural network (GNN) for refining visual CNN features for objects.
However, refining CNN features and detection results of every object are
inefficient and may not be necessary, as that include correct predictions with
low uncertainties. Therefore, we propose to handle uncertain objects by not
only transferring the representation from certain objects (sources) to
uncertain objects (targets) over the directed graph, but also improving CNN
features only on objects regarded as uncertain with their representational
outputs from the GNN. Furthermore, we calculate a training loss by giving
larger weights on uncertain objects, to concentrate on improving uncertain
object predictions while maintaining high performances on certain objects. We
refer to our model as Uncertainty-Aware Graph network for object DETection
(UAGDet). We then experimentally validate ours on the challenging large-scale
aerial image dataset, namely DOTA, that consists of lots of objects with small
to large sizes in an image, on which ours improves the performance of the
existing object detection network.

###### Keywords:

Object Detection, Graph Neural Networks, Uncertainty

##  1 Introduction

Given an input image, the goal of object detection is to find the bounding
boxes and their corresponding classes for objects of interests in the image.
To tackle this task, various object detection models based on conventional
convolutional neural networks (CNNs), including Faster R-CNN [FasterRCNN] and
YOLO [YOLO], to recent transformer-based [Transformers] models, such as DETR
[DETR] and Deformable DETR [deformabletransformer], are proposed, showing
remarkable performances. In other words, there have been considerable
attentions to search for new architectures, for improving their performances
on various object detection tasks, for example, finding objects in aerial
images from Earth Vision [DOTA].

However, despite their huge successes, most existing models are limited in
that they do not consider interactions among objects explicitly, which are
different from how humans perceive images: considering how each object is
spatially and semantically related to every other objects, as well as
capturing visual features of local regions, for contextualizing the given
image. Thus, we suppose that a scheme that does model explicit relationships
among objects in the image is necessary for any object detection networks.
Note that it becomes more important when handling aerial images [DOTA], which
we mainly target, that consist of a large number of objects with varying
scales from extremely small to large than images in the conventional datasets
(i.e., COCO [COCO] or Pascal VOC [PascalVOC]).

Figure 1: Concept Figure. Given an input image, initial object detection is
done, which generates uncertainties of objects as well. Blue objects with low
uncertainties and red objects with high uncertainties are set as source and
target nodes, respectively. A directed graph connecting source nodes to target
nodes, where edges are generated based on spatial-semantic distances between
nodes, is fed into GNNs to obtain refined representations of objects.

The most straightforward approach to consider relationships in the image is to
construct an inherent hierarchy, where local-level features for identifying
local parts are obtained by outputs of CNNs on sub-regions, whereas global-
level features for overall understanding of an entire image are obtained by
aggregating locally obtained CNN features [SSD, FPN, dai2022tardet]. However,
this scheme is highly suboptimal, since it does not explicitly consider
semantic and spatial relationships among objects with object representations,
but it does implicitly model their relationships with CNN features from
different sub-regions. Therefore, in this work, we aim to first detect objects
in the given input image with conventional object detection networks (e.g.,
Faster R-CNN [FasterRCNN] or RoITransformer [RoITransformer]), and then
calculate the edges between them with their distances of semantic and spatial
features. After that, over the constructed graph with objects as nodes and
their distances as edges, we leverage the graph neural networks (GNNs) [GCN,
GNN] to obtain graphical features of objects, which are then combined with
previously obtained visual CNN features for improving final object prediction
performances. Note that, recently, there exist few similar works
[relationnetwork, SGRN] that consider explicit interactions between all
objects by constructing either a fully connected object graph
[relationnetwork] or a spatial-aware sparse graph [SGRN], whose object
representations are used to improve CNN features for object detection.
However, is it necessary to refine CNN features and detection results of every
object?

We suppose that, if the prediction results of object detection networks are
sufficiently certain, it would not be necessary to replace their features and
prediction labels. Therefore, in this work, we aim to improve the prediction
results only on uncertain objects that detection models are mostly confused
about, by leveraging the uncertainty on each object when constructing object
graphs and refining object features in the given image. Specifically, when we
construct a graph composed by objects (nodes) and their both semantic and
spatial similarities (edges), we propose to propagate the information from
certain objects to uncertain ones by defining directions on edges represented
by a directed graph. This scheme allows the model to improve the features of
uncertain objects by contextualizing them with their semantically- and
spatially-related certain ones, but also prevents the certain objects to
receive noisy information from objects that are uncertain, which is also
highly efficient because of ignoring a large number of edges between certain
objects compared against existing models [relationnetwork, SGRN] that consider
all object pairs. Moreover, when we refine CNN features by representations of
the object graph from GNNs, we propose to manipulate the features for only
uncertain objects, rather than changing all object features, which allows the
model to focus on improving uncertain objects while maintaining the high
prediction performances on certain objects. Lastly, to further make the model
focus on uncertain objects, we scale the training loss based on uncertainties,
where uncertain predictions get higher weights in their losses.

We refer our novel object detection framework, which explicitly models
relationships between objects over the directed graph with their
uncertainties, as Uncertainty-Aware Graph network for object DETection
(UAGDet), which is illustrated in Fig. 1. We then experimentally validate our
UAGDet on the large-scale Dataset for Object deTection in Aerial images (DOTA)
[DOTA] with two different versions: DOTA-v1.0 and DOTA-v1.5, containing large
numbers of small and uncertain objects with sufficient interaction existing
among them. Therefore, we suppose our UAGDet is highly beneficial in such a
challenging scenario. Experimentally, we use RoITrans [RoITransformer] as a
backbone object detection network, and the results show that our UAGDet shows
2.32% and 3.82% performance improvements against the backbone RoITrans network
on DOTA-v1.0 and -v1.5 datasets, respectively, with mean Average Precision
(mAP) as an evaluation metric. Also, further analyses show that our graph
construction method considering both semantic and spatial features connects
related objects by edges, and also uncertainties are measured high on
incorrect predictions which are corrected by refined features from graph
representations.

##  2 Related Works

####  2.0.1 Object Detection.

Object detection is the task of localizing and classifying objects of interest
in an image. To mention a few, Faster R-CNN [FasterRCNN] and YOLO [YOLO] are
two CNN-based early models, successfully demonstrating deep learning’s
capability in the object detection task. Recently, following the success of
transformers [Transformers] in natural language processing, DETR [DETR] adopts
transformers in the object detection task, which performs self-attention on
CNN features from different sub-regions of an input image to consider their
relationships. Furthermore, Deformable DETR [deformabletransformer] is
proposed to attend only to the relative spatial locations when performing
self-attention rather than considering all locations, which makes the
transformer-based prediction architecture efficient. However, despite their
huge improvements in regard to developing new object detection networks and
their performances, they largely overlook the relationships among objects. In
particular, CNN-based models rely on constructing hierarchical structures of
local and global features for capturing feature-level relationships via multi-
level feature map, which is known as a feature pyramid network (FPN) [FPN],
but not capturing the explicit object-level relationships. Meanwhile,
transformer-based models only implicitly consider relationships between
candidate queries for objects, but also candidate queries often include
duplicated objects or meaningless background areas which particularly belong
to ’no object’. Thus, unlike these previous works, we propose to explicitly
model object-level relationships, by initially detecting objects and then,
over the graph structure with objects as nodes, representing them with graph
neural networks (GNNs) for obtaining graphical features, which are then
combined with CNN features for final object detection.

####  2.0.2 Graph Neural Networks for Object Detection.

Graph Neural Networks (GNNs), which expressively represent graph structured
data consisting of nodes and edges by iteratively aggregating features from
target node’s neighbors, have gained substantial attention in recent years,
showing successes in various downstream tasks working on graphs [GCN,
graphsage, GAT, GATv2, jo2021edge, GNN]. As such model architectures
explicitly leverage the relationships between connected instances when
representing them, there have been recent attempts to use GNNs on the object
detection task to capture interactions among objects [SGRN, GAR,
RelationRCNN]. For instance, GAR [GAR] constructs a context graph, which
considers interactions between objects and scenes but also between the objects
themselves by forming objects and scenes as nodes and their connections as
edges, and then represents the graph with GNNs. Similarly, Relation R-CNN
[RelationRCNN] generates semantic and spatial relation networks with objects
as nodes, using pre-built co-occurrence matrix and distances between objects,
respectively, for forming edges. However, both GAR [GAR] and Relation R-CNN
[RelationRCNN] have an obvious limitation that the edge generation procedure
is not end-to-end trainable, but based on simple heuristics using pre-
calculated statistics of co-occurrences between instances. On the other hand,
SGRN [SGRN] learns a spatial-aware relation network between objects based on
visual features and spatial distances of initial proposals, which is then
forwarded to GNNs for obtaining representations for objects. Note that our
work has key differences against such a relevant work: instead of working on
all object proposals, we first largely reduce them with non-maximum
suppression (NMS) [dalal2005nms1, felzenszwalb2010nms2], and then aim at
improving only the uncertain objects that the model is largely confused about
by generating semantic and spatial yet directed edges from certain objects to
uncertain ones, which greatly reduces computational costs especially when
dealing with lots of objects in a single image (e.g., DOTA [DOTA]).

####  2.0.3 Uncertainty-Aware Object Detection.

One of the main focuses of Bayesian deep learning in computer vision tasks is
to accurately capture uncertainty of a model’s prediction, which allows the
deep learning models to prevent making incorrect predictions when their
uncertainties are high. Following the successful applications of uncertainty
on semantic segmentation and depth estimation tasks [Gal2016Uncertainty,
WhatUncertainties, UncertDistill], some works propose to utilize uncertainty
in the object detection task as well [varvote, CertainNet]. In particular, Xu
et al. [varvote] propose to merge overlapping bounding boxes based on the
weights of their coordinates obtained from uncertainties (i.e., the larger the
uncertainties, the lower the weights) when performing soft-NMS [SoftNMS],
which contributes to accurately localizing bounding boxes. Also, CertainNet
[CertainNet] measures uncertainties using the distances between predicted
features and learnable class centroids (i.e., the larger the distances at
inference time, the higher the uncertainties), which is based on the work of
deterministic uncertainty quantification (DUQ) [van2020uncertainty]. However,
it does not use uncertainty for improving model’s prediction, but only
calculates uncertainty values of predicted objects. Note that, in contrast to
those previous works [varvote, CertainNet] that use uncertainty either on
combining overlapping bounding boxes or on identifying less certain objects,
we leverage uncertainties in totally different perspectives. That is we
propose to refine representations of uncertain objects by transferring
knowledge from certain objects to uncertain ones over the directed graph
structure, which may correct initially misclassified objects with high
uncertainties.

##  3 Method

####  3.0.1 Faster R-CNN Baseline.

We first define the notations and terms for objection detection by introducing
the object detection pipeline of the Faster R-CNN model [FasterRCNN]. In
Faster R-CNN, initial $N$ proposals $prop_{i}(i=1,...,N)$ are generated from
an input image by Region Proposal Networks (RPN). Then, we obtain $N$ CNN
feature maps $\hat{f}^{conv}_{i}$, each of which is associated with its
proposal $prop_{i}$, with Region of Interest (RoI) extraction modules (e.g.,
RoIPool [RCNN] or RoIAlign [MaskRCNN]), to make use of such features for
finding bounding boxes and their classes. Specifically, extracted CNN features
are fed into two separated fully connected layers, called classification head
and regression head, to obtain the class prediction result $p_{i}$ and the
coordinates of detected bounding box $bbox_{i}$ for each proposal,
respectively. The entire process is formally defined as follows:

$\begin{split}&prop_{i},conv\\_feature=\texttt{RPN}(image),\\\
&\hat{f}^{conv}_{i}=\texttt{RoIAlign}(prop_{i},conv\\_feature),\\\
&\widehat{p_{i}}=\texttt{cls\\_head}(\hat{f}^{conv}_{i}),\;\widehat{bbox_{i}}=\texttt{reg\\_head}(\hat{f}^{conv}_{i}),\end{split}$
(1)

where RPN is a region proposal network for extracting object proposals,
RoIAlign is a CNN feature extraction module for each proposal, cls_head is a
object classification network, and reg_head is a bounding box regression
network. $\widehat{p_{i}}$ and $\widehat{bbox_{i}}$ denote predicted class
label and bounding box coordinates, respectively.

To optimize a model during training, classification loss $L_{cls}$ is
calculated from classification results with CrossEntropy loss, and regression
loss $L_{reg}$ is calculated from regression results with SmoothL1 loss
[FastRCNN], defined as follows:

$L_{cls}=\sum^{N}_{i}\texttt{CrossEntropy}(\widehat{p_{i}},y_{i}),\;L_{reg}=\sum^{N}_{i}\texttt{SmoothL1}(\widehat{bbox_{i}},bbox_{i}),$
(2)

where $y_{i}$ denotes an one-hot vector of a ground-truth label of a proposal
$i$, and $bbox_{i}$ denotes a ground-truth bounding box coordinates. Then,
based on two object detection losses $L_{cls}$ and $L_{reg}$, overall training
loss is formulated as a weighted sum of them:
$L_{obj}=L_{cls}+\lambda_{1}\times L_{reg}$, where $\lambda$ is a scaling
factor. Note that, in the test time, Non-Maximum Suppression (NMS) is applied
so that duplicated bounding boxes with high overlapping areas and non-
promising bounding boxes with low confidences are removed.

####  3.0.2 Overview of Our Uncertainty-Aware Graph Network.

In this work, we aim to further improve the object detection network, for
example, Faster R-CNN, by leveraging the object representations over the
directed graph, which transfers knowledge from certain objects to uncertain
objects by reflecting their semantic-spatial distances via Graph Neural
Networks (GNNs). To achieve this goal, we first measure the uncertainty of
every detected object with minimal costs, while following the initial object
detection pipeline represented in equation 1, which we specify in Subsection
3.1. Then, we construct an object graph consisting of objects as nodes and
their spatial and semantic relatedness as edges, to contextualize all objects
in the input image with a message passing scheme of GNNs. However, since
changing features and prediction results for certain objects may not be
necessary, we focus on improving uncertain objects by leveraging their
contextual knowledge with semantic-spatial relationships to certain objects.
We introduce this uncertainty-based graph generation procedure in Subsection
3.2. After that, based on object representations obtained by GNNs over the
object graph: node features from visual CNN features and initial object
classes; edge weights from pairwise spatial distances among objects, we refine
the features for uncertain objects to improve their prediction performances,
which is described in Subsection 3.3. We finally summarize our overall object
detection pipeline in Subsection 3.4, which is also illustrated in Figure 1.

###  3.1 Uncertainty-Aware Initial Object Detection

Using the Faster R-CNN pipeline defined above, we can predict $N$
classification results and their corresponding bounding boxes. Then, in this
subsection, we describe how to measure the uncertainty of detected objects
along with the Faster R-CNN architecture, to find out the target objects worth
refining.

We first define the uncertainty of each object as $\phi_{i}$. Then, to measure
this, we use MC Dropout [MCDropout]. In particular, MC dropout can approximate
an uncertainty of outputs, obtained from $M$ times of repeated forward passes
while enabling dropout, by calculating a variance of them. To alleviate the
excessive computational cost of MC Dropout caused by repeated forward passes
through every layer, we introduce a slightly modified version of the original
MC dropout. This approach is simple – instead of repeating whole forward
passes, we only repeat forward passes with dropout in last fully connected
layers of Faster R-CNN model’s classification head and regression head, that
are cls_head and reg_head in equation 1. Therefore, we can measure each
object’s uncertainty with minimal additional computational costs. Also, the
suggested uncertainty measure can be easily implemented to other object
detection frameworks, since the only modification required is simply repeating
last forward steps $M$ times with dropout. The formal algorithm of the
lightweight MC Dropout is shown in Algorithm 1.

Algorithm 1 Lightweight MC Dropout

Input: an image $I$, and the number of dropout iterations $M$.
Outputs: $N$ numbers of classification results $\\{p_{1},...,p_{N}\\}$ with
their corresponding bounding boxes $\\{bbox_{1},...,bbox_{N}\\}$ and
uncertainties $\\{\phi_{1},...,\phi_{N}\\}$, where $N$ denotes the number of
object proposals.

1: $\\{(prop_{i},conv\\_feature)\\}_{i=1}^{N}\leftarrow\texttt{RPN}(I)$

2:
$\\{(\hat{f}^{conv}_{i})\\}_{i=1}^{N}\leftarrow\\{\texttt{RoIAlign}(prop_{i},conv\\_feature)\\}_{i=1}^{N}$

3: for $j\leftarrow 1,\ldots,M$ do

4:
$\\{p^{j}_{i}\\}_{i=1}^{N}\leftarrow\\{\texttt{cls\\_head}(\hat{f}^{conv}_{i})\\}_{i=1}^{N}$

5:
$\\{bbox^{j}_{i}\\}_{i=1}^{N}\leftarrow\\{\texttt{reg\\_head}(\hat{f}^{conv}_{i})\\}_{i=1}^{N}$

6: end for//$p$ and $bbox$ are stacked to shape : $(N,M,...)$

7: $p_{i}\leftarrow\texttt{mean}(\\{p^{j}_{i}\\}_{j=1}^{M})$

8: $bbox_{i}\leftarrow\texttt{mean}(\\{bbox^{j}_{i}\\}_{j=1}^{M})$

9: $\phi_{i}\leftarrow\texttt{stdev}(\\{p^{j}_{i}\\}_{j=1}^{M})$

10: return
$\\{p_{1},...,p_{n}\\},\\{bbox_{1},...,bbox_{n}\\},\\{\phi_{1},...,\phi_{n}\\}$

After the object detection step above, we apply NMS with 1/10 of the usual
threshold value to remove redundant and meaningless objects while maintaining
source and targets objects, instead of excessively removing lots of objects
except for the most certain ones. Consequently, after the initial detection
and NMS phases, $N_{nms}$ objects with their features and uncertainties
remain. Note that when any vector existing at this point is used in subsequent
layers, their gradients are detached so that the initial detection model is
left unaffected by the following modules.

###  3.2 Uncertainty-Based Spatial-Semantic Graph Generation

In this subsection, we now explain how to construct an object graph based on
objects’ uncertainties and their spatial-semantic distances. At first, we
first sort all $N_{nms}$ objects based on their uncertainties in ascending
order. Then, the top half with low uncertainties belongs to the source set
$\mathcal{V}_{src}$, and the bottom half with high uncertainties belongs to
the target set $\mathcal{V}_{dst}$. After categorizing all objects into source
and target, we construct a directed bipartite graph $\mathcal{G}$ that
consists of a set of object nodes
$\mathcal{V}=\\{\mathcal{V}_{src}\cup\mathcal{V}_{dst}\\}$ and their edge set
$\mathcal{E}$, where source node $v_{src}$ and target node $v_{dst}$ belong to
source set $\mathcal{V}_{src}$ and target set $\mathcal{V}_{dst}$,
respectively: $v_{src}\in\mathcal{V}_{src}$ and $v_{dst}\in\mathcal{V}_{dst}$.
Note that, in our directed bipartite graph, every edge $e_{i}\in\mathcal{E}$
connects only the source node to the relevant target node but not in the
reverse direction, so that only target nodes (i.e., uncertain nodes) are
affected by source nodes (i.e., certain nodes).

Then, to share the knowledge only across relevant nodes, instead of connecting
every pair of nodes, we aim at connecting a related pair of nodes. To do so,
based on the motivation that each object is likely to be affected by its
nearby objects, our first criterion of relatedness is defined by a spatial
distance between source and target nodes. Specifically, we calculate the
Euclidean distance between two nodes’ coordinates $c_{src}$ and $c_{dst}$ and
consider it as a spatial distance measure, which is formally defined as
follows: $d^{spa}_{(v_{src},v_{dst})}=||c_{src}-c_{dst}||^{2}$. Here, if
$d^{spa}_{(v_{src},v_{dst})}$ is smaller than the certain spatial threshold
value $thr_{spa}$, we add an edge $e_{i}=(v_{src},v_{dst})$ to the graph.
Therefore, every target node has edges to its nearby source nodes.

On the other hand, we also consider the semantic distance between object nodes
to further contextualize them based on their representation-level
similarities. To be specific, we similarly calculate the Euclidean distance
between CNN features of source and target objects, which is considered as a
semantic distance and formally defined as follows:
$d^{sem}_{(v_{src},v_{dst})}=||\hat{f}^{conv}_{src}-\hat{f}^{conv}_{dst}||^{2}$,
where $\hat{f}^{conv}_{src}$ and $\hat{f}^{conv}_{dst}$ denote CNN feature
maps for source and target objects, respectively. After that, similarly in the
edge addition scheme for spatial distances, if $d^{sem}_{(v_{src},v_{dst})}$
is smaller than the semantic threshold $thr_{sem}$, we add an edge
$e_{i}=(v_{src},v_{dst})$.

###  3.3 Feature Refinement via GNNs with Spatial-Semantic Graph

With the spatial-semantic graph $\mathcal{G}=(\mathcal{V},\mathcal{E}$) built
above, we aim at refining the target (i.e., uncertain) node representations by
aggregating features from their source (i.e., certain) node neighbors via
GNNs. Before doing so, we have to define node features and edge weights, which
we describe in the following paragraphs.

At first, we aim to initialize the node features as features of the CNN
outputs and the predicted class. To do so, we first apply 1$\times$1
convolutions to CNN features for each object, to generate a more compact
visual representation $\hat{f}^{down}_{i}$ by downsizing original features
$\hat{f}^{conv}_{i}$ in equation 1, therefore having half of the original
channel dimension. Also, to explicitly make use of the predicted class
information, we regard the index of the maximum value (e.g., argmax) in the
initial probability vector $\widehat{p_{i}}$ as its class, and then embed it
into the representation space. After that, by concatenating both the down-
sized CNN features and the class embedding vector, we initialize the node
features $\hat{f}^{node}_{i}$ for each node. The overall process is formally
represented in equation 3 below.

On the other hand, edge weight used for neighborhood aggregation in GNNs is
defined as a reciprocal of a pairwise spatial distance between source and
target nodes divided by the diagonal length of the input image $I$ for
normalization, formulated as follows:
$w_{i}=1/(d^{spa}_{(v_{src},v_{dst})}\times\texttt{diag\\_len}(I))$ for
$e_{i}=(v_{src},v_{dst})$, where diag_len denotes a function returning the
length of a diagonal of the given image. Note that such an edge weighting
scheme allows the GNN model to give larger weights on nearby objects during
the aggregation of features from neighboring source nodes to the target node.
For GNN, we use two Graph Convolutional Network (GCN) [GCN] layers, and the
overall graph feature extraction procedure is as follows:

$\begin{split}&\hat{f}^{down}_{i}=\texttt{1x1Conv}(\hat{f}^{conv}_{i}),\;\hat{c}_{i}=\texttt{embed}(\texttt{argmax}(\hat{p}_{i}))\\\
&\hat{f}^{node}_{i}=\texttt{concat}(\hat{f}^{down}_{i},\hat{c}_{i}),\;w_{i}=1/(d^{spa}_{(v_{src},v_{dst})}\times\texttt{diag\\_len(I))},\\\
&\hat{f}^{gnn}_{i}=\texttt{GCN}(\mathcal{E},\hat{f}^{node},w),\end{split}$ (3)

where 1x1Conv denotes the 1$\times$1 convolutional operation, argmax returns
the index of the maximum value, embed denotes the class embedding function,
concat denotes the concatenation operation, and we use the GCN for
representing nodes in GCN. Based on equation 3, the resulting node features
for each object $\hat{f}^{gnn}_{i}$ capture visual and class features based on
its relationships to other certain objects as well as itself, but also capture
explicit relatedness between source and target nodes via their spatial
distances.

###  3.4 Final Detection Pipeline and Training Losses

In this subsection, we describe the final detection process, which is done
with contextualized object representations from GNNs. To be specific, for each
object, we first concatenate the visual features $\hat{f}^{conv}_{i}$ from CNN
layers and the graphical features $\hat{f}^{gnn}_{i}$ from GNN layers in a
channel-wise manner. Then, we forward the concatenated features
$\hat{f}^{*}_{i}=\texttt{concat}(\hat{f}^{conv}_{i},\hat{f}^{gnn}_{i})$ to the
fully-connected layers to obtain the final class probability vector
$\hat{p}^{*}_{i}$, which is similar to the class prediction process in
equation 1 for Faster R-CNN, while we use the differently parameterized class
prediction head $\texttt{cls\\_head}^{*}$. That is defined as follows:

$\begin{split}&\hat{f}^{*}_{i}=\texttt{concat}(\hat{f}^{conv}_{i},\hat{f}^{gnn}_{i}),\\\
&\hat{p}^{*}_{i}=\texttt{cls\\_head}^{*}(\hat{f}^{*}_{i}).\\\ \end{split}$ (4)

Note that one of the ultimate goals of our method is to improve the
performance on uncertain objects while maintaining high performance on certain
objects. Therefore, we additionally regulate the CrossEntropy loss based on
the input prediction’s uncertainty value: we give larger weights to objects
with high uncertainties so that the model could focus on those objects. To do
so, we first use the softmax function to normalize the weight values, and then
multiply the number of target objects $N_{nms}$ to keep the scale of the loss:
the sum of all $w_{i}$ from $i=1$ to $N$ equals to the number of objects
$N_{nms}$, as follows:

$\begin{split}w_{i}&=\texttt{softmax}(\phi_{i})\times N_{nms},\\\
&=\frac{\texttt{exp}(\phi_{i}/\tau)}{\sum^{N_{nms}}_{j=1}\texttt{exp}(\phi_{j}/\tau)}\times
N_{nms},\\\ \end{split}$ (5)

where $\tau$ denotes the temperature scaling value for weights. Then, based on
the loss weight for each object proposal, classification loss $L_{ref}$ from
refined features consisting of CNN and GNN representations is defined as
follows:

$\begin{split}&L_{ref}=\sum^{N_{nms}}_{i}w_{i}\times\texttt{CrossEntropy}(\hat{p}^{*}_{i},y_{i}).\\\
\end{split}$ (6)

The overall training loss is then defined by the initial and refined object
detection losses, $L_{obj}$ and $L_{ref}$, as follows:
$L_{total}=L_{obj}+\lambda_{2}\times L_{ref}$, where $\lambda_{2}$ is the
scaling term for the last loss. Note that we refer our overall architecture
calculating $L_{ref}$ as Uncertainty-Aware Graph network for object DETection
(UAGDet), which is jointly trainable with existing object detection layers by
the final objective $L_{total}$, and easily applicable to any object detection
networks. In the test time, we apply NMS to the final object detection results
to remain only the most certain bounding boxes and their classes. Also, during
evaluation, we replace the initial detection results for uncertain objects
associated with target nodes with their final object detection results, while
maintaining the initial detection results for certain objects, to prevent the
possible performance drop.

##  4 Experiments

In this section, we validate the proposed Uncertainty-Aware Graph network for
object DETection (UAGDet) for its object detection performance on the large-
scale Dataset for Object deTection in Aerial images (DOTA) [DOTA].

###  4.1 Datasets

DOTA [DOTA] is widely known as an object detection dataset in aerial images.
There are three versions of DOTA datasets, and we use two of them for
evaluation: DOTA-v1.0, and v1.5. Regarding the dataset statistics, DOTA-v1.0
contains 2,806 large-size images from the aerial view, which are then
processed to have 188,282 object instances in total within 15 object
categories. The 15 object categories for classification are as follows: Plane
(PL), Baseball-diamond (BD), Bridge (BR), Ground track field (GTF), Small
vehicle (SV), Large vehicle (LV), Ship (SH), Tennis court (TC), Basketball
court (BC), Storage tank (ST), Soccer ball field (SBF), Roundabout (RA),
Harbor (HB), Swimming pool (SP), and Helicopter (HC). The number of instances
per class is provided in Table LABEL:tab:result. DOTA-v1.5 uses the same
images as in DOTA-v1.0, while extremely small-sized objects and a new category
named Container Crane (CC) are additionally annotated. Therefore, DOTA-v1.5
contains 403,318 object instances with 16 classes.

Note that the bounding box regression of the DOTA dataset is different from
the conventional datasets, which makes object detection models more
challenging. Specifically, in conventional object detection datasets, such as
COCO [COCO] or Pascal VOC [PascalVOC], the ground truth bounding box of each
object is annotated as $(x,y,w,h)$ format. However, since objects have a wide
variety of orientations in aerial images, the ground truth bounding box in the
DOTA [DOTA] dataset is denoted as $(x,y,w,h,\theta)$ format, with an
additional rotation angle parameter $\theta$. To reflect such an angle in the
object detection architecture, RoITransformer [RoITransformer] uses additional
fully-connected layers to regress additional parameter $\theta$ on top of the
Faster R-CNN architecture [FasterRCNN], which we follow in our experiments.
For model tuning and evaluation on the DOTA dataset, we follow the
conventional evaluation setups [RoITransformer, DOTA, DOTA2]. We first tune
the hyperparameter of our UAGDet on the $val$ dataset while training the model
on the $train$ set. After the tuning is done, we use both the $train/val$ sets
to train the model, and infer on the $test$ set. Inference results on the
$test$ set is uploaded to the DOTA evaluation server [DOTA, DOTA2], to measure
the final performance. For evaluation metrics, we use the Average Precision
for each class result, and also the mAP for all results [DOTA].

###  4.2 Experimental Setups

####  4.2.1 Baselines and Our Model.

We compare our UAGDet with Mask R-CNN [MaskRCNN, DOTA2], RoITransformer
[RoITransformer] and GFNet [GFNet] models. Specifically, Mask R-CNN [MaskRCNN]
is applied to DOTA [DOTA] by viewing bounding box annotations as coarse pixel-
level annotations and finding minimum bounding boxes based on pixel-level
segmentation results [DOTA2]. RoITransformer (RoITrans) [RoITransformer] is
based on Faster R-CNN [FasterRCNN] with one additional FC layers in order to
predict rotation parameter $\theta$ for rotated bounding boxes. GFNet [GFNet]
uses GNNs to effectively merge dense bounding boxes with a cluster structure,
instead of using algorithmic methods such as NMS or Soft-NMS [SoftNMS]. Our
UAGDet uses GNNs for object graphs composed by objects’ uncertainties to
improve uncertain object predictions with graph representations, where we use
RoITrans as a base object detection network.

####  4.2.2 Implementation Details.

Regarding the modules for the architecture including CNN backbone, RPN, and
regression/classification heads, we follow the setting of RoITrans
[RoITransformer]. The number of region proposals is set to 1,250. Based on the
objects’ uncertainties, we set top half nodes as source nodes while bottom
half nodes as target nodes. If $N_{nms}$ is larger then 100, we only consider
top 100 objects, and exclude the rest of highly uncertain objects that are
likely to be noise. We set the dropout ratio as 0.2, $M$ for MC dropout as 50,
$\lambda_{1},\lambda_{2}$ for loss weights as 1, $thr_{spa}$ for sparse graph
generation as 50, and $thr_{sem}$ for semantic graph generation as 10. The
dimensionality of tensors in equation 3 is as follows:
$\hat{f}^{conv}_{i}\in\mathbb{R}^{256\times 7\times 7}$,
$\hat{f}^{down}_{i}\in\mathbb{R}^{128\times 7\times 7}$, and
$\hat{c}_{i}\in\mathbb{R}^{16\times 7\times 7}$. For GNN layers, we use two
GCN layers [GCN] with LeakyReLU activation between them. Concatenated feature
$f^{node}_{i}$ for each node is first fed into 1x1Conv for reducing its
dimension from $\mathbb{R}^{144\times 7\times 7}$ to $\mathbb{R}^{128\times
7\times 7}$, and then fed into GCN, where its dimension is further reduced by
1/2 and 1/4 in two GCN layers, respectively, i.e.,
$\hat{f}^{gnn}_{i}\in\mathbb{R}^{16\times 7\times 7}$. We set a temperature
scale value $\tau$ in softmax of equation 5 as 0.1. The model is trained for
12 epochs with a batch size of 4, a learning rate of 0.01, and a weight decay
of $10^{-4}$, and optimized by a SGD. We use 4 TITAN Xp GPUs.

###  4.3 Quantitative Results

####  4.3.1 Main Results.

Table 1: Object detection results on DOTA-v1.0 and DOTA-v1.5 test datasets.
The CC (Container Crane) category only exists in DOTA-v1.5, thus we remain the
performance of it on DOTA-v1.0 as blank. Best results are marked in bold.