# Leveraging Synthetic Data in Object Detection on Unmanned Aerial Vehicles

Benjamin Kiefer1  University of Tuebingen
<EMAIL_ADDRESS>David Ott1  University of Tuebingen
<EMAIL_ADDRESS>Andreas Zell  University of Tuebingen
<EMAIL_ADDRESS>
###### Abstract

Acquiring data to train deep learning-based object detectors on Unmanned
Aerial Vehicles (UAVs) is expensive, time-consuming and may even be prohibited
by law in specific environments. On the other hand, synthetic data is fast and
cheap to access. In this work, we explore the potential use of synthetic data
in object detection from UAVs across various application environments. For
that, we extend the open-source framework DeepGTAV to work for UAV scenarios.
We capture various large-scale high-resolution synthetic data sets in several
domains to demonstrate their use in real-world object detection from UAVs by
analyzing multiple training strategies across several models. Furthermore, we
analyze several different data generation and sampling parameters to provide
actionable engineering advice for further scientific research. The DeepGTAV
framework is available at https://git.io/Jyf5j.

**footnotetext: Equal contribution. This work has been supported by the German
Ministry for Economic Affairs and Energy, Project Avalon, FKZ: 03SX481B.

##  I Introduction

Unmanned Aerial Vehicles (UAVs) equipped with cameras are increasingly used as
autonomous vision systems in a wide range of applications, such as traffic
surveillance, search and rescue, agriculture and smart cities [1, 2, 3, 4]. In
all of these scenarios, these systems rely on the robust detection of objects
of interest. Although object detection on natural images taken from hand-held
or car-mounted cameras has been studied intensively, object detection from
UAVs trails behind in performance [5]. This is partly due to the limited
amount of annotated publicly available data sets. In turn, this is partly
caused by the highly complex data generating missions, which are subject to
permissions, UAV flying restrictions and environmental factors [6].
Furthermore, there are more degrees of freedom in the UAV domain (camera
angles, position), which account for objects from unnatural perspectives, e.g.
small objects from above. These difficulties are on top of other common
obstacles, such as high and enduring labeling costs.

With more publicly available data sets, object detection on UAVs could be
improved. However, data collection and labeling are expensive and time-
consuming. Furthermore, data set collection raises serious privacy (e.g. GDPR
[7] in Europe) [8] and security concerns because specific locations demand a
long and complicated approval procedure.

Moreover, currently published data sets suffer from large class and domain
imbalances [9, 5, 10, 11]. Both problems are inherently caused by a
problematic capturing procedure, where many variables cannot be controlled.

On the other hand, synthetically generated data for computer vision problems
can help train data demanding visual perception systems because it is
comparably fast and inexpensive to acquire. Furthermore, this data can easily
be tailored to specific requirements. Several works address synthetic data
generation in computer vision. However, most of them focus on driving,
simulating constrained traffic situations [12, 13, 14]. Few works consider the
generation of synthetic data captured from UAVs. However, these only focus on
the capturing process of the sensors and are lacking in world, object and
physics details, or do not feature them at all [12, 15].

---
Figure 1: Real (left) and synthetic (right) image samples in different
applications scenarios with ground truth annotations. Representative objects
are magnified.

While the use of synthetic data in the context of autonomous driving has been
investigated thoroughly, it is not clear whether these findings can be applied
in the UAV setting. As mentioned in the beginning, the problems for general
UAV object detection also apply to synthetic data generation. Most simulation
engines focus on autonomous driving, and therefore rendering is aimed at
looking realistic for these scenarios. Even if these simulation engines could
technically be adapted to the UAV setting, light conditions, shadows,
visibility range, rendered resolution, and more may significantly affect the
quality of the rendered footage. In turn, the generated data may be less
valuable for transfer to the real world.

In this work, we consider the video game Grand Theft Auto V (GTAV) [16] as a
simulation platform. It offers numerous detailed object models that interact
in a large world with realistic graphics and physics simulations. Building on
previous works [17, 18, 19, 14], we extend the DeepGTAV data extraction tool
to work for airborne scenarios by extending its functionalities regarding in-
game agent and camera position and rotation, environment manipulation, object
spawning and metadata extraction.

Using this simulation engine, we create three large-scale high-resolution (4K)
synthetic object detection data sets in different application scenarios. Using
these, we evaluate different training strategies and their transfer
performances on three corresponding real-world data sets. We provide
analytical insights and actionable advice on which settings to choose by doing
extensive experiments and ablations.

In particular, our contributions are as follows:

  * •

We modify and extend the DeepGTAV tool, in particular, to allow the production
of airborne data. We discuss the specific improvements in Section II-A1.

  * •

We provide three large-scale high-resolution metadata annotated data sets in
different UAV application scenarios and make them publicly available.

  * •

We evaluate the applicability of these data sets to improve real-world object
detection and analyze the from-scratch performance.

  * •

We analyze the influence of different parameters of the data generation, e.g.
the graphics quality and the alignment of metadata.

###  I-A Related Work

This section reviews the literature regarding synthetic data generation
engines, synthetic data taken from UAVs and the role of meta/environmental
data in this context.

####  I-A1 Synthetic Data Generation Engines

Many data generation engines focus on autonomous driving. AirSim [20] and
Carla [21] leverage the Unreal Engine [22] to create a simulation engine
suitable for autonomous driving scenarios. While AirSim also provides support
for UAV scenarios, it lacks a world to simulate objects and its physics
implementation, which must be created and modeled first. Instead of
laboriously creating simulation engines, some researchers use the computer
game GTAV [16] to generate data. With DeepGTAV [17], researchers leverage GTAV
to gather data for autonomous driving. PreSIL [14] builds upon this approach
to refine the data acquisition process with their tool DeepGTAV-PreSIL.
However, both systems lack the feature to modify the in-game agent and camera
position and rotation, essentially limiting it to a single autonomous driving
scenario. Further manipulations, such as spawning objects, manipulating the
environment and extracting the metadata, are not possible either. The work at
hand builds upon these two works, improving and extending them to open it for
UAV research.

There are many more synthetic, simulated environments. Please see [23] for an
overview.

####  I-A2 Data Sets Taken on UAVs

The first large-scale real-world object detection data sets taken on UAVs were
VisDrone [24] and UAVDT [9]. Other data sets for object detection and tracking
emerged with many different foci [25, 26, 27, 28, 29, 30, 31, 32, 6, 33].

The need for data sets caused many researchers to focus on synthetic data.
Mid-Air [15] presents a synthetic data set for unstructured environments
captured with the Unreal Engine [22] in combination with AirSim [20]. They
laboriously built a world with landscape and streets for drone navigation.
However, their world is lifeless and does not feature any objects of interest.
The Synthinel-1 [34] data set features synthetic data showing building
footprints with segmentation masks.

An overview comparing the most important data sets is shown in Table I.

####  I-A3 Narrowing the Sim-to-Real Domain Gap

On the model level, many approaches apply synthetic-to-real domain adaptation
[35, 36, 37, 38, 39, 40, 41, 42]. By disentangling the features, these
techniques aim to learn domain invariant features that lead to better transfer
capabilities. A subset of these methods performs image stylization to make the
synthetic images look more similar to their real-world counterparts [43, 44],
referred to as narrowing the appearance gap. Another domain gap arises from
differences in content, called content gap [12]. It depicts the layout and
types of objects featured in the synthetic world instead of in the real world.

Orthogonal to these gaps, we focus on another gap called meta gap. Meta gap
depicts the imaging conditions at the time of capture, such as altitude,
viewing angle and time. It can be seen as a useful abstraction of the content
gap. These metadata are freely available in the synthetic engine and on an
actual UAV. We leverage these metadata to align the distributions of the
synthetic and real-world data set, leading to better performance and data
efficiency.

TABLE I: Comparison with the most prominent annotated aerial object detection data sets in three domains. Data Set | Domain | Data Type | # Images | Platform | Image Widths | Altitude | Angle | Other meta
---|---|---|---|---|---|---|---|---
VisDrone [5] | traffic | real | 10,209 | UAV | 960-2,000 | ✕ | ✕ | ✕
AU-AIR [33] | traffic | real | 32,823 | UAV | 1,920 | ✓ | ✓ | ✓
PreSIL [14] | traffic | synthetic | 40,000 | car | 1,920 | – | – | ✕
DGTA-VisDrone | traffic | synthetic | 50,000 | UAV | 3840 | ✓ | ✓ | ✓
Airbus Ship [45] | maritime | real | 40,000 | satellite | 768 | – | – | ✕
SeaDronesSee [6] | maritime | real | 5,630 | UAV | 3,840-5,456 | ✓ | ✓ | ✓
DGTA-SeaDronesSee | maritime | synthetic | 100,000 | UAV | 3,840 | ✓ | ✓ | ✓
Cattle [46] | agriculture | real | 670 | UAV | 4,000 | ✕ | ✕ | ✕
DGTA-Cattle | agriculture | synthetic | 50,000 | UAV | 3,840 | ✓ | ✓ | ✓

##  II Experimental Setup

###  II-A DeepGTA-UAV

####  II-A1 Tool Description and Improvements

The DeepGTAV framework as used in this work builds upon the DeepGTAV-PreSIL
framework [14, 47], built upon the DeepGTAV framework [48]. Originally,
DeepGTAV was built as a reinforcement learning environment for self-driving
cars, providing functionality to interact with GTAV through a TCP-server and
building upon the functionality of SkriptHookV [49]. The Python library VPilot
[50] interacts with DeepGTAV, which runs in GTAV.

DeepGTA-PreSIL integrates DeepGTAV and GTAVisionExport [51], the technique
presented in [17], to extract depth and stencil buffers from the rendering
pipeline. With those and the world coordinates of objects extracted in GTAV,
pixel-wise object segmentation data can be extracted. From those, object
bounding boxes can be calculated. The DeepGTAV framework also allows
extracting voxel-wise LiDAR segmentation data, which was not used in this
work. The most notable improvements made in this work include:

  1. 1.

Increasing the ease of use and the multitude of scenarios of which synthetic
data can be generated by improving the VPilot interface of DeepGTAV. In
particular, it is now possible to freely modify the in-game position, camera
position and rotation, manipulate the environment (time of day, weather),
spawn objects (e.g. pedestrians, cars) and specify their animations.

  2. 2.

Improving the speed and reliability of the DeepGTAV framework (to obtain
almost no overhead, compared to running GTAV natively).

  3. 3.

Allowing the extraction of metadata of the generated data (like time of day,
height, camera angle, …).

  4. 4.

Providing multiple easily comprehensible and modifiable data generation
scripts for several airborne scenarios and different strategies of metadata
distribution.

  5. 5.

Modifications to capture 4k image data (although we did not analyze the
influence of high resolution data).

In this work, those adaptations were used to capture object detection data
from a UAV perspective (in comparison to an autonomous car scenario in
previous works [17, 14]).

####  II-A2 Modifications of GTAV

For optimal use as a simulation environment, some modifications were made to
the GTAV game files by installing the modifications Simple Increase Traffic
(and Pedestrian) [52], No chromatic aberration lens distortion [53] and Heap
Limit Adjuster [54]. Additionally, the automatic spawning of objects was
modified to match the class distribution in VisDrone.

Furthermore, the bounding box quality was improved by setting the game
resolution to 7680x4320DSR in NVIDIA GeForce Experience [55]. This results in
an upscaling of the rendering buffers to 4k, which is needed to obtain pixel-
perfect object segmentation data for 4k images.

###  II-B Data Set Generation

To assess the usefulness of synthetic data as training data for real-world
scenarios, particularly to assess the usefulness of DeepGTAV and its
adaptability in this context, we examined three different object detection
scenarios.

In choosing those three scenarios, we are bound to existing real-world data
sets to assess the real-world performance. We find that the data sets VisDrone
[56], SeaDronesSee [6] and Cattle [46] are very popular in their respective
application domain and therefore choose these. VisDrone aims for traffic
surveillance in Asian cities with very crowded scenes. SeaDronesSee’s
application domain is search and rescue in open water featuring swimmers and
boats, with the main challenges being reflective regions, shadows and waves or
seafoam. Finally, Cattle aims to bring autonomous vision systems to
agriculture (cattle detection).

Using the DeepGTAV framework, we generate synthetic training data for these
scenarios by specifying VPilot data generation scripts for each scenario. In
the following, we briefly describe the different capturing procedures employed
in the VPilot scripts used to generate the synthetic data sets.

For the generation of all synthetic data sets, the game world of GTAV is
systematically traversed. New images are captured with one frame per second to
obtain mainly distinct images. We export the 4k images but discard the
segmentation and depth maps to focus on pure object detection. Along with
every frame, we export the corresponding ground truth bounding boxes and the
meta labels, i.e. altitude, principal axes (yaw, pitch, roll of camera
rotation), time of the day and weather state.

For the random traversals of the game world, the camera height and angles were
varied. Additional in-game objects were spawned (e.g. vehicles, pedestrians).
See Table II for details. For example, in Cattle, every two seconds, four cows
are spawned $50-250$m in front of the camera with a left-right offset of
$-160-160$m. The objects were spawned in the in-game traffic and pathfinding
and were despawned after 200 seconds. Additionally, a new random travel
location in this area was chosen every 60 seconds to prevent the in-game
navigation from getting stuck.

Concise descriptions of the data sets are given in Tables I and IV.

TABLE II: Data set generation settings. DGTA- | Cattle | SeaDronesSee | VisDrone
---|---|---|---
| GPS
---
$\times 10^{3}$
| $[0,17]\times$
---
$[13,22]$
| $[-28,-18]\times$
---
$[-25,-13]$
| $[-12,14]\times$
---
$[-22,13]$
Altitude |  $10$-$80$m |  $0$-$80$m |  $0$-$40$m
Cam Pitch |  $20$-$90$° |  $20$-$90$° |  $20$-$90$°
| Manual
---
spawn
| 4$\times$cow@2s
---
| 4$\times$ppl@2s
---
4$\times$boat@2
| 3$\times$bike@8
---
1$\times$motor@8
| Spawn y
---
$50$-$250$m |  $50$-$250$m |  $50$-$150$m
| Spawn x
---
$-160$-$160$m |  $-160$-$160$m |  $50$-$150$m

###  II-C Models and Training setup

As object detection models, we take two one-stage real-time detectors,
EfficientDet-$D0$ (E.-$D0$) [57] and Yolov5 [58, 59, 60, 61, 62], both being
on the forefront of real-time object detectors as measured by their
performances on the COCO test set [63, 64]. In particular, EfficientDet-$D0$
is the state-of-the-art model for real-time detectors on large-scale UAVDT [9]
traffic surveillance data set [10]. For that, we use the implementation from
[65] with an image size of 2176px width and anchor scales of (0.3 0.5 0.7).

Yolov5 [62] is a state of the art implementation of the Yolo object detection
model implemented with multiple improvements to the Yolo framework that have
been found in recent years. In this work, we used the unmodified YOLOv5m6
implementation of Yolov5 in release v5.0 [66] with an image size of
1280x1280px and a batchsize of 48. Unless otherwise specified, we used the
provided weights pre-trained on COCO [63].

Furthermore, as a two-stage detector we take the best performing single-model
(no ensemble) on VisDrone from the workshop report [5] (DE-FPN), i.e. a Faster
R-CNN (F.R.) with a ResNeXt-101 64-4d [67] backbone (removing P6), which is
trained using color jitter and random image cropping. The anchor sizes and
strides are decreased to (16, 32, 64, 128, 256) and (4, 8, 16, 32, 64).

We measure the models performances on the popular mean average precision
metric with overlap 0.5, i.e<EMAIL_ADDRESS>[63]. As we are interested in real-world
performance, we test on the test set of the real-world data set unless
indicated otherwise.

##  III Experimental Evaluation

First, we conducted different experiments to show that synthetic training data
could yield good real-world performance or improve the performance of a real-
world object detector. Then we conducted further ablation studies to examine
different factors that could influence or modulate the positive effect of
synthetic training data.

On a general level, we wanted to obtain actionable advice for an engineer
using synthetic data to train an object detector, which factors should be
examined with emphasis and which factors could be ignored. Such factors could
be the graphics quality of the simulation environment or the alignment of
synthetic and real height distributions.

In the following, those experimental conditions and their results will be
described. For better clarity, we discuss the design of each ablation study
and its result individually.

TABLE III: Performance of object detectors for different training strategies given by mAP@50.  | Data set | Synthetic | Real | SyntheticToReal
---|---|---|---|---
E.-$D0$ | Cattle | 29.2 | 78.4 | 85.8
SeaDronesSee | 10.3 | 36.3 | 38.8
VisDrone | 1.2 | 24.6 | 27.2
F.R. | Cattle | 38.8 | 90.5 | 91.5
SeaDronesSee | 14.6 | 54.7 | 59.0
VisDrone | 2.4 | 48.6 | 51.2
YOLO | Cattle | 64.2 | 88.8 | 86.9
SeaDronesSee | 10.5 | 55.8 | 60.3
VisDrone | 10.2 | 43.9 | 45.0

###  III-A General Benefit of Synthetic Data in UAV Object Detection

The main goal of this work is to examine the usefulness of synthetic training
data to train object detectors from scratch or improve the performance of
object detectors trained with real-world data.

From this goal, there naturally arise three conditions which we want to
compare. First, as a baseline, we observe the performance of the object
detector on the real-world data set. Second, we observe the performance of the
object detector trained only on an entirely synthetically generated data set.
Finally, we observe the performance of an object detector which is first pre-
trained on a synthetic data set and then transfer-trained on the real-world
data set. From preliminary experiments, we found these strategies to be
superior to alternative strategies, such as combined training, similar to
previous literature [68].

For all three application scenarios, the corresponding real-world training set
was split into a training, validation and test set. For fully synthetic
training and synthetic pre-training the data set was split into a training and
a validation set, as no testing is conducted on the synthetic data. See the
sizes of the different complete sets in Table IV.

TABLE IV:  Number of images and classes. Note that (awning-)tricycle is abbreviated as (a.-)tri., life jacket as LF and people as ppl. | Number of Images |
---|---|---
Data Set | Train | Val | Test | Classes
VisDrone | 6471 | 548 | 1610 |  | ppl.,bike,car,truck,van
---
motor,tri.,a.-tri.,bus
| Sea-
---
DronesSee
2975 | 859 | 1796 |  | swimmer,floater,boat
---
swimmer†,floater†,LJ
Cattle | 402 | 134 | 134 | Cow
| DGTA-
---
VisDrone
40000 | 10000 | – |  | ppl.,bike,truck
---
car,motor,bus,van
| DGTA-Sea-
---
DronesSee
90000 | 10000 | – |  | swimmer,floater,boat
---
swimmer†,floater†
| DGTA-
---
Cattle
40000 | 10000 | – | Cow

#### Findings

Table III shows that purely training on synthetic data can already provide
minimal working solutions. While all object detectors perform well on the
simple data set Cattle (29.2-64.2 mAP@50), the accuracies on VisDrone and
SeaDronesSee are far lower. This is partly due to missing classes in the
corresponding synthetic data sets ((awning-)tricycle in VisDrone and life
jacket in SeaDronesSee). However, another apparent factor is the difference in
the appearance of certain classes from the synthetic to the real data set. For
example, see Figure 1 to compare the same classes in the synthetic and real
data set. The default appearances of classes vary in some cases significantly.
This appearance gap may be narrowed by manually editing the appearance of
classes to resemble real-world objects. Despite these challenges, synthetic
pre-training with subsequent transfer training boosts performance on VisDrone
and SeaDronesSee significantly across all models. On the SeaDronesSee
evaluation benchmark [69], we can even achieve state-of-the-art performance by
surpassing the best model by +5.6 mAP@50. While the performance improvement is
not as apparent on Cattle, pre-training on synthetic data helps for
EfficientDet-$D0$ and Faster R-CNN.

These experiments show that although synthetic data sets can not replace
corresponding real-world data sets, they enhance the detection performance.
Even the models from [6] can be beaten just by synthetic pre-training.

###  III-B Effect of Data Set Sizes

We wanted to test the influence that the data set sizes had on the performance
in this context. The intuitive hypothesis was that we would observe a curve of
diminishing returns for larger data sets, which is typical in machine
learning. We hypothesized that we would observe such diminishing returns for
the size of the real-world data set, as well as for the size of the synthetic
(pre-training) data set. Furthermore, we hypothesized that the effect of
synthetic pre-training would be more emphasized when using a smaller real-
world data set.

To conduct this ablation, we varied the size of the real-world data set and of
the synthetic data set for VisDrone and SeaDronesSee training.

Figure 2: The effect of the real and synthetic data set size on the model
performance for VisDrone and SeaDronesSee. The color shading specifies the
size of the synthetic data set that was used for pre-training.

#### Findings

Figure 2 shows the effect of different data set sizes in synthetic pre-
training. As hypothesized, we observe an improved mAP@50 with an increase of
the real-world data set size, and an increase of the synthetic pre-training
data set size with a diminishing return for larger data sets. Furthermore, we
observe the hypothesized interactive effect, such that the improvement from
using synthetic pre-training data is disproportionally larger for smaller
real-world data sets.

Our found effects are consistent with the numbers of those reported in [14],
that is, improvements of about +1.0 to +5.0 mAP@50 by using additional
synthetic training data. We note that in [14], a 3D LiDAR object detection
task was examined instead of a 2D object detection task. Compared to [14], in
our work, those improvements are not only on one class but over the whole 10
and 6 classes of VisDrone and SeaDronesSee, respectively.

###  III-C Effect of using pre-trained weights

The use of initial weights pre-trained on large scale image data sets like
ImageNet [70] or COCO [63] has become a standard for many vision machine
learning tasks in recent years. We hypothesized that there could be an
interaction between using such pre-trained weights and synthetic
(pre-)training. For example, such pre-trained weights could encode invariance
to image noise, which is prominent in real world images, but may be missing in
synthetically generated images. The use of pre-trained initial weights would
thereby allow a purely synthetically trained model to obtain the information
it could not obtain from the synthetic data, thereby introducing an
interactive effect. If this hypothesis were true, using pre-trained weights
would improve the performance of a purely synthetically trained model
disproportionally more than it improves the performance of a model trained on
real world data.

TABLE V: Effect of COCO pre-trained weights on performance given by mAP@50 evaluated on VisDrone. | random initial weights | Pre-trained on COCO
---|---|---
Real | 43.1 | 43.9
Synthetic | 7.5 | 10.2
SyntheticToReal | 44.7 | 45.0

#### Findings

Table V shows the effect of using COCO pre-trained weights as initial weights
for the training. Due to the small effect sizes, we would argue that our
results are inconclusive at this time. We observe a more significant absolute
improvement of the mAP@50 when using COCO pre-trained weights on pure DGTA-
VisDrone training than on pure VisDrone training, which would speak for the
hypothesized interactive effect. However, when observing the absolute obtained
mAP@50 of those different conditions, this observed difference in improvements
would also be consistent with an effect of diminishing returns of improvements
for a larger mAP@50. This effect would be that starting from a weaker
performance baseline, the same improvement (using COCO pre-trained weights)
would yield a larger improvement than when starting from a more robust
baseline.

So we conclude that there is at least no strong effect where using pre-trained
weights trained on real-world imagery would disproportionally improve the
performance of synthetic training. However, note that there might be other
factors to consider, such as training time and stability.

###  III-D Effect of Good/Bad graphics settings

One of the parameters that one would intuitively look at when observing
synthetic training data or when trying to improve the use of synthetic
training data is the realism of this training data. This could be framed as
closing the Sim-To-Real gap as discussed above.

A similar but not fully congruent perspective on this coming from game
development is trying to improve the realism of games by improving their
graphics quality. In many cases, those improvements of graphics quality come
with high computational demands (higher polygon models, higher resolution
textures, demanding physics simulations like particle effects and lighting)
and high amounts of labor, e.g. from graphics artists.

Therefore, from an engineering perspective, this raises the question, how well
spent this effort is to obtain the final goal of increasing the synthetically
trained model’s performance.

To answer this question, we conducted experiments comparing the effects of
synthetic training data obtained from GTAV either set to the highest or lowest
possible graphics settings.

TABLE VI: Performance between low quality and high quality in-game settings (given by mAP@50). We compare the performance for only Real training on VisDrone against the performance with synthetic data as DGTA-VisDrone. | Real | Synthetic | SyntheticToReal
---|---|---|---
Real data Baseline | 43.9 | - | -
Low Quality | - | 8.4 | 45.1
High Quality | - | 10.2 | 45.0

#### Findings

Table VI shows the obtained model performance using synthetic data obtained
from GTAV on the lowest and highest graphics settings, respectively. We
observe that for purely synthetic training, the model trained with higher
graphics setting images outperforms the one trained on lower graphics quality
images. This effect, however, is small.

There is no observable effect in the synthetic pre-training condition with
transfer training on the real-world data set.

###  III-E Aligning Domain Distributions

Modifying the DeepGTAV tools to extract metadata allows us to automate parts
of the data generation process by aligning the metadata distributions of the
real and synthetic data sets. Instead of laboriously setting the correct
metadata settings in the synthetic data generation process, one could fall
back to the corresponding real data set to adapt the parameters automatically.
Aligning the distributions may result in higher synthetic data quality or
efficiency.

For instance, in SeaDronesSee, every image is annotated with the capture time
stamp. By bootstrap sampling from the time distribution, we sample a new data
set of 100k synthetic images with the correct time distribution. As before, we
train a Yolov5 model and test it on the SeaDronesSee test set (Synthetic
Only), and we transfer train it on SeaDronesSee (Synthetic-To-Real).

#### Findings

Table VII shows that aligning the time helps in synthetic only training and
synthetic pre-training by increasing the performance over the unaligned
baseline by +4.1 and +0.2 mAP@50, respectively. The performance increase is
mainly due to SeaDronesSee only featuring day-time images, such that sampling
synthetic night images deteriorates the performance.

Similarly, we can also automatically adjust the camera angle. The images in
the Cattle data set have been taken from a downward-facing camera where a
gimbal corrects for UAV angular movement. We sample from DGTA-Cattle only
these images that fall into the range of these angles (with an error threshold
of at most 20 degrees). This reduces the original 40k images to only roughly
10% of the images (3,954). We train an EfficientDet-$D0$ on this subset.

#### Findings

Interestingly, the performance of the synthetic only training improves over
the more extensive unaligned DGTA-Cattle training. This is likely due to the
limited capacity of an EfficientDet-$D0$ model resulting in the model
distributing its performance across many other angular viewpoints, which are
unnecessary for the performance in this use-case. This experiment illustrates
that less but more targeted data may be sufficient to reach the same
performance while reducing the need to filter the data by only aligning the
metadata distributions manually.

TABLE VII: Effect of time and angle alignment on performance (given as mAP@50).  SeaDronesSee | Synthetic | SyntheticToReal
---|---|---
Only real training | - | 55.8
Unaligned Synthetic Pre-training | 10.5 | 60.3
Time Aligned | 14.6 | 60.5
Cattle |  |
Only real training | - | 78.4
Unaligned Synthetic Pre-training | 29.2 | 85.8
Angle Aligned (Subset) | 36.6 | 85.8

##  IV Limitations and Conclusions

We demonstrated that synthetic data can be leveraged for object detection on
UAVs. We can improve the performance over only real training by synthetic pre-
training on multiple application scenarios. Synthetic-only training yields
satisfactory results, but performances are not yet competitive to real
training.

From our ablations, we conclude that the use of more synthetic training data
improves the performance. The use of weights pre-trained on large scale image
data sets constantly improves the performance, although we find no interactive
effect with synthetic training. The graphics quality of the simulation engine
appears to be important for purely synthetic training but not for synthetic
pre-training. In general, metadata alignment is vital for the usefulness and
data efficiency of synthetic training data.

We hope that the adaptation of the DeepGTAV tools helps cast light on object
detection on UAVs via synthetically generated footage. In future works, the
capabilities of the DeepGTAV framework to produce object segmentation data,
LiDAR data and video could be leveraged.

## References

  * [1] T. Adão, J. Hruška, L. Pádua, J. Bessa, E. Peres, R. Morais, and J. J. Sousa, “Hyperspectral imaging: A review on uav-based sensors, data processing and applications for agriculture and forestry,” _Remote Sensing_ , vol. 9, no. 11, p. 1110, 2017.
  * [2] E. Lygouras, N. Santavas, A. Taitzoglou, K. Tarchanidis, A. Mitropoulos, and A. Gasteratos, “Unsupervised human detection with an embedded vision system on a fully autonomous uav for search and rescue operations,” _Sensors_ , vol. 19, no. 16, p. 3542, 2019.
  * [3] R. Geraldes, A. Goncalves, T. Lai, M. Villerabel, W. Deng, A. Salta, K. Nakayama, Y. Matsuo, and H. Prendinger, “Uav-based situational awareness system using deep learning,” _IEEE Access_ , vol. 7, pp. 122 583–122 594, 2019.
  * [4] K. T. San, S. J. Mun, Y. H. Choe, and Y. S. Chang, “Uav delivery monitoring system,” in _MATEC Web of Conferences_ , vol. 151. EDP Sciences, 2018, p. 04011.
  * [5] P. Zhu, L. Wen, D. Du, X. Bian, H. Ling, Q. Hu, Q. Nie, H. Cheng, C. Liu, X. Liu _et al._ , “Visdrone-det2018: The vision meets drone object detection in image challenge results,” in _Proceedings of the European Conference on Computer Vision (ECCV) Workshops_ , 2018, pp. 0–0.
  * [6] L. A. Varga, B. Kiefer, M. Messmer, and A. Zell, “Seadronessee: A maritime benchmark for detecting humans in open water,” _arXiv preprint arXiv:2105.01922_ , 2021.
  * [7] GDPR. General data protection regulation. [Online]. Available: https://gdpr.eu/
  * [8] ExposingAI. Ms-celeb-1m privacy infringment. [Online]. Available: https://exposing.ai/msceleb/
  * [9] D. Du, Y. Qi, H. Yu, Y. Yang, K. Duan, G. Li, W. Zhang, Q. Huang, and Q. Tian, “The unmanned aerial vehicle benchmark: Object detection and tracking,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 370–386.
  * [10] B. Kiefer, M. Messmer, and A. Zell, “Leveraging domain labels for object detection from uavs,” _arXiv preprint arXiv:2101.12677_ , 2021.
  * [11] M. Messmer, B. Kiefer, and A. Zell, “Gaining scale invariance in uav bird’s eye view object detection by adaptive resizing,” _arXiv preprint arXiv:2101.12694_ , 2021.
  * [12] A. Kar, A. Prakash, M.-Y. Liu, E. Cameracci, J. Yuan, M. Rusiniak, D. Acuna, A. Torralba, and S. Fidler, “Meta-sim: Learning to generate synthetic datasets,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 4551–4560.
  * [13] K. Srivastava, A. K. Singh, and G. M. Hegde, “Multi modal semantic segmentation using synthetic data,” _arXiv preprint arXiv:1910.13676_ , 2019\.
  * [14] B. Hurl, K. Czarnecki, and S. Waslander, “Precise synthetic image and lidar (presil) dataset for autonomous vehicle perception,” in _2019 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2019, pp. 2522–2529.
  * [15] M. Fonder and M. V. Droogenbroeck, “Mid-air: A multi-modal dataset for extremely low altitude drone flights,” in _Conference on Computer Vision and Pattern Recognition Workshop (CVPRW)_ , June 2019.
  * [16] R. Games. Grand theft auto v. [Online]. Available: https://www.rockstargames.com/de/games/V
  * [17] M. Johnson-Roberson, C. Barto, R. Mehta, S. N. Sridhar, K. Rosaen, and R. Vasudevan, “Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks?” _arXiv preprint arXiv:1610.01983_ , 2016.
  * [18] M. Angus, M. ElBalkini, S. Khan, A. Harakeh, O. Andrienko, C. Reading, S. Waslander, and K. Czarnecki, “Unlimited road-scene synthetic annotation (ursa) dataset,” in _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2018, pp. 985–992.
  * [19] X. Yue, B. Wu, S. A. Seshia, K. Keutzer, and A. L. Sangiovanni-Vincentelli, “A lidar point cloud generator: from a virtual world to autonomous driving,” in _Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval_ , 2018, pp. 458–464.
  * [20] S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: High-fidelity visual and physical simulation for autonomous vehicles,” in _Field and service robotics_. Springer, 2018, pp. 621–635.
  * [21] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: An open urban driving simulator,” in _Conference on robot learning_. PMLR, 2017, pp. 1–16.
  * [22] W. Qiu and A. Yuille, “Unrealcv: Connecting computer vision to unreal engine,” in _European Conference on Computer Vision_. Springer, 2016, pp. 909–916.
  * [23] S. I. Nikolenko _et al._ , “Synthetic data for deep learning,” _arXiv preprint arXiv:1909.11512_ , vol. 3, 2019.
  * [24] H. Fan, L. Wen, D. Du, P. Zhu, Q. Hu, H. Ling, M. Shah, B. Wang, B. Dong, D. Yuan _et al._ , “Visdrone-sot2020: The vision meets drone single object tracking challenge results,” in _European Conference on Computer Vision_. Springer, 2020, pp. 728–749.
  * [25] Z. Pei, X. Qi, Y. Zhang, M. Ma, and Y.-H. Yang, “Human trajectory prediction in crowded scene using social-affinity long short-term memory,” _Pattern Recognition_ , vol. 93, pp. 273–282, 2019.
  * [26] M. Mueller, N. Smith, and B. Ghanem, “A benchmark and simulator for uav tracking,” in _European conference on computer vision_. Springer, 2016, pp. 445–461.
  * [27] M.-R. Hsieh, Y.-L. Lin, and W. H. Hsu, “Drone-based object counting by spatially regularized regional proposal network,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 4145–4153.
  * [28] T. N. Mundhenk, G. Konjevod, W. A. Sakla, and K. Boakye, “A large contextual dataset for classification, detection and counting of cars with deep learning,” in _European Conference on Computer Vision_. Springer, 2016, pp. 785–800.
  * [29] S. Li and D.-Y. Yeung, “Visual object tracking for unmanned aerial vehicles: A benchmark and new motion models,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 31, no. 1, 2017.
  * [30] R. Krajewski, J. Bock, L. Kloeker, and L. Eckstein, “The highd dataset: A drone dataset of naturalistic vehicle trajectories on german highways for validation of highly automated driving systems,” in _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2018, pp. 2118–2125.
  * [31] J. C. van Gemert, C. R. Verschoor, P. Mettes, K. Epema, L. P. Koh, and S. Wich, “Nature conservation drones for automatic localization and counting of animals,” in _European Conference on Computer Vision_. Springer, 2014, pp. 255–270.
  * [32] F. Ofli, P. Meier, M. Imran, C. Castillo, D. Tuia, N. Rey, J. Briant, P. Millet, F. Reinhard, M. Parkan _et al._ , “Combining human computing and machine learning to make sense of big (aerial) data for disaster response,” _Big data_ , vol. 4, no. 1, pp. 47–59, 2016.
  * [33] I. Bozcan and E. Kayacan, “Au-air: A multi-modal unmanned aerial vehicle dataset for low altitude traffic surveillance,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 8504–8510.
  * [34] F. Kong, B. Huang, K. Bradbury, and J. Malof, “The synthinel-1 dataset: a collection of high resolution synthetic overhead imagery for building segmentation,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 1814–1823.
  * [35] J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell, “Cycada: Cycle-consistent adversarial domain adaptation,” in _International conference on machine learning_. PMLR, 2018, pp. 1989–1998.
  * [36] G. French, M. Mackiewicz, and M. Fisher, “Self-ensembling for visual domain adaptation,” _arXiv preprint arXiv:1706.05208_ , 2017.
  * [37] P. Li, X. Liang, D. Jia, and E. P. Xing, “Semantic-aware grad-gan for virtual-to-real urban scene adaption,” _arXiv preprint arXiv:1801.01726_ , 2018.
  * [38] Y. Zou, Z. Yu, B. Kumar, and J. Wang, “Unsupervised domain adaptation for semantic segmentation via class-balanced self-training,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 289–305.
  * [39] Y. Chen, W. Li, C. Sakaridis, D. Dai, and L. Van Gool, “Domain adaptive faster r-cnn for object detection in the wild,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3339–3348.
  * [40] A. Prakash, S. Boochoon, M. Brophy, D. Acuna, E. Cameracci, G. State, O. Shapira, and S. Birchfield, “Structured domain randomization: Bridging the reality gap by context-aware synthetic data,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 7249–7255.
  * [41] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in _2017 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2017, pp. 23–30.
  * [42] Y.-H. Tsai, W.-C. Hung, S. Schulter, K. Sohn, M.-H. Yang, and M. Chandraker, “Learning to adapt structured output space for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7472–7481.
  * [43] S. R. Richter, H. A. AlHaija, and V. Koltun, “Enhancing photorealism enhancement,” _arXiv preprint arXiv:2105.04619_ , 2021.
  * [44] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2223–2232.
  * [45] Airbus. Airbus ship dataset. [Online]. Available: https://www.kaggle.com/c/airbus-ship-detection
  * [46] W. Shao, R. Kawakami, R. Yoshihashi, S. You, H. Kawase, and T. Naemura, “Cattle detection and counting in uav images based on convolutional neural networks,” _International Journal of Remote Sensing_ , vol. 41, no. 1, pp. 31–52, 2020.
  * [47] B. Hurl. Deepgtav-presil code. [Online]. Available: https://github.com/bradenhurl/DeepGTAV-PreSIL
  * [48] A. Ruano. Deepgtav code. [Online]. Available: https://github.com/aitorzip/DeepGTAV
  * [49] A. Blade. Script hook v. [Online]. Available: http://www.dev-c.com/gtav/scripthookv/
  * [50] A. Ruano. Deepgtav-vpilot code. [Online]. Available: https://github.com/aitorzip/VPilot
  * [51] Gta vision export. [Online]. Available: https://github.com/umautobots/GTAVisionExport
  * [52] dorahamu. Gtav mod - simple increase traffic and pedestrians. [Online]. Available: https://de.gta5-mods.com/misc/simple-increase-traffic-and-pedestrian
  * [53] VenJam1n. Gtav mod - no chromatric aberration and lens distortion. [Online]. Available: https://de.gta5-mods.com/misc/no-chromatic-aberration-lens-distortion-1-41
  * [54] F. S. Division. Gtav mod - heap limit adjuster. [Online]. Available: https://de.gta5-mods.com/tools/heap-limit-adjuster-600-mb-of-heap
  * [55] Nvidia. Nvidia geforce experience. [Online]. Available: https://www.nvidia.com/de-de/geforce/geforce-experience/
  * [56] P. Zhu, L. Wen, X. Bian, H. Ling, and Q. Hu, “Vision meets drones: A challenge,” _arXiv preprint arXiv:1804.07437_ , 2018.
  * [57] M. Tan, R. Pang, and Q. V. Le, “Efficientdet: Scalable and efficient object detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 10 781–10 790.
  * [58] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 779–788.
  * [59] J. Redmon and A. Farhadi, “YOLO9000: better, faster, stronger,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7263–7271.
  * [60] ——, “Yolov3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [61] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “YOLOv4: Optimal Speed and Accuracy of Object Detection,” _arXiv preprint arXiv:2004.10934_ , 2020\.
  * [62] G. Jocher, L. Changyu, A. Hogan, L. Yu, changyu98, P. Rai, and T. Sullivan, “ultralytics/yolov5: Initial Release,” Jun. 2020. [Online]. Available: https://doi.org/10.5281/zenodo.3908560
  * [63] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [64] P. with Code. Object detection on coco test-dev. [Online]. Available: https://paperswithcode.com/sota/
  * [65] S. GmbH, “A pytorch implementation of efficientdet object detection,” _https://github.com/signatrix/efficientdet_ , 2020.
  * [66] G. Jocher, A. Stoken, J. Borovec, NanoCode012, A. Chaurasia, TaoXie, L. Changyu, A. V, Laughing, tkianai, yxNONG, A. Hogan, lorenzomammana, AlexWang1900, J. Hajek, L. Diaconu, Marc, Y. Kwon, oleg, wanghaoyang0106, Y. Defretin, A. Lohia, ml5ah, B. Milanko, B. Fineran, D. Khromov, D. Yiwei, Doug, Durgesh, and F. Ingham, “ultralytics/yolov5: v5.0 - YOLOv5-P6 1280 models, AWS, Supervise.ly and YouTube integrations,” Apr. 2021. [Online]. Available: https://doi.org/10.5281/zenodo.4679653
  * [67] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1492–1500.
  * [68] G. Varol, J. Romero, X. Martin, N. Mahmood, M. J. Black, I. Laptev, and C. Schmid, “Learning from synthetic humans,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 109–117.
  * [69] B. Kiefer and M. Messmer. Seadronessee benchmark. [Online]. Available: https://seadronessee.cs.uni-tuebingen.de/
  * [70] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.