# Progressive Transformation Learning For Leveraging Virtual Images in
Training

Yi-Ting Shen⋆,1, Hyungtae Lee⋆,2, Heesung Kwon2, Shuvra Shikhar Bhattacharyya1

⋆ equal contribution
1 University of Maryland, College Park, 2 DEVCOM Army Research Laboratory,
Adelphi
{ytshen<EMAIL_ADDRESS>{hyungtae.lee<EMAIL_ADDRESS>
###### Abstract

To effectively interrogate UAV-based images for detecting objects of interest,
such as humans, it is essential to acquire large-scale UAV-based datasets that
include human instances with various poses captured from widely varying
viewing angles. As a viable alternative to laborious and costly data curation,
we introduce Progressive Transformation Learning (PTL), which gradually
augments a training dataset by adding transformed virtual images with enhanced
realism. Generally, a virtual2real transformation generator in the conditional
GAN framework suffers from quality degradation when a large domain gap exists
between real and virtual images. To deal with the domain gap, PTL takes a
novel approach that progressively iterates the following three steps: 1)
select a subset from a pool of virtual images according to the domain gap, 2)
transform the selected virtual images to enhance realism, and 3) add the
transformed virtual images to the training set while removing them from the
pool. In PTL, accurately quantifying the domain gap is critical. To do that,
we theoretically demonstrate that the feature representation space of a given
object detector can be modeled as a multivariate Gaussian distribution from
which the Mahalanobis distance between a virtual object and the Gaussian
distribution of each object category in the representation space can be
readily computed. Experiments show that PTL results in a substantial
performance increase over the baseline, especially in the small data and the
cross-domain regime.

##  1 Introduction

Training an object detector usually requires a large-scale training image set
so that the detector can acquire the ability to detect objects’ diverse
appearances. This desire for a large-scale training set is bound to be greater
for object categories with more diverse appearances, such as the human
category whose appearances vary greatly depending on its pose or viewing
angles. Moreover, a person’s appearance becomes more varied in images captured
by an unmanned aerial vehicle (UAV), leading to a wide variety of camera
viewing angles compared to ground-based cameras, making the desire for a
large-scale training set even greater. In this paper, we aim to satisfy this
desire, especially when the availability of UAV-based images to train a human
detector is scarce, where this desire is more pressing.

As an intuitive way to expand the training set, one might consider
synthesizing virtual images to imitate real-world images by controlling the
optical and physical conditions in a virtual environment. Virtual images are
particularly useful for UAV-based object detection since abundant object
instances can be rendered with varying UAV locations and camera viewing angles
along with ground-truth information (e.g., bounding boxes, segmentation masks)
that comes free of charge. Therefore, a large-scale virtual image set covering
diverse appearances of human subjects that are rarely shown in existing UAV-
based object detection benchmarks (Zhu et al., 2021; Barekatain et al., 2017;
ICG, ) can be conveniently acquired by controlling rendering entities and
parameters in a virtual environment, such as poses, camera viewing angles, and
illumination conditions.

To make virtual images usable for training real-world object detection models,
recent works (Hoffman et al., 2018; Mu et al., 2020; Li & Lee, 2021; Qiu et
al., 2021) transform virtual images to look realistic. They commonly use the
virtual2real generator (Mirza & Osindero, 2014) trained with the conditional
GAN framework to transform images in the source domain to have the visual
properties of images in the target domain. Here, virtual and real images are
treated as the source and target domains, respectively. However, the large
discrepancy in visual appearance between the two domains, referred to as the
“domain gap”, result in the degraded transformation quality of the generator.
In fact, the aforementioned works using virtual images validate their methods
where the domain gap is not large (e.g., digit detection (Hoffman et al.,
2018)) or when additional information is available (e.g., animal pose
estimation with additional keypoint information (Mu et al., 2020; Li & Lee,
2021; Qiu et al., 2021)). In our case, real and virtual humans in UAV-based
images inevitably have a large domain gap due to the wide variety of human
appearances.

To address the large domain gap, one critical question inherent in our task is
_how one can accurately measure the domain gap._ Consequently, we estimate the
domain gap in the representation space of a human detector trained on the real
images. The representation space of the detector is learned such that test
samples, which have significantly different properties than training samples
from the perspective of the detector, are located away from the training
samples. In this paper, we show that the feature distribution of object
entities belonging to a certain category, such as the human category, in the
representation space can be modeled with a multivariate Gaussian distribution
if the following two conditions are met: i) the detector uses the sigmoid
function to normalize the final output of the detector and ii) the
representation space is constructed using the output of the penultimate layer
of the detector. This idea is inspired by Lee et al. (2018), which shows that
softmax-based classifiers can be modeled as multivariate Gaussian
distributions. In this paper, we show that the proposition is also applicable
to sigmoid-based classifiers, which are widely used by state-of-the-art (SOTA)
detectors. Based on this modeling, when the two aforementioned conditions are
met, the human category in the representation space can be represented by two
parameters (i.e., mean and covariance) of a multivariate Gaussian distribution
that can be computed on the training images. With the empirically calculated
mean and covariance, the domain gap from a single virtual human image to real
human images (i.e., the training set) can be measured using the Mahalanobis
distance (Mahalanobis, 1936).

Figure 1: Overview of Progressive Transformation Learning (PTL).

To add virtual images to the training set to include more diverse appearances
of objects while preventing the transformation quality degradation caused by
large domain gaps, we introduce _Progressive Transformation Learning (PTL)_
(Figure 1). PTL progressively expands the training set by adding virtual
images through iterating the three steps: 1) transformation candidate
selection, 2) virtual2real transformation, and 3) set update. When selecting
transformation candidates from a pool of virtual images, we use weighted
random sampling, which gives higher weights to images with smaller domain
gaps. The weight takes an exponential term with one hyperparameter controlling
the ratio between images with smaller domain gaps and images with more diverse
appearances. Then, the virtual2real transformation generator is trained via
the conditional GAN framework, taking the selected virtual images (i.e.,
transformation candidates) as the “source” and the images in the training set
as the “target”. After transforming the transformation candidates by applying
the virtual2real transformation generator, the training set is expanded with
the transformed candidates while the original candidates are excluded from the
pool of virtual images.

The main contribution of this paper is that we have validated the utility of
virtual images in augmenting training data via PTL coupled with carefully
designed comprehensive experiments. We first use the task of low-shot
learning, where adequately expanding datasets has notable effects.
Specifically, PTL provides better accuracy on three UAV-view human detection
benchmarks than other previous works that leverage virtual images in training,
as well as methods that only use real images. Then, we validate PTL on the
cross-domain detection task where training and test data are from distinct
datasets, the problem we frequently face in real-world scenarios. Similarly,
PTL consistently presents significantly better accuracy than the baseline. The
experimental results indicate that a high-performance human detection model
can be effectively learned via PTL, even with the significant lack of training
data representing real operational environments.

##  2 Related Works

Leveraging virtual images in training. In this section, we have listed
previous works that demonstrate how virtual images can play a role in a
variety of real-world applications when used for training. In fact, virtual
images are desirable for model training as large-scale labeled datasets can be
built virtually almost free of charge. Unfortunately, when virtual images are
used for model training without proper care, it is shown that the performance
improvement is limited due to the domain gap between the virtual images and
the real test images. Generally, previous works leveraging virtual images
during model training can be summarized into three approaches according to how
they exploit the advantages of virtual images and address the challenges of
using virtual images.

The most intuitive and widely used approach of using virtual images is to pre-
train a model on virtual images and fine-tune the pre-trained model on real
images acquired in the same domain as the test images (Handa et al., 2016; Kim
et al., 2016; Gaidon et al., 2016; Fabbri et al., 2018; Varol et al., 2018; Mu
et al., 2020; Mishra et al., 2022; Guo et al., 2022; Jin et al., 2022; Baek &
Shim, 2022). This approach aims to avoid the domain gap by fine-tuning the
model on real images acquired under the same conditions and environments of
test images. While the first approach seeks to use the representative
capability learned from _large-scale_ virtual datasets, the second approach
seeks to exploit additional information that can be easily _labeled_ on
virtual images. For example, Liu et al. (2022) annotates part segmentation
maps when acquiring virtual vehicle images, and uses the part segmentation
results from the pre-trained model on the virtual images during the fine-
tuning process. Similarly, Yan et al. (2022) uses depth and semantic
information labeled when acquiring virtual images.

The third approach directly builds training batches consisting of both virtual
images and real images. Ros et al. (2016) and Richter et al. (2016) adopt the
most naive approach to build a batch by randomly selecting a fixed number of
images from each of the real and virtual image sets. However, even if the
number of virtual images is several orders of magnitude greater than the
number of real training images, this approach does not provide remarkably
better accuracy and may even provide worse accuracy than its counterparts
using only real images for training. In this case, the effect of using virtual
images during model training does not appear as expected because the large
domain gap between the real (test) images and the virtual images is not
adequately addressed. In this paper, we also use virtual images directly for
model training while considering appropriately reducing this domain gap.

Progressive learning. Progressive learning is a machine learning strategy that
continuously trains a model from easy to hard tasks, primarily for the purpose
of training stabilization or fast optimization. One of the most common
applications for progressive learning is incrementally increasing the network
capacity to improve network capability. The most intuitive approach in this
category is to gradually increase the network size (e.g., depth or width) to
alleviate the training difficulty of very deep networks (Fahlman & Lebiere,
1989; Bengio et al., 2006; Smith et al., 2016; Chen et al., 2016; Wei et al.,
2016; Gong et al., 2019; Li et al., 2022a). Conversely, Zhang & He (2020) uses
progressive learning in the direction of reducing the network size for fast
training. Progressive learning is also used in GAN frameworks to enhance the
generator’s ability to transform input images of larger resolution (Karras et
al., 2018). Curriculum learning (Bengio et al., 2009; Spitkovsky et al., 2009;
Khan et al., 2011; Graves et al., 2017), which continuously raises the level
of training from easy to difficult samples, also falls into this category.

Progressive learning is also used to deal with the scalability of datasets
that are incompletely labeled. In a semi-supervised learning task where
unlabeled samples can be used in training, Wang et al. (2017), Wang et al.
(2022a), and Cao et al. (2022) adopt progressive learning by gradually
increasing the number of unlabeled data used for training. Self-learning in
Kumar et al. (2010) and Kaul et al. (2022) uses progressive learning by
repeating the two steps, assigning labels depending on the current detector
and updating the current detector with these labels.

Our method can be seen as similar to the second approach in that it also
intend to expand the training dataset. However, our method uses progressive
learning to reliably add realistically transformed virtual images to the
training dataset by avoiding quality degradation of the transformation, which
has never been attempted before using the progressive learning strategy.

##  3 Method

###  3.1 Measuring the Domain Gap between Real and Virtual Images

Modeling training set with multivariate Gaussian distribution. The purpose of
adopting progressive transformation learning, which progressively expands the
training set with a subset of the realistically transformed virtual images
instead of expanding it with the full set at once, is to avoid the significant
domain gap between the real and virtual images when training the
transformation generator. Here, the domain gap is measured in the
representation space of the detector, which is learned so that two samples
with different properties from the perspective of the detector are separated
far from each other.

In general, the representation space of the detector refers to the space
formed by the output of the penultimate layer of the detector since all layers
of the detector except for the last layer can be transferred for different
downstream tasks (He et al., 2020; Chen et al., 2020; Grill et al., 2020). In
Lee et al. (2018), it is shown that for the softmax-based classifier, the
distribution of each category in the representation space can be modeled as a
multivariate Gaussian distribution. Object detector generally uses the sigmoid
function (i.e., $f_{\mathit{sigmoid}}({\bf
x})=1/(1+\exp(\scalebox{0.75}[1.0]{$-$}{\bf w}_{c}^{T}{\bf
x}\scalebox{0.75}[1.0]{$-$}b_{c})$ for the category $c$), which does not
consider outputs for other categories, instead of the softmax function (i.e.,
$f_{\mathit{softmax}}({\bf x})=\exp({\bf w}_{c}^{T}{\bf
x}+b_{c})/\sum_{c^{\prime}}{\exp({\bf w}_{c^{\prime}}^{T}{\bf
x}+b_{c^{\prime}})}$ for the category $c$) that competes for outputs for all
categories to normalize the model output to [0 1]. This is because, unlike
classification, the detection task must take into account that two or more co-
located objects may be active on a single output. In the supplementary
material, we show that even for the sigmoid-based detector, the distribution
of each category in the representation space can also be modeled as a
multivariate Gaussian distribution.

Specifically, let ${\bf x}\in\mathcal{X}$ and
$y=\\{y_{c}\\}_{c=1,\cdots,C}\in\mathcal{Y},y_{c}\in\\{0,1\\}$ be an input and
its label, respectively. Then, the representation space of the sigmoid-based
detector can be expressed as follows:

$P(f({\bf x})|y_{c}=1)\sim\mathcal{N}(f({\bf
x})|{\bf\mu}_{c},{\bf\Sigma}_{c}),$ (1)

where $f(\cdot)$ denotes the output of the penultimate layer of the detector.
$\mu_{c}$ and $\Sigma_{c}$ are the mean and the covariance of the multivariate
Gaussian distribution for the category $c$. ${\bf\mu}_{c}$ and
${\bf\Sigma}_{c}$ can be calculated over the entire set of training images as
follows:

${\bf\mu}_{c}=\frac{1}{|\text{D}_{c}|}\sum_{{\bf x}\in\text{D}_{c}}{f({\bf
x})},~{}~{}~{}~{}~{}~{}{\bf\Sigma}_{c}=\frac{1}{|\text{D}_{c}|}\sum_{{\bf
x}\in\text{D}_{c}}{\left(f({\bf x})-{\bf\mu}_{c}\right)\left(f({\bf
x})-{\bf\mu}_{c}\right)^{\top}},~{}$ (2)

where $\text{D}_{c}$ is the set of instances for the category $c$.
Practically, any detection whose IoU with the groundtruth of the category $c$
is greater than 0.5 belongs to $\text{D}_{c}$.

Measuring domain gap. After ${\bf\mu}_{c}$ and ${\bf\Sigma}_{c}$ are
empirically calculated to represent $\text{D}_{c}$, the domain gap between a
new image ${\bf x}_{\mathit{new}}$ and $\text{D}_{c}$ can be measured using
the Mahalanobis distance, as follows:

$d({\bf x}_{\mathit{new}})=\left(f({\bf
x}_{\mathit{new}})-{\bf\mu}_{c}\right)^{\top}{\bf\Sigma}_{c}^{\scalebox{0.75}[1.0]{$-$}1}\left(f({\bf
x}_{\mathit{new}})-{\bf\mu}_{c}\right).$ (3)

This measurement of the domain gap is highly dependent on the detector’s
ability to detect objects in the image. It is commonly known that the
detection capability of a detector is greatly affected by the image size as
well as the object appearance in the image. To mitigate the effect of image
size on measuring the domain gap, the Mahalanobis distance for ${\bf x}_{new}$
is calculated at multiple image scales, and the minimum distance is used as
the domain gap, as follows:

$d({\bf x}_{\mathit{new}})=\min_{s\in S}(\\{d({\bf
x}^{s}_{\mathit{new}})\\}),$ (4)

where ${\bf x}^{s}_{\mathit{new}}$ is the resized image of ${\bf
x}_{\mathit{new}}$ to be $s$$\times$$s$. $S$ is the set of resizing factors.
In our experiments, we use $S=\\{128,256,384,512\\}$.

###  3.2 Progressive Transformation Learning

Figure 2: Progressive Transformation Learning (PTL) pipeline. The red arrow
indicates the processing flow of the virtual images selected to be added to
the training set.

Our objective is to expand the training set consisting of real images by
adding virtual images which are transformed to intimate real images. The
virtual2real transformation can be performed by a generator trained by
treating virtual images and real images as “source” and “target”,
respectively, in the conditional GAN framework. Inevitably, the transformation
quality of the trained generator is degraded when the domain gap between the
source domain and the target domain is large. To prevent the degraded
transformation quality due to the large domain gap, we introduce _Progressive
Transformation Learning (PTL)_ , which progressively and iteratively expands
the training set with a subset of virtual images carefully selected to avoid
the large domain gap. PTL goes through three steps for each iteration (Fig 2):
i) sampling a subset of virtual images from a virtual image pool by giving
heavier weights to images close to the current training set (Transformation
candidate selection), ii) transforming the selected images to be realistic
(Virtual2real transformation), and iii) adding the transformed images to the
training set while excluding the selected images from the virtual image pool
(Set update). The details of each step are described next.

Transformation candidate selection. When selecting transformation candidates,
we must consider two conflicting claims simultaneously: i) to suit the purpose
of PTL, virtual images with a small domain gap should be selected, but ii) to
suit the purpose of expanding the training set, virtual images with
appearances that rarely appear in the training set, which usually implies a
large domain gap, should also be selected.

To jointly consider these two claims, we use weighted random sampling. The
sampling weight takes the exponential term which gives higher weights to
virtual images with smaller domain gaps, while introducing one hyper-parameter
$\tau$ to control the amplitude of the weights, as follows:

$w({\bf x})=\exp{\left(\scalebox{0.75}[1.0]{$-$}\frac{d({\bf
x})}{\tau}\right)},$ (5)

where $d({\bf x})$ is the Mahalanobis distance, which is used to measure the
domain gap of ${\bf x}$ from the current training set (eq 4). Intuitively,
using a small $\tau$ allows a more frequent selection of images with smaller
domain gaps by giving them larger weights than using a large $\tau$. (We use
$\tau$=5.0 throughout all experiments; the detection accuracy is found to be
not very sensitive to $\tau$.)

In practice, transformation candidates are selected from virtual image pool
through the following four steps: i) training the human detector $f_{D}^{t}$
on the current training set of real images ${\bf\text{R}}^{t}$, ii)
calculating ${\bf\mu}^{t}$ and ${\bf\Sigma}^{t}$ on ${\bf\text{R}}^{t}$ as in
eq. 2, iii) calculating weights for each image in the current set of virtual
images ${\bf\text{V}}^{t}$ as in eq. 5, and iv) applying weighted random
sampling to ${\bf\text{V}}^{t}$ to select a pre-defined number $n$ of
transformation candidates. (We use $n$=100 throughout all experiments.)

Virtual2Real transformation. In line with the goal of this paper to obtain a
human detector that can identify humans with diverse appearances captured by a
UAV, we design the virtual2real transformation to focus on the person region
rather than the background of the selected virtual images. To do so, we crop
the person region in the virtual image, apply the transformation only to this
region, and segment the transformed person back to the original background.
For accurate segmentation, the pixel-wise segmentation mask is required.
Obtaining such pixel-wise segmentation mask at no cost is another benefit of
using virtual images.

The conditional GAN framework (Mirza & Osindero, 2014), in which the generator
is trained to transform a given input image from source styles into target
styles, is widely used to transform virtual images to look like real images
(Isola et al., 2017; Li & Lee, 2021). Among many variants of conditional GANs,
we use CycleGAN (Zhu et al., 2017) in which the generator is trained to
minimize the reconstruction error between the input image and the
reconstructed image transformed back to the original style of the input image
after the initial transformation to the target style. It is shown in Mirza &
Osindero (2014) that the transformation with CycleGAN is likely to maintain
the original object pose while changing detailed styles such as patterns
(e.g., transforming a white horse into a zebra in the same pose). We intend to
borrow this characteristic of CycleGAN to transform virtual images in the
direction that makes the detailed styles realistic while maintaining the
overall human appearances, which depend on various viewing angles or human
poses.

In practice, the virtual2real transformation generator $f_{G}^{t}$ is trained
using the CycleGAN framework by treating the selected transformation
candidates ${\bf\text{C}_{\text{V}}}^{t}$ and the current set of real images
${\bf\text{R}}^{t}$ as “source” and “target”, respectively. Then,
${\bf\text{C}_{\text{V}}}^{t}$ are transformed to realistic transformed images
${\bf\text{C}_{\text{R}}}^{t}$ by applying the virtual2real transformation
generator.

Set update. After the transformed images ${\bf\text{C}_{\text{R}}}^{t}$ are
acquired from the selected transformation candidates
${\bf\text{C}_{\text{V}}}^{t}$, PTL updates the current real image set
${\bf\text{R}}^{t}$ and the current virtual image set ${\bf\text{V}}^{t}$ as
follows:

${\bf\text{R}}^{t+1}={\bf\text{R}}^{t}\cup{\bf\text{C}_{\text{R}}}^{t}~{}~{}~{}~{}~{}~{}{\bf\text{V}}^{t+1}={\bf\text{V}}^{t}/{\bf\text{C}_{\text{V}}}^{t}$
(6)

When this progressive learning is terminated, the final human detection model
can be acquired by training on the final set of real images.

In practice, the first two steps of PTL are applied to the tightly cropped
image region of human region, but in the ‘set update’ step, the entire image
including the human region and the background is added to the training set for
training the human detector. More precisely, when training the virtual2real
transformation generator, the tightly cropped image region around each human
from the training images are used as the “target”.

##  4 Experiments

Datasets and evaluation metrics. We perform experiments on three real UAV-
based datasets, VisDrone (Zhu et al., 2021), Okutama-Action (Barekatain et
al., 2017), and ICG (ICG, ), and one virtual dataset, Archangel-Synthetic
(Shen et al., 2022), all including human instances. Archangel-Synthetic
consists of various virtual characters with different poses across a range of
altitudes and circle radii with different camera pitch angles (i.e., 17.3K
images with eight characters, three poses, ten altitudes, six circle radii,
and twelve camera pitch angles). Each image in Archangel-Synthetic accompanies
metadata about the above imaging conditions, allowing us to analyze how the
feature distributions of virtual characters evolve with respect to these
imaging conditions when PTL progresses. We use<EMAIL_ADDRESS>and AP@[.5:.95] as
evaluation metrics for all experiments.

Detector. For the detector, we use RetinaNet (Lin et al., 2017) that uses the
feature pyramid network (FPN) to provide a rich multi-scale feature pyramid
and processes features at all scale levels with the same subnetwork
responsible for the final classification and bounding box regression. It is
important to use the same subnetwork across all scale levels since the domain
gap for each virtual image should be measured in a shared representation space
regardless of the image size. Most other object detectors using FPN (e.g., SSD
(Liu et al., 2016) and v4 or later versions of YOLO (Bochkovskiy et al., 2020;
Jocher, 2020; Li et al., 2022b; Wang et al., 2022b)) use different subnetworks
at each scale level. However, PTL is not structurally limited to RetinaNet as
it can be used with any detector with minor modifications, such as adding one
shared layer across all scale levels.

###  4.1 Properties of Progressive Learning

iter. 1 | iter. 2 | iter. 3 | iter. 4 | iter. 5
---|---|---|---|---
|  |  |  |
|  |  |  |

Figure 3: Analysis of the use of virtual images when PTL progresses. The
figures in the top row show the accumulated distribution of transformation
candidates with respect to camera locations (i.e., altitude and rotation
circle radius from the target human in $x$ and $y$ axes, respectively) for
each PTL iteration. Darker bins indicate that more virtual images have been
added to the training set. The figures in the bottom row ($x$ axis: domain
gap, $y$ axis: the corresponding number of virtual images) show the domain gap
distribution of virtual images measured by eq. 4. These figures are collected
from the experimental setup of using 50 real images from the VisDrone dataset
for training.

Which virtual images are selected for each PTL iteration? The top row of
Figure 3 shows the change in the accumulated distribution of virtual images
added to the training set via each PTL iteration with respect to camera
locations. By examining the distribution, we can identify which camera
locations of the virtual images contribute more to the training set at each
PTL iteration.

It is observed that after the 1st PTL iteration, most virtual images included
in the training set are taken from the camera locations close to the human
subjects. As PTL progresses, the camera locations of the virtual images
included in the training set gradually spread across the UAV altitudes and
rotation circle radii. Consequently, after the 5th PTL iteration, transformed
virtual images with diverse appearances from much broader camera locations are
included in the final training set. This demonstrates that the proposed
transformation candidate selection process is adequately designed to consider
the two conflicting claims together.

How close does the domain gap get as PTL progresses? The bottom row of Figure
3 shows the domain gap between the virtual images and the training set at each
PTL iteration. We can observe the domain gap distribution of virtual images to
the training set gradually becomes narrower and smaller. Additionally, some
virtual images which have not been included in the training set also appear in
the long tail of the distribution.

<EMAIL_ADDRESS>VisDrone |  AP@[.5:.95]: VisDrone |<EMAIL_ADDRESS>Okutama-Action |  AP@[.5:.95]: Okutama-Action |<EMAIL_ADDRESS>ICG |  AP@[.5:.95]: ICG
---|---|---|---|---|---
|  |  |  |  |

Figure 4: Learning curves of the two metrics<EMAIL_ADDRESS>and AP@[.5:.95]) on the
three datasets

Accuracy variation as PTL evolves. Figure 4 shows how the accuracy changes as
PTL progresses on the three datasets. Overall, for<EMAIL_ADDRESS>accuracy increases
rapidly until the 3rd iteration and does not change significantly thereafter.
On the other hand, for AP@[.5:.95], accuracy continues to increase even after
the 3rd iteration on the Okutama-Action and ICG datasets. This can be
interpreted such that human bounding boxes are estimated more accurately as
PTL progresses on these two datasets.

###  4.2 Results on Low-shot Learning

Baselines. In this section, we compare PTL with the method utilizing only real
images for training (i.e., ‘baseline’) and other three methods also leveraging
virtual images in conjunction with real images for training (i.e., ‘pretrain-
finetune’, ‘naive merge’, and ‘naive merge w/ transform’) in terms of human
detection accuracy. ‘Pretrain-finetune’ is the strategy to train a model by
pre-training on virtual images and then fine-tuning on real images, which is
the most widely used approach leveraging virtual images in previous works
(Handa et al., 2016; Kim et al., 2016; Gaidon et al., 2016; Fabbri et al.,
2018; Varol et al., 2018; Mu et al., 2020; Mishra et al., 2022; Guo et al.,
2022; Jin et al., 2022; Baek & Shim, 2022). ’Naive merge’ is the strategy that
uses a training set naively merging from real and virtual images for model
training, which has also been used in previous works (Ros et al., 2016;
Richter et al., 2016). ‘Naive merge w/ transform’ naively adds transformed
virtual images to the training set, where the transformation generator is
trained with the CycleGAN framework by considering all virtual images and all
real images as “source” and “target”.

Table 1: Low-shot learning accuracy with 20 and 50 real images<EMAIL_ADDRESS>and
AP@[.5:.95] are reported in each bin. For PTL, the margin from the baseline
accuracy is shown below the reported accuracy. The best accuracy for each
setting is shown in bold. R and V denote the set of real images and the set of
virtual images, respectively.

method | train set | VisDrone | Okutama-Action | ICG
---|---|---|---|---
20 | 50 | 20 | 50 | 20 | 50
baseline | R |  3.74/ 1.09 |  6.42/ 1.86 |  41.61/ 11.23 |  49.84/ 13.76 |  49.35/ 14.69 |  66.75/ 23.91
pretrain-finetune | R+V |  4.99/ 1.46 |  6.25/ 1.99 |  44.57/ 12.78 |  49.06/ 15.08 |  66.92/ 26.67 |  68.41/ 29.73
naive merge | R+V |  3.41/ 1.02 |  5.18/ 1.65 |  34.26/ 9.21 |  48.33/ 14.61 |  55.95/ 20.76 |  65.68/ 26.73
w/ transform | R+V |  1.26/ 0.49 |  4.02/ 1.37 |  27.37/ 7.84 |  41.36/ 12.64 |  48.02/ 17.62 |  65.03/ 27.21
PTL (5th itr.) | R+V |  6.83/ 1.94 |  9.09/ 2.85 |  52.89/ 15.57 |  59.90/ 18.48 |  69.11/ 27.33 |  74.14/ 31.41
|  | +3.09/+0.85 | +2.67/+0.99 | +11.28/ +4.34 | +10.06/ +4.72 | +19.76/+12.64 |  +7.39/ +7.50
PTL (best) | R+V |  7.52/ 2.13 |  9.33/ 2.94 |  53.82/ 15.59 |  60.65/ 18.48 |  70.23/ 27.33 |  74.14/ 31.41
|  | +3.78/+1.04 | +2.91/+1.08 | +12.21/ +4.36 | +10.81/ +4.72 | +20.88/+12.64 |  +7.39/ +7.50

Main Results. In Table 1, we compare PTL to the baselines in terms of human
detection accuracy in two low-shot detection regimes (i.e., 20 and 50 real
images are used for training) on the three real-world UAV-based datasets. Low-
shot detection is a suitable task to validate the proper use of virtual images
as notable effects can be expected from adequately expanded datasets.

In all cases, the previous methods leveraging virtual images do not present
significantly better, or even worse, accuracy than their counterpart (i.e.,
‘baseline’) using real images only for training. ‘Pretrain-finetune’ is the
only method, except for PTL, that presents better accuracy than the baseline
in some cases. This two-step method is effective in avoiding adverse effects
due to the large domain gap while indirectly taking advantage of the large-
scale dataset. However, the increase in accuracy is marginal as the task-
specific properties (i.e., human detection) of the large-scale virtual dataset
used for pre-training are not fully exploited due to the catastrophic
forgetting issue inherent in this indirect method. In addition, ‘naive merge
w/ transformation’, which is the only previous method that uses transformed
virtual images, presents significantly reduced accuracy, which can be regarded
as the adverse effects (e.g., transformation quality degradation) when the
large domain gap is not properly addressed. The final model obtained after the
5th PTL iteration or the best model achieved when PTL progresses consistently
presents significantly better performance than any compared method. This
demonstrates _the effectiveness of PTL such that accuracy is substantially
improved by expanding the training set using virtual images while the large
domain gap is appropriately addressed_.

Table 2: Varying $\tau$ in PTL (best) on the VisDrone dataset with 50-shot
learning setup.

$\tau$ | 2.0 | 5.0 | 10.0
---|---|---|---
<EMAIL_ADDRESS>| 9.45 | 9.33 | 9.68
AP@[.5:.95] | 2.98 | 2.94 | 2.89

Ablation: the effect of $\tau$. In Table 2, we compare three different $\tau$
values to investigate the effect of $\tau$ used to control the sampling
weights when selecting transformation candidates (eq. 5). In general, the
change in accuracy with respect to $\tau$ is not obvious. Interestingly,
$\tau$=10.0 gives the best accuracy for<EMAIL_ADDRESS>while $\tau$=2.0 is the best for
AP@[.5:.95]. That is, using a smaller $\tau$ to select more virtual images
with small domain gaps leads to more accurate estimation of the bounding
boxes, whereas selecting virtual images with more diverse appearances using a
larger $\tau$ increases the detection accuracy. Taking these two effects into
account, we carry out all experiments with $\tau$=5.0.

###  4.3 Results on Cross-domain Detection

Table 3: Cross-domain detection accuracy. The table shows experiments with
3$\times$3 cross-domain setups. For each setup, datasets shown to the left and
right of the arrow are the training and test sets, respectively. The
accuracies of PTL and the baseline without using virtual images for training
are shown. Setups on the top use training and test images from the same
domain, which provides a baseline accuracy in the cross-domain setups. All
setups in each column are tested on the same dataset and the same low-shot
regime.

method | VisDrone | Okutama-Action | ICG
---|---|---|---
20 | 50 | 20 | 50 | 20 | 50
| Vis $\rightarrow$ Vis | Oku $\rightarrow$ Oku | ICG $\rightarrow$ ICG
baseline |  3.74/ 1.09 |  6.42/ 1.86 |  41.61/ 11.23 |  49.84/ 13.76 |  49.35/ 14.69 |  66.75/ 23.91
PTL (5th itr.) |  6.83/ 1.94 |  9.09/ 2.85 |  52.89/ 15.57 |  59.90/ 18.48 |  69.11/ 27.33 |  74.14/ 31.41
PTL (best) |  7.52/ 2.13 |  9.33/ 2.94 |  53.82/ 15.59 |  60.65/ 18.48 |  70.23/ 27.33 |  74.14/ 31.41
| Oku $\rightarrow$ Vis | Vis $\rightarrow$ Oku | Vis $\rightarrow$ ICG
baseline |  1.62/ 0.47 |  2.04/ 0.57 |  17.13/ 4.53 |  36.82/ 9.87 |  2.92/ 0.56 |  7.46/ 1.83
PTL (5th itr.) |  2.72/ 0.94 |  3.05/ 1.07 |  30.72/ 7.45 |  42.39/ 11.41 |  26.86/ 7.22 |  29.26/ 7.27
PTL (best) |  3.00/ 1.22 |  3.56/ 1.17 |  33.25/ 8.59 |  42.39/ 11.46 |  29.60/ 7.69 |  30.01/ 7.36
| ICG $\rightarrow$ Vis | ICG $\rightarrow$ Oku | Oku $\rightarrow$ ICG
baseline |  0.54/ 0.13 |  0.99/ 0.26 |  3.56/ 0.75 |  10.27/ 2.49 |  5.37/ 1.25 |  5.23/ 1.20
PTL (5th itr.) |  1.09/ 0.33 |  1.61/ 0.50 |  11.19/ 2.58 |  14.20/ 3.56 |  28.98/ 8.14 |  25.39/ 6.53
PTL (best) |  1.58/ 1.02 |  1.70/ 0.63 |  12.82/ 2.96 |  14.20/ 3.71 |  28.98/ 8.14 |  26.62/ 6.53

In Table 3, we show the accuracy for the cross-domain setups on the three
datasets. The “cross-domain detection” setups are used to validate the impact
of using virtual images for training when training and test images have
distinct characteristics, such as when the distributions of human appearances
with respect to human poses and viewing angles are different.

First, it is observed that using PTL yields much better accuracies than the
baseline when using the same real training dataset. Despite the inherent
difficulty of cross-domain learning, it is also observed that leveraging
virtual images through PTL produces results not far behind the accuracy of the
baseline (shown in the first row of Table 3) which uses the training images
from the same domain as the test images. However, the improvement in the
cross-domain detection accuracy when using the ICG dataset for training is
relatively low compared to other cases. This is because the ICG dataset has
very different characteristics from other datasets, and virtual images added
to the training set do not reduce this difference. Note that if the real
images in the initial training set have very distinct appearances, a very
different set of virtual images may be selected in the first PTL iteration.
Thus, the disparity may not be overcome even if the PTL progresses further.
Nevertheless, in general, we can confirm that _human detectors trained using
virtual images through PTL can improve substantially detection accuracy,
regardless of which real dataset is used during training_.

##  5 Discussion

Our method has been proved effective in leveraging virtual images during
training as it presents much better accuracy than any other previous methods
for low-shot learning task, where scaling up the training dataset can
significantly impact. In addition, compared to the baseline which does not use
virtual images, our method also presents remarkable accuracy on cross-domain
detection, where the real training and test datasets are from two distinct
domains with very different characteristics.

Despite the merit of the proposed method, there is still room to improve PTL
further. Specifically, the current version of PTL can only leverage a subset
of the entire virtual images within a limited number of iterations, beyond
which the accuracy may decrease. We hope that more advanced methods can be
developed to address this issue so that PTL can progress for more iterations
with a continuous accuracy increase until all virtual images are leveraged for
training.

#### Acknowledgments

This research was sponsored by the Defense Threat Reduction Agency (DTRA) and
the DEVCOM Army Research Laboratory (ARL) under Grant No. W911NF2120076.

##  Appendix A Gaussian Discriminant Analysis for Modeling Representation
Space of Detector

In this section, we describe modeling the representation space of a general
object detector by fitting a multivariate Gaussian distribution. We denote the
random variable of the input and its label of a linear classifier as ${\bf
x}\in\mathcal{X}$ and
$y=\\{y_{c}\\}_{c=1,\cdots,C}\in\mathcal{Y},y_{c}=\\{0,1\\}$, respectively.
Then, the posterior distribution defined by the linear classifier whose output
formed by the sigmoid function can be expressed as follows:

$P(y_{c}=1|{\bf x})=\frac{1}{1+\exp{\left(\scalebox{0.75}[1.0]{$-$}w_{c}{\bf
x}\scalebox{0.75}[1.0]{$-$}b_{c}\right)}}=\frac{\exp{\left(w_{c}{\bf
x}+b_{c}\right)}}{\exp{\left(w_{c}{\bf x}+b_{c}\right)}+1},$ (7)

where $w_{c}$ and $b_{c}$ are weights and bias of the linear classifier for a
category $c$, respectively.

Gaussian Discriminant Analysis (GDA) models the posterior distribution of the
classifier by assuming that the class conditional distribution ($P({\bf
x}|y)$) and the class prior distribution ($P(y)$) follow the multivariate
Gaussian and the Bernoulli distributions, respectively, as follows:

$\displaystyle P({\bf
x}|y_{c}=0)=\mathcal{N}(\mu_{0},\Sigma_{0}),~{}~{}~{}~{}~{}~{}P({\bf
x}|y_{c}=1)=\mathcal{N}(\mu_{1},\Sigma_{1}),$ $\displaystyle
P(y_{c}=0)=\beta_{0}/\left(\beta_{0}+\beta_{1}\right),~{}~{}~{}~{}~{}~{}P(y_{c}=1)=\beta_{1}/\left(\beta_{0}+\beta_{1}\right),$
(8)

where $\mu_{\\{0,1\\}}$ and $\Sigma_{\\{0,1\\}}$ are the mean and covariance
of the multivariate Gaussian distribution, and $\beta_{\\{0,1\\}}$ is the
unnormalized prior for the category $c$ and the background.

For the special case of GDA where all categories share the same covariance
matrix (i.e., $\Sigma_{0}=\Sigma_{1}=\Sigma_{c}$), known as Linear
Discriminant Analysis (LDA), the posterior distribution ($P(y_{c}|{\bf x})$)
can be expressed with $P({\bf x}|y_{c})$ and $P(y_{c})$ as follows:

$\displaystyle P(y_{c}=1|{\bf x})$ $\displaystyle=$
$\displaystyle\frac{P(y_{c}=1)P({\bf x}|y_{c}=1)}{P(y_{c}=0)P({\bf
x}|y_{c}=0)+P(y_{c}=1)P({\bf x}|y_{c}=1)}$ (9) $\displaystyle=$
$\displaystyle\frac{\exp{\left(\left(\mu_{1}\scalebox{0.75}[1.0]{$-$}\mu_{0}\right)^{\top}\Sigma_{c}^{\scalebox{0.75}[1.0]{$-$}1}{\bf
x}-\frac{1}{2}\mu_{1}^{\top}\Sigma_{c}^{\scalebox{0.75}[1.0]{$-$}1}\mu_{1}+\frac{1}{2}\mu_{0}^{\top}\Sigma_{c}^{\scalebox{0.75}[1.0]{$-$}1}\mu_{0}+\log{\beta_{1}/\beta_{0}}\right)}}{\exp{\left(\left(\mu_{1}\scalebox{0.75}[1.0]{$-$}\mu_{0}\right)^{\top}\Sigma_{c}^{\scalebox{0.75}[1.0]{$-$}1}{\bf
x}-\frac{1}{2}\mu_{1}^{\top}\Sigma_{c}^{\scalebox{0.75}[1.0]{$-$}1}\mu_{1}+\frac{1}{2}\mu_{0}^{\top}\Sigma_{c}^{\scalebox{0.75}[1.0]{$-$}1}\mu_{0}+\log{\beta_{1}/\beta_{0}}\right)}+1}.$

Note that the quadratic term is canceled out since the shared covariance
matrix is used. The posterior distribution derived by GDA in eq. 9 then
becomes equivalent to the posterior distribution of the linear classifier with
the sigmoid function in eq. 7 when
$w_{c}=\left(\mu_{1}\scalebox{0.75}[1.0]{$-$}\mu_{0}\right)^{\top}\Sigma_{c}^{\scalebox{0.75}[1.0]{$-$}1}$
and
$b_{c}=-\frac{1}{2}\mu_{1}^{\top}\Sigma_{c}^{\scalebox{0.75}[1.0]{$-$}1}\mu_{1}+\frac{1}{2}\mu_{0}^{\top}\Sigma_{c}^{\scalebox{0.75}[1.0]{$-$}1}\mu_{0}+\log{\beta_{1}/\beta_{0}}$.
This implies that the representation space formed by x can be modeled by a
multivariate Gaussian distribution.

Based on the above derivation, if ${\bf x}$ is the output of the penultimate
layer of an object detector for a region proposal, and a linear classifier
defined by $w_{c}$ and $b_{c}$ is the last layer of the object detector, it
can be said that the representation space of the object detector for a
category $c$ can be modeled with a multivariate Gaussian distribution. In
other words, the representation space for a category $c$ can be represented by
two parameters $\mu_{1}$ (i.e., $\mu_{c}$) and $\Sigma_{c}$ of the
multivariate Gaussian distribution.

Discussion. The sigmoid function can be viewed as a special case of the
softmax function defined for a single category as both functions take the form
of an exponential term for the category-of-interest normalized by the sum of
exponential terms for all considered categories. Therefore, it is
straightforward to derive the modeling for the sigmoid-based detector from the
previous work by Lee et al. (2018), who shows that the softmax-based
classifier can be modeled with a multivariate Gaussian distribution in the
representation space. However, our derivation is still meaningful in that it
extends the applicability of an existing modeling limited to a certain type of
classifier (i.e., based on softmax) to general object detectors (i.e., based
on sigmoid). Most object detectors, especially one-stage detectors, generally
use the sigmoid function, which does not consider other categories when
calculating the model output for a certain category, since more than one
category can be active on a single output.

##  Appendix B Implementation Details

Multi-scale training. We apply the multi-scale training strategy when
preparing input images, in addition to the 5-level multi-scaling property
provided by the detector’s FPN module, to train the detector. The goal is to
make the detector more robust to the variations of human size in the images.
Since the real and virtual datasets have widely varying human sizes in the
images, we apply different scaling factors for each dataset to share similar
human sizes after image rescaling. Specifically, for the real dataset, the
input image is resized by one of the scaling factors randomly selected from
$\\{768,800,832,864\\}$ for the short side, in which the long side is
constrained not to be larger than 1440, for every training iteration. For the
virtual dataset, the scaling factors are $\\{128,256,384,512\\}$ for the short
side and 512 for the long side.

Network architecture. To obtain a generator, we adopt CycleGAN, where two
generators and two discriminators are involved during model training. For each
generator, we use a 24-layer UNet-like architecture (i.e., the resnet_9blocks
generator in the official repository of CycleGAN (Cyc, )), which contains nine
2-layer residual modules in the middle of the architecture where the encoder
and decoder are connected. For each discriminator, a 5-layer fully
convolutional network (i.e., the basic discriminator in the official
repository of CycleGAN (Cyc, )) is used.

For the detector, we adopt the official RetinaNet architecture implemented in
Detectron2 (Wu et al., 2019) with few modifications. First of all, the feature
dimension of RetinaNet’s classification subnet and box regression subnet are
decreased from 256 to 32 since we are dealing with single category (i.e.,
human) instead of multiple categories (e.g., 80 categories for detectors
trained on the MS COCO dataset). In addition, we switch the activation
function used in these two subnets from ReLU to LeakyReLU to avoid the
singular covariance matrix problem, which may occur when calculating the
Mahalanobis distance, occasionally triggered by the dying ReLU problem.
Finally, the kernel size of the last convolutional layer in the classification
subnet, the layer just in front of the sigmoid layer, is reduced from
3$\times$3 to 1$\times$1 so that each output prediction is associated with
only one feature vector in the feature representation space. Here, ResNet50 is
used as the backbone.

Table 4: Training settings.

(a) Detector training

config | baseline | pretrain-finetune | naive merge | PTL
---|---|---|---|---
pretrain | finetune
optimizer | SGD
momentum | 0.9
weight decay | 0.0001
base $lr$ | 0.001
$lr$ schedule | multi-step $lr$
gamma | 0.1
warmup iter. | 1000
total iter. | 6000 | 6000 | 600 | 6000 | 6000
steps | 5000 | 5000 | 500 | 5000 | 5000
batch size | 16
filter empty annot. | False
box threshold | 10
aspect ratio grouping | False

(b) Generator training

config | naive merge w/ transform | PTL
---|---|---
optimizer | Adam
momentum |  $\beta_{1}$, $\beta_{2}$ = 0.5, 0.999
$lr$ | 0.0002
total epochs | 80 | 100
batch size | 8
load size | 256
preprocess | None

Training details. The settings used for training the human detector through
our method and the other baseline methods are listed in Table 4(a). Without
further specification, we follow all the settings and initialization
strategies defined by the original RetinaNet training (Lin et al., 2017).
Unlike other model training, using fewer iterations for fine-tuning when
adopting the pretrain-finetune method is common because training converges
faster. Although fine-tuning usually uses a lower learning rate than pre-
training, we use the same learning rate (i.e., 0.001) for both stages because
we also fine-tune the ImageNet-pretrained backbone on the virtual images in
the pre-training stage.

The settings used for training the generator through the baseline method using
transformation (i.e., ‘naive merge w/ transformation’) and our method are
listed in Table 4(b). These settings and initialization strategies are taken
from the original CycleGAN training (Zhu et al., 2017). ‘Naive merge w/
transform’ used fewer training epochs than PTL (80 vs 100) because all images
in the virtual set are used for training. In general, as the size of the
dataset increases, the number of epochs required for training decreases.

|  |
---|---|---
VisDrone
|  |
Okutama-Action
|  |
ICG
|  $\cdot$ character: juliet $\cdot$ pose: standing $\cdot$ altitude: 5$m$ $\cdot$ radius: 5$m$ $\cdot$ camera angle: 90∘ $\cdot$ sun angle: 1 |  |  $\cdot$ character: scott $\cdot$ pose: standing $\cdot$ altitude: 5$m$ $\cdot$ radius: 15$m$ $\cdot$ camera angle: 240∘ $\cdot$ sun angle: 2 |  |  $\cdot$ character: juliet $\cdot$ pose: squating $\cdot$ altitude: 5$m$ $\cdot$ radius: 30$m$ $\cdot$ camera angle: 150∘ $\cdot$ sun angle: 3
|  $\cdot$ character: scott $\cdot$ pose: squating $\cdot$ altitude: 25$m$ $\cdot$ radius: 5$m$ $\cdot$ camera angle: 180∘ $\cdot$ sun angle: 2 |  |  $\cdot$ character: juliet $\cdot$ pose: standing $\cdot$ altitude: 25$m$ $\cdot$ radius: 15$m$ $\cdot$ camera angle: 210∘ $\cdot$ sun angle: 3 |  |  $\cdot$ character: scott $\cdot$ pose: standing $\cdot$ altitude: 25$m$ $\cdot$ radius: 30$m$ $\cdot$ camera angle: 330∘ $\cdot$ sun angle: 4
|  $\cdot$ character: scott $\cdot$ pose: squating $\cdot$ altitude: 5$m$ $\cdot$ radius: 5$m$ $\cdot$ camera angle: 270∘ $\cdot$ sun angle: 1 |  |  $\cdot$ character: juliet $\cdot$ pose: squating $\cdot$ altitude: 5$m$ $\cdot$ radius: 15$m$ $\cdot$ camera angle: 180∘ $\cdot$ sun angle: 2 |  |  $\cdot$ character: scott $\cdot$ pose: standing $\cdot$ altitude: 5$m$ $\cdot$ radius: 30$m$ $\cdot$ camera angle: 120∘ $\cdot$ sun angle: 1
|  $\cdot$ character: juliet $\cdot$ pose: standing $\cdot$ altitude: 25$m$ $\cdot$ radius: 5$m$ $\cdot$ camera angle: 60∘ $\cdot$ sun angle: 2 |  |  $\cdot$ character: scott $\cdot$ pose: squating $\cdot$ altitude: 25$m$ $\cdot$ radius: 15$m$ $\cdot$ camera angle: 0∘ $\cdot$ sun angle: 3 |  |  $\cdot$ character: juliet $\cdot$ pose: squating $\cdot$ altitude: 25$m$ $\cdot$ radius: 30$m$ $\cdot$ camera angle: 300∘ $\cdot$ sun angle: 4
Archangel-Synthetic

Figure 5: Sample images from each dataset. For the Archangel-Synthetic
dataset, metadata for each image is shown to the right of the image.

Datasets. In this section, we provide the details of the three real UAV-based
datasets, VisDrone (Zhu et al., 2021), Okutama-Action (Barekatain et al.,
2017), and ICG (ICG, )), and the virtual UAV-based dataset, _Archangel-
Synthetic_ , used in this paper. Sample images for each dataset are shown in
Figure 5.

  * •

VisDrone has four tracks (i.e., object detection (DET), video object detection
(VID), single object tracking (SOT), and multi-object tracking (MOT)) with a
separate dataset for each track. We use the dataset from the DET track and
focus on detecting the person and the pedestrian categories among the ten
object categories covered by the track. The VisDrone dataset in the DET track
consists of 10,209 images, where 6,471 images are used in the training set,
548 images are used in the validation set, 1,580 images are used in the test-
challenge set, and 1,610 images are used in the test-dev set. We use the
training set for model training and the test-dev set for model testing. The
maximal resolution of images in the VisDrone dataset is 2000$\times$1500.

  * •

Okutama-Action was originally created for human action detection from the
aerial view. Although the main task associated with this dataset is human
action detection, the dataset also includes a sub-task of pedestrian
detection. To acquire images from various aerial views, a drone with an
embedded camera flew freely at some altitudes between 10$m$ and 45$m$, and the
camera angle was set at 45 or 90 degrees. The Okutama-Action dataset contains
43 video sequences in 4K resolution (i.e., 3840$\times$2160), where 33 video
sequences are used for the train-val set and the remaining 10 video sequences
are used for the test set. Since adjacent frames in the video sequence are
very similar, we use every tenth frame in both the train-val set and the test
set. Our model is trained on the train-val set and tested with the test set.

  * •

ICG was collected for studying semantic understanding of urban scenes.
Additionally, the ICG dataset also provides information, such as ground-truth
human bounding boxes, for the human detection task. Images in the ICG dataset
were captured from a camera located at some altitudes between 5-50$m$ above
the ground at a resolution of 6000$\times$4000\. The ICG dataset provides a
training set of 400 images for the human detection task, of which we use the
first 320 images for training and the remaining 80 images for testing.

  * •

Archangel-Synthetic is one of the three sub-datasets included in the Archangel
dataset, along with Archangel-Real and Archangel-Mannequin. The Archangel
dataset is a hybrid UAV-based dataset captured with similar imaging conditions
in real and synthetic domains. An important property of the Archangel dataset
that sets it apart from other datasets is that it provides metadata about the
camera positions in terms of UAV altitudes and radii of the rotation circles
for each image. The Archangel-Synthetic dataset was generated by using the
Unity game engine. The dataset includes eight virtual characters in three
different poses captured with camera viewing angles ranging from 0∘ to 358∘ in
increments of 2∘, UAV altitudes and rotation circle radii from 5$m$ to 80$m$
in increments of 5$m$, and four different sun angles. The total number of
images in the Archangel-Synthetic dataset is 4.4M. Considering the significant
difference in dataset size between the Archangel-Synthetic dataset with the
other real datasets, a small subset of the entire dataset (17.6K) was used as
the virtual image set in our experiments. The size of images in the Archangel-
Synthetic dataset is 512$\times$512.

##  Appendix C Additional Analyses

iter. 1 | iter. 2 | iter. 3 | iter. 4 | iter. 5
---|---|---|---|---
|  |  |  |
VisDrone, 20-img
iter. 1 | iter. 2 | iter. 3 | iter. 4 | iter. 5
|  |  |  |
Okutama-Action, 20-img
iter. 1 | iter. 2 | iter. 3 | iter. 4 | iter. 5
|  |  |  |
Okutama-Action, 50-img
iter. 1 | iter. 2 | iter. 3 | iter. 4 | iter. 5
|  |  |  |
ICG, 20-img
iter. 1 | iter. 2 | iter. 3 | iter. 4 | iter. 5
|  |  |  |
ICG, 50-img

Figure 6: Accumulated distributions of transformation candidates with respect
to camera locations in more setups. This figure shows the distributions in
other five experimental setups except for the setup (i.e., VisDrone, 50-img)
shown in Figure 3 of the main manuscript. The $x$ and $y$ axes indicate
altitude and rotation circle radius from the target human.

iter. 1 | iter. 2 | iter. 3 | iter. 4 | iter. 5
---|---|---|---|---
|  |  |  |
VisDrone, 20-img
iter. 1 | iter. 2 | iter. 3 | iter. 4 | iter. 5
|  |  |  |
Okutama-Action, 20-img
iter. 1 | iter. 2 | iter. 3 | iter. 4 | iter. 5
|  |  |  |
Okutama-Action, 50-img
iter. 1 | iter. 2 | iter. 3 | iter. 4 | iter. 5
|  |  |  |
ICG, 20-img
iter. 1 | iter. 2 | iter. 3 | iter. 4 | iter. 5
|  |  |  |
ICG, 50-img

Figure 7: Distributions of virtual images with respect to the domain gap in
more setups. This figure shows the domain gap distributions of virtual images
in other five experimental setups except for the setup (i.e., VisDrone,
50-img) shown in Figure 3 of the main manuscript. The $x$ axis represents the
domain gap and the $y$ axis represents the corresponding number of virtual
images.

Further analysis of the properties of progressive learning. In this section,
we show the distributions of transformation candidates in relation to the
camera position (in Figure 6) and the domain gap (in Figure 7) for other five
cases not shown in Figure 3 of the main manuscript. We intend to show that the
analysis described in the main manuscript can be also applied to these cases.

For the distributions of transformation candidates with respect to camera
locations (Figure 6), the observation shown in the main manuscript is also
applied to the other five cases. That is, as PTL progresses, the camera
locations of virtual images included in the training set are gradually spread
over the entire area. Therefore, the validity of the transformation candidate
selection process of PTL extends to all the experimental setups considered in
this paper.

Note that the distributions of humans in the real training set with respect to
camera locations is likely to be similar to the distributions of
transformation candidates with respect to camera locations at the first PTL
iteration as a virtual image with a smaller domain gap is selected with a
higher probability. Accordingly, we speculate that humans in the real training
set were captured from various camera locations in the Okutama-action dataset.
In contrast, in the other two datasets, most of them were taken at some
similar ranges. In addition, in the ICG dataset, the camera locations where
most images were taken are in the close range, which might be the reason why
diversifying camera locations through PTL does not significantly improve
accuracy in the cross-domain setup.

For the distributions of virtual images with respect to the domain gap (Figure
7), the observation mentioned in the main manuscript that the distribution
becomes narrower and smaller as PTL progresses is still perceived in these
settings. However, the speed of this distribution change is particularly slow
for the ICG dataset compared to the other datasets, as shown in Figure 7. This
observation also implies that the ICG dataset has very different
characteristics compared to the other two datasets.

Table 5: Varying # of virtual images added to the training set per PTL
iteration. These ablation experiments are carried out under the 50-shot and
cross-domain setup, where models are trained on the 50 images from the
VisDrone dataset and tested on all three datasets. The reported accuracies are
obtained by using the best PTL models. The accurices are reported with two
evaluation metrics (i.e<EMAIL_ADDRESS>and AP@[.5:.95]).

# img per itr | VisDrone | Okutama-Action | ICG
---|---|---|---
50 |  9.59/ 2.90 |  41.62/ 11.19 |  25.94/ 6.04
100 |  9.33/ 2.94 |  42.39/ 11.46 |  30.01/ 7.36
200 |  9.04/ 2.91 |  41.29/ 11.18 |  35.50/10.20

Ablation: the effect of $n$. PTL has two hyperparameters: $\tau$, which is
used to control the sampling weights when selecting transformation candidates,
and $n$, which is the number of virtual images added to the training set
(i.e., transformation candidates) per PTL iteration. Ablation experiments on
the effects of $\tau$ are analyzed in the main manuscript.

The number of virtual images included in the training set for each PTL
iteration can affect not only the optimality of training but also the
scalability of models trained by PTL toward other datasets different from the
real dataset used for training. Accordingly, to find the optimal value of $n$
by taking these effects into account, we carry out ablation experiments under
the cross-domain setup as shown in Table 5.

We compare three cases when $n$ is 50, 100, and 200. When the training set and
the test set comes from the same domain (i.e., training and testing on the
VisDrone dataset), the human detection accuracy is not very sensitive to $n$.
The best accuracy is obtained with the fewest virtual images (i.e., $n$=50).
The insensitivity to the hyperparameter $n$ is also observed in the Okutama-
Action dataset, having relatively similar properties to the VisDrone dataset
under the cross-domain setup. However, for ICG, which is considered to have
noticeably different properties from the VisDrone dataset under the cross-
domain setup, the accuracy is significantly improved as more virtual images
(i.e., $n$=200) are used per PTL iteration. This implies that using more
virtual images through more PTL iterations tends to further reduce the cross-
domain gap when the discrepancy between the training set and the test set is
considerably large, such as VisDrone vs. ICG, resulting in improved PTL
scalability. Considering the trade-off between the training optimality and the
PTL scalability, we use $n$=100 throughout all other experiments in this
paper.

Qualitative analysis of transformation. Figures 8, 9, 10, 11, 12, and 13 show
several samples of transformed virtual images included in the training set
using methods with virtual2real transformation (i.e., PTL and ‘naive merge w/
transformation’) in the six experimental setups. First, the quality of the
transformed image by ’naive merge w/transformation’ generally deteriorates a
lot in all six experimental settings, and there are samples, in some severe
cases, in which it is difficult to recognize the human appearance anymore. On
the other hand, in the case of the virtual image transformed by PTL
considering the domain gap when training the transformation generator, it can
be seen that only the pattern is changed to be similar to the image of the
real dataset while the human pose is maintained well. This qualitative
analysis supports the validity of our claim that _the domain gap between
virtual images and real images should be considered when training the
virtual2real transformation generator_.

|  |
---|---|---
|  |
|  |
|  |
|  |
|  |
PTL
|  |
|  |
|  |
|  |
|  |
|  |
Naive merge w/ transform

Figure 8: Sample Virtual2Real transformation output (VisDrone, 20-shot). Each
set consists of three images: original virtual image (left), transformed image
(middle), and transformed image with background (right).

|  |
---|---|---
|  |
|  |
|  |
|  |
|  |
PTL
|  |
|  |
|  |
|  |
|  |
|  |
Naive merge w/ transform

Figure 9: Sample Virtual2Real transformation output (VisDrone, 50-shot). Each
set consists of three images: original virtual image (left), transformed image
(middle), and transformed image with background (right).

|  |
---|---|---
|  |
|  |
|  |
|  |
|  |
PTL
|  |
|  |
|  |
|  |
|  |
|  |
Naive merge w/ transform

Figure 10: Sample Virtual2Real transformation output (Okutama-Action,
20-shot). Each set consists of three images: original virtual image (left),
transformed image (middle), and transformed image with background (right).

|  |
---|---|---
|  |
|  |
|  |
|  |
|  |
PTL
|  |
|  |
|  |
|  |
|  |
|  |
Naive merge w/ transform

Figure 11: Sample Virtual2Real transformation output (Okutama-Action,
50-shot). Each set consists of three images: original virtual image (left),
transformed image (middle), and transformed image with background (right).

|  |
---|---|---
|  |
|  |
|  |
|  |
|  |
PTL
|  |
|  |
|  |
|  |
|  |
|  |
Naive merge w/ transform

Figure 12: Sample Virtual2Real transformation output (ICG, 20-shot). Each set
consists of three images: original virtual image (left), transformed image
(middle), and transformed image with background (right).

|  |
---|---|---
|  |
|  |
|  |
|  |
|  |
PTL
|  |
|  |
|  |
|  |
|  |
|  |
Naive merge w/ transform

Figure 13: Sample Virtual2Real transformation output (ICG, 50-shot). Each set
consists of three images: original virtual image (left), transformed image
(middle), and transformed image with background (right).

## References

  * (1) Cyclegan and pix2pix in pytorch.  https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix.
  * (2) Aerial semantic segmentation drone dataset.  http://dronedataset.icg.tugraz.at.
  * Baek & Shim (2022) Kyungjune Baek and Hyunjung Shim.  Commonality in natural images rescues GANs: Pretraining GANs with generic and privacy-free synthetic data.  In _Proc. CVPR_ , 2022.
  * Barekatain et al. (2017) Mohammadamin Barekatain, Miquel Martí, Hsueh-Fu Shih, Samuel Murray, Kotaro Nakayama, Yutaka Matsuo, and Helmut Prendinger.  Okutama-action: An aerial view video dataset for concurrent human action detection.  In _Proc. CVPRW_ , 2017.
  * Bengio et al. (2006) Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle.  Greedy layer-wise training of deep networks.  In _Proc. NeurIPS_ , 2006.
  * Bengio et al. (2009) Yoshua Bengio, Jérǒme Louradour, Ronan Collobert, and Jason Weston.  Curriculum learning.  In _Proc. ICML_ , 2009.
  * Bochkovskiy et al. (2020) Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao.  YOLOv4: Optimal speed and accuracy of object detection.  arXiv:2004.10934, 2020.  URL https://arxiv.org/abs/2004.10934.
  * Cao et al. (2022) Tianyue Cao, Yongxin Wang, Yifan Xing, Tianjun Xiao, Tong He, Zheng Zhang, Hao Zhou, and Joseph Tighe.  PSS: Progressive sample selection for open-world visual representation learning.  In _Proc. ECCV_ , 2022.
  * Chen et al. (2016) Tianqi Chen, Ian Goodfellow, and Jonathon Shlens.  Net2Net: Accelerating learning via knowledge transfer.  In _Proc. ICLR_ , 2016.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _Proc. ICML_ , 2020.
  * Fabbri et al. (2018) Matteo Fabbri, Fabio Lanzi, Simone Calderara, Andrea Palazzi, Roberto Vezzani, and Rita Cucchiara.  Learning to detect and track visible and occluded body joints in a virtual world.  In _Proc. ECCV_ , 2018.
  * Fahlman & Lebiere (1989) Scott E. Fahlman and Christian Lebiere.  The cascade-correlation learning architecture.  In _Proc. NeurIPS_ , 1989.
  * Gaidon et al. (2016) Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora Vig.  Virtual worlds as proxy for multi-object tracking analysis.  In _Proc. CVPR_ , 2016.
  * Gong et al. (2019) Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tie-Yan Liu.  Efficient training of BERT by progressively stacking.  In _Proc. ICML_ , 2019.
  * Graves et al. (2017) Alex Graves, Marc G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu.  Automated curriculum learning for neural networks.  In _Proc. ICML_ , 2017.
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko.  Bootstrap your own latent - a new approach to self-supervised learning.  In _Proc. NeurIPS_ , 2020.
  * Guo et al. (2022) Xi Guo, Wei Wu, Dongliang Wang, Jing Su, Haisheng Su, Weihao Gan, Jian Huang, and Qin Yang.  Learning video representations of human motion from synthetic data.  In _Proc. CVPR_ , 2022.
  * Handa et al. (2016) Ankur Handa, Viorica Pǎtrǎucean, Vijay Badrinarayanan, Simon Stent, and Roberto Cipolla.  SceneNet: Understanding real world indoor scenes with synthetic data.  In _Proc. CVPR_ , 2016.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proc. CVPR_ , 2020.
  * Hoffman et al. (2018) Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell.  CyCADA: Cycle-consistent adversarial domain adaptation.  In _Proc. ICML_ , 2018.
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros.  Image-to-image translation with conditional adversarial networks.  In _Proc. CVPR_ , 2017.
  * Jin et al. (2022) Zhao Jin, Yinjie Lei, Naveed Akhtar, Haifeng Li, and Munawar Hayat.  Deformation and correspondence aware unsupervised synthetic-to-real scene flow estimation for point clouds.  In _Proc. CVPR_ , 2022.
  * Jocher (2020) Glenn Jocher.  ultralytics/yolov5: v3.1 - Bug Fixes and Performance Improvements.  https://github.com/ultralytics/yolov5, 2020.  URL https://doi.org/10.5281/zenodo.4154370.
  * Karras et al. (2018) Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of GANs for improved quality, stability, and variation.  In _Proc. ICLR_ , 2018.
  * Kaul et al. (2022) Prannay Kaul, Weidi Xie, and Andrew Zisserman.  Label, verify, correct: A simple few shot object detection method.  In _Proc. CVPR_ , 2022.
  * Khan et al. (2011) Faisal Khan, Bilge Mutlu, and Jerry Zhu.  How do humans teach: On curriculum learning and teaching dimension.  In _Proc. NeurIPS_ , 2011.
  * Kim et al. (2016) Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon.  Video panoptic segmentation.  In _Proc. CVPR_ , 2016.
  * Kumar et al. (2010) M. Pawan Kumar, Benjamin Packer, and Daphne Koller.  Self-paced learning for latent variable models.  In _Proc. NeurIPS_ , 2010.
  * Lee et al. (2018) Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  In _Proc. NeurIPS_ , 2018.
  * Li et al. (2022a) Changlin Li, Bohan Zhuang, Guangrun Wang, Xiaodan Liang, Xiaojun Chang, and Yi Yang.  Automated progressive learning for efficient training of vision transformers.  In _Proc. CVPR_ , 2022a.
  * Li & Lee (2021) Chen Li and Gim Hee Lee.  From synthetic to real: Unsupervised domain adaptation for animal pose estimation.  In _Proc. CVPR_ , 2021.
  * Li et al. (2022b) Chuyi Li, Lulu Li, Hongliang Jiang, Kaiheng Weng, Yifei Geng, Liang Li, Zaidan Ke, Qingyuan Li, Meng Cheng, Weiqiang Nie, Yiduo Li, Bo Zhang, Yufei Liang, Linyuan Zhou, Xiaoming Xu, Xiangxiang Chu, Xiaoming Wei, and Xiaolin Wei.  YOLOv6: A single-stage object detection framework for industrial applications.  arXiv:2209.02976, 2022b.  URL https://arxiv.org/abs/2209.02976.
  * Lin et al. (2017) Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In _Proc. ICCV_ , 2017.
  * Liu et al. (2022) Qing Liu, Adam Kortylewski, Zhishuai Zhang, Zizhang Li, Mengqi Guo, Qihao Liu, Xiaoding Yuan, Jiteng Mu, Weichao Qiu, and Alan Yuille.  Learning part segmentation through unsupervised domain adaptation from synthetic vehicles.  In _Proc. CVPR_ , 2022.
  * Liu et al. (2016) Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg.  SSD: Single shot multibox detector.  In _Proc. ECCV_ , 2016.
  * Mahalanobis (1936) Prasanta Chandra Mahalanobis.  On the generalised distance in statistics.  _Proc. National Institute of Sciences of India._ , 1936.
  * Mirza & Osindero (2014) Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  arXiv:1411.1784, 2014.  URL https://arxiv.org/abs/1411.1784.
  * Mishra et al. (2022) Samarth Mishra, Rameswar Panda, Cheng Perng Phoo, Chun-Fu (Richard) Chen, Leonid Karlinsky, Kate Saenko, Venkatesh Saligrama, and Rogerio S. Feris.  Task2Sim: Towards effective pre-training and transfer from synthetic data.  In _Proc. CVPR_ , 2022.
  * Mu et al. (2020) Jiteng Mu, Weichao Qiu, Gregory Hager, and Alan Yuille.  Learning from synthetic animals.  In _Proc. CVPR_ , 2020.
  * Qiu et al. (2021) Haibo Qiu, Baosheng Yu, Dihong Gong, Zhifeng Li, Wei Liu, and Dacheng Tao.  SynFace: Face recognition with synthetic data.  In _Proc. ICCV_ , 2021.
  * Richter et al. (2016) Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun.  Playing for data: Ground truth from computer games.  In _Proc. ECCV_ , 2016.
  * Ros et al. (2016) German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M. Lopez.  The SYNTHIA dataset: A large collection of synthetic images for semantic segmentation of urban scenes.  In _Proc. CVPR_ , 2016.
  * Shen et al. (2022) Yi-Ting Shen, Yaesop Lee, Heesung Kwon, Damon M. Conover, Shuvra S. Bhattacharyya, Nikolas Vale, Joshua D. Gray, G. Jeremy Leong, Kenneth Evensen, and Frank Skirlo.  Archangel: A hybrid UAV-based human detection benchmark with position and pose metadata.  arXiv:2209.00128, 2022.  URL https://arxiv.org/abs/2209.00128.
  * Smith et al. (2016) Leslie N. Smith, Emily M. Hand, and Timothy Doster.  Gradual DropIn of layers to train very deep neural networks.  In _Proc. CVPR_ , 2016.
  * Spitkovsky et al. (2009) Valentin I. Spitkovsky, Hiyan Alshawi, and Daniel Jurafsky.  Baby steps: How “less is more” in unsupervised dependency parsing.  In _Proc. NeurIPS Workshop_ , 2009.
  * Varol et al. (2018) Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, and Cordelia Schmid.  Learning from synthetic humans.  In _Proc. CVPR_ , 2018.
  * Wang et al. (2022a) Can Wang, Sheng Jin, Yingda Guan, Wentao Liu, Chen Qian, Ping Luo, and Wanli Ouyang.  Pseudo-labeled auto-curriculum learning for semi-supervised keypoint localization.  In _Proc. ICLR_ , 2022a.
  * Wang et al. (2022b) Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao.  YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors.  arXiv:2207.02696, 2022b.  URL https://arxiv.org/abs/2207.02696.
  * Wang et al. (2017) Guangcong Wang, Xiaohua Xie, Jianhuang Lai, and Jiaxuan Zhuo.  Deep growing learning.  In _Proc. ICCV_ , 2017.
  * Wei et al. (2016) Tao Wei, Changhu Wang, Yong Rui YONGRUI, and Chang Wen Chen.  Network morphism.  In _Proc. ICML_ , 2016.
  * Wu et al. (2019) Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * Yan et al. (2022) Qi Yan, Jianhao Zheng, Simon Reding, Shanci Li, and Iordan Doytchinov.  CrossLoc: Scalable aerial localization assisted by multimodal synthetic data.  In _Proc. CVPR_ , 2022.
  * Zhang & He (2020) Minjia Zhang and Yuxiong He.  Accelerating training of transformer-based language models with progressive layer dropping.  In _Proc. neurIPS_ , 2020.
  * Zhu et al. (2017) Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _Proc. ICCV_ , 2017.
  * Zhu et al. (2021) Pengfei Zhu, Longyin Wen, Dawei Du, Xiao Bian, Heng Fan, Qinghua Hu, and Haibin Ling.  Detection and tracking meet drones challenge.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 2021.