# Unsupervised Domain Adaptation for Object Detection
via Cross-Domain Semi-Supervised Learning

Fuxun Yu†, Di Wang‡, Yinpeng Chen‡, Nikolaos Karianakis‡, Pei Yu‡,
Dimitrios Lymberopoulos‡, Xiang Chen†
† George Mason University, ‡ Microsoft
†{fyu2<EMAIL_ADDRESS>
‡{wangdi, yiche, nikolaos.karianakis, pei.yu<EMAIL_ADDRESS>
###### Abstract

Current state-of-the-art object detectors can have significant performance
drop when deployed in the wild due to domain gaps with training data.
Unsupervised Domain Adaptation (UDA) is a promising approach to adapt models
for new domains/environments without any expensive label cost. However,
without ground truth labels, most prior works on UDA for object detection
tasks can only perform coarse image-level and/or feature-level adaptation by
using adversarial learning methods. In this work, we show that such
adversarial-based methods can only reduce domain style gap, but cannot address
the domain content distribution gap that are shown to be important for object
detectors. To overcome this limitation, we propose the Cross-Domain Semi-
Supervised Learning (CDSSL) framework by leveraging high quality pseudo labels
to learn better representations from target domain directly. To enable SSL for
cross-domain object detection, we propose fine-grained domain transfer,
progressive-confidence-based label sharpening and imbalanced sampling strategy
to address two challenges: (i) non-identical distribution between source and
target domain data, (ii) error amplification/accumulation due to noisy pseudo
labeling on target domain. Experiment results show that our proposed approach
consistently achieves new state-of-the-art performance (2.2% - 9.5% better
than prior best work on mAP) under various domain adaptation scenarios. Code
will be released at $-link-$.

##  1 Introduction

Recently, deep learning has achieved superior performance for various vision
tasks such as classification, object detection and semantic segmentation [22,
11, 15, 25]. However, real world deployment environments can be highly
uncertain and non-stationary, which can cause domain gaps including
appearance/style difference (e.g., lighting, brightness) and content
distribution mismatch (e.g., object density) with training data, resulting
significantly reduced model performance [8, 26, 35, 36]. Traditional practice
is to collect new training data to retrain models, but labeling these data can
be very costly and time-consuming, especially for complex tasks such as object
detection and segmentation [20, 31]. As a result, Unsupervised Domain
Adaptation (UDA) using labeled source domain and unlabeled target domain has
become a promising approach [8].

Figure 1: Top: Comparison of SSL and UDA settings. In UDA, labeled and
unlabeled data do not come from the same distribution which is different from
traditional SSL assumption. Bottom: Our proposed CDSSL framework: (1) Domain
transfer to generate an intermediate domain to reduce the distribution gap;
(2) Iterative Self-training to improve pseudo label quality on target domain.

Prior UDA works have proposed many methods including metric-based discrepancy
minimization, manifold/subspace alignment, domain mapping and domain-invariant
feature learning, etc [10, 7, 16, 8]. Most of these works target at image
classification, where coarse-grained alignment by aligning features globally
between source and target domain may suffice. On the other hand, UDA for
object detection may require not only coarse-grained/global alignment but also
fine-grained/local alignment such as local object context, object density,
etc. that have been shown to be important for object detectors [41, 32]. The
state-of-the-art UDA works for object detectors [4, 41, 32, 38] mostly use
adversarial feature alignment methods to align features of unpaired source and
target domain images. However, feature alignment cannot distinguish objects
belonging to different classes or in different contexts. Thus, it is prone to
misalignment, e.g., misaligning features of foreground v.s. background,
features of different classes, or features of multiple v.s. single object.
Moreover, due to the lack of target-domain labels, adversarial methods can
only work at a coarse image or feature level and may not be able to align any
content mismatch between domains.

In this work, we take a different approach to reduce domain gaps for object
detectors by leveraging Semi-Supervised Learning (SSL) concept: Self-Training
[27]. Self-Training method utilizes a portion of labeled data to generate
pseudo labels for unlabeled data from the same distribution and combines both
for training [23]. Although such pseudo labels are not as accurate as ground-
truth (GT) labels, they bring several benefits for object detections compared
to prior coarse-grained feature alignment: (i) the detection model learns
object features directly from pseudo labeled boxes, which is more fine-grained
and with less ambiguity, (ii) the overall pseudo labels on the target domain
provide a way to approximate the real data distribution, and (iii) by
improving pseudo label quality, better representations can be learned to
improve detection performance.

However, as shown in Figure 1a, traditional SSL assumes that labeled and
unlabeled data come from the same distribution [29, 1, 12, 30]. Consequently,
directly applying SSL methods into our problem usually cannot work well due to
two main challenges: (1) Non-identical distribution: as shown in Figure 1b,
the source and target domain data typically come from different distributions
with a domain gap; (2) Error amplification/accumulation: due to the domain
gap, the generated pseudo labels on target domain often contain many
annotation errors. These erroneous labels will reinforce themselves during
iterative SSL and generate more erroneous labels which significantly degrade
object detection performance.

To address these challenges, we propose the Cross-Domain Semi-Supervised
Learning (CDSSL) framework for object detectors as shown in Figure 1c. The
CDSSL framework consists of two major steps: (1) Domain Transfer. To reduce
non-identical data distribution gap, an intermediate domain is generated by
transforming source domain image style to match with target domain. A
detection model is trained on this intermediate domain as an initial pseudo
label annotator on target domain. (2) Iterative Self-Training. To improve
pseudo label quality, two key components are proposed: (i) imbalanced mini-
batch sampling by over-sampling source domain samples and under-sampling
target domain samples to reduce the impact of erroneous pseudo labels in each
training iteration; (ii) progressive-confidence-based label sharpening to
reduce pseudo-label error amplification effect by sharpening most confident
predictions as pseudo/hard labels iteratively.

We evaluate our proposed CDSSL framework on several object detector adaptation
benchmarks, including synthetic-to-real, cross-camera, and normal-to-foggy
that represent different degree of domain gaps. Our approach performs
consistently better results than prior best work by 2.2% - 9.5% mAP, achieving
the new state-of-the-art object detection domain adaptation performance.
Through detailed analysis, we demonstrate the importance of each key component
of our CDSSL towards domain adaptation improvements and show how it can help
object detection tasks.

##  2 Related Work

#### Unsupervised Domain Adaptation.

State-of-the-art deep neural networks often face significant performance drop
due to the changing environments. Therefore, many unsupervised domain
adaptation (UDA) techniques are proposed, e.g. MMD distance minimization [13],
sub-space alignment [7], etc. Recently, an adversarial learning based method
with gradient reverse layer for feature-invariant learning achieves great
performance for classification adaptation problems [8]. Many works also
generalize it into other vision tasks including semantic segmentation [17,
39], object detection [4], etc. However, UDA has different implications on
image classification and object detection: (i) each input image for
classification often only contain a single object of a category, while
multiple objects of different categories may exist in a single image for
object detections; (ii) detection task consists of both classification and
localization which may require more than coarse-grained alignment as for
classification.

Figure 2: Cross-Domain Semi-Supervised Learning (CDSSL) Framework Overview.
(a) ${}^{\textbf{\\#1}}$Fine-grained domain transfer first transfers the
source to an intermediate domain to initialize high-quality pseudo labels on
the target domain. (b) Then we conduct iterative semi-supervised self-training
with combined source/intermediate/target domain data. Specifically,
${}^{\textbf{\\#2}}$imbalanced mini-batch sampling is proposed to prevent
pseudo label errors from overwhelming the mini-batch during the detector self-
training. After each round, the new model is adopted for
${}^{\textbf{\\#3}}$pseudo label sharpening on the target domain. This process
can iterate multi-times by replacing with better pseudo labels.

#### Domain Adaptation for Detection.

Most prior works on domain adaptation for object detections used adversarial
learning based methods to learn invariant features between domains. For
example, Chen et al. [4] first use adversarial learning methods to align the
image-level and instance-level features. Following that, Zhu et al. [41]
propose a region-level alignment method to target at a middle granularity.
Some other adversarial methods have also targeted at different levels with
weighting strategies [32, 38, 18]. These methods can well solve the style gaps
between different domains. But due to the challenges like multi-objects,
multi-classes, and varied-locations, they may cause feature misalignment.
Moreover, these methods mostly focus on coarse-granularity feature alignment
and do not account for content distribution gap (e.g., different object
density distribution) which are important for object detections.

#### Semi-Supervised Learning.

Semi-supervised learning (SSL) has become a promising approach utilizing both
labeled data and large amount of unlabeled data to not only reduce the
labeling cost but also improve the model performance. Many SSL methods have
been proposed including EM algorithm [28], self-training [27], co-training
[3], transduction methods [5], as well as deep learning based SSL ones [21,
37, 34]. On the other hand, SSL does not always work well, as discussed in
[29, 12]. One common assumption for SSL to work is the cluster assumption,
i.e., the labeled and unlabeled data tend to form a same or similar
distributed cluster [29, 1]. However, for domain adaptation, this assumption
often cannot be satisfied due to the large domain gaps. Another common
assumption in SSL is the low density prior, i.e., the classifier in real-world
settings favors a low-density separation manifold between classes [2]. Such
prior motivates the commonly-used hard labeling, entropy minimization method,
etc [23, 12]. In this work, when applying SSL in UDA for object detection, we
assume that our problem setting satisfies the low density prior since the
foreground/background and different classes in our datasets (i.e., street
scene datasets) are highly distinguishable.

##  3 Methodology

In this work, we address the unsupervised domain adaptation between the
labeled source domain $(X_{s},~{}Y_{s})$ and the unlabeled target domain
$(X_{t},~{}?)$, where the $?$ denotes that ground-truth annotation $Y_{t}$ in
target domain is unavailable.

#### CDSSL Framework Overview.

The CDSSL framework consists of two major steps as shown in Fig. 2: (a)
Because of the cross-domain setting, the labeled data and unlabeled data does
not meet the identical distribution assumption for semi-supervised learning.
This can cause the subsequent self-training to fail in some complex tasks.
Therefore, we first conduct domain transfer to get an intermediate domain with
smaller gaps from target domain. The initial pseudo label annotator will then
be trained on this intermediate domain to generate higher-quality initial
pseudo labels. This step only needs to be performed once in order to launch
essential next self-training step; (b) We then run semi-supervised self-
training for detector training by combing the source/intermediate domain data
(with GT labels) and the target domain data (with pseudo labels) together. To
prevent the pseudo label errors from overwhelming the training process, we
conduct imbalanced mini-batch sampling during training process. After each
self-training round, the model can often yield better performance on target
domain. Therefore, we utilize the better model for confidence-based hard
labeling to get better pseudo labels. The self-training process could iterate
multiple times till the model performance stops improving because of the
continuing and irreversible error accumulation from erroneous pseudo labels.

###  3.1 Detection-Oriented Domain Transfer

The aim of domain transfer is to transfer source-domain data into an
intermediate domain, which is closer to the target domain distribution. One
common solution is to conduct domain style translation in image level by
cyclegan [40]. In this section, we show that naive cyclegan translation can
only achieve suboptimal translation results for detection tasks. Then we
propose to optimize it by receptive filed restriction for detection-oriented
style translation.

#### Cross-Domain Style Translation.

In our setting, we assume the source domain and target domain images have two
different styles. Therefore, we adopt the popular image translation model
cyclegan [40] for our cross-domain style transfer. For cyclegan training, one
source-domain image will be translated by generators twice: from source to
target domain; and then translated back, ensuring the cycle-consistency
between original and reconstructed images. Meanwhile, two discriminators will
discriminate the generated images from real images to ensure the image styles
match the corresponding domain styles.

However, original cyclegan design does not target at any specific end tasks.
As a result, when applying to detection images, vanilla cyclegan often not
only translates the styles, but also changes the objects and backgrounds. This
is prohibitive for detection task since every object has a matched label in
the dataset. To solve this problem, we optimize it as fine-grained cyclegan to
conduct detection-oriented style translation specialized for object detection
tasks.

Figure 3: CycleGAN with large receptive field tends to translate/change the
objects and its background context, which is suboptimal for object detection
adaptation tasks.

#### Receptive Field Restriction for Fine Granularity.

As we discussed, a detection-oriented image translator should not change
foregrounds and backgrounds (i.e., global content) but focus on the low-level
styles (e.g., local texture, color and brightness). To do so, we propose a
simple yet effective method: imposing restrictions on the receptive field of
cyclegan model. As illustrated in Fig. 3 (a), in training process, we restrict
the input patch size of the generator and discriminator so that the models can
only “see” the local texture difference without knowing the global contents or
major contexts. Thus, cyclegan model can only learn to translate and
discriminate the size-restricted patch style. During testing, by using the
fully-convolutional model structure, the generator can be applied on the full-
size image for fine-grained style translation. Figure 3 (b) and (c) compare
the translation results between fine-/coarse-grained translation. For fine-
grained translation, only local texture in the source domain are changed to
the translated domain, while for coarse-grained translation, global contents
such as the background of source domain are dramatically changed to match
target domain. As shown in Section 5, such translation results often achieve
suboptimal performance compared to our fine-grained translation results.

###  3.2 Semi-Supervised Self-Training

#### Pseudo Label Initialization via Intermediate Domain.

After the domain transfer, we can get an intermediate domain $X_{m}$ that is
closer to the target domain, i.e., $X_{s}\rightarrow X_{m}\rightarrow X_{t}$.
As we mentioned earlier, the basic self-training setting directly uses the
source domain $(X_{s},~{}Y_{s})$ to train the initial annotator. But in our
setting, if source/target domain gap $X_{s}\rightarrow X_{t}$ is too large,
model trained on source domain can often produce many erroneous pseudo labels.
This can cause the following self-training to fail in complex tasks such as
object detection. To reduce the amount of erroneous pseudo labels, we conduct
annotator training on the intermediate domain $(X_{m},~{}Y_{s})$. Since the
gap between $X_{m}\rightarrow X_{t}$ is smaller, the annotator can thus
generate more accurate pseudo labels $Y_{t}^{{}^{\prime}}$ on target domain
$X_{t}$. This initialization step is only executed once, and then our self-
training can iteratively boost pseudo label quality to improve object
detection performance on the target domain.

#### Imbalanced Sampling for Error Mitigation.

Annotated by non-perfect labelers, the pseudo labels in target domain
$(X_{t},~{}Y_{t}^{{}^{\prime}})$ inevitably contain some errors. Without any
precautions, this can greatly hinder the detector’s learning process. For
example, in the RPN training with fore/back-ground classification loss:

$\small
L_{t}=-[~{}y_{i}^{{}^{\prime}}\log(p_{i})+(1-y_{i}^{{}^{\prime}})\log(1-p_{i})~{}],~{}~{}i\in
t,$ (1)

$p_{i}$ is the model predicted foreground probability. If the pseudo label
$y_{i}^{{}^{\prime}}=1$ for some background anchor, the wrong classification
loss $L_{t}$ would mislead the model to predict the $p_{i}$ for this and
similar background anchors’ confidence as 1. The model can then become biased
and more similar incorrect pseudo labels will appear later. This is called
Error Accumulation/Amplification, another major problem that prevents the
self-training from getting better performance.

To mitigate the bias of inaccurate target-domain loss $L_{t}$, we combine
source/intermediate111In practice, combining which one (source or
intermediate) depends on different datasets (which yields some slight
difference, see Supp. materials). Later on, we unify the terms as
‘source’/target domain combination. and target domain data to conduct
imbalanced sampling in each data mini-batch. Since the source-domain samples
have ground-truth labels $(X_{s},~{}Y_{s})$, the loss $L_{s}$ should be
accurate. Therefore, we statistically under-sample the target domain samples
but over-sample the source domain samples during training. In this case, the
loss of each training batch is composed of:

$\small
Loss=\sum_{i=0}^{s}{L_{i}}+\sum_{j=0}^{t}{L_{j}},~{}~{}\textbf{s.t.}~{}~{}s>t.\vspace{-1mm}$
(2)

where $s$ and $t$ denote the sampling amounts of source domain and target
domain data, respectively. In case that the pseudo labels contain errors, the
learning loss of source-domain samples $L_{s}$ can still produce reliable
gradients for each training batch. Therefore, such imbalanced sampling
strategy can provide training stabilization and error-correction effects, as
shown in Section 5.

#### Confidence-based Hard Pseudo Labeling.

At the end of each self-training round, the model’s performance on target
domain could become better than the previous annotator’s. Therefore, we can
update the pseudo labels on the target domain by applying the better model on
the target domain. To control the label quality, we only choose the most
confident predictions above the confidence threshold as pseudo labels. And
based on the low density assumption [2], we sharpen the soft predicted
probability distribution into hard labels to learn more confident
representations:

$\small\begin{split}y_{i}=\left\\{\begin{aligned}
&\operatorname*{arg\,max}_{i}(p_{i}),&\text{$\max(p_{i})>$ conf};\\\
&0,&\text{Otherwise}.\end{aligned}\right.\end{split}\vspace{-2mm}$ (3)

In the case of two classes, this will binarize the prediction $y_{i}$ based on
confidence threshold $conf$. In multi-class situations, the pseudo label would
be chosen if the highest confidence score $\max(p_{i})$ exceeds the confidence
threshold. Otherwise, it will still be considered as background.

During iterative self-training rounds, some error boxes’ confidence will be
reinforced and similar wrong predictions can appear later with higher
confidence. To avoid including too many errors, we progressively increase the
confidence threshold in later rounds to maintain our label quality.

##  4 Experimental Evaluation

###  4.1 Experiments Setup

For experiments setup, we follow the same settings as in [41]. We use Faster-
RCNN with VGG16 backbone as our object detection network. Three benchmarking
domain adaptation scenarios are evaluated, namely Synthetic to Real
(Sim2City), Cross Camera (Kitti2City) and Normal to Foggy (City2Foggy). For
the image size, previous works mainly use two settings: 512 pixels or 600
pixels as image’s shorter side. Experiments using higher resolution (600
pixels) usually achieve better performance. For fair comparison, we evaluate
our framework under both settings. For the evaluation metric, we report the
mean average precision (mAP) at IoU threshold 0.5. Code will be available.

Table 1: SIM10K to Cityscapes Adaptation Performance. Number 512 denotes images are resized with 512 pixels as shorter side, so does 600. For fair comparison, we compare both settings. Methods |  Car AP
---|---
512Baseline [non-adapt] |  33.0
512Faster-rcnn in the wild [4] |  39.0
512SCDA [41] |  43.0
512Ours w/ Self-Train (ST) |  40.6
512Ours w/ Domain-Transfer (DT) |  48.1
512Ours w/ (DT + ST) |  49.0
512Oracle Performance |  61.6
600Strong-Weak [32] |  42.3
600Multi-Level [38] |  42.8
600Ours w/ Domain-Transfer (DT) |  50.8
600Ours w/ (DT + ST) |  52.3
600Oracle Performance |  62.7
Table 2: Multi-Class Cityscapes to Foggy-Cityscapes Adaptation Performance Methods | Person | Rider | Car | Truck | Bus | Train | Motor | Bicycle | mAP
---|---|---|---|---|---|---|---|---|---
512Baseline [non-adapt] | 29.7 | 32.2 | 44.6 | 16.2 | 27.0 | 9.1 | 20.7 | 29.7 | 26.2
512Faster-rcnn in the wild. [4] | 25.0 | 31.0 | 40.5 | 22.1 | 35.3 | 20.2 | 20.0 | 27.1 | 27.6
512SCDA [41] | 33.5 | 38.0 | 48.5 | 26.5 | 39.0 | 23.3 | 28.0 | 33.6 | 33.8
512Ours w/ Self-Train (ST) | 30.0 | 33.1 | 46.5 | 11.2 | 17.7 | 5.0 | 21.1 | 30.6 | 24.4
512Ours w/ Domain-Transfer (DT) | 33.6 | 39.2 | 52.2 | 22.4 | 41.3 | 27.4 | 24.8 | 34.1 | 34.3
512Ours w/ (DT + ST) | 33.9 | 38.7 | 52.1 | 26.3 | 43.4 | 32.9 | 27.5 | 35.5 | 36.3
512Oracle Performance | 40.7 | 44.7 | 61.9 | 28.2 | 51.3 | 33.0 | 31.4 | 40.9 | 41.5
600Strong-Weak [32] | 29.9 | 42.3 | 43.5 | 24.5 | 36.2 | 32.6 | 30.0 | 35.3 | 34.3
600Multi-Level [38] | 33.2 | 44.2 | 44.8 | 28.2 | 41.8 | 28.7 | 30.5 | 36.5 | 36.0
600Ours w/ Domain-Transfer (DT) | 36.5 | 41.7 | 54.6 | 22.6 | 40.7 | 25.3 | 29.3 | 36.9 | 35.9
600Ours w/ (DT + ST) | 38.2 | 42.1 | 55.6 | 25.9 | 43.5 | 27.6 | 33.5 | 39.2 | 38.2
600Oracle Performance | 42.7 | 49.2 | 63.4 | 35.8 | 53.1 | 22.7 | 33.5 | 39.7 | 42.5
Table 3: KITTI to Cityscapes Adaptation Performance Methods |  Car AP
---|---
512Baseline [non-adapt] |  36.4
512Faster-rcnn in the wild [4] |  38.5
512SCDA [41] |  42.5
512Ours w/ Self-Train (ST) |  42.6
512Ours w/ Domain-Transfer (DT) |  41.4
512Ours w/ (DT + ST) |  45.2
512Oracle Performance |  61.6
600Ours w/ Domain-Transfer (DT) |  42.8
600Ours w/ (DT + ST) |  46.4
600Oracle Performance |  62.7

###  4.2 Domain Adaptation Performance

#### Synthetic to Real Adaptation.

We use SIM10K $\rightarrow$ Cityscapes as the domain pairs for synthetic to
real adaptation scenario. SIM10K is a synthetic dataset generated by GTA-V
game engine [20], and Cityscapes consists of images of real street scenes
taken at different cities [6]. Since SIM10K dataset only contains car
annotation, we focus on car detection task. The car detection results of our
approaches are compared with the baseline and state-of-the-art methods [4, 41,
32, 38] as shown in Table 1. Baseline represents applying source-domain
trained car detection model directly on the target domain validation set
without any domain adaptation. The Self-Train setting (ST) denotes only
applying our iterative self-training but without the domain transfer step, and
vice versa for the fine-grained domain transfer (DT) setting. Oracle
represents car detection performance trained on labeled target domain.
Compared to Baseline, applying ST and DT alone improves car AP by +7.6% and
+15.1%, respectively. By combining both methods (DT+ST), the result boosts up
to 49.0% AP. Compared to prior SOTA works, our approach achieves the current
best performance, +6.0% and +9.5% better than prior best work in 512 and 600
pixel resolutions, respectively.

#### Cross Camera Adaptation.

We also evaluate the performance of our framework in cross camera scenarios
using KITTI $\rightarrow$ Cityscapes [9]. As per setting in [4, 41], only car
annotation is used for training and evaluation. The overall results are shown
in Table 3. Compared to Baseline, ST and DT alone improves AP by +6.2% and
+5.0%, respectively. In constrast with synthetic-to-real adaptation, DT
achieves relatively smaller improvement in cross-camera adaptation and we
hypothesize that the domain style gap is smaller between two real-world street
scene datasets. This causes the improvement potential for domain transfer to
be smaller, but in turn can make self-training work better. Combining both
(ST+DT) brings +8.8% improvement over Baseline, which outperforms the previous
best result [41] by 2.7%.

#### Multi-Class Normal to Foggy Adaptation.

Multi-class adaptation scenarios are more challenging than two-classes
fore/background scenarios. In this experiment, we evaluate our framework on
Cityscapes $\rightarrow$ Foggy-Cityscapes [33]. Following [4, 41], eight
categories are used for training and evaluation. As shown in Table 2, our
approach achieves the best performance under both resolution settings,
achieving +2.5% and +2.2% mAP gain compared to prior SOTA performance.
Furthermore, our performance has nearly achieved the oracle model performance
for several classes. This indicates we have nearly closed the domain gaps,
demonstrating the effectiveness of our CDSSL framework.

Interestingly, the self-training only (ST) approach performs worse (24.4%)
than Baseline (26.2%) in such a multi-class adaptation scenario. This is
different from previous single-class results where ST only usually improves
over Baseline. By examining each class, we find that the performance drop lies
in three classes (Truck, Bus and Train). Due to the original large domain gap,
the baseline performance for these classes is significantly worse than other
classes. This causes the initial pseudo label quality for these classes to be
very low, containing a lot of errors. During self-training, such low-quality
pseudo labels will ‘teach’ the following self-trained models to produce even
worse performance: 16.2 $\rightarrow$ 11.2, 27.0 $\rightarrow$ 17.7, 9.1
$\rightarrow$ 5.0. By contrast, after our domain transfer step, these three
classes’ performance become on pair with other classes due to the reduced
domain gap. In addition, combining with self-training (ST+DT) turns to benefit
even more, achieving +10.1% mAP than Baseline, and +2.0% than DT.
Specifically, for the three classes with bad initial performance, the
improvements are even more significant: 16.2 $\rightarrow$ 26.3, 27.0
$\rightarrow$ 43.4, 9.1 $\rightarrow$ 32.9. Such evident results support our
hypothesis: when source-target gaps are too large, the self-training can
hardly work and may even hurt performance compared to baseline approach,
demonstrating the necessity of domain transfer in our overall framework.

##  5 Ablation Study for Design Modules

Figure 4: Pseudo Label Quality Visualization on three classes in Cityscapes-
Foggy. Labelers are models from baseline, self-training only, domain transfer,
and domain transfer + self-training.

#### Impact of CDSSL on Pseudo Label Quality.

In Figure 4, we visualize the effectiveness of our CDSSL framework in
improving pseudo label quality. Due to the large domain gap, Baseline model
generates very low-quality pseudo labels, i.e., erroneous predictions with
very low confidence-score. Directly trained on these Baseline pseudo labels,
the self-trained (ST) model becomes even worse at detecting these classes, as
shown in Table. 2. By contrast, with domain transfer, the DT model
significantly improves prediction accuracy with high confidence. Therefore,
high-quality pseudo labels (i.e., higher precision and recall with GT) are
generated, and Table. 2 shows the quantitative improvement. Furthermore, with
our proposed hard label sharpening, the subsequent self-trained model (ST+DT)
can receive the highest-quality supervision, thus achieving the best
performance among all settings.

Table 4: Effectiveness of Fine-grained Domain Transfer. Note *: The max. patch size is set to 360x360 for Kitti2City since its image max. height is 375. Patch-Size | Sim2City | Kitti2City | City2Foggy
---|---|---|---
Non-Adapt | 33.0 | 36.4 | 26.2
512x512 | 42.4 | 40.5* | 30.0
256x256 | 44.3 | 40.0 | 30.6
128x128 | 48.1 | 41.4 | 34.3

#### Impact of Fine-Grained Domain Transfer.

To study the impact of receptive filed restriction, we trained cyclegan models
under three patch-size settings: $128^{2}$, $256^{2}$, $512^{2}$. These models
are then used to translate the source domain data for training object
detection model using Faster-RCNN. As shown in Table 4: (i) Domain adaption
with different patch sizes all give better for object detection performance
(mAP) than no adaptation (Non-Adapt) which shows the importance reducing
domain style gap; and (ii) As patch size reduces, domain adaptation for object
detection performance (mAP) consistently improves (e.g., 5.7% difference
between $512^{2}$ and $128^{2}$ in Sim2City) as image contents are better
maintained with finer-granularity translation.

Table 5: Imbalanced Sampling Impact. “Source : Target” denotes the sampling ratio. Results are drawn from 1st self-training round. Source : Target | 0:4 | 1:3 | 2:2 | 3:1
---|---|---|---|---
Sim2City [conf=0.6] | 46.2 | 47.0 | 48.2 | 49.0
Kitti2City [conf=0.5] | 41.5 | 40.6 | 42.6 | 43.5
Kitti2City [conf=0.6] | 40.8 | 41.0 | 41.8 | 44.3
Kitti2City [conf=0.7] | 40.5 | 41.4 | 41.3 | 42.0
City2Foggy [conf=0.6] | 32.5 | 35.1 | 35.6 | 35.7

#### Impact of Imbalanced Mini-batch Sampling.

Pseudo labels generated by annotators contain errors, which can hurt the model
learning process. As a remedy, imbalanced sampling aims to mitigate the
errors’ influence during model training. To verify its efficacy, we set
different sampling ratios during training and evaluate the corresponding
training performance. As shown in Table. 5, the source and target sampling
ratio, S:T=0:4 i.e., no source data being sampled gives the worst performance
for most settings. By adding one or more source-domain samples in each mini-
batch, the performance consistently improves under almost all settings and
achieves the best at 3:1 ratio. Note that setting the ratio of S:T to 4:0
deduces to training purely on source domain, which is our Baseline setting.
Such results demonstrate that our imbalanced sampling strategy indeed makes
the training process more robust to label errors. Our hypothesis is the
ground-truth labels in source domain can still provide roughly correct
gradient direction in each training iteration. Therefore, with higher sampling
ratio of source-domain samples, the model can converge to a better minimum in
terms of overall detection loss.

#### Impact of Progressive Confidence Thresholding.

Figure 5: During iterative self-training, some wrong boxes’ confidence can continually increase, i.e., the error accumulation. Because of this, the rpn proposals also become very low-quality.  Table 6: Progressive Conf-Thresholding. Every new self-training round (one column) uses the best annotator from the last column. | Kitti2City | City2Foggy
---|---|---
Conf-Thres | 1st | 2nd | 3rd | 1st | 2nd | 3rd
0.5 | 43.5 | 43.8 | - | 34.1 | - | -
0.6 | 44.3 | 43.7 | 43.8 | 35.7 | 35.1 | 35.0
0.7 | 42.0 | 44.5 | 43.9 | 33.2 | 35.2 | 34.9
0.8 | 42.2 | 45.2 | 43.9 | 33.1 | 36.3 | 34.4
0.9 | 32.5 | 43.3 | 44.7 | - | 35.9 | 35.0

Even with imbalanced sampling strategy, the Error Accumulation problem can
still happen. As shown in Figure 5, erroneous pseudo labels can iteratively
reinforce themselves and increase their confidence scores. Because of such
wrong supervision, the RPN proposals in later self-training rounds also become
highly biased to error regions. Such substantial label errors are the major
problems that prevent iterative self-training evolving with better
performance.

As a mitigation, Table. 6 demonstrates the necessity of progressive confidence
thresholding: In both adaptation scenarios, only setting higher thresholds in
later iterations can help maintain the performance increasing trend. And after
three iterations, the performance on both datasets saturates and cannot
further improve due to the large amounts of label errors. Currently, error
accumulation in self-training is still an open problem [19, 24, 14]. And it is
more complicated for object detection task as the label error can consist of
both classification and regression errors. We plan to explore better error
mitigation approaches in future work.

##  6 Discussion

#### Where does CDSSL improvement come from?

Given that object detection needs to optimize both classification and
regression/localization tasks, it is important to understand the impact of
each individual task under different sources of supervisions (source domain GT
label vs. target domain pseudo label) on the adaptation performance.
Specifically, we set binary masks on classification loss and regression loss
for source domain GT labels and target domain pseudo labels, respectively.
This enables backpropagating the loss of either regression head or
classification head during iterative self-training.

Table 7: Analysis of performance improvement in CDSSL. The evaluated datasets are Kitti2City. Pseudo_City means training on Cityscapes dataset which are pseudo labeled by DT model.  Train Settings | AP
---|---
KITTI [baseline] | 36.4
Domain Transfer (DT) | 41.4
Pseudo_City (labeled by DT model) | 40.8
KITTI + Pseudo_City (+ only regression) | 31.3
KITTI + Pseudo_City (+ only classification) | 43.5
KITTI + Pseudo_City (+ both) | 44.3
KITTI (+ only regression) + Pseudo_City | 44.4
KITTI (+ only classification) + Pseudo_City | 43.5
KITTI (+ both) + Pseudo_City | 44.3

From Table 7, we have several interesting observations: (1) For the target-
domain pseudo labels, their classification loss supervision seems to be more
important than regression one: Enabling classification loss or classification
+ regression losses dramatically improves performance (+7.1$\sim$7.9%), while
the regression loss alone degrades the model performance than baseline
(-5.1%). This could be because the pseudo labels’ box coordinates are mostly
inaccurate, but they still partially cover the right objects, and thus enables
the model to learn better feature representations from the target domain and
provides beneficial classification supervision; (2) For the source-domain GT
labels, both loss components benefit the performance while the regression part
seems to be the major benefit: The regression and classification supervision
of source domain can improve +3.6% and +2.7% AP over Pseudo_City setting,
which is trained without source-domain labels; (3) As expected, Pseudo_City
model trained on only pseudo labels achieves even worse performance than the
pseudo label annotator (-0.6%) due to the error amplification/accumulation
effect of erroneous pseudo labels. This further demonstrates the importance of
combining source-domain GT labels in the self-training process, which can
provide training-stabilization/error-correction effects.

These empirical results show that our CDSSL framework may achieve a synergy
effect by benefiting from the source domain GT-label regression supervision
and the target domain pseudo-label classification supervision.

#### Improvement beyond Style Gap.

In object detection, detectors inherently need to handle multi-scale, multi-
class objects with varied densities in a single image. This can add new
dimension of gaps beyond style gap, which we call content distribution gap
such as object density as shown in Figure 6. Here we empirically show that
previous feature alignment works mainly address the style gap, but our
approach can further reduce the content distribution gap. We compare our
approach with Frcnn in the wild [4], one of the representative feature-level
adversarial learning methods.

Figure 6: Car density distribution comparison, showing that Content Distribution Gap can exist in different detection domains. Table 8: SIM10K to Cityscapes Adaptation on MaskRCNN. Images of full resolution are used for training and evaluation.  Methods |  Car AP
---|---
Baseline [non-adapt] |  45.4
Frcnn in the wild [4] |  52.5
Fine-Grained Domain Trans. (DT) |  62.8
DT + Frcnn in the wild [4] |  63.1 (+0.3)
DT + ST-1st Iter |  66.1 (+3.3)
DT + ST-2nd Iter |  68.1 (+5.3)
DT + ST-3rd Iter |  69.8 (+7.0)

As Table. 8 shows, Frcnn approach can achieve 52.5 AP with 7.1% improvement
over Baseline with no domain adaption. However, by first applying fine-grained
domain transfer and use this style translated dataset as the new source domain
(62.8 AP), Frcnn can barely achieve improvement (only +0.3%). We hypothesize
that the style translation has already reduced the style gap, and consequently
the feature-alignment method cannot achieve further gain since that they are
mostly targeting at the same style gap but at different levels (i.e., pixel
level vs. feature level). In contrast, our CDSSL approach by leveraging
iterative self-training (ST) can give an additional 7% improvement on top of
style translation, far more than the adversarial-based techniques, such as
Frcnn. This implies that beyond reducing the image-level or feature-level gap,
our CDSSL framework may also help reduce the content distribution gap in a
semi-supervised way, suggesting a new direction in addressing UDA for object
detection.

##  7 Conclusion

In this work, we propose CDSSL: a cross-domain semi-supervised learning
framework to address the unsupervised domain adaptation for object detection.
Specifically, we conduct domain transfer to meet the distribution assumption
of SSL and then lauch our iterative self-training. Optimizations including
imbalanced sampling and confidence-based label sharpening are proposed to
mitigate the error accumulation. Experiments demonstrate the effectiveness of
our framework, which consistently outperforms previous SOTA by 2.2% - 9.5% in
various adaptation scenarios.

## References

  * [1] Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien.  Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews].  IEEE Transactions on Neural Networks, 20(3):542–542, 2009.
  * [2] Olivier Chapelle and Alexander Zien.  Semi-supervised classification by low density separation.  In AISTATS, volume 2005, pages 57–64. Citeseer, 2005.
  * [3] Minmin Chen, Kilian Q Weinberger, and John Blitzer.  Co-training for domain adaptation.  In Advances in neural information processing systems, pages 2456–2464, 2011.
  * [4] Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  Domain adaptive faster r-cnn for object detection in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3339–3348, 2018.
  * [5] Ronan Collobert, Fabian Sinz, Jason Weston, and Léon Bottou.  Large scale transductive svms.  Journal of Machine Learning Research, 7(Aug):1687–1712, 2006.
  * [6] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.
  * [7] Basura Fernando, Amaury Habrard, Marc Sebban, and Tinne Tuytelaars.  Unsupervised visual domain adaptation using subspace alignment.  In Proceedings of the IEEE international conference on computer vision, pages 2960–2967, 2013.
  * [8] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  arXiv preprint arXiv:1409.7495, 2014.
  * [9] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  The International Journal of Robotics Research, 32(11):1231–1237, 2013.
  * [10] Bo Geng, Dacheng Tao, and Chao Xu.  Daml: Domain adaptation metric learning.  IEEE Transactions on Image Processing, 20(10):2980–2989, 2011.
  * [11] Ross Girshick.  Fast r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015.
  * [12] Yves Grandvalet and Yoshua Bengio.  Semi-supervised learning by entropy minimization.  In Advances in neural information processing systems, pages 529–536, 2005.
  * [13] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola.  A kernel two-sample test.  Journal of Machine Learning Research, 13(Mar):723–773, 2012.
  * [14] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  In Advances in neural information processing systems, pages 8527–8537, 2018.
  * [15] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [16] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell.  Cycada: Cycle-consistent adversarial domain adaptation.  arXiv preprint arXiv:1711.03213, 2017.
  * [17] Judy Hoffman, Dequan Wang, Fisher Yu, and Trevor Darrell.  Fcns in the wild: Pixel-level adversarial and constraint-based adaptation.  arXiv preprint arXiv:1612.02649, 2016.
  * [18] Han-Kai Hsu, Wei-Chih Hung, Hung-Yu Tseng, Chun-Han Yao, Yi-Hsuan Tsai, Maneesh Singh, Ming-Hsuan Yang, Wayne Treible, Philip Saponaro, Yi Liu, et al.  Progressive domain adaptation for object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1–5, 2019.
  * [19] Wei Hu, Jianfeng Chen, and Yuzhong Qu.  A self-training approach for resolving object coreference on the semantic web.  In Proceedings of the 20th international conference on World wide web, pages 87–96. ACM, 2011.
  * [20] Matthew Johnson-Roberson, Charles Barto, Rounak Mehta, Sharath Nittur Sridhar, Karl Rosaen, and Ram Vasudevan.  Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks?  In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 746–753. IEEE, 2017.
  * [21] Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling.  Semi-supervised learning with deep generative models.  In Advances in neural information processing systems, pages 3581–3589, 2014.
  * [22] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [23] Dong-Hyun Lee.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In Workshop on Challenges in Representation Learning, ICML, volume 3, page 2, 2013.
  * [24] Ming Li and Zhi-Hua Zhou.  Setred: Self-training with editing.  In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 611–621. Springer, 2005.
  * [25] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [26] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan.  Unsupervised domain adaptation with residual transfer networks.  In Advances in Neural Information Processing Systems, pages 136–144, 2016.
  * [27] David McClosky, Eugene Charniak, and Mark Johnson.  Effective self-training for parsing.  In Proceedings of the main conference on human language technology conference of the North American Chapter of the Association of Computational Linguistics, pages 152–159. Association for Computational Linguistics, 2006.
  * [28] Geoffrey McLachlan and Thriyambakam Krishnan.  The EM algorithm and extensions, volume 382.  John Wiley & Sons, 2007.
  * [29] Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow.  Realistic evaluation of deep semi-supervised learning algorithms.  In Advances in Neural Information Processing Systems, pages 3235–3246, 2018.
  * [30] Vishal M Patel, Raghuraman Gopalan, Ruonan Li, and Rama Chellappa.  Visual domain adaptation: An overview of recent advances.  IEEE Signal Processing Magazine, 2014.
  * [31] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun.  Playing for data: Ground truth from computer games.  In European conference on computer vision, pages 102–118. Springer, 2016.
  * [32] Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko.  Strong-weak distribution alignment for adaptive object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6956–6965, 2019.
  * [33] Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  Semantic foggy scene understanding with synthetic data.  International Journal of Computer Vision, 126(9):973–992, 2018\.
  * [34] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In Advances in neural information processing systems, pages 1195–1204, 2017.
  * [35] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell.  Adversarial discriminative domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7167–7176, 2017.
  * [36] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell.  Deep domain confusion: Maximizing for domain invariance.  arXiv preprint arXiv:1412.3474, 2014.
  * [37] Jason Weston, Frédéric Ratle, Hossein Mobahi, and Ronan Collobert.  Deep learning via semi-supervised embedding.  In Neural Networks: Tricks of the Trade, pages 639–655. Springer, 2012.
  * [38] Rongchang Xie, Fei Yu, Jiachao Wang, Yizhou Wang, and Li Zhang.  Multi-level domain adaptive learning for cross-domain detection.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [39] Yang Zhang, Philip David, and Boqing Gong.  Curriculum domain adaptation for semantic segmentation of urban scenes.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2020–2030, 2017.
  * [40] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.
  * [41] Xinge Zhu, Jiangmiao Pang, Ceyuan Yang, Jianping Shi, and Dahua Lin.  Adapting object detectors via selective cross-domain alignment.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 687–696, 2019.

##  8 Supplementary Materials

The supplementary materials are organized as follows to supplement several
details:

Sec. 8.1 visualizes the style translation advantages of small receptive filed
for cyclegan training in domain transfer.

Sec. 8.2 demonstrates the performance difference when combining source or
intermediate domain in iterative self-training.

Sec. 8.3 qualitatively visualizes examples to show the adaptation performance
improvement of our framework.

Sec. 8.4 quantitatively shows the performance improvements by analyzing the
label quality distribution evolution.

###  8.1 Domain Transfer: Results Visualization with Varied Patch Size

In this section, we show the benefits of restricting small receptive field
during cyclegan training. We visualize and compare the translation results of
different cyclegan models in Fig. 7. The first row (Original) is SIM10K images
without translation. Other rows show the translated images by different
cyclegans with corresponding receptive field sizes ($128^{2}\rightarrow
512^{2}$).

  * •

Qualitatively, we can observe that the translation results from small patch-
size cyclegan models ($128^{2}$) reserve the contents fairly well. Most edges
and structures of cars are still distinguishable even for small-size cars.

  * •

By contrast, for the large patch-size cyclegan models ($512^{2}$), the
translation results not only dramatically changed the context (e.g., the
backgrounds in (a) and (c)), but also modified the contents (e.g., cars in red
boxes).

Since the contents is the key of object detection, the generated destructive
artifacts of large-patch cyclegan can mislead the model to learn incorrect
features, and thus perform badly on the real target domain images.

Figure 7: CycleGAN with large receptive filed generates artifacts and corrupts
objects, which are harmful for object detection tasks.

###  8.2 Iterative Self-Training: Combination of Source or Intermediate Domain

In this section, we show the slight difference of combining source or
intermediate domain with target domain during our self-training process. Note
that, the pseudo labels on the target domain are always labeled by the
annotator trained on the intermediate domain.

The results are shown in Table. 9, which are drawn from the first iteration of
self-training experiments. We compare two settings: source+target and
intermediate+target on all three adaptation scenarios.

  * •

In Sim2City and Kitti2City adaptation, combining source+target yields slightly
better performance.

  * •

However, on City2Foggy adaptation, combining intermediate+target gives the
better performance.

The performance variation of different combination lies in the range of
$0.2\sim 1.3$ across all scenarios. Our hypothesis is that on Sim2City and
Kitti2City, since the source/target dataset have totally different contents,
the domain style translation can still introduce some artifacts. As a result,
combining source domain without artifacts may be slightly better. But for
City2Foggy, they have the same contents (cityscapes-foggy are generated by
adding fogs on cityscapes), so the style transfer may introduce less
artifacts. Therefore, combining intermediate domain can produce better
results.

For our experiments in the following self-training iterations, we follow the
best empirical results in each setting to choose the combination, i.e., we
choose source+target for Sim2City and Kitti2City, while intermediate+target
for City2Foggy.

Table 9: The Performance Difference of Self-Training in terms of different combination. All experimental results are from the first iteration of self-training. Note: * denotes a shorter training setting (6 epochs), while all the others are using 12 training epochs.  |  Sim2City |  Kitti2City |  City2Foggy
---|---|---|---
Combination |  0.7* |  0.7 |  0.5 |  0.6 |  0.7 |  0.5 |  0.6
Source+Target |  0.482 |  0.490 |  0.432 |  0.434 |  0.420 |  0.330 |  0.338
Intermediate+Target |  0.474 |  0.485 |  0.425 |  0.423 |  0.422 |  0.341 |  0.351

###  8.3 Qualitative Visualization of Performance Improvement.

In this section, we qualitatively analyze the domain adaptation performance
improvement in this section. Fig. 8 shows several examples demonstrating the
detection performance improvements brought by our fine-grained domain transfer
and iterative self-training. The evaluated dataset pair is SIM10K to
Cityscapes using MASKRCNN with ResNet50 backbone. We can mainly observe two
advantages of our adaptation framework in detection performance improvement:

  * •

The adapted model is better at recognizing highly-overlapped cars, as shown in
Fig. 8(a)-(c). This implies that our adapted models learn better car feature
representation, which is potentially benefited from both domain style transfer
and involving target-domain images in self-training.

  * •

The adapted model can also predict more accurate box coordinates, as shown in
Fig. 8(d)-(e). As we discussed earlier, this might be because we combine the
source-domain GT labels’ regression supervision in the self-training process.
Even though the pseudo labels may be inaccurate, the model can still receive
good regression supervision from source GT labels.

Figure 8: Sim2City Qualitative Visualization. (a)-(c) demonstrates one
advantage of our proposed framework is to handle the densely-overlapped car
scenarios, which is challenging for non-adapted model. (d)-(e) In addition to
that, self-training could also help model to predict more accurate box
coordinates (boxes are more tight to the objects themselves.)

###  8.4 Quantitative Visualization of Performance Improvement.

In this section, we quantitatively demonstrate the the pseudo label quality
improvement after the first iteration of self-training. To do so, we use the
ground-truth (GT) labels and calculate the pseudo labels’ IoU with GT labels
to show their IoU distribution. Note that, this is only for analysis purpose
and we did NOT use any GT labels during the self-training process.

Fig. 9 (a)-(b) shows the pseudo labels’ IoU-Confidence distribution maps on
Cityscapes training dataset. The confidence ($y$-axis) is the prediction
confidence of each box, and the IoU ($x$-axis) is the highest intersection-
over-union of this box with any ground-truth box. Comparing the pseudo label
distribution between DT model (Fig. 9 (a)) and DT+ST model (Fig. 9 (b))), we
can see that the number of high-quality boxes significantly increases (the
upper right corner), demonstrating the performance improvement brought by the
self-training.

In the Fig. 9 (c)-(d), we show the pseudo label’s IoU distribution above the
confidence threshold (0.9). First, we can see that after self-training, the
quantity of high-quality pseudo labels increase a lot (iou$>$0.5). We also
calculate the ground-truth (GT) coverage: the percentage of GT boxes that are
covered by the pseudo labels, which can roughly reflect how many objects are
detected. Clearly, we can see that self-training improves the useful GT
coverage ratio (IoU$>0.5$) from 41.7% to 52.6% (+10.9%). This means that self-
training helps the model detect more objects than the previous model.

Besides, Fig. 9 also quantitatively demonstrates the error accumulation
phenomenon as we mentioned before. Comparing DT and DT+ST model’s IoU
distributions (a) and (b), we can see that with the increase of high-quality
pseudo labels, the low-quality ones (the upper left part) also increase.
Therefore, we conduct imbalanced sampling and progressively increase the
confidence threshold to avoid including too many label errors in the pseudo
labels.

Figure 9: Pseudo Label Quality Improvement Analysis on SIM10K to Cityscapes
adaptation. Top: The IoU-Confidence distribution comparison between (a)-(b)
demonstrates that the self-training improves the pseudo label quality since
more high-IOU boxes are generated (the top right region). Bottom: The
comparison of GT box coverage ratios (c)-(d) also shows that after self-
training, the model prediction can cover more ground-truth boxes, e.g., from
41.71% to 52.60%.


# Totally real points in the Mandelbrot set

Xavier Buff<EMAIL_ADDRESS>Institut de Mathématiques de
Toulouse
Université Paul Sabatier
118, route de Narbonne
31062 Toulouse Cedex
France  and  Sarah Koch<EMAIL_ADDRESS>Department of Mathematics
530 Church Street
East Hall
University of Michigan
Ann Arbor MI 48109
United States

###### Abstract.

Recently, Noytaptim and Petsche proved that the only totally real parameters
$c\in{\overline{\mathbb{Q}}}$ for which $f_{c}(z):=z^{2}+c$ is postcritically
finite are $0$, $-1$ and $-2$ [NP]. In this note, we show that the only
totally real parameters $c\in{\overline{\mathbb{Q}}}$ for which $f_{c}$ has a
parabolic cycle are $\frac{1}{4}$, $-\frac{3}{4}$, $-\frac{5}{4}$ and
$-\frac{7}{4}$.

The research of the second author was supported in part by the NSF

## Introduction

Consider the family of quadratic polynomials
$f_{c}:{\mathbb{C}}\to{\mathbb{C}}$ defined by

$f_{c}(z):=z^{2}+c,\quad c\in{\mathbb{C}}.$

The Mandelbrot set $M$ is the set of parameters $c\in{\mathbb{C}}$ for which
the orbit of the critical point $0$ under iteration of $f_{c}$ remains
bounded:

$M:=\bigl{\\{}c\in{\mathbb{C}}~{}|~{}\forall n\geq 1,~{}f_{c}^{\circ
n}(0)\in\overline{D}(0,2)\bigr{\\}}.$

######  Definition 1.

A parameter $c\in{\mathbb{C}}$ is postcritically finite if the orbit of $0$
under iteration of $f_{c}$ is finite.

######  Definition 2.

A parameter $c\in{\mathbb{C}}$ is parabolic if $f_{c}$ has a periodic cycle
with multiplier a root of unity.

Postcritically finite parameters and parabolic parameters are algebraic
numbers contained in $M$. More precisely, $c\in{\mathbb{C}}$ is a
postcritically finite parameter if and only if $c$ is an algebraic integer
whose Galois conjugates all belong to $M$ (see [M] and [Bu]). In addition, if
$c\in{\mathbb{C}}$ is a parabolic parameter, then $4c$ is an algebraic integer
(see [Bo]); moreover, $4c$ is an algebraic unit in
$\overline{\mathbb{Z}}/2\overline{\mathbb{Z}}$ (see [M, Remark 3.2]).

######  Definition 3.

An algebraic number $c\in{\overline{\mathbb{Q}}}$ is totally real if its
Galois conjugates are all in ${\overline{\mathbb{Q}}}\cap{\mathbb{R}}$.

Recently, Noytaptim and Petsche [NP] completely determined the totally real
postcritically finite parameters.

######  Proposition 1 (Noytaptim-Petsche).

The only totally real parameters $c\in{\overline{\mathbb{Q}}}$ for which
$z\mapsto z^{2}+c$ is postcritically finite are $-2$, $-1$ and $0$.

Their proof relies on the fact that the Galois conjugates of a postcritically
finite parameter are also postcritically finite parameters, thus contained in
$M$, and on the fact that $M\cap{\mathbb{R}}=[-2,\frac{1}{4}]$ has small
arithmetic capacity. In this note, we revisit their proof. We then determine
the totally real parabolic parameters.

######  Proposition 2.

The only totally real parameters $c\in{\overline{\mathbb{Q}}}$ for which
$z\mapsto z^{2}+c$ has a parabolic cycle are $\frac{1}{4}$, $-\frac{3}{4}$,
$-\frac{5}{4}$ and $-\frac{7}{4}$.

Acknowledgments. We thank Valentin Huguin for useful discussions, and Curtis
McMullen for introducing us to these questions.

##  1\. Postcritically finite parameters

We first revisit the proof of Noytaptim and Petsche [NP].

###### Proof of Proposition 1.

Assume that $c$ is a totally real postcritically finite parameter. Then, $c$
and all of its Galois conjugates are real postcritically finite parameters,
thus lie in the interval $[-2,0]$. Indeed,

  * •

for $c\in(0,\frac{1}{4})$, the orbit of $0$ under iteration $f_{c}$ is
infinite, converging to an attracting fixed point of $f_{c}$;

  * •

for $c=\frac{1}{4}$ the orbit of $0$ under iteration $f_{c}$ is infinite,
converging to a parabolic fixed point of $f_{c}$ at $z=\frac{1}{2}$;

  * •

for $c\in(-\infty,-2)\cup(\frac{1}{4},+\infty)$ the orbit of $0$ under
iteration $f_{c}$ is infinite, converging to $\infty$.

Let $a$ be a solution of $a+1/a=c$; that is, $a^{2}-ca+1=0$. Then $a$ is an
algebraic integer of modulus $1$ with nonpositive real part, and all of its
Galois conjugates also have modulus $1$ and nonpositive real part.

By Kronecker’s theorem, $a$ is a root of unity. And since the Galois
conjugates of $a$ all have nonpositive real part, the only possibilities are
the following:

  * •

$a=-1$, which is mapped to $c=-2$;

  * •

$a={\rm e}^{\pm{\rm i}2\pi/3}$, which is mapped to $c=-1$;

  * •

$a=\pm{\rm i}$, which is mapped to $c=0$.

Therefore, the only postcritically finite parameters that are totally real are
$-2$, $-1$, and $0$. ∎

##  2\. Parabolic parameters

We now present the proof of Proposition 2. Note that $c=\frac{1}{4}$,
$c=-\frac{3}{4}$, $c=-\frac{5}{4}$ and $c=-\frac{7}{4}$ are indeed parabolic
parameters. Indeed,

  * •

$f_{\frac{1}{4}}$ has a fixed point with multiplier $1$ at $z=\frac{1}{2}$;

  * •

$f_{-\frac{3}{4}}$ has a fixed point with multiplier $-1$ at $z=-\frac{1}{2}$;

  * •

$f_{-\frac{5}{4}}$ has a cycle of period $2$ with multiplier $-1$ consisting
of the two roots of $4z^{2}+4z-1$;

  * •

$f_{-\frac{7}{4}}$ has a cycle of period $3$ with multiplier $1$ consisting of
the three roots of $8z^{3}+4z^{2}-18z-1$.

###### Proof of Proposition 2.

Assume that $c$ is a totally real parabolic parameter. Then, the Galois
conjugates of $c$ also are parabolic parameters. Either $c=\frac{1}{4}$,
$c=-\frac{3}{4}$, $c=-\frac{5}{4}$, or $c$ and all of its Galois conjugates
lie in the interval $[-2,-\frac{5}{4})$. Indeed, a parabolic cycle must
attract the orbit of $0$ under iteration of $f_{c}$. However,

  * •

for $c\in(-\frac{3}{4},\frac{1}{4})$, the orbit of $0$ under iteration $f_{c}$
converges to an attracting fixed point of $f_{c}$;

  * •

for $c\in(-\frac{5}{4},-\frac{3}{4})$ the orbit of $0$ under iteration $f_{c}$
converges to an attracting cycle of period $2$ of $f_{c}$;

  * •

for $c\in(-\infty,-2)\cup(\frac{1}{4},+\infty)$ the orbit of $0$ under
iteration $f_{c}$ converges to $\infty$.

Let us assume that $c\in[-2,-\frac{5}{4})$. Then, $b:=4c+6$ and all of its
Galois conjugates lie in the interval $[-2,1)$. Let $a$ be a solution of
$a+1/a=b$; that is, $a^{2}-ba+1=0$. Then $a$ is an algebraic integer of
modulus $1$ with real part less than $\frac{1}{2}$, and all of its Galois
conjugates also have modulus $1$ and real part less than $\frac{1}{2}$.

By Kronecker’s theorem, $a$ is a root of unity. And since the Galois
conjugates of $a$ all have real part less than $\frac{1}{2}$, the only
possibilities are the following:

  * •

$a=-1$, $b=-2$ and $c=-2$; this is not a parabolic parameter;

  * •

$a={\rm e}^{\pm{\rm i}2\pi/3}$, $b=-1$ and $c=-\frac{7}{4}$; this is indeed a
parabolic parameter;

  * •

$a=\pm{\rm i}$, $b=0$ and $c=-\frac{3}{2}$; in that case $4c=-6$ is not an
algebraic unit in $\overline{\mathbb{Z}}/2\overline{\mathbb{Z}}$ and so, $c$
is not a parabolic parameter;

  * •

$a={\rm e}^{\pm{\rm i}2\pi/5}$, $b=2\cos(\frac{2\pi}{5})$ and
$c=\frac{\sqrt{5}-13}{8}$; in this case, $f_{c}$ has an attracting cycle of
period $4$ and so, $c$ is not a parabolic parameter;

  * •

$a={\rm e}^{\pm{\rm i}4\pi/5}$, $b=2\cos(\frac{4\pi}{5})$ and
$c=\frac{-\sqrt{5}-13}{8}$; then the Galois conjugate $\frac{\sqrt{5}-13}{8}$
is not a parabolic parameter and so, $c$ is not a parabolic parameter.

This completes the proof of the proposition. ∎

Remark: the following proof that $-\frac{3}{2}$ is not a parabolic parameter
was explained to us by Valentin Huguin. It follows from [Bo] that for all
$n\geq 1$,

${\rm discriminant}\bigl{(}f_{c}^{\circ
n}(z)-z,z\bigr{)}=P_{n}(4c)\quad\text{with}\quad
P_{n}(b)\in{\mathbb{Z}}[b]\quad\text{and}\quad\pm P_{n}\text{ monic}.$

As an example,

$P_{1}(b)=-b+1,\quad P_{2}(b)=(b-1)(b+3)^{3},\quad
P_{3}(z)=(b-1)(b+7)^{3}(b^{2}+b+7)^{4},$

and

$P_{4}(z)=(b-1)(b+3)^{3}(b+5)^{6}(b^{3}+9b^{2}+27b+135)^{4}(b^{2}-2b+5)^{5}.$

Note that this yields an alternate proof that $c=\frac{1}{4}$,
$c=-\frac{3}{4}$, $c=-\frac{5}{4}$ and $c=-\frac{7}{4}$ are parabolic
parameters. In addition,

$P_{n}(0)={\rm discriminant}\bigl{(}z^{2^{n}}-z,z\bigr{)}\equiv 1{\ ({\rm
mod}\ 2)}.$

As a consequence

$P_{n}(-6)\equiv 1{\ ({\rm mod}\ 2)}.$

Thus, for all $n\geq 1$, the roots of $f_{-\frac{3}{2}}^{\circ n}(z)-z$ are
simple, which shows that $f_{-\frac{3}{2}}$ has no parabolic cycle.

## References

  * [Bo] T. Bousch, Les racines des composantes hyperboliques de $M$ sont des quarts d’entiers algébriques, Frontiers in Complex Dynamics: a volume in honor of John Milnor’s 80th birthday, A. Bonifant, M. Lyubich, S. Sutherland, editors. Princeton University Press 2014, 25–26.
  * [Bu] X. Buff, On Postcritically Finite Unicritical Polynomials, New York J. Math. 24 (2018) 1111–1122.
  * [M] J. Milnor, Arithmetic of unicritical polynomial maps, Frontiers in Complex Dynamics: a volume in honor of John Milnor’s 80th birthday, A. Bonifant, M. Lyubich, S. Sutherland, editors. Princeton University Press 2014, 15–24.
  * [NP] C. Noytaptim $\&$ C. Petsche, Totally real algebraic integers in short intervals, Jacobi polynomials, and unicritical families in arithmetic dynamics, Preprint (2022).