11institutetext:  Hikvision Research Institute, China 22institutetext:
Zhejiang University, Hangzhou, China

# Learning a Domain Classifier Bank for Unsupervised Adaptive Object Detection

Sanli Tang 11 Zhanzhan Cheng 2121 Shiliang Pu 11
Dashan Guo 11 Yi Niu 11 Fei Wu 22

###### Abstract

In real applications, object detectors based on deep networks still face
challenges of the large domain gap between the labeled training data and
unlabeled testing data. To reduce the gap, recent techniques are proposed by
aligning the image/instance-level features between source and unlabeled target
domains. However, these methods suffer from the suboptimal problem mainly
because of ignoring the category information of object instances. To tackle
this issue, we develop a fine-grained domain alignment approach with a well-
designed domain classifier bank that achieves the instance-level alignment
respecting to their categories. Specifically, we first employ the mean teacher
paradigm to generate pseudo labels for unlabeled samples. Then we implement
the class-level domain classifiers and group them together, called domain
classifier bank, in which each domain classifier is responsible for aligning
features of a specific class. We assemble the bare object detector with the
proposed fine-grained domain alignment mechanism as the adaptive detector, and
optimize it with a developed crossed adaptive weighting mechanism. Extensive
experiments on three popular transferring benchmarks demonstrate the
effectiveness of our method and achieve the new remarkable state-of-the-arts.

###### Keywords:

Object Detection, Domain Classifier Bank, Domain Alignment

##  1 Introduction

Deep neural networks have shown great power on various tasks [10, 16, 21], but
heavily rely on the amount of labelled data. In the real world, it is much
costly to annotate a large-scale dataset especially for object detection
tasks. Thus, training a model on label-rich dataset (source domain) and then
transferring to the unlabelled data (target domain), namely the unsupervised
domain adaptation (_abbr._ UDA), is a promising solution [27, 23]. For
example, the auto-annotated vehicles from self-driving simulation system such
as GTA can be used to help improve the vehicle detection performance in the
real-world.

Early researches aim at shrinking the domain gap [39] by aligning the model’s
activating responses to data from both the source and the target domains [7,
23, 36]. Inspired by the adversarial training techniques [9, 35] in image
classification task, recent two methods [4, 28] attempted to directly
incorporate the bare detectors with domain classifiers to extract the image-
level or instance-level domain-invariant features, and achieved significant
results.

Figure 1: Illustration of domain alignment. (a) shows the instance-level
feature alignment regarding to the object instances in source or target domain
where all instances share the same class. (b) shows the class-level alignment
by considering the category labels of object instances. The hollow and solid
circles are corresponding to the instance labels in target and source domains,
respectively. The arrows represent the aligning directions.

However, the image-level domain alignment (_abbr._ ImDA) strategy, such as
[28], takes no account of the significant difference in object number, size
and even the layouts in different domains. This method can be treated only as
a coarse solution and has very limited effects. Though the instance-level
(_abbr._ InDA) strategy, such as [4], considers the issues in ImDA, it still
suffers from the suboptimal problem due to lacking of considering their
category information. To be specific, it is unreasonable to train domain
classifiers by regarding all instances from source/target domain as the same
class, which prevents the model from drawing a clear distinction among
different object categories. As a result, the detector is easy to be confused.
Figure 1 (a) illustrates InDA that each instance from target domain is aligned
to its closest instance from source domain, which inevitably leads to the
misalignment between different instance categories.

Considering above issues, a more promising way is to align instance features
according to their ground truth or pseudo labels. It means the detected
objects with higher confidence in target domain should be paid more attention
on aligning the instance features regarding to their categories. For example
in Figure 1(b), the target objects predicted as class-2 (denoted as brown
hollow circles) should be aligned to the class-2 objects in source domain
(denoted as brown solid circles). Here, we treat it as class-level domain
alignment. To achieve class-level alignment in all classes, a group of domain
classifiers can be established as a domain classifier bank in which each
classifier takes the charge of aligning features of a specific class. In this
way, the closest pairs of instances between the source and target domains
refer to those objects sharing the same class from the perspective of the
detector. Note that, predicted results of domain classifiers also reveal the
effects of feature alignment, i.e., the more extent of alignment, the more
confused prediction will be made by the domain classifiers, as addressed in
[37]. Thereby, for those well-aligned features, their pseudo labels could be
added more weight when training the detector on unlabeled data from the target
domain. Then the detection performance can be further enhanced.

In this paper, we propose a fine-grained unsupervised domain adaptation method
for object detection, which consists of a domain classifier bank integrating
with a teacher-student framework. Concretely, we group the class-level domain
classifiers together to form as a bank, named as DCBank, in which each
classifier is responsible for aligning features of the specific class between
the source and target domains. Since images in target domain are unlabeled,
mean teacher [33] is employed to provide pseudo labels, e.g. the locations and
classes of the objects. The generated pseudo labels can be used to train the
DCBank. We integrate the bare object detector with mean teacher as well as the
DCBank into an unsupervised adaptive detection framework named as MDBank, and
optimize it with a crossed adaptive weighting mechanism. Here, the crossed
adaptive weights are calculated from the predicting confidence of the detector
as well as the entropy of the DCBank, and can improve both the detector and
the DCBank.

The contributions are summarized as follows: (1) We address the class-level
domain adaptation problem, and develop the domain classifier bank mechanism to
align instance-level features according to their categories. (2) We assemble a
bare detector with mean teacher as well as the designed DCBank into an
adaptive detection framework. The whole framework is jointly optimized with a
crossed weighting strategy which can improve both the detector and DCBank. (3)
Extensive experiments on three popular datasets demonstrate the effectiveness
of our method.

##  2 Related works

###  2.1 Object Detection

Object detectors based on deep neural networks can be roughly divided into two
categories: the two-stage and the one-stage. Faster R-CNN [26] is a
representative two-stage detector, where a Region Proposal Network (RPN) is
designed to provide object proposals, e.g. the coarse bounding boxes and the
probabilities of their being the foreground category. Then the cropped and
resized features are fed into a classifier in the second stage to predict
their categories and refined locations. A series of improvements [3, 11, 17]
based on Faster R-CNN have also been explored to further boost the
performance. While for one-stage detectors, YOLO [24] directly regressed the
bounding boxes and the confidence of being multiple categories, which achieved
competitive performance in a high efficiency manner. SSD [20] aimed to
increase the detection rates of objects in different scales, especially the
small ones by predicting from multiple feature maps at different resolutions.
After that, [19, 25, 34] further advanced the one-stage detectors by revising
the network structure or applying delicate training skills.

###  2.2 Domain Adaptation

Many researches [7, 30, 36] on domain adaptation struggle for bridging the gap
between the source and target domain. Earlier works tried to minimize the
discrepancy between two domains, which was defined in statistics, e.g. the
Maximum Mean Discrepancy (MMD) [1, 22, 38] or CORAL distance [31, 32]. Recent
methods [35] based on adversarial training aligned the feature distribution by
cheating domain classifiers that were trained to distinguish the image
features from different domain. [37] proposed the transferable attention that
assigns different weights to feature maps according to the predicting
confidence of the domain classifiers. Derived from Mean Teacher [33] in semi-
supervised learning, self-ensembling [7] was proposed to extract domain-
invariant features by minimizing the outputs of the teacher and the student
with augmented inputs. All above methods were examined in image classification
tasks.

Recently, researchers start to pay attention to domain adaptation in object
detection. In general, existing methods could be summarized as three types of
domain alignments: the input-level, the feature-level and the output-level.
(1) The input-level aligning techniques usually adopted generative models to
directly transfer input images from the source domain to the target domain
while keeping the labels unchanged [13]. Then, the generated labeled images
could be utilized to train a detector in a fully supervised manner. (2) For
feature-level alignment, [4] aligned both top features in backbone and
instance features by adversarial training, where the instance-level domain
classifier treated the object features as the same class only if they come
from the same domain. Strong weak domain adaptation (SWDA) [28] argued that
the precisely matching on global features was likely to hurt the performance
confronting with large domain gap, and adopted a weak image-level domain
classifier by focal loss [19] to align features. (3) For output alignment,
mean teacher with object relation (MTOR) [2] made three kinds of consistency
regularization based on two relational graphs in teacher and student networks,
which showed a promising way of self-ensembling framework in UDA object
detection tasks. [15] addressed the UDA detection problem by training a
detector on the target domain with noisy object bounding box. Here, we focus
on the feature-level domain alignment.

Unlike previous feature-level alignment methods in which aligning domain
features regardless of their classes, in this paper, we try to achieve a fine-
grained instance-level domain alignment by regarding to instance categories.

##  3 Preliminary Work

We build the proposed framework MDBank based on Faster R-CNN and mean teacher,
as illustrated in Figure 2. Notice that we select Faster R-CNN [26] as the
bare detector for fair comparison with previous methods [2, 4, 28].

Faster R-CNN detector. Faster R-CNN [26] is a two-stage detector, which
consists of a feature extractor backbone $\mathcal{F}_{\rm conv}$, a region
proposal network (RPN) $\mathcal{F}_{\rm RPN}$ and a region convolutional
neural network (RCNN) $\mathcal{F}_{\rm RCNN}$. For the input data $\bf x$,
the image-level features are first calculated by ${\bf f}=\mathcal{F}_{\rm
conv}({\bf x})$, and the object proposals are represented by ${\bf
r}=\mathcal{F}_{\rm RPN}({\bf f})$. Then the instance-level features $\bf
f_{r}$ is obtained according to ${\bf r}$, and the bounding boxes b and the
category probabilities p can be predicted by $({\bf b,p})=\mathcal{F}_{\rm
RCNN}(\bf f_{r})$.

Mean Teacher in Faster R-CNN. Mean teacher [33] $\mathcal{T}$ is used to
provide relatively robust pseudo labels for unlabelled samples, which is
established as the same network structure to the student $\mathcal{S}$. Its
parameters $\mathcal{W}^{\mathcal{T}}$ at $t$-th iteration are updated in a
moving average manner:
$\mathcal{W}^{\mathcal{T}}_{t}={\alpha}\mathcal{W}^{\mathcal{T}}_{t-1}+(1-\alpha)\mathcal{W}^{\mathcal{S}}_{t}$,
where $\mathcal{W}^{\mathcal{S}}_{t}$ are student’s parameters at $t$-th
iteration and $\alpha$ is the moving average factor for controlling the update
speed of the teacher. Following [2], the proposals ${\bf r}^{\mathcal{T}}$
from the teacher $\mathcal{T}$ are fed into $\mathcal{F}_{\rm RCNN}$ of both
the student and teacher detectors, respectively.

Figure 2: The Faster R-CNN with mean teacher framework. For data from the
source domain, it is trained in a supervised routine by minimizing the
detecting objectives in Faster R-CNN [26]. For unlabeled data from the target
domain, it is trained by optimizing the consistency regularization between the
teacher’s and the student’s prediction. Teacher detector shares its proposals
with the student when training on the unlabeled data.

For the labeled data $\bf(x_{s},y_{s})$ from source domain ${\bf D_{s}}$, the
normal supervised routine in [26] is applied to train the student detector by
minimizing the supervised detection objective $\mathcal{L}_{\rm det}$. For the
unlabeled data $\bf x_{t}$ from target domain ${\bf D_{t}}$, the teacher model
is used to obtain object proposals ${\bf r}^{\mathcal{T}}$ and their pseudo
labels $({\bf b^{\mathcal{T}},p^{\mathcal{T}}})$. Then the augmented input
$\bf\tilde{x}_{t}$ is fed into the student detector along with ${\bf
r}^{\mathcal{T}}$ to obtain the predictions $({\bf
b^{\mathcal{S}},p^{\mathcal{S}}})$. Thus, the consistency regularization of
mean teacher could be calculated as following:

$L_{\rm mt}=L_{p}({\bf p}_{r}^{\mathcal{T}},{\bf
p}_{r}^{\mathcal{S}})+L_{b}({\bf b}_{r}^{\mathcal{T}},{\bf
b}_{r}^{\mathcal{S}}),$ (1)

where $L_{p}(\cdot,\cdot)$ and $L_{b}(\cdot,\cdot)$ are consistency objectives
for the classification and bounding box regression between the teacher and the
student detector, respectively.

##  4 Methodology

Figure 3: The architecture of MDBank for UDA detection tasks. Based on Faster
R-CNN detector and mean teacher, the teacher detector shares the proposals
with the student to further align the instance-level feature in the same
region. DCBank consists of a group of domain classifiers that performs class-
level feature alignment in an adversarial learning manner. Crossed adaptive
weighting mechanism is applied on instance-level feature $f_{r}$ between the
consistency regularization in mean teacher and adversarial objective in
DCBank. The confidence from the second stage of teacher detector
$\mathcal{F}_{\rm RCNN}^{\mathcal{T}}$ is used as a gate function for training
the DCBank module while the entropies from the DCBank weight the consistency
objective of different categories, respectively. The ’w-ent’ and ’w-mse’ are
the abbreviations of weighted cross-entropy and weighted mean square error,
respectively.

In this section, we describe the proposed framework MDBank specifically, shown
in Figure 3.

###  4.1 Domain Classifier Bank

The existing adaptive detection methods [4, 28] used a single domain
classifier to align instance-level features of different labels, which might
prevent the detectors from distinguishing their categories. To align the
features according to their categories, a group of domain classifiers are
established and each of them is responsible for aligning features of a
specific class.

Formally, we establish the domain classifier bank (DCBank) as
$\bf{\mathcal{D}}=\\{\mathcal{D}_{i}\\}_{i=1}^{C+1}$, where $C+1$ denotes the
number of domain classifiers corresponding to $C$ object categories and the
background category. All classifiers don’t share parameters with each other.
These domain classifiers are trained to distinguish the instance level
features from either the source domain or the target domain. For the instance
level feature ${\bf f}_{r,s}$ from source domain with category label
$y_{r,s}$, only the domain classifier $\mathcal{D}_{y_{r,s}}$ in the DCBank is
activated to align the region features. For the instance level feature ${\bf
f}_{r,t}$ from the target domain with unknown category, the prediction
$p_{r}^{\mathcal{T}}$ of the teacher model is referred as the pseudo label to
activate the specific domain classifiers to align the region features, as
illustrated in Figure 3. For example, the category of maximum predicted
confidence $\tilde{y}_{r,t}={\rm argmax}({\bf p_{r}}^{\mathcal{T}})$ can be
regard as its pseudo label such that only the $\tilde{y}_{r,t}$-th domain
classifier $\mathcal{D}_{\tilde{y}_{r,t}}$ is adopted to make the instance
level feature alignment. In fact, a more soft and robust way is to
simultaneously select several domain classifiers for domain alignment
according to the uncertainty of the teacher model (detailed in Section 4.2).

According to the label of the instance-level features, domain classifiers can
be trained by minimizing the objective $L_{\rm dcbank}$:

$\begin{split}L_{\bf{\mathcal{D}}}({\bf f_{r}})&=\frac{1}{|{\bf
r}_{K}^{\mathcal{T}}|}\sum_{r{\in}{\bf
r}_{K}^{\mathcal{T}}}\sum_{i=1}^{C+1}G(y_{r,t}){\rm
log}(1-\mathcal{D}_{i}({\bf f}_{r,t}))\\\ &+\frac{1}{|{\bf
r}_{K}^{\mathcal{S}}|}\sum_{r{\in}{\bf
r}_{K}^{\mathcal{S}}}\sum_{i=1}^{C+1}G(y_{r,s}){\rm log}(\mathcal{D}_{i}({\bf
f}_{r,s}))\\\ &=L_{\bf{\mathcal{D}},s}({\bf
f}_{r,s})+L_{\bf{\mathcal{D}},t}({\bf f}_{r_{t}}),\end{split}$ (2)

where $G(\cdot)$ is the activation function to decide which domain classifier
is trained. And the instance-level and class-level feature alignment is
achieved by adversarial training:

$\begin{split}L_{\rm adv}=\max\limits_{\mathcal{F}_{\rm
conv}^{\mathcal{S}}}&\min\limits_{\bf{\mathcal{D}}}E_{{\bf x}\sim{\bf
x_{s}}}[L_{\bf{\mathcal{D}},s}(\mathcal{F}_{\rm conv}^{\mathcal{S}}({\bf
x})_{\bf r})]\\\ &+E_{{\bf x}\sim{\bf
x_{t}}}[L_{\bf{\mathcal{D}},t}(\mathcal{F}_{\rm conv}^{\mathcal{S}}({\bf
x_{t}})_{\bf r})].\end{split}$ (3)

That is, the domain classifiers in DCBank try to distinguish the domain label
of the instance-level features conditioned by their GT/pesudo labels, while
the feature extractors are trained to generate domain-invariant features to
cheat those classifiers. Inspired by the Gradient Reversal Layer (GRL) [8]
where signs of output gradients are flipped, the adversarial loss in Equation
3 can be easily implemented by adding GRL onto the instance-level features
${\bf f_{r}}$ before the domain classifier bank module.

Since the DCBank module is deployed to align instance-level features, which
are much smaller than the image-level feature, it will add a little storage
and computational effort when training. When testing, it can be completely
omitted without any addition cost.

###  4.2 Crossed Adaptive Weighting

Moreover, domain classifiers in DCBank can be trained in a robust manner by
incorporating the prediction confidence $\bf{p_{r,t}}$ from the teacher model
such that a soft gate function $G(\cdot)$ can be deployed to weight the domain
classifiers. Here, we define two types of the gate function:

$\displaystyle G_{1}(\bf{p_{r,t}})$ $\displaystyle={\rm
onehot}(\mathop{{\arg\max}}_{1{\leq}c{\leq}C+1}(\bf{p_{r,t}}))$ (4)
$\displaystyle G_{2}(\bf{p_{r,t}})$ $\displaystyle=\bf{p_{r,t}}^{\gamma},$ (5)

where $G_{1}(\cdot)$ only activates the domain classifier at the index of
maximum prediction score, and $G_{2}(\cdot)$ activates all the domain
classifiers regarding to their confidence score. Since prediction of the
teacher model might be incorrect, DCBank with the soft gate function $G_{2}$
is believed to be more robust comparing to the hard gate function $G_{1}$.

In the meanwhile, the prediction of DCBank can reveal the extent of current
instance-level feature to be aligned. The more extent of the feature
alignment, the more consistency should be made between the teacher and the
student. Since the domain classifier bank plays a discriminator role by
adversarial training, the entropy of the prediction could be used to weight
the consistency discrepancy regularization. Formally, given the output scores
${\bf d_{r}}=(d_{r,1},d_{r,2},\cdots,d_{r,C+1})$ of the DCBanks where
$d_{r,i}$ is the output of the $i$-th domain classifier predicted on the
$r$-th region-level feature, the entropy of the domain classifier bank can be
calculated as

$\bf{e_{r}}=-(\bf{p_{r}}{\log}\bf{p_{r}}+(1-\bf{p_{r}}){\log}(1-\bf{p_{r}})).$
(6)

Therefore, the consistency regularization in Equation 1 can be rewritten as

$L_{\rm mt}=||{\bf
e_{r}\odot(p_{r}^{\mathcal{T}}-p_{r}^{\mathcal{S}})}||_{2}+||{\bf
e_{r}\odot(b_{r}^{\mathcal{T}}-b_{r}^{\mathcal{S}})}||_{2},$ (7)

where $\odot$ represents the element-wise product. For simplicity, we use
$l_{2}$ norm to measure both the classification and bounding box regression
discrepancy between the teacher and student detector. In Figure 3, we
illustrate the adaptive weighting mechanism between the consistency
regularization and the adversarial objective.

###  4.3 Overall Objective

The overall objective consists of three parts: the normal supervised routine
of Faster R-CNN $L_{\rm det}$, the weighted instance level consistency
regularization from mean teacher $L_{\rm mt}$ and the adversarial objective
from DCBank $L_{\rm adv}$:

$L_{\rm MDBank}=L_{\rm det}+{\eta}(L_{\rm mt}+{\lambda}L_{\rm adv}),$ (8)

in which the parameter $\eta$ controls the weight of MDBank training on the
unlabelled data from the target domain, $\lambda$ is the trade-off between
output-level alignment from teacher detector and the class-level alignment
from MDBank.

##  5 Experiments

We evaluate the proposed method on three public domain shift benchmarks:
Sim10k [14] to CityScape [5], PASCAL VOC [6] to Clipart [13] and CityScapes to
Foggy CityScapes [29], which represent three scenarios: synthetic to real
scenario, normal to foggy weather scenario and photographic to comic scenario,
respectively.

###  5.1 Dataset

SIM10K is a synthetic dataset containing 10,000 training images collected from
a synthetic driving game Grand Theft Auto V (GTA5) with bounding box
annotation only for cars. CityScapes is an urban street dataset whose images
are captured by a car-mounted camera. Since these images are annotated in
pixel for semantic segmentation task, following [4, 28], we generate the
tightest axis-aligned rectangles as the bounding box labels according to the
instance segmentation mask. Foggy CityScapes is established based on
CityScapes where the images are synthetically rendered with fog according to
the depth map, in which each image is rendered in three different levels:
$\beta=0.005,0.01,0.02$. Following [2, 28], the heaviest foggy images
($\beta=0.02$) are used in our experiments. PASCAL VOC contains images of 20
categories with bounding box annotations. Following the common evaluation
protocol, we use both the training and validation split of PASCAL VOC 2007 and
2012 for training, which leads to about 15k images. Clipart contains 1k comic
images sharing the same categories as the PASCAL VOC dataset. Following the
setting in [28], all images are used for unsupervised training and testing.

###  5.2 Implement Details

The backbone network of Faster-RCNN is implemented by ResNet-50 [12] with a
Feature Pyramid Network (FPN) [18]. In training stage, each image is resized
as the short size between (960, 1440) pixels while keeping the ratio
unchanged. In particular, images for student model are applied additional
augmentation by randomly adjusting the image contrast in scales$(0.5,1.5)$,
the saturation in scales $(0.5,1.5)$ and the brightness in $(-32,32)$ RGB
value. Notice that the above data augmentations keep the objects’ categories
and bounding boxes unchanged. The proposed model is trained on 2 GPUs with
batch-size=2. We follow the hyper-parameter setting of the base Faster R-CNN
to train the images from source domain. While in target domain, since the
labels are unknown, we select the top 512 proposals with highest confidence
scores produced by RPN. Then, those instance-level features are further
aligned in the proposed DCBank module. The moving average weight $\alpha$ for
updating the teacher model is set to $0.99$ by default.

###  5.3 Declaration for Fair Comparison

We verify our method by considering the following settings:

  * •

The origin faster R-CNN model trained on the source domain without any
adaptation is treated our baseline, denoted as Faster.

  * •

To evaluate the effectiveness of DCBank, we replace the DCBank with a single
instance-level domain classifier without class-level alignment, denoted as
${\rm MT}_{\rm ins}$.

  * •

To evaluate the effectiveness of crossed adaptive weighting mechanism, we
removes the entropy weighting for consistency regularization and replace
confidence weighting for adversarial objective by hard label as the gate
function $G_{1}$ in Equation 4, denoted as ${\rm MDBank}_{\rm H}$.

  * •

The Oracle model directly trained on the target domain in a fully supervised
manner, denoted as Oracle.

The performance of Source/Target only can be regarded as the lower/upper bound
without domain adaptation strategy, which are trained in fully supervised
manner.

We also compare MDBank with current state-of-the-arts: (1) DA [4] using image-
level and instance-level domain classifiers as well as a consistency
regularizer, (2) SWDA [28] adopting strong local and weak global feature
alignment as well as a context-vector based regularization, and (3) MTOR [2]
incorporating instance-level, inter-graph and intra-graph consistency based on
a relationship graph without domain classifiers. For fairly comparison, we
here re-implement DA, SWDA and MTOR with region level consistency with the
same ResNet-50 with FPN structure.

###  5.4 Cross Domain Detection

Table 1: Experiment results on CityScapes to Foggy CityScapes transfer. The mean average precision (mAP) is evaluated on 8 categories under Foggy CityScapes validation set. The notations G, I, C and AW separately indicate the global-level (or image-level) alignment, instance-level alignment, class-level alignment and adaptive weighting, respectively. method | G | I | C | AW | person | rider | car | truck | bus | train | mcycle | bicycle | mAP
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Faster |  |  |  |  | 30.6 | 41.5 | 40.2 | 6.2 | 38.3 | 48.9 | 7.1 | 13.8 | 28.3
DA [4] | ✓ | ✓ |  |  | 39.4 | 48.1 | 48.8 | 31.0 | 42.9 | 54.9 | 7.8 | 18.1 | 36.4
SWDA [28] | ✓ |  |  |  | 45.8 | 49.2 | 56.2 | 31.1 | 47.0 | 57.5 | 11.2 | 21.9 | 40.0
MTOR [2] |  |  |  |  | 40.4 | 49.7 | 57.6 | 30.1 | 47.9 | 58.6 | 16.9 | 27.1 | 41.0
${\rm MT}_{\rm ins}$ |  | ✓ |  |  | 46.1 | 48.4 | 54.2 | 30.8 | 45.9 | 56.4 | 20.9 | 25.9 | 41.1
${\rm MDBank}_{\rm H}$ |  | ✓ | ✓ |  | 45.8 | 49.9 | 57.2 | 32.9 | 49.3 | 59.1 | 21.0 | 29.1 | 43.0
MDBank |  | ✓ | ✓ | ✓ | 44.3 | 50.0 | 58.4 | 34.9 | 48.7 | 59.1 | 26.1 | 28.7 | 43.8
Oracle |  |  |  |  | 46.4 | 54.0 | 65.7 | 41.3 | 54.6 | 64.8 | 34.4 | 30.5 | 48.9

####  5.4.1 Normal to foggy weather images

The comparison results for normal weather to foggy weather (CityScapes to
Foggy CityScape) domain are summarized in Table 1, which illustrates the AP of
8 common objects in urban street scenario and their mean AP (mAP). Our MDBank
achieves the best 43.8% mAP with the margin 2.8% comparing to the second
runner MTOR [2]. Since the foggy images in target domain are rendered directly
from source domain, the domain gap between them is smaller than other
scenarios. Naturally, the teacher model can provide more convincible
predicting results on unlabeled images. Thus the proposed MDBank with hard
pseudo label ${\rm MDBank}_{\rm H}$ shows similar performance as using soft
label 43.0% v.s. 43.8% mAP. We also note that MTOR and ${\rm MT}_{\rm ins}$
achieve similar performance, which indicates that applying a single domain
classifier to simply align instance-level feature has limited effects for
adaptive object detection. In summary, with the help of DCBank, our MDBank
surpasses MTOR and ${\rm MT}_{\rm ins}$ by around 2.7% mAP. In Figure 5, we
illustrate the detection result under Foggy Cityscapes.

Table 2: Experiment results on SIM10K to CityScapes transfer. The average precision (AP) is evaluated on the car category. The notation G, I, C, AW is following Table 1. method | G | I | C | AW | car AP on target
---|---|---|---|---|---
Faster |  |  |  |  | 34.9
DA [4] | ✓ | ✓ |  |  | 43.1
SWDA [28] | ✓ |  |  |  | 47.8
MTOR [2] |  |  |  |  | 54.9
${\rm MT}_{\rm ins}$ |  | ✓ |  |  | 55.8
${\rm MDBank}_{\rm H}$ |  | ✓ | ✓ |  | 56.3
MDBank |  | ✓ | ✓ | ✓ | 56.3
Oracle |  |  |  |  | 65.9

####  5.4.2 Synthetic to real images

We first evaluate the performance of MDBank on the synthetic (SIM10K) to real
(CityScapes) domain. Table 2 shows the performance under CityScapes validation
set of average precision rate (AP) on car category. Since there are only one
category (car) to be detected, the MDBank performs closed to the ${\rm
MT}_{\rm ins}$ and ${\rm MDBank}_{\rm H}$, where DCBank only consists of two
domain classifiers. Own to the class-level feature alignment by DCBank, our
MDBank still outperforms MTOR by 1.4%, which achieves the best 56.3% of AP.

####  5.4.3 Photographic to comic images

In this experiment, we analyze our method by evaluating on images from real
(PASCAL VOC) to artistic (Clipart) domain. The evaluation results are shown in
Table 3 with the AP of 20 common objects and their mAP. Though the source and
target domain are much dissimilar, as illustrated in Figure 5, our MDBank
method also achieves best performance of 45.4% mAP, which improves about 3%
comparing to current best result achieved by MTOR. Note that ${\rm MT}_{\rm
ins}$ even performs worse than the MTOR, which reveals that the simple
instance-level alignment regardless of categories information might have side
effect as the number of category growing. MDBank surpassing ${\rm MDBank}_{\rm
H}$ by 1.5% verifies the effectiveness of the crossed adaptive weighting
strategy. Since the all images are used for unsupervised adaptive training
[28], the performance of the oracle is meaningless that it is trained on the
validation set in a fully supervised manner.

Table 3: Experiment results on dissimilar domain transfer from PASCAL VOC to Clipart datasets. The mean average precision (mAP) is evaluated on 20 categories on all 1k images in Clipart. The mAP of oracle is only for referenced since all the images from target domain are used for training following the setting in [28]. The table is transposed for fully visualization. method | Faster[26] | DA[4] | SWDA[28] | MTOR[2] | ${\rm MT}_{\rm ins}$ | ${\rm MDBank}_{H}$ | MDBank | Oracle
---|---|---|---|---|---|---|---|---
areo | 18.6 | 12.7 | 37.0 | 39.9 | 33.5 | 40.2 | 42.5 | 62.5
bycle | 34.6 | 40.3 | 67.2 | 74.5 | 67.1 | 76.0 | 70.4 | 77.8
bird | 17.1 | 21.4 | 30.6 | 22.8 | 24.8 | 26.3 | 37.8 | 77.0
boat | 11.2 | 16.2 | 28.3 | 39.4 | 42.7 | 39.4 | 38.7 | 60.5
bottle | 23.8 | 35.7 | 44.6 | 45.8 | 51.3 | 38.5 | 48.8 | 61.1
bus | 44.3 | 29.3 | 65.0 | 58.8 | 55.9 | 69.1 | 58.0 | 86.6
car | 23.4 | 29.6 | 41.9 | 56.3 | 56.2 | 59.1 | 57.7 | 81.8
cat | 11.3 | 0.6 | 13.1 | 13.0 | 13.2 | 9.7 | 19.5 | 67.1
chair | 35.5 | 29.6 | 52.2 | 56.9 | 58.0 | 58.2 | 52.9 | 77.9
cow | 5.2 | 39.1 | 42.7 | 18.6 | 31.2 | 36.9 | 30.6 | 76.9
table | 22.8 | 18.4 | 22.3 | 36.7 | 34.9 | 31.6 | 36.5 | 77.4
dog | 6.0 | 10.9 | 7.2 | 5.8 | 13.9 | 10.8 | 15.3 | 75.1
horse | 21.2 | 19.7 | 26.9 | 29.4 | 30.0 | 32.2 | 38.0 | 74.8
bike | 40.8 | 61.5 | 76.3 | 79.0 | 58.9 | 80.2 | 85.1 | 87.2
person | 29.0 | 54.1 | 53.1 | 58.7 | 63.9 | 65.8 | 66.3 | 85.5
plant | 35.4 | 41.2 | 55.0 | 64.5 | 57.1 | 56.9 | 57.1 | 73.2
sheep | 0.4 | 16.9 | 11.2 | 11.4 | 8.9 | 18.3 | 17.3 | 76.6
sofa | 18.2 | 16.5 | 24.1 | 19.9 | 25.6 | 24.8 | 17.2 | 69.0
train | 24.9 | 16.3 | 48.9 | 56.3 | 45.1 | 41.9 | 58.5 | 70.4
tv | 22.8 | 33.7 | 51.2 | 62.0 | 58.6 | 61.2 | 59.4 | 81.3
mAP | 22.3 | 27.2 | 40.0 | 42.5 | 41.5 | 43.9 | 45.4 | 75.0

Figure 4: The effect of the trade off parameters $\lambda$ and $\eta$ under
CityScapes to Foggy CityScapes transfer.

####  5.4.4 Analysis of hyper-parameter $\lambda$ and $\eta$

$\lambda$ controls the trade-off of the student detector learned from the
source and target domain, while $\eta$ weights the objective of the DCBank
module. Figure 4 shows the mAP under different value of hyper-parameters
$\lambda$ and $\eta$, respectively. We find that MDBank is relatively robust
to the trade off parameter $\lambda$ in a wide range. MDBank can achieve the
best 44.1% mAP when $\lambda=0.1$ and $\eta=5.0$.

####  5.4.5 Visualization on the target domain

In Figure 5, we illustrate examples of detection results by the proposed
MDBank on the target domain. The detection results show that the MDBank is
relative robust to the similar and dissimilar domain transfer. In Figure 6, we
show the instance-level feature distribution of the proposed MDBank and DA
[4]. Notice that DA applies instance-level domain alignment without
considering their categories. Though both MDBank and DA achieve similar
alignment referring to their domain labels as in Figure 6 (a) and (c), our
MDBank shows more distinguishable boundaries among instances of different
categories as illustrated in 6 (d). Since there are much more instance
categories than the previous transfer tasks, MDBank outperforms DA with a
large gap by applying the class-level feature alignment.

Figure 5: The examples of detection results from the proposed MDBank on the
target domain. From left to right is PASCAL VOC to Clipart, SIM10K to
CityScapes and CityScapes to Foggy CityScapes transfer. For clarity, the
confidence score is omitted on images from CityScapes and Foggy Cityscapes.
Notice that ’car’ is the only object to be detected in SIM10K to CityScapes
transfer.

(a) DA/domain

(b) DA/category

(c) MDBank/domain

(d) MDBank/category

Figure 6: Evidences of instance-level feature distribution on PASCAL VOC to
Clipart transfer. Features belonging to the first 10 classes in Table 1 are
illustrated for better visualization. Here we compare the proposed MDBank and
DA[4]. (a) and (c) show the instance-level feature distribution with red/blue
color representing instances from source/target domain. (b) and (d) show their
category labels with different colors. Although the instance-level features
are well-aligned both in MDBank and DA by referring to (a) and (c), the MDBank
achieves more accurate class-level alignment by referring to (b) and (d).

##  6 Conclusions

In this paper, we present an MDBank framework including a mean teacher with a
domain classifier bank for domain adaptation object detection problem in
unsupervised manner. Our key contribution is the domain classifier bank module
that respectively aligns the instance-level features according to their
category labels. To align the unlabelled data from target domain, a mean
teacher paradigm is incorporated to provide robust pseudo labels while
applying instance-level prediction consistency between the teacher and student
detector. Besides, a crossed weighting mechanism is then proposed to
adaptively connect the DCBank and mean teacher to boost their performance.
Experiment shows that our MDBank achieves the new state-of-the-arts on
CityScapes, Foggy CityScapes, SIM10K, PASCAL VOC and Clipart Datasets for
unsupervised domain adaptation detection.

## References

  * [1] Borgwardt, K.M., Gretton, A., Rasch, M.J., Kriegel, H.P., Schölkopf, B., Smola, A.J.: Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics 22(14), e49–e57 (2006)
  * [2] Cai, Q., Pan, Y., Ngo, C.W., Tian, X., Duan, L., Yao, T.: Exploring object relation in mean teacher for cross-domain detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 11457–11466 (2019)
  * [3] Cai, Z., Vasconcelos, N.: Cascade r-cnn: Delving into high quality object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6154–6162 (2018)
  * [4] Chen, Y., Li, W., Sakaridis, C., Dai, D., Van Gool, L.: Domain adaptive faster r-cnn for object detection in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3339–3348 (2018)
  * [5] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3213–3223 (2016)
  * [6] Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. International journal of computer vision 88(2), 303–338 (2010)
  * [7] French, G., Mackiewicz, M., Fisher, M.: Self-ensembling for domain adaptation. arXiv preprint arXiv:1706.05208 (2017)
  * [8] Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. arXiv preprint arXiv:1409.7495 (2014)
  * [9] Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., Lempitsky, V.: Domain-adversarial training of neural networks. The Journal of Machine Learning Research 17(1), 2096–2030 (2016)
  * [10] Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision. pp. 1440–1448 (2015)
  * [11] He, K., Gkioxari, G., Dollár, P., Girshick, R.B.: Mask r-cnn. 2017 IEEE International Conference on Computer Vision (ICCV) pp. 2980–2988 (2017)
  * [12] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [13] Inoue, N., Furuta, R., Yamasaki, T., Aizawa, K.: Cross-domain weakly-supervised object detection through progressive domain adaptation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5001–5009 (2018)
  * [14] Johnson-Roberson, M., Barto, C., Mehta, R., Sridhar, S.N., Rosaen, K., Vasudevan, R.: Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks? arXiv preprint arXiv:1610.01983 (2016)
  * [15] Khodabandeh, M., Vahdat, A., Ranjbar, M., Macready, W.G.: A robust learning approach to domain adaptive object detection. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 480–490 (2019)
  * [16] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems. pp. 1097–1105 (2012)
  * [17] Li, Z., Peng, C., Yu, G., Zhang, X., Deng, Y., Sun, J.: Light-head r-cnn: In defense of two-stage object detector. arXiv preprint arXiv:1711.07264 (2017)
  * [18] Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2117–2125 (2017)
  * [19] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 2980–2988 (2017)
  * [20] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: Ssd: Single shot multibox detector. In: European conference on computer vision. pp. 21–37. Springer (2016)
  * [21] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3431–3440 (2015)
  * [22] Long, M., Zhu, H., Wang, J., Jordan, M.I.: Deep transfer learning with joint adaptation networks. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70. pp. 2208–2217. JMLR. org (2017)
  * [23] Raj, A., Namboodiri, V.P., Tuytelaars, T.: Subspace alignment based domain adaptation for rcnn detector. arXiv preprint arXiv:1507.05578 (2015)
  * [24] Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 779–788 (2016)
  * [25] Redmon, J., Farhadi, A.: Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 (2018)
  * [26] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems. pp. 91–99 (2015)
  * [27] Saenko, K., Kulis, B., Fritz, M., Darrell, T.: Adapting visual category models to new domains. In: European conference on computer vision. pp. 213–226. Springer (2010)
  * [28] Saito, K., Ushiku, Y., Harada, T., Saenko, K.: Strong-weak distribution alignment for adaptive object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6956–6965 (2019)
  * [29] Sakaridis, C., Dai, D., Van Gool, L.: Semantic foggy scene understanding with synthetic data. International Journal of Computer Vision 126(9), 973–992 (2018)
  * [30] Sankaranarayanan, S., Balaji, Y., Castillo, C.D., Chellappa, R.: Generate to adapt: Aligning domains using generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8503–8512 (2018)
  * [31] Sun, B., Feng, J., Saenko, K.: Return of frustratingly easy domain adaptation. In: Thirtieth AAAI Conference on Artificial Intelligence (2016)
  * [32] Sun, B., Saenko, K.: Deep coral: Correlation alignment for deep domain adaptation. In: European conference on computer vision. pp. 443–450. Springer (2016)
  * [33] Tarvainen, A., Valpola, H.: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In: NIPS (2017)
  * [34] Tian, Z., Shen, C., Chen, H., He, T.: Fcos: Fully convolutional one-stage object detection. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 9627–9636 (2019)
  * [35] Tzeng, E., Hoffman, J., Saenko, K., Darrell, T.: Adversarial discriminative domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7167–7176 (2017)
  * [36] Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., Darrell, T.: Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474 (2014)
  * [37] Wang, X., Li, L., Ye, W., Long, M., Wang, J.: Transferable attention for domain adaptation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 5345–5352 (2019)
  * [38] Yan, H., Ding, Y., Li, P., Wang, Q., Xu, Y., Zuo, W.: Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2272–2281 (2017)
  * [39] Yao, T., Ngo, C.W., Zhu, S.: Predicting domain adaptivity: redo or recycle? In: Proceedings of the 20th ACM international conference on Multimedia. pp. 821–824 (2012)