# Adversarial Unsupervised Domain Adaptation with Conditional and Label Shift:
Infer, Align and Iterate

Xiaofeng Liu1, Zhenhua Guo2, Site Li3, Fangxu Xing1, Jane You4, C.-C. Jay
Kuo5,
Georges El Fakhri1, Jonghye Woo1

1Gorden Center for Medical Imaging, Massachusetts General Hospital and Harvard
Medical School,
Boston, MA, USA.
2Alibaba Group, Hangzhou, Zhejiang, China.
3Carnegie Mellon University, Pittsburgh, PA, USA.
4Dept. of Computing, The Hong Kong Polytechnic University, Hong Kong.
5Dept. of ECE, University of Southern California, Los Angeles, CA, USA.
<EMAIL_ADDRESS>
###### Abstract

In this work, we propose an adversarial unsupervised domain adaptation (UDA)
method under inherent conditional and label shifts, in which we aim to align
the distributions w.r.t. both $p(x|y)$ and $p(y)$. Since labels are
inaccessible in a target domain, conventional adversarial UDA methods assume
that $p(y)$ is invariant across domains and rely on aligning $p(x)$ as an
alternative to the $p(x|y)$ alignment. To address this, we provide a thorough
theoretical and empirical analysis of the conventional adversarial UDA methods
under both conditional and label shifts, and propose a novel and practical
alternative optimization scheme for adversarial UDA. Specifically, we infer
the marginal $p(y)$ and align $p(x|y)$ iteratively at the training stage, and
precisely align the posterior $p(y|x)$ at the testing stage. Our experimental
results demonstrate its effectiveness on both classification and segmentation
UDA and partial UDA.

##  1 Introduction

Deep learning methods are highly reliant on the large volume of labeled
training datasets and the independent and identically distributed (i.i.d.)
assumption of the training and testing data [6, 28]. The real world
implementation scenarios, however, can be significantly diverse, and it can be
costly to label datasets in every target environment [10, 41]. To address
this, unsupervised domain adaptation (UDA) can be used to transfer knowledge
learned from a labeled source domain to different unlabeled target domains
[13, 33, 32].

Figure 1: Plots of the UDA performance with different levels of label shift
using $KL(p_{s}(y),p_{t}(y))$ as a metric. Our CLS is more robust to label
shift than conventional adversarial training (TDDA), maximum discrepancy
(MCD), dropout (ADR), and self-training (CBST).

One of the predominant streams in UDA makes use of the adversarial training
[12, 31, 27, 30], which hinges on a discriminator to enforce
$p_{s}(f(x))=p_{t}(f(x))$, where $f(\cdot)$ is a feature extractor. It results
in a domain invariant representation under the $covariate$ $shift$ assumption
[38]. However, the $covariate$ $shift$ assumption is not realistic in lieu of
$conditional$ $shift$ w.r.t. $p(x|y)$, since each class may have its own
appearance shift protocol. For example, a street lamp is sparkly in the night,
while a pedestrian is shrouded in darkness.

Essentially, we would expect the fine-grained class-wise alignment w.r.t.
$p(x|y)$, while $p(y)$ in a target domain is inaccessible in UDA. Assuming
that there is no concept shift (i.e., $p_{s}(y|f(x))=p_{t}(y|f(x))$) and label
shift (i.e., $p_{s}(y)=p_{t}(y))$), and given the Bayes’ theorem,
$p(f(x)|y)=\frac{p(y|f(x))p(f(x))}{p(y)}$, adversarial UDA can align
$p(f(x)|y)$ if $p_{s}(f(x))=p_{t}(f(x))$. The label shift, $i.e.,$ different
class proportions, however, is quite common [26], e.g., a motorbike is more
common in Taipei than in Tokyo.

The adversarial learning based UDA with this unrealistic assumption has been
outperformed by self-training [65], dropout [47], and moment matching methods
[39] in most of the benchmarks.

Either the $conditional$ $shift$ [35] or $label$ $shift$ [5, 23] has a long
research history. However, it is ill-posed to only consider one of them in UDA
without $p_{t}(y)$ [63, 23]. [63, 15, 9] take both the $conditional$ and
$label$ shifts into account from a causal interpretation view. However, its
linearity assumption of $p_{s}(x|y)$ and $p_{t}(x|y)$ does not necessarily
hold. Unfortunately, there are few practical methods available for real-world
applications. Moreover, the theoretical analysis and methodology under the
conditional and label shift assumptions in recently flourished adversarial
learning-based UDA warrant a specific discussion as well.

In this work, we theoretically analyze the inequilibrium of conventional
adversarial UDA methods under the different shifts. The discrepancy between
the marginal distributions $p_{s}(y)$ and $p_{t}(y)$ is measured via the
$KL$-divergence as the semi-supervised learning with the selective bias
problem [61]. The impact of a label shift is empirically illustrated in Fig.
1.

Figure 2: Illustration of our alternative training scheme for conditional and
label shifts: infer $p(y)$, align $p(f(x)|y)$, and iterate.

We believe that the conditional and label shift hypothesis is a better and
minimal assumption in many real-world scenarios. We first assume the target
label distribution $p_{t}(y)$ can be estimated, and further propose to align
the conditional shift between $p_{s}(x|y)$ and $p_{t}(x|y)$. To make the
discriminator aware of the class label $y$, we adopt the joint discriminator
and classifier parameterization and give a theoretical analysis from a domain
adaptation theory perspective. A class-level balancing parameter $w.r.t.$ the
estimated $p_{t}(y)$ can be simply applied to adversarial UDA methods,
following a plug-and-play manner. Since we are quantifying the transferability
of each category instead of each example [4], it is more promising for large-
scale tasks, especially for the segmentation with pixel-wise classification.

Then, we can estimate $p_{t}(y)$, assuming that $p_{s}(x|y)$ and $p_{t}(x|y)$
are aligned using feature-level mean matching, which is incorporated into an
alternative optimization framework. The precise target label distribution and
a decent classifier are expected after sufficient round-based iterations. At
last, we use these two parts to align the posterior $p_{t}(y|x)$, i.e., target
classifier.

Our contributions are summarized as follows:

$\bullet$ We explore both the conditional and label shifts in adversarial UDA,
and give a thorough theoretical and empirical analysis of the conventional UDA
methods under this assumption.

$\bullet$ An alternative optimization scheme is proposed to align the
conditional and label distributions at the training stage, following the
infer, align, and iterate scheme. Finally, we align the posterior $p_{t}(y|x)$
at the testing stage.

$\bullet$ We propose a practical and scalable method to align the conditional
shift with the class-level balancing parameter.

We extensively evaluate our method on both popular UDA classification and
semantic segmentation benchmarks. It can be naturally generalized to partial
UDA.

##  2 Related Work

Unsupervised domain adaptation is aimed at transferring knowledge learned from
a labeled source domain to an unlabeled target domain [23]. The solutions can
be mainly classified into statistic moment matching (e.g., maximum mean
discrepancy (MMD) [35]), domain style transfer [50], self-training [65, 29,
32], and feature-level adversarial learning [13, 18, 19, 34]. As one of the
mainstreams, adversarial learning based UDA proposes to encourage the features
$f(x)$ from different domains indistinguishable. Conditional and label shifts,
however, are ubiquitous in real-world tasks. At present, the moment matching
and self-training usually dominate the top place in classification and
segmentation tasks, respectively.

Domain shifts [23] can be categorized into four classes based on the shift
content, as summarized in Fig. 3. Conventionally, each shift is studied
independently, by assuming that the other shift conditions are invariant
across domains [23].

The conditional shift [35] is more realistic than the covariate shift
assumption [23]. However, without $p_{t}(y)$, estimation of $p_{t}(x|y)$ is,
in general, ill-posed [63].

The label shift [5] (a.k.a. the target shift) occurs when the proportion of
each class is different between source and target domains. [26] propose a test
distribution estimator to detect the label shift. [1] introduce a
regularization approach to correct the label shift. [7] consider the label
shift in an optimal transportation-based UDA task, which is also related to
the class imbalance problem in the MMD framework. [56] propose an
asymmetrically-relaxed alignment in adversarial UDA. These methods, however,
assume that there is no conditional shift.

Moreover, in partial UDA [4], classes in a target domain are a subset of those
in a source domain. It can be regarded as a special case of the label shift in
that some classes in the target domain are zero. Instead of calculating the
transferability of each source example, which can be each pixel in the
segmentation task [4], we directly estimate the probability of each class and
demonstrate its generality and scalability for a novel segmentation partial
UDA task. In a related development, the open set adaptation [40] also has new
target categories and treats them as unknown, which is different from the
above settings.

The concept shift [23] can arise, when classifying a tomato as a vegetable or
fruit in different countries. It is usually not a common problem in popular
object classification or semantic segmentation tasks.

[63, 15, 9] consider the conditional and target shifts in a casual system
($y\rightarrow{x}$) with a somewhat unrealistic linearity assumption.
Unfortunately, they are not aligned with the modern UDA methods and sensitive
to the scale of the dataset. To the best of our knowledge, this is the first
attempt at considering both the conditional and label shifts in adversarial
UDA.

Figure 3: A summary of the possible shifts. The red area indicates the common
shifts in UDA. $p(x)$ can be aligned, if $p(x|y)$ is aligned with the law of
total probability [63].

##  3 Methodology

In UDA, given a source domain $p_{s}(x,y)$ and a target domain $p_{t}(x,y)$, a
labeled set $\mathcal{D_{S}}$ is drawn $i.i.d.$ from $p_{s}(x,y)$, and an
unlabeled set $\mathcal{D_{T}}$ is drawn $i.i.d.$ from the marginal
distribution $p_{t}(x)$. The goal of UDA is to build a good classifier in the
target domain, with the training on $\mathcal{D_{S}}$ and $\mathcal{D_{T}}$.
$\mathcal{Y}=\left\\{1,2,\dots,c\right\\}$ is the set of the class label.
Adversarial UDA [13, 54] is motivated by the following theorem [23]:

Theorem 1 For a hypothesis $h$

$\displaystyle\mathcal{L}_{t}(h)\leq\mathcal{L}_{s}(h)+d[p_{s}(x),p_{t}(x)]+$
(1) $\displaystyle{\rm{min}}[\mathbb{E}_{x\sim
p_{s}}|p_{s}(y|x)-p_{t}(y|x)|,\mathbb{E}_{x\sim
p_{t}}|p_{s}(y|x)-p_{t}(y|x)|],$

where $\mathcal{L}_{s}(h)$ and $\mathcal{L}_{t}(h)$ denote the expected loss
with hypothesis $h$ in the source and target domains, respectively, $d[\cdot]$
is equivalent to the Jensen–Shannon divergence in conventional adversarial UDA
[49].

Based on Theorem 1 and assuming that performing feature transform on $x$ will
not increase the values of $\mathcal{L}_{t}(h)$ and last terms of Eq. 1,
adversarial UDAs [13, 54] apply a feature extractor $f(\cdot)$ onto $x$,
hoping to obtain a feature representation $f(x)\in\mathbb{R}^{K}$ that has a
lower value of $d[p_{s}(f(x)),p_{t}(f(x))]$. To this end, $f(\cdot)$ is
trained to make the feature distributions of the two domains
indistinguishable, by using a domain discriminator
$Dis:\mathbb{R}^{K}\rightarrow(0,1)$, and training a classifier $Cls$ to
correctly classify the source data.
$Cls:\mathbb{R}^{K}\times\mathcal{Y}\rightarrow(0,1)$ outputs the probability
of an extracted feature $f(x)$ being a class $y$ among $c$ categories, i.e.,
$C(f(x),y)=p(y|f(x);Cls)$. The objective of different modules can be

$\displaystyle~{}_{Cls}^{\rm{max}}~{}~{}^{\mathbb{~{}~{}E}}_{x\sim p_{s}}{\rm
log}{{C}}(f(x),y)$ (2)
$\displaystyle~{}_{Dis}^{\rm{max}}~{}~{}^{\mathbb{~{}~{}E}}_{x\sim p_{s}}{\rm
log}(1-Dis(f(x))+^{\mathbb{~{}~{}E}}_{x\sim p_{t}}{\rm log}Dis(f(x))$ (3)
$\displaystyle~{}_{~{}f}^{\rm{max}}~{}^{\mathbb{~{}~{}E}}_{x\sim p_{s}}{\rm
log}{{C}}(f(x),y)+\lambda^{\mathbb{~{}~{}E}}_{x\sim p_{t}}{\rm
log}(1-Dis(f(x)),$ (4)

where $\lambda\in\mathbb{R}^{+}$ balances between the classification and
adversarial loss. Here, we follow the conventional adversarial UDA methods
that make use of the easily implemented three $max$ [51, 49]. Note that
maximizing $~{}^{\mathbb{~{}~{}E}}_{x\sim p_{t}}{\rm log}Dis(f(x))$ for $Dis$
in Eq 3, while maximizing $~{}^{\mathbb{~{}~{}E}}_{x\sim p_{t}}{\rm
log}(1-Dis(f(x)))$ for $f$ in Eq 4 has made the $minmax$ adversarial game. We
omit the subscript $y$ for simplicity, since only the source domain has $y$ in
UDA.

###  3.1 Motivation

We first analyze how the $label$ $shift$ affects the adversarial UDA, when
enforcing the the distribution of feature representation $f(x)$ to be
invariant across domains. We start the analysis under the label shift only
assumption in adversarial learning, followed by taking a step to introduce the
conditional shift.

Theorem 2 Given $p_{s}(x|y)=p_{t}(x|y)$ and $p_{s}(y)\neq p_{t}(y)$, if a
decent $f(\cdot)$ makes $p_{s}(f(x))=p_{t}(f(x))$, then we have
$p_{s}(y|f(x))\neq p_{t}(y|f(x))$, i.e., the concept shift of $f(x)$, and
$\exists~{}p_{s}(y=i|f(x))=p_{s}(y)$, $p_{t}(y=i|f(x))=p_{t}(y=i)$.

From Theorem 2, under the label shift, aligning $p_{s}(f(x))$ with
$p_{t}(f(x))$ results in the posterior w.r.t. $f(x)$ inconsistent in different
domains, i.e., $concept$ $shift$ is induced. We can also deduce that enforcing
$f(x)$ to be domain invariant may mix the feature of all classes in the
feature space, since $p_{s}(y=i|f(x))$ or $p_{t}(y=i|f(x))$ will be a constant
for any $x$. Let us make a specific example that all samples $x$ are mapped
onto a consistent point $f_{0}$ in the feature manifold of $f(x)$; we have
$p_{s}(f(x)=f_{0})=p_{t}(f(x)=f_{0})=1$. As a result, $f(x)$ is domain
invariant as expected, but the classifier will completely lose the
discriminative ability. Actually, strictly enforcing $p_{s}(f(x))=p_{t}(f(x))$
in this case will make the classifier less disciminative, i.e., increasing the
target loss $\mathcal{L}_{t}(h)$ or its upper bound $\mathcal{L}_{s}(h)$ in
Theorem 1 [23].

Furthermore, under the $joint$ $conditional$ $and$ $label$ $shift$ assumption,
there is also a conflict between minimal domain discrepancy and superior
classification performance. Following the law of total probability, the
optimization objective of adversarial UDA to align $p_{s}(f(x))$ with
$p_{t}(f(x))$ can be reformulated as

$\displaystyle\sum_{i=1}^{c}{p_{s}(f(x)|y=i)p_{s}(y=i)}=\sum_{i=1}^{c}{p_{t}(f(x)|y=i)}p_{t}(y=i).$
(5)

Suppose that the samples of class $i$, $x\in\mathcal{X}_{i}$, are completely
distinguishable from the samples in the other classes in $f(x)$, with a decent
$f(\cdot)$, $i.e.,$

$\displaystyle p_{s}(f(x\in\mathcal{X}_{i})|y=i)>0\Leftrightarrow
p_{s}(f(x\in\mathcal{X}_{i})|y\neq i)=0$ (6) $\displaystyle
p_{t}(f(x\in\mathcal{X}_{i})|y=i)>0\Leftrightarrow
p_{t}(f(x\in\mathcal{X}_{i})|y\neq i)=0.$

Therefore, advUDA is enforcing $p_{s}(f(x\in\mathcal{X}_{i})|y=i)$
$p_{s}(y=i)$ $={p_{t}(f(x\in\mathcal{X}_{i})|y=i)}p_{t}(y=i)$. Taking the
integral of $x$ over $\mathcal{X}_{i}$ for both sides of the above equation,
we have $p_{s}(y=i)=p_{t}(y=i)$. This deduction contradicts with the label
shift setting $p_{s}(y)\neq p_{t}(y)$.

Based on the above analysis, we have the conclusion that the label and
conditional shifts are non-negligible in adversarial UDA. Since $p_{t}(y)$ is
not available in UDA, we propose to align the class-dependent conditional
distribution and estimate the label distribution, following a turn-based
alternative strategy in each iteration, as shown in Fig. 4.

Figure 4: Illustration of our alternative training-based CLS framework, which
consists of feature encoder $f$, joint $Dis$ and $Cls$, and $p_{t}(y)$
estimator. The posterior alignment is only applied at the testing stage. The
overall objectives are shown in Eqs. (10-13), which are optimized iteratively.
CE indicates cross-entropy.

###  3.2 Conditional Shift Alignment

To incorporate the knowledge of $p_{t}(y)$ to align $p_{s}(x|y)$ with
$p_{t}(x|y)$, we introduce a class-wise balancing parameter
$\gamma\in\mathbb{R}^{c}$. We expect that
$\gamma_{i}{p_{s}(y=i)}={p_{t}(y=i)}$ and $\gamma^{*}$ denote the vector that
makes this equation hold. Therefore, we propose to multiply $\gamma$ to the
source domain term of the optimization objective in Eq. (5), $i.e.,$
$\sum_{i=1}^{c}{p_{s}(f(x)|y=i)p_{s}(y=i)\gamma_{i}}=\sum_{i=1}^{c}{p_{t}(f(x)|y=i)}p_{t}(y=i)$.
Since $\gamma_{i}=\frac{{p_{t}(y=i)}}{{p_{s}(y=i)}}$, it can be written as
$\sum_{i=1}^{c}{p_{s}(f(x)|y=i){\cancel{p_{s}(y=i)}}\frac{{p_{t}(y=i)}}{\cancel{p_{s}(y=i)}}}=\sum_{i=1}^{c}{p_{t}(f(x)|y=i)}p_{t}(y=i)$
and the estimated $p_{t}(y=i)$ is a constant in each iteration. Theoretically,
with a precisely estimated $\gamma$, we can align the conditional distribution
using adversarial UDA.

Besides, from an implementation perspective, it is important to make the
discriminator be aware of classification. As discussed in [48], the three-
player ($f,Cls,Dis$) game should fail to consider the relationship between the
target samples and the task-specific decision boundary, when aligning
distributions. Here, we simply adopt the jointly parameterized classifier and
discriminator [49, 51] to incorporate the classification into the domain
alignment. Next, we align $p_{s}(x|y)$ and $p_{t}(x|y)$ with the estimated
$p_{t}(y)$, even though the up-to-date class-aware domain adaptation method
can be added, following a plug-and-play fashion.

We configure $c+1$ output units to represent the $c$ class and target domain,
respectively. The score function ${\overline{C}}$ (joint $cls$ and $dis$) is
defined on $\mathbb{R}^{K}\times\\{1,\cdots,c+1\\}$. Since the sum of the
probability that $f(x)$ belongs to one of the $c$-class is not 1, we need to
reparamiterize the conditional score:

$\displaystyle{\overline{C}}(f(x),y|\mathcal{Y})=\frac{{\overline{C}}(f(x),y)}{1-{\overline{C}}(f(x),c+1)}$
$\displaystyle\forall y\leq c;~{}~{}~{}{\overline{C}}(f(x),c+1|\mathcal{Y})=0$
(7)

Its optimization objective is given by

$\displaystyle~{}_{Dis\&Cls}^{\rm{~{}~{}max}}~{}\\{~{}$
${}^{\mathbb{~{}~{}E}}_{x\sim p_{s}}{\rm
log}{\overline{C}}(f(x),y|\mathcal{Y})$
$\displaystyle+^{\mathbb{~{}~{}E}}_{x\sim p_{s}}{\rm
log}(1-{\overline{C}}(f(x),c+1))$ $\displaystyle+^{\mathbb{~{}~{}E}}_{x\sim
p_{t}}{\rm log}({\overline{C}}(f(x),c+1))\\}$ (8)
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}_{~{}~{}f}^{\rm{max}}~{}\\{~{}$
${}^{\mathbb{~{}~{}E}}_{x\sim p_{s}}{\rm
log}{\overline{C}}(f(x),y|\mathcal{Y})$
$\displaystyle+\lambda^{\mathbb{~{}~{}E}}_{x\sim p_{t}}{\rm
log}(1-{\overline{C}}(f(x),c+1))\\}~{}~{}~{}~{}~{}~{}$ (9)

The empirical effectiveness of the joint parameterization of $Cls$ and $Dis$
in semi-supervised learning has been shown in [49]. [51] give an intuitive
explanation that all target examples yield the gradient of the same direction
in the conventional three-player game. In this paper, we give a theoretical
analysis from the domain adaptation theory perspective.

Proposition 1 The joint parameterization can lead to a better approximation of
$\mathcal{L}_{t}(h)$ by $\mathcal{L}_{s}(h)$ in Theorem 1.

This means that the performance in the target domain can be less affected by
the domain shift. Please note that using only the joint parameterization
cannot guarantee the conditional alignment, but can provide a powerful
backbone for our conditional alignment scheme. We also give the detailed
ablation study in our experiments.

In general, the expectation function in Eqs. (8-9) has the property
$~{}^{\mathbb{~{}~{}E}}_{x\sim p_{s}(x)}[\cdot]\equiv$
$~{}^{\mathbb{~{}~{}~{}~{}~{}~{}~{}~{}E}}_{x\sim
p_{s}(x|y)p_{s}(y)}[\cdot]\equiv\sum_{i=1}^{c}p_{s}(y=i)~{}^{\mathbb{~{}~{}~{}~{}~{}~{}~{}~{}E}}_{x\sim(p_{s}(x|y=i))}[\cdot]$,
where $[\cdot]$ is a deterministic function of $x$ $w.r.t.$ $f$. Therefore, to
take the target label distribution into account, we can make Eqs. (8-9) be
aware of the estimated $\gamma$, by rewriting them as

${}_{D\&C}^{\rm{~{}max}}\\{$
$\displaystyle\sum_{i=1}^{c}\gamma_{i}p_{s}(y=i)^{\mathbb{~{}~{}E}}_{x\sim
p_{s}(x|y=i)}{\rm
log}{\overline{C}}(f(x),y|\mathcal{Y})~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}$
$\displaystyle+^{\mathbb{~{}~{}E}}_{x\sim p_{s}}{\rm
log}(1-{\overline{C}}(f(x),c+1))$ $\displaystyle+^{\mathbb{~{}~{}E}}_{x\sim
p_{t}}{\rm log}({\overline{C}}(f(x),c+1))\\}$ (10)
$\displaystyle~{}~{}~{}~{}_{~{}~{}f}^{\rm{max}}~{}\\{$
$\displaystyle\sum_{i=1}^{c}\gamma_{i}p_{s}(y=i)~{}^{\mathbb{~{}~{}E}}_{x\sim
p_{s}(x|y=i)}{\rm
log}{\overline{C}}(f(x),y|\mathcal{Y})~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}$
$\displaystyle+\lambda^{\mathbb{~{}~{}E}}_{x\sim p_{t}}{\rm
log}(1-{\overline{C}}(f(x),c+1))\\}.$ (11)

When $\gamma_{i}=1,\forall i\in\mathcal{Y}$, this alignment will degrade to
conventional adversarial UDA. $\gamma_{i}$ can also be regarded as a class-
level weight for each class in the source domain. This is essentially
different from the sample-level weight in [4, 62]. Actually, we multiply
$\gamma_{i}$ to the $p_{s}(y=i)$ term, $i.e.,$ revising the source label
distribution to the estimated target label distribution
$\frac{{{p_{t}}(y=i)}}{\cancel{{p_{s}}(y=i)}}{\cancel{{p_{s}}(y=i)}}$.

Table 1: Experimental results for the VisDA17-val setting. We use ResNet101 as
our backbone except for [43, 50] and CLS:Res152.

Method | Aero | Bike | Bus | Car | Horse | Knife | Motor | Person | Plant | Skateboard | Train | Truck | Mean
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Source-Res101 [14] | 55.1 | 53.3 | 61.9 | 59.1 | 80.6 | 17.9 | 79.7 | 31.2 | 81.0 | 26.5 | 73.5 | 8.5 | 52.4
DANN (Baseline) [13] | 81.9 | 77.7 | 82.8 | 44.3 | 81.2 | 29.5 | 65.1 | 28.6 | 51.9 | 54.6 | 82.8 | 7.8 | 57.4
MCD [48] | 87.0 | 60.9 | 83.7 | 64.0 | 88.9 | 79.6 | 84.7 | 76.9 | 88.6 | 40.3 | 83.0 | 25.8 | 71.9
ADR [47] | 87.8 | 79.5 | 83.7 | 65.3 | 92.3 | 61.8 | 88.9 | 73.2 | 87.8 | 60.0 | 85.5 | 32.3 | 74.8
DEV [59] | 81.83 | 53.48 | 82.95 | 71.62 | 89.16 | 72.03 | 89.36 | 75.73 | 97.02 | 55.48 | 71.19 | 29.17 | 72.42
TPN [39] | 93.7 | 85.1 | 69.2 | 81.6 | 93.5 | 61.9 | 89.3 | 81.4 | 93.5 | 81.6 | 84.5 | 49.9 | 80.4
CRST [65] | 89.2 | 79.6 | 64.2 | 57.8 | 87.8 | 79.6 | 85.6 | 75.9 | 86.5 | 85.1 | 77.7 | 68.5 | 78.1
TDDA [14] | 88.2 | 78.5 | 79.7 | 71.1 | 90.0 | 81.6 | 84.9 | 72.3 | 92.0 | 52.6 | 82.9 | 18.4 | 74.03
PANDA [22] | 90.9 | 50.5 | 72.3 | 82.7 | 88.3 | 88.3 | 90.3 | 79.8 | 89.7 | 79.2 | 88.1 | 39.4 | 78.3
DMRL [55] | - | - | - | - | - | - | - | - | - | - | - | - | 75.5
CLS w/o Joint Dis+Cls | 91.8$\pm$1.2 | 84.4$\pm$1.9 | 73.3$\pm$1.7 | 71.5$\pm$1.6 | 87.2$\pm$1.0 | 80.8$\pm$1.4 | 85.6$\pm$1.8 | 76.3$\pm$1.5 | 85.6$\pm$1.8 | 88.5$\pm$1.3 | 85.7$\pm$1.1 | 71.0$\pm$1.4 | 80.8$\pm$0.6
CLS w/o fade-in $p_{t}(y)$ | 92.0$\pm$1.4 | 84.2$\pm$1.2 | 72.8$\pm$1.6 | 72.5$\pm$1.2 | 87.6$\pm$1.5 | 82.7$\pm$2.0 | 86.4$\pm$1.7 | 77.0$\pm$1.3 | 86.3$\pm$1.7 | 89.6$\pm$1.5 | 85.5$\pm$1.7 | 72.1$\pm$1.4 | 81.3$\pm$0.5
CLS | 92.6$\pm$1.5 | 84.5$\pm$1.4 | 73.7$\pm$1.5 | 72.7$\pm$1.1 | 88.5$\pm$1.4 | 83.3$\pm$1.2 | 89.1$\pm$1.4 | 77.6$\pm$1.5 | 89.5$\pm$1.5 | 89.2$\pm$1.3 | 85.8$\pm$1.2 | 72.7$\pm$1.3 | 81.6$\pm$0.4
CLS+TDDA w/o Joint Dis&Cls | 92.3$\pm$1.5 | 84.6$\pm$1.2 | 74.4$\pm$1.5 | 73.7$\pm$1.8 | 89.6$\pm$1.8 | 82.5$\pm$1.9 | 88.4$\pm$1.5 | 82.8$\pm$1.5 | 89.1$\pm$1.6 | 91.9$\pm$1.5 | 85.6$\pm$1.6 | 73.5$\pm$1.3 | 82.2$\pm$0.5
CLS+TDDA | 94.0$\pm$1.6 | 85.4$\pm$1.6 | 75.4$\pm$1.0 | 74.5$\pm$1.4 | 92.2$\pm$1.5 | 83.4$\pm$1.5 | 89.8$\pm$1.6 | 82.8$\pm$1.2 | 90.8$\pm$1.2 | 91.8$\pm$1.4 | 87.5$\pm$1.2 | 74.1$\pm$0.9 | 82.9$\pm$0.5
SimNet:Res152 [43] | 94.3 | 82.3 | 73.5 | 47.2 | 87.9 | 49.2 | 75.1 | 79.7 | 85.3 | 68.5 | 81.1 | 50.3 | 72.9
GTA:Res152 [50] | - | - | - | - | - | - | - | - | - | - | - | - | 77.1
CLS:Res152 | 94.2$\pm$1.3 | 86.8$\pm$1.2 | 74.7$\pm$1.0 | 75.3$\pm$1.5 | 90.5$\pm$0.6 | 88.6$\pm$1.6 | 89.2$\pm$1.6 | 83.2$\pm$1.6 | 91.9$\pm$1.3 | 92.6$\pm$1.8 | 86.1$\pm$1.4 | 80.9$\pm$1.4 | 83.8$\pm$0.5

###  3.3 Target Label Distribution Estimation

Assuming that the conditional distribution is aligned ($i.e.,$
$p_{s}(f(x)|y)=p_{t}(f(x)|y)$), we can estimate the target label distribution
$p_{t}(y)$ using the marginal matching [63]. Instead of working on raw data,
in this work, we carry out our analyses in the feature space. Note that
$p_{s}(y)$ can be obtained, by counting each labeled sample.

Targeting this, the mean matching [16] is a simple yet efficient method. The
target label proportion $p_{t}(y)$ can be estimated, by minimizing the loss
function

$\displaystyle\mathcal{L}_{M}(p_{t}(y))=||M_{s}p_{t}(y)-\mu_{t}||_{2}^{2},$
(12)

where $M_{s}$ is the vector of
$[\mu_{s}(f(x)|y=1),\mu_{s}(f(x)|y=2),\cdots,\mu_{s}(f(x)|y=c))]$, i.e., the
empirical sample means from the source domain, and $\mu_{t}$ is the encoded
feature mean of the target. Note that the above loss is equivalent to a
standard linearly constrained quadratic problem, resulting in the estimated
target label proportions $p_{t}(y)$. Typically, this can be solved by gradient
descent in each minibatch.

Suppose that the conditional shift can be aligned, then the target label
distribution estimation scheme is asymptotically consistent, which can be
expressed below.

Theorem 3 Assuming $p_{s}(f(x)|y)=p_{t}(f(x)|y)$, the variance in the feature
space is not infinite, and all of the label proportions are not zero. As the
number of training and testing samples becomes infinity, the estimated
${p}_{t}(y)$ is asymptotically consistent for real ${p}_{t}(y)$, if
$M_{s}^{\rm T}M_{s}$ is invertible.

Here the superscript ${\rm T}$ represents transpose. While this theorem only
considers a single source domain, it can be easily extended to multiple source
domains.

We note that the L2 loss used in $\mathcal{L}_{M}(p_{t}(y))$ can also be
replaced by the other distribution loss, $e.g.,f$-divergence, Wasserstein
loss, or MMD to form the distribution matching. In some cases, the model can
be benefited, by matching more than the first moment.

Considering that the segmentation task is to perform pixel-wise
classification, the label distribution is essentially the proportion of the
class-wise pixel numbers in a dataset.

Table 2: Experimental results for GTA5 to Cityscapes.

Method | Base Net | Road | SW | Build | Wall | Fence | Pole | TL | TS | Veg. | Terrain | Sky | PR | Rider | Car | Truck | Bus | Train | Motor | Bike | mIoU
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Source | DRN26 | 42.7 | 26.3 | 51.7 | 5.5 | 6.8 | 13.8 | 23.6 | 6.9 | 75.5 | 11.5 | 36.8 | 49.3 | 0.9 | 46.7 | 3.4 | 5.0 | 0.0 | 5.0 | 1.4 | 21.7
CyCADA [21] |  | 79.1 | 33.1 | 77.9 | 23.4 | 17.3 | 32.1 | 33.3 | 31.8 | 81.5 | 26.7 | 69.0 | 62.8 | 14.7 | 74.5 | 20.9 | 25.6 | 6.9 | 18.8 | 20.4 | 39.5
Source | DRN105 | 36.4 | 14.2 | 67.4 | 16.4 | 12.0 | 20.1 | 8.7 | 0.7 | 69.8 | 13.3 | 56.9 | 37.0 | 0.4 | 53.6 | 10.6 | 3.2 | 0.2 | 0.9 | 0.0 | 22.2
MCD [48] |  | 90.3 | 31.0 | 78.5 | 19.7 | 17.3 | 28.6 | 30.9 | 16.1 | 83.7 | 30.0 | 69.1 | 58.5 | 19.6 | 81.5 | 23.8 | 30.0 | 5.7 | 25.7 | 14.3 | 39.7
Source | PSPNet | 69.9 | 22.3 | 75.6 | 15.8 | 20.1 | 18.8 | 28.2 | 17.1 | 75.6 | 8.00 | 73.5 | 55.0 | 2.9 | 66.9 | 34.4 | 30.8 | 0.0 | 18.4 | 0.0 | 33.3
DCAN [57] |  | 85.0 | 30.8 | 81.3 | 25.8 | 21.2 | 22.2 | 25.4 | 26.6 | 83.4 | 36.7 | 76.2 | 58.9 | 24.9 | 80.7 | 29.5 | 42.9 | 2.50 | 26.9 | 11.6 | 41.7
Source | ResNet-101 | 75.8 | 16.8 | 77.2 | 12.5 | 21.0 | 25.5 | 30.1 | 20.1 | 81.3 | 24.6 | 70.3 | 53.8 | 26.4 | 49.9 | 17.2 | 25.9 | 6.5 | 25.3 | 36.0 | 36.6
DPR [53] |  | 92.3 | 51.9 | 82.1 | 29.2 | 25.1 | 24.5 | 33.8 | 33.0 | 82.4 | 32.8 | 82.2 | 58.6 | 27.2 | 84.3 | 33.4 | 46.3 | 2.2 | 29.5 | 32.3 | 46.5
Source | ResNet-101 | 73.8 | 16.0 | 66.3 | 12.8 | 22.3 | 29.0 | 30.3 | 10.2 | 77.7 | 19.0 | 50.8 | 55.2 | 20.4 | 73.6 | 28.3 | 25.6 | 0.1 | 27.5 | 12.1 | 34.2
PyCDA [24] |  | 90.5 | 36.3 | 84.4 | 32.4 | 28.7 | 34.6 | 36.4 | 31.5 | 86.8 | 37.9 | 78.5 | 62.3 | 21.5 | 85.6 | 27.9 | 34.8 | 18.0 | 22.9 | 49.3 | 47.4
Source | ResNet101 | 71.3 | 19.2 | 69.1 | 18.4 | 10.0 | 35.7 | 27.3 | 6.8 | 79.6 | 24.8 | 72.1 | 57.6 | 19.5 | 55.5 | 15.5 | 15.1 | 11.7 | 21.1 | 12.0 | 33.8
CRST [65] |  | 89.0 | 51.2 | 79.4 | 31.7 | 19.1 | 38.5 | 34.1 | 20.4 | 84.7 | 35.4 | 76.8 | 61.3 | 30.2 | 80.7 | 27.4 | 39.4 | 10.2 | 32.2 | 43.3 | 46.6
Source | ResNet-101 | 75.8 | 16.8 | 77.2 | 12.5 | 21.0 | 25.5 | 30.1 | 20.1 | 81.3 | 24.6 | 70.3 | 53.8 | 26.4 | 49.9 | 17.2 | 25.9 | 6.5 | 25.3 | 36.0 | 36.6
APODA [58] |  | 85.6 | 32.8 | 79.0 | 29.5 | 25.5 | 26.8 | 34.6 | 19.9 | 83.7 | 40.6 | 77.9 | 59.2 | 28.3 | 84.6 | 34.6 | 49.2 | 8.0 | 32.6 | 39.6 | 45.9
CLS+[58] |  | 94.7 | 59.2 | 82.6 | 31.1 | 26.7 | 42.3 | 38.5 | 29.2 | 86.3 | 39.0 | 78.9 | 67.6 | 36.1 | 85.8 | 28.3 | 45.6 | 21.8 | 36.8 | 46.4 | 51.2
CLS+[58] w/o Joint Dis+Cls |  | 94.1 | 54.9 | 81.6 | 32.6 | 22.0 | 41.4 | 36.0 | 20.8 | 87.6 | 36.9 | 77.4 | 64.3 | 32.4 | 84.0 | 23.5 | 39.6 | 25.2 | 33.2 | 40.4 | 50.8
Source | ResNet-101 | 75.8 | 16.8 | 77.2 | 12.5 | 21.0 | 25.5 | 30.1 | 20.1 | 81.3 | 24.6 | 70.3 | 53.8 | 26.4 | 49.9 | 17.2 | 25.9 | 6.5 | 25.3 | 36.0 | 36.6
IAST-MST [37] |  | 94.1 | 58.8 | 85.4 | 39.7 | 29.2 | 25.1 | 43.1 | 34.2 | 84.8 | 34.6 | 88.7 | 62.7 | 30.3 | 87.6 | 42.3 | 50.3 | 24.7 | 35.2 | 40.2 | 52.2
CLS+[37] |  | 94.7 | 60.1 | 85.6 | 39.5 | 24.4 | 44.1 | 39.5 | 20.6 | 88.7 | 38.7 | 80.3 | 67.2 | 35.1 | 86.5 | 37.0 | 45.4 | 39.0 | 37.9 | 46.2 | 53.0
CLS+[37] w/o Joint Dis+Cls |  | 94.5 | 58.6 | 84.9 | 32.2 | 26.5 | 42.4 | 39.3 | 28.4 | 88.6 | 42.3 | 79.5 | 67.4 | 36.5 | 83.3 | 34.8 | 43.2 | 24.6 | 36.5 | 48.3 | 52.5

###  3.4 Alternative Optimization with Fade-in $p_{t}(y)$

In summary, we have three to-be learned players in the game at our training
stage, i.e., feature extractor $f(\cdot)$, joint $Dis\&Cls$, and a $p_{t}(y)$
estimator. In each iteration, we update them following an alternative
optimization scheme:

[Infer]: Fix $f(\cdot)$ and update the target label distribution estimator
with $\frac{\partial\mathcal{L}_{M}(p_{t}(y))}{\partial p_{t}(y)}$.

[Align]: Fix $p_{t}(y)$ estimator first and update the feature extractor and
joint $Dis\&Cls$ following the GAN’s protocol according to Eq. (10) and Eq.
(11), respectively.

Considering that it is difficult to precisely estimate $p_{t}(y)$ at the start
of the training, we simply initialize it as uniform distribution, i.e.,
$p_{t}(y=i)=\frac{1}{c}$ and construct the target label distribution with
$\frac{1}{1+\alpha}\left\\{p_{t}(y)+\alpha p_{s}(y)\right\\}$ in each round.
$\alpha$ is decreased from 1 to 0 gradually at the training stage, i.e.,
$\alpha=\frac{1}{1+N}$ for epoch number $N\leq$5 and $\alpha=0$ for $N>$5.

###  3.5 Posterior Alignment

Finally, we obtain the target domain classifier $p_{t}(y=i|f(x))$ in testing.
Suppose that we have successfully aligned the conditional shift and precisely
estimated the target label distribution at the training stage, following [5],
the posterior in the target domain can be
$p_{t}(y=i|f(x))=\frac{\gamma_{i}^{*}p_{s}(y=i|f(x))}{\sum_{j=1}^{c}p_{s}(y=j|f(x))}$.
Since we cannot practically know the exact $\gamma*$, we choose the estimated
$p_{t}(y)$ to calculate $\gamma$ in each iteration, and approximate the
posterior by

$\displaystyle
p_{t}(y=i|f(x))\leftarrow\frac{\gamma_{i}p_{s}(y=i|f(x))}{|\gamma|_{1}\sum_{j=1}^{c}p_{s}(y=j|f(x))}.$
(13)

##  4 Experimental Results

We provide comprehensive evaluations of our framework under the conditional
and label shift (CLS) on image classification and semantic segmentation
(partial) UDA tasks.

Table 3: Experimental results for partial unsupervised domain adaptation on
Office-31 dataset (ResNet-50).

Method | A31$\rightarrow$W10 | D31$\rightarrow$W10 | W31$\rightarrow$D10 | A31$\rightarrow$D10 | D31$\rightarrow$A10 | W31$\rightarrow$A10 | Mean
---|---|---|---|---|---|---|---
ResNet-50 [20] | 75.59$\pm$1.09 | 96.27$\pm$0.85 | 98.09$\pm$0.74 | 83.44$\pm$1.12 | 83.92$\pm$0.95 | 84.97$\pm$0.86 | 87.05$\pm$0.94
DAN [36] | 59.32$\pm$0.49 | 73.90$\pm$0.38 | 90.45$\pm$0.36 | 61.78$\pm$0.56 | 74.95$\pm$0.67 | 67.64$\pm$0.29 | 71.34$\pm$0.46
DANN [13] | 73.56$\pm$0.15 | 96.27$\pm$0.26 | 98.73$\pm$0.20 | 81.53$\pm$0.23 | 82.78$\pm$0.18 | 86.12$\pm$0.15 | 86.50$\pm$0.20
ADDA [54] | 75.67$\pm$0.17 | 95.38$\pm$0.23 | 99.85$\pm$0.12 | 83.41$\pm$0.17 | 83.62$\pm$0.14 | 84.25$\pm$0.13 | 87.03$\pm$0.16
IWAN [62] | 89.15$\pm$0.37 | 99.32$\pm$0.32 | 99.36$\pm$0.24 | 90.45$\pm$0.36 | 95.62$\pm$0.29 | 94.26$\pm$0.25 | 94.69$\pm$0.31
SAN [3] | 93.90$\pm$0.45 | 99.32$\pm$0.52 | 99.36$\pm$0.12 | 94.27$\pm$0.28 | 94.15$\pm$0.36 | 88.73$\pm$0.44 | 94.96$\pm$0.36
PADA [2] | 86.54$\pm$0.31 | 99.32$\pm$0.45 | 100.00$\pm$0.00 | 82.17$\pm$0.37 | 92.69$\pm$0.29 | 95.41$\pm$0.33 | 92.69$\pm$0.29
ETN [4] | 94.52$\pm$0.20 | 100.00$\pm$0.00 | 100.00$\pm$0.00 | 95.03$\pm$0.22 | 96.21$\pm$0.27 | 94.64$\pm$0.24 | 96.73$\pm$0.16
BAUS [25] | 98.98$\pm$0.28 | 100.0$\pm$0.00 | 98.73$\pm$0.00 | 99.36$\pm$0.00 | 94.82$\pm$0.05 | 94.99$\pm$0.08 | 97.81
CLS | 99.64$\pm$0.28 | 100.00$\pm$0.00 | 100.00$\pm$0.00 | 97.26$\pm$0.30 | 97.94$\pm$0.23 | 98.32$\pm$0.34 | 98.24$\pm$0.15
CLS w/o Joint Dis+Cls | 98.91$\pm$0.35 | 99.62$\pm$0.43 | 100.00$\pm$0.00 | 96.36$\pm$0.27 | 98.05$\pm$0.25 | 97.75$\pm$0.42 | 97.94$\pm$0.32

###  4.1 UDA for Image Classification

We first consider the challenging VisDA17 benchmark [42]. It is a 12-class UDA
classification problem. We follow the evaluation protocols as in [50, 47],
where a source domain incorporates $152,409$ synthetic images and a target
domain has $55,400$ real-world images.

For fair comparisons, we adopt the standard ResNet101 [20] as our backbones
for VisDA17. We use ImageNet [11] for pre-training, and fine-tune the network
in the source domain with SGD and set $lr=1\times 10^{-3}$. The batch size is
set to 32 consistently. We simply set $\lambda=1$ in all the experiments.

We show the results on VisDA17 in Table 1 in terms of the per-class accuracy
and mean accuracy. To report the standard deviation of each method, we
independently run five times. The CLS w/o Joint Dis+Cls, and CLS w/o fade-in
$p_{t}(y)$ denote the CLS with separate discriminator and classifier and
without fade-in $p_{t}(y)$, respectively.

DANN [13] is a pioneer of vanilla adversarial UDA, and its performance on
VisDA is reported in [14]. Benefited by the conditional and label alignments,
CLS outperforms DANN by a large margin and the recent adversarial UDA method,
TDDA, significantly, while the joint parameterization further boosts its
performance. Although the full training of CLS takes 1.5 times more training
time than DANN, it only requires 2$\times$ less training time to obtain
comparable results than DANN. Moreover, CLS has the same inference speed as
conventional adversarial UDA methods at its testing stage. Note that the CDAN
[35] method is ill-posed as it only considers the conditional shift in UDA
(without $p_{t}(y)$), and a direct comparison with that can be helpful.
However, its relatively old backbone (ResNet50) is not adopted by recent
state-of-the-art approaches (SOTA). Our CLS with ResNet50 can also achieve a
mean accuracy of 76.2% on VisDA17, which is a significant improvement over
CDAN (70%). Besides, Our CLS outperforms DEV on VisDA17 (all with ResNet101),
and DEV outperforms CDAN on Office31 (all with ResNet50).

Figure 5: The KL-divergence between the predicted $p_{t}(y)$ and the ground
truth $p_{t}(y)$ at the testing stage. Left: VisDA17 task, right: GTA5 to
Cityscapes task. The smaller the value is the more accurate the estimation is.

Moreover, our CLS can be orthogonal with other advanced adversarial UDA
methods. For example, the regularization losses used in TDDA [14] can be
simply add-on and further improve the performance. To the best of our
knowledge, all CLSs, including CLS w/o Joint Dis+Cls, and CLS w/o fade-in
$p_{t}(y)$ achieve the STOA of adversarial UDA in VisDA 17 with ResNet 101 as
the backbone.

More appealingly, CLS achieves on par or better performance than the recent
adversarial training [55, 14], self-training [65], dropout [47], and moment
matching methods [39, 22], demonstrating the potential of adversarial learning
in UDA. The more powerful backbones have also been applied and shown better
results [43, 50]. The CLS with the ResNet152 backbone outperforms both CLS
with the ResNet 101 backbone and [43, 50] significantly.

###  4.2 UDA for Semantic Segmentation

Then, we investigate a more challenging segmentation adaptation case that is
from synthetic to real images. We select the GTA5 [44] to Cityscapes task,
which has 19 shared segmentation labels. There are more than 24,000 labeled
game engine rendered images in the GTA5 dataset. According to the evaluation
protocol in previous works [58], the whole GTA5 dataset is used as the labeled
source domain, and the training set of Cityscape with 2,975 images is used as
the unlabeled target domain training data. During testing, we also report the
performance in the testing set of Cityscapes with 500 images.

For fair comparisons with the other methods, we adopt the ResNet-101 as our
backbone network for the GTA5 to Cityscapes task. Note that PSPNET [64] and
Wide ResNet38 are the stronger basenet than ResNet101. We compare CLSs with
the other methods in Table 2. Based on the adversarial training scheme of
[58], CLS outperforms [58], by more than 4% w.r.t. the mean IoUs.

When compared with other state-of-the-art adversarial UDAs, CLS methods lead
to a new state-of-the-art performance, even compared with a generative pixel-
level domain adaptation method or adversarial and self-training combination
methods, which are the relatively complex algorithm in terms of both
architecture and objectives. We note that the additional self-training UDA can
also be applied to our CLS, following the combination scheme in [37], yielding
53.0 mIoU. This clearly demonstrates the generality of our CLS. Introducing
our CLS to AdaptSegNet can significantly improve the segmentation performance.
To investigate the target label estimation performance, we plot the KL-
divergence between the predicted $p_{t}(y)$ and the ground truth $p_{t}(y)$ at
the testing stage of both classification and segmentation tasks as in Fig. 5.

Consistently with the classification, our CLS performs better, or on par with
self-training-based methods [65, 24], indicating the adversarial learning can
still be a powerful methodology in UDA. Of note, the moment matching is
usually not well scalable to segmentation.

Table 4: Experimental results for partial UDA on Cityscapes (merged 14-class)
$\rightarrow$ NTHU (10-class). *Our reproduced results.

Method | Base Net | Road | SW | Build | TL | TS | Veg. | Sky | PR | Car | Bus | mIoU
---|---|---|---|---|---|---|---|---|---|---|---|---
Source | Dilation-Frontend [60] | 80.3 | 24.8 | 69.3 | 8.2 | 5.5 | 70.4 | 76.5 | 40.2 | 55.4 | 18.7 | 38.8
GCAA [52]* |  | 81.0 | 34.2 | 69.6 | 11.8 | 13.4 | 74.5 | 63.1 | 42.4 | 60.1 | 20.5 | 41.5
CBST [66]* |  | 82.6 | 35.1 | 77.4 | 6.6 | 28.5 | 80.2 | 75.7 | 49.6 | 69.8 | 4.8 | 45.4
CLS |  | 84.6 | 38.7 | 75.4 | 19.3 | 32.6 | 77.4 | 69.6 | 49.3 | 70.2 | 30.5 | 49.3
Source | ResNet-101 | 83.2 | 36.0 | 70.6 | 13.5 | 12.8 | 75.3 | 61.2 | 43.5 | 63.8 | 20.4 | 42.0
GCAA [52]* |  | 81.6 | 27.1 | 73.5 | 16.4 | 23.8 | 80.2 | 87.5 | 52.3 | 68.1 | 19.3 | 47.6
CBST [66]* |  | 84.5 | 34.2 | 79.6 | 10.5 | 29.1 | 82.2 | 79.4 | 54.7 | 74.0 | 5.7 | 48.2
CLS |  | 88.1 | 42.3 | 83.6 | 26.7 | 37.5 | 87.3 | 64.6 | 45.4 | 79.2 | 24.5 | 50.7

Table 5: Experimental results for partial unsupervised domain adaptation on
ImageNet-Caltech dataset (ResNet-50).

Method | Img1000$\rightarrow$Cal84 | Cal256$\rightarrow$Img84 | Avg
---|---|---|---
ResNet-50 [20] | 69.69$\pm$0.78 | 71.29$\pm$0.74 | 70.49$\pm$0.76
DAN [36] | 71.30$\pm$0.46 | 60.13$\pm$0.50 | 65.72$\pm$0.48
DANN [13] | 70.80$\pm$0.66 | 67.71$\pm$0.76 | 69.23$\pm$0.71
ADDA [54] | 71.82$\pm$0.45 | 69.32$\pm$0.41 | 70.57$\pm$0.43
IWAN [62] | 78.06$\pm$0.40 | 73.33$\pm$0.46 | 75.70$\pm$0.43
SAN [3] | 77.75$\pm$0.36 | 75.26$\pm$0.42 | 76.51$\pm$0.39
PADA [2] | 75.03$\pm$0.36 | 70.48$\pm$0.44 | 72.76$\pm$0.40
ETN [4] | 83.23$\pm$0.24 | 74.93$\pm$0.28 | 79.08$\pm$0.26
BAUS [25] | 84.00$\pm$0.15 | 83.35$\pm$0.28 | 83.68
CLS | 85.13$\pm$0.35 | 84.47$\pm$0.42 | 84.76$\pm$0.39
CLS w/o Joint Dis+Cls | 83.05$\pm$0.37 | 84.14$\pm$0.45 | 83.52$\pm$0.41

###  4.3 Partial UDA for Image Classification

To demonstrate it generally, we also apply our method to partial UDA task,
i.e., a specific label shift setting, although CLS is developed for a more
general problem. We note that partial UDA is not the extreme of label shift,
when we use KL-divergence as the measurement.

Office-31 dataset [46] is the $defacto$ standard benchmark for domain
adaptation. It contains 4,652 images with 31 classes from three domains:
Amazon (A), Webcam (W) and DSLR (D), each containing $2,817$, $795$ and $498$
images respectively.

Following the experimental setting in [4, 2], we select 10 classes shared by
Office-31 and Caltech-256 to form a new target domain. Note that there are 31
classes in the source domain and 10 classes in the target domain, and 21
classes have zero probability in the target label distribution.

We compare the performances of different methods on Office-31 with the same
base net ResNet-50 in Table 3 left. All CLSs achieve better or comparable
performance, compared with the state-of-the-art methods. This may be
benefited, by also explicitly involving the conditional shift alignment along
with addressing the label shift. The joint parameterization can consistently
help the framework to align conditional and label distributions, respectively.

Besides, ImageNet-Caltech is a large scale dataset, which consists of
ImageNet1K [45] and Caltech [17] datasets. Since they share 84 classes, we
typically use 1,000 categories ImageNet data as a source domain, while using
84 categories Caltech data as a target domain, or choose 256 categories
Caltech as a source domain, while using 84 categories ImageNet as a target
domain. Considering that the networks are usually trained on ImageNet, we
follow the conventional partial UDA methods [4, 2] for
Caltech(256)$\rightarrow$ImageNet(84) adaptation, which uses the validation
set of ImageNet as a target domain.

We give the comparison results in Table 5 right. It is worth noticing that
without considering the conditional and label distribution shift, DAN, DANN,
and ADDA obtain worse performance than the ResNet-50 backbone in many tasks.
To avoid the negative transfer in partial UDA setting, [4] propose a weighting
mechanism to quantify the transferability of each source example. CSL
outperforms or on par with the single example-based weighting methods [4] on
most of the tasks, showing the efficiency of the class-based balancing scheme.
Of note, the to-be-learned weights are much fewer than example-based
weighting.

###  4.4 Partial UDA for Semantic Segmentation

To investigate a more challenging partial UDA scenario, we propose a new task
for semantic segmentation. The Cityscapes to NTHU is a cross-city task with
real-world images [8]. We choose the Tokyo subset of the NTHU.

The NTHU [8] has 13 segmentation classes shared with the Cityscapes dataset.
Following the standard protocol in NTHU, the pole, fence, and wall are
considered as a building. In addition, the truck is regarded as the car class,
and the terrain is regarded as the vegetation class. According to the previous
works [8, 66], the 100 images in each city are separated to ten folds for the
leave-one-out cross-validation. NTHU does not have the category of the train.
To construct a more uneven target label distribution, we remove all of the
images with the motor, bike, and rider classes in NTHU. Since the fact that
motors are prohibited in many cities of China, and bikes are not allowed in
highway can be used as prior knowledge.

As shown in Table 4, our CLS works well with different popular backbones
$i.e.,$ Dilation-Frontend Convolution as in [52] and ResNet101 as in [66]. We
note that the example weighted adversarial partial UDA [4] and moment
matching-based methods usually cannot be well scalable to the pixel-level
classification-based segmentation task. CLS can be a strong baseline of
partial UDA for semantic segmentation.

##  5 Conclusion

In this paper, we tackle the problem of both conditional and label shifts in
the context of adversarial UDA. We carry out a thoroughly theoretical analysis
to demonstrate that it gives a more realistic and even minimal assumption in
many tasks. Then, we propose to align these two shifts in an alternative
optimization with a fade-in $p_{t}(y)$ framework which consists of three to-be
learned players, i.e., feature extractor, joint parameterized discriminator
and classifier, and target label distribution estimator. The merits of joint
parameterization are also investigated. After the training, the posterior
alignment is applied for real-world deployment. Extensive experiments on
popular (partial) UDA benchmarks verify our analysis and revoke the
adversarial learning, which can still be effective as the recent fast
developed self-training, dropout or moment matching methods.

## Acknowledgments

This work was supported in part by the Hong Kong GRF 152202/14E, PolyU Central
Research Grant G-YBJW, and Jiangsu NSF (BK20200238).

## References

  * [1] Kamyar Azizzadenesheli, Anqi Liu, Fanny, and Animashree.  Regularized learning for domain adaptation under label shifts.  ICLR, 2019.
  * [2] Cao, Ma, Long, and Wang.  Partial adversarial domain adaptation.  In ECCV, 2018.
  * [3] Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Michael I Jordan.  Partial transfer learning with selective adversarial networks.  In CVPR, 2018.
  * [4] Zhangjie Cao, Kaichao You, Mingsheng Long, Jianmin Wang, and Qiang.  Learning to transfer examples for partial domain adaptation.  In CVPR, 2019.
  * [5] Chan and Hwee.  Word sense disambiguation with distribution estimation.  In IJCAI, 2005.
  * [6] Tong Che, Xiaofeng Liu, Site Li, Yubin Ge, Ruixiang Zhang, Caiming Xiong, and Yoshua Bengio.  Deep verifier networks: Verification of deep discriminative models with deep generative models.  AAAI, 2021.
  * [7] Qingchao Chen, Yang Liu, Zhaowen Wang, Ian Wassell, and Kevin Chetty.  Re-weighted adversarial adaptation network for unsupervised domain adaptation.  In CVPR, 2018.
  * [8] Yi-Hsin Chen, Wei-Yu Chen, Yu-Ting Chen, Bo-Cheng Tsai, Yu-Chiang Frank Wang, and Min Sun.  No more discrimination: Cross city adaptation of road scene segmenters.  In ICCV, 2017.
  * [9] Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoff Gordon.  Domain adaptation with conditional distribution matching and generalized label shift.  arXiv:2003.04475, 2020.
  * [10] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.
  * [11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248–255. Ieee, 2009.
  * [12] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  arXiv preprint arXiv:1409.7495, 2014.
  * [13] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor.  Domain-adversarial training of neural networks.  JMLR, 2016.
  * [14] Behnam Gholami, Pritish Sahu, Minyoung Kim, and Vladimir Pavlovic.  Task-discriminative domain alignment for unsupervised domain adaptation.  In ICCV, 2019.
  * [15] Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf.  Domain adaptation with conditional transferable components.  In ICML, 2016.
  * [16] Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard.  Covariate shift by kernel mean matching.  2009\.
  * [17] Gregory Griffin, Alex Holub, and Pietro Perona.  Caltech-256 object category dataset.  2007\.
  * [18] Gewen He, Xiaofeng Liu, Fangfang Fan, and Jane You.  Classification-aware semi-supervised domain adaptation.  In CVPR Workshops, pages 964–965, 2020.
  * [19] Gewen He, Xiaofeng Liu, Fangfang Fan, and Jane You.  Image2audio: Facilitating semi-supervised audio emotion recognition with facial expression image.  In CVPR Workshops, 2020.
  * [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [21] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell.  Cycada: Cycle-consistent adversarial domain adaptation.  In ICML, 2018.
  * [22] Dapeng Hu, Jian Liang, Qibin Hou, Hanshu Yan, Shuicheng Yan, and Jiashi Feng.  Panda: Prototypical unsupervised domain adaptation.  ECCV, 2020.
  * [23] Wouter M Kouw.  An introduction to domain adaptation and transfer learning.  arXiv, 2018.
  * [24] Lian, Lv, and Duan.  Constructing self-motivated pyramid curriculums for cross-domain semantic segmentation: A non-adversarial approach.  ICCV, 2019.
  * [25] Jian Liang, Yunbo Wang, Dapeng Hu, Ran He, and Jiashi Feng.  A balanced and uncertainty-aware approach for partial domain adaptation.  In ECCV, 2020.
  * [26] Zachary Lipton, Yu-Xiang Wang, and Alexander Smola.  Detecting and correcting for label shift with black box predictors.  In ICML, 2018.
  * [27] Xiaofeng Liu, Yang Chao, Jane J You, C-C Jay Kuo, and Bhagavatula Vijayakumar.  Mutual information regularized feature-level frankenstein for discriminative recognition.  IEEE T-PAMI, 2021.
  * [28] Xiaofeng Liu, Bo Hu, Linghao Jin, Xu Han, Fangxu Xing, Jinsong Ouyang, Jun Lu, Georges El Fakhri, and Jonghye Woo.  Domain generalization under conditional and label shifts via variational bayesian inference.  In IJCAI, 2021.
  * [29] Xiaofeng Liu, Bo Hu, Xiongchang Liu, Jun Lu, Jane You, and Lingsheng Kong.  Energy-constrained self-training for unsupervised domain adaptation.  In ICPR, 2021.
  * [30] Xiaofeng Liu, Linghao Jin, Xu Han, and Jane You.  Mutual information regularized identity-aware facial expression recognition in compressed video.  Pattern Recognition, 2021.
  * [31] Xiaofeng Liu, Site Li, Lingsheng Kong, Wanqing Xie, Ping Jia, Jane You, and BVK Kumar.  Feature-level frankenstein: Eliminating variations for discriminative recognition.  In CVPR, 2019.
  * [32] Xiaofeng Liu, Fangxu Xing, Maureen Stone, Jiachen Zhuo, Reese Timothy, Jerry L Prince, Georges El Fakhri, and Jonghye Woo.  Generative self-training for cross-domain unsupervised tagged-to-cine mri synthesis.  MICCAI, 2021.
  * [33] Xiaofeng Liu, Fangxu Xing, Chao Yang, Georges El Fakhri, and Jonghye Woo.  Adapting off-the-shelf source segmenter for target medical image segmentation.  MICCAI, 2021.
  * [34] Xiaofeng Liu, Yang Zou, Lingsheng Kong, Zhihui Diao, Junliang Yan, Jun Wang, Site Li, Ping Jia, and Jane You.  Data augmentation via latent space interpolation for image classification.  In ICPR. IEEE, 2018.
  * [35] Long, Cao, Wang, and Jordan.  Conditional adversarial domain adaptation.  In NIPS, 2018.
  * [36] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I Jordan.  Learning transferable features with deep adaptation networks.  ICML, 2015.
  * [37] Ke Mei, Chuang Zhu, Jiaqi Zou, and Shanghang Zhang.  Instance adaptive self-training for unsupervised domain adaptation.  ECCV, 2020.
  * [38] Jose G Moreno-Torres, Troy Raeder, RocíO Alaiz-RodríGuez, Nitesh V Chawla, and Francisco.  A unifying view on dataset shift in classification.  PR.
  * [39] Yingwei Pan, Ting Yao, Yehao Li, and Tao Mei.  Transferrable prototypical networks for unsupervised domain adaptation.  In CVPR, 2019.
  * [40] Pau Panareda Busto and Juergen Gall.  Open set domain adaptation.  In ICCV, 2017.
  * [41] Vishal M Patel, Raghuraman Gopalan, Ruonan Li, and Rama Chellappa.  Visual domain adaptation: A survey of recent advances.  IEEE signal processing magazine, 32(3):53–69, 2015.
  * [42] Xingchao Peng, Ben Usman, Neela Kaushik, Dequan Wang, Judy Hoffman, Kate Saenko, Xavier Roynard, Jean-Emmanuel Deschaud, Francois Goulette, Tyler L Hayes, et al.  Visda: A synthetic-to-real benchmark for visual domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 2021–2026, 2018.
  * [43] Pedro O Pinheiro.  Unsupervised domain adaptation with similarity learning.  In CVPR, 2018.
  * [44] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun.  Playing for data: Ground truth from computer games.  In ECCV, 2016.
  * [45] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [46] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell.  Adapting visual category models to new domains.  In European conference on computer vision, pages 213–226. Springer, 2010.
  * [47] Saito, Ushiku, Harada, and Saenko.  Adversarial dropout regularization.  ICLR, 2018.
  * [48] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya.  Maximum classifier discrepancy for unsupervised domain adaptation.  In CVPR, 2018.
  * [49] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  In NIPS, 2016.
  * [50] Swami Sankaranarayanan, Yogesh Balaji, Carlos D Castillo, and Rama Chellappa.  Generate to adapt: Aligning domains using generative adversarial networks.  In CVPR, 2018.
  * [51] Tran, Sohn, Yu, Liu, and Chandraker.  Gotta adapt all joint pixel and feature-level domain adaptation for recognition in the wild.  In CVPR, 2019.
  * [52] Tsai, Hung, Schulter, Sohn, Yang, and Chandraker.  Learning to adapt structured output space for semantic segmentation.  In CVPR, 2018.
  * [53] Yi-Hsuan Tsai, Kihyuk Sohn, and Samuel.  Domain adaptation for structured output via discriminative representations.  ICCV, 2019.
  * [54] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell.  Adversarial discriminative domain adaptation.  In CVPR, 2017.
  * [55] Yuan Wu, Diana Inkpen, and Ahmed El-Roby.  Dual mixup regularized learning for adversarial domain adaptation.  ECCV, 2020.
  * [56] Yifan Wu, Ezra Winston, Divyansh, and Lipton.  Domain adaptation with asymmetrically-relaxed distribution alignment.  In ICML, 2019.
  * [57] Zuxuan Wu, Xintong Han, Yen-Liang Lin, Mustafa Gokhan Uzunbas, Tom Goldstein, Ser Nam Lim, and Larry S.  Dcan: Dual channel-wise alignment networks for unsupervised scene adaptation.  In ECCV, 2018.
  * [58] Jihan Yang, Ruijia Xu, Ruiyu Li, Xiaojuan Qi, Xiaoyong Shen, Guanbin Li, and Liang Lin.  An adversarial perturbation oriented domain adaptation approach for semantic segmentation.  2020\.
  * [59] Kaichao You, Ximei Wang, Mingsheng Long, and Jordan.  Towards accurate model selection in deep unsupervised domain adaptation.  In ICML, 2019.
  * [60] Fisher Yu and Vladlen Koltun.  Multi-scale context aggregation by dilated convolutions.  arXiv preprint arXiv:1511.07122, 2015.
  * [61] Bianca Zadrozny.  Learning and evaluating classifiers under sample selection bias.  In ICML, 2004.
  * [62] Jing Zhang, Zewei Ding, Wanqing Li, and Philip Ogunbona.  Importance weighted adversarial nets for partial domain adaptation.  In CVPR, 2018.
  * [63] Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun.  Domain adaptation under target and conditional shift.  In ICML, 2013.
  * [64] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890, 2017.
  * [65] Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang.  Confidence regularized self-training.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5982–5991, 2019.
  * [66] Yang Zou, Zhiding Yu, B. V. K. Vijaya Kumar, and Jinsong Wang.  Unsupervised domain adaptation for semantic segmentation via class-balanced self-training.  In ECCV, 2018.


11institutetext: Imperial College London, UK 22institutetext: Université
Paris-Saclay, France 22email<EMAIL_ADDRESS><EMAIL_ADDRESS>
# A Stochastic Analysis Approach to Tensor Field Theories

Ajay Chandra1 Léonard Ferdinand2

###### Abstract

We present two different arguments using stochastic analysis to construct
super-renormalizable tensor field theories, namely the $\mathrm{T}^{4}_{3}$
and $\mathrm{T}^{4}_{4}$ models. The first approach is the construction of a
Langevin dynamic [ParisiWu, GH21] combined with a PDE energy estimate while
the second is an application of the variational approach of Barashkov and
Gubinelli [BG20]. By leveraging the melonic structure of divergences,
regularising properties of non-local products, and controlling certain random
operators, we demonstrate that for tensor field theories these arguments can
be significantly simplified in comparison to what is required for
$\Phi^{4}_{d}$ models.

###### Contents

  1. 1 Introduction

##  1 Introduction

A class of heavily studied and paradigmatic models in constructive quantum
field theory are the $\Phi^{4}_{d}$ measures. In finite volume, these are non-
Gaussian probability measures $\mu$ supported on distributions (over space)
which are formally written

$\mathrm{d}\mu(\phi)\propto\exp\Big{(}-\frac{\lambda}{4}\|\phi\|_{L^{4}({{\mathchoice{\hbox{\bf
T}}{\hbox{\bf T}}{\hbox{\scriptsize\bf T}}{\hbox{\tiny\bf
T}}}}^{d})}^{4}+\frac{a}{2}\|\phi\|_{L^{2}({{\mathchoice{\hbox{\bf
T}}{\hbox{\bf T}}{\hbox{\scriptsize\bf T}}{\hbox{\tiny\bf
T}}}}^{d})}^{2}\Big{)}\mathrm{d}\mcb{g}(\phi)\,.{\hbox{\rm(1)}\kern
5.69046pt\kern-5.69046pt}q:local_{p}hi_{4}{=}e0{1}$


# Double-relativistic-electron-shell laser proton acceleration

Yongsheng Huang http://www.anianet.com/adward<EMAIL_ADDRESS>China
Institute of Atomic Energy, Beijing 102413, China.  Naiyan Wang  Xiuzhang Tang
Yijin Shi  China Institute of Atomic Energy, Beijing 102413, China.  Yan
Xueqing  Institute of Heavy Ion Physics, Peking University, Beijing 100871,
China  Zhang Shan  Beijing Normal University, Beijing 100875, China

###### Abstract

A new laser-proton acceleration structure combined by two relativistic
electron shells, a suprathermal electron shell and a thermal electron cloud is
proposed for $a\gtrapprox 80\sigma_{0}$, where a is the normalized laser field
and $\sigma_{0}$ is the normalized plasma surface density. In the new region,
a uniform energy distribution of several GeV and a monoenergetic hundreds-of-
MeV proton beam have been obtained for $a=39.5$. The first relativistic
electron shell maintains opaque for incident laser pulse in the whole process.
A monoenergetic electron beam has been generated with energy hundreds of MeV
and charge of hundreds of pC. It is proposed a stirring solution for
relativistic laser-particle acceleration.

###### pacs:

52.38.Kd,41.75.Jv,52.40.Kh,52.65.-y

Laser-ion acceleration has been an international research focusMakoTajima ;
Machnisms ; Esirkepov , however it is still a challenge to obtain mono-
energetic proton beams larger than $100\mathrm{MeV}$. Although the field in
the laser-plasma acceleration is three to four orders higher than that of the
classic accelerators, it decreases to zero quickly in several pulse durations
for target normal sheath acceleration (TNSA)Machnisms  for
$a\ll\sigma_{0}=\frac{n_{e}l_{e}}{n_{c}\lambda}$, where $a=eE_{l}/m\omega c$,
$E_{l}$ is the electric field of the laser pulse, $\omega$ is the laser
frequency, $e$ is the elementary charge, $m$ is the electron mass, $c$ is the
light velocity, $n_{0}$ is the initial plasma density, $n_{c}$ is the critical
density, $\lambda$ is the wave length, $n_{e}$ is the electron density,
$l_{e}$ is the thickness of the electron shell. As a promising method to
generate relativistic mono-energetic protons, radiation pressure acceleration
(RPA) has attracted more attentionEsirkepov ; Henig ; YanxqPRL ; unlimitedRPA
and becomes dominant in the interaction of the ultra-intense laser pulse with
thin foils if $a\approx\sigma_{0}$. Even in the unlimited ion
accelerationunlimitedRPA , only the ions trapped in the electron shell can
obtain efficient acceleration, therefore, the total charge of the ion beam is
quite limited due to the transverse expansionunlimitedRPA . Although in RPA
region, the energy dispersion will become worse with time.

Fortunately, for $a\gtrapprox 80\sigma_{0}$, in the relativistic case, a new
acceleration region appears: double relativistic electron shells come into
being. The ions between the two electron shells will be accelerated most
efficiently and obtain a uniform energy distribution. The first electron shell
is ultra-relativistic and is totally separated from the ions. The second
electron shell comes into being in the potential well induced by the electron
recirculation and will also be relativistic. It is in the ion region and
follows the ion front and forms a potential well which traps energetic ions
and accelerates them to be quasi-mono-energetic and relativistic. Following
the second electron shell, a suprathermal electron beam comes into being and
induces another potential well which can also trap lots of ions and accelerate
them to obtain a monoenergetic relativistic one. On the whole, the maximum ion
energy can reach several GeV and a relativistic monoenergetic ion beam with
relative energy dispersion smaller then $5\%$ can be obtained.

As a ultraintense laser pulse is shot on a ultra-thin plasma foil for
$a\gtrapprox 80\sigma_{0}$, the electron shell is compressed to ultra-high
density and pushed forward to be separated from the ion shell totally, and
gains ultra-relativistic energy that can make sure it opaque for the laser
pulse. In the whole process, the first electron shell keeps opaque for the
incident laser pulse and is pushed by it continuously. According to Eq.
(22)unlimitedRPA , it can be satisfied that the opaqueness condition of
electron shell for laser in the acceleration:

$a_{0}\leq\pi(\gamma_{e}+p_{e})\hat{n}_{e}\hat{l}_{e},$ (1)

since $d\ln(p\hat{n}_{e}\hat{l}_{e})/dt>0$, as pointed by Bulanov, $p\propto
t^{1/3}$ is the normalized electron momentum, and $n_{e}l_{e}=n_{0}l_{0}$ in
the no transverse expansion case, where the electron density $n_{e},n_{0}$ are
normalized by $n_{c}$ and $l_{e}$ is normalized by $\lambda$,
$a_{0}=eE_{0}/m\omega_{0}c$, $E_{0}$ is the electric field of the laser pulse,
$\omega_{0}$ is the laser frequency, $e$ is the elementary charge, $m$ is the
electron mass, $c$ is the light velocity, $n_{0}$ is the initial plasma
density, $n_{c}$ is the critical density, $\lambda$ is the wave length. In the
ultra-relativistic case, the r.h.s. of Eq. (1) is approximate $2\pi
p\hat{n}_{e}\hat{l}_{e}$.

For $a=39.5$ and $\hat{n}_{e}\hat{l}_{e}=49\times 0.01=0.49$, $a\approx
81\hat{n}_{e}\hat{l}_{e}$. When the laser pulse interacts with the plasma
shell, the electrons are compressed to high density and pushed forward to be
separated from the ion shell. As shown by Figure 1 (c) and (d), at
$t=25\mathrm{fs}$, the normalized momentum of the electron shell reaches
$20-100$. With Eq. (1), the compressed high-density electron shell is opaque
for the laser pulse as shown by Figure 1 (f). The electrons will be
accelerated efficiently and continuously by the radiation pressure of the
laser.

Figure 1:  (Color online) Simulation results by one-dimensional VORPAL at
$t=25$fs: the first opaque relativistic electron shell forms and is totally
separated from ions. (a) and (c): the phase space distribution of ions and
electrons. The electron shell is totally separated from the ions and is
relativistic and opaque for laser pulse. (b) and (d): the density distribution
of the ions and electrons with normalized momentum. The ion energy
distribution is uniform. The electron energy distribution contains several
monoenergetic ones. (e): the longitudinal space charge separation field, which
is similar to the field in a capacity, is uniform and is $9.95\times
10^{12}$V/m. (f): the potential and laser field. The electron shell is
relativistic and opaque for the laser pulse.

This high-density relativistic electron shell is called the first relativistic
electron shell. Between the first electron shell and the ions, a uniform
space-charge separation field forms and accelerates the ions at the ion front
and drags the electrons as shown by Figure 1 (e). The ”capacity” field is
decided by the surface density of the electron shell:

$E_{cap}=\frac{en_{e}l_{e}}{\epsilon_{0}},$ (2)

For $n_{e}=5.5\times 10^{22}\mathrm{/cm^{3}}$, $l_{e}=10\mathrm{nm}$, the
stable field is $9.95\times 10^{12}\mathrm{V/m}$, which accelerates the ions
at the rear of the ion shell continuously. Therefore the maximum ion energy is
proportional to the acceleration length, $d_{acc}$,

$E_{i}=E_{cap}d_{acc}(\mathrm{eV}),$ (3)

before the electron shell breaks up.

After several hundreds of femtoseconds, some electrons leak out from the
electron shell continuously and move backward and round again and follow the
ion front, however, they can not catch the ion front. Figure 2 (c), (e) and
(f) shows the electron recirculation, the decrease of the separation field,
and the formation of the potential well for electrons at $t=450\mathrm{fs}$
respectively. The normalized maximum electron momentum reaches about $500$ as
shown by Figure 2 (d). Figure 2 (f) shows that the deepness of the potential
well for electrons increases about to $0.3\mathrm{GeV}$. From Figure 2 (a) and
(b), the maximum ion energy is about $500\mathrm{MeV}$. The acceleration
length is about $65\mathrm{\mu m}$. A monoenergetic electron beam of
$186\mathrm{MeV}$ and $132$pC is obtained as shown by Figure 2 (d). It is
obvious that the first relativistic electron shell is still opaque for the
laser pulse.

Figure 2:  (Color online) Simulation results by one-dimensional VORPAL at
$t=450$fs: electron recirculation begins and generates a potential well for
electrons, where $p_{i},p_{e}$ is momentum of ion and electron respectively,
$M$ is the proton mass. (a) and (c): the phase space of ions and electrons
respectively. The electron recirculation happens and some of them move in the
opposite direction relativistically. (b) and (d): the energy distribution of
ions and electrons respectively. The first electron shell is accelerated most
efficiently and continuously. (e) the longitudinal field decreases due to the
electron recirculation. However it is still uniform between the ion front and
the first electron shell. (f): a potential hill for ions and a potential well
for electrons forms due to the electron recirculation. The first relativistic
electron shell maintains opaque for the laser pulse.

As shown by Figure 3 (d), the recirculating electrons cumulate and drive up
the electron potential. In front of and behind the accumulating electrons, two
potential wells are forming for electrons. The potential well I still traps
and accelerates the accumulating electrons to generate the second relativistic
high-density electron shell. In the potential well II, some electrons at the
end of the second relativistic electron shell drop into it and will be trapped
and accelerated to form a suprathermal electron shell as shown by Figure 5
(d). At the same time and at the local position of the second relativistic
electron shell, potential well III for ions traps lots of ions and accelerates
them to be relativistic and monoenergetic.

Figure 3:  (Color online) Simulation results by one-dimensional VORPAL at
$t=1.25$ps: the second relativistic electron shell forms and traps ions to be
accelerated to relativistic. (a): the second relativistic electron shell forms
in the potential well. It drives up the potential and forms two potential well
I and II for electrons in front of and behind itself, a potential well III for
ions at the local position of itself. It is shown clearly in (d). (b): the
number density distribution of electrons. (c): the electron recirculation
decreases the longitudinal field continuously. It is still uniform between the
ion front and the first electron shell. Figure 4:  (Color online) Simulation
results by one-dimensional VORPAL at $t=3.075$ps: the suprathermal electron
shell forms and traps ions to obtain a flap-top $3-47$MeV energy distribution.
(a) and (c): the phase space of ions and electrons. The suprathermal electron
shell forms in the potential well II and then potential V for electrons and
potential IV are generated. (b) and (d) the energy distribution of ions and
electrons. A monoenergy ion beam with energy of $627$MeV is obtained in the
second relativistic electron shell. (e): the longitudinal field induced by the
double electron shell. (f): the potential IV for ions is induced by the
suprathermal electron shell and traps ions and improves the energy dispersion
of the $3-47$MeV ion beam. The potential V can trap slow electrons and
thermalize them to obtain thermal electron cloud. Potential III is nearly
filled to be flat and the energy dispersion of the 627MeV monoenergetic ion
beam will become worse.

With time, the slow recirculating electrons can also be trapped in potential
well V as shown in Figure 4(f) and the third suprathermal electron shell forms
in the potential well II as shown by Figure 5. At the position of the shell,
potential well IV traps the ions and accelerates them to obtain a quasi-
monoenergetic distribution of $171\pm 10\mathrm{MeV}$. Behind the shell, a
potential well for electrons traps them and a thermal electron cloud is
generated. The ions between the double relativistic electron shell have a
uniform distribution from $1\mathrm{GeV}$ to $2.18\mathrm{GeV}$. Trapped by
the second relativistic electron shell, the maximum energy reach
$981\mathrm{MeV}$. As shown in Figure 4 (f), the potential well III has been
filled up nearly, then the energy dispersion will become worse. The ions with
larger energy will coast down the following potential slope and get into the
ion beam between the double relativistic electron shell. The ion number has a
steep descent for the energy larger than $981\mathrm{MeV}$ and has a slow drop
for the energy smaller than $981\mathrm{MeV}$. In the electron energy
distribution, there is a monoenergetic one of $385\pm 10\mathrm{MeV}$ and
$163\mathrm{pC}$, a ultra-relativistic one of $1\mathrm{GeV}$ and a Maxwellian
one which contains the thermal electron cloud and the suprathermal electron
shell. As shown by Figure 5(c) and (f), the ion front is between the double
electron shell. The ions between the double electron shell coast down the
potential slope and obtain relativistic energy as shown in Figure 5(h) and
(b).

Figure 5:  (Color online) Simulation results by one-dimensional VORPAL at
$t=4.025$ps: the thermal electron cloud and the suprathermal electron shell
come into being. (a) and (d): the phase-space of ions and electrons
respectively. The electrons contain four main parts: the double relativistic
electron shells, the suprathermal electron shell, the thermal electron cloud.
(b) and (e): the energy distribution of ions and electrons respectively. A
monoenergetic ion beam with energy of $171\pm 10$MeV is obtained by the
suprathermal electron shell. Trapped and accelerated by the second
relativistic electron shell, the ion energy distribution drops down at
$981$MeV. (c) and (f) the number density of ions and electrons respectively.
(g) the longitudinal field. (h) the potential for ions and electrons. (i) the
laser pulse field. The first electron shell maintains opaque for laser pulse.

In conclusion, the double relativistic electron shells, the suprathermal
electron shell and the thermal electron cloud induce a new region of laser
particle acceleration. In the process, several potential wells for ions and
electrons are generated. On the whole, the double relativistic electron shells
induce two relativistic platforms of the ion energy distribution. The
suprathermal electron shell traps and accelerates a monoenergetic ion beam
with several hundreds of MeV, whose relative energy dispersion is near $5\%$.
Together with the thermal electron cloud, a thermal Maxwellian ion beam has
been obtained.

###### Acknowledgements.

The authors would like to thank Dr. Hong-Yu Wang for useful discussion. The
computation was carried out at the HSCC of Beijing Normal University. This
work was supported by the Key Project of Chinese National Programs for
Fundamental Research (973 Program) under contract No. $2011CB808104$ and the
Chinese National Natural Science Foundation under contract No. $10834008$.

## References

  * (1) F. Mako and T. Tajima, Phys. Fluids 27, 1815 (1984).
  * (2) H. Schwoerer, S. Pfotenhauer, O. Jackel, et al., Nature 439, 445 (2006). electronacc
  * (3) T. Esirkepov, M. Borghesi, S. V. Bulanov, G. Mourou, and T. Tajima, Phys. Rev. Lett. 92, 175003 (2004).
  * (4) A. Henig, S. Steinke, M. Schn rer, et al., Phys. Rev. Lett. 103, 245003 (2009).
  * (5) H. Schwoerer, S. Pfotenhauer, O. Jackel, et al., Nature 439, 445 (2006).
  * (6) X. Q. Yan, C. Lin, Z.M. Sheng, et al., Phys. Rev. Lett. 100, 135003 (2008).
  * (7) S. V. Bulanov, E. Yu. Echkina, T. Zh. Esirkepov, et al., Phys. Rev. Lett. 104, 135003 (2010).
  * (8) X. Q. Yan, T. Tajima, M. Hegelich, et al., Appl. Phys. B, 98 711-721 (2010).
  * (9) Y. S. Huang,
  * (10) T. P. Yu, A. Pukhov, G. Shvets, and M. Chen, Phys. Rev. Lett. 105, 065002 (2010).
  * (11) A. Macchi, F. Cattani, T. V. Liseykina and F. Cornolti, Phys. Rev. Lett. 94, 165003 (2005).