# Deep Distributionally Robust Learning for Calibrated Uncertainties under
Domain Shift

Haoxuan Wang
Department of CSE
Shanghai Jiao Tong University
Shanghai, China, 200240
<EMAIL_ADDRESS>
&Anqi Liu
Department of CMS
California Institute of Technology
Pasadena, CA 91106
<EMAIL_ADDRESS>
&Zhiding Yu
NVIDIA
<EMAIL_ADDRESS>
&Junchi Yan
Department of CSE
Shanghai Jiao Tong University
Shanghai, China, 200240
<EMAIL_ADDRESS>
&Yisong Yue
Department of CMS
California Institute of Technology
Pasadena, CA 91106
<EMAIL_ADDRESS>
&Anima Anandkumar
Department of CMS
California Institute of Technology
Pasadena, CA 91106
<EMAIL_ADDRESS>

###### Abstract

We propose a deep distributionally robust learning framework for calibrated
uncertainties under domain shifts. We consider cases where the source
(training) distribution differs significantly from the target (test)
distribution. In addition to the standard class predictor, our framework
contains a binary domain classifier which estimates the density ratio between
the source and target domains. We incorporate both with neural networks and
train them end-to-end. The framework is demonstrated to generate calibrated
uncertainties that benefit many downstream tasks, including unsupervised
domain adaptation (UDA) and semi-supervised learning (SSL) where methods such
as self-training and FixMatch use uncertainties to select confident pseudo-
labels. Our experiments show that the introduction of DRL to these methods
leads to significant improvements in cross-domain performance. We also
demonstrate that the produced density ratio estimates show agreement with the
human selection frequencies, suggesting a match with the human perceived
uncertainties. Source code of this work will be made publicly available.

##  1 Introduction

Uncertainty estimation is an important machine learning problem [1] and is
becoming increasingly relevant to trustworthy AI [2, 3, 4]. In addition, many
important downstream applications rely heavily on the correct estimation of
uncertainties. This includes unsupervised domain adaptation (UDA) [5] and
semi-supervised learning (SSL) [6] where uncertainties are used to solicit
pseudo-labels for training on unlabeled data. It is shown that reliable
pseudo-labels help avoid error propagation and catastrophic failures in early
iterations [7].

However, obtaining reliable uncertainty estimation is challenging. In contrast
to semantic recognition, obtaining the ground truths of uncertainties from
real-world data can be costly and difficult. It is also known that commonly
used probabilistic proxies in deep neural networks, such as softmax output,
tend to give overconfident estimates [8, 9]. In many real-world applications,
the models are often deployed on a different target domain with out-of-
distribution (OOD) samples. Such distributional shifts tend to aggravate the
existing issues in uncertainty estimation, leading to false but overconfident
predictions on unfamiliar samples [10].

There have been many methods proposed to calibrate the confidence of deep
learning models so that the uncertainty level of a model prediction reflects
the likelihood of the true event [8]. Label smoothing is a popular approach to
reduce overconfidence and promote more uniform outputs [11]. Temperature
scaling is another method where the logit scores are rescaled by a calibrated
temperature [12]. Approaches such as Monte-Carlo sampling [9] and Bayesian
inference [13, 14] model uncertainties from a Bayesian learning perspective
but are computationally more expensive. Even though these methods lead to more
calibrated uncertainties in general, recent studies show that their results
cannot be fully trusted under domain shift [15].

|
---|---
(a) The end-to-end framework of DRL | (b) $P_{s}(\bm{x})/P_{t}(\bm{x})$ for target images: 1.004 (L) v.s. 2.132 (R)
Figure 1: (a) Architecture for end-to-end training of the DRL framework. We
present training details in Sec. 2.3. (b) Density ratios
($P_{s}(\bm{x})/P_{t}(\bm{x})$) and example images for category “Train” in
VisDA. Deep DRL outputs lower density ratio (more uncertain predictions) on
"Train" that is not well-represented in the source domain (left bottom).

Our approach. To mitigate domain shifts, a key idea in this work is to
understand “how overlapped” are the training data and the test data. Such
information measures how informative the source distribution is to a test
sample, thus providing guidance to the confidence calibration. We incorporate
this idea through density ratio. Recall that the probability output of a
standard neural network can be expressed as
$P(y|\bm{x})\propto\exp\left(\bm{\theta}_{y}\cdot\bm{\phi}(\bm{x})\right)$,
where $\bm{x}$ is the predictor input, $\bm{\phi}(\bm{x})$ is the data
feature, and $\bm{\theta}_{y}$ is the model parameter of the $y$-th class. We
instead consider the following predictive form:

$P(y|\bm{x})\propto\exp\left(\frac{P_{\text{s}}(\bm{x})}{P_{\text{t}}(\bm{x})}\bm{\theta}_{y}\cdot\bm{\phi}(\bm{x})\right),$
(1)

where $P_{\text{s}}(\bm{x})$ and $P_{\text{t}}(\bm{x})$ are the densities of a
data sample under the source and target distributions, respectively.
Intuitively, the density ratio $P_{\text{s}}(\bm{x})/P_{\text{t}}(\bm{x})$
prevents the model from being overconfident on target inputs far away from the
source domain. When the target domain is close to the source domain
($P_{\text{s}}(\bm{x})/P_{\text{t}}(\bm{x})\rightarrow 1$), the prediction is
close to the original classifier. However, when there exists a domain gap and
the target data $\bm{x}$ is far away from any source data
($P_{\text{s}}(\bm{x})/P_{\text{t}}(\bm{x})\rightarrow 0$), the prediction is
closer to a uniform distribution.

Our contributions. We show that Eq. (1) falls under a deep distributionally
robust learning (DRL) framework. While many previous DRL methods [16, 17]
operate in low dimensional spaces using kernel density estimators, our
framework needs to work with deep features, thus requiring density estimation
in high dimensional spaces. We show that the deep DRL framework scales to
real-world computer vision tasks, and is able to produce calibrated
uncertainties under domain shift.

Our technical contributions can be summarized as follows:

  * 1)

We propose a deep DRL framework for uncertainty estimation under domain shift.
We make the framework end-to-end differentiable for training at scale, by
introducing an additional binary domain classifier network which learns to
predict density ratios between source and target domain (See Figure 1(a)). We
also introduce a regularized DRL framework to further promote smoothed model
prediction and improve the calibration. We derive the prediction form (Thm. 1)
and train the entire regularized model end-to-end.

  * 2)

We demonstrate that the estimated density ratio reflects the relative distance
of an instance from both domains (Figure 1(b)). We show that it is correlated
with human selection frequency which can be regarded as a proxy of the human
uncertainty perception.

  * 3)

We empirically show that the top-1 class predictions are more calibrated than
the empirical risk minimization (ERM) based training and temperature scaling.

  * 4)

We integrate our method as a plugin module in downstream applications such as
unsupervised domain adaption and semi-supervised learning, leading to
significant improvements. For example, incorporating self-training [5] with
DRL leads to state-of-the-art performance on VisDA-2017 [18]. Incorporating
Fixmatch [6] with DRL improves the original Fixmatch by a relative 17%
increase in accuracy under the cross-domain setting.

Our framework involves differentiable density ratio estimation which is
jointly trained with the tasks instead of being “plugged-in". Such a property
allows our framework to be more task-driven with desirable benefits. Recall
that we are considering an unsupervised setting where the target domain labels
are unavailable. Thus a challenge of our method is that the training loss will
require these target labels. However, this can be bypassed thanks to the DRL
formulation where the gradients from the target loss do not depend on the
target labels but only involve the source distribution, thus making our
framework end-to-end trainable. More details are described in Sec. 2.3.

##  2 Deep distributionally robust learning

In this section, we first review the preliminaries of DRL (Sec. 2.1), followed
by a proposed variant of DRL with class regularization (Sec. 2.2). We then
propose an instantiation of deep DRL with a differentiable density ratio
estimation network (Sec. 2.3). Finally, we show the application of deep DRL in
UDA and SSL tasks (Sec. 2.4).

###  2.1 Preliminaries

Notations and definitions: We denote the input and labels by random variables
$X$ and $Y$. We also use $\bm{x}\in\mathbb{R}^{d}$ and $\mathcal{X}$ to
represent the realization and sample space of $X$. Our goal is to find a
predictor

$\displaystyle\bm{F}:\mathbb{R}^{d}\mapsto\mathbb{R}^{C}\triangleq\\{\bm{f}(\bm{x})|\bm{x}\in\mathcal{X},\bm{f}(\bm{x})\in\mathbb{R}^{C}\cap\Delta\\}$
(2)

close to the true underlying $P_{t}(Y|X)$. Here, $d$, $C$ and $\Delta$ denote
the input dimension, class number, and probabilistic simplex, respectively. We
consider the problem with labeled data sampled from a source distribution
$P_{\text{s}}(X,Y)$ and unlabeled data sampled from a target distribution
$P_{\text{t}}(X)$, and use $e{P}_{s}(X,Y)$ to represent the empirical source
distribution. In this work, we consider an important form of domain shift with
the covariate shift assumption $P_{\text{s}}(X)\neq P_{\text{t}}(X)$,
$P_{\text{s}}(Y|X)=P_{\text{t}}(Y|X)$.

Motivation: Traditional empirical risk minimization (ERM) frameworks tend to
fail under covariate shift since ERM empirically learns a predictor
$\hat{P}_{\text{s}}(Y|X)$ from the finite source data that usually fails to
generalize to the target distribution. DRL was previously proposed to overcome
this issue. DRL can be formulated as a two-player adversarial risk
minimization game [19] with the predictor player minimizing a loss, while the
adversary player maximizing the loss. The adversary is allowed to perturb the
labels, subject to certain feature-matching constraints to ensure data-
compatibility.

Formulation: Under covariate shift, DRL [20] deals with the mismatch between
the expected loss and the training data, and is defined on the target data
distribution:

$\displaystyle\hat{P}_{\text{t}}(Y|X)=\operatornamewithlimits{argmin}_{\bm{F}}\;\max_{\bm{G}\in\Sigma}\;\mathbb{E}_{\bm{x}\sim
P_{\text{t}}(X)}\mathcal{L}\left(\bm{f}(\bm{x}),\bm{g}(\bm{x})\right),$ (3)

where $\bm{f}(\bm{x}),\bm{g}(\bm{x})\in\mathbb{R}^{C}$ are the conditional
label distributions given an input $\bm{x}$ and $\bm{F}$, $\bm{G}$ are the
entire distribution over all the input. $\mathbb{E}_{x\sim
P_{\text{t}}}\mathcal{L}(\cdot)$ is an expected log loss on the target input
defined as:

$\displaystyle\mathbb{E}_{\bm{x}\sim
P_{\text{t}}(X)}\mathcal{L}(\bm{f}(\bm{x}),\bm{g}(\bm{x}))\triangleq\mathbb{E}_{\bm{x}\sim
P_{\text{t}}(X)}[-\bm{g}(\bm{x})\cdot\log\bm{f}(\bm{x})].$ (4)

$\bm{F}$ is the predictor player minimizing the loss function while $\bm{G}$
is the adversary maximizing the loss function. After solving this game,
$\bm{F}$ is our estimate of $\hat{P}_{t}(Y|X)$, which we will use for the
classification on the target domain.

Eq. (3) is defined on the target domain only. How could a predictor be
properly trained while there are no target labels available? The answer is
that the adversary $\bm{G}$ is implicitly constrained by the source features.
We use the following constraints to make sure that $\bm{G}$ is close to
$P_{s}(Y|X)$:

$\displaystyle\Sigma=\\{\bm{G}|{\textstyle\sum}_{i}g_{y}\bm{\phi}(\bm{x}_{i})={\textstyle\sum}_{i}\mathbb{I}[y_{i}=y]\bm{\phi}(\bm{x}_{i}),\forall
y\\},$ (5)

where $\bm{x}_{i}\sim P_{s}(X)$ and $g_{y}$ is the $y$-th dimension of
$\bm{g}$. Eq. (5) is a necessary but not sufficient condition for
$\bm{G}=P_{s}(Y|X)$, thus serving as an implicit constraint for $\bm{G}$ to be
close to the true $P_{t}(Y|X)$ under the covariate shift assumption. Given a
predefined feature function $\bm{\phi}$, when the adversary perturbs the
conditional label distribution, certain aggregate function of $\bm{\phi}$ on
$\bm{g}$ should equal to the counterpart on the empirical source data.

From DRL to density ratio: When using the expected target logloss in Eq. (3),
Eq. (1) is derived by solving the predictor $\bm{F}$. Here we refer the
derivation details to [20] but emphasize the important properties of the
prediction as follows.

Representation-level conservativeness: One of the main properties of Eq. (1)
is that the predictions have higher certainty for inputs closer to the source
domain when $P_{\text{s}}(\bm{x})/P_{\text{t}}(\bm{x})$ is large. On the
contrary, when $P_{\text{s}}(\bm{x})/P_{\text{t}}(\bm{x})$ is small, the
prediction is more uncertain. This property reflects the model’s ability to
convey information about what it does not know through the model uncertainty.

###  2.2 Class-regularized distributionally robust learning

Inspired by label smoothing [11] and regularization of neural network outputs
[21], we can further add class-regularization to the conservative predictions
in Eq. (1). Instead of doing it in a post-hoc way, we study how to incorporate
class-regularization into the DRL framework. We propose to use a weighted
logloss to penalize high confidence in the adversary’s label prediction:

$\displaystyle\hat{P}_{\text{t}}(Y|X)=\operatornamewithlimits{argmin}_{\bm{F}}\;\max_{\bm{G}\in\Sigma}\;\mathbb{E}_{\bm{x}\sim
P_{\text{t}}(X)}[-\bm{g}(\bm{x})\cdot\log\bm{f}(\bm{x})]-r\mathbb{E}_{\bm{x}\sim
P_{\text{t}}(X)}\left[\bm{y}\odot\bm{g}(\bm{x})\cdot\log\bm{f}(\bm{x})\right]$
(6)

where $\bm{y}$ is the one-hot class vector, $\odot$ is the element-wise
product, and $r\in[0,1]$ is a hyper-parameter that controls the level of
regularization. $\Sigma$ here is the same as in (3). We observe the whole loss
function is a convex-concave function in terms of $\bm{f}$ and $\bm{g}$.
According to the strong duality, we switch the order of the $\min$ and $\max$.
With a fixed $\bm{g}$, $\bm{f}=\bm{g}$ is the optimal solution of the inner
$\min$ problem. So we have the following lemma (we refer all the proofs to the
appendix):

######  Lemma 1.

Eq. (6) can be reduced to a regularized maximum entropy problem with the
estimator constrained:

$\displaystyle\max_{\bm{F}\in\Sigma}\;\mathbb{E}_{\bm{x}\sim
P_{\text{t}}(X)}[-\bm{f}(\bm{x})\cdot\log\bm{f}(\bm{x})]-r\mathbb{E}_{\bm{x}\sim
P_{\text{t}}(X)}\left[\bm{y}\odot\bm{f}(\bm{x})\cdot\log\bm{f}(x)\right]$ (7)

where $\Sigma$ is the same as Eq. (5), meaning $\bm{F}$ should be close to the
empirical source $P_{s}(Y|X)$.

######  Theorem 1.

The solution of Eq. (7) for takes the form:
$\bm{f}_{\theta,r}(y|\bm{x})\propto\exp\left(\frac{\frac{P_{\text{s}}(\bm{x})}{P_{\text{t}}(\bm{x})}\bm{\theta}_{y}\cdot\bm{\phi}(\bm{x})+r\mathbb{I}(y)}{r\mathbb{I}(y)+1}\right),$
where $\bm{\theta}$ represents the model parameters and $\mathbb{I}(y)$ is the
$y$th dimension of the one-hot encoding $\bm{y}$.

The proof of Theorem 1 follows the same principles when Eq. (1) is derived
from Eq. (3). The training process of this model also follows the same
procedure in [20], except that we use Theorem 1 to compute the prediction
results. In training, we can compute the gradients using source labels where
$\bm{y}$ is the one-hot encoding of each class. In testing, we set $\bm{y}$ to
be an all-one vector.

Class-level regularization: Hyper-parameter $r$ adjusts the smoothness of
$\bm{g}$’s label prediction in Eq. (6). It translates to the $r\bm{y}$ terms
in the prediction form. Intuitively, this regularization increases the correct
label’s prediction result when it is small and decreases it when it is large,
using a threshold of $1$ to discriminate between "large" and "small". This
provides additional regularization and smoothness to the conservative
prediction.

###  2.3 Differentiable density ratio estimation

Density ratio $P_{\text{s}}(\bm{x})/P_{\text{t}}(\bm{x})$ is an important
factor in DRL. However, density ratio estimation [22] can be challenging,
especially in high-dimensional spaces. Usually a plug-in estimator for the
density ratio $P_{\text{s}}(\bm{x})/P_{\text{t}}(\bm{x})$ is used. But when
the estimation performance is far from satisfactory, the plug-in estimates are
sub-optimal for downstream tasks. We propose an end-to-end training procedure
for deep DRL such that the density ratio estimator is trained together with
the target classifier. The key insight here is that we use two neural networks
for classification and density ratio estimation, respectively. See Figure
1(a). We now first introduce differentiable density ratio estimation before
introducing joint training loss and the parameter learning of the two
networks.

Differentiable density ratio estimation: Based on the Bayes’ rule, the density
ratio $\frac{P_{\text{t}}(\bm{x})}{P_{\text{s}}(\bm{x})}$ can be computed from
a conditional domain classifier [23]:
$\frac{P_{\text{t}}(\bm{x})}{P_{\text{s}}(\bm{x})}=\frac{P(\bm{x}|\text{t})}{P(\bm{x}|\text{s})}=\frac{P(\text{t}|\bm{x})P(\text{s})}{P(\text{s}|\bm{x})P(\text{t})}$.
Concretely speaking, the density ratio can be estimated through binary
classification using unlabeled source and target data
($\frac{P(\text{s})}{P(\text{t})}$ is a constant relating to number of source
and target samples). On the other hand, we make the observation that the
density ratio in Eq. (1) and Theorem 1 can be a trainable weight for each
sample, and can receive gradients from the training objective of DRL.
Therefore, we propose to train a discriminative neural network to
differentiate the two domains, which receives training signals from both the
target classification loss in DRL and the binary classification loss.

Joint training loss: Assume $\bm{\phi}(\bm{x},\bm{w}_{r})$ is the
representation learning neural network with parameter $\bm{w}_{r}$. We further
define $P_{d}(X)$ as the joint distribution of both source data and target
data with their domain labels $D=\\{\bm{d}(\bm{x})\\}$. We denote
$\bm{\tau}(\bm{x},\bm{w}_{d})=(\tau_{s},\tau_{t})$ as the probability output
of source and target domains from a domain classifier with parameter
$\bm{w}_{d}$. Our joint training loss is defined as:

$\displaystyle\min_{\bm{w_{r}},\theta,\bm{w_{d}}}\mathbb{E}_{\bm{x}\sim
P_{\text{t}}(X)}\left[-\bm{g}_{t}(\bm{x})\cdot\log\bm{f}(\bm{x};\bm{w}_{r},\bm{\theta},\bm{w}_{d})\right]+\mathbb{E}_{\bm{x}\sim
P_{d}(X)}\left[-\bm{d}(\bm{x})\cdot\log\bm{\tau}(\bm{x},\bm{w}_{d})\right]$
(8)

where $\bm{g}_{t}(\bm{x})=P_{t}(Y|X=\bm{x})$ and
$\bm{f}(\bm{x};\bm{w}_{r},\bm{\theta},\bm{r}_{d})$ takes the form in Theorem
1:

$\displaystyle\bm{f}(\bm{x};\bm{w}_{r},\bm{\theta},\bm{r}_{d})\propto\exp\left(\left(\frac{\tau_{s}(\bm{x},\bm{w}_{d})}{\tau_{t}(\bm{x},\bm{w}_{d})}\bm{\theta}\cdot\bm{\phi}(\bm{x},\bm{w}_{r})+r\mathbb{I}(y)\right)/(r\mathbb{I}(y)+1)\right)$
(9)

Parameter learning: Note that $\bm{g}_{t}$ in the first loss term of Eq. (8)
concerns the conditional label distribution on target which is assumed to be
not available. However, with the deep DRL formulation, we can compute and
evaluate the gradients of $\theta$ and $\phi(\bm{x},\bm{w}_{r})$ directly with
the change of measure in the derivation of the gradients [20], since the
gradients are only associated with the source data and labels. We show this in
Figure 1(a) that the representation learning network $\phi$ only has source
data as the input. We then update $\theta$ and $\bm{w}_{r}$ using the computed
gradients and also directly back-propagate from the second loss term to update
$\bm{w}_{d}$. Finally, we treat the densities as trainable variables and
derive gradients for them from the first loss term. According to the Bayes
rule, $\frac{P_{s}(\bm{x})}{P_{t}(\bm{x})}=\frac{\tau_{s}P(t)}{\tau_{t}P(s)}$,
where $P(t)$ and $P(s)$ are the amount of unlabeled data from each of the
domain during the training process. Since we usually use the same amount of
source and target data in each batch, they cancel out by following
$\frac{P(s)}{P(t)}=1$. Then $\bm{f}$ in Theorem 1 is reduced to Eq. 9.
Therefore, in addition to the binary classification loss, parameter
$\bm{w}_{d}$ of the discriminative network is also trained with gradients from
the first loss term. We summarize the procedure in Algorithm 1, and refer to
the appendix for more details.

Algorithm 1 End-to-end Training for DRL 1: Input: Source data, target data, DNN $\bm{\phi}$, DNN $\tau$, SGD optimizer SGD1, SGD2, learning rate $\gamma_{1}$ and $\gamma_{2}$, epoch number $T$.  2: Initialization: $\phi,\tau\leftarrow$ random initialization, epoch $\leftarrow 0$ 3: While epoch $<T$ 4: For each mini-batch  5: Update $\tau$ by SGD${}_{1}(\gamma_{1})$ using the combined gradients from both loss terms;  6: Compute $\bm{f}$ using $\theta$, $\bm{w}_{r}$, and $\bm{w}_{d}$;  7: Update $\phi,\theta$ by SGD${}_{2}(\gamma_{2})$ using derived gradients;  8: epoch $\leftarrow$ epoch $+1$ 9: Output: Trained networks $\bm{\phi}$, $\tau$. |  Figure 2: Formulation of pseudo label based UDA or SSL methods with deep DRL. The unsupervised losses represent the loss function user can impose on the unlabeled target data. DRST conducts this procedure for multiple iterations, while DRSSL minimizes the unsupervised losses on the augmented target data.
---|---

###  2.4 Applications to UDA and SSL

We show when and how deep DRL can be end-to-end incorporated with framework
that takes unlabeled data for training. We first introduce the general setting
with two specific examples: self-training UDA and cross-domain SSL, followed
by a new self-training algorithm DRST and a new SSL algorithm DRSSL.

General settings: In many cases, the source domain where models are trained
may have abundant labels but the target domain lacks enough labels. Typical
problems under this setting include unsupervised domain adaptation and [5] and
semi-supervised learning [6]. In both cases, a common strategy is to treat the
prediction results on the target data as pseudo labels to train the model on
target data. In these methods, model confidence (softmax output) is often
leveraged as the proxy to rank the reliability of pseudo labels, with the
underlying assumption that there exists a positive correlation between model
confidence and pseudo label quality. However, such assumption requires
accurate uncertainty estimation to avoid false usage of wrong pseudo labels
which may poison the training. To this end, we incorporate deep DRL into these
frameworks with to provide calibrated uncertainties. An illustration of this
setting is shown in Figure 2.

Distributionally robust self-training: In UDA, we are given labeled source
data and unlabeled target data and aim to achieve adaptation from the source
to the target domain. Self-training is an effective method for UDA [5], where
the training procedure in Figure 2 is conducted multiple times. Here we
propose DRST to plug the class-regularized deep DRL model into self-training.
The idea is to regard each training epoch as a new domain shift problem in
deep DRL. After each training epoch, we make predictions on the target domain
and select more confident data and generate pseudo-labels for them to merge
into the source training data. Then the labeled source data and the newly
pseudo-labeled target data become the new source set for the next training
epoch.

Distributionally robust semi-supervised learning: In cross domain SSL, we have
few labeled source data and much unlabeled target data. We aim to utilize
unlabeled data in the target domain to help representation learning and save
the effort needed for labeling. One simple and effective strategy is to use
pseudo-labels generated from weakly-augmented data to supervise strongly-
augmented data [6]. Here “weakly" means simple flip-and-shift data
augmentation while “strongly" follows the same strategy with [6]. Using deep
DRL’s prediction, we propose DRSSL, which assigns pseudo-labels more
conservatively. We plug deep DRL into Fixmatch [6]. Here the unsupervised loss
in Figure 2 is the “consistency loss" in [6]:
$\mathcal{L}_{u}=\frac{1}{M}\sum_{m=1}^{M}\mathbb{I}(\max(\hat{P}(y^{w}_{t}|x^{w}_{t}))>\tau)H(\hat{y}^{w}_{t},\hat{P}(y^{s}_{t}|x^{s}_{t}))$,
where $x_{t}^{w}$ and $y_{t}^{w}$ represent the weakly-augmented target data,
$x_{t}^{s}$ and $y_{t}^{s}$ represent the strongly-augmented version of the
same image data, and $\tau$ is a threshold for generating pseudo-labels
$\hat{y}^{w}_{t}$.

##  3 Experiments

In this section, we evaluate the performance of our method on benchmark
datasets. Deep DRL is evaluated as a method providing more calibrated
uncertainties (Sec. 3.1), DRST as an effective UDA method (Sec. 3.2) and DRSSL
as a cross domain SSL method. We show additional results and experimental
details in the appendix.

Datasets and methods: We use Office31 [24], Office-Home [25] and VisDA [18]
for evaluating deep DRL’s uncertainties. We compare deep DRL with temperature
scaling (TS) and source-only. We also train models using ImageNet [26] as the
source domain and ImageNetV2 [27] as the target domain to check the
relationship between our density ratios and the human selection frequencies
(HSF). VisDA2017 is used to evaluate DRST, for which we compare with (1)
traditional UDA baselines: MMD [28], MCD [29] and ADR [30]; (2) recent self-
training UDA baselines: CBST [31] and CRST [5]; (3) other uncertainty
quantification or UDA methods combined with self-training baselines: AVH [32]
\+ CBST and DeepCORAL[33]+CBST. In addition, we use CIFAR10, STL10 [34], MNIST
[35] and SVHN [36] to construct settings with few source labeled data and much
unlabeled target data and show DRSSL’s advantages in cross domain SSL over
Fixmatch [6].

Evaluation metrics: Apart from accuracy, we also use Brier score [37],
expected calibration error (ECE) [8], and reliability plot [38, 8, 39] to
evaluate the performance of our proposed method and the baselines. Brier score
measures the mean squared difference between the predicted probability
assigned to the possible outcome and the actual outcome. Despite potential
problems with ECE [40], it is still the most prevelant metric for the top-1
prediction. ECE is defined as the sum of average difference between prediction
accuracy and confidence of different confidence bins (we use 5 bins in
practice). For both Brier score and ECE, the lower the score, the better the
model is calibrated.

Table 1: ECE scores on ImageNetV2

HSF | Source | TS | DRL
---|---|---|---
[0.0, 0.2] | 0.4363 | 0.2924 | 0.0147
[0.2, 0.4] | 0.4257 | 0.2487 | 0.0144
[0.4, 0.6] | 0.4526 | 0.2172 | 0.0140
[0.6, 0.8] | 0.3987 | 0.1535 | 0.0139
[0.8, 1.0] | 0.3302 | 0.0939 | 0.0136

Experiment setup: For Office31 and Office-Home tasks, we use ResNet50 [41] as
the backbone for all models to make comparison fair. We train by SGD for $100$
epochs and set the learning rate to $0.001$. For VisDA, we use the ResNet101
[41] backbone and also the SGD optimizer. During the $20$ epochs of training,
the initial learning rate is set as $10^{-5}$ and the weight decay parameter
is set as $5\times 10^{-4}$. For ImageNet, we follow the standard training
process of AlexNet [42] and VGG-19 [43], where the initial lerning rate is
$0.01$ and we decay the learning rate by a factor of 10 every 30 epochs.

###  3.1 Calibrated uncertainties from DRL

---
(a) Office31 (A$\rightarrow$W) (b) Office-Home (P$\rightarrow$A) (c) VisDA
Figure 3: Brier score (upper) and reliability diagrams (lower) on Office31,
Office-Home and
VisDA. DRL generates more calibrated uncertainties than source-only and
temperature scaling.
We adopt the same diagram plotting protocol as in [8] and [44].

DRL’s calibrated confidence: Fig. 3 demonstrates our more calibrated
uncertainty. Deep DRL tends to be underconfident and conservative but still
stays closer to the calibration line (dashed line). Table 3 also shows our
better ECE on ImageNetV2 in different human selection frequency (HSF) bins.
Our additional results in appendix show that the accuracy of deep DRL is also
competitive.

Figure 4: Density ratios v.s. HSF.

Density ratio v.s. human selection frequency: Fig. 4 shows that our density
ratio matches the human selection frequency (HSF) in ImageNetV2 [45] under
different neural network architectures. Here we regard [$0,0.2$] as low HSF
and $(0.2,1]$ as high HSF. Images with low HSF have smaller density ratios
($P_{s}(\bm{x})/P_{t}(\bm{x})$), indicating that they are visually harder for
human to select compared with other images.

###  3.2 Unsupervised domain adaptation

Initialization with ASG: The quality of the source model has a great impact on
the final self-training performance. It is recently proposed that automated
synthetic-to-real generalization (ASG) using ImageNet pretrained knowledge can
be used to improve synthetic training through a distillation loss [46].
Initializing with ASG trained model leads to significant improvements in self-
training. Hence, we take the ASG pretrained model as one type of
initialization.

Accuracy and calibration: Fig. 5(a) and Table 2 demonstrate that DRST
outperforms all the baselines in accuracy. Our vanilla version of DRST
outperforms CRST by more than 5% and with ASG initialization, we improve the
SOTA self-training accuracy on VisDA by more than 1%. Note that in the rest of
the results, we directly use DRST to represent DRST-ASG. Fig. 5(b)
demonstrates that DRST also achieves more calibrated confidence.

Table 2: Accuracy comparison with different methods on VisDA2017. “Skate”
denotes “Skateboard”.

Method | Aero | Bike | Bus | Car | Horse | Knife | Motor | Person | Plant | Skate | Train | Truck | Mean
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Source [30] | 55.1 | 53.3 | 61.9 | 59.1 | 80.6 | 17.9 | 79.7 | 31.2 | 81.0 | 26.5 | 73.5 | 8.5 | 52.4
MMD [28] | 87.1 | 63.0 | 76.5 | 42.0 | 90.3 | 42.9 | 85.9 | 53.1 | 49.7 | 36.3 | 85.8 | 20.7 | 61.1
MCD [29] | 87.0 | 60.9 | 83.7 | 64.0 | 88.9 | 79.6 | 84.7 | 76.9 | 88.6 | 40.3 | 83.0 | 25.8 | 71.9
ADR [30] | 87.8 | 79.5 | 83.7 | 65.3 | 92.3 | 61.8 | 88.9 | 73.2 | 87.8 | 60.0 | 85.5 | 32.3 | 74.8
CBST [31] | 87.2 | 78.8 | 56.5 | 55.4 | 85.1 | 79.2 | 83.8 | 77.7 | 82.8 | 88.8 | 69.0 | 72.0 | 76.4
CRST [5] | 88.0 | 79.2 | 61.0 | 60.0 | 87.5 | 81.4 | 86.3 | 78.8 | 85.6 | 86.6 | 73.9 | 68.8 | 78.1
CBST-AVH [32] | 93.3 | 80.2 | 78.9 | 60.9 | 88.4 | 89.7 | 88.9 | 79.6 | 89.5 | 86.8 | 81.5 | 60.0 | 81.5
CBST-DCORAL [33] | 92.1 | 78.9 | 83.0 | 73.6 | 93.2 | 94.7 | 89.0 | 83.0 | 89.8 | 81.2 | 85.5 | 44.9 | 82.4
DRST (ours) | 93.47 | 86.30 | 65.74 | 68.03 | 93.99 | 95.08 | 87.34 | 83.30 | 92.97 | 88.65 | 83.66 | 66.42 | 83.75
ASG [46] | 88.81 | 68.55 | 65.31 | 78.06 | 95.78 | 9.11 | 84.89 | 29.58 | 82.13 | 33.76 | 86.00 | 12.04 | 61.17
CBST-ASG [46] | 95.12 | 86.53 | 79.83 | 76.01 | 94.61 | 92.34 | 85.94 | 75.08 | 89.23 | 82.16 | 73.42 | 56.49 | 82.23
CRST-ASG [46] | 92.38 | 81.30 | 74.63 | 84.40 | 90.90 | 92.43 | 91.65 | 83.78 | 94.92 | 88.12 | 74.88 | 61.10 | 84.21
DRST-ASG (ours) | 94.51 | 85.58 | 76.50 | 77.18 | 94.39 | 95.33 | 88.89 | 81.23 | 94.22 | 90.36 | 81.75 | 63.10 | 85.25

Ablation study: Fig. 5(a)(b) include two ablation methods. In the first
ablation, we set $r$ to 0 so that there is no class regularization in DRL (“r
= 0”). The prediction then follows the form in Eq. (1). In the second
ablation, we set density ratio to 1 to mute the differentiable density ratio
estimation in our method so that there is no representation level
conservativeness (“R = 1”). DRST achieves the best performance when both
components are present.

|  |  |
---|---|---|---
(a) Test accuracy | (b) Brier score | (c) Covariate shift | (d) Harder cases
Figure 5: (a)-(b) Main results on VisDA-17 (5 random seeds) with the test
accuracy and Brier score. The results show that DRST outperforms the baselines
significantly. (c) Distribution gap
$P_{s}(\bm{\phi}(\bm{x}))/P_{t}(\bm{\phi}(\bm{x}))-P_{s}(\bm{\phi}(\bm{x}),y)/P_{t}(\bm{\phi}(\bm{x}),y)$
as a proxy of covariate shift. DRST helps to further reduce this gap over DRL
with self-training. (d) Accuracy vs. density ratios. The improvement from DRST
increases on harder examples (lower density ratios).

Covariate shift: Fig. 5(c) shows
$P_{s}(\bm{\phi}(\bm{x}))/P_{t}(\bm{\phi}(\bm{x}))-P_{s}(\bm{\phi}(\bm{x}),\bm{y})/P_{t}(\bm{\phi}(\bm{x}),\bm{y})$
using discriminative density ratio estimators (per class). This serves as a
proxy of covariate shift as it becomes 0 when covariate shift holds. We can
see that the gap decreases with self-training, which shows that even though
covariate shift may not hold in the beginning, self-training helps to promote
this assumption with better aligned domains and more discriminative feature
distributions.

Improvement on hard examples: Fig. 5 (d) demonstrates that DRST achieves
larger performance gain compared to the baselines on target samples with
smaller density ratios. Recall that data is not well-represented in the source
domain when $P_{\text{s}}(\bm{x})/P_{\text{t}}(\bm{x})$ is small (Fig. 1(b)).
Therefore, DRST provides robust performance on harder examples in the target
domain.

Density ratios: In Fig. 1(b), a more typical “train” image obtains higher
density ratio than a “train” on a busy platform, whose shape is not obvious.
DRL is able to give higher confidence to images that are better-represented in
the synthetic training domain. More examples are given in the appendix.

Improved attention:  We visualize the network attention of DRST and the
baselines in Fig. 6 using Grad-CAM [47], where DRST is shown to render
improved attention with better object coverage.

---
Figure 6: Model attention visualized using Grad-Cam [47]. We also show the
predicted labels by different methods. Our method captures the shape features
of the image better.

Table 3: Accuracies for cross domain SSL.

Test set | CIFAR10 | STL10 | MNIST | SVHN
---|---|---|---|---
Fixmatch [6] | 91.60 | 59.61 | 99.43 | 26.50
DRSSL | 95.17 | 69.38 | 99.46 | 30.96

###  3.3 Distributionally robust semi-supervised Learning

We use two pairs of source and target domains in this experiment: Source:
CIFAR10 / Target: STL10 and Source: MNIST / Target: SVHN. Table 3.2
demonstrates that DRL-powered SSL method improves the original Fixmatch
significantly in cross-domain semi-supervised learning tasks. Note our setting
is different from UDA where source labeled data is abundant. SSL focuses on
using unlabeled augmented data to help learning from few labeled data. In the
CIFAR10 to STL10 case, we only have 4k labeled source data. And in the MNIST
to SVHN case, we have 40k labeled source data. Our results provide evidence
that DRL is beneficial for generating high-quality pseudo supervision for
unlabeled data under the cross domain SSL setting.

##  4 Related work

Distributionally robust learning: Different learning algorithms can be derived
from this framework with different loss functions and feature constraints [20,
48, 49, 50, 51]. One limitation of this line of work is that it is usually
only suitable for low-dimensional data. Another inconvenience is that it
requires the density ratio to be estimated beforehand. Instead of having the
adversary as the conditional labeling function, an alternative is to perturb
the covariate variable [52, 53, 54, 55, 56]. This line of work has been mostly
focusing on model robustness against adversarial perturbations. Instead, in
this work, we focus on the benefits of DRL’s uncertainty estimation for
downstream tasks like UDA and SSL. We apply neural networks and propose end-
to-end learning method to handle high-dimensional image data.

UDA and SSL: There is a rich literature in UDA focusing on representation
alignment between the source and target domains [57, 58, 59, 60, 61, 62]. It
is also prevalent to use a discriminator network to differentiate two learned
representations (adversarially) to locate such a feature space [63, 64, 65,
66]. Another line of work focuses on more specific shift assumptions, such as
covariate shift [67] and label shift [68, 69]. For the covariate shift case,
even though various density ratio estimation methods were investigated before
[22], only few works explore the possibility to apply them to high-dimensional
data [70, 71, 72]. Recently, multiple self-training methods have been proposed
for UDA [31, 5]. It is also helpful to use a query selection strategy based on
angular visual hardness [32]. More theoretical understanding of self-training
are also developed [7, 73]. Similarly, rich literature also exist in pseudo-
labeling for SSL [74, 75, 76, 6]. In this paper, we incorporate ideas from
CRST to smooth DRL’s model uncertainty and demonstrate that DRL’s model
confidence help with choosing better pseudo-labeled data from the target
domain. We also integrate DRL’s model uncertainty to generate the pseudo-
labels for the consistency loss in Fixmatch [6] to provide better supervision
for unlabeled data.

Uncertainty quantification with deep models: Bayesian methods largely
contribute to complementing deep learning with uncertainty quantification.
MCMC [77, 78, 79], (stochastic) variational inference for deep learning [80,
81, 13, 14], dropout variational inference [82, 83, 1], and SGD-based
approximations [84] aim to approximate the Bayesian inference from different
perspectives in the deep learning setting. However, with a large number of
parameters in the neural network, Bayesian deep learning may suffer from
computational inefficiency. Other calibration methods include temperature
scaling [12, 85], deep ensembles [86], calibration regression [87], quantile
regression [88] and trainable calibration measure [89]. These methods are not
applicable to our domain shift setting. In this paper, we utilize the model
confidence from the DRL framework and introduce the domain classifier for
differentiating source and target domains, which can be fulfilled by end-to-
end learning.

##  5 Conclusion and future work

In this paper, we study uncertainty estimation under distribution shift using
the distributionally robust learning framework. We demonstrate that density
estimation can be integrated into the learning process by using a domain
classifier. We propose differentiable density ratio estimation and develop
end-to-end training techniques for our proposed method. Using DRL’s more
calibrated model confidence helps to solicit better pseudo-labels for self-
training in UDA and cross domain SSL. We also empirically demonstrate that the
density ratios learned from our domain classifier reflect the hardness of an
image accurately, matching the human selection frequencies. In the future, we
are interested in relaxing the assumptions made and study different shifts in
the DRL framework.

## Broader Impact

Societal impacts: This paper do not explicitly study any fairness issues
within our method. We also do not study safety-critical applications. So when
the data is human-related or the task is safety-critical, our algorithm should
be used with extra care.

Density ratio estimation: Our density ratios are estimated from a
differetiable domain classifier and is not guaranteed to match the true
density ratios. Thus our density ratios should not be used in circumstances
where true density ratios are absolutely required. However, we show that our
density ratios do reflect the relation between the two distributions and
benefit the downstream tasks.

Covariate shift assumption: Covariate shift is a relatively strong assumption
in real-world applications. However, this can be partially remediated with
stronger and more compositional features $\bm{\phi}(\bm{x})$. We analyzed how
well this assumption holds in Fig. 5(c). In practice, users are encouraged to
judge whether the assumption holds to see whether the proposed method is
applicable.

## References

  * Kendall and Gal [2017] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  In _NIPS_ , 2017.
  * Antifakos et al. [2005a] Stavros Antifakos, Nicky Kern, Bernt Schiele, and Adrian Schwaninger.  Towards improving trust in context-aware systems by displaying system confidence.  In _International Conference on Human Computer Interaction with Mobile Devices and Services_ , 2005a.
  * Tomsett et al. [2020] Richard Tomsett, Alun Preece, Dave Braines, Federico Cerutti, Supriyo Chakraborty, Mani Srivastava, Gavin Pearson, and Lance Kaplan.  Rapid trust calibration through interpretable and uncertainty-aware ai.  _Patterns_ , 2020.
  * Antifakos et al. [2005b] Stavros Antifakos, Nicky Kern, Bernt Schiele, and Adrian Schwaninger.  Towards improving trust in context-aware systems by displaying system confidence.  In _International Conference on Human Computer Interaction with Mobile Devices and Services_ , 2005b.
  * Zou et al. [2019] Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang.  Confidence regularized self-training.  In _ICCV_ , 2019.
  * Sohn et al. [2020] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  In _NeurIPS_ , 2020.
  * Kumar et al. [2020] Ananya Kumar, Tengyu Ma, and Percy Liang.  Understanding self-training for gradual domain adaptation.  In _ICML_ , 2020.
  * Guo et al. [2017a] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.  On calibration of modern neural networks.  In _ICML_ , 2017a.
  * Gal and Ghahramani [2016] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _ICML_ , 2016.
  * Li and Hoiem [2020] Zhizhong Li and Derek Hoiem.  Improving confidence estimates for unfamiliar examples.  In _CVPR_ , 2020.
  * Szegedy et al. [2016] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _CVPR_ , 2016.
  * Platt et al. [1999] John Platt et al.  Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.  _Advances in large margin classifiers_ , 1999.
  * Blundell et al. [2015] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.  Weight uncertainty in neural network.  In _ICML_ , 2015.
  * Riquelme et al. [2018] Carlos Riquelme, George Tucker, and Jasper Snoek.  Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling.  _arXiv preprint arXiv:1802.09127_ , 2018.
  * Snoek et al. [2019] Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, D Sculley, Joshua Dillon, Jie Ren, and Zachary Nado.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  In _NeurIPS_ , 2019.
  * Liu et al. [2020] Anqi Liu, Guanya Shi, Soon-Jo Chung, Anima Anandkumar, and Yisong Yue.  Robust regression for safe exploration in control.  In _Learning for Dynamics and Control_ , 2020.
  * Nakka et al. [2020] Yashwanth Kumar Nakka, Anqi Liu, Guanya Shi, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung.  Chance-constrained trajectory optimization for safe exploration and learning of nonlinear systems.  _IEEE Robotics and Automation Letters_ , 2020.
  * Peng et al. [2017] Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko.  Visda: The visual domain adaptation challenge, 2017.
  * Grünwald et al. [2004] Peter D Grünwald, A Philip Dawid, et al.  Game theory, maximum entropy, minimum discrepancy and robust bayesian decision theory.  _Annals of statistics_ , 2004.
  * Liu and Ziebart [2014] Anqi Liu and Brian Ziebart.  Robust classification under sample selection bias.  In _NIPS_ , 2014.
  * Pereyra et al. [2017] Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton.  Regularizing neural networks by penalizing confident output distributions.  _arXiv preprint arXiv:1701.06548_ , 2017.
  * Sugiyama et al. [2012] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori.  _Density ratio estimation in machine learning_.  Cambridge University Press, 2012.
  * Bickel et al. [2007] Steffen Bickel, Michael Brückner, and Tobias Scheffer.  Discriminative learning for differing training and test distributions.  In _ICML_ , 2007.
  * Saenko et al. [2010] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell.  Adapting visual category models to new domains.  In _ECCV_. Springer, 2010.
  * Venkateswara et al. [2017] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In _CVPR_ , 2017.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _CVPR_ , 2009.
  * Recht et al. [2019a] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar.  Do imagenet classifiers generalize to imagenet?  2019a.
  * Long et al. [2015] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan.  Learning transferable features with deep adaptation networks.  In _ICML_ , 2015.
  * Saito et al. [2018a] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada.  Maximum classifier discrepancy for unsupervised domain adaptation.  In _CVPR_ , 2018a.
  * Saito et al. [2018b] Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko.  Adversarial dropout regularization.  In _ICLR_ , 2018b.
  * Zou et al. [2018] Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang.  Unsupervised domain adaptation for semantic segmentation via class-balanced self-training.  In _ECCV_ , 2018.
  * Chen et al. [2020a] Beidi Chen, Weiyang Liu, Zhiding Yu, Jan Kautz, Anshumali Shrivastava, Animesh Garg, and Anima Anandkumar.  Angular visual hardness.  In _ICML_ , 2020a.
  * Sun and Saenko [2016] Baochen Sun and Kate Saenko.  Deep coral: Correlation alignment for deep domain adaptation.  In _ECCV Workshops_ , 2016.
  * Coates et al. [2011] Adam Coates, Andrew Ng, and Honglak Lee.  An analysis of single-layer networks in unsupervised feature learning.  In _AISTATS_ , 2011.
  * Lecun and Bottou [1998] Y. Lecun and L. Bottou.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 1998.
  * svh [2011] Reading digits in natural images with unsupervised feature learning.  _NIPS workshop on deep learning and unsupervised feature learning_ , 2011.
  * Brier [1950] Glenn W Brier.  Verification of forecasts expressed in terms of probability.  _Monthly weather review_ , 1950.
  * Dawid [1982] A Philip Dawid.  The well-calibrated bayesian.  _Journal of the American Statistical Association_ , 1982.
  * Naeini et al. [2015] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht.  Obtaining well calibrated probabilities using bayesian binning.  In _AAAI_ , 2015.
  * Nixon et al. [2019] Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran.  Measuring calibration in deep learning.  In _CVPR Workshops_ , 2019.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _NIPS_ , 2012.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Maddox et al. [2019] Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson.  A simple baseline for bayesian uncertainty in deep learning.  _NeurIPS_ , 2019.
  * Recht et al. [2019b] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In _ICML_ , 2019b.
  * Chen et al. [2020b] Wuyang Chen, Zhiding Yu, Zhangyang Wang, and Animashree Anandkumar.  Automated synthetic-to-real generalization.  In _ICML_ , 2020b.
  * Selvaraju et al. [2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-CAM: Visual explanations from deep networks via gradient-based localization.  In _ICCV_ , 2017.
  * Liu and Ziebart [2017] Anqi Liu and Brian D Ziebart.  Robust covariate shift prediction with general losses and feature views.  _arXiv preprint arXiv:1712.10043_ , 2017.
  * Chen et al. [2016] Xiangli Chen, Mathew Monfort, Anqi Liu, and Brian D Ziebart.  Robust covariate shift regression.  In _AISTATS_ , 2016.
  * Fathony et al. [2016] Rizal Fathony, Anqi Liu, Kaiser Asif, and Brian Ziebart.  Adversarial multiclass classification: A risk minimization perspective.  In _NIPS_ , 2016.
  * Fathony et al. [2018] Rizal Fathony, Kaiser Asif, Anqi Liu, Mohammad Ali Bashiri, Wei Xing, Sima Behpour, Xinhua Zhang, and Brian D Ziebart.  Consistent robust adversarial prediction for general multiclass classification.  _arXiv preprint arXiv:1812.07526_ , 2018.
  * Ben-Tal et al. [2013] Aharon Ben-Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen.  Robust solutions of optimization problems affected by uncertain probabilities.  _Management Science_ , 2013.
  * Shafieezadeh Abadeh et al. [2015] Soroosh Shafieezadeh Abadeh, Peyman M Mohajerin Esfahani, and Daniel Kuhn.  Distributionally robust logistic regression.  _NIPS_ , 2015.
  * Sinha et al. [2018] Aman Sinha, Hongseok Namkoong, Riccardo Volpi, and John Duchi.  Certifying some distributional robustness with principled adversarial training.  2018\.
  * Hu et al. [2018] Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama.  Does distributionally robust supervised learning give robust classifiers?  In _ICML_ , 2018.
  * Najafi et al. [2019] Amir Najafi, Shin-ichi Maeda, Masanori Koyama, and Takeru Miyato.  Robustness to adversarial perturbations in learning from incomplete data.  In _NeurIPS_ , 2019.
  * Fernando et al. [2014] Basura Fernando, Amaury Habrard, Marc Sebban, and Tinne Tuytelaars.  Subspace alignment for domain adaptation.  _arXiv preprint arXiv:1409.5241_ , 2014.
  * Pan et al. [2010] Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang.  Domain adaptation via transfer component analysis.  _IEEE Trans. Neural Networks_ , 2010.
  * Sun et al. [2016] Baochen Sun, Jiashi Feng, and Kate Saenko.  Return of frustratingly easy domain adaptation.  In _AAAI_ , 2016.
  * Wang et al. [2018] Jindong Wang, Wenjie Feng, Yiqiang Chen, Han Yu, Meiyu Huang, and Philip S Yu.  Visual domain adaptation with manifold embedded distribution alignment.  In _Proceedings of the 26th ACM international conference on Multimedia_ , pages 402–410, 2018.
  * Baktashmotlagh et al. [2013] Mahsa Baktashmotlagh, Mehrtash T Harandi, Brian C Lovell, and Mathieu Salzmann.  Unsupervised domain adaptation by domain invariant projection.  In _ICCV_ , 2013.
  * Gretton et al. [2012] Arthur Gretton, Karsten M. Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander J. Smola.  A kernel two-sample test.  2012\.
  * Tzeng et al. [2017] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell.  Adversarial discriminative domain adaptation.  In _CVPR_ , 2017.
  * Ajakan et al. [2014] Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand.  Domain-adversarial neural networks.  _arXiv preprint arXiv:1412.4446_ , 2014.
  * Long et al. [2018] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan.  Conditional adversarial domain adaptation.  In _NeurIPS_ , 2018.
  * Sankaranarayanan et al. [2018] Swami Sankaranarayanan, Yogesh Balaji, Carlos D Castillo, and Rama Chellappa.  Generate to adapt: Aligning domains using generative adversarial networks.  In _CVPR_ , 2018.
  * Shimodaira [2000] Hidetoshi Shimodaira.  Improving predictive inference under covariate shift by weighting the log-likelihood function.  _Journal of statistical planning and inference_ , 2000.
  * Lipton et al. [2018] Zachary Lipton, Yu-Xiang Wang, and Alexander Smola.  Detecting and correcting for label shift with black box predictors.  In _ICML_ , 2018.
  * Azizzadenesheli et al. [2019] Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar.  Regularized learning for domain adaptation under label shifts.  2019\.
  * Khan et al. [2019] Haidar Khan, Lara Marcuse, and Bülent Yener.  Deep density ratio estimation for change point detection.  _arXiv preprint arXiv:1905.09876_ , 2019.
  * Moustakides and Basioti [2019] George V Moustakides and Kalliopi Basioti.  Training neural networks for likelihood/density ratio estimation.  _arXiv preprint arXiv:1911.00405_ , 2019.
  * Park et al. [2020] Sangdon Park, Osbert Bastani, James Weimer, and Insup Lee.  Calibrated prediction with covariate shift via unsupervised domain adaptation.  In _AISTATS_ , 2020.
  * Chen et al. [2020c] Yining Chen, Colin Wei, Ananya Kumar, and Tengyu Ma.  Self-training avoids using spurious features under domain shift.  In _NeurIPS_ , 2020c.
  * Berthelot et al. [2019] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel.  Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring.  _arXiv preprint arXiv:1911.09785_ , 2019.
  * Arazo et al. [2020] Eric Arazo, Diego Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness.  Pseudo-labeling and confirmation bias in deep semi-supervised learning.  In _International Joint Conference on Neural Networks_ , 2020.
  * Miyato et al. [2018] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii.  Virtual adversarial training: a regularization method for supervised and semi-supervised learning.  _IEEE Trans. PAMI_ , 2018.
  * Gal [2016] Yarin Gal.  Uncertainty in deep learning.  2016\.
  * Welling and Teh [2011] Max Welling and Yee W Teh.  Bayesian learning via stochastic gradient langevin dynamics.  In _ICML_ , 2011.
  * Chen et al. [2014] Tianqi Chen, Emily Fox, and Carlos Guestrin.  Stochastic gradient hamiltonian monte carlo.  In _ICML_ , 2014.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Louizos and Welling [2017] Christos Louizos and Max Welling.  Multiplicative normalizing flows for variational bayesian neural networks.  In _ICML_. JMLR. org, 2017.
  * Gal and Ghahramani [2015] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  2015\.
  * Gal et al. [2017] Yarin Gal, Jiri Hron, and Alex Kendall.  Concrete dropout.  In _NIPS_ , 2017.
  * Mandt et al. [2017] Stephan Mandt, Matthew D Hoffman, and David M Blei.  Stochastic gradient descent as approximate bayesian inference.  _JMLR_ , 2017.
  * Guo et al. [2017b] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.  On calibration of modern neural networks.  In _ICML_ , 2017b.
  * Lakshminarayanan et al. [2017] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  In _NIPS_ , 2017.
  * Kuleshov et al. [2018] Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon.  Accurate uncertainties for deep learning using calibrated regression.  In _ICML_ , 2018.
  * Romano et al. [2019] Yaniv Romano, Evan Patterson, and Emmanuel Candes.  Conformalized quantile regression.  In _NeurIPS_ , 2019.
  * Kumar et al. [2018] Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain.  Trainable calibration measures for neural networks from kernel mean embeddings.  In _ICML_ , 2018.

##  Appendix A Simulation on the plug-in estimator for the density ratios

The density ratios are important factors adjusting the representation-level
conservativeness for DRL. Like many other machine learning methods using
density ratios such as, transfer learning 111Sinno Jialin Pan and Qiang Yang.
A survey on transfer learning. IEEE Transactions on knowledge and data
engineering, 22(10):1345–1359, 2009. , and off-policy evaluation 222Miroslav
Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and
learning. arXiv preprint arXiv:1103.4601, 2011. , we can use a plug-in
estimator for the density ratio $P_{\text{s}}(\bm{x})/P_{\text{t}}(\bm{x})$.
However, density ratio estimation 333Masashi Sugiyama, Taiji Suzuki, and
Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge
University Press, 2012. , especially in the high-dimensional data, is rather
difficult. Here, we ask the question: whether a more accurate density ratio
estimate leads to a greater predictive performance in the downstream tasks?

We show a two-dimensional binary classification example in Figure 7 to
demonstrate relation between the performance of the density (ratio) estimation
and the performance of the ultimate target learning tasks. We use the RBA
method 444Anqi Liu and Brian Ziebart. Robust classification under sample
selection bias. In Advances in neural information processing systems, pages
37–45, 2014. as an example. We conduct kernel density estimation (KDE) and
evaluate the average log-likelihood of the source and target data on the
source and target distributions. We take the ratio of the density from KDE and
plug in the RBA method. We can see that the case with higher log-likelihood
actually fails to give informative predictions in RBA. One of the reasons is
that the density (ratio) estimation task, as an independent learning task,
does not share information with the downstream prediction tasks that use the
ratios. Therefore, if the density ratio estimation over-optimize for its own
objective, it may even hurts the performance for the downstream tasks.

|  |  |
---|---|---|---
(a) | (b) | (c) | (d)
Figure 7: (a) Source and target data points are drawn from two Gaussian
distributions. Solid line: source, dashed line: target. The underlying true
decision boundary for the binary classes is the same between the two domains.
(b)-(c) Prediction with density ratios from low and high density estimation
likelihoods. With more accurate density estimation, the RBA predictor gives
overly conservative predictions on the target domain. The colormap is the
confidence $P(``1"|x)$. (d) With larger likelihood in density estimation, the
target log loss becomes worse.

However, does it mean that a good density ratio estimation is not needed? No.
We demonstrate that our density ratios estimated from the deep DRL model match
the relative distance between the two distributions. We further show that it
is matching the human visual hardness. In fact, a good density ratio is very
important for both the model calibration and the predictive performance.
Therefore, we still expect a good density ratio, but not just for its own
sake.

##  Appendix B Proof of Lemma 1 and Theorem 1

Lemma 1 reduce the original minimax problem into a constrained maximum entropy
problem.

###### Proof.

We want to solve the following minimax problem:

$\displaystyle\min_{\bm{F}}\;\max_{\bm{G}\in\Sigma}\;\mathbb{E}_{\bm{x}\sim
P_{\text{t}}(X)}[-\bm{g}(\bm{x})\cdot\log\bm{f}(\bm{x})]-r\mathbb{E}_{\bm{x}\sim
P_{\text{t}}(X)}\left[\bm{y}\odot\bm{g}(\bm{x})\cdot\log\bm{f}(\bm{x})\right]$
(10)

We first observe that the loss function is convex-concave in terms of $\bm{g}$
and $\bm{f}$. Then we can switch the order of the min player and the max play:

$\displaystyle\max_{\bm{G}\in\Sigma}\;\min_{\bm{F}}\;\mathbb{E}_{\bm{x}\sim
P_{\text{t}}(X)}[-\bm{g}(\bm{x})\cdot\log\bm{f}(\bm{x})]-r\mathbb{E}_{\bm{x}\sim
P_{\text{t}}(X)}\left[\bm{y}\odot\bm{g}(\bm{x})\cdot\log\bm{f}(\bm{x})\right]$
(11)

We then first solve the inner problem and observe that the optimal solution
for the inner min problem is $\bm{f}=\bm{g}$. Then we use $\bm{f}$ to replace
$\bm{g}$ in the outer max problem and have the following:

$\displaystyle\max_{\bm{F}\in\Sigma}\;\mathbb{E}_{\bm{x}\sim
P_{\text{t}}(X)}[-\bm{f}(\bm{x})\cdot\log\bm{f}(\bm{x})]-r\mathbb{E}_{\bm{x}\sim
P_{\text{t}}(X)}\left[\bm{y}\odot\bm{f}(\bm{x})\cdot\log\bm{f}(\bm{x})\right]$
(12)

∎

We then prove Theorem 1:

###### Proof.

To solve the constrained maximum entropy problem, we first write out the
constraints $\bm{f}$ needs to satisfy. $\bm{f}$ needs to reside in the
conditional probability simplex and also must satisfy the constraints that
$\forall y\in\mathcal{Y}$:

$\displaystyle{\textstyle\sum}_{i}f_{y}\bm{\phi}(\bm{x}_{i})={\textstyle\sum}_{i}\mathbb{I}[y_{i}=y]\bm{\phi}(\bm{x}_{i}),$
(13)

where $\bm{x}\sim P_{s}(\bm{x})$. So before approximating using the finite
samples, the constraint can be written as $\forall y\in\mathcal{Y}$:

$\displaystyle\sum_{\bm{x}\in\mathcal{X}}P_{s}(\bm{x})f_{y}(\bm{x})\bm{\phi}(\bm{x})=\sum_{\bm{x}\in\mathcal{X}}P_{s}(\bm{x})\bm{g}^{y}_{s}(\bm{x})\bm{\phi}(\bm{x}),$
(14)

where $\bm{g}^{y}_{s}(\bm{x})$ denotes the ground-truth conditional
distribution in the source data distribution. By the definition of covariate
shift, we have $\bm{g}_{s}(\bm{x})=\bm{g}_{t}(\bm{x})$. The right-hand-side is
just a fixed vector given the source training data. Therefore, we just use a
$\tilde{\bm{c}}_{y}$ to represent it for each $y$. The constrained
optimization problem then can be written as:

$\displaystyle\max_{\bm{F}}-\sum_{\bm{x}\in\mathcal{X}}P_{\text{t}}(\bm{x})\sum_{y\in\mathcal{Y}}f_{y}(\bm{x})(1+r\mathbb{I}(y))\log\bm{f}(\bm{x})$
$\displaystyle\text{such that: }\forall
y\in\mathcal{Y}:\sum_{\bm{x}\in\mathcal{X}}{P_{\text{s}}}(\bm{x})[f_{y}(\bm{x})\bm{\phi}(\bm{x})]=\tilde{\bm{c}}$
$\displaystyle\qquad\qquad\text{$\forall\bm{x}\in\mathcal{X}$:
}\sum_{y\in\mathcal{Y}}f_{y}(\bm{x})=1$
$\displaystyle\qquad\qquad\text{$\forall\bm{x}\in\mathcal{X},y\in\mathcal{Y}$:
}f_{y}(\bm{x})\geq 0.$

Note that the final constraint is superfluous since the domain of the
objective function is the positive real numbers. The Lagrangian associated
with this problem is:

$\displaystyle\mathcal{L}(\theta,\lambda)=$
$\displaystyle-\sum_{\bm{x}\in\mathcal{X}}P_{\text{t}}(\bm{x})\sum_{y\in\mathcal{Y}}f_{y}(\bm{x})(1+r\mathbb{I}(y))\log
f_{y}(\bm{x})+$ (15)
$\displaystyle\cdot\left[\sum_{y\in\mathcal{Y}}\bm{\theta}_{y}\cdot\sum_{\bm{x}\in\mathcal{X}}{P_{\text{s}}}(x)f_{y}(\bm{x})\bm{\phi}(\bm{x})-\tilde{\bm{c}}_{y}\right]+\sum_{\bm{x}\in\mathcal{X}}\lambda(x)\left[\sum_{y\in\mathcal{Y}}f_{y}(\bm{x})-1\right],$

where $\theta$ and $\lambda(\bm{x})$ are the Lagrangian multipliers. Taking
the derivative with respect to a specific $f_{y}(\bm{x})$,

$\displaystyle\frac{\partial}{\partial
f_{y}(\bm{x})}\mathcal{L}(\theta,\lambda)$ $\displaystyle=$ $\displaystyle-
P_{\text{t}}(\bm{x})(1+r\mathbb{I}(y))\log
f_{y}(\bm{x})-P_{\text{t}}(\bm{x})+\bm{\theta}_{y}\cdot{P_{\text{s}}}(\bm{x})\bm{\phi}(\bm{x})+\lambda(x),$

setting equal to zero,
$\frac{\partial}{\partial\bm{f}(\bm{x})}\mathcal{L}(\theta,\lambda)=0$, and
solving, we get:

$\displaystyle(1+r\mathbb{I}(y))\log f_{y}(\bm{x}))$ $\displaystyle=$
$\displaystyle-1+\bm{\theta}_{y}\cdot\frac{{P_{\text{s}}}(\bm{x})}{P_{\text{t}}(\bm{x})}\bm{\phi}(\bm{x})+\frac{\lambda(\bm{x})}{P_{\text{t}}(\bm{x})}+\frac{(1+r\mathbb{I}(y))}{P_{\text{t}}(\bm{x})}.$

Therefore, we conclude:

$\displaystyle f_{y}(\bm{x})\propto
e^{\frac{\bm{\theta}_{y}\cdot\frac{{P_{\text{s}}}(\bm{x})}{P_{\text{t}}(\bm{x})}\bm{\phi}(\bm{x})+r\mathbb{I}(y)}{1+r\mathbb{I}(y)}},$

The derivation of gradient then resembles the derivation in Theorem 1 in
555Anqi Liu and Brian Ziebart. Robust classification under sample selection
bias. In Advances in neural information processing systems, pages 37–45,
2014.. ∎

---
A$\rightarrow$W A$\rightarrow$D W$\rightarrow$A
W$\rightarrow$D D$\rightarrow$A D$\rightarrow$W
A$\rightarrow$W A$\rightarrow$D W$\rightarrow$A
W$\rightarrow$D D$\rightarrow$A D$\rightarrow$W
Figure 8: Accuracy and brier score comparison of DRL with source-only and LL-
SVI on Office31.

##  Appendix C Gradients of the Densities

As mentioned in the paper, the gradients of the densities come from both of
the two loss terms. Let’s define the first term
$\mathcal{L}_{1}\triangleq\mathbb{E}_{\bm{x}\sim
P_{\text{t}}(X)}\left[-\bm{g}_{t}(\bm{x})\cdot\log\bm{f}(\bm{x};\bm{w}_{r},\bm{\theta},\bm{w}_{d})\right]$
and the second loss term $\mathcal{L}_{2}\triangleq\mathbb{E}_{\bm{x}\sim
P_{d}(X)}\left[-\bm{d}(\bm{x})\cdot\log\bm{\tau}(\bm{x},\bm{w}_{d})\right]$.
The gradients of $\mathcal{L}_{2}$ with respect to $\bm{\tau}$is obvious. Here
we only discuss the gradients of $\mathcal{L}_{1}$ with respect to
$\bm{\tau}$. As mentioned in the paper, $\bm{\tau}$ is a two-dimensional
vector $(\tau_{s},\tau_{t})$. We give the gradient for each of them
respectively.

We first show that the first loss term $\mathcal{L}_{1}$ is equivalent to
plugging $\bm{f}$ into the Lagrange $\mathcal{L}(\theta,\lambda)$. Here, for
simplicity, we use Eq. (1) as the $\bm{f}(\bm{x})$.

$\displaystyle\mathcal{L}(\theta,\lambda)=$
$\displaystyle-\sum_{\bm{x}\in\mathcal{X}}P_{\text{t}}(\bm{x})\sum_{y\in\mathcal{Y}}f_{y}(\bm{x})\cdot\left(\bm{\theta}_{y}\cdot\frac{P_{\text{s}}(\bm{x})}{P_{\text{t}}(\bm{x})}\bm{\phi}(\bm{x})-\log
Z_{\bm{\theta}}(\bm{x})\right)$
$\displaystyle+\left[\sum_{y\in\mathcal{Y}}\bm{\theta}_{y}\cdot\sum_{\bm{x}\in\mathcal{X}}P_{\text{s}}(\bm{x})f_{y}(\bm{x})\bm{\phi}(\bm{x})-\tilde{\bm{c}}_{y}\right]$
$\displaystyle=$
$\displaystyle\sum_{\bm{x}\in\mathcal{X}}P_{\text{t}}(\bm{x})\log
Z_{\bm{\theta}}(\bm{x})-\sum_{y\in\mathcal{Y}}\bm{\theta}_{y}\cdot\tilde{{\bf
c}}_{y}.$

On the other hand,

$\displaystyle\mathcal{L}_{1}$ $\displaystyle=\mathbb{E}_{\bm{x}\sim
P_{\text{t}}(X)}\left[-\bm{g}_{t}(\bm{x})\cdot\log\bm{f}(\bm{x})\right]$ (16)
$\displaystyle=\sum_{\bm{x}\in\mathcal{X}}P_{\text{t}}(\bm{x})\log
Z_{\bm{\theta}}(\bm{x})-\sum_{\bm{x}\in\mathcal{X}}P_{\text{t}}(\bm{x})\sum_{y\in\mathcal{Y}}g^{y}_{t}(\bm{x})\bm{\theta}_{y}\cdot\frac{P_{\text{s}}(\bm{x})}{P_{\text{t}}(\bm{x})}\bm{\phi}(\bm{x})$
(17) $\displaystyle=\sum_{\bm{x}\in\mathcal{X}}P_{\text{t}}(\bm{x})\log
Z_{\bm{\theta}}(\bm{x})-\sum_{y\in\mathcal{Y}}\bm{\theta}_{y}\cdot\tilde{{\bf
c}}_{y}$ (18) $\displaystyle=\mathcal{L}(\theta,\lambda)$ (19)

Then, we take the derivative of a specific $\bm{x}$’s densities,
$\tau_{s}(\bm{x})$ and $\tau_{t}(\bm{x})$.

$\centering\begin{split}\frac{\partial\mathcal{L}_{1}}{\partial\tau_{s}}&=\frac{\partial}{\partial\tau_{s}}\left(\sum_{\bm{x}\in\mathcal{X}}P_{\text{t}}(\bm{x})\log
Z_{\bm{\theta}}(\bm{x})-\sum_{y\in\mathcal{Y}}\bm{\theta}_{y}\cdot\tilde{{\bf
c}}_{y}\right)\\\
&=\frac{1}{\tau_{t}}\sum_{\bm{x}\in\mathcal{X}}P_{\text{t}}(\bm{x})\sum_{y\in\mathcal{Y}}\bm{\theta}_{y}\cdot\frac{\exp(\bm{\theta}_{y}\cdot\frac{P_{\text{s}}(\bm{x})}{P_{\text{t}}(\bm{x})}\bm{\phi}(\bm{x}))}{Z_{\bm{\theta}}(\bm{x})}\bm{\phi}(\bm{x})\\\
&=\frac{1}{\tau_{t}}\mathbb{E}_{\bm{x}\sim
P_{t}(X)}\left[\sum_{y\in\mathcal{Y}}\bm{\theta}_{y}\cdot
f_{y}(\bm{x})\bm{\phi}(\bm{x})\right].\end{split}\@add@centering$ (20)

In the same spirit, the gradient of $\mathcal{L}_{1}$ over target densities
$\tau_{t}$ is:

$\centering\begin{split}\frac{\partial\mathcal{L}_{1}}{\partial\tau_{t}}&=\frac{\partial}{\partial\tau_{t}}\left(\sum_{\bm{x}\in\mathcal{X}}P_{\text{t}}(\bm{x})\log
Z_{\bm{\theta}}(\bm{x})-\sum_{y\in\mathcal{Y}}\bm{\theta}_{y}\cdot\tilde{{\bf
c}}_{y}\right)\\\
&=\frac{\tau_{s}}{\tau_{t}^{2}}\sum_{\bm{x}\in\mathcal{X}}P_{\text{t}}(\bm{x})\sum_{y\in\mathcal{Y}}\bm{\theta}_{y}\cdot\frac{\exp(\bm{\theta}_{y}\cdot\frac{P_{\text{s}}(\bm{x})}{P_{\text{t}}(\bm{x})}\bm{\phi}(\bm{x}))}{Z_{\bm{\theta}}(\bm{x})}\bm{\phi}(\bm{x})\\\
&=\frac{\tau_{s}}{\tau_{t}^{2}}\mathbb{E}_{\bm{x}\sim
P_{t}(X)}\left[\sum_{y\in\mathcal{Y}}\bm{\theta}_{y}\cdot
f_{y}(\bm{x})\bm{\phi}(\bm{x})\right].\end{split}\@add@centering$

Here, $f_{y}$ is the $y$-th dimension of the prediction $\bm{f}$ and
$Z_{\bm{\theta}}(\bm{x})$ is the normalization term of $\bm{f}$. Note that,
the gradients are not relevant to the target labels, but only target input
data. We then use finite samples in the target domain to approximate these
gradients.

##  Appendix D Additional Results

In this section, we provide results for DRL, DRST and DRSSL in addition to the
main paper.

###  D.1 Additional DRL Results (for Section 3.1)

Accuracy Comparison of Deep DRL on Office31 Note that DRL not only achieves
better uncertainties than the baselines, but also does not hurt the prediction
accuracy too much and may even achieve better accuracy. We show the DRL
results on all of the tasks of Office31 in Figure 8 by comparing accuracy with
Source model and a Bayesian method: LL-SVI 666Riquelme, C., Tucker, G., and
Snoek, J. Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian
Deep Networks for Thompson Sampling. In ICLR, 2018..

Additional Brier Score Results on Office31 Figure 8 (Bottom) also demonstrates
that our method generate more calibrated uncertainty estimates than Source and
LL-SVI.

Additional Reliability Plots on Office31 Figure 9 shows all the reliability
plots in all tasks in Office31 (blue line for source-only, green line for TS
and orange line for DRL).

|  |
---|---|---
|  |
(a) |  (b) |  (c)
Figure 9: Office31 results on reliability plots, which is a complement for
Figure 4 in main text. DRL is compared with source-only and temperature
scaling. Note that there are some missing points for source-only and TS in
some plots. This is due to the fact that the corresponding models do not have
data prediction results with confidence in those ranges.
---
Office31 (A$\rightarrow$W) Office-Home (P$\rightarrow$A) VisDA
Figure 10: Comparison of DRL and source-only of misclassification entropy on
different datasets.

Misclassification Entropy Comparison on Office31 Figure 10 shows the
misclassification entropy comparison between source model and the deep DRL
model. Misclassification entropy is calculated as
$S_{i}=\frac{1}{n}\mathop{\sum}_{i=1}^{n}\mathop{\sum}_{j=1}^{m}p_{ij}\log
p_{ij}$ , where $n$ is the number of samples and $m$ is the number of
categories in the dataset, and $p_{ij}$ indicates the prediction probability
of the $i$th sample on the $j$th category. The larger misclassification
entropy is, the more uncertain the model prediction result is on the wrong
predictions. This means the model would fail more gently.

ECE Results on Office31 and VisDA Table 4 demonstrates the expected
calibration error of Source-Only, LL-SVI , TS and our method on different
tasks in the Office31 datasets. LL-SVI is an approximate Bayesian inference
method for the parameters of the last layer only. Deep DRL achieves much
better ECE than all the baselines. Table 5 shows the ECE score on VisDA2017
for Source-Only, TS and DRST. DRST also avhieves better ECE than the other two
baselines.

Table 4: ECE score comparison on Office31 Tasks | A$\rightarrow$W | A$\rightarrow$D | W$\rightarrow$A | W$\rightarrow$D | D$\rightarrow$W | D$\rightarrow$A
---|---|---|---|---|---|---
Source-Only | 0.193 | 0.140 | 0.329 | 0.003 | 0.026 | 0.304
LL-SVI | 0.669 | 0.747 | 0.790 | 0.905 | 0.887 | 0.782
TS | 0.128 | 0.136 | 0.241 | 0.001 | 0.000 | 0.263
DRL | 0.031 | 0.041 | 0.045 | 0.000 | 0.029 | 0.018
Table 5: ECE score comparison on VisDA2017 Methods | Source-Only | TS | DRST
---|---|---|---
ECE | 0.108 | 0.096 | 0.089

Full density Ratio Results for Different HSF Bins on ImageNetV2 When comparing
the density ratios of different HSFs (Figure 5), we treat low HSF as having
HSF in range $[0,0.2]$, and high HSF from $(0.2,1]$. Results on ResNet50 and
DenseNet121 are shown in Table 6.

Table 6: Additional results on density ratio v.s. HSF HSF | ResNet50 | DenseNet121
---|---|---
Low HSF | 1.797 | 0.657
High HSF | 1.804 | 0.665

We also show more detailed comparison between density ratio and HSF. Here we
divide the HSF into 5 bins, with ranges of $[0.0,0.2]$, $[0.2,0.4]$,
$[0.4,0.6]$, $[0.6,0.8]$,$[0.8,1.0]$. We show the numbers in Table 7. Note
that the numbers might be inconsistent with the binary case, because we
calculated the numbers based on different models for variance.

Table 7: Additional detailed results on density ratio v.s. HSF HSF | [0.0, 0.2] | [0.2, 0.4] | [0.4, 0.6] | [0.6, 0.8] | [0.8, 1.0]
---|---|---|---|---|---
AlexNet | 2.583 | 3.326 | 3.315 | 3.197 | 4.089
VGG-19 | 1.138 | 2.206 | 1.760 | 1.688 | 1.557
ResNet50 | 2.060 | 2.064 | 2.072 | 2.074 | 2.072
DenseNet121 | 0.686 | 0.693 | 0.692 | 0.692 | 0.693

Example Image from ImageNet for Density Ratio v.s. HSF We demonstrate two
examples in Figure 11 and Figure 12 with their learned density ratios from
deep DRL and their HSF values. The higher density ratios correspond with
higher human selection frequencies.

---
Figure 11: Example image from ImageNet with high and low human selection
frequencies and corresponding high and low density ratios.
---
Figure 12: Example image from ImageNet with high and low human selection
frequencies and corresponding high and low density ratios.

###  D.2 Additional DRST Results (for Section 3.2)

Source: |  |  |  |
---|---|---|---|---
|  |  |  |
Target: |  |  |  |
|  |  |  |
| ASG | ASG+CBST | ASG+CRST | ASG+DRST
Figure 13: TSNE visualization of the learned classifier using different
methods. Using DRST, the classes are well-separated.

TSNE Comparison for DRST and Baselines on VisDA We demonstrate the TSNE plot
of the learned decision boundaries for CBST, CRST and DRST in Figure 13.

Additional Attention Plots for DRST and Baselines on VisDA Figure 14
demonstrates additional model attention visualization by using Grad-Cam
777Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna
Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from
deep networks via gradient-based localization. In Proceedings of the IEEE
international conference on computer vision, pages 618–626, 2017..

Example Target Data with Different Density Ratios on VisDA Figure 15 shows
additional target examples with high and low density ratios. Our model is able
to find noisy and less-represented image from the target data by estimating
the density ratios. DRST would be more uncertain on those data.

###  D.3 Additional DRSSL Results (for Section 3.3)

We present more results on different degree of data deficiency in Table 8 and
Table 9. We use even smaller number of source labeled data to start the SSL
process. We show the results for the two pairs of source and target domains in
the paper: Source:CIFAR10/ Target: STL10 and Source:MNIST/ Target: SVHN. We
show the result on both the source test set and the target test set.

Table 8: Additional results on CIFAR10$\rightarrow$STL10 Labeled Samples | 250 | 40
---|---|---
Test Set | CIFAR10 | STL10 | CIFAR10 | STL10
Fixmatch | 65.70 | 47.92 | 50.98 | 23.50
DRSSL | 93.58 | 66.01 | 92.50 | 63.69
Table 9: Additional results on MNIST$\rightarrow$SVHN Labeled Samples | 400 | 40
---|---|---
Test Set | MNIST | SVHN | MNIST | SVHN
Fixmatch | 99.25 | 25.62 | 97.26 | 25.84
DRSSL | 99.29 | 32.19 | 97.28 | 27.06
---
Figure 14: Model attention visualized using Grad-Cam. Our method can also
capture the domain knowledge well. For example, the first row input image
contains a giraffe and a car. However, giraffe is not a existing label for
VisDA2017. While CBST and CRST capture the wrong information, DRST is able to
correctly capture the domain information.
---
Figure 15: Additional examples for density ratio estimation for different
categories. We can observe that data less well-represented in the source data
has much lower density ratio. This shows that our learned density ratio is a
good measure of the level of representation of data in source and target
domains.

##  Appendix E Experiment details

In this section, we include the experimental details that was not shown in
section 3 due to space limit.

###  E.1 Metric explanation

We calculate Brier score as the mean squared difference between the predicted
probability $p$ assigned to the possible outcome and the actual outcome $y$:
$BS=\frac{1}{n}\sum\limits_{i=1}^{n}\sum\limits_{j=1}^{m}(p_{ij}-y_{ij})^{2}\,\\!$.
ECE is calculated as
$\sum_{j=1}^{J}\frac{B_{j}}{N}|\text{acc}(B_{j})-\text{conf}(B_{j})|$, where
$B_{j}$ is a set of predictions binned by confidence values,
$\text{acc}(B_{j})$ is the average accuracy of the $B_{j}$ , and
$\text{conf}(B_{j})$ is the average confidence of the most likely label.

###  E.2 Computation resources

All of our training experiments are done on the DGX V100 Tesla V100 GPUs with
32GB memory. The main packages and corresponding versions we use are: Python
3.6, PyTorch 0.4.0, CUDA 10.1.

###  E.3 Details for section 3.1

We train DRL on Office31 and OfficeHome using the ResNet50 backbone, and use
the ResNet101 backbone for VisDA. DRL on ImageNetV2 uses the DenseNet121
backbone. For each task, we use the same backbone for the source model and the
TS model to make comparison fair. TS models adopt a temperature value of 1.5,
and is scaled by using the target dataset containing both images and labels.
The DRL models we train use different update frequencies for the density ratio
estimator and the classifier. The classifier is trained for every batch
iteration, and the density ratio estimator is trained once per 5 batch
iterations and we guarantee that the estimator sees all of the target domain
data at least once. For training on Office31, OfficeHome and ImageNet, we use
the SGD optimizer with learning rate selected from $[0.03,0.01,0.001,0.0001]$,
momentum $0.9$ and weight decay rate of $0.0001$.

We have done experiments on ImageNet using four different network
architectures: AlexNet, VGG-19, ResNet50 and DenseNet121. The training details
follow the official PyTorch training process. The models are not trained from
scratch but rather use pretrained models as the starting point. For AlexNet,
VGG-19 and ResNet50, we use a learning rate of $10^{-4}$; For DenseNet121, the
learning rate is $10^{-6}$. Note that when comparing the scores where there
consist of the TS (temperature scaling) baseline (such as Figure 4 in main
text), we only use half of the ImageNetV2 dataset to do the evaluation. The
other half of the dataset is used to train the TS model. This is for the fair
comparison of TS and the other models so that they all cannot access the test
data during the training process.

###  E.4 Details for section 3.2

In this subsection, we focus on self-training for UDA. We follow the same
implementation as CBST and only change the model and few hyperparameter
values. We set batch size as 16, use the SGD optimizer with learning rate
$10^{-5}$ and momentum 0.9. The initial value of source data portion is
$6.5\%$ and we add $0.85\%$ of source data number of data into the training
set every epoch. The maximum portion is set as $16.5\%$.

###  E.5 Details for section 3.3

For Fixmatch, under the single domain setting, the model is trained using
source labeled data and source unlabeled data. However, under the cross-domain
setting, we train the model using source labeled data and target unlabeled
data. We follow the implementation of Fixmatch and use the same augmentation
methods. The unlabeled dataset uses all of the target domain train data, while
the labeled dataset samples the same number of data for every class from the
source domain data.

We test both the source test data performance and the target test data
performance, to evaluate label efficiency under different settings. We tested
the performance when using different pretrained models for DRSSL. The
pretrained models are achieved by using Fixmatch but under different settings,
including the source-only setting (original Fixmatch setting) and the cross-
domain setting (the setting which we use). We find that for
CIFAR10$\rightarrow$STL10 task, the source-only setting helps to achieve
better final performance ($5\%$ more) while for MNIST$\rightarrow$SVHN, the
two settings yield similar performance.

For the CIFAR10$\rightarrow$STL10 task, note that the ten classes in two
datasets are different. CIFAR10 and STL10 only have 9 classes in common, thus
we eliminate the "frog" samples in CIFAR10 and the "deer" class samples in
STL10. Also note that we regard the "automobile" class in CIFAR10 and "car"
class in STL10 as the same. We also match the order in which the classes are
represented.

##  Appendix F Code Repository

Code can be referred to at: https://anonymous.4open.science/r/Deep-
Distributionally-Robust-Learning-for-Calibrated-Uncertainties-under-Domain-
Shift-7C2E