heightadjust=object capbtabboxtable capbfigboxfigure

#  S4T: Source-free domain adaptation for semantic segmentation via self-
supervised selective self-training

Viraj Prabhu∗ Shivam Khare Deeksha Kartik Judy Hoffman
Georgia Institute of Technology  Equal contribution. Correspondence to Viraj
Prabhu<EMAIL_ADDRESS>
###### Abstract

Most modern approaches for domain adaptive semantic segmentation rely on
continued access to source data during adaptation, which may be infeasible due
to computational or privacy constraints. We focus on _source-free_ domain
adaptation for semantic segmentation, wherein a source model must adapt itself
to a new target domain given only unlabeled target data. We propose Self-
Supervised Selective Self-Training (S4T), a source-free adaptation algorithm
that first uses the model’s pixel-level predictive consistency across diverse
views of each target image along with model confidence to classify pixel
predictions as either reliable or unreliable. Next, the model is self-trained,
using predicted pseudolabels for reliable predictions and pseudolabels
inferred via a selective interpolation strategy for unreliable ones. S4T
matches or improves upon the state-of-the-art in source free adaptation on 3
standard benchmarks for semantic segmentation within a single epoch of
adaptation.

##  1 Introduction

In this work, we focus on the challenging problem of _source-free_ domain
adaptation [1, 2, 3, 4] for semantic segmentation [5, 6]. Consider a deep
model trained to perform semantic segmentation deployed atop an autonomous
vehicle. While unsupervised domain adaptation (DA) has been extensively
studied [7, 8, 9, 10, 11], most prior DA methods assume continued access to
labeled source data during adaptation. In our example, this may be impractical
due to the limitations of on-board compute and memory, particularly so for a
compute-heavy task such as segmentation. Further, such access to source data
on-board may also be subject to and limited by privacy regulations.

Concretely, our goal is to adapt a trained semantic segmentation model to a
new target domain given only its trained parameters and unlabeled target data.
The absence of source data for regularization makes this source-free
adaptation setting very challenging and highly susceptible to divergence from
original task training, leading to catastrophic performance loss.

Recent work has proposed parameter constrained self-training as a means to
source-free domain adaptation. For example, Test-time adaptation by entropy
minimization or TENT [5], constrains optimization to only update the model’s
batch-norm parameters (both affine and normalization), and self-trains on
unlabeled target data by minimizing a conditional entropy [12] loss. By
keeping all other parameters frozen, TENT is able to prevent task drift in the
source-free adaptation setting.

While TENT leads to modest performance improvements on standard domain shifts,
it performs self-training on _all_ model predictions. Under a domain shift,
many of the model’s predictions may initially be incorrect, and entropy
minimization encourages the model to increase its confidence even on such
incorrect predictions! As a result, unconstrained self-training leads to error
accumulation [13, 14, 15], particularly on categories on which the source
model does poorly to begin with (See Fig. 1).

To address this, we propose _selective_ self-training – specifically, we first
identify pixel predictions with a high likelihood of being correctly
classified, and only self-train on those. Naturally, identifying such reliable
pixels is challenging without access to labels – while one may use model
confidence, predictive distributions from deep networks are often
miscalibrated under a domain shift [16], and relying solely on them for
selection degrades performance. We instead propose a novel self-supervised
selection criteria that combines _pixel-level predictive consistency_ with
model confidence.

Specifically, we generate two views of each target image that vary in scale
and spatial context via a random crop and resize strategy. We then obtain
aligned model predictions in both views and mark pixels for which the model
makes identical predictions in both views as “reliable”. Note that while
several recent works have used contrastive learning to encourage consistency
between predictions across different views [17, 18, 19], we instead propose
_measuring_ this consistency to identify reliable pixels for self-training.
Further, we also mark pixels that are in the top-K (where K is a
hyperparameter) percentile by confidence _per-category_ as reliable.

Next, we mark all pixels not marked as reliable as “unreliable”. To assign a
pseudolabel to such pixels, we propose a selective label interpolation
strategy that infers a pseudolabel as an average of model predictions on all
its reliable neighbors (for a given neighborhood size). We then self-train the
model on predicted pseudolabels (for reliable predictions) and interpolated
pseudolabels (for unreliable ones). To prevent task drift in the source-free
setting, we match TENT [5] to constrain weight updates to only the model’s
batch-norm parameters. We make the following contributions:

Figure 1: We study source-free domain adaptation for semantic segmentation,
where the goal is to adapt a well-trained source model to a target domain
given only unlabeled data from the target. Left. Under a domain shift, many
model predictions on the target domain may initially be incorrect. Top right:
Unconstrained self-training on such model predictions reinforces model errors
and degrades performance over the course of training (yellow box denoting
missed categories). Bottom right: We propose S4T, a selective self-training
algorithm that identifies reliable predictions on which to self-train based on
pixel-level predictive consistency across diverse target image views, and
leads to state-of-the art performance in the source-free setting.

  1. 1.

We propose Self-supervised Selective Self-training (S4T), a source-free
adaptation algorithm for semantic segmentation that i) identifies reliable
pixel predictions on which to self-train by combining pixel-level predictive
consistency across diverse target image views and class-conditioned confidence
thresholding, and ii) self-trains the model to predict the current (for
reliable) or selectively interpolated (for unreliable) pseudolabels.

  2. 2.

S4T improves upon the state-of-the-art for source-free adaptation from GTA5
[20]$\to$Cityscapes [21] (+1.7 mIoU) and Cityscapes$\to$Dark Zurich [22] (+5.4
mIoU), and matches it on SYNTHIA [23]$\to$Cityscapes, all within a single
epoch of adaptation.

##  2 Related Work

Unsupervised domain adaptation (UDA) for semantic segmentation. Initial
approaches to UDA for semantic segmentation employed pixel-level adversarial
learning [24], adversarial learning in the output space [25], or adversarial
learning with significance-based feature purification [26]. Follow-up work has
incorporated additional pixel-level constraints such as image-to-image
translation [27], bidirectional translation [28], cycle consistency and
inconsistency [29, 30] and several others [31, 32, 33, 34, 35, 36, 36, 37, 38,
39, 40]. We propose a source-free UDA approach for semantic segmentation that
does not require adversarial training or generative modeling, but instead
makes use of self-supervised signals in the form of pixel-level predictive
consistency for selective self-training.

Self-training for UDA. Recently, training on model predictions or _self-
training_ has been shown to be an effective UDA strategy [41, 11, 15, 42].
Self-training typically involves simple supervised training on the model’s
predictions on unlabeled target instances (_pseudolabels_) to minimize a
cross-entropy [43], conditional entropy [11, 37] or max-squares objective
[17], often with additional confidence regularization [41] or other
constraints [44, 45, 46]. However, unconstrained self-training leads to error
accumulation under a domain shift (see Sec 1). Recent work has sought to
resolve this via prototype-based pseudolabel denoising [47] or self-training
only on confident predictions [43]. We propose a selective self-training
approach that first identifies reliable (and unreliable) pixel predictions
based on predictive consistency across diverse image views, and then
selectively self-trains the model on predicted pseudolabels (for reliable
pixels) and interpolated pseudolabels (for unreliable ones).

Source-free and test-time adaptation. Recent work has studied the _source-
free_ UDA setting, wherein a trained source model must adapt itself to a
target given access only to unlabeled target data [1, 2, 3, 4]. These works
focus on image classification and propose approaches based on source
hypothesis transfer [1], class-conditional generative modeling [2],
progressive pseudolabeling [3], and source-similarity based weighting [4]. Liu
_et al_. [6] proposes the first source-free UDA method for semantic
segmentation that makes use of data-free distillation, self-training, and
patch-level self-supervision. While source-free UDA typically adapts models to
target train data and evaluates performance on a held-out target test set, the
related task of test-time adaptation [48, 5] directly adapts model to target
test data [48, 5]. We match the source-free UDA setting from Liu  _et al_. [6]
and propose a selective self-training approach that adapts the model in a
single epoch of training.

Batch normalization and domain adaptation. Some works have focused on the role
of batch normalization [49] (BN) layers in adapting to new domains, proposing
adaptive BN [50] and domain-specific BN [51]. Recent works propose test-time
updates to BN parameters as a means of overcoming covariate shift without
source data [52, 5] – Nado  _et al_. proposes re-estimating BN normalization
parameters on test data, whereas Wang  _et al_. propose TENT, which
additionally updates BN channel-wise affine parameters over test/target data
to minimize predictive entropy. Motivated by the success of these works in
source-free adaptation and recent findings on the large expressive power of BN
layers [53], we also constrain optimization to the network’s batch-norm
parameters during training.

Predictive consistency. Predictive consistency under image transformations has
been successfully applied as a regularizer in supervised learning [54], self-
supervised learning [17, 18, 19], semi-supervised learning [55, 56, 57, 58],
and unsupervised DA [59, 42]. Unlike prior work which _optimizes_ for
consistent predictions across augmentations, we propose _measuring_ the
model’s pixel-level predictive consistency across diverse views of each target
image to detect reliable (and unreliable) predictions on which to self-train.
Closer to our work is Bahat  _et al_. [60], which also uses inconsistency
under image transformations, but focuses on identifying natural and
adversarial errors in classification models whereas we focus on source-free
UDA for semantic segmentation.

##  3 S4T: Self-Supervised Selective Self-Training

Setup and Notation. We study the problem of source-free domain adaptation for
semantic segmentation. In semantic segmentation we are given an input image,
$\mathbf{x}\in\mathbb{R}^{H\times W\times 3}$, and the goal is to label every
pixel, $\mathbf{x}_{ij}$, with one of $C$ semantic labels,
$y_{ij}\in\\{1,2,\dots,C\\}$, producing an output label image,
$y\in\mathbb{R}^{H\times W}$. To do this, we will learn a function $h$ (CNN in
our case) which takes images as input and produces a probabilistic output over
$C$ classes for each output pixel:
$h:\mathbf{x}\rightarrow\mathbf{p}\in\mathbb{R}^{H\times W\times C}$. We
produce a _pseudolabel_ by taking the argmax of the output probabilities:
$\hat{y}=\operatorname*{argmax}\mathbf{p}$. In source-free unsupervised domain
adaptation, we assume access to a model trained on labeled source
($\mathcal{S}$) data, $h_{\mathcal{S}}$, as well as $N$ _unlabeled_ instances
$\mathbf{x}_{\mathcal{T}}\sim\mathcal{P}_{\mathcal{T}}(\mathcal{X})$ from a
target domain $\mathcal{T}$.

Overview. The main goal of our algorithm is to learn a target model, $h$, by
leveraging a trained source model ($h=h_{\mathcal{S}}$ at initialization) and
unlabeled target data. Standard unconstrained self-training under a domain
shift suffers from the problem of error accumulation. To address this, we
present S4T, a selective self-training strategy for source-free domain
adaptive semantic segmentation (see Fig. 2). Our method first uses a random
crop and resize strategy to generate two aligned predictive views of each
target image that capture model predictions across varying object scale and
spatial context (Sec. 3.1). Next, S4T identifies _reliable_ model predictions
on which to self-train using self-supervised signals in the form of pixel-
level predictive consistency across the two aligned views, as well as model
confidence. Finally, the model is self-trained, using pseudolabels for
reliable predictions and selectively interpolated pseudolabels for unreliable
ones (Sec. 3.2).

Figure 2: Overview of Self-supervised Selective Self-Training (S4T). Left:
First, the model makes predictions on two views of each target image that
differ in scale and spatial context and are generated via a random crop &
resize strategy (Sec. 3.1). Blue and orange arrows indicate the control flows
for generating the first and second predictive views. Right: Next, reliable
pixel predictions for self-training are identified based on pixel-level
consistency across aligned predictions and class-conditioned confidence
thresholding. Finally, the model is self-trained to predict the current (for
reliable) or selectively interpolated (for unreliable) pseudolabels (Sec.
3.2).

###  3.1 Aligned predictive view generation

A key facet of our approach will be to identify pixels for which model
predictions are deemed reliable. To do this we ensemble model predictions over
random image regions that differ in scale and spatial context. We begin by
randomly selecting a bounding box with coordinates,
$(r_{1},c_{1},r_{2},c_{2})$, for each target image that satisfies two
constraints: i) it spans an area that is 25-50% of the area of the original
image and ii) it matches the aspect ratio of the original image (i.e.
$(r_{2}-r_{1})/(c_{2}-c_{1})=H/W$).

View 1 (resized crop of prediction): To create the first output prediction, we
pass the original image, $\mathbf{x}_{\mathcal{T}}$, through the current
model, $h$, to produce an output probabilistic prediction,
$\mathbf{p}=h(\mathbf{x}_{\mathcal{T}})$. This original output prediction will
be cropped using the random bounding box coordinates and resized to the
original output image size:
$V=\texttt{resize}(\operatorname*{argmax}\mathbf{p}[r_{1}:r_{2},c_{1}:c_{2}],H,W)$

View 2 (prediction on resized image crop): For our second output prediction we
use the same bounding box coordinates to extract a cropped image region and
resize that region to the original image size to produce a rescaled image
view,
$\tilde{\mathbf{x}}_{\mathcal{T}}=\texttt{resize}(\mathbf{x}_{\mathcal{T}}[r_{1}:r_{2},c_{1}:c_{2}],H,W)$.
This resized cropped image is then passed through the model to produce a
probabilistic output, $\tilde{\mathbf{p}}=h(\tilde{\mathbf{x}}_{\mathcal{T}})$
and associated predicted view,
$\tilde{V}=\operatorname*{argmax}\tilde{\mathbf{p}}$.

We thus obtain aligned predictive views $V$ and $\tilde{V}$, which capture
model predictions made at varying object scale (_e.g._ in Fig. 2, the size of
the car in the secondary view is larger than in the original) and spatial
context (_e.g._ additional cars and lane markings are absent in the secondary
view).

###  3.2 Selective Self-Training

To increase the robustness of self-training, our method identifies pixel
predictions on which to self-train using two self-supervised reliability
signals: predictive consistency across aligned views and model confidence. The
two sources offer complementary information about the model’s stability across
augmentation and its intrinsic confidence. For instances deemed reliable, we
may directly self-train, while care is taken to avoid compounded errors when
learning using the unreliable instances.

####  3.2.1 Measuring Reliability

Pixel-level predictive consistency. First, we measure pixel-level consistency
between the model’s aligned predictions $V$ and $\tilde{V}$, and mark pixels
with identical predictions ($V_{ij}==\tilde{V}_{ij}$) across the two views as
“consistent” and those with different predicted labels as “inconsistent”. We
note that while invariance across augmented views has been used extensively in
prior work, particularly in recent work on semi- [55, 56, 57, 58] and self-
supervised [17, 18, 19] learning, we instead propose using such predictive
consistency to _detect_ reliable predictions on which to self-train. In Sec. 4
we empirically demonstrate that such consistency is indeed a reliable proxy
for correctness.

Class-conditioned confidence thresholding. In addition to predictive
consistency, we also aim to capture a notion of the intrinsic model
confidence. Ideally, the instances on which the model is highly confident
should be trusted as reliable. However, a single static threshold will be
inadequate for this goal since output probability distributions differ by
category [41]. Therefore, we compute a per-category empirical range to choose
an adaptive per-category confidence threshold. Given a batch, we gather all
output probabilities and select a confidence threshold per category,
$t_{c}\in\mathbb{R}$, corresponding to the top K-th percentile (K=50 in our
experiments) of observed confidence values for category $c$. We consider an
output prediction to be high confidence if its max score is greater than the
corresonding category threshold:
$\max\mathbf{p}_{ij}>t_{\operatorname*{argmax}\mathbf{p}_{ij}}$. Confidence
thresholding is performed over the second view, $\tilde{V}$, as we later opt
to self-train on this view. In this manner, we are able to incorporate class-
conditioned model confidence into our selection strategy.

Overall, for a pixel, $\mathbf{x}_{ij}$, with per-view probabilistic and
categorical predictions, $\mathbf{p},V$ and $\tilde{\mathbf{p}},\tilde{V}$, we
define a binary reliability value, $r_{ij}$, in the following way:

$r_{ij}=\begin{cases}1&\text{if
}\overbrace{V_{ij}=\tilde{V}_{ij}}^{\text{consistent}}\;\text{or}\;\overbrace{\max\;\tilde{\mathbf{p}}_{ij}>t_{\tilde{V}_{ij}}}^{\text{confident}}\\\
0&\text{otherwise}\\\ \end{cases}$ (1)

####  3.2.2 Learning Objectives

Using the reliability measure defined above, we now specify a set of learning
objectives for each unlabeled target pixel. For pixels deemed reliable, we
make the assumption (and empirically confirm – Sec. 4) that the current pixel
pseudolabel is likely to be accurate. Thus, we train using a standard self-
training cross-entropy loss on the predicted pseudolabel. Notably this is
different from prior approaches which optimize pseudolabel predictions over
all samples [11, 61, 59, 5] or on those with high model confidence alone [37,
43]. Further, for pixels deemed unreliable we aim to determine whether there
is some salient information which may be incorporated into a learning
objective.

To do this, we consider two cases of an unreliable pixel: 1) reliable
neighborhood where the pixel is unreliable but some adjacent neighbors are
reliable and 2) unreliable neighborhood where both the pixel and all its
adjacent neighbors are unreliable. Intuitively, if the model output is
unreliable for a particular pixel, but nearby pixels are reliable, there may
be some information from adjacent pixels that can be helpful for learning. To
capture this notion, we introduce selective label interpolation to produce an
updated pseudolabel in the reliable neighborhood case.

Selective label interpolation. We assign a pseudolabel to predictions
identified as unreliable, but which have adjacent neighbors with reliable
predictions, via a selective label interpolation strategy. Specifically, we
compute a pseudolabel distribution for each unreliable prediction as an
average of model predictions for all its _reliable_ neighbors (for a given
neighborhood size $k$). We then compute the argmax over this interpolated
distribution to yield a pseudolabel for each unreliable pixel prediction.

Concretely, for an unreliable pixel, $r_{ij}=0$, we can define a notion of
neighborhood reliability as the number of reliable pixels in a $k$ sized
neighborhood, $N_{ij}=\sum_{l=i-k/2,m=j-k/2}^{i+k/2,j+k/2}r_{lm}$. If any
neighbors are deemed reliable ($N_{ij}>0$), we compute i) a new interpolated
pseudolabel, $\hat{y}^{\text{int}}_{ij}$, by averaging all reliable model
predictions within the neighborhood, and ii) an associated interpolated
pseudolabel weight, $w_{ij}^{\text{int}}$, which captures the notion of
_neighborhood_ reliability by computing the fraction of reliable predictions
within a neighborhood:

$\hat{y}^{\text{int}}_{ij}=\operatorname*{argmax}\sum_{l=i-k/2}^{i+k/2}\sum_{m=j-k/2}^{j+k/2}r_{lm}\cdot\tilde{\mathbf{p}}_{lm}\quad;\quad
w_{ij}^{\text{int}}=\frac{N_{ij}}{k^{2}-1}$ (2)

For unreliable pixels with no reliable neighbors, we assume there is no
salvagable information available and thus consider optimization against a
uniform prior over the output space. This may be efficiently optimized by
maximizing predictive entropy to penalize model overconfidence [62, 15].

Selective self-training. Having obtained pseudolabels for all pixels, we
update model parameters via self-training. In a source-free setting, we do not
have access to any source data or labels and instead only receive a trained
source model and unlabeled target data. In such a setting, optimizing all
model parameters causes the model to rapidly diverge from its original task.
To address this, we update only the model’s batch-norm parameters (affine and
normalization), as proposed in Wang  _et al._ [5]. To build in data
augmentation we opt to backpropagate predictions made on the secondary view
($\tilde{V}$).

Further, to address the label imbalance across categories which poses a
significant challenge in semantic segmentation [37], we maintain a running
average of model predictions over the last-$Q$ batches (before interpolation),
denoted by $q\in\mathbb{R}^{C}$ (for $C$ categories), and employ log-inverse
frequency loss-weighting: We compute a per category loss weight $\lambda_{c}$
based on its normalized log inverse frequency [63]:
$\lambda_{c}=\log\left[\frac{\Sigma_{c=1}^{C}q_{c}}{q_{c}^{\eta}}\right]$,
where $\eta<1$ is a damping factor. We then minimize a cross-entropy loss
$L_{CE}$ between model predictions and pseudolabels (original and
interpolated). Let $\alpha$ denote a loss weight for unreliable predictions.
The self-training objective we minimize is:

$L_{SST}(\mathbf{x}_{ij})=\underbrace{r_{ij}\lambda_{\tilde{V}_{ij}}L_{CE}(\tilde{\mathbf{p}}_{ij},\tilde{V}_{ij})}_{\text{reliable}}+\underbrace{\alpha(1-r_{ij})\Big{[}\underbrace{\lambda_{\hat{y}^{\text{int}}_{ij}}w_{ij}^{\text{int}}L_{CE}(\tilde{\mathbf{p}}_{ij},\hat{y}^{\text{int}}_{ij})}_{\text{reliable
neighborhood}}+\underbrace{\mathbbm{1}[w_{ij}^{\text{int}}=0]L_{\text{entmax}}(\tilde{\mathbf{p}}_{ij})\Big{]}}_{\text{unreliable
neighborhood}}}_{\text{unreliable}}$

(3)

Finally, to encourage the model to make diverse predictions over the target
domain, we add a target “information entropy” loss ${L}_{IE}$ proposed in Li
_et al_. [59], which helps prevent trivial solutions when self-training in the
presence of label imbalance. To implement this, we update the model to
maximize entropy over the running average of its predictions $q$. ${L}_{IE}$
is given by:
$L_{IE}(\mathbf{x}_{ij})=\sum_{c=1}^{C}\tilde{\mathbf{p}}_{ijc}\log q_{c}$

For IE loss weight $\beta$, the complete S4T loss objective that is
backpropagated is given by:

$L_{\texttt{S4T}}=\mathbb{E}_{\mathbf{x}\sim\mathcal{P}_{\mathcal{T}}}\left[\frac{1}{HW}\sum_{i=1,j=1}^{H,W}L_{SST}(\mathbf{x}_{ij})+\beta
L_{IE}(\mathbf{x}_{ij})\right]$ (4)

##  4 Experiments

###  4.1 Datasets, Metrics, Implementation Details, and Baselines

We evaluate S4T on three shifts for semantic segmentation adaptation.

GTA5$\to$Cityscapes. GTA5 [20] contains $\sim$25k synthetic images (of
resolution 1914x1052) with pixel-level annotations extracted from the GTA5
game. Cityscapes [21] comprised of real images (of resolution 2048x1024) of
daytime street scenes from 50 European cities. Following prior work [10, 11,
41], we use the unlabeled train split of Cityscapes for adaptation (2975
images) and report 19-way classification performance over its validation split
(500 images).

SYNTHIA$\to$Cityscapes. SYNTHIA [23] is a semantic segmentation dataset
comprised of 9400 images of synthetic scenes of size 1280x760. 16 categories
overlap between SYNTHIA and Cityscapes; following prior work we report
performance after adaptation for 16-way and 13-way classification (excluding 3
challenging categories: wall, fence, and pole).

Cityscapes$\to$Dark Zurich Night. The Dark Zurich dataset [22] consists of
2416 nighttime, 2920 twilight, and 3041 daytime images of resolution
1920x1080. We only use the 2416 nighttime images for adaptation (skipping
twilight and daytime), tune performance on the Dark-Zurich validation set (50
images), and report adaptation performance on the Dark-Zurich test set (151
images).

Metric. We match prior work and report per-category Intersection-over-Union
(IoU) and its mean across classes (mIoU).

Implementation details. For S4T, we only update batch norm (BN) affine
parameters during training, report performance after 1 epoch of adaptation for
all settings, and use batch statistics to compute BN normalization parameters
at test-time. A models are trained with PyTorch [64] on 4 Nvidia RTX 2080
GPU’s with the Adam [65] optimizer, with weight decay $5\times 10^{-4}$, loss
weights $\alpha=0.1$ (to downweight relatively noisy unreliable pixels),
$\beta=0.1$, $\eta=0.5$, and neighborhood size $k=3$.

For shifts, GTA5$\to$Cityscapes & SYNTHIA$\to$Cityscapes, we resize images to
1024x512 on Cityscapes, 1024x560 on GTA5 and 1280x760 on SYNTHIA. Matching Liu
_et al_. [6], we use DeepLabV3 [66] with a ResNet50 [67] backbone and
initialize models with ImageNet [68] weights. We train on the source domain
for 20 epochs (GTA5) and 10 epochs (SYNTHIA), making use of Gaussian blur and
random flip augmentations. We use a batch size of 12, and learning rate of
$1\times 10^{-4}$ for GTA5 and $5\times 10^{-5}$ for SYNTHIA. On SYNTHIA, we
L2-normalize per-category classifier weights in the last FC layer and set bias
to 0 following [69]. For the Cityscapes $\to$ Dark Zurich Night shift, we
match Sakaridis  _et al_. [22] and use a DeepLabV2 [70] architecture with
ResNet-101 [67] backbone. We set learning rate to $2.5\times 10^{-4}$ and use
a batch size of 8.

Baselines. To our knowledge, only two prior works have studied source-free
domain adaptation for semantic segmentation: i) TENT [5]: TENT is a source-
free, test-time adaptation algorithm that learns batch-norm parameters so as
to minimize predictive entropy over unlabeled target data, and ii) SFDA [6]:
SFDA is the current state-of-the art in source-free DA for semantic
segmentation. SFDA makes use of data-free distillation via a dual attention
module for knowledge transfer from the source model, followed by adaptation
via self-training and patch-level self-supervised learning in the target
domain. We also benchmark performance for Test-time BN [52], a test-time
adaptation method that only updates batch-norm normalization statistics over
the entire test data before making a prediction.

For completeness, we also report performance for a few _non_ source-free
baselines reported in Liu  _et al._ [6]: MinEnt [11], AdvEnt [11], AdaptSegNet
[25], CBST [61], and MaxSquares [61] (details in appendix). Unlike TENT and
SFDA, these baselines assume access to labeled source data during adaptation
and prevent task drift by minimizing an additional supervised loss on source
data.

Method | SF |  road |  building |  vegetation |  car |  sidewalk |  sky |  pole |  person |  terrain |  fence |  wall |  bicycle |  sign |  bus |  truck |  rider |  light |  train |  motorcycle | mIoU
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
source | - | 74.6 | 77.6 | 77.9 | 71.1 | 16.0 | 79.4 | 14.8 | 57.9 | 9.0 | 16.6 | 12.5 | 11.0 | 15.1 | 18.8 | 25.0 | 24.1 | 35.6 | 0.5 | 15.4 | 34.36
Test-time BN [52] | ✓ | 79.5 | 79.5 | 81.3 | 72.7 | 29.8 | 74.1 | 28.0 | 58.8 | 25.3 | 22.2 | 19.4 | 11.0 | 22.6 | 17.0 | 24.5 | 19.9 | 34.3 | 2.4 | 14.7 | 37.73
TENT [5] | ✓ | 87.3 | 79.8 | 83.8 | 85.0 | 39.0 | 77.7 | 21.2 | 57.9 | 34.7 | 19.6 | 24.3 | 4.5 | 16.6 | 20.8 | 24.9 | 17.8 | 25.1 | 2.0 | 16.6 | 38.86
TENT [5]+MS | ✓ | 88.6 | 76.6 | 83.3 | 83.0 | 37.9 | 71.4 | 12.9 | 57.8 | 35.2 | 14.7 | 20.2 | 4.0 | 18.5 | 17.5 | 22.4 | 18.3 | 19.6 | 3.3 | 16.2 | 36.89
SFDA [6] | ✓ | 84.2 | 82.7 | 82.4 | 80.0 | 39.2 | 85.3 | 25.9 | 58.7 | 30.5 | 22.1 | 27.5 | 30.6 | 21.9 | 31.5 | 33.1 | 22.1 | 31.1 | 3.6 | 27.8 | 43.16
S4T (ours) | ✓ | 88.2 | 84.4 | 86.3 | 85.4 | 44.2 | 79.2 | 38.6 | 58.7 | 41.0 | 27.6 | 28.9 | 8.4 | 34.6 | 26.4 | 20.1 | 25.3 | 41.5 | 6.3 | 10.8 | 43.98
S4T \+ MS (ours) | ✓ | 89.7 | 84.4 | 86.8 | 85.0 | 46.7 | 79.3 | 39.5 | 61.2 | 41.8 | 29.0 | 25.7 | 9.3 | 36.8 | 28.2 | 19.3 | 26.7 | 45.1 | 5.3 | 11.8 | 44.83
MinEnt [11] | ✗ | 80.2 | 81.4 | 83.2 | 75.5 | 31.9 | 76.2 | 24.6 | 55.2 | 18.0 | 20.8 | 25.1 | 22.9 | 17.5 | 31.2 | 33.2 | 24.6 | 30.2 | 4.4 | 27.4 | 40.18
AdaptSegNet [25] | ✗ | 81.6 | 79.5 | 81.6 | 79.1 | 26.6 | 81.2 | 23.7 | 52.4 | 26.7 | 20.5 | 20.7 | 26.2 | 22.6 | 28.8 | 36.0 | 20.2 | 29.9 | 7.5 | 24.7 | 40.50
CBST [37] | ✗ | 84.8 | 80.4 | 83.5 | 79.2 | 41.5 | 82.3 | 24.7 | 54.7 | 29.6 | 22.4 | 19.5 | 34.9 | 20.4 | 32.3 | 34.5 | 25.3 | 30.2 | 6.8 | 29.0 | 42.94
MaxSquares [61] | ✗ | 85.8 | 82.4 | 83.2 | 81.0 | 33.6 | 79.8 | 26.5 | 57.8 | 32.9 | 25.0 | 25.3 | 32.4 | 18.7 | 32.6 | 32.1 | 22.2 | 33.3 | 5.2 | 29.8 | 43.14

Table 1: GTA5$\to$Cityscapes: IoU on the Cityscapes validation set. Categories
are in descending order of frequency. SF = Source Free, MS = Multi-scale
evaluation. Best mIoU is bolded and second-best is underlined.

Method | SF |  road |  building |  vegetation |  car |  sidewalk |  sky |  pole∗ |  person |  fence∗ |  wall∗ |  bicycle |  sign |  bus |  rider |  light |  motorcycle | mIoU | mIoU∗
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
source | - | 50.4 | 78.3 | 71.8 | 44.0 | 18.8 | 78.4 | 24.0 | 51.9 | 0.2 | 2.5 | 18.1 | 6.5 | 5.9 | 13.5 | 5.3 | 0.2 | 29.36 | 34.07
Test-time BN [52] | ✓ | 76.3 | 75.4 | 74.5 | 65.9 | 33.9 | 81.0 | 23.2 | 44.1 | 0.5 | 6.3 | 34.4 | 12.1 | 4.9 | 15.9 | 6.5 | 5.9 | 35.03 | 40.82
TENT [5] | ✓ | 88.1 | 74.4 | 77.3 | 77.6 | 44.9 | 82.8 | 21.8 | 52.9 | 0.1 | 4.3 | 15.8 | 7.8 | 7.5 | 9.7 | 2.0 | 0.2 | 35.46 | 41.62
SFDA [6] | ✓ | 81.5 | 80.6 | 83.1 | 81.3 | 43.5 | 87.6 | 19.9 | 36.8 | 0.7 | 1.4 | 31.7 | 7.1 | 22.7 | 9.5 | 4.2 | 8.6 | 39.20 | 45.89
S4T (ours) | ✓ | 84.9 | 79.5 | 80.7 | 77.4 | 43.2 | 82.4 | 26.3 | 52.4 | 0.3 | 7.2 | 37.9 | 11.7 | 9.6 | 18.7 | 7.8 | 9.5 | 39.34 | 45.81
MinEnt [11] | ✗ | 78.2 | 81.9 | 81.1 | 75.8 | 39.6 | 87.7 | 26.2 | 37.7 | 0.2 | 4.3 | 25.1 | 4.1 | 24.9 | 7.2 | 2.2 | 4.6 | 36.30 | 42.31
AdaptSegNet [25] | ✗ | 79.7 | 79.3 | 80.0 | 79.8 | 38.6 | 85.4 | 25.4 | 40.8 | 0.8 | 5.6 | 30.5 | 5.5 | 21.4 | 11.7 | 3.6 | 5.2 | 37.08 | 43.19
CBST [37] | ✗ | 81.4 | 80.4 | 81.4 | 82.1 | 44.2 | 89.5 | 25.6 | 39.7 | 0.7 | 7.9 | 32.9 | 12.4 | 21.9 | 10.6 | 5.2 | 6.3 | 38.88 | 45.23
MaxSquares [61] | ✗ | 81.0 | 82.6 | 85.3 | 84.7 | 39.8 | 90.1 | 23.2 | 39.9 | 0.5 | 8.7 | 33.4 | 12.4 | 19.4 | 8.4 | 6.6 | 10.2 | 39.12 | 45.65

Table 2: SYNTHIA$\to$Cityscapes: IoU on the Cityscapes validation set.
Categories are in descending order of frequency. SF = Source Free. Best mIoU
is bolded and second-best is underlined. mIoU and mIoU∗ are calculated over 16
and 13 categories (excluding categories with a ∗) respectively.

###  4.2 Source-Free Unsupervised Adaptation Results

GTA5$\to$Cityscapes. Table 1 presents our results for adaptation from
GTA5$\to$Cityscapes. We evaluate our method (S4T) after a _single pass_ over
the unlabeled target data (_i.e._ one epoch), in two-settings: i) at standard
scale, and ii) after averaging predictions made at 4 scales, following [45].
S4T outperforms the state-of-the-art SFDA [6] by 0.82 and 1.67 mIoU in each
setting, and on 13/19 categories. S4T also significantly outperforms TENT [5]
by (+5.12/+7.94). Finally, we find S4T to also outperform the other non
source-free baselines from Liu  _et al._ [6] that make use of the same
backbone architecture. However, we acknowledge that several recent works have
reported significantly higher performance in the standard DA (source
accessible) setting with alternative model architectures, and thus restrict
our performance claims strictly to the source-free setting.

SYNTHIA$\to$Cityscapes. Table 2 presents our results for adaptation from
SYNTHIA$\to$Cityscapes. As seen, S4T achieves an mIoU/mIoU* of 39.34/45.81,
matching performance of the state-of-the-art SFDA method (39.20/45.89). We
note here that SFDA makes use of 120 epochs of adversarial learning whereas
S4T is only trained for a single epoch of adaptation. Further, S4T also
considerably outperforms TENT (by 3.88/4.19 mIoU). We do not find multi-scale
evaluation to help in this setting.

Cityscapes$\to$Dark Zurich Night. See Table 3. S4T again outperforms TENT [5]
(by 5.9 and 6.8 points, without and with multi-scale eval) and Test-time BN
[52] (by 4.5 and 5.4 points). Note that we restrict comparisons to methods
that only use nighttime images for adaptation.

Method | SF |  road |  building |  vegetation |  car |  sidewalk |  sky |  pole |  person |  terrain |  fence |  wall |  bicycle |  sign |  bus |  truck |  rider |  light |  train |  motorcycle | mIoU
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
source | - | 79.0 | 53.0 | 43.5 | 64.1 | 21.8 | 18.0 | 22.5 | 37.4 | 10.4 | 11.2 | 13.3 | 7.4 | 22.1 | 0.0 | 6.4 | 33.8 | 20.2 | 52.3 | 30.4 | 28.8
Test-Time BN [52] | ✓ | 77.9 | 50.0 | 44.6 | 48.4 | 36.8 | 0.6 | 34.8 | 33.8 | 7.0 | 15.0 | 19.8 | 15.4 | 25.9 | 0.0 | 0.1 | 38.1 | 16.9 | 36.0 | 30.9 | 28.0
TENT [5] | ✓ | 79.1 | 43.8 | 13.5 | 61.0 | 31.9 | 0.0 | 29.1 | 39.3 | 0.8 | 14.1 | 8.7 | 12.8 | 23.2 | 0.0 | 19.1 | 30.2 | 17.1 | 51.5 | 30.7 | 26.6
S4T (ours) | ✓ | 85.6 | 48.8 | 58.3 | 61.5 | 47.8 | 0.4 | 34.6 | 34.4 | 13.6 | 17.6 | 31.2 | 18.6 | 28.0 | 0.0 | 7.6 | 30.3 | 16.7 | 53.2 | 29.8 | 32.5
S4T +MS (ours) | ✓ | 84.0 | 47.4 | 56.7 | 65.8 | 46.2 | 0.1 | 37.5 | 38.5 | 12.7 | 17.5 | 21.2 | 22.1 | 31.1 | 0.3 | 9.4 | 38.8 | 20.1 | 53.9 | 31.6 | 33.4
AdaptSegNet [25] | ✗ | 86.1 | 55.1 | 37.2 | 68.2 | 44.2 | 1.2 | 21.1 | 35.9 | 8.4 | 4.8 | 22.2 | 15.6 | 16.7 | 0.0 | 45.1 | 26.7 | 5.6 | 50.1 | 33.9 | 30.4
Advent [11] | ✗ | 85.8 | 55.5 | 32.1 | 64 | 37.9 | 2 | 23.1 | 39.9 | 8.7 | 14.5 | 27.7 | 20.7 | 21.1 | 0.0 | 13.8 | 16.6 | 14 | 58.8 | 28.5 | 29.7

Table 3: Cityscapes$\to$Dark Zurich Night: IoU on the DZ-Night test set.
Categories are in descending order of frequency. SF = Source Free, MS = Multi-
scale evaluation. Best mIoU is bolded and second-best is underlined.

###  4.3 Ablating S4T

In Table 4 we present ablations of the components of S4T by reporting mIoU
over all categories, as well as over groupings of “thing” (_i.e._ object
instance) and “stuff” categories [71] 111“thing” categories in Cityscapes:
car, pole, person, bicycle, sign, bus, truck, rider, light, train, motorcycle.
We observe:

| Regularization | Selection strategy | Class balancing | Loss | mIoU $\uparrow$ |
---|---|---|---|---|---|---
# | H(Y) reg. | Confidence | Consistency | Loss wts. | Reliable | Unreliable | all | thing | stuff |
1 | (Unconstrained self-training on all predictions) | entmin | entmin | 38.86 | 26.57 | 55.76 |
2 | CE | CE | 37.77-1.09 | 27.39+0.82 | 52.03-3.73 |
3 | ✓ |  |  |  | CE | CE | 42.67+4.90 | 31.69+4.3 | 57.77+5.74 |
4 | ✓ | ✓ |  |  | CE | None | 27.48-15.2 | 23.71-7.98 | 32.66-25.1 |
5 | ✓ |  | ✓ |  | CE | None | 44.08+16.6 | 32.47+8.76 | 60.04+27.4 |
6 | ✓ | ✓ | ✓ |  | CE | None | 44.61+0.53 | 33.13+0.66 | 60.40+0.36 |
7 | ✓ | ✓ | ✓ | ✓ | CE | None | 44.76+0.15 | 33.34+0.21 | 60.43+0.03 |
8 | ✓ | ✓ | ✓ | ✓ | CE | CE${}_{\text{int}}$ | 44.83+0.07 | 33.47+0.14 | 60.43+0.00 |

Table 4: Ablating S4T on GTA$\to$Cityscapes. We report mIoU over all
categories, as well as over only “thing” and “stuff” categories. Subscript
denotes change over previous row, color coded for increase and decrease.
entmin = entropy minimization. CE / CE${}_{\text{int}}$ = cross-entropy
against predicted / interpolated pseudolabels (Eq. 3).

$\triangleright$ Unconstrained self-training leads to suboptimal performance
(Row 1-2). We first experiment with self-training on all pixel predictions to
minimize predictive entropy (matching TENT [5]), or cross-entropy with respect
to predictions. As seen, this achieves 38.86 (Row 1) and 37.77 (Row 2) mIoU,
outperforming the source model (mIoU=34.36, Tab. 1) by a relatively small
margin.

$\triangleright$ Pixel-level predictive consistency is an effective selection
strategy (Row 3-6). To regularize self-training, we first add a target
information entropy regularizer (Sec 3) and find that mIoU increases to 42.67
(Row 3). Next, to validate our selection criterion, we first use only
confidence for selection (we select predictions in the top-50 %ile by
confidence per-category) – despite careful tuning this leads to a low mIoU of
27.48 mIoU (Row 4), indicating that confidence alone is a poor indicator of
reliability.

Next, we experiment with using predictive consistency for selection, and find
this improves mIoU to 44.08 (Row 5), validating the hypothesis that predictive
consistency is an effective proxy for reliability (more analysis in Sec. 4).
Finally, combining consistency and confidence obtains the best performance of
44.61 (Row 6), particularly boosting performance on “thing“ categories
(+0.66).

$\triangleright$ Loss-weighting and selective interpolation further improve
performance, particularly on “thing” categories (Row 7-8). Finally we add the
loss-weighting and selective interpolation components of our method, and
observe modest performance gains of 0.15 (Row 7) and 0.07 mIoU (Row 8). We
again find a majority of these gains to arise from improvements on “thing”
categories.

$\triangleright$ Reliability oracle. As an upper bound, we train our method
with perfect reliability maps (using ground truth labels) and find this
achieves an mIoU of 48.20 within a single epoch – the 3.37 mIoU gain over our
method captures the performance drop due to imperfect reliability maps. In
appendix we analyze the relationship between reliabilty mapping accuracy and
mIoU.

In the appendix, we try alternatively training all model parameters instead of
just batch norm (achieves an mIoU 3.23 points lower than S4T), and vary the
neighborhood size $k$ ($k=3$ works best).

###  4.4 Analyzing S4T

Fig. 3 shows qualitative segmentation results achieved by our method. We
further analyze:

Figure 3: Qualitative segmentation results of the source model, TENT [5], and
S4T. White boxes highlight categories recovered by S4T, whereas red boxes show
some failure cases.

Pseudolabel type | accuracy (%)
---|---
reliable | 85.93
unreliable | 26.37
– reliable neigbhorhood | 27.90$\to$42.83∗
– unreliable neigbhorhood | 25.87

∗Accuracy of unreliable pixel with reliable
neighborhood increases throughout training.

(a) Pseudolabel accuracies by type

(b) % pseudolabel types vs iter.

(c) Reliability Precision

Figure 4: Analyzing reliability measure and selective interpolation components
of S4T. NBHD = neighborhood.

Evaluating reliability measure and selective interpolation (Fig. 4). To
evaluate whether S4T’s selection strategy is indeed a good indicator of
reliability, we first measure the accuracy of pseudolabels marked as reliable
and unreliable – as Fig. 4(a) shows, reliable pseudolabels have an accuracy of
85.93%, whereas unreliable ones have a low accuracy of 26.36%. Further, we
find that S4T’s selective interpolation strategy improves accuracy on
unreliable predictions in reliable neighborhoods from 27.90% to 42.83%
(+14.94%). Fig. 4(b) presents the breakdown of predictions as reliable,
unreliable (reliable neigbhorhood), and unreliable (unreliable neigbhorhood),
as training progresses – as seen, a majority (>80%) of predictions are
considered reliable, and this proportion increases over time.

In Fig. 4(c), we evaluate the reliability measure by category. For each
category, we report i) _precision_ of reliability with respect to correctness
(when a pixel prediction is reliable, how often is it actually correctly
classified?), and ii) precision of unreliability with respect to
incorrectness. As seen, unreliable predictions are highly correlated with
being incorrect across categories, but the precision of the reliability
measure is significantly higher for head categories (_e.g._ road, building,
car) than for the tail (_e.g._ bicycle, bus, train). In fact, we observe a
high correlation between the precision of reliabilty for a given category and
its mIoU after adaptation (Pearson correlation coefficient=0.88).

Computational efficiency. S4T is a lightweight adaptation method, requiring i)
only one additional forward pass for the secondary view during training, and
ii) updates only to batchnorm parameters.

In the appendix, we analyze S4T’s training stability (for >1 epochs) and the
role of multi-scale evaluation, try other combinations of consistency and
confidence, and visualize its selection strategy.

##  5 Discussion and Limitations

Despite S4T’s versatility, its effectiveness is limited by the correlation
between predictive consistency and correctness, which varies across
categories. Modifying source training to improve this correlation may be
promising future work. Further, S4T requires an extra forward pass and
confidence thresholding, which makes it slightly slower than TENT [5].
Finally, while S4T seeks to make source-free adaptation more robust, such
unsupervised training still has the potential to fail silently. A deeper
investigation into the the failure modes of such methods is required before
deployment in sensitive applications. Despite these limitations, S4T is a step
towards developing an effective source-free adaptation algorithm that can be
used in the real-world at practical computational cost.

Acknowledgements. We thank Prithvijit Chattopadhyay for feedback on the draft.
This work was supported in part by funding from the DARPA LwLL project.

## References

  * [1] J. Liang, D. Hu, and J. Feng, “Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation,” in International Conference on Machine Learning, pp. 6028–6039, PMLR, 2020.
  * [2] R. Li, Q. Jiao, W. Cao, H.-S. Wong, and S. Wu, “Model adaptation: Unsupervised domain adaptation without source data,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9641–9650, 2020.
  * [3] Y. Kim, S. Hong, D. Cho, H. Park, and P. Panda, “Domain adaptation without source data,” arXiv preprint arXiv:2007.01524, 2020.
  * [4] J. N. Kundu, N. Venkat, R. V. Babu, et al., “Universal source-free domain adaptation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4544–4553, 2020.
  * [5] D. Wang, E. Shelhamer, S. Liu, B. Olshausen, T. Darrell, U. Berkeley, and A. Research, “Tent: Fully test-time adaptation by entropy minimization,” in International Conference on Learning Representations, vol. 4, p. 6, 2021\.
  * [6] Y. Liu, W. Zhang, and J. Wang, “Source-free domain adaptation for semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2021.
  * [7] K. Saenko, B. Kulis, M. Fritz, and T. Darrell, “Adapting visual category models to new domains,” in European conference on computer vision, pp. 213–226, Springer, 2010.
  * [8] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in International Conference on Machine Learning, pp. 1180–1189, 2015.
  * [9] M. Long, Y. Cao, J. Wang, and M. Jordan, “Learning transferable features with deep adaptation networks,” in International Conference on Machine Learning, pp. 97–105, 2015.
  * [10] J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell, “Cycada: Cycle-consistent adversarial domain adaptation,” in International Conference on Machine Learning, pp. 1989–1998, 2018.
  * [11] T.-H. Vu, H. Jain, M. Bucher, M. Cord, and P. Pérez, “Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2517–2526, 2019.
  * [12] Y. Grandvalet, Y. Bengio, et al., “Semi-supervised learning by entropy minimization.,” in CAP, pp. 281–296, 2005.
  * [13] C. Chen, W. Xie, W. Huang, Y. Rong, X. Ding, Y. Huang, T. Xu, and J. Huang, “Progressive feature alignment for unsupervised domain adaptation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 627–636, 2019.
  * [14] X. Jiang, Q. Lao, S. Matwin, and M. Havaei, “Implicit class-conditioned domain alignment for unsupervised domain adaptation,” in International Conference on Machine Learning, pp. 4816–4827, PMLR, 2020.
  * [15] V. Prabhu, S. Khare, D. Kartik, and J. Hoffman, “Sentry: Selective entropy optimization via committee consistency for unsupervised domain adaptation,” arXiv preprint arXiv:2012.11460, 2020.
  * [16] J. Snoek, Y. Ovadia, E. Fertig, B. Lakshminarayanan, S. Nowozin, D. Sculley, J. Dillon, J. Ren, and Z. Nado, “Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift,” in Advances in Neural Information Processing Systems, pp. 13969–13980, 2019.
  * [17] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in International conference on machine learning, pp. 1597–1607, PMLR, 2020.
  * [18] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738, 2020.
  * [19] Z. Xie, Y. Lin, Z. Zhang, Y. Cao, S. Lin, and H. Hu, “Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning,” arXiv preprint arXiv:2011.10043, 2020.
  * [20] S. R. Richter, V. Vineet, S. Roth, and V. Koltun, “Playing for data: Ground truth from computer games,” in European conference on computer vision, pp. 102–118, Springer, 2016.
  * [21] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213–3223, 2016.
  * [22] C. Sakaridis, D. Dai, and L. V. Gool, “Guided curriculum model adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7374–7383, 2019.
  * [23] G. Ros, L. Sellart, J. Materzynska, D. Vazquez, and A. M. Lopez, “The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3234–3243, 2016.
  * [24] J. Hoffman, D. Wang, F. Yu, and T. Darrell, “Fcns in the wild: Pixel-level adversarial and constraint-based adaptation,” arXiv preprint arXiv:1612.02649, 2016.
  * [25] Y.-H. Tsai, W.-C. Hung, S. Schulter, K. Sohn, M.-H. Yang, and M. Chandraker, “Learning to adapt structured output space for semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7472–7481, 2018.
  * [26] Y. Luo, P. Liu, T. Guan, J. Yu, and Y. Yang, “Significance-aware information bottleneck for domain adaptive semantic segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6778–6787, 2019\.
  * [27] Z. Murez, S. Kolouri, D. Kriegman, R. Ramamoorthi, and K. Kim, “Image to image translation for domain adaptation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4500–4509, 2018.
  * [28] Y. Li, L. Yuan, and N. Vasconcelos, “Bidirectional learning for domain adaptation of semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6936–6945, 2019.
  * [29] J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell, “Cycada: Cycle-consistent adversarial domain adaptation,” in International conference on machine learning, pp. 1989–1998, PMLR, 2018\.
  * [30] G. Kang, Y. Wei, Y. Yang, Y. Zhuang, and A. G. Hauptmann, “Pixel-level cycle association: A new perspective for domain adaptive semantic segmentation,” arXiv preprint arXiv:2011.00147, 2020.
  * [31] Y.-C. Chen, Y.-Y. Lin, M.-H. Yang, and J.-B. Huang, “Crdoco: Pixel-level domain transfer with cross-domain consistency,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1791–1800, 2019.
  * [32] H. Fu, M. Gong, C. Wang, K. Batmanghelich, K. Zhang, and D. Tao, “Geometry-consistent generative adversarial networks for one-sided unsupervised domain mapping,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2427–2436, 2019.
  * [33] Y. Yang and S. Soatto, “Fda: Fourier domain adaptation for semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4085–4095, 2020.
  * [34] Y. Yang, D. Lao, G. Sundaramoorthi, and S. Soatto, “Phase consistent ecological domain adaptation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9011–9020, 2020.
  * [35] Y. Chen, W. Li, and L. Van Gool, “Road: Reality oriented adaptation for semantic segmentation of urban scenes,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7892–7901, 2018.
  * [36] Y. Zhang, P. David, H. Foroosh, and B. Gong, “A curriculum domain adaptation approach to the semantic segmentation of urban scenes,” IEEE transactions on pattern analysis and machine intelligence, vol. 42, no. 8, pp. 1823–1841, 2019.
  * [37] Y. Zou, Z. Yu, B. Kumar, and J. Wang, “Unsupervised domain adaptation for semantic segmentation via class-balanced self-training,” in Proceedings of the European conference on computer vision (ECCV), pp. 289–305, 2018.
  * [38] F. Pan, I. Shin, F. Rameau, S. Lee, and I. S. Kweon, “Unsupervised intra-domain adaptation for semantic segmentation through self-supervision,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3764–3773, 2020.
  * [39] R. Sun, X. Zhu, C. Wu, C. Huang, J. Shi, and L. Ma, “Not all areas are equal: Transfer learning for semantic segmentation via hierarchical region selection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4360–4369, 2019.
  * [40] G. Li, G. Kang, W. Liu, Y. Wei, and Y. Yang, “Content-consistent matching for domain adaptive semantic segmentation,” in European Conference on Computer Vision, pp. 440–456, Springer, 2020.
  * [41] Y. Zou, Z. Yu, X. Liu, B. Kumar, and J. Wang, “Confidence regularized self-training,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 5982–5991, 2019.
  * [42] C. Wei, K. Shen, Y. Chen, and T. Ma, “Theoretical analysis of self-training with deep networks on unlabeled data,” in International Conference on Learning Representations, 2021.
  * [43] S. Tan, X. Peng, and K. Saenko, “Class-imbalanced domain adaptation: An empirical odyssey,” in Proceedings of the European Conference on Computer Vision (ECCV) Workshops, September 2020.
  * [44] Q. Lian, F. Lv, L. Duan, and B. Gong, “Constructing self-motivated pyramid curriculums for cross-domain semantic segmentation: A non-adversarial approach,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6758–6767, 2019.
  * [45] K. Mei, C. Zhu, J. Zou, and S. Zhang, “Instance adaptive self-training for unsupervised domain adaptation,” 2020.
  * [46] S. Tang, P. Tang, Y. Gong, Z. Ma, and M. Xie, “Unsupervised domain adaptation via coarse-to-fine feature alignment method using contrastive learning,” arXiv preprint arXiv:2103.12371, 2021.
  * [47] P. Zhang, B. Zhang, T. Zhang, D. Chen, Y. Wang, and F. Wen, “Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation,” arXiv preprint arXiv:2101.10979, 2021.
  * [48] Y. Sun, X. Wang, Z. Liu, J. Miller, A. A. Efros, and M. Hardt, “Test-time training for out-of-distribution generalization,” 2019.
  * [49] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in International conference on machine learning, pp. 448–456, PMLR, 2015.
  * [50] Y. Li, N. Wang, J. Shi, J. Liu, and X. Hou, “Revisiting batch normalization for practical domain adaptation,” arXiv preprint arXiv:1603.04779, 2016\.
  * [51] W.-G. Chang, T. You, S. Seo, S. Kwak, and B. Han, “Domain-specific batch normalization for unsupervised domain adaptation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7354–7362, 2019.
  * [52] Z. Nado, S. Padhy, D. Sculley, A. D’Amour, B. Lakshminarayanan, and J. Snoek, “Evaluating prediction-time batch normalization for robustness under covariate shift,” arXiv preprint arXiv:2006.10963, 2020.
  * [53] J. Frankle, D. J. Schwab, and A. S. Morcos, “Training batchnorm and only batchnorm: On the expressive power of random features in {cnn}s,” in International Conference on Learning Representations, 2021.
  * [54] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, “Randaugment: Practical automated data augmentation with a reduced search space,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702–703, 2020.
  * [55] M. Sajjadi, M. Javanmardi, and T. Tasdizen, “Regularization with stochastic transformations and perturbations for deep semi-supervised learning,” arXiv preprint arXiv:1606.04586, 2016.
  * [56] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, “Mixmatch: A holistic approach to semi-supervised learning,” in Advances in Neural Information Processing Systems, pp. 5049–5059, 2019\.
  * [57] Q. Xie, Z. Dai, E. Hovy, M.-T. Luong, and Q. V. Le, “Unsupervised data augmentation for consistency training,” arXiv preprint arXiv:1904.12848, 2020.
  * [58] K. Sohn, D. Berthelot, C.-L. Li, Z. Zhang, N. Carlini, E. D. Cubuk, A. Kurakin, H. Zhang, and C. Raffel, “Fixmatch: Simplifying semi-supervised learning with consistency and confidence,” arXiv preprint arXiv:2001.07685, 2020\.
  * [59] B. Li, Y. Wang, T. Che, S. Zhang, S. Zhao, P. Xu, W. Zhou, Y. Bengio, and K. Keutzer, “Rethinking distributional matching based domain adaptation,” arXiv preprint arXiv:2006.13352, 2020.
  * [60] Y. Bahat, M. Irani, and G. Shakhnarovich, “Natural and adversarial error detection using invariance to image transformations,” arXiv preprint arXiv:1902.00236, 2019.
  * [61] M. Chen, H. Xue, and D. Cai, “Domain adaptation for semantic segmentation with maximum squares loss,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2090–2099, 2019.
  * [62] G. Pereyra, G. Tucker, J. Chorowski, Ł. Kaiser, and G. Hinton, “Regularizing neural networks by penalizing confident output distributions,” ICLR, 2017.
  * [63] J. Ramos et al., “Using tf-idf to determine word relevance in document queries,” in Proceedings of the first instructional conference on machine learning, vol. 242, pp. 29–48, Citeseer, 2003.
  * [64] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., “Pytorch: An imperative style, high-performance deep learning library,” arXiv preprint arXiv:1912.01703, 2019.
  * [65] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  * [66] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking atrous convolution for semantic image segmentation,” arXiv preprint arXiv:1706.05587, 2017.
  * [67] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [68] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al., “Imagenet large scale visual recognition challenge,” International journal of computer vision, vol. 115, no. 3, pp. 211–252, 2015.
  * [69] B. Kang, S. Xie, M. Rohrbach, Z. Yan, A. Gordo, J. Feng, and Y. Kalantidis, “Decoupling representation and classifier for long-tailed recognition,” arXiv preprint arXiv:1910.09217, 2019.
  * [70] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 4, pp. 834–848, 2017.
  * [71] Z. Wang, M. Yu, Y. Wei, R. Feris, J. Xiong, W.-m. Hwu, T. S. Huang, and H. Shi, “Differential treatment for stuff and things: A simple unsupervised domain adaptation method for semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12635–12644, 2020.

##  6 Appendix

###  6.1 S4T: Additional ablations

####  6.1.1 Noisy reliability oracles

Figure 5:  GTA$\rightarrow$Cityscapes: Reliability oracles with varying amount
of noise

In Sec 4.3 of the main paper, we presented the performance for a reliability
oracle designed as S4T with perfect reliability maps (using ground truth
labels). We found that this achieves an mIoU of 48.20 within a single epoch –
a 3.37 mIoU gain over our method, capturing the performance drop due to
imperfect reliability maps.

In Fig. 5 we extend this experiment to train reliability oracles with a
varying amount of noise. Concretely, we begin with a perfect reliability
mapping and randomly flip the mapping for P-% of pixels in each image – we
vary P across $\\{0,20,40,60,80\\}$, where P$=0$ corresponds to the perfect
reliability oracle reported in the main paper. As seen, performance is
approximately the same upto P=20 but begins to dip at P=40, dropping off
steeply afterwards to below-source mIoU values. Clearly, a good reliability
measure is critical to the success of our method, and our combination of
pixel-level consistency and confidence is an effective choice that achieves an
mIoU of 44.83 (higher than the oracle with 40% noise, which achieves 43.61)
despite only using self-supervised signals.

####  6.1.2 Training all parameters.

Recall that in order to prevent divergence from the source task when adapting
with S4T, we opt to only train batch norm parameters similar to Wang  _et al_.
[5]. We now experiment with alternatively updating _all_ model parameters.
Using the same learning rate we use in the paper (1$\times 10^{-4}$), we
observe rapid overfitting and an mIoU of 0.4. After careful tuning, we achieve
an mIoU of 41.60 with a learning rate of 1$\times 10^{-5}$, still 3.23 points
lower than when only updating batch norm parameters during adaptation.
Clearly, constrained optimization of only channel-wise affine and
normalization parameters of batch-normalization layers is an effective
strategy for source-free adaptation.

####  6.1.3 Varying neighborhood size.

We vary neighorhood size used for selective interpolation across values
$k=\\{3,5,7,9\\}$, and observe a gradually decreasing trend in mIoU:
$\\{44.83,44.79,44.74,44.71\\}$. At larger values of $k$, while it becomes
possible to interpolate a pseudolabel for a larger proportion of unreliable
pixels (_i.e._ most unreliable pixels will be in a reliable neighborhood as
the neighborhood size increases), the accuracy of interpolation also decreases
as labels interpolated from distant neighbors are more frequently incorrect.
As a result, we find that interpolating from immediate neighbors (using $k=3$)
works best.

###  6.2 S4T: Additional analysis

####  6.2.1 Training stability

Figure 6:  GTA$\rightarrow$Cityscapes: 5 epochs of S4T.

#####  6.2.1.1 Convergence.

Recall that in the main paper, we report performance obtained after a single
epoch of adaptation via S4T. To test the training stability of our approach
when adapting for longer periods of time, as well as its convergence, we
experiment with adapting with S4T on GTA5$\to$Cityscapes for 5 epochs with a
learning rate of 2$\times 10^{-5}$ (1/5th the learning rate used for 1 epoch)
and a weight decay of 1$\times 10^{-4}$. As Fig. 6 shows, mIoU increases
rapidly until 2 epochs, finally peaking at 4 epochs (at 44.3) and dipping
slightly after (to 43.7). Overall, we observe stable training without
divergence even when training for longer time horizons despite the absence of
labeled source data.

#####  6.2.1.2 Variance over 3 runs.

We run S4T for adapting from GTA5$\to$Cityscapes with 3 random seeds, and
report mIoU mean and 1 standard deviation – we observe mIoU’s of 44.29 $\pm$
0.38 (with multi-scale evaluation) and 43.76 $\pm$ 0.16 (without multi-scale
evaluation). We note that in both cases, we still outperform the next best
method (SFDA [6] – mIoU=43.16, Table 1 of main paper), despite only adapting
the model for a single epoch.

####  6.2.2 Pseudolabel accuracy by type over time

Figure 7:  GTA$\rightarrow$Cityscapes: Pseudolabel accuracy over the course of
training (1 epoch). nbhd=neighborhood.

Recall that in Sec. 4.4 of the main paper, we reported the aggregate accuracy
of pseudolabels marked as reliable and unreliable, finding that i) reliable
pseudolabels are indeed significantly more accurate (85.93%) than unreliable
ones (26.36%), and that ii) S4T’s selective interpolation strategy improves
accuracy on unreliable predictions in reliable neighborhoods from 27.90% to
42.83% (+14.94%). In Fig. 7, we plot these accuracies for pseudolabels of
different types as training progresses.

As seen, we observe similar trends throughout the course of adaptation –
reliable pixels are indeed significantly more accurate than unreliable ones
throughout the course of training, and their accuracy gradually increases over
time as the model adapts to the target domain. Further, unreliable pixels are
subtantially less accurate, although our interpolation strategy consistently
refines predictions for unreliable pixels in reliable neighborhoods (see
before v/s after for “unreliable, reliable nbhd”).

####  6.2.3 Role of multi-scale evaluation

In Section 4.1 in the main paper, we additionally presented results for S4T
with multi-scale evaluation – following Mei  _et al_. [45], we make
predictions using our trained model at 4 different image resolutions (see
Table 5), and average to yield final predictions. We find such multi-scale
evaluation to improve performance over single-scale evaluation (at the default
resolution) by 0.85 points on GTA5$\to$Cityscapes and 0.9 points on
Cityscapes$\to$Dark Zurich. Since S4T self-trains on the secondary augmented
predicted view (see Sec. 3 in the main paper), it naturally learns objects at
different scales and is thus complemented by multi-scale evaluation at test
time.

In Table 5, we perform a fine-grained analysis of performance at each
individual scale. As seen, the first, second, and third scales perform best on
9/19, 5/19, and 4/19 categories respectively, which explains the performance
improvement observed. Further, the ensembled predictions outperform single-
scale evaluation at any given scale on 12/19 categories. We notice that
evaluating at higher resolutions (_e.g._ 640 $\times$ 1280, 768 $\times$ 1536)
particularly improves performance on small object categories (person, sign,
fence, light, motorcyle). The largest resolution (900 $\times$ 1800) does not
improve performance on any category, suggesting it may be dropped from the
list of scales to evaluate at.

Resolution |  road |  building |  vegetation |  car |  sidewalk |  sky |  pole |  person |  terrain |  fence |  wall |  bicycle |  sign |  bus |  truck |  rider |  light |  train |  motorcycle | mIoU
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
512$\times$1024 | 88.2 | 84.4 | 86.3 | 85.4 | 44.2 | 79.2 | 38.6 | 58.7 | 41.0 | 27.6 | 28.9 | 8.4 | 34.6 | 26.4 | 20.1 | 25.3 | 41.5 | 6.3 | 10.8 | 43.98
640$\times$1280 | 88.9 | 84.2 | 86.1 | 84.9 | 45.5 | 78.7 | 38.2 | 59.3 | 40.4 | 28.4 | 26.4 | 10.0 | 35.7 | 27.9 | 18.1 | 25.5 | 42.1 | 5.5 | 11.4 | 44.06
768$\times$1536 | 89.7 | 83.2 | 86.0 | 83.6 | 46.2 | 77.0 | 38.0 | 59.4 | 39.3 | 28.2 | 22.3 | 9.9 | 36.0 | 26.7 | 16.4 | 25.2 | 41.5 | 4.6 | 11.4 | 43.39
900$\times$1800 | 88.8 | 80.0 | 85.5 | 80.9 | 43.4 | 74.8 | 35.1 | 58.5 | 37.1 | 24.0 | 18.0 | 8.6 | 35.6 | 27.7 | 12.8 | 25.3 | 39.2 | 3.0 | 10.1 | 41.49
MS average | 89.7 | 84.4 | 86.8 | 85.0 | 46.7 | 79.3 | 39.5 | 61.2 | 41.8 | 29.0 | 25.7 | 9.3 | 36.8 | 28.2 | 19.3 | 26.7 | 45.1 | 5.3 | 11.8 | 44.83

Table 5: GTA5$\to$Cityscapes: Role of multi-scale evaluation. We evaluate S4T
at 4 different scales and report per-category and mean IoU at each scale, as
well as for the averaged predictions. MS=multi-scale.

Surprisingly, while multi-scale evaluation improves performance on both
GTA5$\to$Cityscapes and Cityscapes$\to$Dark Zurich, it does not generalize to
SYNTHIA$\to$Cityscapes, achieving 45.34/39.04 on 13 and 16 categories
respectively (slightly worse than 45.81/39.34 with single-scale evaluation at
512 $\times$ 1024). A similar per-scale analysis reveals that the original
resolution performs best on 11/16 categories and is only narrowly worse on the
others. We conjecture that this may potentially be due to a lack of scale
invariance learnt by the source model when trained on the small and less-
diverse SYNTHIA dataset (compared to GTA5), but leave further investigation to
future work.

Figure 8: GTA5$\to$Cityscapes: Visualizing reliability mapping and selective
interpolation steps of S4T. We use white to denote “reliable” and black to
denote “unreliable” pixels in the second-last row (reliability mapping).
Dashed boxes in the last row indicate a few regions where the selective
interpolation strategy refines pseudolabels.

####  6.2.4 Visualizing reliability mappings and interpolation

In Fig. 8 we include qualitative examples illustrating the different steps of
our algorithm, using a source model without any adaptation (_i.e._ at epoch
0). The first two rows correspond to the original target image and the
secondary view that is generated via a random crop and resize strategy. Rows 4
and 5 demonstrate aligned model predictions made on the two views (Row 3 shows
the corresponding ground truth annotations). Row 6 visualizes the self-
supervised reliability mappings obtained by combining pixel-level predictive
consistency across the two aligned predictive views with model confidence
(white corresponds to “reliable” and black to “unreliable”). As seen, our
selection strategy correctly identifies several incorrect pixel predictions as
unreliable (black). Finally, the last row shows model predictions on the
secondary view after our selective interpolation strategy (dashed black boxes
indicate a few regions where pseudolabels have been refined).

####  6.2.5 Alternative combinations of consistency and confidence

Recall that we opt to select as reliable pixel predictions that are either
consistent across predictive views or in the top-K %-ile of model confidence
for each category, and everything else as unreliable. As alternative
combinations of consistency and confidence, we experiment with treating
predictions that are consistent AND confident as reliable, and marking i)
everything else (=inconsistent OR unconfident) as unreliable, or ii) only
inconsistent AND unconfident pixels as unreliable, while ignoring the
remaining pixels that do not fall into either bucket.

In both cases, we observe low mIoU’s of 18.22 and 19.82, respectively. Note
that we skip the remaining combination (reliable=consistent OR confident,
unreliable=inconsistent OR unreliable) as it would lead to marking certain
pixels as both reliable and unreliable and is thus ill-defined. Altogether, we
find that consistency and confidence complement each other when used
individually – only selecting pixels that are both consistent AND confident is
a highly restrictive criterion that leads to selecting a very small percentage
of pixels for self-training (improving the precision with respect to
correctness but at a large cost to the recall). Secondly, the small percentage
of pixels being marked as reliable leads to a large percentage of the
remaining unreliable predictions not having a reliable neighbor from which to
selectively interpolate a pseudolabel, which further hinders the pseudolabel
refinement strategy. Both of these contributing factors ultimately lead to
poor performance.

####  6.2.6 Additional qualitative examples

In Fig. 9 we include additional qualitative segmentation results achieved by
S4T and TENT [5].

Figure 9: GTA5$\to$Cityscapes: Qualitative segmentation results of the source
model, TENT [5], and S4T.

###  6.3 Additional training details

####  6.3.1 Code

We will publicly release all our code and pretrained models.

####  6.3.2 Datasets

GTA5: Images in the GTA5 dataset were collected from the GTA5 game and are
distributed under the MIT license which makes it free to use for research.

SYNTHIA: SYNTHIA is distributed under a Creative Commons license which allows
free use for non-commercial purposes.

Cityscapes: The Cityscapes dataset license allows free use for non-commercial
purposes.

Dark Zurich: The Dark Zurich dataset is released under a Creative Commons
license which allows free use for non-commerical purposes.

To the best of our knowledge, none of the above datasets contain personally
identifiable information (in fact, GTA5 and SYNTHIA comprise of purely
synthetic images), or offensive content.

####  6.3.3 Baselines

We briefly describe the source-equipped baselines from Liu  _et al_. [6], that
we report in Sec. 4, and refer readers to the original work for more details.

i) MinEnt and AdvEnt [11]: Performs conditional entropy minimization and
adversarial entropy minimization of the model’s predictions over unlabeled
target data.

ii) AdaptSegNet [25]: Performs adversarial learning in the output space for
domain alignment.

iii) CBST [61]: Adopts iterative self-training in combination with a
pseudolabel-based class-balancing strategy for adaptation.

iv) MaxSquares [61]: Proposes a “maximum squares” loss which leads to more
balanced gradients when self-training on poorly-classified target samples than
traditional entropy minimization.

###  6.4 Clarification of self-supervision used in S4T

We clarify the “self-supervised” portion of our selective self-training loss
objective ($L_{SST}$) in Eq.(3) from the main paper (replicated below as Eq. e
for convenience):

$L_{SST}(\mathbf{x}_{ij})=\underbrace{r_{ij}\lambda_{\tilde{V}_{ij}}L_{CE}(\tilde{\mathbf{p}}_{ij},\tilde{V}_{ij})}_{\text{reliable}}+\underbrace{\alpha(1-r_{ij})\Big{[}\underbrace{\lambda_{\hat{y}^{\text{int}}_{ij}}w_{ij}^{\text{int}}L_{CE}(\tilde{\mathbf{p}}_{ij},\hat{y}^{\text{int}}_{ij})}_{\text{reliable
neighborhood}}+\underbrace{\mathbbm{1}[w_{ij}^{\text{int}}=0]L_{\text{entmax}}(\tilde{\mathbf{p}}_{ij})\Big{]}}_{\text{unreliable
neighborhood}}}_{\text{unreliable}}$

(e)

While this formulation resembles a standard (selective) self-training
objective, the reliability mappings $r_{ij}$ are generated in a fully self-
supervised manner by combining pixel-level consistency across predictive views
with intrinsic model confidence (Eq.(1) in the main paper). This combination
gives us our full approach of Self-Supervised Selective Self-Training (S4T).