capbtabboxtable[][]

11institutetext: Information Technology University, Lahore, Pakistan
11email<EMAIL_ADDRESS>
# Learning from Scale-Invariant Examples for Domain Adaptation in Semantic
Segmentation

M.Naseer Subhani  Mohsen Ali

###### Abstract

Self-supervised learning approaches for unsupervised domain adaptation (UDA)
of semantic segmentation models suffer from challenges of predicting and
selecting reasonable good quality pseudo labels. In this paper, we propose a
novel approach of exploiting scale-invariance property of the semantic
segmentation model for self-supervised domain adaptation. Our algorithm is
based on a reasonable assumption that, in general, regardless of the size of
the object and stuff (given context) the semantic labeling should be
unchanged. We show that this constraint is violated over the images of the
target domain, and hence could be used to transfer labels in-between
differently scaled patches. Specifically, we show that semantic segmentation
model produces output with high entropy when presented with scaled-up patches
of target domain, in comparison to when presented original size images. These
scale-invariant examples are extracted from the most confident images of the
target domain. Dynamic class specific entropy thresholding mechanism is
presented to filter out unreliable pseudo-labels. Furthermore, we also
incorporate the focal loss to tackle the problem of class imbalance in self-
supervised learning. Extensive experiments have been performed, and results
indicate that exploiting the scale-invariant labeling, we outperform existing
self-supervised based state-of-the-art domain adaptation methods.
Specifically, we achieve 1.3% and 3.8% of lead for GTA5 to Cityscapes and
SYNTHIA to Cityscapes with VGG16-FCN8 baseline network.

##  1 Introduction

Deep learning based semantic segmentation models [yu2017dilated,
chen2018deeplab, zhao2017pyramid, zhao2018icnet] have made considerable
progress in last few years. Exploiting hierarchical representation, these
models report state-of-the-art results over the large datasets. However, these
models do not generalize well; when presented with out of domain images, their
accuracies drops. This behavior is attributed to the shift between the source
domain, one over which model has been trained, and target, over which its
being tested. Most of semantic segmentation algorithms are trained in a
supervised fashion, requiring pixel-level, labor extensive and costly
annotations. Collecting such fine-grain annotations for every scene variation
is not feasible. To avoid this pain-sticking task, road scene segmentation
algorithm use synthetic but photo-realistic datasets, like GTA5
[richter2016playing], Synthia [ros2016synthia], etc., for training. However,
they are evaluated on the real datasets like Cityscapes
[cordts2016cityscapes], thus amplifying the domain shift.

Figure 1:  Scale-invraince property of semantic segmentation model Original
image and patch extracted from it and resized, are assigned same semantic
labels by the model f at the corresponding locations. Left: An image $x_{s}$
from the source domain, labels assigned to it by model f. $x_{s}$ belongs to
the source domain. Self-entropy map $E$ shows small values. Yellow box on
$x_{s}$ indicate patch location Right: Extracted patch resized to original
image size. Assigned labels are similar to ones of original and self-entropy
is similar that of original image.

Over the years, many unsupervised domain adaptation (UDA) methods have been
proposed to overcome the domain shift, employing adversarial learning
[chen2017no, ganin2016domain, sankaranarayanan2017unsupervised,
zhu2017unpaired], self-supervised learning [zhao2017pyramid,
zou2018unsupervised, Iqbal_2020_WACV], etc. or their combination. Where
adversarial learning methods are dependent upon how good (input, feature or
output) translation could be performed, self-supervised learning methods have
to deal with challenges of generating so-called good quality pseudo-labels and
selection of confident images for the learning from the domain.

In this paper we propose a novel method of generating pseudo-labels for self-
supervised adaptation for semantic segmentation, by exploiting scale-
invariance property of the model. Our proposed solution is based on an
assumption that regardless of the size of an object in the image, the model’s
prediction should not be change, as shown in Fig. 1. To support our algorithm,
we introduce three other novel components to be incorporated in the self-
supervised method. A class-based sorting mechanism image selection process to
identify images that should be used for the self-learning. To filter out
pixels with non-confident pseudo-labels from learning process, we design an
automatic process of estimating class specific dynamic entropy-threshold
allowing ”easy” classes to have tighter threshold than the ones that are
”difficult” to adapt. To further reduce the effect of class imbalance over
adaptation process, we also incorporate the focal loss [lin2017focal] in our
loss. Below we define the concept of scale-invariance.

Scale-invariance:  In general one can assume that depending on the camera
location, pose and other parameters, objects in images will appear at varying
sizes. In the road scene imagery, such as GTA5, Cityscapes, etc., due to
movement of the vehicle and dynamic nature of environment, objects and other
scene elements (like road, building) appear at multiple scales. These
variations are readily visible in Fig. 2. Its reasonable to assume that the
semantic segmentation model trained on such dataset that will assign objects
and stuff with same semantic labels regardless of their size. This could be
seen in Fig. 1, where when an image and a resized patch extracted from same
image are presented to segmentation model we get similar semantic labels at
(almost all) corresponding regions. For both, image and resized patch, self-
entropy is also indicating that the decision was made with low uncertainty.
Semantic segmentation model, when presented with an image, from the out of
source but somewhat visually similar domain, and the patches extracted from
that image, we see considerable difference between the labels assigned for
patches and ones assigned to corresponding areas of original image.
Comparative increase in the self-entropy indicates that labels assigned to
patches are not reliable . In this work, as shown in Fig. 2, we propose to use
semantic labels assigned to the image to create pseudo-labels of corresponding
patches. Our objective is to preserve the scale-invariance property of the
semantic segmentation model and use it to direct our adaptation process.

Figure 2: Objects and scene-elements exhibit the scale variations naturally in
road scene images, as shown in the frames sampled from Cityscapes
[cordts2016cityscapes] and GTA5[richter2016playing] datasets. As the vehicle
moves, near by objects and other scene elements might become afar or vice-
versa, resulting in scale changes. Matching color boxes highlight changing
size of cars, buildings, and other regions as vehicle moves.

We summarize our contribution as bellow.

  * •

We propose a novel approach of exploiting scale-invarince property of the
model to generate pseudo-labels for the self-supervised domain adaptation of
semantic segmentation model.

  * •

Class specific dynamic entropy thresholding is introduced so that pixels
belonging to classes at different adaptation stage could be judged differently
when being made included in the loss function.

  * •

To eliminate the effect of the class imbalance problem, we incorporate the
focal loss to boost the performance of smaller classes. And Class-based target
image sorting algorithm is proposed so that selected images have equal
representation of all the classes.

Although, part of our algorithm is generic, we show our results on the
adaptation from synthetic to real road scene segmentation. We report state-of-
the-art results over the GTA to Cityscapes and Synthia to Cityscapes for the
self-supervised based domain adaptation algorithms. VGG16 [simonyan2014very]
and ResNet101 [he2016deep] are used as our baseline architectures.

##  2 Related works

Semantic Segmentation: There is an intensive amount of research has been done
in semantic segmentation due to its importance in the field of computer
vision. State of the art methods in semantic segmentation have gained huge
success for their contribution. Recently, many researchers have proposed
algorithm for semantic segmentation such as DRN (Dilated Residual Network)
[yu2017dilated], DeepLab [chen2018deeplab] etc. [badrinarayanan2017segnet,
zhao2017pyramid, wang2018understanding].yu2017dilated [yu2017dilated] have
proposed a dilated convolution neural network in semantic segmentation to
increase the depth resolution of the model without effecting its receptive
field. In this work, we have utilized FCN8s[long2015fully] with
VGG16[simonyan2014very] and DeepLab [chen2017deeplab] with ResNet101
[he2016deep] as our baseline architectures of semantic segmentation.

Domain Adaptation: Domain adaptation is a popular research area in computer
vision, especially in classification and detection problems. The goal of
domain adaptation is to minimize the distribution gap between source and
target domain. Many of the algorithms have already developed for domain
adaptation like [zou2018unsupervised, vu2018advent,
sankaranarayanan2018learning, hoffman2017cycada, tsai2018learning,
zhang2017curriculum, hoffman2016fcns, zhu2017unpaired, Iqbal_2020_WACV]. In
this paper, we are focused in self-supervised domain adaptation to tackle the
problem of domain diversity. Previous methods have been applied Maximum Mean
Discrepancy (MMA) [quinonero2008covariate] to minimize the distribution
difference. Recently, there has been an enormous interest in developing domain
adaptation methods with the help of unsupervised and semi-supervised learning.

Adversarial Domain Adaptation in Semantic Segmentation: Adversarial training
for unsupervised domain adaptation is the most explored approach for semantic
segmentation. hoffman2016fcns [hoffman2016fcns] are the first ones to
introduce domain adaptation in semantic segmentation. vu2018advent
[vu2018advent] have proposed an entropy minimization, based on domain
adaptation in which they have minimized the self-entropy with the help of
adversarial learning. In [tsai2018learning], they have applied adversarial
learning at the output space to minimize the distribution at the pixel level
between the source and the target domain. chen2018road [chen2018road] presents
Reality-Oriented-Adaptation-Network (ROAD) to learn invariant features of
source and target domain by target guided distillation and spatial-aware
adaptation. luo2019taking [luo2019taking] has also introduced a categorical-
level adversarial network (CLAN) in which they have aligned the features of
each class by adaptive adjusting the weight on adversarial loss specific to
each class. There are other methods with the generative part for adversarial
training in semantic segmentation. In generative methods, they are trying to
generate the target images with a condition of the source domain.
zhu2017unpaired [zhu2017unpaired] have proposed a pixel level adaptation to
generate image similar in visual perception with target distribution. In
[hoffman2017cycada], they have used pixel level and feature level adaptation
to overcome the distribution gap between the source and the target domain.
They incorporate cycle consistency loss to generate the target image condition
on the source domain. They have also utilized the feature space adaptation and
generate target images from the source features and vice-versa.

Self-Supervised Domain Adaptation in Semantic Segmentation: The idea behind
self-supervised learning is to adapt the model by the pseudo labels generated
for unlabeled data from the previous state of the model. laine2016temporal
[laine2016temporal] proposed a method of self-supervised learning from the
assembling of the output from different models and latter train the model by
generating pseudo labels of unlabeled data. tarvainen2017mean
[tarvainen2017mean] developed an algorithm based on a teacher network where
the model is adapted by averaging the different weights for better performance
on the target domain. Recently, self-supervised learning has also gained
popularity in the semantic segmentation task. zou2018unsupervised
[zou2018unsupervised] proposed a class-balanced-self-training (CBST) for
domain adaptation by generating class-balanced pseudo-labels from images which
were assigned labels with most confidence by last state of model. To help
guide the adaptation, spatial priors were incorporated. du2019ssf[du2019ssf]
have also contributed their research in self-supervised learning by generating
pseudo labels with a progressive reliable strategy. They have excluded less
confident classes with a constant threshold and have trained the model on
generated pseudo labels. In this research, we filter out the less confident
classes by applying a dynamic threshold that is calculated for each class
separately during the training process. lian2019constructing
[lian2019constructing] have proposed a self-motivated pyramid curriculum
domain adaptation (PyCDA) for semantic segmentation. They have included the
curriculum domain adaptation by constructing the pyramid of pixel squares at
different sizes, which has included the image itself. The model trained on
these pyramids of the pixel by capturing local information at different
scales. Iqbal and Ali [Iqbal_2020_WACV]’s spatially independent and
semantically consistent (SISC) pseudo-generation method could be closest to
our work. However, they only explore the spatial invariance by creating
multiple translated versions of same image. Since they don’t have knowledge of
which version has results in better inference they aggregate inference
probabilities from all to create a single version, leading to smoothed out
pseudo-labels. We on the other hand, define a relationship between the scale
of the image and the self-entropy; therefore instead of aggregating we use the
inference for image of original scale to create pseudo-labels for the up-
scaled patch extracted from same image. Along with it, we present a
comprehensive strategy of overcoming class imbalance and selecting the
reliable psuedo-labels.

##  3 Methodology

In this section, we briefly describe our propose domain adaptation algorithm
by learning from self-generated scale-invariant examples for semantic
segmentation. In this work, we assume that the predictions of these confident
images on target data are the approximation of their actual labels.

###  3.1 Preliminaries

Let $X_{S}$ be set of images belonging to the source domain, such that for
each image $x_{s}\in{\mathbb{R}}^{H\times W\times 3}$, in the source domain we
have respective ground-truth one-hot encoded matrix
$y_{s}\in{\mathbb{R}}^{H\times W\times C}$. Where $C$ is the number of classes
and $H\times W$ is the spatial size of the image. Similarly, let $X_{T}$ be
set of images belonging target domain. We train a fully convolution neural
network, f, in a supervised fashion over the source domain for the task of
semantic segmentation. Let $P=\textbf{{f}}(x)$ be soft-max output volume of
size $H\times W\times C$, representing predicted semantic class probabilities
for each pixel. The segmentation loss for any image $x$ with the given ground-
truth labels $y$ and predicted probabilities $P$ is given by

$\mathcal{L}_{seg}(x,y)=-\sum_{h,w,c}^{H,W,C}y^{h,w,c}log(P^{h,w,c})$ (1)

In later cases to increase readability we just use $h,w,c$ with summation
sign, to indicate the summation over total height, width and channels. Source
model f has been trained by minimizing
$\mathcal{L}_{seg}^{S}=\sum_{s}^{S}\mathcal{L}_{seg}(x_{s},y_{s})$.

For target domain, since we do not have ground-truth labels, self-supervised
learning method requires us to generate pseudo-labels. Let $x_{t}\in X_{T}$ be
an image in the target domain, $P_{t}=\textbf{{f}}(x_{t})$ be output
probability volume, one hot encoded pseudo-labels $\hat{y}_{t}$ could be
generated by assigning label at each pixel to the class with maximum predicted
probability. Since, source model is not accurate on the target domain,
therefore a binary map $F_{t}\in{\mathbb{B}}^{H\times W}$ is defined to select
the pixels whose prediction loss has to be back-propagated.

$\mathcal{L}_{seg}(x_{t},\hat{y}_{t})=-\sum_{h,w,c}^{H,W,C}F_{t}^{h,w}\hat{y}_{t}^{h,w,c}log(P_{t}^{h,w,c})$
(2)

In general, for self-supervised learning, we minimize the loss in Eq. 2 over
the selected subset of images from the target domain.

###  3.2 Class-Based Sorting for Target Subset Selection

To train the model with self-supervised learning, we need to extract the
pseudo-labels which are reliable. A binary filter defined in Eq. 2, helps
select pixels with so-called ǵoodṕseudo-labels, however, does not give us
global view of how good are predictions in the whole image. Calculating an
average of maximum probability per location of $\hat{y}_{t}$ can help us
define the confidence of predictions on the $x_{t}$, for readability we call
it confidence of image $x_{t}$. A subset selected on the base of the above
defined confidence can lead to a class-imbalance with more images with pseudo-
labels belonging to large and frequently appearing classes. That in turn leads
to adaptation failing for the smaller objects or infrequent classes. We design
a class based image subset selection process from the target domain (Algo. 1)
to mitigate this effect.

Instead of calculating confidence for each image globally, we calculate the
confidence with respect to each class c for every image in target data XT. For
each class, $X_{T}$ is sorted with respect to the class specific confidence
$U_{c}$ and a subset, of size $p$, is selected. Union of these subsets form
our confident target images subset $X_{t}^{{}^{\prime}}$, note that repeated
entries are removed. The algorithm of class-based sorting shown in Algorithm
1. For $X^{{}^{\prime}}_{t}$ the model prediction are relatively of more
confidence than rest of the set and can be utilized to adapt the model by
self-supervised learning.

Input : Model f(w), Target data Xt, portion p

Output : Confident images X′t of target domain , Entropy threshold hc

1 for  _t = 1 to T_ do

2 P_x_t = f (w,x_t)

3 M_P_x_t = max (P_x_t, axis = 0)

4 A_P_x_t = argmax (P_x_t, axis = 0)

5 for  _c = 1 to C_ do

6 M_P_x_t,c = M_P_x_t [ A_P_x_t== c ]

7 U_c = [ U_c, mean (M_P_x_t,c) ]

8 X_t,c = [ X_t,c, x_t ]

9

10 end for

11

12 end for

13for  _c = 1 to C_ do

14 X_t,c,sort = sort (X_t,c w.r.t U_c, descending order)

15 len_th = length (X_t,c,sort) ×(p/C) $\rightarrow$ (p/C) is the portion of
class c

16 X′t = [ X′t, X_t,c,sort [ 0 : len_th - 1 ] ]

17 Calculate hc for each class

18 x_l = X_t,c,sort [ len_th - 1 ]

19 P_x_l = f (w,x_l )

20 A_P_x_l = argmax (P_x_l, axis = 0)

21 E_P_x_l = entropy (P_x_l) $\rightarrow$ normalized to [0 ,1]

22 h_c = mean ( E_P_x_l [ A_P_x_l == c ] )

23 end for

return X′t , hc

Algorithm 1 Class-Based Sorting Figure 3: Exploiting Scale-Invariance property
for generated pseudo labels: For an image ${x^{_{t}}}$ belonging to target
domain and its zoomed-in version scale-invariance property is violated. (a)
Image $x^{_{t}}$ and its extracted patch Ii. (b) High self-entorpy values
computed from the output probabilities indicate source model f is not
confident about the labels assigned to resized patch. (c) comparison of the
labels indicate violation of scale-invariance property (d) Since original
image exhibit low self-entropy we can use predictions over it as the pseudo-
labels for the resized patch.

###  3.3 Dynamic entropy threshold for class dependent filter selection

The class based sorting takes in consideration all the pixels and does not
make distinction between pixel-wise reliable and unreliable predictions. We
define reliable or good predictions as by how low is the self-entropy of the
prediction. If the entropy is low the prediction is more confident, if its
high it means that the model is undecided which semantic label should be
assigned to the pixel. Let,
$P(x^{{}^{\prime}}_{t})=\textbf{{f}}(x^{{}^{\prime}}_{t})$ be the predicted
probability volume, and
$E_{x^{{}^{\prime}}_{t}}=-\sum_{c}\,P_{c}(x^{\prime}_{t})\,\,\textit{log}(P_{c}(x^{\prime}_{t}))$
be entropy computed at each location. A binary filter map
$F_{x^{{}^{\prime}}_{t}}$ is generated by thresholding the entropy at every
location, by a class specific threshold.

$F_{x^{{}^{\prime}}_{t}}(h,w)=\left\\{\begin{array}[]{ll}1&E_{x^{{}^{\prime}}_{t}}(h,w)\leq
h_{c}\text{\leavevmode\nobreak\ \leavevmode\nobreak\ ;\leavevmode\nobreak\
\leavevmode\nobreak\ where\leavevmode\nobreak\
}c=argmax(P(x^{{}^{\prime}}_{t})(h,w))\\\
0&\text{otherwise}\end{array}\right.$ (3)

Instead of being $h_{c}$ a global and constant hyper-parameter, $h_{c}$ is
different for every class and depends upon predicted probabilities pixels
belonging to that class in the selected confident set $X_{t}^{{}^{\prime}}$.
As the adaptation for that class improves the filter selection for that class
becomes more tighter (Algo. 1).

###  3.4 Self-Generated Scale-Invariant Examples

Based on a reasonable assumption that a source domain consists of images with
scene elements and objects of same class appearing in scale variations, we
claim that model trained on such dataset should label same object with same
semantic label regardless of its size in the image. We define this as scale-
invariance property of the model. As shown in Fig. 3 such a property is
violated when target domain images are presented to the source model and could
be used to guide the domain adaptation process. Specifically, lets assume
$x^{{}^{\prime}}_{t}\in{\textit{X${}^{\prime}$}}\textsubscript{t}$ be the one
of the selected images, $F_{x^{{}^{\prime}}_{t}}$ be the binary mask, and
$P(x^{{}^{\prime}}_{t})=\textbf{{f}}(x^{{}^{\prime}}_{t})$ is the output
probability volume. Let $\textbf{{R}}(x^{{}^{\prime}}_{t},rec_{i})$ be the
operation applied on $x^{{}^{\prime}}_{t}$ to extract $i^{th}$ patch from
location $rec_{i}=(r_{i},c_{i},w_{i},h_{i})$ and resized to spatial size of
$H\times W$. Then we can define,
$I_{i}^{t}=\textbf{{R}}(x^{{}^{\prime}}_{t},rec_{i})$,
$F_{x^{{}^{\prime}}_{t}}^{i}=\textbf{{R}}(F_{x^{{}^{\prime}}_{t}},rec_{i})$
and
$P_{x^{{}^{\prime}}_{t}}^{i}=\textbf{{R}}(P_{x^{{}^{\prime}}_{t}},rec_{i})$ be
the corresponding extracted and resized versions. We compute $\hat{y}_{t}^{i}$
is the one-hot encoded pseudo labels created from
$P_{x^{{}^{\prime}}_{t}}^{i}$. Then loss for violating the scale invariance
could be computed by Eq. 4.

$\mathcal{L}_{seg}(I_{i}^{t},\hat{y}_{t}^{i})=-\sum_{h,w,c}^{H,W,C}F_{t}^{i,h,w}\hat{y}_{t}^{i,h,w,c}log(\textbf{{f}}(I_{i}^{t})^{h,w,c}).$
(4)

Figure 4: Algorithm Overview: Our algorithm consists of three main steps. (a)
First, we have calculated the confidence of each target images Xt with
reference to each class c. We have sort out these images X′t,c of each class c
in descending order on the basis of their confidence value. After that, we
have selected the top portion from these sorted images X′t,c to form confident
target data X′t. (b) Second, we have extracted the random patches Ii from each
confident images x′t of target domain Xt. These patches are the scale-
invariant with full-sized image. The model performs inconsistent on these
patches and predict an output with high entropy prediction. To filter out the
less confident pixels we have generated a filter map for each confident images
x′t by calculating their entropy with the help of threshold hc for each class
c. (c) Third, we have trained the model by given loss function on these scale-
invariant examples with their pseudo labels that are generated from the
previous state of the model.

###  3.5 Leveraging Focal Loss for Class-Imbalance

Self-supervised approach for domain adaptation highly dependent on information
represented in selected confident images of the target domain. Biased
distribution, i.e. number of pixels per class, in the road scenes creates a
class imbalance problem. Even after the class based sorting (Sec. 3.2) and
class dependent entropy thresholding, classes with high volume of pixels in
target dataset (such as road, building, vegetation, etc.) end up having more
contribution towards loss function. Classes which appear infrequently and/or
have less volume of pixels per image will contribute less and hence adaptation
will be slow. To eliminate the effect of class imbalance problem, we
incorporate the focal loss [lin2017focal], so that cross-entropy function of
each pixel is weighted by the based on pixel confidence. Focal loss balanced
the loss for each pixel based on their confident level. This approach of
applying focal loss balance the learning process of self-supervised learning
equally to each class. In this work, we apply focal loss during the training
of scale-invariant examples. Eq. 5 shows the formulation of focal loss.

$\mathcal{L}_{FL}(I_{i}^{t},\hat{y}_{t}^{i})=-\sum_{h,w,c}^{H,W,C}\,\hat{y}_{t}^{i,h,w,c}\,\,\textbf{{log}}(\textbf{{f}}(I_{i}^{t})^{h,w,c})(1-\textbf{{f}}(I_{i}^{t})^{h,w,c})^{\gamma}\\\
$ (5)

Where $\gamma$ is the hyperparameter that controls the focus and generally
have value between 0 to 5. Low value bring it closer cross-entropy and high
value focusing only on the hard examples. We set $\gamma$ to middle value,3.

Table 1: Results from GTA5 to Cityscapes. We report the results of our
algorithm by presenting IoU of each class and also overall mIoU. ‘V’ and ‘R’
represents VGG-FCN8 and ResNet101 as our baseline network. ‘ST’ and ‘AT’
represents self-training and adversarial training respectively. We report the
best results in bold.

GTA5 to Cityscapes |
---|---
|  Arch. |  Meth. |  road |  sidewalk |  building |  wall |  fence |  pole |  light |  sign |  veg |  terrain |  sky |  person |  rider |  car |  truck |  bus |  train |  mbike |  bike | mIoU
FCN wild [hoffman2016fcns] | V | AT | 70.4 | 32.4 | 62.1 | 14.9 | 5.4 | 10.9 | 14.2 | 2.7 | 79.2 | 21.3 | 64.6 | 44.1 | 4.2 | 70.4 | 8.0 | 7.3 | 0.0 | 3.5 | 0.0 | 27.1
CyCADA [hoffman2017cycada] | V | AT | 85.2 | 37.2 | 76.5 | 21.8 | 15.0 | 23.8 | 22.9 | 21.5 | 80.5 | 31.3 | 60.7 | 50.5 | 9.0 | 76.9 | 17.1 | 28.2 | 4.5 | 9.8 | 0.0 | 35.4
ROAD [chen2018road] | V | AT | 85.4 | 31.2 | 78.6 | 27.9 | 22.2 | 21.9 | 23.7 | 11.4 | 80.7 | 29.3 | 68.9 | 48.5 | 14.1 | 78.0 | 19.1 | 23.8 | 9.4 | 8.3 | 0.0 | 35.9
| R | AT | 76.3 | 36.1 | 69.6 | 28.6 | 22.4 | 28.6 | 29.3 | 14.8 | 82.3 | 35.3 | 72.9 | 54.4 | 17.8 | 78.9 | 27.7 | 30.3 | 4.0 | 24.9 |  |

12.6 & 39.4 CLAN [luo2019taking] V AT 88.0 30.6 79.2 23.4 20.5 26.1 23.0 14.8
81.6 34.5  72.0 45.8 7.9 80.5 26.6 29.9 0.0 10.7 0.0 36.6 R AT 87.0 27.1 79.6
27.3 23.3 28.3 35.5 24.2 83.6 27.4 74.2 58.6 28.0 76.2  33.1  36.7  6.7  31.9
31.4 43.2 Curr. DA [zhang2017curriculum] V AT 74.9 22.0 71.7 6.0 11.9 8.4 16.3
11.1 75.7 13.3 66.5 38.0 9.3 55.2 18.8 18.9 0.0 16.8 14.6 28.9 AdvEnt
[vu2018advent] V AT,ST 86.9 28.7 78.7 28.5 25.2 17.1 20.3 10.9 80.0 26.4 70.2
47.1 8.4 81.5 26.0 17.2 18.9 11.7 1.6 36.1 R AT,ST 89.4 33.1 81.0 26.6 26.8
27.2 33.5 24.7 83.9 36.7 78.8 58.7 30.5 84.8 38.5 44.5 1.7 31.6  32.4 45.5
SSF-DAN [du2019ssf] V ST,AT 88.7 32.1 79.5 29.9 22.0 23.8 21.7 10.7 80.8 29.8
72.5 49.5 16.1 82.1 23.2 18.1 3.5 24.4 8.1 37.7 R ST,AT 90.3 38.9 81.7  24.8
22.9  30.5  37.0  21.2  84.8  38.8 76.9  58.8  30.7  85.7 30.6  38.1  5.9
28.3  36.9  45.4

CBST [zou2018unsupervised] V ST 66.7 26.8 73.7 14.8 9.5 28.3 25.9 10.1 75.5
15.7 51.6 47.2 6.2 71.9 3.7 2.2 5.4 18.9 32.4  30.9

PyCDA[lian2019constructing] V ST 86.7 24.8  80.9 21.4  27.3 30.2 26.6 21.1
86.6 28.9 58.8 53.2  17.9 80.4 18.8 22.4 4.1 9.7 6.2 37.2 R  ST  90.5 36.3
84.4 32.4 28.7 34.6 36.4  31.5  86.8 37.9  78.5  62.3 21.5  85.6  27.9  34.8
18.0 22.9 49.3 47.4 LSE V ST 80.2 26.6 78.1 28.4 17.3 19.8 27.6 12.2 78.6 23.6
72.0 50.8 14.8 81.2 22.5 20.3 4.0 20.1 14.5 36.4

LSE + FL V ST 86.0 26.0 76.7 33.1 13.2 21.8 30.1 16.5 78.8 25.8 74.7 50.6
18.7 81.8 22.5 30.5 12.3 16.9 25.4 39.0

LSE + FL R ST 90.2 40.0 83.5 31.9  26.4  32.6  38.7 37.5 81.0  34.2 84.6 61.6
33.4 82.5 32.8 45.9 6.7  29.1 30.6  47.5

###  3.6 Adaptation

During adaptation, for each round $r$, we perform class based sorting of
target dataset to create subset $X^{{}^{\prime}}_{T}$. For each
$x^{{}^{\prime}}_{t}\in X^{{}^{\prime}}_{T}$, $k$ patches are extracted. Out
total loss is defined as

$\mathcal{L}_{LSE}=\sum_{x_{s}\in
X_{S}}\mathcal{L}_{seg}(x_{s},y_{s})+\mathcal{L}_{adapt}(X^{{}^{\prime}}_{T})$
(6)

where first term is cross entropy loss over source domain Xs to prevent the
model from forgetting the previous knowledge. Second term, is adaptation loss
computed as summation of focal loss Eq. 5 and segmentation loss (Eq. 4),
trying to minimize loss of violating scale-invariance.

$\mathcal{L}_{adapt}(X^{{}^{\prime}}_{T})=\sum_{x^{{}^{\prime}}_{t}\in
X^{{}^{\prime}}_{T}}\sum_{i}^{k}\beta\mathcal{L}_{FL}(I_{i}^{t},\hat{y}_{t}^{i})+\mathcal{L}_{seg}(I_{i}^{t},\hat{y}_{t}^{i}),$
(7)

$\beta$ is a hyperparameter that controls the effect of focal loss on self-
supervised domain adaptation. In the end, we adapt the model with an iterative
process for each rounds r.

Table 2: mIoU (16-categories) and mIoU* (13-categories) results from SYNTHIA
to Cityscapes. ‘V’ and ‘R’ represent VGG-FCN8 and ResNet101 as our baseline
network. ‘ST’ and ‘AT’ represent self-training and adversarial training,
respectively. We have reported the highest results in bold.

SYNTHIA to Cityscapes |
---|---
|  Arch. |  Meth. |  road |  sidewalk |  building |  wall |  fence |  pole |  light |  sign |  veg |  sky |  person |  rider |  car |  bus |  mbike |  bike | mIoU | mIoU*
ROAD [chen2018road] | V | AT | 77.7 | 30.0 | 77.5 | 9.6 | 0.3 | 25.8 | 10.3 | 15.6 | 77.6 | 79.8 | 44.5 | 16.6 | 67.8 | 14.5 | 7.0 | 23.8 | 36.2 | -
CLAN [luo2019taking] | V | AT | 80.4 | 30.7 | 74.7 | - | - | - | 1.4 | 8.0 | 77.1 | 79.0 | 46.5 | 8.9 | 73.8 | 18.2 | 2.2 | 9.9 | - | 39.3
| R | AT | 81.3 | 37.0 | 80.1 | - | - | - | 16.1 | 13.7 | 78.2 | 81.5 | 53.4 | 21.2 | 73.0 | 32.9 | 22.6 | 30.7 | - | 47.8
Curr. DA [zhang2017curriculum] | V | AT | 65.2 | 26.1 | 74.9 | 0.1 | 0.5 | 10.7 | 3.7 | 3.0 | 76.1 | 70.6 | 47.1 | 8.2 | 43.2 | 20.7 | 0.7 | 13.1 | - | 34.8
AdvEnt [vu2018advent] | V | AT,ST | 67.9 | 29.4 | 71.9 | 6.3 | 0.3 | 19.9 | 0.6 | 2.6 | 74.9 | 74.9 | 35.4 | 9.6 | 67.8 | 21.4 | 4.1 | 15.5 | 31.4 | 36.6
| R | AT,ST | 85.6 | 42.2 | 79.7 | 8.7 | 0.4 | 25.9 | 5.4 | 8.1 | 80.4 | 84.1 | 57.9 | 23.8 | 73.3 | 36.4 | 14.2 | 33.0 | 41.2 | 48.0
SSF-DAN [du2019ssf] | V | ST,AT | 87.1 | 36.5 | 79.7 | - | - | - | 13.5 | 7.8 | 81.2 | 76.7 | 50.1 | 12.7 | 78.0 | 35.0 | 4.6 | 1.6 | - | 43.4
| R | ST,AT | 84.6 | 41.7 | 80.8 | - | - | - | 11.5 | 14.7 | 80.8 | 85.3 | 57.5 | 21.6 | 82.0 | 36.0 | 19.3 | 34.5 |  |

- & 50.0 CBST [zou2018unsupervised] V ST 69.6 28.7 69.5  12.1 0.1 25.4 11.9 13.6 82.0 81.9  49.1 14.5 66 6.6 3.7 32.4  35.4 36.1

PyCDA[lian2019constructing] V ST 80.6 26.6 74.5 2.0 0.1 18.1 13.7 14.2 80.8
71.0 48.0 19.0 72.3 22.5 12.1 18.1 35.9 42.6 R ST 75.5 30.9 83.3 20.8 0.7 32.7
27.3 33.5 84.7 85.0 64.1 25.4 85.0 45.2 21.2 32.0 46.7 53.3 LSE V ST 82.2 38.4
79.0 2.2 0.5 25.3 9.6 20.7 78.6 77.4 51.7 18.0 72.9 21.7 11.1 22.2 38.2 44.9
LSE + FL V ST 83.6 39.6  79.3 3.6  0.9 25.3 14.1  26.1 79.4 76.5 51.0  18.1
75.7 22.5 12.0 32.1 40.0 47.0 LSE + FL R ST 82.9 43.1 78.1 9.3 0.6  28.2 9.1
14.4 77.0 83.5 58.1 25.9 71.9 38.0 29.4 31.2 42.6 49.4

##  4 Experiments and Results

In this section, we provide implementation details and experimental setup of
our proposed approach. We evaluate the proposed self-supervised learning
strategy on standard synthetic to real domain adaptation setup and present a
detailed comparison with state-of-the-art methods.

Figure 5: Qualitative results of our algorithm with self-supervised domain
adaptation for GTA5 to Cityscapes. For each example, we show images without
adaptation and with adaptation as our result. We also show the ground truth
for each image.

###  4.1 Experimental Details

Network Architecture: For a fair comparison we follow the standard practice of
using FCN-8s [long2015fully] with VGG16 and DeepLab-v2 [chen2017deeplab] with
ResNet-101 [he2016deep] as our baseline approaches. We have used pretrained
models for further adaptation towards the target domain

Datasets and Evaluation Metric: To evaluate the proposed approach, we have
used benchmark synthetic datasets, e.g., GTA5 [richter2016playing] and
SYNTHIA-RAND-CITYSCAPES [ros2016synthia] as our source domain datasets and
real imagery Cityscapes[cordts2016cityscapes] as our target domain dataset.
The GTA5 dataset consists of 24966 high resolution (1052 x 1914) densely
annotated images captured from the GTA5 game. Similarly, SYNTHIA contains 9400
labeled images with a spatial resolution of 760 x 1280. The Cityscapes
datasets has 2975 training images and 500 validation images. We use mean
intersection over union (mIoU) as the evaluation metric and evaluate the
proposed approach on compatible 19 and 16 classes for GTA to Cityscapes and
SYNTHIA to Cityscapes adaptation respectively. Due to GPU memory limitations
we use the highest spatial size of $512\times 1024$.

Implementation Details: We have used PyToch deep learning framework to
implement our algorithm with a Tesla k80 GPU having 12GB of memory. To select
number of high confident images for each class, we choose $p=0.1$ and after
each round increment it with 0.05. $k=4$ number of patches, of spatial size of
$256\times 512$, are chosen randomly and resized to $512\times 1024$. For
focal loss, we use $\gamma=3$ and $\beta=0.1$ in-order to focus on hard
examples. We used Adam optimizer [kingma2014adam] with learning rate and
momentum of 1x10-6 and 0.9 respectively.

###  4.2 Comparisons with state-of-the-art Methods

To compare with existing methods, we perform experiments of adapting to
Cityscapes from two different synthetic datasets, GTA5 and SYNTHIA. All
experiments were done under the standard settings.

GTA5 to Cityscapes: Table 3.5 shows the comparison of our result with existing
state of the art domain adaptation methods in semantic segmentation from GTA5
to Cityscapes respectively. Proposed approach reports state-of-the-art results
on VGG16-FCN8 [long2015fully] and ResNet101 [he2016deep], for self-training
based adaptation methods. It outperforms most of the non self-training methods
and complex methods too, and is comparative to state-of-the-art. We report the
results with and without the focal loss to see the effect on the model
regarding class balance adaptation. Due to focal loss, the small/infrequent
objects benefit specifically.

Figure 6: Effect of focal loss on each class after the first and the fourth
round of domain adaptation with self-supervised learning for semantic
segmentation, evaluated for GTA5 to Cityscape with VGG16-FCN8 baseline
network.

SYNTHIA to Cityscapes: Table 3.6 describes the quantitative results of LSE and
a detailed comparison with existing methods. Like previous methods
[Iqbal_2020_WACV], we report both the mIoU (16 classes) and mIoU* (13 classes)
for the classes compatible with Cityscapes. The LSE+FL performs comparative to
other complex methods based on adversarial learning, however, in self-training
setting LSE+FL shows 4.1% mIoU gain over state-of-the-art PyCDA
[zhao2017pyramid]

Performance Table
---
GTA5 to Cityscapes (VGG16-FCN8)
Method | Oracle | mIoU % | gap (%)
FCN wild [hoffman2016fcns] | 64.6 | 27.1 | -37.5
CyCADA [hoffman2017cycada] | 60.3 | 35.4 | -24.9
ROAD[chen2018road] | 64.6 | 35.9 | -28.7
CLAN[luo2019taking] | 64.6 | 36.6 | -28.0
AdvEnt [vu2018advent] | 61.8 | 36.1 | -25.7
SSF-DAN[du2019ssf] | 65.1 | 37.7 | -27.4
CBST [zou2018unsupervised] | 65.1 | 30.9 | -34.2
PyCDA[lian2019constructing] | 65.1 | 37.2 | -27.9
Ours | 60.3 | 39.9 | -21.3

Table 3: Comparisons of performance gap of adaptation algorithms vs oracle
scores

###  4.3 Analysis

To demonstrate the reasoning of the working principle for the proposed
algorithm, we evaluate different aspect of our algorithm. Effect of Focal
Loss: To verify the effect of focal loss on each class equally, we calculate
the number of images selected for each class after a few rounds. Focal loss
can affect the smaller classes for each class on different rounds, as shown in
Figure 6. The graph demonstrates the effect on different classes to balance
the effect of learning for self-supervised domain adaptation. For each class,
the Figure 6 shows three bars, red shows the number of images selected on the
first round of adaptation, whereas the orange and green are the corresponding
values of selected images after fourth round and with and without focal loss
respectively. It can be seen that the focal loss balances the selection
process especially for infrequent classes, by maximizing their prediction
probabilities.

Performance Gap: We also compare the performance of our algorithm using the
performance gap with other state-of-the-art methods of domain adaptation.
Table 3.5 shows the performance gap of different algorithms with their oracle
values. Our algorithm clearly shows the best results with a gap -21.3 as
compared to other algorithms we mentioned.

##  5 Conclusion

In this paper, we have proposed a novel approach of self-supervised domain
adaptation method by exploiting the scale-invariance properties of the
semantic segmentation model. In general images in dataset, especially road-
scene dataset, contains objects in varying sizes and scene elements closer and
far away from the. The scale invarance property of the model is defined as
ability to assign same semantic labels to scaled instance of the image or
parts of image as it will assign to the original image. In simple words
regardless of size variation of object it should be similarly semantically
labeled. We show that for the target domain this property is violated and
could be used to direct the adaptation label by using the pseudo-labels for
the original size images as pseudo-labels for the zoomed in region. Multiple
strategies were employed to counter the class imbalance problem and pseudo-
label selection problem. Class specific sorting algorithm is desinged to
select images from target dataset such that all classes are equally
represented at image level. Dynamic class dependent entropy threshold
mechanism is presented to allow classes at different levels of adaptation have
different threshold. Finally, a focal loss is introduced to guide the
adaptation process. Our experimenal results are competitive to state-of-the-ar
algorithms and outpeform state-of-the-art self-training methods.