# Domain Adaptive Semantic Segmentation by Optimal Transport

Yaqian Guo  Xin Wang  Shaoyi Du  Ce Li  Shihui Ying<EMAIL_ADDRESS>Department of Mathematics, Shanghai University, Shanghai, 200444, China
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University,
Xi’an 710049, China  College of Electrical and Information Engineering,
Lanzhou University of Technology, Lanzhou, 730050, China

###### Abstract

Scene segmentation is widely used in the field of autonomous driving for
environment perception, and semantic scene segmentation (3S) has received a
great deal of attention due to the richness of the semantic information it
contains. It aims to assign labels to pixels in an image, thus enabling
automatic image labeling. Current approaches are mainly based on convolutional
neural networks (CNN), but they rely on a large number of labels. Therefore,
how to use a small size of labeled data to achieve semantic segmentation
becomes more and more important. In this paper, we propose a domain adaptation
(DA) framework based on optimal transport (OT) and attention mechanism to
address this issue. Concretely, first we generate the output space via CNN due
to its superiority of feature representation. Second, we utilize OT to achieve
a more robust alignment of source and target domains in output space, where
the OT plan defines a well attention mechanism to improve the adaptation of
the model. In particular, with OT, the number of network parameters has been
reduced and the network has been better interpretable. Third, to better
describe the multi-scale property of features, we construct a multi-scale
segmentation network to perform domain adaptation. Finally, in order to verify
the performance of our proposed method, we conduct experimental comparison
with three benchmark and four SOTA methods on three scene datasets, and the
mean intersection-over-union (mIOU) has been significant improved, and
visualization results under multiple domain adaptation scenarios also show
that our proposed method has better performance than compared semantic
segmentation methods.

###### keywords:

Semantic scene segmentation, Unsupervised domain adaptation, Optimal
transport, Deep learning

mytitlenote1mytitlenote1footnotetext: This research is supported by the
National Natural Science Foundation of China (No. 11971296) and National Key R
& D Program of China (No. 2021YFA1003004)mytitlenote2mytitlenote2footnotetext:
Fully documented templates are available in the elsarticle package on CTAN.

##  1 Introduction

Scene segmentation is widely used in the field of autonomous driving for
environment perception, as the scene contains a lot of complex semantic
information, and 3S has received a great deal of attention. As one of core
tasks in computer vision, it focuses on complete scene understanding of the
surrounding environment in intelligent transportation systems. Unlike
classical computer vision research, such as image classification, object
detection and instance segmentation, current mainstream deep learning-based 3S
uses algorithms to densely predict each pixel of an image and assign a label
to each pixel. The results of deep semantic segmentation allow computers to
have a more detailed and accurate understanding of images, and have a wide
range of applications such as autonomous driving, medical imaging, and robot
simulation [59].

Traditional 3S methods directly conduct a mapping from the image space to the
label space. They mainly include thresholding methods, K-means clustering,
support vector machine (SVMs), edge-based approaches and region-growing
methods. Besides, kernel method is adopted to deal with inseparable tasks
through mapping current dimensional features to the high dimensional space. In
recent years, as we known, the usage of deep learning algorithms for domain
adaptation has received extensive attention in computer vision [57, 40, 27,
37]. Due to the excellent feature representation, CNN obtains remarkable
results on image classification, segmentation, and detection. However, the
divergence of natural scene images is large, which leads the transferring of
labels is difficult.

For this issue, DA offers an efficient way to achieve the labels transfer,
which transfers the knowledge from the source domain to the target domain
based on a small number of annotations. Unfortunately, due to the existence of
the huge domain gap between the source and the target domains, we need
transfer the knowledge learned on the labeled source domain to the unlabeled
target domain, i.e., we should address the unsupervised domain adaptation
(UDA) problem in 3S.

There are two kinds of approaches to solve such domain drift problem. One is
the self-training-based approaches, which train the target data with pseudo-
labels and the labeled source data achieve cross-domain alignment
simultaneously. They improve the discriminative performance of the model on
the target data. However, self-training methods usually need assign the
pseudo-labels to the target data based on the confidence, and the obtained
pseudo-labels are usually noisy. Such noise may weaken the generalization
ability of the model. In addition, they require more pre-training steps, and
hence the model complexity is generally high. On the other hand, adversarial
learning-based UDA methods reduce domain differences by aligning the
distribution of two domains in terms of appearances, features [4, 23], or the
output space [59, 32, 58]. In 3S, we note that the output space usually
contains rich information both globally and locally. That is, even if the
images from two domains have different appearances, their segmentation output
images will have substantial similarities in the overall spatial layout and
locally (see, Fig1). Based on this observation, Tsai et al. address the pixel-
level domain adaption problem in the output space. However, the transfer
mapping of the source and target domains obtained by network approximation may
lead to overfitting phenomenon due to the large amount of network parameters,
at the same time, it has few interpretability.

Figure 1: The motivation for distribution alignment in the output space. Even
if the two images in the source and target domains are very different in
appearance, their corresponding output images may be very similar in spatial
layout and localization.

.

Fortunately, OT, as a method of transferring information between samples of
two domains, has been applied in the DA problem. It can be used to calculate
the Wasserstein distance between two probability distributions to achieve
distribution alignment. The feasibility of using OT for DA has been verrified
in [21], where the discrete OT is used to match the edge distributions of two
domains under class regularity constraints. Deep Joint Optimal Transfer
(DeepJDOT) uses the same logic to transfer the source domain samples to the
target domain through the coupling matrix [2]. Seg-JDOT [1] is an attempt to
use this idea for medical image segmentation, which enables the source and the
target domain samples sharing similar representations. Recently OT is applied
to DA tasks using weighting methods [65, 36] to make the transmission of
classification image samples more accurate. To our best knowledge, there is
almost no OT-based method for the 3S, and hence in this paper, we introduce OT
into 3S.

In this paper, we use deep network to implement the UDA problem based on OT in
SSS. Firstly, we input the source and target images into the segmentation
network, which generates the output space via CNN due to its superiority of
feature representation. Secondly, we utilize OT to achieve a more robust
alignment of source and target domains in output space. By gradually adjusting
OT coupling in segmentation network training, different weights (which can
also be interpreted as an attention mechanism) are provided for sample
information transmission between the source and target domain. The OT part
achieves DA and interprets alignment process of source domain and target
domain. At the same time, it reduces the number of network parameters.
Moreover, low-level features and high-level features contain different feature
information respectively, and transferring knowledge from the high-level to
guide the adaptation of low-level features is necessary. Inspired by [35], we
perform DA on both low and high-level features in an integrated manner. It
bridges the output of low-level features and high-level features. In other
words, we construct a multi-scale segmentation network to perform adaptation
in the output space at different feature levels.

The contributions of our work are as follows.

  * 1.

We utilize the OT to achieve the UDA and embed it into the semantic
segmentation network, and further form an End-to-End CNN-based framework for
3S. It indeed reduces the number of network, as well as offers an explanation
for the semantic segmentation process.

  * 2.

we introduce a multiscale strategy to better model the low-level and high-
level feature representations, which realizes the label transfer under
different scales.

  * 3.

We apply the model to the semantic segmentation of road scenes, and
experimental results on real and synthetic datasets show that our proposed
method has a better segmentation performance and a lower network complexity.

This paper is organized as follows: Related work is introduced in Section 2.
Section 3 introduces some preliminaries of OT. Section 4 proposes our method
and then presents a optimization strategy for solving this model. Section 5
presents experimental results implemented on synthetic and real data sets to
verify the superiority of our proposed method over other state-of-the-arts
methods. Finally, Section 6 draws a conclusion.

##  2 Related works

Deep learning based semantic segmentation. Current semantic segmentation
methods are basically based on deep learning methods, existing methods are
divided into three categories of decoder-based, information fusion-based and
recurrent neural network-based(RNN) approaches. In 2017, Shelhamer et al.
proposed a semantic segmentation method based on fully convolutional neural
networks (FCNs), which is the pioneering work in deep learning algorithms for
semantic segmentation. It not only answers the question of how CNNs can be
trained end-to-end for the semantic segmentation problem, but also effectively
solves the problem of how to produce semantic predictions of pixel-level
outputs for inputs of arbitrary size. However, the method also has some
limitations, in that it tends to lose pixel location information when
performing sampling, thus affecting segmentation accuracy. In 2017, the SegNet
algorithm proposed by Badrinarayanan et al. designed an encoder-decoder
network for semantic segmentation of road scenes, which improved image
segmentation by retaining the pooling layer index when pooling was performed
and reducing the training parameters through improvements, while being able to
recover image boundary information more accurately, but its segmentation
accuracy at object boundaries still needs further improvement.

To further optimize the semantic segmentation effect and make full use of the
target spatial information, it is usually necessary to fuse different levels
of information, which are mainly classified as pixel-level fusion, feature map
fusion and multi-scale fusion. Compared with CNN, Conditional Random Fields
(CRF) can better learn the correlation between pixels. Chen et al.[6] proposed
DeepLabv1 to use the CRF model as an independent optimisation link in their
network to achieve detail enhancement on segmentation results. DeepLabv2,
proposed in 2016, introduced a pyramidal hole pooling (ASPP) module on top of
DeepLabv1[7] to select different sampling rates of convolution with holes to
process feature maps and improve segmentation accuracy. DeepLabv3[9] continued
to optimize the ASPP structure and introduced the Resnet block module to
effectively extract expressive features by cascading multiple cavity
convolution structures. In 2018, Chen et al.[5] proposed DeepLabv3+ which used
DeepLabv3 as an encoder and the Xception network structure as a benchmark
model, and designed a novel decoder structure that achieved better test
results. Another common approach to information fusion for FCNs in semantic
segmentation problems is to perform feature graph fusion. Feature graph fusion
refers to combining the global feature graph extracted from the front layer of
the network with the local feature graph extracted from the back layer. The
main representative methods are ParseNet[38], SharpMask[46], PSPNet[67], etc.
Another idea to achieve information fusion is the multi-scale fusion
approach[47], [54], [24], [63], where multiple networks at different scales
are selected and combined with their predictions to produce a comprehensive
output.

Recurrent neural networks (RNNs) combine pixel-level and local information for
successful application in modelling global information and improving semantic
segmentation results. Long short-term memory networks (LSTM)[12] and gated
recurrent units (GRU)[62] are two mainstream RNN structures. In summary, the
application of deep learning to solve the image semantic segmentation problem
has gained rapid momentum. In addition to the above methods, many new ideas
and methods have still emerged in recent years[64], [10], [56], and have shown
strong competitiveness.

Domain adaptation based semantic segmentation. The method proposed in this
paper is based on CNN training to solve the UDA problem in semantic
segmentation. UDA aims to reduce the domain difference between the labeled
source domain and the unlabeled target domain, which could improve the
generalization ability of the model on the target data. The current widely
used UDA semantic segmentation methods are mainly divided into two categories:
self-training to improve the adaptive ability of segmentation models using
pseudo labeling and domain alignment through adversarial learning, and this
paper mainly improves the UDA semantic segmentation method based on
adversarial learning.

The semantic segmentation model of UDA based on adversarial learning mainly
consists of two networks one network is used as a segmentation model to
predict the output results, and the other network is used as a discriminator
to determine whether the input is from the source segmentation output or the
target segmentation output, and the training goal of the segmentation model is
to deceive the discriminator so that the output space [59, 32, 8954024, 58] or
feature layer[4, 23] achieve domain alignment.

In all these methods, the distribution of the source and target domains are
aligned by the discriminator, a network structure. And in our work, the
discriminator network is explained in terms of OT theory to achieve DA. In
addition, the attention mechanism is used to obtain the target region that
needs to be focused on, which can obtain more details and key information
about the current task in semantic segmentation and improve the accuracy of
segmentation results. Current researchers have proposed numerous models of
attention mechanisms for semantic segmentation[68, 26, 42]. The OT coupling
computed in this paper can also be used as an attention mechanism, which is
also an important improvement of the model.

OT based domain adaptation. Domain adaptation problems can be broadly
classified into two categories, semi-supervised domain adaptation and
unsupervised domain adaptation, according to whether the samples in the target
domain are partially labeled. In the semantic segmentation problem, due to the
high cost of obtaining pixel-level label annotations, the current research
focuses on the unsupervised domain adaptation problem in semantic
segmentation, and the key to the UDA problem is to solve the source and target
domain alignment problem.

The OT[55] was early proposed by the French mathematician Gaspard Monge to
study the scheme of transforming one distribution to another at minimum cost,
and OT provides efficient methods to compute the optimal mapping to transform
one distribution to another, and to determine the distance between them. OT
has been used for DA[20, 45] to learn transitions between domains, with
associated theoretical guarantees[48]. Deep Joint Optimal Transfer
(DeepJDOT)[2] has performed well on many tasks by transferring the source
domain samples to the target domain through a coupling matrix. Recently OT has
been applied as a weighted method in domain adaptive tasks[65, 36] with more
accurate transmission on image classification tasks. However, there is no
method to make segmented image samples accurate transmission. In the SSS task,
the computation of OT is a challenging problem, its computational cost being
of order $O(n^{3}log(n))$, where $n$ is the number of samples.

To solve this problem, [22]introduced the entropy regular term and proposed
the sinkhorn algorithm with $O(n^{2})$ computational complexity in time and
space, however, the computation of OT still requires a large cost when n is
large. In this paper, we solve this problem by distributed training, thus
achieving the exact transmission of segmented image samples between two
domains.

##  3 Preliminaries of OT

In this section, we discuss the problem of achieving domain-adaptive optimal
transmission. OT is a better method to establish nonlinear correspondence
between samples in the source and target domains. In the first part, we
introduced the OT optimization problem over discrete empirical distributions.
Then, we discussed a regularized variant of this discrete optimal transport
problem.

###  3.1 Discrete OT

First, we denote by $\mu_{s}$ and $\mu_{t}$ the discrete forms of the marginal
probability distributions of samples $X_{s}$ and $X_{t}$ on the source and
target domains, that is

$\mu_{s}=\sum_{i=1}^{n_{s}}p^{s}_{i}\delta_{x^{s}_{i}},\quad\mu_{t}=\sum_{i=1}^{n_{t}}p^{t}_{i}\delta_{x^{t}_{i}}$
(1)

where $\delta_{x_{i}}$ is the Dirac function at location
$x_{i}\in\mathbb{R}^{d}$. $p^{s}_{i}$ and $p^{t}_{i}$ are probability masses
accociated to the $i$th sample, they satisfy
$\sum\nolimits_{i=1}^{n_{s}}p^{s}_{i}=\sum\nolimits_{i=1}^{n_{t}}p^{t}_{i}=1$.
The Kantorovitch problem is to find the probabilistic coupling $\gamma$ that
minimizes the optimal transport distance and is defined in the following form:

$\gamma^{*}=\mathop{\arg\min}\limits_{\gamma\in\prod(\mu_{s},\mu_{t})}\langle\gamma,C\rangle_{F}$
(2)

where $\langle.,.\rangle_{F}$ is the Frobenius dot product and $C$ is the cost
matrix, and $C_{ij}=c(x^{s}_{i},x^{t}_{j})$ defines the cost of moving the
probability mass $x^{s}_{i}$ to $x^{t}_{j}$. As previously detailed, this cost
is choosen as the squared Euclidean distance between the two locations, i.e.,
$C_{ij}=\big{\|}x^{s}_{i}-x^{t}_{j}\big{\|}^{2}_{2}$.
$\prod(\mu_{s},\mu_{t})=\\{\gamma\in\mathbb{R}^{N_{s}\times
N_{t}}_{+}\mid\gamma 1=\mu_{s},\gamma^{T}1=\mu_{t}\\}$, which denotes the
probability coupling space of two empirical distributions. A unique optimal
solution $\gamma^{*}$ satisfying the above equation can be obtained by solving
problem (2). Based on the above conditions, we have the Wasserstein distance
[60] on the probability space of the source and target domains as follows:

$W(\mu_{s},\mu_{t})=\mathop{\min}\limits_{\gamma\in\prod(\mu_{s},\mu_{t})}\langle\gamma,C\rangle_{F}$
(3)

It should be pointed out that the Problem (2) is the original Kantorovitch
problem, which is a constrained linear programming problem. However, it is
complex and difficult to calculate, which can be solved by entropy
regularization.

Figure 2: Overview of our proposed method. Given images from source and target
domains, we pass them through the segmentation network to obtain output
predictions. For source prediction, a segmentation loss is computed based on
the source ground truth. To make target prediction closer to the source ones,
We propose the OT module to achieve the distribution alignment of the source
and target domains on the output space and to generate the OT loss. The source
domain segmentation loss and OT loss are back propagated to the segmentation
network.

###  3.2 Regularized OT

To solve the above computational complexity problem, Cuturi [22] proposes to
add entropy regularization term of probabilistic coupling to the equation. The
entropy regularized version of the optimal transmission optimization problem
is as follows:

$\gamma^{*}=\mathop{\arg\min}\limits_{\gamma\in\prod(\mu_{s},\mu_{t})}\langle\gamma,C\rangle_{F}-\frac{1}{\lambda}\Omega(\gamma)$
(4)

where $\Omega(\gamma)=-\sum_{i,j}\gamma_{ij}$log$\gamma_{ij}$, since most
elements in $\gamma^{*}$ have a high probability of being zero, the entropy
regular term is added to the original OT problem in order to reduce its
sparsity and make the transfer matrix smoother. $\lambda$ is the weight
coefficient of the regular term. The larger $\lambda$ is, the lower the
sparsity of $\gamma^{*}$. In other words, in the source domain tend to
transmit their information to more target samples.

The entropy-regularized OT can be approximated by applying the Sinkhorn
algorithm to find the approximate solution, which significantly reduces the
computational complexity.

##  4 Proposed Method

In this section, we first introduce the proposed OT-based 3S framework and its
basic framework, and then describe its process of implementing DA at the
single-level and multi-level, respectively.

###  4.1 The Framework of OT-based Scene Segmentation

In this paper, we propose a UDA segmentation model based on OT. In the source
domain, given the source images $X_{S}=\\{x_{s}\in\mathbb{R}^{H\times W\times
3}\\}^{n_{s}}_{j=1}$ and the corresponding pixel-wise one-hot labels
$Y_{S}=\\{y_{s}\in\\{0,1\\}^{H\times W\times C}\\}^{n_{s}}_{j=1}$, while only
target images $X_{T}=\\{x_{t}\in\mathbb{R}^{H\times W\times
3}\\}^{n_{t}}_{j=1}$ are available in the target domain. Here, $H$,$W$,$C$
denote the height, width of images and the number of classes, respectively. We
aim to train a segmentation network that can correctly categorize pixels for
target data $X_{T}$.

As shown in Fig2, we first input the source domain images $X_{S}$ with
annotations into the segmentation network $G$ for training, and then use $G$
to make segmentation predictions for the images $X_{T}$ without annotations in
the target domain to obtain the softmax output $P_{t}$. Since our goal is to
make the segmentation predictions ($P_{s}$ and $P_{t}$) in the source and
target domains closer, the feasibility of DA with OT has been verified[21].
Here we use OT instead of the discriminator in the adversarial-based training
network[59] to achieve the alignment of the distributions of the two domains.
That is, the segmentation network $G$ trained in the source domain is adapted
to the target domain by encouraging the segmentation network $G$ to generate a
segmentation distribution in the target domain that is similar to that in the
source domain.

With the proposed method, we formulate the domain adaptation task containing
two loss functions:

$\mathcal{L}_{ST}=\mathcal{L}^{S}_{seg}(X_{S},Y_{S})+\mathcal{L}_{OT}(X_{S},X_{T},Y_{S},Y_{T})$
(5)

where $\mathcal{L}^{S}_{seg}(X_{S},Y_{S})$ is the cross-entropy loss using
ground truth annotations in the source domain, and $\mathcal{L}_{OT}$ is the
OT loss that adapts predicted segmentations of source images to the
distribution of target predictions.

###  4.2 Output Space Adaptation

Compared with the high-dimensional feature space, the segmentation outputs are
in the low-dimensional space, but they contain rich information such as the
overall layout of the scene and local information. The basic idea of our
proposed model is that regardless of whether the images are from the source or
target domain, their segmentation should have strong spatial and local
similarity. Tsai et al. [59] exploits this property by an adversarial learning
scheme to fit the low-dimensional softmax output of the segmentation
prediction. In this paper, we further improve the adversarial learning scheme
in [59] to achieve the distribution alignment of segmentation outputs on the
source and target domains by OT theory.

####  4.2.1 Single-level Joint Distribution Optimal Transport

Courty et al [19] proposed the Joint Distribution Optimal Transfer (JDOT)
method for two-step adaptation (i.e., first adapting the representation and
then learning the classifier on the adapted features) by directly learning the
classifier embedded in the cost function $C$. The basic idea is to align joint
feature/label distributions instead of only considering feature distributions.
In the UDA semantic segmentation scenario, we use the weighted combination of
costs in the output space and label space as a general loss for the
distribution alignment of the source and target domains:

$d(p^{s}_{i},y^{s}_{i};p^{t}_{j},y^{t}_{j})=\alpha
c(p^{s}_{i},p^{t}_{j})+\beta\mathcal{L}(y^{s}_{i},y^{t}_{j})$ (6)

For the $i$-th source and $j$-th target sample, where $Y_{S}$ is the ground
truth annotations for source images, $P_{S}=G(X_{S})$ and $P_{T}=G(X_{T})$ are
the segmentation outputs of the source and target domains, respectively. In
our problem, $c(p^{s}_{i},p^{t}_{j})$ is chosen as the Kullback-Leibler
divergence between the class distributions of the segmentation output space of
the source and target domains instead of $\ell^{2}_{2}$ distance, and
$\mathcal{L}(y^{s}_{i},y^{t}_{j})$ is a cross-entropy classification loss.
Parameters $\alpha$ and $\beta$ are two values weighing the contributions of
distance terms. Since target labels $y^{t}_{j}$ are unknown, they are replaced
by the segmentation output $p^{t}_{j}$. Based on the idea of OT, we want to
match samples in the source and target domains that have similar segmentation
outputs and share the same labels, and then we obtain the following
optimization problem.

$\mathcal{L}_{OT}=\mathop{\min}\limits_{G,\gamma\in\prod(\mu_{s},\mu_{t})}\langle\gamma,D_{G}\rangle_{F}$
(7)

where $D_{G}$ depends on $G$ and gathers all the pairwise cost
$d(p^{s}_{i},y^{s}_{i};p^{t}_{j},y^{t}_{j})$, and $\prod(\mu_{s},\mu_{t})$ is
the optimal transport scheme space from source domain to target domain. The
solution to problem (7) can be achieved by minimizing the following objective
function:

$\mathop{\min}\limits_{G,\gamma\in\prod(\mu_{s},\mu_{t})}\sum_{i}\sum_{j}\gamma_{ij}d(p^{s}_{i},y^{s}_{i},p^{t}_{j})$
(8)

where $d(p^{s}_{i},y^{s}_{i};p^{t}_{j},y^{t}_{j})=\alpha
KL(p^{s}_{i}\big{\|}p^{t}_{j})+\beta\mathcal{L}(y^{s}_{i},p^{t}_{j})$, and
$\mathcal{L}(y^{s}_{i},p^{t}_{j})$ is defined in the same way as the
segmentation loss definition for the source domain below, and $\alpha$,
$\beta$ are the parameters controlling the tradeoff between the two terms, as
in equation(6). We use the KL divergence in the discrete case,
$KL(p^{s}_{i}\big{\|}p^{t}_{j})=\sum_{c}p^{s}_{i}log\frac{p^{s}_{i}}{p^{t}_{j}}$,
that is, we want to use the segmentation output distribution $p^{s}_{i}$ of
the source domain to guide the split output distribution $p^{t}_{j}$ of the
target domain.

In addition we have to consider the segmentation performance of the
segmentation network $G$ on the source domain samples, and according to
equation (1), we define the segmentation loss in (5) as the cross-entropy loss
for images from the source domain:

$\mathcal{L}_{seg}(X_{S},Y_{S})=-\sum_{h,w}\sum_{c}Y^{(h,w,c)}_{s}log(P^{(h,w,c)}_{s})$
(9)

####  4.2.2 Multi-level Joint Distribution Optimal Transport

Although performing OT alignment in the output space enables adaptive
prediction in the target domain, the low-level feature and high-level feature
spaces usually contain different information, and similar to the deep
supervision approach using auxiliary loss for semantic segmentation in [35],
we added an OT module to the low-level feature space to enhance the adaption.
The final training objective of the segmentation network G is:

$\displaystyle\mathcal{L}_{ST}=\sum_{i}\lambda^{i}_{seg}\mathcal{L}^{i}_{seg}(X_{S},Y_{S})+\sum_{i}\lambda^{i}_{OT}\mathcal{L}^{i}_{OT}(X_{S},X_{T},Y_{S})\quad(10)$

where $i$ denotes the network layer that predicts the segmentation output, and
$\mathcal{L}^{i}_{seg}(X_{S},Y_{S})$ and
$\mathcal{L}^{i}_{OT}(X_{S},X_{T},Y_{S})$ remain in the same form as in (9)
and (7), respectively. However, for large sample sizes the constraint of
computing a full $\gamma$ yields a computationally problem, both in terms of
memory and time complexity. In the next section, we propose a stochastic
optimization method based on distributed training.

###  4.3 Optimization Strategy

In this part, we describe the approximate optimization procedure for solving
problem (10). Equation (10) involves two variables to be optimized: the OT
matrix $\gamma$ and the segmentation network $G$. Due to the huge sample size
in the semantic segmentation scenario, computing a complete $\gamma$ generates
computational problems in terms of memory and time complexity, we propose a
small-batch distributed training method with random sampling from the source
and target domains for optimization in each training.

There are two steps in each training optimization, first construct the cost
matrix $C$ in equation (2), we construct the cost matrix $C$ by calculating
the two-by-two similarity between the source and target domain samples in each
small batch sampling. That is,
$C_{ij}=\big{\|}x^{s}_{i}-x^{t}_{j}\big{\|}^{2}_{2}$, where $x^{s}_{i}$ and
$x^{t}_{j}$ denote the initial input images of the source and target domains,
respectively.

Then the OT coupling $\gamma$ is calculated by the sinkhorn algorithm, and
when the $\gamma$ is fixed, the optimization of $G$ is a classical deep
learning problem.

Based on the idea of OT theory, the calculated $\gamma_{ij}$ represents the
probability of transmitting the information from the $i$-th sample in the
source domain to the $j$-th sample in the target domain at the minimum
transport cost. From equation (8), this probability value can also be regarded
as the weight of aligning two samples in the loss function, which can be seen
as an attention mechanism. That is, for the more similar samples in the source
and target domains, a greater weight is assigned when performing the domain
adaptive alignment, which further improves the efficiency of model adaption.

We summarize this approach in Alogorithm 1, and the solution algorithm
converges during the training process (see Figure3).

Algorithm 1 Stochastic optimization algorithm

Input: $x^{s}$:source domain samples, $x^{t}$:target domain samples,
$y^{s}$:source domain labels
Parameter: $\alpha$, $\beta$, $\lambda^{i}_{seg}$, $\lambda^{i}_{OT}$
Output: $G$:segmentation network

1: for each source batch($x^{s}_{b}$,$y^{s}_{b}$) and target
batch($x^{t}_{b}$) do

2: Calculate $C$, and $C_{ij}=\big{\|}x^{s}_{i}-x^{t}_{j}\big{\|}^{2}_{2}$.

3: Calculate $\gamma$ for the given batch by Sinkhorn algorithm.

4: fix $\gamma$, and use gradient descent to update $G$.

5: end for

Figure 3: Loss variation diagram during model training. Table 1: Results of
adapting GTA5 to CityScapes. We not only compare the results of performing OT
alignment on the output space with other UDA semantic segmentation methods
based on adversarial learning, but also compare the results of domain
alignment without the inclusion of the OT module as an ablation experiment.
Our baseline model is AdaptSegNet. Other previous state-of-the-art methods
include CycleGAN[69], DLOW[28], TGCF-DA+SE[13], SIBAN[39], CLAN.

GTA5$\to$CityScapes
---
Method |  road |  sidewalk |  building |  wall |  fence |  pole |  light |  sign |  veg |  terrain |  sky |  person |  rider |  car |  truck |  bus |  train |  mbike |  bike | mIoU
Source only | 75.8 | 16.8 | 77.2 | 12.5 | 21.0 | 25.5 | 30.1 | 20.1 | 81.3 | 24.6 | 70.3 | 53.8 | 26.4 | 49.9 | 17.2 | 25.9 | 6.5 | 25.3 | 36.0 | 36.6
CycleGAN | 89.3 | 45.1 | 816 | 27.5 | 18.6 | 29.0 | 35.7 | 17.3 | 79.3 | 29.4 | 71.5 | 59.7 | 15.7 | 85.3 | 18.2 | 14.8 | 1.4 | 21.9 | 12.5 | 39.6
DLOW | 87.1 | 33.5 | 80.5 | 24.5 | 13.2 | 29.8 | 29.5 | 26.6 | 82.6 | 26.7 | 81.8 | 55.9 | 25.3 | 78.0 | 33.5 | 38.7 | 0.0 | 22.9 | 34.5 | 42.3
AdaptSegNet | 86.5 | 36.0 | 79.9 | 23.4 | 23.3 | 23.9 | 35.2 | 14.8 | 83.4 | 33.3 | 75.6 | 58.5 | 27.6 | 73.7 | 32.5 | 35.4 | 3.9 | 30.1 | 28.1 | 42.4
TGCF-DA+SE | 90.2 | 51.5 | 81.1 | 15.0 | 10.7 | 37.5 | 35.2 | 28.9 | 84.1 | 32.7 | 75.9 | 62.7 | 19.9 | 82.6 | 22.9 | 28.3 | 0.0 | 23.0 | 25.4 | 42.5
SIBAN | 88.5 | 35.4 | 79.5 | 26.3 | 24.3 | 28.5 | 32.5 | 18.3 | 81.2 | 40.0 | 76.5 | 58.1 | 25.8 | 82.6 | 30.3 | 34.4 | 3.4 | 21.6 | 21.5 | 42.6
CLAN | 87.0 | 27.1 | 79.6 | 27.3 | 23.3 | 28.3 | 35.5 | 24.2 | 83.6 | 27.4 | 74.2 | 58.6 | 28.0 | 76.2 | 33.1 | 36.7 | 6.7 | 31.9 | 31.4 | 43.2
Ours(only DA) | 81.2 | 24.8 | 78.6 | 19.9 | 22.0 | 26.7 | 32.6 | 26.5 | 78.1 | 14.5 | 74.3 | 59.7 | 29.1 | 73.2 | 28.5 | 22.0 | 3.4 | 26.0 | 32.0 | 39.6
Ours(single OT+DA) | 87.4 | 29.8 | 81.1 | 26.2 | 23.7 | 25.1 | 32.8 | 16.9 | 83.1 | 32.4 | 75.3 | 58.0 | 29.7 | 78.1 | 35.3 | 38.8 | 4.5 | 27.7 | 40.5 | 43.5
Ours(multi OT+DA) | 87.8 | 31.5 | 80.5 | 24.7 | 23.0 | 26.1 | 33.8 | 15.9 | 84.2 | 33.6 | 74.4 | 57.6 | 27.7 | 83.0 | 41.2 | 41.5 | 8.4 | 27.5 | 39.0 | 44.3

##  5 Experiments and Discussions

The main purpose of our experiments is not only to evaluate the accuracy of
our proposed method, but also to verify its lightness. In this section, we
first introduce the datasets used in our experiments. Then we evaluate our
method in different UDA semantic scene segmentation tasks and compare it with
current methods, at the same time, we show our advantages in terms of model
size. Finally, we visualize the scene segmentation results and the feature
representations, which shows that our proposed method has better performance
than compared semantic segmentation methods.

###  5.1 Datasets

Due to the high cost of obtaining real pixel-level labels of segmented images,
we chose to use large-scale synthetic datasets with artificial annotations,
GTA5[50] and SYNTHIA[53]. For model training, we used the real-world dataset
Cityscapes[18]as the target domain, hoping to adapt the trained model to it.
Based on this setup, we conduct experimental comparision with three benchmark
and four SOTA methods to validate our proposed approach.

We first evaluate the performance of our method on two synthetic-to-real
semantic segmentation tasks, two synthetic datasets are GTA5[50] and
SYNTHIA[53], and the real dataset is CityScapes[18]. And the two evaluation
scenarios are GTA5$\to$CityScapes and SYNTHIA$\to$CityScapes.

GTA5 dataset contains 24966 images captured from a video game based on the
city of Los Angeles. Pixel-wise annotations with 33 classes are provided, but
only 19 classes are utilized for compatibility with CityScapes. SYNTHIA
consists of 9400 synthetic images compatiable with the cityscapes annotated
classes, and annotations with 13 classes are used for adaptation. CityScapes
is a real-world semantic segmentation dataset collected in driving scenarios.
It contains 2975 unlabeled images in the training set, which are used as the
target domain during training, and another 500 labeled images with manual
annotations as the validation set for evaluation.

In order to verify the performance of our proposed method, we conduct
experimental comparison with state-of-the-art methods on the scene datasets,
the methods used for comparison include AdaptSegNet, CycleGAN[69], DLOW[28],
TGCF-DA+SE[13], SIBAN[39], CLAN, GIO-Ada[10], AdvEnt[61].

Table 2: Performance gap between the adapted model and the fully-
supervised(oracle)model. We first compare results with state-of-the-art
methods using the VGG based model, and then show our result using the ResNet
one. Our baseline model is AdaptSegNet[59]. Other previous state-of-the-art
methods include CDA[66], CyCADA[31].

GTA5$\to$CityScapes
---
Method | Baseline | Adapt | Oracle | mIoU Gap
CDA |  | 28.9 | 60.3 | -31.4
CyCADA |  | 34.8 | 60.3 | -25.5
AdaptSegNet(single) | VGG-16 | 35.0 | 61.8 | -26.8
Ours(single) |  | 35.8 | 62.4 | -26.6
AdaptSegNet(multi) | ResNet-101 | 42.4 | 65.1 | -22.7
Ours(multi) |  | 44.3 | 66.2 | -21.9

Table 3: Results of adapting SYNTHIA to CityScapes. We not only compare the
results of performing OT alignment on the output space with other UDA semantic
segmentation methods based on adversarial learning, but also compare the
results of domain alignment without the inclusion of the OT module as an
ablation experiment. Our baseline model is AdaptSegNet. Other previous state-
of-the-art methods include CycleGAN[69], GIO-Ada[10], DLOW[28], TGCF-
DA+SE[13], SIBAN[39], CLAN, AdvEnt[61].

SYNTHIA$\to$CityScapes
---
Method |  road |  sidewalk |  building |  light |  sign |  veg |  sky |  person |  rider |  car |  bus |  mbike |  bike | mIoU
Source only | 64.3 | 21.3 | 73.1 | 7.0 | 27.7 | 63.1 | 67.6 | 42.2 | 19.9 | 73.1 | 15.3 | 10.5 | 38.9 | 40.3
CycleGAN | 58.8 | 20.4 | 71.6 | 2.7 | 8.5 | 73.5 | 73.1 | 45.3 | 16.2 | 67.2 | 14.9 | 7.9 | 24.7 | 37.3
GIO-Ada | 78.3 | 29.2 | 76.9 | 10.8 | 17.2 | 81.7 | 81.9 | 45.8 | 15.4 | 68.0 | 15.9 | 7.5 | 30.4 | 43.0
SIBAN | 82.5 | 24.0 | 79.4 | 16.5 | 12.7 | 79.2 | 82.8 | 58.3 | 18.0 | 79.3 | 25.3 | 17.6 | 25.9 | 46.3
TGCF-DA+SE | 90.1 | 48.6 | 80.7 | 3.2 | 14.3 | 82.1 | 78.4 | 54.4 | 16.4 | 82.5 | 12.3 | 1.7 | 21.8 | 46.6
AdaptSegNet | 84.3 | 42.7 | 77.5 | 4.7 | 7.0 | 77.9 | 82.5 | 54.3 | 21.0 | 72.3 | 32.2 | 18.9 | 32.3 | 46.7
CLAN | 81.3 | 37.0 | 80.1 | 16.1 | 13.7 | 78.2 | 81.5 | 53.4 | 21.2 | 73.0 | 32.9 | 22.6 | 30.7 | 47.8
AdvEnt | 85.6 | 42.2 | 79.7 | 5.4 | 8.1 | 80.4 | 84.1 | 57.9 | 23.8 | 73.3 | 36.4 | 14.2 | 33.0 | 48.0
Ours(only DA) | 75.5 | 35.4 | 76.3 | 10.1 | 12.5 | 80.0 | 81.0 | 53.5 | 13.9 | 53.1 | 23.0 | 8.0 | 19.8 | 41.7
Ours(single OT+DA) | 84.2 | 18.0 | 77.4 | 28.9 | 17.4 | 80.9 | 72.2 | 52.9 | 24.1 | 69.8 | 24.4 | 21.0 | 34.6 | 46.6
Ours(multi OT+DA) | 87.6 | 43.8 | 80.6 | 11.2 | 12.1 | 81.1 | 81.2 | 56.7 | 20.1 | 74.8 | 33.7 | 16.8 | 34.2 | 48.8

Table 4: Performance gap between the adapted model and the fully-
supervised(oracle)model. We first compare results with state-of-the-art
methods using the VGG based model, and then show our result using the ResNet
one. Our baseline model is AdaptSegNet[59]. Other previous state-of-the-art
methods include CDA[66], Cross-City[11].

SYNTHIA$\to$CityScapes
---
Method | Baseline | Adapt | Oracle | mIoU Gap
CDA |  | 34.8 | 69.6 | -34.8
Cross-City |  | 35.7 | 73.8 | -38.1
AdaptSegNet(single) | VGG-16 | 37.6 | 68.4 | -30.8
Ours(single) |  | 38.5 | 69.0 | -30.5
AdaptSegNet(multi) | ResNet-101 | 46.7 | 71.7 | -25.0
Ours(multi) |  | 48.8 | 72.5 | -23.7

Table 5: Comparison of the computational results of the space complexity
(Params) and time complexity (Flops) of AdaptSegNet[59] and our proposed
model.

Method | Params | Flops
---|---|---
AdaptSegNet | 49.11M | 705468.37M
Ours | 43.55M | 651190.14M

###  5.2 Network Architecture and Training

Segmentation Network. We adapt the DeepLab-v2[8] framework with pre-trained
ResNet-101[30] encoder as our segmentation net. After the last layer, we use
the Atrous Spatial Pyramid Pooling(ASPP)[8] as the final classifier. Finally,
we apply an up-sampling layer along with the softmax output to match the size
of the input image.

We construct the above-mentioned segmentation net and apply the OT module to
the output layer as our proposed model.

Network Training. To train the proposed adaptation model, in each train batch,
we first forward the source image $X_{s}$ to optimize the segmentation network
for $\mathcal{L}^{s}_{seg}$ in (9) and generate the output $P_{s}$. For the
target image $X_{t}$, we obtain the segmentation output $P_{t}$ and pass it
along with $P_{s}$ for optimizing $\mathcal{L}_{OT}$ in (7). For the multi-
level training objective in (10), we simply repeat the same procedure for each
adaptation module.

Our method is implemented with the PyTorch library on Nvidia GPU GTX 2080Ti
with 12GB memory. To train the segmentation network, we use the Stochastic
Gradient Descent(SGD) optimizer with Nesterov acceleration where the momentum
is 0.9 and the weight decay is 5 $\times$ $10^{-4}$. The initial learning rate
is set as 2.5 $\times$ $10^{-4}$ and is decreased using the polynomial decay
with power of 0.9 as mentioned in [8], and the maximum iteration number is
60000.

The performances of our method are evaluated by the widely utilized
performance metrics, intersection-over-union(IoU) of each class and the mean
IoU(mIoU).

###  5.3 Results on GTA CityScapes

We first evaluate the performance of our proposed method in the
GTA5$\to$CityScapes scenario, and the corresponding results are listed in
Table 1. For a fair comparison, all the competed models adopt DeepLab-v2
network framework with pre-trained ResNet-101 as encoder. Our model is an
improvement of the adversarial-based UDA method and shows better segmentation
performance compared to other adversarial training-based domain alignment
methods[59, 39, 28, 69, 13]. Our model surpasses all these models with a
promising mIoU of 44.3%, it is demonstrated that our proposed method can
effectively replace the discriminator part of the generative adversarial
network and achieve better segmentation performance than it.

In addition, we use another factor to evaluate the adaptation performance,
that is, measure how much gap is narrowed between the adaptation model and the
fully-supervised model. So we train the model using annotated ground truths in
the CityScapes datasets as the oracle results. The gap under different
baseline models are showed in Table2. We compare the mIoU under two benchmark
models, VGG-16 and ResNet-101, and it is clear that the gap is larger for the
VGG one. This suggests that using a deeper benchmark model with larger
capacity would be a greater improvement to our approach.

###  5.4 Results on SYNTHIA CityScapes

We then utilize SYNTHIA dataset as the source domain and display comparison
results of our method and other state-of-the-art methods[59, 10, 39, 61, 69,
13] that are adversarial training-based domain alignment on the validation set
of CityScapes, as listed in the Table 3. We consider the IoU and mIoU of a
subset of 13 classes following the standard experimental setting[43]. Our
method still achieves promising results in comparison to other competed
methods. Specifically, the proposed method achieves 48.8% mIoU of 13
categories.

In addition, with the same experimental setup as GTA5$\to$CityScapes, we
measure how much gap is narrowed between the adaptation model and the fully-
supervised model. So we train the model using annotated ground truths in the
CityScapes datasets as the oracle results. The gap under different baseline
models are showed in Table4. We compare the mIoU under two benchmark models,
VGG-16 and ResNet-101, and it is clear that the gap is larger for the VGG one,
too.

###  5.5 Model size comparison

To further demonstrate the simplicity of our proposed model, we compared the
number of network parameters (Params) and the amount of computation (Flops),
i.e., the spatial complexity and the temporal complexity of the model, between
our method and the AdaptSegNet model[59]. These two metrics for both models
are presented in Table5. According to the computational results, our proposed
model is further improved by reducing 11.32% in spatial complexity and 7.7% in
temporal complexity compared to AdaptSegNet, while ensuring that the model
segmentation results do not degrade and have some improvement.

Figure 4: Qualitative results of UDA semantic segmentation. (a)Target image,
(b)Ground truth, (c)source only model, (d)AdaptSegNet[59], (e)Ours.

###  5.6 Visualization Results

Segmentation Visualization. As illustrated in Figure4, we provide some typical
qualitative segmentation results of target data on two evaluation scenarios
GTA$\to$CityScapes and SYNTHIA$\to$CityScapes. Obviously, the adversarial
training-based UDA method[59] could significantly promote the performance in
comparison to the source model. Besides, our proposed model has better
scalability to small-scale objectives(e.g.,’train’), and showed better
segmentation performance on all other categories. Since OT is added as an
attention mechanism in the adaptive process, the domain distribution alignment
pays more attention to the category with higher similarity in both domains, so
our proposed method provides more accurate supervision information and thus
avoids some mislabeling and produces more reasonable segmentation results.

Feature Visualization. We use t-SNE[33] to visualize the feature
representations of our model and AdaptSegNet model[59] as illustrated in
Figure 5. It is observed that Our model learns a more compact and better
separated pixel embedding, which suggests that our segmentation network can
produce more discriminative features. This observation demonstrates that our
method can provide correct supervision signal for target data through the OT
part.

Figure 5:  The t-SNE visualization of embedded features.

##  6 Conclusion

In this paper, we propose a UDA framework based on OT, and use a small size of
labeled data to achieve semantic segmentation. Based on the fact that although
the two images in the source and target domains have large differences in
appearance, their segmentation outputs will have large similarities in the
overall spatial layout or local structure, we achieve domain alignment of
source and target domains on the segmented output. First we generate the
output space via CNN, then we utilize OT to achieve DA of source and target
domains in output space. In particular, the computed OT coupling matrix is
used as an attention mechanism, which assigns higher weights to the samples
with higher similarity in the two domains, and hence improves the adaptation
of the model. Further, to better describe the multi-scale property of
features, we construct multi-scale segmentation network on different feature
layers to perform DA. Experimental results show that our proposed method
performs favorably against three benchmark and four state-of-the-art methods,
and visualization results under multiple domain adaptation 3S also show that
our method has better performance.

## References

  * Ackaouy et al. [2020] Ackaouy, A.; Courty, N.; Vallée, E.; Commowick, O.; Barillot, C.; and Galassi, F. 2020.  Unsupervised Domain Adaptation With Optimal Transport in Multi-Site Segmentation of Multiple Sclerosis Lesions From MRI Data.  _Frontiers in Computational Neuroscience_ , 14.
  * [2] Bhushan Damodaran, B.; Kellenberger, B.; Flamary, R.; Tuia, D.; and Courty, N.  DeepJDOT: Deep Joint Distribution Optimal Transport for Unsupervised Domain Adaptation.  arXiv:1803.10081.
  * Bouville [2008] Bouville, M. 2008.  Crime and punishment in scientific research.  arXiv:0803.4058.
  * Chen et al. [2019] Chen, C.; Dou, Q.; Chen, H.; Qin, J.; and Heng, P. A. 2019.  Synergistic Image and Feature Adaptation: Towards Cross-Modality Domain Adaptation for Medical Image Segmentation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33: 865–872.
  * Chen et al. [2018a] Chen, L.; Zhu, Y.; Papandreou, G.; Schroff, F.; and Adam, H. 2018a.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation.  _CoRR_ , abs/1802.02611.
  * Chen et al. [2014] Chen, L. C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; and Yuille, A. L. 2014.  Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs.  In _International Conference on Learning Representations_.
  * Chen et al. [2016] Chen, L. C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; and Yuille, A. L. 2016.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs.
  * Chen et al. [2018b] Chen, L. C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; and Yuille, A. L. 2018b.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 40(4): 834–848.
  * Chen et al. [2017a] Chen, L. C.; Papandreou, G.; Schroff, F.; and Adam, H. 2017a.  Rethinking Atrous Convolution for Semantic Image Segmentation.
  * Chen et al. [2018c] Chen, Y.; Li, W.; Chen, X.; and Gool, L. V. 2018c.  Learning Semantic Segmentation from Synthetic Data: A Geometrically Guided Input-Output Adaptation Approach.
  * Chen et al. [2017b] Chen, Y. H.; Chen, W. Y.; Chen, Y. T.; Tsai, B. C.; and Min, S. 2017b.  No More Discrimination: Cross City Adaptation of Road Scene Segmenters.  In _2017 IEEE International Conference on Computer Vision (ICCV)_.
  * Cheng, Dong, and Lapata [2016] Cheng, J.; Dong, L.; and Lapata, M. 2016.  Long Short-Term Memory-Networks for Machine Reading.
  * [13] Choi, J.; Kim, T.; and Kim, C. ????  Self-Ensembling with GAN-based Data Augmentation for Domain Adaptation in Semantic Segmentation.  _International Conference on Computer Vision_.
  * Clancey [1979] Clancey, W. J. 1979.  _Transfer of Rule-Based Expertise through a Tutorial Dialogue_.  Ph.D. diss., Dept. of Computer Science, Stanford Univ., Stanford, Calif.
  * Clancey [1983] Clancey, W. J. 1983.  Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education.  In _Proceedings of the Eighth International Joint Conference on Artificial Intelligence (IJCAI-83)_ , 556–560. Menlo Park, Calif: IJCAI Organization.
  * Clancey [1984] Clancey, W. J. 1984.  Classification Problem Solving.  In _Proceedings of the Fourth National Conference on Artificial Intelligence_ , 45–54. Menlo Park, Calif.: AAAI Press.
  * Clancey [2021] Clancey, W. J. 2021.  The Engineering of Qualitative Models.  Forthcoming.
  * Cordts et al. [2016] Cordts, M.; Omran, M.; Ramos, S.; Rehfeld, T.; Enzweiler, M.; Benenson, R.; Franke, U.; Roth, S.; and Schiele, B. 2016.  The Cityscapes Dataset for Semantic Urban Scene Understanding.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 3213–3223.
  * Courty et al. [2017a] Courty, N.; Flamary, R.; Habrard, A.; and Rakotomamonjy, A. 2017a.  Joint distribution optimal transportation for domain adaptation.  In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Courty, Flamary, and Tuia [2014] Courty, N.; Flamary, R.; and Tuia, D. 2014.  Domain Adaptation with Regularized Optimal Transport.  In Calders, T.; Esposito, F.; Hüllermeier, E.; and Meo, R., eds., _Machine Learning and Knowledge Discovery in Databases_ , 274–289. Berlin, Heidelberg: Springer Berlin Heidelberg.  ISBN 978-3-662-44848-9.
  * Courty et al. [2017b] Courty, N.; Flamary, R.; Tuia, D.; and Rakotomamonjy, A. 2017b.  Optimal Transport for Domain Adaptation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 39(9): 1853–1865.
  * Cuturi [2013] Cuturi, M. 2013.  Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances.  _Advances in Neural Information Processing Systems_ , 26: 2292–2300.
  * Du et al. [2020] Du, L.; Tan, J.; Yang, H.; Feng, J.; Xue, X.; Zheng, Q.; Ye, X.; and Zhang, X. 2020\.  SSF-DAN: Separated Semantic Feature Based Domain Adaptation Network for Semantic Segmentation.  In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_.
  * Eigen and Fergus [2014] Eigen, D.; and Fergus, R. 2014.  Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture.
  * Engelmore and Morgan [1986] Engelmore, R.; and Morgan, A., eds. 1986.  _Blackboard Systems_.  Reading, Mass.: Addison-Wesley.
  * Fu et al. [2019] Fu, J.; Liu, J.; Tian, H.; Li, Y.; Bao, Y.; Fang, Z.; and Lu, H. 2019.  Dual Attention Network for Scene Segmentation.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 3141–3149.
  * Ganin et al. [2016] Ganin, Y.; Ustinova, E.; Ajakan, H.; Germain, P.; Larochelle, H.; Laviolette, F.; Marchand, M.; and Lempitsky, V. 2016.  Domain-Adversarial Training of Neural Networks.  _Journal of Machine Learning Research_ , 17(1): 2096–2030.
  * Gong et al. [2019] Gong, R.; Li, W.; Chen, Y.; and Gool, L. V. 2019.  DLOW: Domain Flow for Adaptation and Generalization.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Goodfellow et al. [2014] Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014.  Generative Adversarial Nets.  In _Neural Information Processing Systems_.
  * He et al. [2016] He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep Residual Learning for Image Recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 770–778.
  * Hoffman et al. [2018] Hoffman, J.; Tzeng, E.; Park, T.; Zhu, J.-Y.; Isola, P.; Saenko, K.; Efros, A.; and Darrell, T. 2018.  CyCADA: Cycle-Consistent Adversarial Domain Adaptation.  In Dy, J.; and Krause, A., eds., _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , 1989–1998. PMLR.
  * Kim and Byun [2020] Kim, M.; and Byun, H. 2020.  Learning Texture Invariant Representation for Domain Adaptation of Semantic Segmentation.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 12972–12981.
  * Laurens and Hinton [2008] Laurens, V. D. M.; and Hinton, G. 2008.  Visualizing Data using t-SNE.  _Journal of Machine Learning Research_ , 9(2605): 2579–2605.
  * Lecun et al. [1998] Lecun, Y.; Bottou, L.; Bengio, Y.; and Haffner, P. 1998.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11): 2278–2324.
  * Lee et al. [2014] Lee, C.-Y.; Xie, S.; Gallagher, P.; Zhang, Z.; and Tu, Z. 2014.  Deeply-Supervised Nets.  arXiv:1409.5185.
  * Li et al. [2020] Li, M.; Zhai, Y.-M.; Luo, Y.-W.; Ge, P.-F.; and Ren, C.-X. 2020.  Enhanced Transport Distance for Unsupervised Domain Adaptation.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 13933–13941.
  * Liu and Tuzel [2016] Liu, M. Y.; and Tuzel, O. 2016.  Coupled Generative Adversarial Networks.
  * Liu, Rabinovich, and Berg [2015] Liu, W.; Rabinovich, A.; and Berg, A. C. 2015.  ParseNet: Looking Wider to See Better.  _Computer ence_.
  * Luo et al. [2019a] Luo, Y.; Liu, P.; Guan, T.; Yu, J.; and Yang, Y. 2019a.  Significance-Aware Information Bottleneck for Domain Adaptive Semantic Segmentation.  In _2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019_ , 6777–6786. IEEE.
  * Luo et al. [2017] Luo, Z.; Zou, Y.; Hoffman, J.; and Fei-Fei, L. 2017.  Label Efficient Learning of Transferable Representations across Domains and Tasks.
  * NASA [2015] NASA. 2015.  Pluto: The ’Other’ Red Planet.  https://www.nasa.gov/nh/pluto-the-other-red-planet.  Accessed: 2018-12-06.
  * Niu et al. [2022] Niu, R.; Sun, X.; Tian, Y.; Diao, W.; Chen, K.; and Fu, K. 2022.  Hybrid Multiple Attention Network for Semantic Segmentation in Aerial Images.  _IEEE Transactions on Geoscience and Remote Sensing_ , 60: 1–18.
  * Pan et al. [2020] Pan, F.; Shin, I.; Rameau, F.; Lee, S.; and Kweon, I. 2020.  Unsupervised Intra-Domain Adaptation for Semantic Segmentation Through Self-Supervision.  _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Pearlmutter [1995] Pearlmutter, B. 1995.  Gradient calculations for dynamic recurrent neural networks: a survey.  _IEEE Transactions on Neural Networks_ , 6(5): 1212–1228.
  * Perrot et al. [2016] Perrot, M.; Courty, N.; Flamary, R.; and Habrard, A. 2016.  Mapping Estimation for Discrete Optimal Transport.  In Lee, D. D.; Sugiyama, M.; von Luxburg, U.; Guyon, I.; and Garnett, R., eds., _Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain_ , 4197–4205.
  * Pinheiro et al. [2016] Pinheiro, P. O.; Lin, T. Y.; Collobert, R.; and Dollár, P. 2016.  Learning to Refine Object Segments.
  * Raj, Maturana, and Scherer [2015] Raj, A.; Maturana, D.; and Scherer, S. 2015.  Multi-Scale Convolutional Architecture for Semantic Segmentation.
  * Redko, Habrard, and Sebban [2016] Redko, I.; Habrard, A.; and Sebban, M. 2016.  Theoretical Analysis of Domain Adaptation with Optimal Transport.
  * Rice [1986] Rice, J. 1986.  Poligon: A System for Parallel Problem Solving.  Technical Report KSL-86-19, Dept. of Computer Science, Stanford Univ.
  * Richter et al. [2016] Richter, S. R.; Vineet, V.; Roth, S.; and Koltun, V. 2016.  Playing for Data: Ground Truth from Computer Games.  _Springer International Publishing_.
  * Robinson [1980a] Robinson, A. L. 1980a.  New Ways to Make Microcircuits Smaller.  _Science_ , 208(4447): 1019–1022.
  * Robinson [1980b] Robinson, A. L. 1980b.  New Ways to Make Microcircuits Smaller—Duplicate Entry.  _Science_ , 208: 1019–1026.
  * Ros et al. [2016] Ros, G.; Sellart, L.; Materzynska, J.; Vazquez, D.; and Lopez, A. M. 2016.  The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 3234–3243.
  * Roy and Todorovic [2016] Roy, A.; and Todorovic, S. 2016.  A Multi-scale CNN for Affordance Segmentation in RGB Images.  In _European Conference on Computer Vision_.
  * Rüschendorf [2009] Rüschendorf, L. 2009.  Optimal Transport. Old and New.  _Jahresbericht der Deutschen Mathematiker-Vereinigung_ , 111(2): 18–21.
  * Souly, Spampinato, and Shah [2017] Souly, N.; Spampinato, C.; and Shah, M. 2017.  Semi Supervised Semantic Segmentation Using Generative Adversarial Network.  In _2017 IEEE International Conference on Computer Vision (ICCV)_.
  * Sun and Saenko [2016] Sun, B.; and Saenko, K. 2016.  Deep CORAL: Correlation Alignment for Deep Domain Adaptation.  _Springer International Publishing_.
  * Tsai et al. [2019] Tsai, Y.; Sohn, K.; Schulter, S.; and Chandraker, M. 2019.  Domain Adaptation for Structured Output via Discriminative Patch Representations.  In _2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019_ , 1456–1465. IEEE.
  * Tsai et al. [2018] Tsai, Y.-H.; Hung, W.-C.; Schulter, S.; Sohn, K.; Yang, M.-H.; and Chandraker, M. 2018.  Learning to Adapt Structured Output Space for Semantic Segmentation.  In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 7472–7481.
  * Villani and Cédric [2014] Villani; and Cédric. 2014.  _Optimal transport : old and new_.  Optimal transport : old and new.
  * Vu et al. [2019] Vu, T. H.; Jain, H.; Bucher, M.; Cord, M.; and Perez, P. 2019.  ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Wu and King [2016] Wu, Z.; and King, S. 2016.  Investigating gated recurrent networks for speech synthesis.
  * Xiao, Lim, and Ning [2016] Xiao, B.; Lim, S. N.; and Ning, Z. 2016.  Multiscale fully convolutional network with application to industrial inspection.  In _2016 IEEE Winter Conference on Applications of Computer Vision (WACV)_.
  * Xin et al. [2017] Xin, L.; Jie, Z.; Wei, W.; Liu, C.; and Feng, J. 2017.  FoveaNet: Perspective-Aware Urban Scene Parsing.
  * Xu et al. [2020] Xu, R.; Liu, P.; Wang, L.; Chen, C.; and Wang, J. 2020.  Reliable Weighted Optimal Transport for Unsupervised Domain Adaptation.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 4393–4402.
  * Zhang, David, and Gong [2017] Zhang, Y.; David, P.; and Gong, B. 2017.  Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes.  In _IEEE International Conference on Computer Vision_.
  * Zhao et al. [2016] Zhao, H.; Shi, J.; Qi, X.; Wang, X.; and Jia, J. 2016.  Pyramid Scene Parsing Network.  In _IEEE Computer Society_.
  * Zhao et al. [2017] Zhao, H.; Shi, J.; Qi, X.; Wang, X.; and Jia, J. 2017.  Pyramid Scene Parsing Network.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 6230–6239.
  * Zhu et al. [2017] Zhu, J. Y.; Park, T.; Isola, P.; and Efros, A. A. 2017.  Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks.  _IEEE_.