# Learning to Relate Depth and Semantics for Unsupervised Domain Adaptation

Suman Saha*
ETH Zurich  Anton Obukhov*
ETH Zurich  Danda Pani Paudel
ETH Zurich  Menelaos Kanakis
ETH Zurich  Yuhua Chen
ETH Zurich  Stamatios Georgoulis
ETH Zurich  Luc Van Gool
ETH Zurich  KU Leuven

###### Abstract

We present an approach for encoding visual task relationships to improve model
performance in an Unsupervised Domain Adaptation (UDA) setting. Semantic
segmentation and monocular depth estimation are shown to be complementary
tasks; in a multi-task learning setting, a proper encoding of their
relationships can further improve performance on both tasks. Motivated by this
observation, we propose a novel Cross-Task Relation Layer (CTRL), which
encodes task dependencies between the semantic and depth predictions. To
capture the cross-task relationships, we propose a neural network architecture
that contains task-specific and cross-task refinement heads. Furthermore, we
propose an Iterative Self-Learning (ISL) training scheme, which exploits
semantic pseudo-labels to provide extra supervision on the target domain. We
experimentally observe improvements in both tasks’ performance because the
complementary information present in these tasks is better captured.
Specifically, we show that: (1) our approach improves performance on all tasks
when they are complementary and mutually dependent; (2) the CTRL helps to
improve both semantic segmentation and depth estimation tasks performance in
the challenging UDA setting; (3) the proposed ISL training scheme further
improves the semantic segmentation performance. The implementation is
available at https://github.com/susaha/ctrl-uda.

Figure 1:  Semantic segmentation improvement with our approach to unsupervised
domain adaptation over the state-of-the-art DADA [62] method. Left to right:
Cityscapes test images, DADA, and the proposed method (CTRL). Our model
correctly segments the “bus”, “rider”, and “wall” classes underrepresented in
the target domain (highlighted).

##  1 Introduction

††Corresponding author: Suman Saha (suman.saha@vision.ee.ethz.ch)††* Equal
contribution.

Semantic segmentation and monocular depth estimation are two important
computer vision tasks that allow us to perceive the world around us and enable
agents’ reasoning, e.g., in an autonomous driving scenario. Moreover, these
tasks have been shown to be complementary to each other, i.e., information
from one task can improve the other task’s performance [29, 42, 60]. Domain
Adaptation (DA) [11] refers to maximizing model performance in an environment
with a smaller degree of supervision (the target domain) relative to what the
model was trained on (the source domain). Unsupervised Domain Adaptation (UDA)
assumes only access to the unannotated samples from the target domain at train
time – the setting of interest in this paper, explained in greated detail in
Sec. 6.

Recent domain adaptation techniques [34, 62] proposed to leverage depth
information available in the source domain to improve semantic segmentation on
the target domain. However, they lack an explicit multi-task formulation to
relate depth and semantics, that is to say, how each semantic category relates
to different depth levels. The term depth levels refers to different discrete
ranges of depth values, i.e., “near” (1-5m); “medium-range” (5-20m), or “far”
($>$20m). This paper aims to design a model that learns explicit relationships
between different visual semantic classes and depth levels within the UDA
context.

To this end, we design a network architecture and a new multitask-aware
feature space alignment mechanism for UDA. First, we propose a Cross-Task
Relation Layer (CTRL) – a novel parameter-free differentiable module tailored
to capture the task relationships given the network’s semantic and depth
predictions. Second, we utilize a Semantics Refinement Head (SRH) that
explicitly captures cross-task relationships by learning to predict semantic
segmentation given predicted depth features. Both CTRL and SRH boost the
model’s ability to effectively encode correlations between semantics and
depth, thus improving predictions on the target domain. Third, we employ an
Iterative Self Learning (ISL) scheme. Coupled with the model design, it
further pushes the performance of semantic segmentation. As a result, our
method achieves state-of-the-art semantic segmentation performance on three
challenging UDA benchmarks (Sec. 4). Fig. 1 demonstrates our method’s
effectiveness by comparing semantic predictions of classes underrepresented in
the target domain to predictions made by the previous state-of-the-art method.
The paper is organized as follows: Sec. 2 discusses the related work; Sec. 3
describes the proposed approach to UDA, the network architecture, and the
learning scheme; Sec. 4 presents the experimental analysis with ablation
studies; Sec. 5 concludes the paper.

##  2 Related Work

Semantic Segmentation. refers to the task of assigning a semantic label to
each pixel of an image. Conventionally, the task has been addressed using
hand-crafted features combined with classifiers, such as Random Forests [53],
SVMs [16], or Conditional Random Fields [31]. Powered by the effectiveness of
Convolutional Neural Networks (CNNs) [33], we have seen an increasing number
of deep learning-based models. Long et al. [38] were among the first to use
fully convolutional networks (FCNs) for semantic segmentation. Since then,
this design has quickly become a state-of-the-art method for the task. The
encoder-decoder design is still widely used [67, 5, 1, 72, 4].

Cross-domain Semantic Segmentation. Training deep networks for semantic
segmentation requires large amounts of labeled data, which presents a
significant bottleneck in practice, as acquiring pixel-wise labels is a labor-
intensive process. A common approach to address the issue is to train the
model on a source domain and apply it to a target domain in a UDA context.
However, this often causes a performance drop due to the domain shift. Domain
Adaptation aims to solve the issue by aligning the features from different
domains. DA is a highly active research field, and techniques have been
developed for various applications, including image classification [18, 36,
39, 40], object detection [8], fine-grained recognition [19], etc.

More related to our method are several works on unsupervised domain adaptation
for semantic segmentation [69, 52, 74, 9, 61, 25, 65, 73, 48, 66]. This
problem has been tackled with curriculum learning [69], GANs [52], adversarial
training on the feature space [9], output space [55], or entropy maps [61],
self-learning using pseudo- or weak labels [74, 48, 25]. However, prior works
typically only consider adapting semantic segmentation while neglecting any
multi-task correlations. A few methods [7, 62] model correlations between
semantic segmentation and depth estimation, similarly to our work, yet – as
explained in Sec. 1 – these works come with crucial limitations.

Monocular Depth Estimation. Similar to semantic segmentation, monocular depth
estimation is dominated by CNN-based methods [13, 15, 32, 35]. [13] introduced
a CNN-based architecture for depth estimation, which regresses a dense depth
map. Their approach was then improved by incorporating techniques such as a
CRF [37, 35] and multi-scale CRF techniques [64]. Besides, improvements in the
loss design itself also lead to better depth estimation. Examples include the
reverse Huber (berHu) loss [46, 75], and the ordinal regression loss [15].

Multi-task Learning for Semantic Segmentation and Depth Estimation. Within the
context of multi-task learning, semantic segmentation is shown to be highly
correlated with depth estimation, and vice versa [68, 63, 29, 70, 71, 42, 54,
60, 59, 28]. To leverage this correlation, some authors have proposed to learn
them jointly [50, 27, 6]. In particular, [45, 27, 58, 3] proposed to share the
encoder and use multiple decoders, whereas a shared conditional decoder is
used in [6]. Semantic segmentation was also demonstrated to help guide the
depth training process [21, 26].

In this paper, we build upon these observations. We argue that task
relationships, like the ones between depth and semantics, are not entirely
domain-specific. As a result, if we correctly model these relationships in one
domain, they can be transferred to another domain to help guide the DA
process. The proposed method and its components are explicitly designed around
this hypothesis.

Inputimage ($x$)($\mathcal{F}_{e}$)

Backbone

Decoder

SharedfeaturemapSemantics head ($\mathcal{F}_{s}$)SRH head
($\mathcal{F}_{r}$)Depth head ($\mathcal{F}_{d}$)PredictionsSemantics
($\hat{y}_{s}$)Refinedsemantics ($\hat{y}_{r}$)Continuousdepth
($\hat{z}$)$\mathcal{L}_{\text{CE}}$$\mathcal{L}_{\text{CE}}$$\mathcal{L}_{\text{berHu}}$Cross-
Task Relation LayerDiscretizationmoduleDescretedepth
($\hat{z}^{\prime}$)Entropy map generatorEntropymap
($E$)Domaindiscriminator($\mathcal{D}$)Domain of $x$$\mathcal{L}_{\text{adv}}$

Figure 2:  Overview of the proposed neural architecture (Sec. 3.2) and the
CTRL module (Sec. 3.4). Supervised losses (in the middle) are applied only on
the source domain; the rest of the data flow is domain-agnostic. Legend:
learned modules, predictions, loss functions; rounded corners denote
operators, rectangles denote activations.

##  3 Method

In this section, we describe our approach to UDA in the autonomous driving
setting. Sec. 3.1 presents an overview of the proposed approach; Sec. 3.2
explains the notation and problem formulation; Sec. 3.3 describes supervision
on the source domain; Sec. 3.4 presents the CTRL module design; Sec. 3.5
describes the ISL technique; Sec. 3.6 prescribes the rest of the network
architecture details.

###  3.1 Overview

The primary hypothesis behind our approach is that task dependencies persist
across domains, i.e., most semantic classes fall under a finite depth range.
We can exploit this information from source samples and transfer it to target
using adversarial training. As our goal is to train the network in a UDA
setting, we follow an adversarial training scheme [24, 55] to learn domain
invariant representations.

Unlike [62] that directly aligns a combination of semantics and depth
features, we wish to design a joint feature space for domain alignment by
fusing the task-specific and the cross-task features and then learn to
minimize the domain gap through adversarial training. To this end, we propose
CTRL – a novel module that constructs the joint feature space by computing
entropy maps of both the semantic label and discretized depth distributions
(Fig. 2). Thus, CTRL entropy maps, generated on the source and target domains,
are expected to carry similar information.

Further enhancement of semantic segmentation performance appears possible by
utilizing the Iterative Self-Learning (ISL) training scheme, which does not
require expensive patch-based pseudo-label generation like [25]. As our CTRL
helps the network to predict high-quality predictions (Fig. 1), ISL training
exploits high-confidence predictions as supervision (pseudo-labels) on the
target domain.

###  3.2 Problem Formulation

Let $\mathfrak{D}^{(s)}$ and $\mathfrak{D}^{(t)}$ denote the source and target
domains, with samples from them represented by tuples
$(x^{(s)},y^{(s)},z^{(s)})$ and $(x^{(t)})$ respectively, where
$x\in\mathbb{R}^{H\times W\times 3}$ are color images,
$y\in\\{1,...,C\\}^{H\times W}$ are semantic annotations with $C$ classes, and
$z\in[Z_{\text{min}},Z_{\text{max}}]^{H\times W}$ are depth maps from a finite
frustum. Furthermore, $\mathcal{F}_{e}$ is the shared feature extractor, which
includes a pretrained backbone, and a decoder; $\mathcal{F}_{s}$ and
$\mathcal{F}_{d}$ are the task-specific semantics and depth heads,
respectively; $\mathcal{F}_{r}$ is the SRH (Fig. 2).

First, $\mathcal{F}_{e}$ extracts a shared feature map to be used by SRH and
task-specific semantics and depth heads. The semantics head $\mathcal{F}_{s}$
predicts a semantic segmentation map
$\hat{y}_{s}=\mathcal{F}_{s}(\mathcal{F}_{e}(x))$ with $C$ channels per pixel,
denoting predicted class probabilities. The depth head $\mathcal{F}_{d}$
predicts a real-valued depth map
$\hat{z}=\mathcal{F}_{d}(\mathcal{F}_{e}(x))$, where each pixel is mapped into
the finite frustum specified in the source domain. We further employ SRH to
learn the cross-task relationship between semantics and depth by making it
predict semantics from the shared feature map, attenuated by the predicted
depth map. Formally, the shared feature map is point-wise multiplied by the
predicted depth map, and then SRH predicts a second (auxiliary) semantic
segmentation map:
$\hat{y}_{r}=\mathcal{F}_{r}(\hat{z}\odot\mathcal{F}_{e}(x))$.

We refer to the part of the model enclosing the
$\mathcal{F}_{e},\mathcal{F}_{s},\mathcal{F}_{r},\mathcal{F}_{d}$ modules as a
prediction network. The predictions made by the network on the source and
target domains are denoted as
$(\hat{y}_{s}^{(s)},\hat{y}_{r}^{(s)},\hat{z}^{(s)})$ and
$(\hat{y}_{s}^{(t)},\hat{y}_{r}^{(t)},\hat{z}^{(t)})$, respectively. We
upscale these predictions along the spatial dimension to match the original
input image dimension $H\times W$ before any further processing. Given these
semantics and depth predictions on the source and target domains, we optimize
the network cost using supervised loss on the source domain, and unsupervised
domain alignment loss on the target domain within the same training process.

###  3.3 Supervised Learning

Since the semantic segmentation predictions $\hat{y}_{s}^{(s)}$,
$\hat{y}_{r}^{(s)}$ and ground truth $y^{(s)}$ are represented as pixel-wise
class probabilities over $C$ classes, we employ the standard cross-entropy
loss with the semantic heads:

$\mathcal{L}_{\text{CE}}(\hat{y},y)=-\sum_{i=1}^{C}y_{i}\log\hat{y}_{i}.$ (1)

We use the berHu loss (the reversed Huber criterion [32]) for penalizing depth
predictions:

$\begin{split}\mathcal{L}_{\text{berHu}}(\hat{z},z)&=\left\\{\begin{array}[]{ll}|z-\hat{z}|&|z-\hat{z}|\leq
L,\\\ \frac{(z-\hat{z})^{2}+L^{2}}{2L}&|z-\hat{z}|>L,\end{array}\right.\\\
L&=0.2\max(|z-\hat{z}|).\end{split}$ (2)

Following [29], we regress inverse depth values (normalized disparity), which
is shown to improve the precision of predictions on the full range of the view
frustum. The parameters of the network $\theta_{e}$, $\theta_{s}$,
$\theta_{r}$, $\theta_{d}$ (parameterizing $\mathcal{F}_{e}$,
$\mathcal{F}_{s}$, $\mathcal{F}_{r}$, $\mathcal{F}_{d}$ modules), collectively
denoted as $\theta_{\text{net}}$, are learned to minimize the following
supervised objective on the source domain:

$\begin{split}\min_{\theta_{\text{net}}}\mathop{\mathbb{E}}_{\mathfrak{D}^{(s)}}\big{[}&\mathcal{L}_{\text{CE}}(\hat{y}_{s}^{(s)},y^{(s)})+\lambda_{r}\mathcal{L}_{\text{CE}}(\hat{y}_{r}^{(s)},y^{(s)})\;+\\\
&\lambda_{d}\mathcal{L}_{\text{berHu}}(\hat{z}^{(s)},z^{(s)})\big{]}\end{split}$
(3)

where $\lambda_{r}$ and $\lambda_{d}$ are the hyperparameters weighting
relative importance of the SRH and depth supervision.

###  3.4 Cross-Task Relation Layer

In the absence of ground truth annotations for the target samples, we train
the network on the target images using unsupervised domain alignment loss.
Existing works either align source and target domain in a semantic space [61]
or a depth-aware semantic space [62] by fusing the continuous depth
predictions with predicted semantic maps. Here, we argue that simple fusion of
the continuous depth prediction into the semantics does not enable the network
to learn useful semantic features at different depth levels. Instead, explicit
modeling is required to achieve this goal.

Humans learn to relate semantic categories at each discrete depth level
differently. For example, “sky” is “far away” (large depth), “vehicles” are
“nearby”, “road” appears to be both “far” and “nearby”. Taking inspiration
from the way humans relate semantic and depth, we design a CTRL (Fig. 2) that
captures the semantic class-specific dependencies at different discrete depth
levels. Moreover, CTRL also preserves task-specific information by fusing
task-specific and task-dependent features learned by the semantics, depth, and
refinement (SRH) heads. CTRL consists of a depth discretization, an entropy
map generation, and a fusion layer described in the following subsections.

####  3.4.1 Depth Discretization Module

The prediction made by the depth head $\hat{z}$ contains continuous depth
values. We want to map it to a discrete probability space to learn visual
semantic features at different depth levels. We quantize the view frustum
depth range into a set of representative discrete values following the
spacing-increasing discretization (SID) [15]. Such discretization assigns
progressively large depth sub-ranges further away from the point of view into
separate bins, which allows us to simulate the human perception of depth
relations in the scene, with a finite number of categories.

Given the depth range $[Z_{\text{min}},Z_{\text{max}}]$ and the number of
depth bins $K$, SID outputs a $K$-dimensional vector of discretization bin
centers $b$ as follows:

$b_{i}=Z_{\text{min}}^{1-(2i+1)/2K}\cdot Z_{\text{max}}^{(2i+1)/2K},\quad
i=0,\dots,K{-}1$ (4)

We can now assign probabilities of the predicted depth values falling into the
defined bins:

$\hat{z}^{\prime}=\text{softmax}(-(\hat{z}-b)^{2}).$ (5)

####  3.4.2 Joint Space for Domain Alignment

The task-dependency $\hat{y}_{r}$ (output by SRH), alongside the task-specific
semantics $\hat{y}_{s}$ and depth $\hat{z}^{\prime}$ probability maps, can be
considered as discrete distributions over semantic classes and depth levels.
As we do not have access to the ground truth labels for the target domain, one
way to train the network to predict high-confidence predictions is by
minimizing the uncertainty (or entropy) in the predicted distributions over
the target domain [61]. The source and target domains share similar spatial
features, and it is recommended to align them in the structured output space
[23].

To this end, we propose a novel UDA training scheme, where task-specific and
task-dependent knowledge is transferred from the source to the target domain
by constraining the target distributions to be similar to the source by
aligning the entropy maps of $\hat{y}_{r}$, $\hat{y}_{s}$, and
$\hat{z}^{\prime}$. Note that unlike [62, 61], which constrain only on the
task-specific space ($\hat{y}_{s}$ in our case) for domain alignment, we train
the network to output highly certain predictions by aligning features in the
task-specific and task-dependent spaces.

We argue that aligning source and target distributions jointly in task-
specific and task-dependent spaces helps to bridge the domain gap for
underrepresented classes, which are learned poorly without the presence of a
joint representation. To encode such a joint representation, we generate
entropy maps as follows:

$\begin{split}\begin{gathered}\mathcal{E}(p)=-p\odot\log p\\\
E_{r}=\mathcal{E}(\hat{y}_{r}),\;E_{s}=\mathcal{E}(\hat{y}_{s}),\;E_{d}=\mathcal{E}(\hat{z}^{\prime}).\end{gathered}\end{split}$
(6)

We then concatenate these maps along the channel dimension to get the fused
entropy map $E=\text{concat}(E_{r},E_{s},E_{d})$ and employ adversarial
training on it.

For aligning the source and target domain distributions, we train the proposed
segmentation and depth prediction network (parameterized by
$\theta_{\text{net}}$) and the discriminator network $\mathcal{D}$
(parameterized by $\theta_{\mathcal{D}}$) following an adversarial learning
scheme. More specifically, the discriminator is trained to correctly classify
the sample domain being either source or target given only the fused entropy
map:

$\begin{split}\min_{\theta_{\mathcal{D}}}\big{\\{}&\mathop{\mathbb{E}}_{\mathfrak{D}^{(s)}}\big{[}\log\mathcal{D}(E^{(s)})\big{]}+\\\
&\mathop{\mathbb{E}}_{\mathfrak{D}^{(t)}}\big{[}\log\big{(}1-\mathcal{D}(E^{(t)})\big{)}\big{]}\big{\\}}\end{split}$
(7)

At the same time, the prediction network parameters are learned to maximize
the domain classification loss (i.e., fooling the discriminator) on the target
samples using the following optimization objective:

$\min_{\theta_{\text{net}}}\mathop{\mathbb{E}}_{\mathfrak{D}^{(t)}}\left[\log\mathcal{D}(E^{(t)})\right]$
(8)

We use the hyperparameter $\lambda_{\text{adv}}$ weighing the relative
importance of the adversarial loss (8). Our training scheme jointly optimizes
the model parameters of the prediction network ($\theta_{\text{net}}$) and the
discriminator ($\theta_{\mathcal{D}}$). Updates to the prediction network and
the discriminator happen upon every training iteration; however, when updating
the prediction network, the discriminator parameters are kept fixed.
Parameters of the discriminator are updated separately using the domain
classification objective (Eq. 7).

###  3.5 Iterative Self Learning

Following prior work [74], we train our network end-to-end using an ISL scheme
using Algorithm 1. We first train the prediction ($\theta_{\text{net}}$) and
discriminator ($\theta_{\mathcal{D}}$) networks for $Q_{1}$ iterations. We
then generate semantic pseudo-labels ($\widetilde{y}^{(t)}$) on the target
training samples $x^{(t)}$ using the trained prediction network.

We further train the prediction network on the target training samples using
pseudo-labels supervision and a masked cross-entropy loss (Eq. 1), masking
target prediction pixels with confidence less than $0.9$, for $Q_{3}$
iterations. Instead of training the prediction network using SL only once, we
iterate over generating high-confidence pseudo-labels and self-training
$Q_{2}$ times to refine the pseudo-labels, further resulting in better quality
semantics output on the target domain.

We show in the ablation studies (Sec. 4.4) that our ISL scheme outperforms the
simple SL. The discriminator network parameters ($\theta_{\mathcal{D}}$) are
kept fixed during self-training.

Algorithm 1
ISL$(\mathfrak{D}^{(s)},\mathfrak{D}^{(t)},\theta_{\text{net}},\theta_{\mathcal{D}})$

1:Train prediction ($\theta_{\text{net}}$) and discriminator
($\theta_{\mathcal{D}}$) networks on source and target domains for $Q_{1}$
iterations;

2:for $Q_{2}$ times do

3: Generate
$\widetilde{y}_{s}^{(t)}=\mathcal{F}_{s}(\mathcal{F}_{e}(x^{(t)}))$ using
trained $\theta_{\text{net}}$;

4: Train $\theta_{\text{net}}$ on $(x^{(t)},\widetilde{y}_{s}^{(t)})$ for
$Q_{3}$ iterations;

5:end for

###  3.6 Network Architecture

The shared part of the prediction network $\mathcal{F}_{e}$ consists of a
ResNet-101 backbone and a decoder (Fig. 2). The decoder consists of four
convolutional layers; its outputs are fused with the backbone output features,
which are denoted as the “shared feature map”. This shared feature map is then
fed forward to the respective semantics and semantics refinement heads.
Following the residual auxiliary block [43] (as in [62]), we place the depth
prediction head between the last two convolutional layers of the decoder. In
the supplementary materials, we show that our proposed approach is not
sensitive to the residual auxiliary block and performs equally well with a
standard multi-task learning network architecture (i.e., a shared encoder
followed by multiple task-specific decoders). We adopt the Deeplab-V2 [4]
architectural design with Atrous Spatial Pyramid Pooling (ASPP) for the
prediction heads. We use DC-GAN [49] as our domain discriminator for
adversarial learning.

##  4 Experiments

Table 1:  Semantic segmentation performance (IoU and mIoU, %) comparison to
the prior art. All models are trained and evaluated using the EP1 protocol.
mIoU* is computed on a subset of $13$ classes, excluding those marked with *.
For our method, we report the results of the run giving the best mIoU, as well
as 68% confidence interval over five runs as mean$\scriptstyle\pm$std.

|  | SYNTHIA $\rightarrow$ Cityscapes (16 classes)
---|---|---
Models | Depth |  road |  sidewalk |  building |  wall* |  fence* |  pole* |  light |  sign |  veg |  sky |  person |  rider |  car |  bus |  mbike |  bike | mIoU $\uparrow$ | mIoU* $\uparrow$
SPIGAN-no-PI [34] |  | 69.5 | 29.4 | 68.7 | 4.4 | 0.3 | 32.4 | 5.8 | 15.0 | 81.0 | 78.7 | 52.2 | 13.1 | 72.8 | 23.6 | 7.9 | 18.7 | 35.8 | 41.2
SPIGAN [34] | ✓ | 71.1 | 29.8 | 71.4 | 3.7 | 0.3 | 33.2 | 6.4 | 15.6 | 81.2 | 78.9 | 52.7 | 13.1 | 75.9 | 25.5 | 10.0 | 20.5 | 36.8 | 42.4
AdaptSegnet [55] |  | 79.2 | 37.2 | 78.8 | - | - | - | 9.9 | 10.5 | 78.2 | 80.5 | 53.5 | 19.6 | 67.0 | 29.5 | 21.6 | 31.3 | - | 45.9
AdaptPatch [56] |  | 82.2 | 39.4 | 79.4 | - | - | - | 6.5 | 10.8 | 77.8 | 82.0 | 54.9 | 21.1 | 67.7 | 30.7 | 17.8 | 32.2 | - | 46.3
CLAN [41] |  | 81.3 | 37.0 | 80.1 | - | - | - | 16.1 | 13.7 | 78.2 | 81.5 | 53.4 | 21.2 | 73.0 | 32.9 | 22.6 | 30.7 | - | 47.8
Advent [61] |  | 87.0 | 44.1 | 79.7 | 9.6 | 0.6 | 24.3 | 4.8 | 7.2 | 80.1 | 83.6 | 56.4 | 23.7 | 72.7 | 32.6 | 12.8 | 33.7 | 40.8 | 47.6
DADA [62] | ✓ | 89.2 | 44.8 | 81.4 | 6.8 | 0.3 | 26.2 | 8.6 | 11.1 | 81.8 | 84.0 | 54.7 | 19.3 | 79.7 | 40.7 | 14.0 | 38.8 | 42.6 | 49.8
Ours (best mIoU) | ✓ | 88.1 | 39.1 | 81.8 | 19.6 | 0.7 | 30.2 | 7.7 | 13.8 | 82.4 | 83.0 | 60.2 | 22.0 | 81.7 | 47.6 | 24.8 | 35.0 | 44.9 | 51.3
Ours (confidence) | ✓ |  | 85.5
---
$\scriptstyle\pm$3.3
| 40.6
---
$\scriptstyle\pm$4.4
| 80.0
---
$\scriptstyle\pm$1.1
| 16.5
---
$\scriptstyle\pm$3.3
| 0.73
---
$\scriptstyle\pm$0.2
| 28.5
---
$\scriptstyle\pm$1.6
| 7.2
---
$\scriptstyle\pm$3.5
| 11.7
---
$\scriptstyle\pm$3.4
| 80.7
---
$\scriptstyle\pm$1.6
| 82.4
---
$\scriptstyle\pm$0.6
| 60.2
---
$\scriptstyle\pm$1.1
| 22.2
---
$\scriptstyle\pm$4.6
| 79.1
---
$\scriptstyle\pm$5.1
| 47.2
---
$\scriptstyle\pm$5.7
| 26.0
---
$\scriptstyle\pm$4.8
| 40.6
---
$\scriptstyle\pm$5.6
| 44.3
---
$\scriptstyle\pm$0.6
| 51.0
---
$\scriptstyle\pm$0.7

Table 2:  Semantic segmentation performance (IoU and mIoU, %) comparison to
the prior art. All models are trained and evaluated using the EP2 and EP3
protocols at different resolutions, as indicated in the resolution (“Res.”)
column. For our method, we report the results of the run giving the best mIoU,
as well as 68% confidence interval over five runs as mean$\scriptstyle\pm$std.

|  |  | (a) SYNTHIA $\rightarrow$ Cityscapes (7 classes) |  | (b) SYNTHIA $\rightarrow$ Mapillary (7 classes)
---|---|---|---|---|---
Res. | Model | Depth |  flat |  const |  object |  nature |  sky |  human |  vehicle | mIoU $\uparrow$ |  |  flat |  const |  object |  nature |  sky |  human |  vehicle | mIoU $\uparrow$
$320\times 640$ | SPIGAN-no-PI [34] |  | 90.3 | 58.2 | 6.8 | 35.8 | 69.0 | 9.5 | 52.1 | 46.0 |  | 53.0 | 30.8 | 3.6 | 14.6 | 53.0 | 5.8 | 26.9 | 26.8
SPIGAN [34] | ✓ | 91.2 | 66.4 | 9.6 | 56.8 | 71.5 | 17.7 | 60.3 | 53.4 |  | 74.1 | 47.1 | 6.8 | 43.3 | 83.7 | 11.2 | 42.2 | 44.1
Advent [61] |  | 86.3 | 72.7 | 12.0 | 70.4 | 81.2 | 29.8 | 62.9 | 59.4 |  | 82.7 | 51.8 | 18.4 | 67.8 | 79.5 | 22.7 | 54.9 | 54.0
DADA [62] | ✓ | 89.6 | 76.0 | 16.3 | 74.4 | 78.3 | 43.8 | 65.7 | 63.4 |  | 83.8 | 53.7 | 20.5 | 62.1 | 84.5 | 26.6 | 59.2 | 55.8
Ours (best mIoU) | ✓ | 90.9 | 78.3 | 16.6 | 77.4 | 83.6 | 43.5 | 67.6 | 65.4 |  | 85.8 | 59.8 | 18.4 | 71.0 | 89.7 | 44.7 | 64.1 | 61.9
Ours (confidence) | ✓ |  | 90.2
---
$\scriptstyle\pm$0.6
| 77.9
---
$\scriptstyle\pm$0.7
| 16.1
---
$\scriptstyle\pm$0.5
| 76.5
---
$\scriptstyle\pm$1.0
| 82.9
---
$\scriptstyle\pm$0.9
| 42.0
---
$\scriptstyle\pm$5.0
| 68.7
---
$\scriptstyle\pm$1.4
| 64.9
---
$\scriptstyle\pm$0.3
|  | 85.5
---
$\scriptstyle\pm$0.2
| 58.6
---
$\scriptstyle\pm$1.5
| 19.1
---
$\scriptstyle\pm$1.3
| 69.2
---
$\scriptstyle\pm$2.5
| 88.7
---
$\scriptstyle\pm$1.1
| 44.0
---
$\scriptstyle\pm$0.9
| 63.6
---
$\scriptstyle\pm$1.8
| 61.3
---
$\scriptstyle\pm$0.5
Full | Advent [61] |  | 89.6 | 77.8 | 22.1 | 76.3 | 81.4 | 54.7 | 68.7 | 67.2 |  | 86.9 | 58.8 | 30.5 | 74.1 | 85.1 | 48.3 | 72.5 | 65.2
DADA [62] | ✓ | 92.3 | 78.3 | 25.0 | 75.5 | 82.2 | 58.7 | 72.4 | 69.2∗ |  | 86.7 | 62.1 | 34.9 | 75.9 | 88.6 | 51.1 | 73.8 | 67.6
Oracle (only-target) |  | 97.6 | 87.9 | 46.0 | 87.9 | 88.8 | 69.1 | 88.6 | 80.8 |  | 95.0 | 84.2 | 54.8 | 87.7 | 97.2 | 70.2 | 87.5 | 82.4
Ours (best mIoU) | ✓ | 93.0 | 81.3 | 29.0 | 78.8 | 83.2 | 63.2 | 77.5 | 72.3 |  | 88.9 | 66.5 | 35.7 | 78.9 | 92.0 | 61.3 | 79.2 | 71.8
Ours (confidence) | ✓ |  | 92.1
---
$\scriptstyle\pm$0.7
| 80.7
---
$\scriptstyle\pm$0.7
| 28.3
---
$\scriptstyle\pm$0.5
| 77.8
---
$\scriptstyle\pm$1.7
| 83.1
---
$\scriptstyle\pm$1.1
| 60.9
---
$\scriptstyle\pm$3.8
| 76.4
---
$\scriptstyle\pm$1.0
| 71.3
---
$\scriptstyle\pm$0.9
|  | 88.5
---
$\scriptstyle\pm$0.3
| 64.8
---
$\scriptstyle\pm$2.8
| 34.4
---
$\scriptstyle\pm$1.2
| 78.9
---
$\scriptstyle\pm$1.0
| 90.5
---
$\scriptstyle\pm$2.0
| 61.3
---
$\scriptstyle\pm$1.4
| 78.1
---
$\scriptstyle\pm$1.1
| 70.9
---
$\scriptstyle\pm$0.7
∗ The correct mean of class IoU values in Table 2 of [62]. |

###  4.1 UDA Benchmarks

We use three standard UDA evaluation protocols (EPs) to validate our model:
EP1: SYNTHIA $\rightarrow$ Cityscapes (16 classes), EP2: SYNTHIA $\rightarrow$
Cityscapes (7 classes), and EP3: SYNTHIA $\rightarrow$ Mapillary (7 classes).
A detailed explanation of these settings can be found in [62]. In all
settings, the SYNTHIA dataset [51] is used as the synthetic source domain. In
particular, we use the SYNTHIA-RAND-CITYSCAPES split consisting of 9,400
synthetic images and their corresponding pixel-wise semantic and depth
annotations. For target domains, we use Cityscapes [10] and Mapillary Vistas
[44] datasets. Following EP1, we train models on 16 classes common to SYNTHIA
and Cityscapes; in EP2 and EP3, models are trained on 7 classes common to
SYNTHIA, Cityscapes, and Mapillary. We use intersection-over-union to evaluate
segmentation: IoU (class-IoU) and mIoU (mean-IoU). To promote reproducibility
and emphasize significance of our results, we report two outcomes: the best
mIoU, and the confidence interval. The latter is denoted as
mean$\scriptstyle\pm$std collected over five runs, thus describing a 68%
confidence interval centered at mean111 Class-IoU values of the ”best mIoU”
setting can be less than the mean of the class confidence interval at the
expense of other classes performance. . For depth, we use Absolute Relative
Difference ($|\text{Rel}|$), Squared Relative Difference (Rel2), Root Mean
Squared Error (RMS), its log-variant LRMS; and the accuracy metrics [14] as
denoted by $\delta_{1}$, $\delta_{2}$, and $\delta_{3}$. For each metric, we
use $\uparrow$ and $\downarrow$ to denote the improvement direction.

###  4.2 Experimental Setup

All our experiments are implemented in PyTorch [47]. Backbone network is a
ResNet-101 [22] initialized with ImageNet [12] weights. The prediction and
discriminator networks are optimized with SGD [2] and Adam [30] with learning
rates $2.5\times 10^{-4}$ and $10^{-4}$ respectively. Throughout our
experiments, we use $\lambda_{r}=1.0$, $\lambda_{d}=\lambda_{adv}=10^{-3}$.
For generating depth bins, we use $Z_{\text{min}}=1$m,
$Z_{\text{max}}=655.36$m, and $K=15$. In all ISL experiments, parameters of
the algorithm are: $Q_{1}=65K$, $Q_{2}=5$, $Q_{3}=5K$. Link to the project
page with source code is in the Abstract.

Figure 3:  Qualitative semantic segmentation results with EP1: SYNTHIA
$\rightarrow$ Cityscapes (16 classes). (a) Images from Cityscapes validation
set; (b) ground truth annotations; (c) DADA [62] predictions; (d) our model
predictions. Our method demonstrates notable improvements over [62] on “bus”,
“person”, “motorbike”, and “bicycle” classes as highlighted using the yellow
boxes.  Figure 4:  Qualitative semantic segmentation results with EP3: SYNTHIA
$\rightarrow$ Mapillary-Vista ($7$ classes). Top: images from Mapillary
validation set; Middle: ground truth annotations; Bottom: our model
predictions.

###  4.3 Comparison to Prior Art

####  4.3.1 EP1

Table 1 reports semantic segmentation performance of our proposed model
trained and evaluated following EP1. For a fair comparison with [55, 56, 41],
we also report results on 13 classes and the standard 16 classes settings. Our
method achieves SOTA performance in EP1 on both $16$ and $13$ classes,
outperforming [62, 34] by large margins. Now we can identify the major class-
specific improvements of our method over the SOTA [62] DADA. The major gains
come from the following classes – “wall” ($+12.8\%$), “motorbike” ($+10.8\%$),
“bus” ($+6.9\%$), “person” ($+5.5\%$), “rider” ($+2.7\%$) and “car”
($+2.0\%$). Moreover, our method shows consistent improvements on classes
underrepresented in the target domain: “motorbike” ($+10.8\%$), “pole”
($+4.0\%$), “sign” ($+2.7\%$), and “bicycle” ($+1.8\%$). Fig. 3 shows the
results of the qualitative comparison of our method with DADA [62]. Note that
our model delineates small objects like “human”, “bicycle”, and “motorbike”
more accurately than DADA.

####  4.3.2 EP2 and EP3

Table 2 presents the semantic segmentation results in EP2 and EP3 benchmarks.
The models are evaluated on the Cityscapes and Mapillary validation sets on
their common $7$ classes. We also train and evaluate our model on the
$320\times 640$ resolution to obtain a fair comparison with the reference low-
resolution models. In a similar vein, the proposed method outperforms the
prior works in EP2 and EP3 benchmarks for both full- and low-resolution
($640\times 320$) settings. We further show in Sec. 4.5 that our approach
achieves state-of-the-art performance without ISL in EP2 and EP3 in both full-
and low-resolution settings. The proposed CTRL coupled with SRH demonstrates
consistent improvements over three challenging benchmarks by capitalizing on
the inherent semantic and depth correlations. In EP2 and EP3, our models show
noticeable improvements over the state-of-the-art [62] with mIoU gains of
$+3.1\%$ (EP2-full-res), $+2.0\%$ (EP2-low-res), $+4.2\%$ (EP3-full-res),
$+6.1\%$ (EP3-low-res). Despite the challenging domain gap between SYNTHIA and
Mapillary, our model shows significant improvement ($+6.1\%$) in a low-
resolution setting, which suggests robustness to scale changes.

Table 3:  Ablation study of our method from Sec. 4.4.

Conf |  SemSup |  DepSup |  SRHSup |  SemAdv |  DepAvd |  SRHAdv |  SL |  ISL | mIoU (%) $\uparrow$
---|---|---|---|---|---|---|---|---|---
$C1$ | ✓ |  |  |  |  |  |  |  | 30.7
$C2$ | ✓ | ✓ |  |  |  |  |  |  | 35.2
$C3$ |  | ✓ | ✓ |  |  |  |  |  | 33.7
$C4$ | ✓ | ✓ | ✓ |  |  |  |  |  | 33.1
$C5$ | ✓ |  |  | ✓ |  |  |  |  | 40.8
$C6$ | ✓ | ✓ |  | ✓ | ✓ |  |  |  | 40.2
$C7$ |  | ✓ | ✓ |  | ✓ | ✓ |  |  | 39.5
$C8$ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  |  | 42.1
$C9$ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  | 44.1
$C10$ | ✓ | ✓ |  | ✓ | ✓ |  |  | ✓ | 42.8
$C11$ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  | ✓ | 44.9

###  4.4 Ablation Studies

A comprehensive ablation study is reported in Table 3. We trained $11$
different models, each having a different configuration; these are denoted as
$C1,...,C11$. We use the following shortcuts in Table 3 to represent different
combinations of settings: “Sem” – semantic, “Dep” – depth, “Sup” –
supervision, “Adv” – adversarial, and “Conf” – configuration. Configurations
$C1$ to $C4$ denote supervised learning settings without any adversarial
training. These models are trained on the SYNTHIA dataset and evaluated on
Cityscapes validation set. Configurations from $C5$ to $C7$ denote different
combinations of supervised and adversarial losses on the semantics, depth, and
semantics refinement heads. $C8$ is the proposed model with CTRL, but without
ISL. $C9$ to $C11$ are models trained with SL or ISL with or without SRH. $C5$
to $C11$ follow EP1 protocol: SYNTHIA $\rightarrow$ Cityscapes UDA training
and evaluation setting.

$C1$ is trained using semantics label supervision without any depth
information or adversarial learning. By enabling parts of the model and
training procedure, we observed the following tendencies: $C2$ & $C3$ : depth
supervision (either direct or through SRH) improves performance; $C4$:
however, adding SRH on top of the depth head in the supervised learning
setting does not bring improvements; $C5$: effectiveness of entropy map domain
alignment in semantics feature space [61]; $C6$ and $C7$: domain alignment in
the depth or refined semantics feature spaces do not bring any further
improvements; $C8$: a combination of depth and SRH with task-specific
semantics improves the performance (i.e., our CTRL model); $C9$: SL brings
further improvement but not as good as with our ISL training scheme; $C10$:
emphasizes the improvement over $C6$ with ISL enabled; $C11$: positive
contribution of the SRH towards improving the overall model performance.
Finally, we achieve state-of-the-art segmentation results (mIoU $44.9\%$) by
combining the proposed CTRL, SRH, and ISL (configuration $C11$).

Table 4:  Effectiveness of the joint feature space learned by our method (w/o
ISL) for robust domain alignment. Performance in mIoU; legend for “Ours” as in
Table 2.

Model |  $\text{S}{\rightarrow}\text{C}$ (FR) |  $\text{S}{\rightarrow}\text{M}$ (FR) |  $\text{S}{\rightarrow}\text{C}$ (LR) |  $\text{S}{\rightarrow}\text{M}$ (LR)
---|---|---|---|---
DADA [62] | 69.2 | 67.6 | 63.4 | 55.8
Ours (best mIoU) |  70.7 |  67.7 |  64.7 | 58.7
Ours (confidence) |  | 70.2
---
$\scriptstyle\pm$0.6
| 66.8
---
$\scriptstyle\pm$0.9
| 64.1
---
$\scriptstyle\pm$0.5
| 58.0
---
$\scriptstyle\pm$0.7

Table 5:  Improvement over the state-of-the-art [62] in monocular depth
estimation. The models are trained following SYNTHIA $\rightarrow$ Cityscapes
(16 classes) UDA setting w/o ISL and evaluated on the Cityscapes validation
set.

Model |  $|\text{Rel}|{\downarrow}$ |  Rel${}^{2}{\downarrow}$ |  RMS$\downarrow$ |  LRMS$\downarrow$ | $\delta_{1}{\uparrow}$ | $\delta_{2}{\uparrow}$ | $\delta_{3}{\uparrow}$
---|---|---|---|---|---|---|---
DADA [62] | 0.6 | 10.8 | 17.0 | 4.4 | 0.14 | 0.28 | 0.41
Ours | 0.3 | 6.3 | 14.8 | 0.6 | 0.30 | 0.58 | 0.77

###  4.5 Additional Experimental Analysis

####  4.5.1 Effectiveness of the Joint UDA Feature Space

This section analyzes the effectiveness of joint feature space learned by the
CTRL for unsupervised domain alignment. We train and evaluate our CTRL model
without ISL on two UDA benchmarks: (a) EP2: SYNTHIA to Cityscapes $7$ classes
(S $\rightarrow$ C) and (b) EP3: SYNTHIA to Mapillary $7$ classes (S
$\rightarrow$ M) in both full- and low-resolution (FR and LR) settings. In
Table 4, we show the segmentation performance of our model on these four
different benchmark settings and compare it against the state-of-the-art DADA
model [62]. The proposed CTRL model (w/o ISL) outperforms the DADA model with
mIoU gains of $+1.5\%$, $+0.1\%$, $+1.3\%$, and $+2.9\%$ on all four UDA
benchmark settings attesting the effectiveness of the joint feature space
learned by the proposed CTRL.

Besides, we train both DADA and our model with ISL and notice improvements in
both the models with mIoU $43.5\%$ (DADA) and $44.9\%$ (ours). The superior
quality of the predictions of our model, when used as pseudo labels, provides
better supervision to the target semantics; the same can be observed in both
our quantitative (Tables 1 and 2) and qualitative results (Figs. 3 and 4).

####  4.5.2 Monocular Depth Estimation Results

In this section, we show that our model not only improves semantic
segmentation but also learns a better representation for monocular depth
estimation. This intriguing property is of great importance for multi-task
learning. According to [43], paying too much attention to depth is detrimental
to the segmentation performance. Following [43], DADA [62] uses depth as
purely auxiliary supervision. We observed that depth predictions of [62] are
noisy (also admitted by the authors), resulting in failure cases. We
conjecture that a proper architectural design choice coupled with a robust
multi-tasking feature representation (encoding task-specific and cross-task
relationship) improves both semantics and depth. In Table 5, we report the
depth estimation evaluation results on the Cityscapes validation set of our
method and compare it against the DADA model [62]. Training and evaluation are
done following the EP1 protocol: SYNTHIA $\rightarrow$ Cityscapes (16
classes). We use Cityscapes disparity maps as ground truth depth pseudo-labels
for evaluation. Table 5 demonstrates a consistent improvement of depth
predictions with our method over [62].

##  5 Conclusion

We proposed a novel approach to semantic segmentation and monocular depth
estimation within a UDA context. The main highlights of this work are: (1) a
Cross-Task Relation Layer (CTRL), which learns a joint feature space for
domain alignment; the joint space encodes both task-specific features and
cross-task dependencies shown to be useful for UDA; (2) a semantic refinement
head (SRH) aids in learning task correlations; (3) a depth discretizing
technique facilitates learning distinctive relationship between different
semantic classes and depth levels; (4) a simple yet effective iterative self-
learning (ISL) scheme further improves the model’s performance by capitalizing
on the high confident predictions in the target domain. Our comprehensive
experimental analysis demonstrates that the proposed method consistently
outperforms prior works on three challenging UDA benchmarks by a large margin.

Acknowledgments. The authors gratefully acknowledge the support by armasuisse.
We thank Amazon Activate for EC2 credits and the anonymous reviewers for the
valuable feedback and time spent.

Learning to Relate Depth and Semantics for Unsupervised Domain Adaptation
Supplementary Materials

Figure 5: Overview of the UDA setting

In this document, we provide supplementary materials for our main paper
submission. First, Sec. 6 provides a bird-eye view of the assumed UDA setting
and how CTRL fits into it. The main paper reported our experimental results
using three standard UDA evaluation protocols (EPs) where the SYNTHIA dataset
[51] is used as the synthetic domain. To demonstrate our proposed method’s
effectiveness on an entirely new UDA setting, in Sec. 7, we report semantic
segmentation results of our method on a new EP: Virtual KITTI $\rightarrow$
KITTI. In this setup, we use synthetic Virtual KITTI [17] as the source domain
and real KITTI [20] as the target domain. We show that our proposed method
consistently outperforms the SOTA DADA method [62] when evaluated on this new
EP with different synthetic and real domains. In Sec. 8, we present a t-SNE
[57] plot comparing our method with [62]. We also share additional qualitative
results on SYNTHIA $\rightarrow$ Cityscapes (16 classes). Sec. 9 details our
network design. To demonstrate that the proposed CTRL is not sensitive to a
particular network design (in our case, the residual auxiliary block [43]), we
train a standard multi-task learning network architecture (i.e., a shared
encoder followed by multiple task-specific decoders without any residual
auxiliary block) with CTRL and notice a similar improvement trend over the
baselines. The set of experiments and the results are discussed in Sec. 10.

##  6 Overview of the UDA setting

Unsupervised Domain Adaptation (UDA) aims at training high-performance models
with no label supervision on the target domain. As seen in Fig. 5, label
supervision is applied only on the source domain predictions, whereas tuning
the model to perform well on the target domain is the task of adversarial
supervision. Since both types of supervision are applied within the same
training protocol, adversarial supervision is responsible for teaching the
model the specificity of the target domain by means of bridging the domain
gap. When dealing with multi-modal predictions, it is crucial to choose the
joint feature space subject to adversarial supervision correctly. CTRL
provides such rich feature space, which allows training much better models
using the same training protocols. This allows us to leverage the abundance of
samples in the synthetic source domain and produce high-quality predictions in
the real target domain.

##  7 Virtual KITTI $\rightarrow$ KITTI

Following [7], we train and evaluate our model on $10$ common classes of
Virtual KITTI and KITTI. In KITTI, the ground-truth label is only available
for the training set; thus, we use the official unlabelled test images for
domain alignment. We report the results on the official training set following
[7]. The model is trained on the annotated training samples of VKITTI and
unannotated samples of KITTI. For this experiment, we train our model without
(w/o) ISL. Table 6 reports the semantic segmentation performance (mIoU%) of
our approach. Our model outperforms DADA [62], with significant gains coming
from the following classes: “sign” ($+8.1\%$), “pole” ($+5.7\%$), “building”
($+2.7\%$), and “light” ($+1.9\%$). Notably, these classes are practically
highly relevant to an autonomous driving scenario. In Figure 7, we present
some qualitative results of DADA and our models trained following the new
Virtual KITTI $\rightarrow$ KITTI UDA protocol.

Table 6:  Semantic segmentation performance (IoU and mIoU, higher is better, %) comparison to the prior art. All models are trained and evaluated using the UDA evaluation protocol Virtual KITTI $\rightarrow$ KITTI.  |  | VKITTI $\rightarrow$ KITTI (10 classes)
---|---|---
Models | Depth |  road |  building |  pole |  light |  sign |  veg |  terrain |  sky |  car |  truck | mIoU $\uparrow$
Chen et al.[7] | ✓ | 81.4 | 71.2 | 11.3 | 26.6 | 23.6 | 82.8 | 56.5 | 88.4 | 80.1 | 12.7 | 53.5
DADA [62] | ✓ | 90.9 | 76.2 | 12.4 | 30.3 | 30.8 | 73.5 | 24.1 | 88.4 | 86.8 |  17.2 | 53.0
Ours (w/o ISL) | ✓ | 90.9 | 78.9 | 18.1 | 32.2 | 38.9 | 73.7 | 22.0 | 88.2 | 86.2 | 16.7 | 54.6

Table 7: Semantic segmentation performance (mIoU) of two variants of the
proposed model. Both models outperform DADA [62] attesting the robustness of
features learned by the proposed CTRL.

UDA Protocol | DADA | Ours$*$ | Ours
---|---|---|---
S $\rightarrow$ C 16 cls | 42.6 | 43.7$\scriptstyle\pm$0.2 | 44.3$\scriptstyle\pm$0.6
S $\rightarrow$ C (LR) 7 cls | 63.4 | 63.8$\scriptstyle\pm$0.5 | 64.9$\scriptstyle\pm$0.3
S $\rightarrow$ M (LR) 7 cls | 55.8 | 61.5$\scriptstyle\pm$0.6 | 61.3$\scriptstyle\pm$0.5
S $\rightarrow$ C (FR) 7 cls | 69.2 | 71.3$\scriptstyle\pm$0.5 | 71.3$\scriptstyle\pm$0.9
S $\rightarrow$ M (FR) 7 cls | 67.6 | 70.1$\scriptstyle\pm$0.5 | 70.9$\scriptstyle\pm$0.7

Figure 6: t-SNE comparison of features learned by DADA [62] and CTRL. It leads
to more structured feature space and better class separation in the target
domain. Circled classes have a better separation than the other method.

Figure 7:  Qualitative semantic segmentation results with VKITTI $\rightarrow$
KITTI (10 classes) UDA evaluation protocol. (a) Input image from the target
domain KITTI; (b) ground truth annotations; (c) DADA [62] predictions; (d) our
model predictions. We follow the color encoding scheme of Cityscapes to
colorize the label maps.

##  8 SYNTHIA $\rightarrow$ Cityscapes

This section presents a t-SNE [57] plot of the feature embeddings learned by
the proposed model guided by CTRL, and [62]. Fig. 6 shows 10 top-scoring
classes of each method; distinct classes are circled. As can be seen from the
figure, CTRL leads to more structured feature space, which concurs with our
analysis of the main paper. Both models are trained and evaluated following
the UDA protocol SYNTHIA $\rightarrow$ Cityscapes (16 classes). Furthermore,
we present additional qualitative results of our model for semantic
segmentation and monocular depth estimation. Figures 8, 9 show the results of
the qualitative comparison of our method with [62]. Note that our proposed
method has higher spatial acuity in delineating small objects like “human”,
“bicycle”, and “person” compared to [62]. Figure 10 shows some qualitative
monocular depth estimation results.

Figure 8:  Qualitative semantic segmentation results with EP1: SYNTHIA
$\rightarrow$ Cityscapes (16 classes). (a) Images from Cityscapes validation
set; (b) ground truth annotations; (c) DADA [62] predictions; (d) our model
predictions. Our method demonstrates notable improvements over [62] on “bus”,
“person”, and “bicycle” classes as highlighted using the yellow boxes.  Figure
9:  Qualitative semantic segmentation results with EP1: SYNTHIA $\rightarrow$
Cityscapes (16 classes). (a) Images from Cityscapes validation set; (b) ground
truth annotations; (c) DADA [62] predictions; (d) our model predictions. Our
method demonstrates notable improvements over [62] on “bus”, “person”, and
“bicycle” classes as highlighted using the yellow boxes.  Figure 10:
Qualitative monocular depth estimation results with EP1: SYNTHIA $\rightarrow$
Cityscapes (16 classes). (a) Images from Cityscapes validation set; (b) ground
truth annotations; (c) DADA [62] predictions; (d) our model predictions.

##  9 Network Architecture Design

The shared part of the semantic and depth prediction network $\mathcal{F}_{e}$
consists of a ResNet-101 backbone and a decoder. The decoder consists of four
convolutional layers, each followed by a Rectified Linear Unit (ReLU). The
decoder outputs a feature map that is shared among both semantics and depth
heads. This shared feature map is fed forward to the respective semantic
segmentation, monocular depth estimation, and semantics refinement heads. For
the task-specific and task-refinement heads, we use Atrous Spatial Pyramid
Pooling (ASPP) with sampling rates $[6,12,18,24]$ and the Deeplab-V2 [4]
architecture. Our DC-GAN [49] based domain discriminator takes as input a
feature map with channel dimension $2\times C+K$, where $C$ is the number of
semantic classes, and $K$ is the number of depth levels.

##  10 Robustness to Different Network Design

Our proposed model adopts the residual auxiliary block [43] (as in [62]),
which was originally proposed to tackle a particular MTL setup where the
objective was to improve one primary task by leveraging several other
auxiliary tasks. However, unlike [62] which doesn’t have any decoder for
depth, we introduce a DeepLabV2 decoder for depth estimation to improve both
task performances. Our qualitative and quantitative experimental results show
an improvement of depth estimation performance over [62]. Furthermore, we are
interested to see the proposed model’s performance when used with a standard
MTL architecture (a common encoder followed by multiple task-specific decoders
without any residual auxiliary blocks). To this end, we make necessary changes
to our existing network design to have a standard MTL network design. We then
train it following UDA protocols. The details of our experimental analysis are
given below.

For the standard MTL model (denoted as “Ours*” in Table 7), the depth head is
placed after the shared feature extractor $\mathcal{F}_{e}$. The shared
feature extractor consists of a ResNet backbone and decoder network (see Fig.
2). For the second model with residual auxiliary block (denoted as “Ours”), we
positioned the depth head after the decoder’s third convolutional layer. The
semantic segmentation performance of these two variants of the proposed model
is shown in Table 7. Both models are evaluated on the five different UDA
protocols and outperform state-of-the-art DADA [62] results. The results show
that our proposed CTRL is not sensitive to architectural changes and can be
used with standard encoder-decoder MTL frameworks. Our findings may be found
beneficial for the domain-adaptive MTL community, e.g., in answering a
question whether learning additional complementary tasks (surface normals,
instance segmentation) performs domain alignment.

## References

  * [1] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  IEEE Trans. Pattern Anal. Mach. Intell., 39(12):2481–2495, 2017\.
  * [2] Léon Bottou.  Large-scale machine learning with stochastic gradient descent.  In Proceedings of COMPSTAT’2010, pages 177–186. Springer, 2010\.
  * [3] David Bruggemann, Menelaos Kanakis, Stamatios Georgoulis, and Luc Van Gool.  Automated search for resource-efficient branched multi-task networks.  Brit. Mach. Vis. Conf., 2020.
  * [4] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE Trans. Pattern Anal. Mach. Intell., 40(4):834–848, 2017.
  * [5] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Eur. Conf. Comput. Vis., pages 801–818, 2018.
  * [6] Po-Yi Chen, Alexander H Liu, Yen-Cheng Liu, and Yu-Chiang Frank Wang.  Towards scene understanding: Unsupervised monocular depth estimation with semantic-aware representation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 2624–2632, 2019\.
  * [7] Yuhua Chen, Wen Li, Xiaoran Chen, and Luc Van Gool.  Learning semantic segmentation from synthetic data: A geometrically guided input-output adaptation approach.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 1841–1850, 2019\.
  * [8] Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  Domain adaptive faster r-cnn for object detection in the wild.  In CVPR, 2018.
  * [9] Yuhua Chen, Wen Li, and Luc Van Gool.  Road: Reality oriented adaptation for semantic segmentation of urban scenes.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 7892–7901, 2018\.
  * [10] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 3213–3223, 2016\.
  * [11] Gabriela Csurka.  A comprehensive survey on domain adaptation for visual applications.  In Domain adaptation in computer vision applications, pages 1–35. Springer, 2017.
  * [12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 248–255. Ieee, 2009.
  * [13] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  In Adv. Neural Inform. Process. Syst., 2014.
  * [14] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  In Adv. Neural Inform. Process. Syst., pages 2366–2374, 2014.
  * [15] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao.  Deep ordinal regression network for monocular depth estimation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 2002–2011, 2018\.
  * [16] Brian Fulkerson, Andrea Vedaldi, and Stefano Soatto.  Class segmentation and object localization with superpixel neighborhoods.  In Int. Conf. Comput. Vis., pages 670–677. IEEE, 2009.
  * [17] Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora Vig.  Virtual worlds as proxy for multi-object tracking analysis.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4340–4349, 2016.
  * [18] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  In International Conference on Machine Learning (ICML), pages 1180–1189, 2015.
  * [19] Timnit Gebru, Judy Hoffman, and Li Fei-Fei.  Fine-grained recognition in the wild: A multi-task domain adaptation approach.  In Int. Conf. Comput. Vis., 2017.
  * [20] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  The International Journal of Robotics Research, 32(11):1231–1237, 2013.
  * [21] Vitor Guizilini, Rui Hou, Jie Li, Rares Ambrus, and Adrien Gaidon.  Semantically-guided representation learning for self-supervised monocular depth.  In Int. Conf. Learn. Represent., 2020.
  * [22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 770–778, 2016\.
  * [23] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell.  Cycada: Cycle-consistent adversarial domain adaptation.  In International conference on machine learning, pages 1989–1998. PMLR, 2018.
  * [24] Judy Hoffman, Dequan Wang, Fisher Yu, and Trevor Darrell.  Fcns in the wild: Pixel-level adversarial and constraint-based adaptation.  arXiv preprint arXiv:1612.02649, 2016.
  * [25] Javed Iqbal and Mohsen Ali.  Mlsl: Multi-level self-supervised learning for domain adaptation with spatially independent and semantically consistent labeling.  In The IEEE Winter Conference on Applications of Computer Vision, pages 1864–1873, 2020.
  * [26] Huaizu Jiang, Deqing Sun, Varun Jampani, Zhaoyang Lv, Erik Learned-Miller, and Jan Kautz.  Sense: A shared encoder network for scene-flow estimation.  In Int. Conf. Comput. Vis., pages 3195–3204, 2019.
  * [27] Jianbo Jiao, Ying Cao, Yibing Song, and Rynson Lau.  Look deeper into depth: Monocular depth estimation with semantic booster and attention-driven loss.  In Eur. Conf. Comput. Vis., pages 53–69, 2018.
  * [28] Menelaos Kanakis, David Bruggemann, Suman Saha, Stamatios Georgoulis, Anton Obukhov, and Luc Van Gool.  Reparameterizing convolutions for incremental multi-task learning without task interference.  In Eur. Conf. Comput. Vis., 2020.
  * [29] Alex Kendall, Yarin Gal, and Roberto Cipolla.  Multi-task learning using uncertainty to weigh losses for scene geometry and semantics.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 7482–7491, 2018\.
  * [30] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  Int. Conf. Learn. Represent., 2014.
  * [31] L’ubor Ladickỳ, Paul Sturgess, Karteek Alahari, Chris Russell, and Philip HS Torr.  What, where and how many? combining object detectors and crfs.  In Eur. Conf. Comput. Vis., pages 424–437. Springer, 2010.
  * [32] Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab.  Deeper depth prediction with fully convolutional residual networks.  In 2016 Fourth international conference on 3D vision (3DV), pages 239–248. IEEE, 2016.
  * [33] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [34] Kuan-Hui Lee, German Ros, Jie Li, and Adrien Gaidon.  Spigan: Privileged adversarial learning from simulation.  Int. Conf. Learn. Represent., 2019.
  * [35] Bo Li, Chunhua Shen, Yuchao Dai, Anton Van Den Hengel, and Mingyi He.  Depth and surface normal estimation from monocular images using regression on deep features and hierarchical crfs.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 1119–1127, 2015\.
  * [36] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales.  Deeper, broader and artier domain generalization.  In Int. Conf. Comput. Vis., 2017.
  * [37] Fayao Liu, Chunhua Shen, Guosheng Lin, and Ian Reid.  Learning depth from single monocular images using deep convolutional neural fields.  IEEE Trans. Pattern Anal. Mach. Intell., 38(10):2024–2039, Oct. 2016.
  * [38] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 3431–3440, 2015\.
  * [39] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I Jordan.  Learning transferable features with deep adaptation networks.  In International Conference on Machine Learning (ICML), pages 97–105, 2015.
  * [40] Hao Lu, Lei Zhang, Zhiguo Cao, Wei Wei, Ke Xian, Chunhua Shen, and Anton van den Hengel.  When unsupervised domain adaptation meets tensor representations.  In Int. Conf. Comput. Vis., 2017.
  * [41] Yawei Luo, Liang Zheng, Tao Guan, Junqing Yu, and Yi Yang.  Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 2507–2516, 2019\.
  * [42] Kevis-Kokitsi Maninis, Ilija Radosavovic, and Iasonas Kokkinos.  Attentive single-tasking of multiple tasks.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 1851–1860, 2019\.
  * [43] Taylor Mordan, Nicolas Thome, Gilles Henaff, and Matthieu Cord.  Revisiting multi-task learning with rock: a deep residual auxiliary block for visual detection.  In Adv. Neural Inform. Process. Syst., pages 1310–1322, 2018.
  * [44] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder.  The mapillary vistas dataset for semantic understanding of street scenes.  In Int. Conf. Comput. Vis., pages 4990–4999, 2017.
  * [45] Davy Neven, Bert De Brabandere, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool.  Fast scene understanding for autonomous driving.  arXiv preprint arXiv:1708.02550, 2017.
  * [46] Art B Owen.  A robust hybrid of lasso and ridge regression.  Contemporary Mathematics, 443(7):59–72, 2007.
  * [47] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch, 2017.
  * [48] Sujoy Paul, Yi-Hsuan Tsai, Samuel Schulter, Amit K Roy-Chowdhury, and Manmohan Chandraker.  Domain adaptive semantic segmentation using weak labels.  arXiv preprint arXiv:2007.15176, 2020.
  * [49] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  arXiv preprint arXiv:1511.06434, 2015.
  * [50] Pierluigi Zama Ramirez, Matteo Poggi, Fabio Tosi, Stefano Mattoccia, and Luigi Di Stefano.  Geometry meets semantics for semi-supervised monocular depth estimation.  In ACCV, pages 298–313. Springer, 2018.
  * [51] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez.  The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 3234–3243, 2016\.
  * [52] Swami Sankaranarayanan, Yogesh Balaji, Arpit Jain, Ser Nam Lim, and Rama Chellappa.  Unsupervised domain adaptation for semantic segmentation with gans.  arXiv preprint arXiv:1711.06969, 2:2, 2017.
  * [53] Jamie Shotton, Matthew Johnson, and Roberto Cipolla.  Semantic texton forests for image categorization and segmentation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 1–8. IEEE, 2008\.
  * [54] Trevor Standley, Amir R Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese.  Which tasks should be learned together in multi-task learning?  International Conference on Machine Learning (ICML), 2019.
  * [55] Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker.  Learning to adapt structured output space for semantic segmentation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 7472–7481, 2018\.
  * [56] Yi-Hsuan Tsai, Kihyuk Sohn, Samuel Schulter, and Manmohan Chandraker.  Domain adaptation for structured output via discriminative patch representations.  In Int. Conf. Comput. Vis., pages 1456–1465, 2019.
  * [57] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of Machine Learning Research, 9(86):2579–2605, 2008.
  * [58] Simon Vandenhende, Stamatios Georgoulis, Bert De Brabandere, and Luc Van Gool.  Branched multi-task networks: deciding what layers to share.  arXiv preprint arXiv:1904.02920, 2019.
  * [59] Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, Dengxin Dai, and Luc Van Gool.  Revisiting multi-task learning in the deep learning era.  arXiv preprint arXiv:2004.13379, 2020.
  * [60] Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool.  Mti-net: Multi-scale task interaction networks for multi-task learning.  Eur. Conf. Comput. Vis., 2020.
  * [61] Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez.  Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 2517–2526, 2019\.
  * [62] Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez.  Dada: Depth-aware domain adaptation in semantic segmentation.  In Int. Conf. Comput. Vis., pages 7364–7373, 2019.
  * [63] Dan Xu, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe.  Pad-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 675–684, 2018\.
  * [64] Dan Xu, Elisa Ricci, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe.  Multi-scale continuous crfs as sequential deep networks for monocular depth estimation.  In IEEE Conf. Comput. Vis. Pattern Recog., 2017.
  * [65] Jinyu Yang, Weizhi An, Sheng Wang, Xinliang Zhu, Chaochao Yan, and Junzhou Huang.  Label-driven reconstruction for domain adaptation in semantic segmentation.  arXiv preprint arXiv:2003.04614, 2020.
  * [66] Jinyu Yang, Weizhi An, Chaochao Yan, Peilin Zhao, and Junzhou Huang.  Context-aware domain adaptation in semantic segmentation.  arXiv preprint arXiv:2003.04010, 2020.
  * [67] Fisher Yu and Vladlen Koltun.  Multi-scale context aggregation by dilated convolutions.  In Int. Conf. Learn. Represent., 2016.
  * [68] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese.  Taskonomy: Disentangling task transfer learning.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 3712–3722, 2018\.
  * [69] Yang Zhang, Philip David, and Boqing Gong.  Curriculum domain adaptation for semantic segmentation of urban scenes.  In Int. Conf. Comput. Vis., 2017.
  * [70] Zhenyu Zhang, Zhen Cui, Chunyan Xu, Zequn Jie, Xiang Li, and Jian Yang.  Joint task-recursive learning for semantic segmentation and depth estimation.  In Eur. Conf. Comput. Vis., pages 235–251, 2018.
  * [71] Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, and Jian Yang.  Pattern-affinitive propagation across depth, surface normal and semantic segmentation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 4106–4115, 2019\.
  * [72] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 2881–2890, 2017\.
  * [73] Qianyu Zhou, Zhengyang Feng, Guangliang Cheng, Xin Tan, Jianping Shi, and Lizhuang Ma.  Uncertainty-aware consistency regularization for cross-domain semantic segmentation.  arXiv preprint arXiv:2004.08878, 2020.
  * [74] Yang Zou, Zhiding Yu, BVK Vijaya Kumar, and Jinsong Wang.  Unsupervised domain adaptation for semantic segmentation via class-balanced self-training.  In Eur. Conf. Comput. Vis., pages 289–305, 2018.
  * [75] Laurent Zwald and Sophie Lambert-Lacroix.  The berhu penalty and the grouped effect.  arXiv preprint arXiv:1207.6868, 2012.


[meshStyle] (P1014) – (P403) – (P1066) – cycle;
[meshStyle] (P37) – (P1014) – (P1066) – cycle;
[meshStyle] (P1014) – (P257) – (P1171) – cycle;
[meshStyle] (P403) – (P1014) – (P1171) – cycle;
[meshStyle] (P339) – (P257) – (P383) – cycle;
[meshStyle] (P16) – (P339) – (P383) – cycle;
[meshStyle] (P339) – (P109) – (P413) – cycle;
[meshStyle] (P257) – (P339) – (P413) – cycle;
[meshStyle] (P574) – (P120) – (P656) – cycle;
[meshStyle] (P109) – (P574) – (P656) – cycle;
[meshStyle] (P574) – (P7) – (P601) – cycle;
[meshStyle] (P120) – (P574) – (P601) – cycle;
[meshStyle] (P403) – (P120) – (P743) – cycle;
[meshStyle] (P743) – (P37) – (P1066) – cycle;
[meshStyle] (P403) – (P743) – (P1066) – cycle;
[meshStyle] (P403) – (P109) – (P656) – cycle;
[meshStyle] (P120) – (P403) – (P656) – cycle;
[meshStyle] (P1133) – (P263) – (P1139) – cycle;
[meshStyle] (P244) – (P1133) – (P1139) – cycle;
[meshStyle] (P98) – (P263) – (P1133) – cycle;
[meshStyle] (P972) – (P263) – (P1051) – cycle;
[meshStyle] (P37) – (P972) – (P1051) – cycle;
[meshStyle] (P972) – (P244) – (P1139) – cycle;
[meshStyle] (P263) – (P972) – (P1139) – cycle;
[meshStyle] (P971) – (P252) – (P1145) – cycle;
[meshStyle] (P244) – (P971) – (P1145) – cycle;
[meshStyle] (P971) – (P13) – (P993) – cycle;
[meshStyle] (P252) – (P971) – (P993) – cycle;
[meshStyle] (P252) – (P98) – (P1133) – cycle;
[meshStyle] (P1133) – (P244) – (P1145) – cycle;
[meshStyle] (P252) – (P1133) – (P1145) – cycle;
[meshStyle] (P372) – (P257) – (P415) – cycle;
[meshStyle] (P98) – (P372) – (P415) – cycle;
[meshStyle] (P372) – (P16) – (P383) – cycle;
[meshStyle] (P257) – (P372) – (P383) – cycle;
[meshStyle] (P1014) – (P263) – (P1123) – cycle;
[meshStyle] (P257) – (P1014) – (P1123) – cycle;
[meshStyle] (P1014) – (P37) – (P1051) – cycle;
[meshStyle] (P263) – (P1014) – (P1051) – cycle;
[meshStyle] (P263) – (P98) – (P415) – cycle;
[meshStyle] (P415) – (P257) – (P1123) – cycle;
[meshStyle] (P263) – (P415) – (P1123) – cycle;
[meshStyle] (P1154) – (P324) – (P1158) – cycle;
[meshStyle] (P315) – (P1154) – (P1158) – cycle;
[meshStyle] (P1154) – (P99) – (P1052) – cycle;
[meshStyle] (P324) – (P1154) – (P1052) – cycle;
[meshStyle] (P1004) – (P324) – (P1053) – cycle;
[meshStyle] (P41) – (P1004) – (P1053) – cycle;
[meshStyle] (P1004) – (P315) – (P1158) – cycle;
[meshStyle] (P324) – (P1004) – (P1158) – cycle;
[meshStyle] (P1005) – (P312) – (P1162) – cycle;
[meshStyle] (P315) – (P1005) – (P1162) – cycle;
[meshStyle] (P1005) – (P17) – (P994) – cycle;
[meshStyle] (P312) – (P1005) – (P994) – cycle;
[meshStyle] (P1154) – (P312) – (P995) – cycle;
[meshStyle] (P99) – (P1154) – (P995) – cycle;
[meshStyle] (P1154) – (P315) – (P1162) – cycle;
[meshStyle] (P312) – (P1154) – (P1162) – cycle;
[meshStyle] (P607) – (P208) – (P688) – cycle;
[meshStyle] (P688) – (P99) – (P759) – cycle;
[meshStyle] (P607) – (P688) – (P759) – cycle;
[meshStyle] (P604) – (P607) – (P760) – cycle;
[meshStyle] (P8) – (P604) – (P760) – cycle;
[meshStyle] (P208) – (P607) – (P604) – cycle;
[meshStyle] (P324) – (P208) – (P459) – cycle;
[meshStyle] (P459) – (P41) – (P1053) – cycle;
[meshStyle] (P324) – (P459) – (P1053) – cycle;
[meshStyle] (P688) – (P324) – (P1052) – cycle;
[meshStyle] (P99) – (P688) – (P1052) – cycle;
[meshStyle] (P208) – (P324) – (P688) – cycle;
[meshStyle] (P328) – (P122) – (P659) – cycle;
[meshStyle] (P110) – (P328) – (P659) – cycle;
[meshStyle] (P753) – (P328) – (P1072) – cycle;
[meshStyle] (P41) – (P753) – (P1072) – cycle;
[meshStyle] (P122) – (P328) – (P753) – cycle;
[meshStyle] (P575) – (P122) – (P605) – cycle;
[meshStyle] (P7) – (P575) – (P605) – cycle;
[meshStyle] (P575) – (P110) – (P659) – cycle;
[meshStyle] (P122) – (P575) – (P659) – cycle;
[meshStyle] (P289) – (P315) – (P1155) – cycle;
[meshStyle] (P110) – (P289) – (P1155) – cycle;
[meshStyle] (P289) – (P17) – (P1005) – cycle;
[meshStyle] (P315) – (P289) – (P1005) – cycle;
[meshStyle] (P1004) – (P328) – (P1163) – cycle;
[meshStyle] (P315) – (P1004) – (P1163) – cycle;
[meshStyle] (P1004) – (P41) – (P1072) – cycle;
[meshStyle] (P328) – (P1004) – (P1072) – cycle;
[meshStyle] (P328) – (P110) – (P1155) – cycle;
[meshStyle] (P1155) – (P315) – (P1163) – cycle;
[meshStyle] (P328) – (P1155) – (P1163) – cycle;
[meshStyle] (P645) – (P103) – (P647) – cycle;
[meshStyle] (P83) – (P645) – (P647) – cycle;
[meshStyle] (P645) – (P19) – (P618) – cycle;
[meshStyle] (P103) – (P645) – (P618) – cycle;
[meshStyle] (P576) – (P103) – (P1015) – cycle;
[meshStyle] (P17) – (P576) – (P1015) – cycle;
[meshStyle] (P576) – (P83) – (P647) – cycle;
[meshStyle] (P103) – (P576) – (P647) – cycle;
[meshStyle] (P577) – (P138) – (P674) – cycle;
[meshStyle] (P83) – (P577) – (P674) – cycle;
[meshStyle] (P577) – (P1) – (P579) – cycle;
[meshStyle] (P138) – (P577) – (P579) – cycle;
[meshStyle] (P645) – (P138) – (P578) – cycle;
[meshStyle] (P19) – (P645) – (P578) – cycle;
[meshStyle] (P645) – (P83) – (P674) – cycle;
[meshStyle] (P138) – (P645) – (P674) – cycle;
[meshStyle] (P801) – (P651) – (P923) – cycle;
[meshStyle] (P84) – (P801) – (P923) – cycle;
[meshStyle] (P801) – (P99) – (P796) – cycle;
[meshStyle] (P651) – (P801) – (P796) – cycle;
[meshStyle] (P710) – (P651) – (P797) – cycle;
[meshStyle] (P19) – (P710) – (P797) – cycle;
[meshStyle] (P710) – (P84) – (P923) – cycle;
[meshStyle] (P651) – (P710) – (P923) – cycle;
[meshStyle] (P709) – (P607) – (P924) – cycle;
[meshStyle] (P84) – (P709) – (P924) – cycle;
[meshStyle] (P709) – (P8) – (P760) – cycle;
[meshStyle] (P607) – (P709) – (P760) – cycle;
[meshStyle] (P801) – (P607) – (P759) – cycle;
[meshStyle] (P99) – (P801) – (P759) – cycle;
[meshStyle] (P801) – (P84) – (P924) – cycle;
[meshStyle] (P607) – (P801) – (P924) – cycle;
[meshStyle] (P312) – (P103) – (P806) – cycle;
[meshStyle] (P806) – (P99) – (P995) – cycle;
[meshStyle] (P312) – (P806) – (P995) – cycle;
[meshStyle] (P1015) – (P312) – (P994) – cycle;
[meshStyle] (P17) – (P1015) – (P994) – cycle;
[meshStyle] (P103) – (P312) – (P1015) – cycle;
[meshStyle] (P618) – (P651) – (P934) – cycle;
[meshStyle] (P103) – (P618) – (P934) – cycle;
[meshStyle] (P618) – (P19) – (P797) – cycle;
[meshStyle] (P651) – (P618) – (P797) – cycle;
[meshStyle] (P806) – (P651) – (P796) – cycle;
[meshStyle] (P99) – (P806) – (P796) – cycle;
[meshStyle] (P806) – (P103) – (P934) – cycle;
[meshStyle] (P651) – (P806) – (P934) – cycle;
[meshStyle] (P214) – (P213) – (P552) – cycle;
[meshStyle] (P125) – (P214) – (P552) – cycle;
[meshStyle] (P1006) – (P214) – (P1063) – cycle;
[meshStyle] (P18) – (P1006) – (P1063) – cycle;
[meshStyle] (P213) – (P214) – (P1006) – cycle;
[meshStyle] (P465) – (P213) – (P476) – cycle;
[meshStyle] (P33) – (P465) – (P476) – cycle;
[meshStyle] (P552) – (P465) – (P996) – cycle;
[meshStyle] (P125) – (P552) – (P996) – cycle;
[meshStyle] (P213) – (P465) – (P552) – cycle;
[meshStyle] (P1116) – (P211) – (P1119) – cycle;
[meshStyle] (P195) – (P1116) – (P1119) – cycle;
[meshStyle] (P125) – (P211) – (P1116) – cycle;
[meshStyle] (P937) – (P211) – (P997) – cycle;
[meshStyle] (P8) – (P937) – (P997) – cycle;
[meshStyle] (P937) – (P195) – (P1119) – cycle;
[meshStyle] (P211) – (P937) – (P1119) – cycle;
[meshStyle] (P936) – (P214) – (P1121) – cycle;
[meshStyle] (P195) – (P936) – (P1121) – cycle;
[meshStyle] (P936) – (P18) – (P1063) – cycle;
[meshStyle] (P214) – (P936) – (P1063) – cycle;
[meshStyle] (P214) – (P125) – (P1116) – cycle;
[meshStyle] (P1116) – (P195) – (P1121) – cycle;
[meshStyle] (P214) – (P1116) – (P1121) – cycle;
[meshStyle] (P223) – (P163) – (P407) – cycle;
[meshStyle] (P173) – (P223) – (P407) – cycle;
[meshStyle] (P935) – (P223) – (P1106) – cycle;
[meshStyle] (P18) – (P935) – (P1106) – cycle;
[meshStyle] (P163) – (P223) – (P935) – cycle;
[meshStyle] (P360) – (P163) – (P340) – cycle;
[meshStyle] (P5) – (P360) – (P340) – cycle;
[meshStyle] (P360) – (P173) – (P407) – cycle;
[meshStyle] (P163) – (P360) – (P407) – cycle;
[meshStyle] (P202) – (P213) – (P1117) – cycle;
[meshStyle] (P173) – (P202) – (P1117) – cycle;
[meshStyle] (P202) – (P33) – (P476) – cycle;
[meshStyle] (P213) – (P202) – (P476) – cycle;
[meshStyle] (P1006) – (P223) – (P1122) – cycle;
[meshStyle] (P213) – (P1006) – (P1122) – cycle;
[meshStyle] (P1006) – (P18) – (P1106) – cycle;
[meshStyle] (P223) – (P1006) – (P1106) – cycle;
[meshStyle] (P223) – (P173) – (P1117) – cycle;
[meshStyle] (P1117) – (P213) – (P1122) – cycle;
[meshStyle] (P223) – (P1117) – (P1122) – cycle;
[meshStyle] (P826) – (P664) – (P925) – cycle;
[meshStyle] (P129) – (P826) – (P925) – cycle;
[meshStyle] (P826) – (P118) – (P823) – cycle;
[meshStyle] (P664) – (P826) – (P823) – cycle;
[meshStyle] (P764) – (P664) – (P824) – cycle;
[meshStyle] (P41) – (P764) – (P824) – cycle;
[meshStyle] (P764) – (P129) – (P925) – cycle;
[meshStyle] (P664) – (P764) – (P925) – cycle;
[meshStyle] (P765) – (P597) – (P926) – cycle;
[meshStyle] (P129) – (P765) – (P926) – cycle;
[meshStyle] (P765) – (P33) – (P737) – cycle;
[meshStyle] (P597) – (P765) – (P737) – cycle;
[meshStyle] (P826) – (P597) – (P736) – cycle;
[meshStyle] (P118) – (P826) – (P736) – cycle;
[meshStyle] (P826) – (P129) – (P926) – cycle;
[meshStyle] (P597) – (P826) – (P926) – cycle;
[meshStyle] (P596) – (P122) – (P671) – cycle;
[meshStyle] (P118) – (P596) – (P671) – cycle;
[meshStyle] (P596) – (P7) – (P605) – cycle;
[meshStyle] (P122) – (P596) – (P605) – cycle;
[meshStyle] (P753) – (P664) – (P929) – cycle;
[meshStyle] (P122) – (P753) – (P929) – cycle;
[meshStyle] (P753) – (P41) – (P824) – cycle;
[meshStyle] (P664) – (P753) – (P824) – cycle;
[meshStyle] (P671) – (P664) – (P823) – cycle;
[meshStyle] (P118) – (P671) – (P823) – cycle;
[meshStyle] (P671) – (P122) – (P929) – cycle;
[meshStyle] (P664) – (P671) – (P929) – cycle;
[meshStyle] (P524) – (P208) – (P551) – cycle;
[meshStyle] (P551) – (P125) – (P1067) – cycle;
[meshStyle] (P524) – (P551) – (P1067) – cycle;
[meshStyle] (P524) – (P41) – (P459) – cycle;
[meshStyle] (P208) – (P524) – (P459) – cycle;
[meshStyle] (P211) – (P208) – (P604) – cycle;
[meshStyle] (P604) – (P8) – (P997) – cycle;
[meshStyle] (P211) – (P604) – (P997) – cycle;
[meshStyle] (P211) – (P125) – (P551) – cycle;
[meshStyle] (P208) – (P211) – (P551) – cycle;
[meshStyle] (P1069) – (P465) – (P1203) – cycle;
[meshStyle] (P129) – (P1069) – (P1203) – cycle;
[meshStyle] (P1069) – (P125) – (P996) – cycle;
[meshStyle] (P465) – (P1069) – (P996) – cycle;
[meshStyle] (P465) – (P33) – (P765) – cycle;
[meshStyle] (P765) – (P129) – (P1203) – cycle;
[meshStyle] (P465) – (P765) – (P1203) – cycle;
[meshStyle] (P764) – (P524) – (P1204) – cycle;
[meshStyle] (P129) – (P764) – (P1204) – cycle;
[meshStyle] (P41) – (P524) – (P764) – cycle;
[meshStyle] (P1069) – (P524) – (P1067) – cycle;
[meshStyle] (P125) – (P1069) – (P1067) – cycle;
[meshStyle] (P1069) – (P129) – (P1204) – cycle;
[meshStyle] (P524) – (P1069) – (P1204) – cycle;
[ultra thick, red, , line cap=round, line join=round]  (0,0) – (1, 0) – (1,1) – (0, 1) – (0,0);
at (0.87, 0.87) \(K\);

[fill=TUblue!30!white] (1, 0.5) – (1, 1) – (0.5, 1) – cycle;