Method | Cifar10 | Cifar100 | Tiny ImageNet | SOMe ImageNet
---|---|---|---|---
fs(50) | linear | nonlinear | fs(50) | linear | nonlinear | fs(50) | linear | nonlinear | fs(50) | linear | nonlinear
D-CSNN-B1 | $\mathbf{41.59}$ | $73.73$ | $77.83$ | $27.29$ | $45.17$ | $49.80$ | $13.70$ | $13.52$ | $3.56$ | $\mathbf{68.92}$ | $70.69$ | $49.99$
D-CSNN-B1-Aug | - | $\mathbf{75.68}$ | $\mathbf{81.55}$ | - | $\mathbf{46.82}$ | $\mathbf{52.66}$ | - | $13.32$ | $\mathbf{5.20}$ | - | $\mathbf{70.80}$ | $\mathbf{75.29}$
D-CSNN-B1-Cifar10 | $\mathbf{41.59}$ | $73.73$ | $77.83$ | $\mathbf{27.96}$ | $45.68$ | $49.70$ | $13.46$ | $\mathbf{14.36}$ | $2.03$ | $47.65$ | $65.34$ | $65.59$
D-CSNN-B1-Cifar100 | $41.43$ | $71.60$ | $76.46$ | $27.29$ | $45.17$ | $49.80$ | $\mathbf{13.72}$ | $13.57$ | $1.61$ | $45.68$ | $63.79$ | $65.13$

#####  3.4.8 Generalization

To test generalizability, we train models on one dataset and retrain the
classifiers for each dataset to which we transfer the models. Following that,
we report the test set performance of the dataset for which we transferred the
models. As shown in Table 3.3, our models achieve surprising generalization
capability, where differences in accuracy between the model trained and the
model transferred to the dataset lie in the low percentage range. Furthermore,
we observe good performance on the larger $256\times 256$ SOMe ImageNet
images.

Figure 3.5: Neuron utilization and few-shot learning. (top) Neuron utilization
of models trained with various batch sizes and mask learning rules on Cifar10.
(bottom) 10-fold few-shot classifier accuracy of our models with increasing
shot sizes from $1$ to $50$ on Cifar10.

#####  3.4.9 Neuron Utilization

In the upper part of Figure 3.5, we observe that the neuron utilization — the
percentage of used neurons per input — is getting smaller for deeper layers,
indicating that the SOM neurons of deeper layers specialize to certain inputs.
Compared to learning rule 3.13, learning rule 3.12 leads to a higher neuron
utilization, which may be explained by the forgetting term of Oja’s rule,
which is not present in Hebb’s rule. We want to note that a higher neuron
utilization does not necessarily lead to higher performance since a lower
neuron utilization indicates sparser representations that may be more
distinguishable.

#####  3.4.10 Few-shot Performance

The lower part of Figure 3.5 shows the increase in accuracy of the linear
classifier as the number of representations per class used to train the linear
classifier increases. Our models reach an accuracy of 41.49% on Cifar10,
27.29% on Cifar100, 13.70% on Tiny ImageNet, and 68.92% on SOMe ImageNet when
trained on 50 examples per class. The comparatively higher accuracy in SOMe
ImageNet indicates that the model can utilize the additional patches obtained
from the larger input images.

Figure 3.6: BMU images for each layer of the D-CSNN model during training.
Images have been created by replacing each patch with the SOM weight of the
BMU, which is reshaped into the 3D patch shape (RGB image). For deeper layers,
we only show a slice of depth three through the BMU weight, since its hard to
visualize kernel depths larger then three. Since the D-CSNN-B1 model contains
three maps, we use the best BMU from all maps of the layer. Best viewed in
color. Figure 3.7: Representations and reconstructions. a) The Cifar10 input
image. b) The reconstruction of the input image using the representation of
the D-CSNN-B1 model. c) The average representation per map for each class
created by taking the mean over the spatial dimension of each test sample’s
representation and reshaping the resulting vector to SOM grid shape. d) RGB
image of the average representation (three maps lead to an RGB image). e)
Differences of the average representations calculated by element-wise
subtracting the average representation of the respective class from the
average representation of the class on the right-hand side. Best viewed in
color.

#####  3.4.11 Visual Interpretations

In Figure 3.6 we show BMU images for each layer during the training of the
D-CSNN model. These images are created by replacing the image patch with the
reshaped BMU weights or a slice of these weights. For earlier training steps,
BMUs that have been calculated from randomly initialized neurons can be seen
for layers 2 - 4. After training the individual layers, patterns emerge. For
example, for layer 1, we observe patterns corresponding to the content of
specific regions of the input image. These patterns become less interpretable
at deeper layers. Moreover, we observe that the SOM weights are not only
sensitive to colors since different backgrounds result in similar BMU
patterns, for example, in image 5 of layer 2.

As shown in row b) of Figure 3.7, the input images can be reconstructed with a
high level of detail from the representation of our unsupervised CSNN models.
This is surprising and shows the amount of information present in the
representation of our unsupervised CSNN models. Moreover, in rows c), d), and
e), it can be observed that "similar" classes lead to similar average
representations: 1) In rows c) and d) exist similar spots with high average
neuron activities. 2) The classes have been sorted according to $L_{1}$
distance of their average representation from the average representation of
the truck class. The average car class representation has the shortest
distance to the average truck class representation, followed by the average
representations of "similar" looking animal classes. 3) Row e) shows images
where the average representation of the right-hand class has been subtracted
element-wise from the average representation of the respective class. For
example, the first column of e) equals the truck’s average representation
minus the car’s average representation. Therefore, row e) shows the remaining
distinguishable differences between average representations of the respective
class and the right-hand class. We observe that this remaining information is
lower when a "similar" class is on the right-hand side (e.g., dog minus cat).

Additional quantitative and qualitative results can be found in Appendix A.

####  3.5 Conclusion

In this chapter, we have introduced the modular building blocks of CSNNs to
learn representations in an unsupervised manner without backpropagation. With
the proposed CSNN modules and learning rules — which combine CNNs, SOMs and
Hebbian learning of masks — a new, alternative way for learning unsupervised
feature hierarchies has been explored. Along the way, we have discussed the
Scalar Bow Tie Problem and the objective function mismatch: two problems in
the field of deep learning that we believe can potentially be solved together
and provide an interesting direction for future research.

## Evaluating Visual
Unsupervised Repre-
sentation Learning

### Investigating the Objective Function Mismatch

####  4.1 Motivation

Unsupervised representation learning is a promising approach to learn useful
features from huge amounts of data without human annotation effort. Thereby a
common evaluation pattern is to train an unsupervised pretext model on
different datasets and then test its performance on several target tasks.
Because of the huge variety of target tasks and preferred representation
characteristics, the evaluation of these methods is challenging. In preceding
work, several evaluation metrics have been proposed [174, 112, 199, 177], but
due to the fast changes in unsupervised learning methodologies, only a few of
them can be used across the wide spectrum of promising approaches. This is one
reason why the linear evaluation protocol is commonly used [85, 144, 291, 59,
206, 104, 42], which trains a linear model for a target task on-top of the
representations of an unsupervised pretext model. In this chapter, we show
that simply training a target model for different layers of the pretext model
does not yield the entire picture of the training process and leads to a loss
of useful temporal information about learning. It is already known in
literature that succeeding in a pretext task can be the reason why the model
fails on the target task. Here we propose that the linear evaluation protocol
does not capture this properly.

####  4.2 Contributions

Figure 4.1: The objective function mismatch. (bottom left) Evaluation loss of
a pretext autoencoder trained on Cifar10. (top left) Best evaluation losses of
linear target models trained for classification on the representations of the
pretext autoencoder from different pretext training epochs. (right) Evaluation
loss curves from each linear target model. Colors correspond to the current
epoch of pretext task training, and each value is obtained by 5-fold cross-
validation. An objective function mismatch occurs around pretext training
epoch 40, from which the target loss increases. Best viewed in color.

We extend the linear and nonlinear evaluation protocols and address the
question of when succeeding in a pretext task hurts performance and how much.
We train target models on representations obtained from different training
steps or epochs of the pretext model and plot target and pretext model metrics
in comparison, as shown in Figure 4.1. Thereby we observe that training an
unsupervised pretext model too long can lead to an objective function mismatch
[185, 238] between the objectives used to train both models. This mismatch
leads to a drop in performance on the target task, while the pretext model and
the target models still converge correctly, which can be seen in Figure 4.1.
To quantify our results, we define soft and hard versions for two simple and
general evaluation metrics - the metrics mismatch and the objective function
mismatch \- formally. With these metrics, we then evaluate different image-
based pretext task types for self-supervised learning by using the linear
evaluation protocol.

Our contributions can be summarized as follows:

  * •

We propose hard and soft versions of general metrics to measure and compare
mismatches of (unsupervised) representation learning methods across different
target tasks (Sections 4.3 and 4.4). To the best of our knowledge, this has
not been done before. (RQ-E2)

  * •

We discuss the usability and stability of our protocols on a variety of
pretext and target tasks (Section 4.6.2). (RQ-E2)

  * •

In our experiments, we qualitatively show dependencies of the objective
function mismatch with respect to the pretext model’s representation size
(Section 4.6.3), target model complexity (Section 4.6.4), pretext and target
augmentations (Section 4.6.5), as well as pretext and target task types
(Section 4.6.6). (RQ-E3)

  * •

We find that the objective function mismatch can reduce performance on various
benchmarks. Specifically, we observe a performance decrease of $\sim$0.1-5.0%
for Cifar10, Cifar100, and PCam, and up to $\sim$25-59% in extreme cases for
the 3dshapes dataset (Section 4.6). (RQ-E4)

####  4.3 Hard Metrics Mismatch

With the objective function mismatch, we want to measure the mismatch of two
objectives while training a model on a (unsupervised) pretext task and using
its representations to train another model on a target task. In general we can
measure the mismatch of two comparable metrics, if one metric is captured
during training of a single pretext model and the other is captured for each
target model fully trained on the representations of different steps or epochs
of the pretext model. Two comparable metrics, for example, are classification
accuracies for the pretext and target task because they use the same
measurement unit and scale. As illustrated in Figure 4.2, the metric values of
the target models form a curve over the course of learning. Between a metric
value on this curve and the corresponding metric value on the pretext model
curve we can define the metrics mismatch ($\mathrm{M3}$) for a certain step
(or epoch) in training by calculating their distance.

More formally, let $M^{P}=(m_{1}^{P},...,m_{n}^{P})$ denote an $n$-tuple of
values from a metric used to measure pretext model $P$ for different steps
$S=(s_{1},...,s_{n})$. The length $n$ of the tuple is usually given by a
convergence criterion $C$ on the metric of model $P$ during training.
Furthermore, let $M^{T}=(m_{1}^{T},...,m_{n}^{T})$ denote an $n$-tuple of
values from a comparable metric used to measure target model $T$. $M^{T}$ is
of the same length and order as $M^{P}$ and all values are calculated at the
same training steps $S$ of $M^{P}$. Thereby the target model $T$ is fully
retrained for every step $s_{i}$ in $S$ on the representations of model $P$ at
this step before we measure $m_{i}^{T}$.

Figure 4.2: $\mathrm{M3}$ and $\mathrm{MM3}$. (left) The intuition behind
$\mathrm{MM3}$: In this case, both metrics measure a classification error in
%. The pretext metric (solid curve) is measured on the pretext task of
predicting rotations with a ResNet18 model, and the target metric (dotted
curve) on the fully trained Cifar10 classification task. When divided by the
number of measurements, the discrete area enclosed by the target and pretext
task curves corresponds to the $\mathrm{MM3}$ of the entire training process.
(right) The Metrics Mismatch $\mathrm{M3}$ plotted during training: We observe
a common behavior where the mismatch increases as training progresses.
Additionally, we show the stability ($+,-$) of $\mathrm{M3}$, and
$\mathrm{MM3}$ across a 5-fold cross-validation.

###### Definition 1

The hard Metrics MisMatch ($\mathrm{M3}$) between $m_{i}^{T}$ and $m_{i}^{P}$
at step $s_{i}$ is defined as:

$\mathrm{M3}(m_{i}^{T},m_{i}^{P})\coloneqq m_{i}^{T}-m_{i}^{P}$ (4.1)

where $m_{i}^{T}$ and $m_{i}^{P}$ are single values measured with comparable
metrics at step $s_{i}$.111Note that we define our measurements only for the
case where lower metric values correspond to better performance. The
definition for the opposite case arises naturally by changing maximum and
minimum operations and/or subtraction orders.

If $\mathrm{M3}>0$, the performance of the target model is lower then the
performance of the pretext model at step $s_{i}$. In contrast,
$\mathrm{M3}\leq 0$ represents the desired case in unsupervised representation
learning, where the target model performance is the same or above the pretext
model performance at step $s_{i}$. In our case, we measure $m_{i}^{T}$ and
$m_{i}^{P}$ over the entire evaluation dataset for every step $s_{i}$ in $S$.
We plot $\mathrm{M3}(m_{i}^{T},m_{i}^{P})$ for the pretext task of predicting
rotations and the target task of Cifar10 classification during training in
Figure 4.2. This shows that our metric captures the behavior of the target
task performance regarding the pretext task performance, and we observe an
increasing mismatch as training progresses. To capture the mismatch of the
entire training procedure with respect to the target task in a single value,
we can now define the mean hard metrics mismatch ($\mathrm{MM3}$) as the mean
bias error between $M^{T}$ and $M^{P}$.

###### Definition 2

The Mean hard Metrics MisMatch ($\mathrm{MM3}$) between $M^{T}$ and $M^{P}$ is
defined as:

$\mathrm{MM3}({M}^{T},M^{P})\coloneqq\frac{1}{n}\sum\limits_{0<i\leq
n}(m_{i}^{T}-m_{i}^{P})$ (4.2)

where $M^{T}$ and $M^{P}$ are tuples measured with comparable metrics until
the pretext model converges at step $s_{n}$.

$\mathrm{MM3}$ measures the bias of the target model metric to the pretext
model metric. For positive or negative values of $\mathrm{MM3}$, we can make
similar observations as for $\mathrm{M3}$, but they now account for the
tendency of the entire training process and not for a single step $s_{i}$. In
general, the mean bias error can convey useful information, but it should be
interpreted cautiously because there are special cases where positive and
negative values cancel each other out. In our case, this can happen, for
example, when learning the pretext task is very useful for the target task
early in training but hurts the target performance equally strong later on
when the pretext task is sufficiently solved. We capture this behavior simply
by measuring and plotting $\mathrm{M3}$ individually for the metric values of
each step, as in Figure 4.2, analogous to the way a loss is measured and
plotted during training.

#####  4.3.1 Hard Objective Function Mismatch

Naively we could compare the objective functions of the target and pretext
task by using $\mathrm{M3}$, which we define as the hard objective function
mismatch. In most cases, however, the objective functions used to train the
pretext model and the target models are not directly comparable. This is due
to the usage of different objective functions for both model types, which,
i.a., use different (non-)linearities. But for some pretext tasks simple
comparable metrics can be defined. These metrics can be used as a proxy to
measure the objective function mismatch in a general and comparable manner. A
well-known example is the accuracy metric, which can be used on the self-
supervised tasks of predicting rotations [85] and the state-of-the-art
approach of contrastive learning [42]. But comparable metric pairs can not
always be found easily. For example, if we train a variational autoencoder and
later use its representation for a classification target task, it does not
make sense to define a pixel-wise error between the given and generated images
as a comparable pretext task metric. To achieve a comparable measurement for
this situation and on the loss curves in general, one could think of
individual normalization techniques between objective function pairs. However,
we want to be practical and define a measure that can be used independently of
the objective function pairs for every pretext and target model combination.
Furthermore, in practice we might be especially interested in how much the
target task mismatches with the pretext task if a mismatch decreases target
performance. This is why we define soft versions of our measurements.

Figure 4.3: $\mathrm{OFM}$, $\mathrm{cFM}$, $\mathrm{mOFM}$, and
$\mathrm{MOFM}$. We measure the $\mathrm{OFM}$ instead of $\mathrm{SM3}$ by
normalizing the metric values with Equation 4.11. For visualization, we
additionally shift the normalized metric values such that they lie in
$[0,100]$ by subtracting the minimal measurement. (left) Intuition for the
$\mathrm{MOFM}$: When divided by the number of measurements, the discrete area
enclosed by the target metric values and their previous minimal target metric
values correspond to the $\mathrm{OFM}$ of the entire training process. The
red arrow shows the $\mathrm{cOFM}$, and the black arrow the $\mathrm{mOFM}$.
In this case, the target metric measures the cross-entropy loss of each fully
trained target model on a Cifar10 validation set. (right) The $\mathrm{OFM}$
plotted during training: We observe an increasing mismatch starting around
epoch 600. Additionally, we show the range ($+,-$) of the $\mathrm{OFM}$ and
$\mathrm{MOFM}$ across a 5-fold cross-validation.

####  4.4 Soft Metrics Mismatch

To bypass objective function pair normalization we define the soft metrics
mismatch ($\mathrm{SM3}$) directly on the target metric. Thereby we no longer
take the exact improvement of the pretext metric into account, we only care
about its convergence. Since we now have no exact information about the
pretext metric curve, we define $\mathrm{SM3}$ for the current step $s_{i}$
between the current target metric value and the previously or currently
occurred minimal target metric value:

###### Definition 3

The Soft Metrics Mismatch ($\mathrm{SM3}$) between $M^{T}$ and $M^{P}$ at step
$s_{i}$ is defined as:

$\mathrm{SM3}(m_{i}^{T})\coloneqq m_{i}^{T}-\min_{0<j\leq i}(m_{j}^{T})$ (4.3)

where $\min_{0<j\leq i}(m_{j}^{T})$ is the previously or currently occurred
minimal target metric value.

$\mathrm{SM3}$ has a slightly different meaning compared to $\mathrm{M3}$: It
equals zero if $m_{i}^{T}$ is a minimal metric value and is positive if
$m_{i}^{T}$ is higher than the previously occurred minimal metric value. We
want to point out that the only way we incorporate the pretext metrics into
this measurement is by making sure that the pretext model does not overfit and
has not yet converged. Again, we measure $m_{i}^{T}$ and $m_{i}^{P}$ over the
entire evaluation dataset for every step $s_{i}$ in $S$ and plot
$\mathrm{SM3}(m_{i}^{T})$. A common case is shown in Figure 4.3, which
captures the behavior of pretext model training with respect to the target
model. Here we observe zero soft mismatch early in training followed by
increasing soft mismatch until pretext model convergence. Again, we can
capture the mismatch of the pretext task with respect to the target task for
the entire training process until pretext model convergence as the mean bias
error of every metric value $m_{i}^{T}$ and its minimal metric value:

###### Definition 4

The Mean Soft Metrics Mismatch ($\mathrm{MSM3}$) between $M^{T}$ and $M^{P}$
is defined as:

$\mathrm{MSM3}(M^{T})\coloneqq\frac{1}{n}\sum\limits_{0<i\leq
n}\left(m_{i}^{T}-\min_{0<j\leq i}(m_{j}^{T})\right)$ (4.4)

when the pretext model convergences at step $s_{n}$.

$\mathrm{MSM3}$ can either be zero if no mismatch occurs or positive if there
is a mismatch. Therefore, using $\mathrm{MSM3}$ brings the benefit that
positive values can not be canceled out by negative values. Furthermore, we
define the maximum occurring mismatch $\mathrm{mSM3}$ and the mismatch at
pretext model convergence $\mathrm{cSM3}$. We are especially interested in
$\mathrm{cSM3}$ since it measures the representations one would naively take
for the target task:

$\displaystyle\mathrm{cSM3}(M^{T})\coloneqq\mathrm{SM3}(m_{n}^{T})$ (4.5)
$\displaystyle\mathrm{mSM3}(M^{T})\coloneqq\max_{0<i\leq
n}\left(\mathrm{SM3}(m_{i}^{T})\right)$ (4.6)

#####  4.4.1 Soft Objective Function Mismatch

Now we can use $\mathrm{SM3}$ to measure a soft form of the objective function
mismatch on the loss curve obtained by the target models. However, the values
of these measurements lie in a range that depends on the target objective
function. Therefore, they are not directly comparable to the measurements on
loss curves from other target tasks. This is why we normalize the measurements
of the target metric to the percentage range and define the objective
functions mismatch ($\mathrm{OFM}$) as follows:

###### Definition 5

The Soft Objective Function Mismatch  ($\mathrm{OFM}$) between $M^{T}$ and
$M^{P}$ at step $s_{i}$ is defined as:

$\displaystyle\mathrm{OFM}(m^{T}_{i})\coloneqq\mathrm{SM3}\left(\mathrm{N}(m^{T}_{i})\right)$
(4.7)
$\displaystyle\mathrm{N}(x)\coloneqq\left\\{\begin{array}[]{ll}\frac{100\times
x}{m_{1}^{T}-m_{b}^{T}}&\quad m_{1}^{T}>x\geq m_{b}^{T}\\\ 0&\quad
m_{1}^{T}=m_{b}^{T}=x\\\ \infty\,(special\,case)&\quad
m_{1}^{T}=m_{b}^{T}<x\\\ \end{array}\right.$ (4.11)

where $m_{1}^{T}$ is the loss value of the target model trained on an
untrained pretext model ($s_{1}=0$) and $b=\mathrm{argmin}_{0<i\leq
n}(m^{T}_{i})$ denotes the index of the minimal target loss value. We then use
$\hat{M}^{T}=(\mathrm{N}(m^{T}_{1}),...,\mathrm{N}(m^{T}_{n}))$ to calculate
the $\mathrm{OFM}$.

The intuition behind this normalization is that we declare $m_{b}^{T}$ as the
value where the pretext model has learned all of the target objective it was
able to learn (with this setting) and $m_{1}^{T}$ as the value where the model
has learned nothing of the target objective. Now we measure with
$\mathrm{OFM}(m^{T}_{i})$ for what percentage the learning of a pretext
objective hurts the maximum achieved target performance at step $s_{i}$. An
example is illustrated in Figure 4.3. Furthermore, we can normalize the other
soft measurements from Equation 4.5 and 4.6 analog to Equation 4.7.

The $\mathrm{OFM}$ is a general measure that can be used for pretext and
target models where no good proxy metrics can be defined. With the
$\mathrm{OFM}$, we are able to compare mismatches across different pretext and
target task objectives and their combinations. We propose these measurements
to obtain quantitative and therefore comparable results for individual pretext
tasks. To get the best information about the training process we encourage to
plot the curves formed by our metrics as well. We want to point out that our
metrics are not intended to measure target task performance, they measure how
much the performance on a target task can decrease when an (ill-posed) pretext
task is learned too long. Now, to understand the $\mathrm{OFM}$ further, we
take a look at some cases:

$\mathrm{OFM}(m^{T}_{i})=0$: In this case, solving the pretext task objective
has not hurt the performance of the target task objective at this point in
training.

$\mathrm{OFM}(m^{T}_{i})=x$: Solving the pretext objective has hurt the
performance of the target objective at this point in training by $x\%$ of what
the model has learned. Therefore, we should have stopped training earlier. It
is not guaranteed that longer training would hurt performance even more, but a
growing $\mathrm{OFM}$ curve or $\mathrm{MOFM}$ is a good indicator of that.

$\mathrm{OFM}(m^{T}_{i})>100$: The target objective performance is worse than
for the untrained model at this point in training.

$\mathrm{MOFM}(M^{T})=\infty$: Solving the pretext objective hurts the
performance of the target objective from the point of initialization. Because
we have learned essentially 0% about the target objective in the training
process, there is no interval to be used for normalization. Therefore, we
interpret this case as if the model has an infinite mismatch as soon as the
model forgets something about the target objective.

####  4.5 Experimental Setup

In our experiments, we focus on image-based self-supervised learning. However,
it is likely that other target domains show mismatches as well, e.g., [58].

Pretext Tasks. For generation-based self-supervision, we evaluate the
approaches of autoencoding data from autoencoders (CAE) and color restoration
(CCAE) as suggested in [42]. To evaluate context structure generation, we use
denoising autoencoders (DCAE). Transformation prediction (spatial context
structure) is evaluated via autoencoding tranformations by predicting
rotations [85] (RCAE). For contrastive methods (context-based similarity), we
evaluate SimCLR [42] (SCLCAE). We refer to the literature for first glances
into mismatches for VAEs [185], and meta-learning [185] and to chapter 3 for
self-organization [238].

Pretext Models. Unless stated otherwise, we use a four-layer CNN as the
encoder. For the autoencoding data approaches, we use a four-layer decoder
with transposed convolutions, for rotation prediction a single dense layer,
and for contrastive learning a nonlinear head, as suggested in [42]. We show
that mismatches account for other architectures as well by carrying out
additional evaluations using ResNets [105] in Table 4.3 and Appendix B.

Target Tasks. We evaluate our metrics on image-based target tasks. For coarse-
grained classification, we use Cifar10, Cifar100 [145], and the coarse-grained
labels of 3dshapes [29]. For fine-grained classification, we use the PCam
dataset [259] and the fine-grained labels of 3dshapes.

Target Models.: Following the linear evaluation protocol, we use a single,
linear dense layer (FC) as the target model with a softmax activation. To
evaluate our metrics for other target models, we use a two-layer MLP (2FC) and
a three-layer MLP (3FC).

Augmentations. We make sure not to compare augmentations instead of pretext
tasks by following [42] for our base augmentations, to which we add the
pretext task-specific augmentations for pretext task training and evaluation.
For the target task, we use the base training and evaluation augmentations of
[42].

Optimization. Our models are trained using the Adam optimizer [141] with
standard parameters and a batch size of $2048$ without any regularization
instead of batch normalization. For our ResNets, we additionally use a weight
decay of $1\mathrm{e}{-4}$.

Mismatch Evaluation. All reported values are determined by 5-fold cross-
validation. We use standard early stopping (from tf.keras) as a convergence
criterion on the pretext evaluation curve with a minimum delta (threshold) of
$0$ and patience of $3$. We change the patience in some experiments of Tables
4.1 and 4.3 to get a reasonable convergence epoch. For more details we refer
to Appendix B. When calculating our metrics, we estimate target values of
missing epochs with linear interpolation to save computation time. In our
case, $\mathrm{SM3}$ and $\mathrm{MM3}$ are measured on the target task
accuracy.

Implementation. Our implementation is available at
https://github.com/BonifazStuhr/OFM.

####  4.6 Evaluation

In the following, we show the results of most pretext and target tasks we have
evaluated. We refer to Appendix B for additional, more detailed evidence.
Since we capture our metrics during training, all mismatches are measured on
the evaluation dataset.

#####  4.6.1 Mismatch and Convergence

For our measurements we make sure to use metric value pairs from models that
do not overfit. We achieve this by applying a convergence criterion on the
pretext task and by using the best metric values from each target model
evaluation curve. As shown in Appendix B, most observations in our experiments
are independent of the use of a convergence criterion if pretext models are
trained long enough and without overfitting. Furthermore, we observe a common
behavior in Figure 4.1: Target models trained on higher epochs of the pretext
model tend to converge faster. This indicates that longer training of the
pretext task tends to create easier separable representations, which may
mismatch with the class label.

Figure 4.4: $\mathrm{OFM}$ ablation study. (top) Impact of different pretext
model representation sizes on the $\mathrm{OFM}$. (middle) $\mathrm{OFM}$ for
the linear target model and nonlinear target models trained on our pretext
model. (bottom) $\mathrm{OFM}$ for the linear target model and the pretext
models trained on fewer augmentations. First, we removed the color jitter and
then the vertical flip from the augmentations. The target models of SCLCAE
were trained on 3dshapes to predict the object hue. Best viewed in color.

Table 4.1: $\mathrm{MOFM}$, $\mathrm{cSM3}$ and $\mathrm{MM3}$ of the models
from Figure 4.4. $\mathrm{MM3}$ is measured on the target and pretext task
classification error. $\mathrm{cSM3}$ is measured on the target task
classification error and corresponds to the accuracy we would lose when we
naively train the target model after pretext model convergence. Values are
obtained by 5-fold cross-validation. We show the stability of each measurement
in Tables B.2, B.3, and B.4 of Appendix B.

|  CAE (Cifar10) |  DCAE (Cifar10) |  CCAE (Cifar100) |  CCAE (PCam) |  RCAE (PCam) |  SCLCAE (3dshapes)
---|---|---|---|---|---|---
| $\mathrm{cSM3}$ | $\mathrm{MOFM}$ | $\mathrm{cSM3}$ | $\mathrm{MOFM}$ | $\mathrm{cSM3}$ | $\mathrm{MOFM}$ | $\mathrm{cSM3}$ | $\mathrm{MOFM}$ | $\mathrm{cSM3}$ | $\mathrm{MOFM}$ | $\mathrm{MM3}$ | $\mathrm{cSM3}$ | $\mathrm{MOFM}$ | $\mathrm{MM3}$
Rep. Size: |  |  |  |  |  |  |  |  |  |  |  |  |  |
2x2x4 | 0.00 | 0.00 | 0.05 | 0.00 | 0.28 | 1.54 | 4.98 | 9.28 | 5.38 | 4.15 | -22.26 | 26.34 | $\infty$ | -7.99
2x2x32 | 0.07 | 1.99 | 0.00 | 3.20 | 0.65 | 3.64 | 5.17 | 34.30 | 3.34 | 7.92 | -21.09 | 12.98 | 36.39 | -57.67
2x2x128 | 0.20 | 10.10 | 0.06 | 5.51 | 0.51 | 0.81 | 0.32 | 0.10 | 1.03 | 4.04 | -23.47 | 8.14 | 22.65 | -66.19
2x2x256 | 0.75 | 11.14 | 0.69 | 5.17 | 0.17 | 0.00 | 0.43 | 0.87 | 0.44 | 0.00 | -27.60 | 6.52 | 27.65 | -65.77
2x2x512 | 0.43 | 5.28 | 0.36 | 1.25 | 0.00 | 0.00 | 0.20 | 0.07 | 0.18 | 0.00 | -28.03 | 5.96 | 27.78 | -63.61
2x2x1024 | 0.24 | 0.25 | 0.03 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.09 | 0.00 | -26.56 | 4.76 | 32.70 | -57.92
Target Model: |  |  |  |  |  |  |  |  |  |  |  |  |  |
FC | 0.75 | 11.14 | 0.69 | 5.17 | 0.00 | 0.00 | 0.00 | 0.00 | 1.03 | 4.04 | -23.47 | 6.52 | 27.65 | -65.77
2FC | 0.03 | 5.68 | 0.00 | 5.14 | 0.08 | 0.00 | 0.00 | 0.00 | 0.31 | 0.76 | -28.56 | 1.84 | 103.30 | -70.49
3FC | 0.03 | 3.94 | 0.00 | 3.17 | 0.12 | 0.02 | 0.08 | 0.00 | 0.37 | 0.61 | -29.61 | 0.91 | 258.18 | -71.26
Augmentations: |  |  |  |  |  |  |  |  |  |  |  |  |  |
All | 0.75 | 11.14 | 0.69 | 5.17 | 0.17 | 0.00 | 0.43 | 0.87 | 1.03 | 4.04 | -23.47 | 6.52 | 27.65 | -65.77
NoJitter | 0.99 | 10.99 | 0.50 | 2.33 | - | - | - | - | 0.58 | 0.10 | -28.60 | 0.00 | 0.01 | -37.92
NoJitterNoFlip | 1.00 | 12.51 | 0.55 | 1.73 | - | - | - | - | 1.51 | 7.33 | -10.60 | 0.00 | 0.00 | -35.95
NoFlip | - | - | - | - | 0.20 | 0.00 | 0.41 | 0.04 | - | - | - | - | - | -

#####  4.6.2 Stability

To evaluate the stability of our measurements, we show the mismatches of the
entire training process and their range ($+,-$) using 5-fold cross-validation
in Tables 4.2 and 4.3. The range of all other models we have trained is shown
in Appendix B. We observe that $\mathrm{M3}$ generally seems more stable than
$\mathrm{SM3}$ or the $\mathrm{OFM}$ since it does not rely so heavily on the
target metric values, which can be quite unstable. The instability of the
target task mismatch is captured in $\mathrm{M3}$ but does not matter much in
the overall measurement for most cases. This is favorable if a stable value is
desired and unfavorable if one wants to capture the instability of the target
task training process explicitly. Furthermore, $\mathrm{M3}$ is able to
compensate target fluctuations with pretext fluctuations. In general, we
observe that as long as we calculate the $\mathrm{OFM}$ across a fair amount
of cross-validations (in our case $5$), we can make statements about the
mismatch. We measure our metrics on the mean losses during 5-fold cross-
validation instead of calculating them five times and taking the average. For
$\mathrm{M3}$, both variants are equivalent, and for the $\mathrm{OFM}$
measuring on the mean losses leads to a lower bound in the case where all
models converge at step $s_{n}$ (see Appendix B for the simple proofs). We
prefer to measure our metrics on the mean losses since this avoids mismatches
occurring only in some validation cycles due to small fluctuations in the
underlying training procedure. An example is shown in translucent red in
Figure 4.3 at the beginning of training. We want to point out that the
training and validation data differ slightly in every training round because
of the cross-validation setup. This increases the instability but shows the
general behavior of the metrics for the underlying data distribution. In
Figures B.2 and B.1 of Appendix B, we compare the instability of partially
measured mismatches using linear interpolation with mismatches measured for
every pretext training epoch and observe a similar instability. However, when
using the $\mathrm{OFM}$ in practice to compare models on a finer scale, we
recommend searching for the actual minimal target metric value since the
$\mathrm{OFM}$ relies on this value at each step. When tuning a model for
maximum performance, one searches for this value. Thereby looking at the
$\mathrm{OFM}$ curve gives good indications of which interval one should
search. This makes this protocol useful for performance tuning if enough
computational power is available.

Figure 4.5: Pretext and target losses on the 3dshapes dataset. (bottom)
Pretext losses of our model trained for color restoration (CCAE), prediction
rotations (RCAE), and contrastive learning (SCLCAE). (top) Best target losses
of linear models trained for the different prediction tasks of 3dshapes. Best
viewed in color.

Table 4.2: $\mathrm{cSM3}$, $\mathrm{MOFM}$ and $\mathrm{MM3}$ on the 3dshapes
dataset. $\mathrm{SM3}$ and $\mathrm{MM3}$ are measured on the target task
accuracy. Values are obtained by 5-fold cross-validation. We show the
stability of each measurement in Tables B.5, B.6, and B.7 of Appendix B.

| CAE | DCAE | CCAE | RCAE | SCLCAE
---|---|---|---|---|---
| $\mathrm{cSM3}$ | $\mathrm{MOFM}$ | $\mathrm{cSM3}$ | $\mathrm{MOFM}$ | $\mathrm{cSM3}$ | $\mathrm{MOFM}$ | $\mathrm{cSM3}$ | $\mathrm{MOFM}$ | $\mathrm{MM3}$ | $\mathrm{cSM3}$ | $\mathrm{MOFM}$ | $\mathrm{MM3}$
floor_hue | 0.01 | 0.95 | 0.00 | 1.28 | 0.02 | 0.00 | 56.68 | $\infty$ | 44.67 | 28.18 | 268.27 | -48.38
wall_hue | 0.02 | 32.03 | 0.00 | 24.43 | 0.10 | 0.00 | 25.17 | $\infty$ | 7.80 | 0.29 | 0.46 | -76.40
object_hue | 0.38 | 22.71 | 0.43 | 24.55 | 1.55 | 0.63 | 59.65 | $\infty$ | 40.1 | 2.87 | 8.69 | -73.17
scale | 0.41 | 0.00 | 0.27 | 0.00 | 0.10 | 0.00 | 2.60 | 0.13 | 31.78 | 2.43 | 0.00 | -44.80
shape | 0.07 | 0.00 | 0.08 | 0.00 | 0.03 | 0.00 | 0.20 | 0.06 | -2.48 | 1.67 | 2.16 | -67.54
orientation | 0.00 | 0.00 | 0.00 | 0.00 | 0.23 | 0.00 | 0.48 | 0.00 | 22.26 | 2.50 | 6.68 | -9.11
average | 0.15 | 9.28 | 0.13 | 8.21 | 0.34 | 0.11 | 24.13 | $\infty$ | 24.02 | 6.32 | 47.71 | -53.23

#####  4.6.3 Dependence on Representation Size

We hypothesize that large representation sizes tend to lower the
$\mathrm{OFM}$, which could be one reason why representation sizes are large
in unsupervised learning. To affirm this hypothesis empirically, we train our
pretext models with varying representation sizes on different target tasks
while fixing all other model parameters. Figures 4.4 and 4.1 show that the
$\mathrm{OFM}$ tends to decrease when we enlarge the representation size. A
reason for that might be that target models can exploit the high dimensional
space of large representations to find better-fitting clusters for their
target task. We found an exception to this behavior, where we use larger
representations of SCLCAE for the easy task of object hue prediction. Here,
the target models trained on the untrained pretext models with larger
representation sizes already achieve high performance due to a larger number
of color-selective random features. Further learning of the pretext model does
not lead to a high performance gain in this case, and forgetting these
sensitive random features during training leads to a high mismatch.
Additionally, we observe that mismatches decrease when we decrease the
representation size for generation-based methods. A reason could be that the
pretext models are forced to generalize to solve the target task for small
representation sizes due to the limited amount of features in the bottleneck
or simply underfit on the pretext task.

#####  4.6.4 Dependence on Target Model Complexity

In Figures 4.4 and 4.1, we observe an $\mathrm{OFM}$ spike early in training
for more complex target models. This spike occurs probably because nonlinear
target models make better sense of specific random features at pretext task
initialization, in contrast to the linear target model. Besides early spikes,
mismatches tend to decrease when we add complexity to the target model. A
model with increased nonlinearity has more freedom to disentangle
representations that do not fit properly with the target task. Again we found
an exception where the $\mathrm{MOFM}$ is lower for linear models when
predicting the object hue after contrastive learning, which can be appointed
to the color-selective, random features of the untrained pretext model.

#####  4.6.5 Dependence on Augmentations

We vary the augmentations used for the pretext and target model by removing
the color jitter and the image flip from our base augmentations successively.
Figure 4.4 shows that augmentations can have a positive or negative impact on
the mismatch. E.g., when predicting the object’s hue, the ill-posed color
jitter augmentation increases the mismatch significantly.

#####  4.6.6 Dependence on Target Task Type

Here we use our metrics to examine findings stated in [58], [144], [289], and
[261], where it is argued that some pretext tasks are better suited for
different target tasks. We fix the underlying data distribution by using the
3dshapes dataset and train our target models for the different tasks. These
tasks require a generic understanding of the scene like coarse-grained
knowledge about the object’s type and hue and fine-grained knowledge about
shapes, positions, and scales. In Figures 4.5 and 4.2, we observe that pretext
models tend to learn pretext task-specific features and discard features that
are not needed to solve the pretext task during training. Therefore, these
models mismatch with ill-posed target tasks. For example, rotation prediction
discards features corresponding to the hue while it learns much about the
orientation of the object.

Table 4.3: Mismatches of ResNets with convergence criterium. $\mathrm{ACC}$
stands for the best accuracy on the target task of all target models trained
on the pretext model. $\mathrm{cSM3}$ corresponds to the accuracy we would
lose when we naively train the target model after pretext model convergence.
Values are obtained by 5-fold cross-validation. The mismatches are measured
with a convergence criterium.

| ResNet
---|---
| $\mathrm{ACC}$ | $\mathrm{cSM3}$ | $\mathrm{MOFM}$ | $\mathrm{MM3}$
RCAE (Cifar10) | $54.64^{+1.80}_{-2.01}$ | $3.98^{+1.74}_{-3.60}$ | $4.87^{+4.42}_{-3.11}$ | $31.82^{+0.75}_{-0.68}$
SCLCAE (PCam) | $96.25^{+0.44}_{-0.23}$ | $0.37^{+0.44}_{-0.37}$ | $0.86^{+1.00}_{-0.60}$ | $-53.26^{+0.52}_{-0.38}$

|
---|---
|
Figure 4.6: $\mathrm{OFM}$ for different pretext tasks trained with a ResNet18
model as the backbone. The mismatches are shown for the entire training
process.

#####  4.6.7 Applying our Metrics to ResNet Models

In Figure 4.6, we apply our metrics to ResNet models for several pretext and
target tasks. For contrastive learning, we observe a small $\mathrm{OFM}$ for
Cifar10 and Cifar100, which occurs late in training after pretext model
convergence. However, when we use contrastive learning as a pretext task for
fine-grained tumor detection on the PCam dataset, we observe a mismatch before
pretext model convergence. For the well-known rotation prediction pretext
task, we observe a high mismatch in Cifar10 classification early in pretext
training. In Table 4.3, we show the corresponding mismatches measured until
pretext model convergence.

####  4.7 Future Work

In future work, our metrics can be used to create, tune and evaluate (self-
supervised) representation learning methods for different target tasks and
datasets. These metrics make it possible to quantify the extent to which a
pretext task matches a target task and to determine whether the pretext task
learns the right kind of representation throughout the entire training
process. This enables a comparison of methods on benchmarks across different
pretext tasks and models. The dependencies of the objective function mismatch
on different parts of the self-supervised setup (e.g., representation size)
can be explored in future work in more detail to evaluate our findings further
and to create pretext tasks and model architectures that are robust against
mismatches. Our metrics are defined for setups where the target models are
trained on pretext model representations in general. Therefore, they can also
be applied to other representation learning areas such as supervised, semi-
supervised, few-shot, or biological plausible representation learning.

####  4.8 Conclusion

In this work, we have used the linear evaluation protocol as the basis to
define and discuss metrics to measure the metrics mismatch and the objective
function mismatch. With soft and hard versions of our metrics, we have
collected evidence of how these mismatches relate to the pretext model’s
representation size, target model complexity, pretext and target
augmentations, as well as pretext and target task types. Furthermore, we have
observed that the epoch of target task peak performance varies strongly for
different datasets and pretext tasks. This highlights the importance of the
protocol and shows that comparing approaches after a fixed number of epochs
does not yield the entire picture of their capability. Our protocols make it
possible to define benchmarks across different target tasks, where the goal is
not to mismatch with the target metrics while achieving the best possible
performance.

## Unsupervised Transfer of
Visual Representations

### A Lane Detection Benchmark for Multi-Target Domain Adaptation

####  5.1 Motivation

Vision-based deep learning systems for autonomous driving have made
significant progress in the past years [201, 77, 212, 192, 116]. Recent state-
of-the-art methods achieve remarkable results on public, real-world benchmarks
but require labeled, large-scale datasets. Annotations for these datasets are
often hard to acquire, mainly due to the high expenses of labeling in terms of
cost, time, and difficulty. Instead, simulation environments for autonomous
driving, such as CARLA [62], can be utilized to generate abundant labeled
images automatically. However, models trained on data from simulation often
experience a significant performance drop in a different domain, i.e., the
real world, mainly due to the domain shift [223]. Unsupervised domain
adaptation methods [75, 268, 176, 308, 76, 256, 277, 245] try to mitigate the
domain shift by transferring models from a fully-labeled source domain to an
unlabeled target domain. This eliminates the need for annotating images but
assumes that the target domain is accessible at training time. While
unsupervised domain adaptation has been applied to complex tasks for
autonomous driving, such as object detection [192, 278] and semantic
segmentation [302, 272], only few works focus on lane detection [78, 116].
This can be attributed to the lack of public unsupervised domain adaptation
datasets for lane detection.

####  5.2 Contributions

| MoLane | TuLane
---|---|---
Source |  |  |  |
Target |  |  |  |
Figure 5.1: Images sampled from our CARLANE Benchmark. Best viewed in color.

To compensate for this data scarcity and encourage future research, we
introduce CARLANE, a sim-to-real domain adaptation benchmark for lane
detection. We use the CARLA simulator for data collection in the source domain
with a free-roaming waypoint-based agent and data from two distinct real-world
domains as target domains. This enables us to construct a benchmark that
consists of three datasets:

(1) MoLane focuses on abstract lane markings in the domain of a 1/8th Model
vehicle. We collect 80K labeled images from simulation as the source domain
and 44K unlabeled real-world images from several tracks with two lane markings
as the target domain. Further, we apply domain randomization as well as data
balancing. For evaluation, we annotate 2,000 validation and 1,000 test images
with our labeling tool.

(2) TuLane incorporates 24K balanced and domain-randomized images from
simulation as the source domain and the well-known TuSimple [255] dataset with
3,268 real-world images from U.S. highways with up to four labeled lanes as
the target domain. The target domain of MoLane is a real-world abstraction
from the target domain of TuLane, which may result in interesting insights
about unsupervised domain adaptation.

(3) MuLane is a balanced combination of MoLane and TuLane with two target
domains. For the source domain, we randomly sample 24K images from MoLane and
combine them with TuLane’s synthetic images. For the target domains, we
randomly sample 3,268 images from MoLane and combine them with TuSimple. This
allows us to investigate multi-target unsupervised domain adaptation for lane
detection.

To establish baselines and investigate unsupervised domain adaptation on our
benchmark, we evaluate several adversarial discriminative methods, such as
DANN [76], ADDA [256], and SGADA [4]. Additionally, we propose SGPCS, which
builds upon PCS [286] with a pseudo-labeling approach to achieve state-of-the-
art performance.

In summary, our contributions are three-fold: (1) We introduce CARLANE, a
3-way sim-to-real benchmark, allowing single- and multi-target unsupervised
domain adaptation. (RQ-E5) (2) We provide several dataset tools, i.e., an
agent to collect images with lane annotations in CARLA and a labeling tool to
annotate the real-world images manually. (3) We evaluate several well-known
unsupervised domain adaptation methods - as well as our own SGPCS method - to
establish baselines and discuss results on both single- and multi-target
unsupervised domain adaptation. (RQ-T1 and RQ-E6) To the best of our
knowledge, we are the first to adapt a lane detection model from simulation to
multiple real-world domains.

####  5.3 Data Generation

To construct our benchmark, we gather image data from a real 1/8th model
vehicle and the CARLA simulator [62]. Ensuring the verification of results and
transferability to real driving scenarios, we extend our benchmark with the
TuSimple dataset [255]. This enables gradual testing, starting from
simulation, followed by model cars, and ending with full-scale real word
experiments. Data variety is achieved through domain randomization in all
domains. However, naively performing domain randomization might lead to an
imbalanced dataset. Therefore, similar driving scenarios are sampled across
all domains, and a bagging approach is utilized to uniformly collect lanes by
their curvature with respect to the camera position. We strictly follow
TuSimple’s data format [255] to maintain consistency across all our datasets.

#####  5.3.1 Real-World Environment

As shown in Figure 5.2, we build six different 1/8th race tracks, where each
track is available in two different surface materials (dark and light gray).
We vary between dotted and solid lane markings, which are pure white and
$50\text{\,}\mathrm{mm}$ thick. The lanes are constantly
$750\text{\,}\mathrm{mm}$ wide, and the smallest inner radius is
$250\text{\,}\mathrm{mm}$. The track layouts are designed to roughly contain
the same proportion of straight and curved segments to obtain a balanced label
distribution. We construct these tracks in four locations with alternating
backgrounds and lighting conditions.

(a)

(b)  (c)

(d)

(e)

(f)

Figure 5.2: Overview of our track types for MoLane. (a) - (d) show the black
version of the training and validation tracks. These tracks are also
constructed using a light gray surface material. (e) and (f) depict our test
tracks.

#####  5.3.2 Real-World Data Collection

Raw image data is recorded from a front-facing Stereolabs ZEDM camera with 30
FPS and a resolution of $1280\times 720$ pixels. A detailed description of the
1/8th car can be found in Appendix C. The vehicle is moved with a quasi-
constant velocity clockwise and counter-clockwise to cover both directions of
each track. All collected images from tracks (e) and (f) are used for the test
subset. In addition, we annotate lane markings with our labeling tool for
validation and testing, which is made publicly available.

#####  5.3.3 Simulation Environment

We utilize the built-in APIs from CARLA to randomize multiple aspects of the
agent and environment, such as weather, daytime, ego vehicle position, camera
position, distractor vehicles, and world objects (i.a., walls, buildings, and
plants). Weather and daytime are varied systematically by adapting parameters
for cloud density, rain intensity, puddles, wetness, wind strength, fog
density, sun azimuth, and sun altitude. For further details, we refer to our
implementation. To occlude the lanes similar to real-world scenarios, up to
five neighbor vehicles are spawned randomly in the vicinity of the agent. We
consider five different CARLA maps in urban and highway environments (Town03,
Town04, Town05, Town06, and Town10) to collect our dataset, as the other
towns’ characteristics are not suitable for our task (i.a., mostly straight
lanes). In addition, we collect data from the same towns without world objects
to strengthen the focus on lane detection, similar to our model vehicle target
domain.

#####  5.3.4 Simulation Data Agent

We implement an efficient agent based on waypoint navigation, which roams
randomly and reliably in the aforementioned map environments and collects
$1280\times 720$ images. In each step, the waypoint navigation stochastically
traverses the CARLA road map with a fixed lookahead distance of one meter. In
addition, we sample offset values $\Delta y_{k}$ from the center lane within
the range $\pm 1.20\text{\,}\mathrm{m}$.

To avoid saturation at the lane borders, which would occur with a sinusoidal
function, we use the triangle wave function:

$\Delta y_{k}=\frac{2m}{\pi}\arcsin(\sin(i_{k}))$ (5.1)

where $m$ is the maximal offset and $i_{k}$ is incremented by $0.08$ for each
simulation step $k$. Per frame, our agent moves to the next waypoint with an
increment of one meter, enabling the collection of highly diverse data in a
fast manner. We use a bagging approach for balancing, which allows us to
define lane classes based on their curvature.

####  5.4 The CARLANE Benchmark

Table 5.1: Dataset overview. Unlabeled images are denoted by *, partially
labeled images are denoted by **.

Dataset | domain | total images | train | validation | test | lanes
---|---|---|---|---|---|---
MoLane | CARLA simulation | 84,000 | 80,000 | 4,000 | - |  $\leq$ 2
model vehicle | 46,843 | 43,843* | 2,000 | 1,000 |  $\leq$ 2
TuLane | CARLA simulation | 26,400 | 24,000 | 2,400 | - |  $\leq$ 4
TuSimple [255] | 6,408 | 3,268 | 358 | 2,782 |  $\leq$ 4
MuLane | CARLA simulation | 52,800 | 48,000 | 4,800 | - |  $\leq$ 4
model vehicle + TuSimple [255] | 12,536 | 6,536** | 4,000 | 2,000 |  $\leq$ 4

The CARLANE Benchmark consists of three distinct sim-to-real datasets, which
we build from our three different domains. The details of the individual
subsets can be found in Table 5.1.

MoLane consists of images from CARLA and the real 1/8th model vehicle. For the
abstract real-world domain, we collect 46,843 images with our model vehicle,
of which 2,000 validation and 1,000 test images are labeled. For the source
domain, we use our simulation agent to gather 84,000 labeled images. To match
the label distributions between both domains, we define five lane classes
based on the relative angle $\beta$ of the agent to the center lane for our
bagging approach: strong left curve ($\beta\leq$$None$), soft left curve
($None$ $<\beta\leq$ $None$), straight ($None$ $<\beta<$ $None$), soft right
curve ($None$ $\leq\beta<$ $None$), and strong right curve
($None$$\leq\beta$). In total, MoLane encompasses 130,843 images.

TuLane consists of images from CARLA, and a cleaned version of the TuSimple
dataset [255], which is licensed under the Apache License, Version 2.0. To
clean test set annotations, we utilize our labeling tool to ensure that the up
to four lanes closest to the car are correctly labeled. We adapt the bagging
classes to align the source dataset with TuSimple’s lane distribution: left
curve ($None$ $<\beta\leq$ $None$), straight ($None$ $<\beta<$ $None$), and
right curve ($None$ $\leq\beta<$ $None$).

| MoLane | TuLane | MuLane
---|---|---|---
Source |  |  |
Target |  |  |
Figure 5.3: Lane annotation distributions of the three subsets of CARLANE.
Since the real-world training data of MoLane and MuLane is unlabeled, we
utilize their validation data for visualization.

MuLane is a multi-target unsupervised domain adaptation dataset and is a
balanced mixture of images from MoLane and TuLane. For MuLane’s entire
training set and its source domain validation and test set, we use all
available images from TuLane and sample the same amount of images from MoLane.
We adopt the 1,000 test images from MoLane’s target domain and sample 1,000
test images from TuSimple to form MuLane’s test set. For the validation set,
we use the 2,000 validation images from MoLane and 2,000 of the remaining
validation and test images of TuLane’s target domain. In total, MuLane
consists of 65,336 images.

To further analyze CARLANE, we visualize the ground truth lane distributions
in Figure 5.3. We observe that the lane distributions of source and target
data from our datasets are well aligned.

MoLane, TuLane, and MuLane are publicly available at
https://carlanebenchmark.github.io and licensed under the Apache License,
Version 2.0.

#####  5.4.1 Dataset Format

For each dataset, we split training, validation, and test samples into source
and target subsets. Lane annotations are stored within a .json file containing
the lanes’ y-values discretized by raw anchors, the lanes’ x-values, and the
image file path following the data format of TuSimple [255]. Additionally, we
adopt the method from [212] to generate .png lane segmentations and a .txt
file containing the linkage between the raw images and their segmentation as
well as the presence and absence of a lane.

#####  5.4.2 Dataset Tasks

The main task of our datasets is unsupervised domain adaptation for lane
detection, where the goal is to predict lane annotations
$Y_{t}\in\mathbb{R}^{R\times G\times N}$ given the input image
$X_{t}\in\mathbb{R}^{H\times W\times 3}$ from the unlabeled target domain
$\mathcal{D}_{\mathcal{T}}=\\{(X_{t})\\}_{t\in\mathcal{T}}$. $R$ defines the
number of row anchors, $G$ the number of griding cells, and $N$ the number of
lane annotations available in the dataset, where the definition of $Y_{t}$
follows [255]. During training time, the images $X_{s}\in\mathbb{R}^{H\times
W\times 3}$, corresponding labels $Y_{s}\in\mathbb{R}^{H\times W\times C}$
from the source domain
$\mathcal{D}_{\mathcal{S}}=\\{(X_{s},Y_{s})\\}_{s\in\mathcal{S}}$, and the
unlabeled target images $X_{t}$ are available. Additionally, MuLane focuses on
multi-target unsupervised domain adaptation, where
$\mathcal{D}_{\mathcal{T}}=\\{(X_{t_{1}})\cup(X_{t_{2}})\\}_{t_{1}\in\mathcal{T}_{1},t_{2}\in\mathcal{T}_{2}}$.

Although we focus on sim-to-real unsupervised domain adaptation, our datasets
can be used for unsupervised and semi-supervised tasks and partially for
supervised learning tasks. Furthermore, a real-to-real transfer can be
performed between the target domains of our datasets.

####  5.5 Benchmark Experiments

We conduct experiments on our CARLANE Benchmark for several unsupervised
domain adaptation methods from the literature and our proposed method.
Additionally, we train fully supervised baselines on all domains.

#####  5.5.1 Metrics

For evaluation, we use the following metrics:

(1) Lane Accuracy (LA) [212] is defined by $\textrm{LA}=\frac{p_{c}}{p_{y}}$,
where $p_{c}$ is the number of correctly predicted lane points and $p_{y}$ is
the number of ground truth lane points. Lane points are considered as correct
if their $L_{1}$ distance is smaller than the given threshold
$t_{pc}=\frac{20}{\cos(a_{yl})}$, where $a_{yl}$ is the angle of the
corresponding ground truth lane.

(2) False Positives (FP) and False Negatives (FN) [212]: To further determine
the error rate and to draw more emphasis on mispredicted or missing lanes, we
measure false positives with $\textrm{FP}=\frac{l_{f}}{l_{p}}$ and false
negatives with $\textrm{FN}=\frac{l_{m}}{l_{y}}$, where $l_{f}$ is the number
of mispredicted lanes, $l_{p}$ is the number of predicted lanes, $l_{m}$ is
the number of missing lanes, and $l_{y}$ is the number of ground truth lanes.
Following [212], we classify lanes as mispredicted if the $LA<85\%$.

#####  5.5.2 Baselines

We use Ultra Fast Structure-aware Deep Lane Detection (UFLD) [212] as baseline
and strictly adopt its training scheme and hyperparameters. UFLD treats lane
detection as a row-based classification problem and utilizes the row anchors
defined by TuSimple [255]. To achieve a lower bound for the evaluated
unsupervised domain adaptation methods, we train UFLD as a supervised baseline
on the source simulation data (UFLD-SO). Furthermore, we train our baseline on
the labeled real-world training data for a surpassable fully-supervised
performance in the target domain (UFLD-TO). Since the training images from
MoLane and MuLane have no annotations, we train UFLD-TO in these cases on the
labeled validation images and validate our model on the entire test set.

#####  5.5.3 Compared Unsupervised Domain Adaptation Methods

We evaluate the following feature-level unsupervised domain adaptation methods
on the CARLANE Benchmark by adopting their default hyperparameters and tuning
them accordingly. Each model is initialized with the pre-trained feature
encoder of our baseline model (UFLD-SO). The optimized hyperparameters can be
found in Table 5.2.

(1) DANN [76] is an adversarial discriminative method that utilizes a shared
feature encoder and a dense domain classifier connected via a gradient
reversal layer.

(2) ADDA [256] employs a feature encoder for each domain and a dense domain
discriminator. Following ADDA, we freeze the weights of the pre-trained
classifier of UFLD-SO to obtain final predictions.

(3) SGADA [4] builds upon ADDA and utilizes its predictions as pseudo labels
for the target training images. Since UFLD treats lane detection as a row-
based classification problem, we reformulate the pseudo label selection
mechanism. For each lane, we select the highest confidence value from the
griding cells of each row anchor. Based on their griding cell position, the
confidence values are divided into two cases: absent lane points and present
lane points. Thereby, the last griding cell represents absent lane points as
in [212]. For each case, we calculate the mean confidence over the
corresponding lanes. We then use the thresholds defined by SGADA to decide
whether the prediction is treated as a pseudo label.

(4) SGPCS (ours) builds upon PCS [286] and performs in-domain contrastive
learning and cross-domain self-supervised learning via cluster prototypes. Our
overall objective function comprises the in-domain and cross-domain loss from
PCS, the losses defined by UFLD, and our adopted pseudo loss from SGADA. We
adjust the momentum for memory bank feature updates to $0.5$ and use spherical
k-means [132] with $K=2,500$ to cluster them into prototypes.

Table 5.2: Optimized hyperparameters to achieve the reported results. $C$
denotes domain classifier parameters, $D$ denotes domain discriminator
parameters, adv the adversarial loss from [256], cls the classifier loss, sim
the similarity loss, and aux the auxiliary loss from [212]. Loss weights are
set to $1.0$ unless stated otherwise.

Method | Initial Learning Rate | Scheduler | Batch Size | Epochs | Losses | Other Changes
---|---|---|---|---|---|---
UFLD-SO | $4e^{-4}$ | Cosine Annealing | 4 | 150 | cls, sim, aux | -
DANN |  $1e^{-5}$, $C$: $1e^{-3}$ | $\frac{1e^{-5}}{(1+10p)^{0.75}}$ | 4 | 30 |  cls, sim, aux, adv [76] |  $C$: 3 fc layers (1024-1024-2)
ADDA |  $1e^{-6}$, $D$: $1e^{-3}$ | Constant | 16 | 30 |  map [256], adv [256] |  $D$: 3 fc layers (500-500-2)
SGADA | $1e^{-6}$, $D$: $1e^{-3}$ | Constant | 15 | 10 |  map [256], adv [256], | Pseudo label selection
pseudo: 0.25
SGPCS | $4e^{-4}$ | Cosine Annealing | 16 | 10 |  in-domain [286], | -
cross-domain [286],
cls, sim, aux, pseudo: 0.25
UFLD-TO | $4e^{-4}$ | Cosine Annealing | 4 | 300 | cls, sim, aux | -

#####  5.5.4 Implementation Details

We implement all methods in PyTorch 1.8.1 and train them on a single machine
with four RTX 2080 Ti GPUs. Tuning all methods has required a total amount of
compute of approximately 3.5 petaflop/s-days. The training times for each
model range from 4-13 days for UFLD baselines and 6-44 hours for domain
adaption methods. In addition, we have found that applying output scaling on
the last linear layer of the model yields slightly better results. Therefore,
we divide the models’ output by 0.5. Our implementation is publicly available
at https://carlanebenchmark.github.io.

#####  5.5.5 Evaluation

Quantitative Evaluation. In Table 5.3, we report the results on MoLane,
TuLane, and MuLane across five different runs. We observe that UFLD-SO is able
to generalize to a certain extent to the target domain. This is mainly due to
the alignment of semantic structure from both domains. ADDA, SGADA, and our
proposed SGPCS manage to adapt the model to the target domain slightly and
consistently. However, DANN suffers from negative transfer [263] when trained
on MoLane and MuLane. The negative transfer of DANN for complex domain
adaptation tasks is also observed in other works [250, 71, 263, 139] and can
be explained by the source domain’s data distribution and the model complexity
[263]. In our case, the source domain contains labels not present in the
target domain, as shown in Figure 5.3, which is more pronounced in MoLane and
MuLane.

We want to emphasize that with an accuracy gain of a maximum of 5.14% (SGPCS)
and high false positive and false negative rates, the domain adaptation
methods are not able to achieve comparable results to the supervised baselines
(UFLD-TO). Furthermore, we observe that false positive and false negative
rates increase significantly on MuLane, indicating that the multi-target
dataset forms the most challenging task. False positives and false negatives
represent wrongly detected and missing lanes which can lead to crucial impacts
on autonomous driving functions. These results affirm the need for the
proposed CARLANE Benchmark to further strengthen the research in unsupervised
domain adaptation for lane detection.

Table 5.3: Performance on the test set. Lane accuracy (LA), false positives
(FP), and false negatives (FN) are reported in %.

ResNet-18 | MoLane | TuLane | MuLane
---|---|---|---
LA | FP | FN | LA | FP | FN | LA | FP | FN
UFLD-SO | 88.15 | 34.35 | 28.45 | 87.43 | 34.21 | 23.48 | 79.61 | 44.78 | 33.36
DANN [76] |  85.25$\pm$0.49 |  39.07$\pm$1.21 |  36.18 $\pm$1.50 |  88.74$\pm$0.32 |  32.71$\pm$0.52 |  21.64$\pm$0.65 |  78.25$\pm$0.62 |  48.67$\pm$1.17 |  41.69$\pm$1.80
ADDA [256] |  92.10$\pm$0.21 |  19.71$\pm$0.77 |  11.46$\pm$0.92 |  90.72$\pm$0.15 |  29.73$\pm$0.36 |  17.67$\pm$0.42 |  81.25$\pm$0.33 |  41.69$\pm$0.40 |  28.58$\pm$0.57
SGADA [4] |  93.15$\pm$0.12 |  16.23$\pm$0.22 |  7.68$\pm$0.26 |  91.70$\pm$0.13 |  28.42$\pm$0.34 |  16.10$\pm$0.43 |  82.05$\pm$0.10 |  40.37$\pm$0.33 |  25.95$\pm$0.36
SGPCS (ours) |  93.29$\pm$0.07 |  16.22$\pm$0.17 |  7.29$\pm$0.13 |  91.55$\pm$0.13 |  28.52$\pm$0.21 |  16.16$\pm$0.26 |  82.91$\pm$0.19 |  40.76$\pm$0.57 |  26.14$\pm$0.75
UFLD-TO | 97.15 | 0.96 | 0.05 | 94.97 | 18.05 | 3.84 | 87.64 | 29.48 | 11.52
ResNet-34 | LA | FP | FN | LA | FP | FN | LA | FP | FN
UFLD-SO | 88.76 | 31.30 | 26.70 | 89.42 | 32.35 | 21.19 | 80.70 | 43.63 | 31.40
DANN [76] |  89.58$\pm$0.63 |  28.75$\pm$1.53 |  23.24$\pm$2.16 |  91.06$\pm$0.14 |  30.17$\pm$0.20 |  18.54$\pm$0.25 |  80.40$\pm$0.15 |  43.52$\pm$0.37 |  31.53$\pm$0.66
ADDA [256] |  91.76$\pm$0.29 |  21.04$\pm$0.81 |  13.06$\pm$1.02 |  91.39$\pm$0.16 |  28.76$\pm$0.30 |  16.63$\pm$0.36 |  81.64$\pm$0.34 |  40.74$\pm$0.48 |  27.50$\pm$0.78
SGADA [4] |  92.59$\pm$0.18 |  18.32$\pm$0.24 |  9.88$\pm$0.29 |  92.04$\pm$0.09 |  28.18$\pm$0.20 |  15.99$\pm$0.24 |  82.93$\pm$0.03 |  39.45$\pm$0.11 |  24.98$\pm$0.13
SGPCS (ours) |  92.82$\pm$0.27 |  17.10$\pm$0.73 |  8.77$\pm$0.95 |  93.29$\pm$0.18 |  25.68$\pm$0.48 |  12.73$\pm$0.59 |  82.87$\pm$0.17 |  40.13$\pm$0.28 |  25.38$\pm$0.45
UFLD-TO | 96.92 | 0.94 | 0.03 | 94.43 | 20.74 | 7.20 | 87.62 | 29.19 | 11.08

UFLD-SO | DANN | ADDA | SGADA | SGPCS
---|---|---|---|---
|  |  |  |
Figure 5.4: t-SNE visualization of MuLane dataset. The source domain is marked
in blue, the real-world model vehicle target domain in red, and the TuSimple
domain in green. Best viewed in color.

Qualitative Evaluation. We use t-SNE [258] to visualize the features of the
features encoders for the source and target domains of MuLane in Figure 5.4.
t-SNE visualizations of MoLane and TuLane can be found in Appendix C. In
accordance with the quantitative results, we observe only a slight adaptation
of the source and target domain features for ADDA, SGADA, and SGPCS compared
to the supervised baseline UFLD-SO. Consequently, the examined well-known
domain adaptation methods have no significant effect on feature alignment. In
addition, we show results from the evaluated methods in Figure 5.5 and observe
that the models are able to predict target domain lane annotations in many
cases but are not able to achieve comparable results to the supervised
baseline (UFLD-TO).

In summary, we find quantitatively and qualitatively that the examined domain
adaptation methods do not significantly improve the performance of lane
detection and feature adaptation. For this reason, we believe that the
proposed benchmark could facilitate the exploration of new domain adaptation
methods to overcome these problems.

| MoLane | TuLane | MuLane
---|---|---|---
UFLD-SO |  |  |  |
DANN |  |  |  |
ADDA |  |  |  |
SGADA |  |  |  |
SGPCS |  |  |  |
UFLD-TO |  |  |  |
Figure 5.5: Qualitative results of target domain predictions. Ground truth
lane annotations are marked in blue, and predictions in red. Best viewed in
color.

####  5.6 Conclusion

We present CARLANE, the first unsupervised domain adaptation benchmark for
lane detection. CARLANE was recorded in three domains and consists of three
datasets: the single-target datasets MoLane and TuLane and the multi-target
dataset MuLane, which is a balanced combination of both. Based on the UFLD
model, we have conducted experiments with different unsupervised domain
adaptation methods on CARLANE and found that the selected methods are able to
adapt the model to target domains slightly and consistently. However, none of
the methods achieve comparable results to the supervised baselines. The most
significant performance differences are noticeable in the high false positive
and false negative rates of the unsupervised domain adaptation methods
compared to the target-only baselines, which is even more pronounced in the
MuLane multi-target task. These false-positive and false-negative rates can
negatively impact autonomous driving functions since they represent
misidentified and missing lanes. Furthermore, as shown in the t-SNE plots of
Figure 5.4, the examined well-known domain adaptation methods have no
significant effect on feature alignment. The current difficulties of the
examined unsupervised domain adaptation methods to align the source and target
domains adequately confirm the need for the proposed CARLANE benchmark. We
believe that CARLANE eases the development and comparison of unsupervised
domain adaptation methods for lane detection. In addition, we open-source all
tools for dataset creation and labeling and hope that CARLANE facilitates
future research in these directions.

Limitations. One limitation of our work is that we only use a fixed set of
track elements within our 1/8th scaled environment. These track elements
represent only a limited number of distinct curve radii. Furthermore, neither
buildings nor traffic signs exist in MoLane’s model vehicle target domain.
Moreover, the full-scale real-world target domain of TuLane is derived from
TuSimple. TuSimple’s data was predominantly collected under good and medium
conditions and lacks variation in weather and time of day. In addition, we
want to emphasize that collecting data for autonomous driving is still an
ongoing effort and that datasets such as TuSimple do not cover all possible
real-world driving scenarios to ensure safe, practical use. For the
synthetically generated data, we limited ourselves to using existing CARLA
maps without defining new simulation environments. Despite these limitations,
CARLANE serves as a supportive dataset for further research in the field of
unsupervised domain adaptation.

Ethical and Responsible Use. Considering the limitations of our work,
unsupervised domain adaptation methods trained on TuLane and MuLane should be
tested with care and under the right conditions on a full-scale car. However,
real-world testing with MoLane in the model vehicle domain can be carried out
in a safe and controlled environment. Additionally, TuLane contains open-
source images with unblurred license plates and people. This data should be
treated with respect and in accordance with privacy policies. In general, our
work contributes to the research in the field of autonomous driving, in which
a lot of unresolved ethical and legal questions are still being discussed. The
step-by-step testing possibility across three domains makes it possible for
our benchmark to include an additional safety mechanism for real-world
testing.

### Content-Consistent Translation with Masked Discriminators

(a) PFD$\rightarrow$Cityscapes

(b) Day$\rightarrow$Night

(c) Viper$\rightarrow$Cityscapes

(d) Clear$\rightarrow$Snowy

Figure 6.1: Results of our method. Best viewed in color.

####  6.1 Motivation

Unpaired image-to-image translation aims at transferring images from a source
domain to a target domain when no paired examples are given. Recently, this
field has attracted increasing interest and has advanced several use cases,
such as photorealism [209, 216, 128], neural rendering [102], domain
adaptation [113, 219], the translation of seasons or daytime [130, 127, 209],
and artistic style transfer [119, 283, 140]. Current work has primarily
focused on improving translation quality [193, 127], efficiency [160, 235],
multi-modality [120, 164], and content consistency [216, 128]. Due to the ill-
posed nature of the unpaired image-to-image translation task and biases
between datasets, content consistency is difficult to achieve. To mitigate
content inconsistencies, several methods have been proposed that constrain the
generator of GANs [307, 20, 74, 164, 295, 303, 170, 120, 233, 282]. However,
only constraining the generator leads to an unfair setup, as biases in the
datasets can be detected by the discriminator: The generator tries to achieve
content consistency by avoiding biases in the output, while the discriminator
is still able to detect biases between both datasets and, therefore, forces
the generator to include these biases in the output, for example, through
hallucinations. Constraining the discriminator [161, 216, 251] or improving
the sampling of training pairs [137, 216] is currently underexplored,
especially for content consistency on a global level, where the discriminator
has a global view on larger image crops instead of a local view on small
crops.

####  6.2 Contributions

In this work, we propose masked conditional discriminators, which operate on
masked global crops of the inputs to mitigate content inconsistencies. We
combine these discriminators with an efficient sampling strategy based on a
pre-trained robust segmentation model to sample similar global crops.
Furthermore, we argue that when transferring feature statistics from the
content stream of the source image to the generator stream, content-unrelated
feature statistics from the content stream could affect image quality if the
generator is unable to ignore this information since the output image should
mimic the target domain. Therefore, we propose a feature-attentive
denormalization (FATE) block that extends feature-adaptive denormalization
(FADE) [130] with an attention mechanism. This block allows the generator to
selectively incorporate statistical features from the content stream into the
generator stream. In our experiments, we find that our method achieves state-
of-the-art performance on most of the benchmarks shown in Figure 6.1.

Our contributions can be summarized as follows:

  * •

We propose an efficient sampling strategy that utilizes robust semantic
segmentations to sample similar global crops. This reduces biases between both
datasets induced by semantic class misalignment. (RQ-T2)

  * •

We combine this strategy with masked conditional discriminators to achieve
content consistency while maintaining a more global field of view. (RQ-T2)

  * •

We extend our method with an unmasked local discriminator. This discriminator
operates on local, partially class-aligned patches to minimize the
underrepresentation of frequently masked classes and associated artifacts.
(RQ-T2)

  * •

We propose a feature-attentive denormalization (FATE) block, which selectively
fuses statistical features from the content stream into the generator stream.
(RQ-T3)

  * •

We propose the class-specific Kernel VGG Distance (cKVD) that builds upon the
semantically aligned Kernel VGG Distance (sKVD) [216] and uses robust
segmentations to incorporate class-specific content inconsistencies in the
perceptual image quality measurement. (RQ-E7)

  * •

In our experiments, we show that our method achieves state-of-the-art
performance on photo-realistic sim-to-real transfer and the translation of
weather and performs well for daytime translation. (RQ-E8)

####  6.3 Method

Figure 6.2: Method overview. In our method, similar image crops from both
domains ($i_{a}$, $i_{b}$) and their corresponding conditions ($c_{a}$,
$c_{b}$, $z_{a}$) are selected via a sampling procedure. In this sampling
procedure, a mask $M_{ab}$ is created from the conditions $C_{a}$ and $C_{b}$.
This mask is used to sample crops from both datasets for which the semantic
classes align by at least 50%. The cropped mask $m_{ab}$ is also used to mask
the generated fake image $f_{b}$, the real images $i_{b}$, and the
corresponding conditions for the global conditional discriminators. Through
the mask, these discriminators can only see the parts of the crop where the
semantic classes align. To further improve image quality, a local
discriminator is introduced that works on a batch of small patches selected
from the crop using our sampling technique. This discriminator is not masked
and works on patches where the semantic classes do not fully align. Best
viewed in color.

We propose an end-to-end framework for unpaired image-to-image translation
that transfers an image $I_{a}\in\mathbb{R}^{3\times h\times w}$ from a source
domain $a$ to an image $F_{b}\in\mathbb{R}^{3\times h\times w}$ from a target
domain $b$. Our goal is to design a method for content-consistent translations
that utilizes a simple masking strategy for the global crops seen by the
discriminators. We achieve this by combining an efficient segmentation-based
sampling method that samples large crops from the input image with a masked
discriminator that operates on these global crops. This is in contrast to EPE
[216], which achieves content-consistent translation at the local level by
sampling small, similar image crops from both domains. To further improve
image quality, we use a local discriminator that operates on a batch of small
image patches sampled from the global input crops utilizing our sampling
method. An overview of our method is shown in Figure 6.2. Furthermore, we
propose a feature-attentive denormalization (FATE) block that extends feature-
adaptive denormalization (FADE) [130] with an attention mechanism, allowing
the generator to selectively incorporate statistical features from the content
stream of the source image into the generator stream.

#####  6.3.1 Contend-based Similarity Sampling

To minimize the bias between both datasets in the early stage of our method,
we sample similar image crops with an efficient sampling procedure. This
procedure uses the one-hot encoded semantic segmentations
$C_{a}\in\mathbb{R}^{d\times h\times w}$ and $C_{b}\in\mathbb{R}^{d\times
h\times w}$ of both domains, where $d$ is the channel dimension of the one-hot
encoding. In our case, these segmentations are created with the robust pre-
trained MSeg model [150]. First, a mask $M_{ab}\in\mathbb{R}^{1\times h\times
w}$ is computed from the segmentations:

$M_{ab}=\operatorname*{max}_{d}(C_{a}\circ C_{b}),$ (6.1)

where $\circ$ denotes the Hadamard product. We can now sample semantically
aligned image crops $i_{a}$ and $i_{b}$ from the images $I_{a}$ and $I_{b}$
with the crop $m_{ab}$ from mask $M_{ab}$. Thereby, we calculate the
percentage of overlap of semantic classes between both image crops as follows:

$\mathcal{P}_{match}(i_{a})=\\{i_{b}\mid\operatorname*{mean}(m_{ab})>t\\},$
(6.2)

where $t$ is the similarity sampling threshold. In our case, we sample crops
where more than $50\%$ of the semantic classes align ($t>0.5$). We use this
procedure to sample crops $c_{a}$, $c_{b}$, and $z_{b}$ from the discriminator
conditions $C_{a}$, $C_{b}$, and the generator condition $Z_{b}$ as well. The
cropped mask $m_{ab}$ is also used for our masked conditional discriminator.s

#####  6.3.2 Contend-based Discriminator Masking

To train a discriminator with a global field of view that facilitates the
usage of global properties of the scene, while maintaining content
consistency, we mask the discriminator input from both domains with a content-
based mask $m_{ab}$. This mask erases all pixels from the discriminator input
where the semantic classes do not align. This removes the bias between both
datasets caused by the underlying semantic class distribution of the two
domains without directly restricting the generator. The objective function of
a conditional GAN with a masked discriminator that transfers image crops
$i_{a}$ to domain $b$ can be then defined as follows:

$\begin{split}&\mathcal{L}_{madv}=\leavevmode\nobreak\
\mathbb{E}_{i_{b},c_{b},m_{ab}}[\log D(i_{b}\circ m_{ab}|c_{b}\circ
m_{ab})]\\\
&+\mathbb{E}_{i_{a},z_{a},c_{a},m_{ab}}[\log(1-D(G(i_{a}|z_{a})\circ
m_{ab}|c_{a}\circ m_{ab}))].\end{split}$ (6.3)

To ensure that the discriminator does not use the segmentation maps as
learning shortcuts, we follow [216] and create the segmentations of both
datasets using a robust segmentation model such as MSeg [150]. With this
setting, we are able to train discriminators with large crop sizes with
significantly reduced hallucinations in the translated image.

#####  6.3.3 Local Discriminator

Masking the input of the discriminator may lead to the underrepresentation of
some semantic classes. Therefore, we additionally train a local discriminator
that operates on a batch of small patches sampled from the global crop. Our
local discriminator is not masked but only sees patches where a certain amount
of the semantic classes align. In our case, we sample patches with 1/8th the
size of the global input crop where more than $50\%$ of the semantic classes
align. We use our sampling procedure from subsection 6.3.1 to sample these
patches. Using small, partially aligned patches ensures that semantic classes
are less underrepresented while maintaining content consistency.

#####  6.3.4 Feature-attentive Denormalization (FATE)

Figure 6.3: FADE and FATE.

Spatially adaptive denormalization (SPADE) [205] fuses resized semantic
segmentation maps as content into the generator stream. Feature-adaptive
denormalization (FADE) [130] generalizes SPADE to features learned through a
content stream. As shown in Figure 6.3, the normalized features $N(h)$ of the
generator are modulated with the features $f$ of the content stream using the
learned functions $\gamma$ and $\beta$ as follows:

$\operatorname*{FADE}(h,f)=N(h)\circ\gamma(f)+\beta(f),$ (6.4)

where $\gamma$ and $\beta$ are one-layer convolutions. This denormalization is
applied in several layers of the generator. However, we argue that
denormalization with content features is not always appropriate for
transferring images to another domain because, as shown in [79, 156, 159,
119], image feature statistics contain not only content information but also
style information. When transferring feature statistics from the content
stream of the source image to the generator stream, style information from the
source image could affect the final image quality if the generator cannot
ignore this information since the output image should mimic the style of the
target domain. Therefore, we propose an additional attention mechanism to
selectively incorporate statistics from the content stream into the generator
stream. This allows the model to only fuse the statistical features from the
source image into the generator stream that are useful for the target domain.
As shown in Figure 6.3, this attention mechanism relies on the features of the
content stream and the features of the generator stream and attends to the
statistics $\gamma$ and $\beta$. With this attention mechanism, we can extend
FADE to feature-attentive denormalization (FATE) as follows:

$\operatorname*{FATE}(h,f)=N(h)\circ
A(h,f)\circ\gamma(f)+A(h,f)\circ\beta(f),$ (6.5)

where $A$ is the attention mechanism and $A(h,f)$ is the attention map for the
statistics. We use a lightweight two-layer CNN with sigmoid activation in the
last layer as the attention mechanism. More details can be found in Appendix
D.

#####  6.3.5 Training Objective

Our training objective consists of three losses: a global masked adversarial
loss $\mathcal{L}_{madv}^{global}$, a local adversarial loss
$\mathcal{L}_{adv}^{local}$, and the perceptual loss $\mathcal{L}_{perc}$ used
in [130]. We define the final training objective as follows:

$\mathcal{L}=\lambda_{madv}^{global}\mathcal{L}_{madv}^{global}+\lambda_{adv}^{local}\mathcal{L}_{adv}^{local}+\lambda_{perc}\mathcal{L}_{perc},$
(6.6)

where we use a hinge loss to formulate the adversarial losses and
$\lambda_{madv}^{global}$, $\lambda_{madv}^{local}$, $\lambda_{perc}$ are the
corresponding loss weights.

####  6.4 Experiments

#####  6.4.1 Experimental Settings

Implementation details. Our method is implemented in PyTorch 1.10.0 and
trained on an A100 GPU (40 GB) with batch size $1$. For training, we
initialize all weights with the Xavier normal distribution [86] with a gain of
$0.02$ and use an Adam optimizer [141] with $\beta_{1}=0.9$ and
$\beta_{2}=0.999$. The initial learning rates of the generator and
discriminators are set to $0.0001$ and halved every $d_{e}$ epochs. Learning
rate decay is stopped after reaching a learning rate of $0.0000125$. We
formulate our adversarial objective with a hinge loss [163] and weight the
individual parts of our loss function as follows:
$\lambda_{madv}^{global}=1.0,\lambda_{madv}^{local}=1.0$,
$\lambda_{perc}=1.0$. In addition, we use a gradient penalty on target images
[97, 184] with $\lambda_{rp}=0.03$. The images of both domains are resized and
cropped to the same size and randomly flipped before the sampling strategy is
applied. In our experiments, we show that we achieve the best performance by
cropping global patches of size 352$\times$352\. We crop local patches with
1/8th the size of the global crop (i.a., 44$\times$44). The global
discriminators are used on two scales. Crops are scaled down by a factor of
two for the second scale. We train all our models for $\sim\\!400$K
iterations. Training a model takes 4-8 days, depending on the dataset, model,
and crop size. We report all results as an average across five different runs.
We refer to Appendix D for more details regarding the training and model. Our
implementation is publicly available at
https://github.com/BonifazStuhr/feamgan.

Memory usage. Our best model requires $\sim$25 GB of VRAM at training time and
performs inference using $\sim$12 GB for an image of size 957$\times$526\. Our
small model, with a slight performance decrease, runs on consumer graphic
cards with $\sim$9 GB of VRAM at training time and performs inference using
$\sim$8 GB for an image of size 957$\times$526\.

Datasets. We conduct experiments on four translation tasks across four
datasets. For all datasets, we compute semantic segmentations with MSeg [150],
which we use as a condition for our discriminator and to calculate the
discriminator masks.

(1) PFD [218] consists of images of realistic virtual world gameplay. Each
frame is annotated with pixel-wise semantic labels, which we use as additional
input for our generator. We use the same subset as [216] to compare with
recent work.

(2) Viper [217] consists of sequences of realistic virtual world gameplay.
Each frame is annotated with different labels, where we use the pixel-wise
semantic segmentations as additional input for our generator. Since Cityscapes
does not contain night sequences, we remove them from the dataset.

(3) Cityscapes [50] consists of sequences of real street scenes from 50
different German cities. We use the sequences of the entire training set to
train our models. We use datasets (1-3) for the sim-to-real translation tasks
PFD$\rightarrow$Cityscapes and Viper$\rightarrow$Cityscapes.

(4) BDD100K [285] is a large-scale driving dataset. We use subsets of the
training and validation data for the following translation tasks:
Day$\rightarrow$Night, Clear$\rightarrow$Snowy.

Compared methods. We compare our work with the following methods.

  * •

Color Transfer (CT) [214] performs color correction by transferring
statistical features in lαβ space from the target to the source image.

  * •

MUNIT [120] achieves multimodal translation by recombining the content code of
an image with a style code sampled from the style space of the target domain.
It is an extension of CycleGAN [307] and UNIT [170].

  * •

CUT [204] uses a patchwise contrastive loss to achieve one-sided unsupervised
image-to-image translation.

  * •

TSIT [130] achieves one-sided translation by fusing features from the content
stream into the generator on multiple scales using FADE and utilizing a
perceptual loss between the translated and source images.

  * •

QS-Attn [117] builds upon CUT [204] with an attention module that selects
significant anchors for the contrastive loss instead of features from random
locations of the image.

  * •

EPE [216] relies on a variety of gbuffers as input. Techniques such as
similarity cropping, utilizing segmentations for both domains generated by a
robust segmentation model as input to the conditional discriminators, and
small patch training are used to achieve content consistency.

Since EPE [216] provides inferred images of size 957$\times$526 for the
PFD$\rightarrow$Cityscapes task, comparisons are performed on this resolution.
For the Viper$\rightarrow$Cityscapes, Day$\rightarrow$Night, and
Clear$\rightarrow$Snowy tasks, we train the models using their official
implementations. Furthermore, we retrain models as additional baselines for
the PFD$\rightarrow$Cityscapes task.

Evaluation metrics. Following prior work [216], we use the Fréchet Inception
Distance (FID) [108], the Kernel Inception Distance (KID) [23], and the
semantically aligned Kernel VGG Distance (sKVD) [216] to evaluate image
translation quality quantitatively. The sKVD metric was introduced in [216]
and improved over previous metrics for mismatched layouts in source and target
data. In addition, we propose the class-specific Kernel VGG Distance (cKVD),
where a robust segmentation model is used before the sKVD calculation to mask
input crops by class (or category). Thereby, for each given class, all source
and target image crops are filtered using their segmentations by erasing the
pixels of all other classes. We select crops where more then $5$% of the
pixels belong to the respective class. Then, the sKVD is calculated class-wise
on the filtered crops. Afterward, we can report the cKVD as an average over
all classes or separately for each class to achieve a more fine-grained
measurement. We follow [216] and use a crop size of $1/8$ and sample source
and target crop pairs with an similarity threshold of $0.5$ between unmasked
source and target segmentation crops. More information on the classes used in
the cKVD metric can be found in Table D.1 of Appendix D. For the KID, sKVD,
and cKVD metrics, we multiply the measurements by $1000$ to improve the
readability of results.

(a) Input

(b) EPE

(c) FeaMGAN (ours)

(d) Generator Input Types

Figure 6.4: Qualitative comparison to EPE. We compare our method with the
provided inferred images of EPE [216]. Best viewed in color.

Table 6.1: Quantitative comparison to the baselines provided by EPE. We
calculate all metrics on the provided inferred images of EPE and its baselines
[216].

Method | FID | KID | sKVD | cKVD
---|---|---|---|---
AVG |  AVGsp | sky | ground | road | terrain | vegetation | building | roadside-obj. | person | vehicle | rest
ColorTransfer | 84.34 | 88.17 | 16.65 | 36.01 | 33.12 | 32.40 | 12.97 | 16.13 | 20.94 | 19.24 | 29.92 | 74.79 | 62.78 | 41.79 | 49.16
MUNIT | 45.00 | 35.05 | 16.51 | 38.57 | 34.81 | 29.80 | 16.93 | 17.62 | 29.52 | 19.29 | 24.28 | 79.14 | 77.34 | 40.13 | 51.61
CUT | 47.71 | 42.01 | 18.03 | 35.31 | 33.26 | 25.96 | 15.32 | 17.87 | 20.09 | 22.72 | 25.00 | 74.02 | 60.99 | 41.71 | 49.37
EPE | 44.06 | 33.66 | 13.87 | 35.22 | 30.21 | 27.14 | 13.54 | 13.56 | 24.77 | 20.77 | 26.75 | 50.58 | 83.34 | 41.29 | 50.45
FeaMGAN-S (ours) | 43.27 | 32.59 | 12.98 | 40.23 | 32.69 | 38.10 | 13.29 | 15.34 | 26.29 | 20.17 | 27.32 | 61.57 | 102.65 | 42.83 | 54.73
FeaMGAN (ours) | 40.32 | 28.59 | 12.94 | 40.02 | 31.78 | 46.70 | 13.72 | 15.60 | 23.23 | 17.69 | 25.57 | 66.65 | 99.24 | 39.38 | 52.40

#####  6.4.2 Comparison to the State of the Art

We compare our models quantitatively and qualitatively with different
baselines. First, we compare our results with EPE and the baselines provided
by EPE [216]. Then, we train our own baselines on the four translation tasks
for further comparison.

Comparison to EPE. A set of inferred images is provided for EPE and each of
the baselines [216]. Therefore, we train our models on the same training set
and use the inferred images from our best models for this comparison. We
select our best models based on scores of various visual metrics and visual
inspections of translated images. As shown in Figure 6.4 a) and b), our model
relies solely on segmentation maps as additional input compared to EPE, which
uses a variety of gbuffers. In addition, our model is trained with
significantly fewer steps ($\sim\\!400$K iterations) compared to EPE and the
baselines ($1$M iterations). As shown in Table 6.1, our model outperforms the
baselines and EPE in all commonly used metrics (FID and KID) and the sKVD
metric. More surprisingly, our small model, which can be trained on consumer
GPUs, outperforms all baselines and EPE as well.

However, our cKVD metric shows that our models have difficulty with the person
and sky classes. Therefore, the average cKVD values are high and become low
when we remove both classes from the average calculation (AVGsp). A possible
reason for the weaker performance on the person class is our masking
procedure. Since the masking procedure requires overlapping samples in both
domains, the person class is not seen frequently during training. This can
lead to inconsistencies (a glow) around the person class, as seen in Figure
6.11 of our limitations. The masking procedure also leads to a drop in
performance in the sky class, as seen in Table 6.3 of our ablation study.

As shown in the first row of Figure 6.4 and the results of Figures D.7 and D.8
of Appendix D, our model translates larger structures, such as lane markings,
more consistently, but fails to preserve some in-class characteristics from
the source dataset. This is evident, for example, in the structure of
translated streets and the corresponding cKVD value (road). As shown in the
second row and Appendix D, EPE achieves visually superior modeling of the
reflective properties of materials (e.g., the car) but suffers from
inconsistencies (erased objects) regarding the vegetation, which can be seen
in the palm trees and the corresponding cKVD value (vegetation). The superior
modeling of reflective properties can be attributed to the availability of
gbuffers (i.a., glossines) in EPE’s input.

By surpassing EPE in all commonly used quantitative metrics while maintaining
content consistency, we are able to show that our model improves overall
quantitative translation performance. However, our method has specific
drawbacks that we discussed with the help of the cKVD metric and visual
comparisons.

PFD$\rightarrow$Cityscapes

Viper$\rightarrow$Cityscapes

Day$\rightarrow$Night

Clear$\rightarrow$Snowy

(a) Input

(b) MUNIT

(c) CUT

(d) TSIT

(e) QS-Attn

(f) FeaMGAN (ours)

Figure 6.5: Qualitative comparison to prior work. Models were trained using
their official implementations. Randomly sampled results can be found in
Figure D.9 of Appendix D. Best viewed in color.

Table 6.2: Quantitative comparison to prior work. Models were trained using
their official implementations. Results are reported as the average across
five runs. We refer to Table D.4 of Appendix D for an extended version of this
table.

Method |  PFD$\rightarrow$Cityscapes |  Viper$\rightarrow$Cityscapes |  Day$\rightarrow$Night |  Clear$\rightarrow$Snowy
---|---|---|---|---
FID | KID | sKVD | cKVD | FID | KID | sKVD | cKVD | FID | KID | sKVD | cKVD | FID | KID | sKVD | cKVD
Color Transfer | 91.01 | 94.82 | 18.16 | 50.87 | 89.30 | 83.51 | 20.20 | 51.23 | 125.90 | 140.60 | 32.58 | 56.52 | 46.85 | 19.44 | 14.91 | 42.89
MUNIT | 40.36 | 29.98 | 14.99 | 43.24 | 47.96 | 30.35 | 14.14 | 59.62 | 42.53 | 31.83 | 15.02 | 50.83 | 44.74 | 17.48 | 11.65 | 48.10
CUT | 49.55 | 44.25 | 16.85 | 37.53 | 60.35 | 49.48 | 16.80 | 51.02 | 34.36 | 20.54 | 10.16 | 53.55 | 46.03 | 15.70 | 14.71 | 43.91
TSIT | 38.70 | 28.70 | 10.80 | 42.35 | 45.26 | 28.40 | 8.47 | 50.03 | 54.96 | 33.21 | 12.71 | 57.91 | 79.28 | 40.02 | 12.97 | 41.52
QS-Attn | 49.41 | 42.87 | 14.01 | 38.57 | 55.62 | 39.31 | 12.99 | 63.22 | 46.67 | 21.47 | 7.58 | 52.02 | 60.91 | 18.85 | 14.19 | 44.00
FeaMGAN-S (ours) | 45.16 | 34.93 | 13.87 | 40.50 | 52.79 | 35.92 | 14.34 | 45.38 | 70.40 | 51.30 | 14.68 | 46.66 | 57.93 | 16.24 | 11.88 | 38.28
FeaMGAN (ours) | 46.12 | 36.56 | 13.69 | 41.19 | 51.56 | 34.63 | 14.01 | 47.21 | 66.39 | 46.96 | 13.14 | 46.88 | 56.78 | 14.77 | 11.36 | 41.72

Comparisons to retrained baselines. We find that retraining the baselines with
their original training setup for the PFD$\rightarrow$Cityscapes task
significantly improves their performance on commonly used metrics compared to
the baselines provided by EPE, as can be seen in Table 6.2. However, as shown
in Figure 6.5 and the random results of Figure D.9 of Appendix D, content-
consistency problems remain. This indicates again that simply relying on
commonly used metrics does not provide a complete picture if content
consistency is taken into account. When qualitatively comparing our model to
the baselines for the PFD$\rightarrow$Cityscapes and
Viper$\rightarrow$Cityscapes tasks in Figure 6.5, we observe that our method
significantly reduces content inconsistencies. However, a limitation of our
masking strategy are class boundary artifacts, which are particularly evident
in the Day$\rightarrow$Night translation task (Figure 6.11). Since masking
allows our method to focus on specific classes, we achieve state-of-the-art
performance for the Clear$\rightarrow$Snowy translation task.

(a) Input

(b) Full

(c) w/o Dis. Mask

(d) w/o Local Dis.

(e) w/ FADE w/o FATE

Figure 6.6: Qualitative ablations. Results are selected from the best model.
Randomly sampled results can be found in Figure D.11 of Appendix D. Best
viewed in color.

Table 6.3: Quantitative evaluation for ablation study. Results are reported as
the average across five runs. We refer to Table D.5 of Appendix D for an
extended version of this table.

Method | FID | KID | sKVD | cKVD
---|---|---|---|---
AVG | sky | ground | road | terrain | vegetation | building | roadside-obj. | person | vehicle | rest
FeaMGan (Full) | 46.12 | 36.56 | 13.69 | 41.19 | 42.69 | 14.97 | 17.35 | 26.51 | 20.25 | 26.34 | 64.64 | 102.23 | 42.38 | 54.52
w/o Dis. Mask | 37.10 | 25.88 | 14.73 | 39.65 | 26.70 | 15.81 | 16.65 | 31.02 | 22.97 | 25.39 | 67.01 | 93.78 | 44.23 | 52.91
w/ FADE w/o FATE | 45.46 | 35.73 | 13.17 | 40.90 | 41.49 | 13.78 | 16.78 | 25.30 | 20.58 | 27.21 | 63.12 | 104.43 | 42.44 | 53.83
w/ Random Crop | 47.88 | 38.48 | 13.37 | 40.18 | 39.88 | 12.90 | 14.65 | 25.09 | 21.89 | 27.32 | 64.32 | 98.81 | 43.08 | 53.86
w/ VGG Crop | 51.23 | 42.46 | 13.56 | 40.62 | 40.32 | 13.38 | 15.67 | 26.47 | 21.09 | 27.28 | 65.23 | 99.61 | 43.19 | 53.94
w/o Local Dis. |  |  |  |  |  |  |  |  |  |  |  |  |  |
\- w/ 256$\times$256 Crop | 48.57 | 38.89 | 12.89 | 41.26 | 42.31 | 13.57 | 15.98 | 25.28 | 22.18 | 26.56 | 61.13 | 107.48 | 42.44 | 55.62
\- w/ 352$\times$352 Crop | 47.26 | 37.75 | 14.38 | 39.30 | 34.44 | 13.09 | 15.84 | 25.83 | 21.50 | 27.20 | 61.24 | 98.25 | 42.24 | 53.38
\- w/ 464$\times$464 Crop | 46.61 | 37.25 | 15.04 | 38.62 | 31.60 | 13.13 | 15.38 | 27.06 | 22.23 | 29.67 | 63.38 | 87.51 | 44.41 | 51.77
\- w/ 512$\times$512 Crop | 55.89 | 49.12 | 15.94 | 39.35 | 36.48 | 14.68 | 16.06 | 26.87 | 19.61 | 27.37 | 62.40 | 98.90 | 40.32 | 50.86

#####  6.4.3 Ablation Study

Effectiveness of masked discriminator. As shown in Figure 6.6 and the random
samples in Figure D.11 of Appendix D, our masking strategy for the
discriminator positively impacts content consistency. Without masking,
inconsistencies occur that correlate with biases between the class
distributions of the source and target domains. As shown in [216], the
distributions of certain classes in the spatial image dimension vary greatly
between the PFD dataset and the Cityscapes dataset. For example, trees in
Cityscapes appear more frequently in the top half of the image, resulting in
hallucinated trees when the images are translated without accounting for
biases. In the first and second row of Figure 6.6, we show that our masking
strategy (Full) prevents these inconsistencies in contrast to our model
trained without masking (w/o Dis. Mask). However, as shown in Table 6.3, this
comes with a quantitative tradeoff in performance on commonly used metrics.

Effectiveness of local discriminator. We compare our model trained with a
local discriminator (Full) to the model trained without a local discriminator
(w/o Local Dis. 352x352). As shown in Figure 6.6, the local discriminator
leads to an increase in quantitative performance. Furthermore, we show the
qualitative effects of the local discriminator in Figure 6.6, where we observe
a decrease in glowing objects and a significant decrease of erased objects in
the translation. An example of a glowing object is the palm tree in row two of
Figure 6.6. An example of erased objects are the missing houses in the
background of the images from row three. In addition, small inconsistencies
near object boundaries are reduced, as shown by the randomly sampled results
in Figure D.11 of Appendix D (e.g., the wheels of the car in row one and
three). Overall, we can conclude that local discriminators can reduce local
inconsistencies, which might arise from the robust but not flawless
segmentation maps used for masking.

Effectiveness of segmentation-based sampling. We compare our segmentation-
based sampling method with random sampling and sampling based on VGG features.
For the sampling strategy based on VGG features, we follow EPE [216] to
calculate scores for 352$\times$352 crops of the input images. Crops with a
similarity score higher than $0.5$ are selected for training. As shown in
Table 6.3, our segmentation-based sampling strategy (Full) slightly
outperforms the other sampling strategies in overall translation performance.

(a) 957$\times$526 Input

(b) 957$\times$526 Output

(c) 957$\times$526 Layer

(d) 478$\times$263 Layer

(e) 119$\times$65 Layer

(f) 59$\times$32 Layer

Figure 6.7: FATE attention maps. Results are selected from the best model.
Best viewed in color.

Effectiveness of FATE. For each spatial point ("pixel") in the input feature
map, our feature-attentive denormalization block selects the features in the
feature dimension to be incorporated into the output stream of the generator
by denormalization. We show the attention values of our feature-attentive
denormalization block in Figure 6.7 by visualizing all attention values for a
single feature across the entire feature map. Since a single feature
represents a property of the input, a spatial pattern should emerge. This is
expected especially in earlier layers, where the spatiality of the
convolutional model’s feature map is best preserved. As shown in Figure 6.7,
our attention mechanism learns to attend to features that correlate with a
property. Examples are the shadows of a scene (row 1), cars and their lighting
(row 2), and vegetation (row 3). In addition, we find increasingly more white
feature maps in deeper layers. This can be interpreted positively as an
indication that the learned content (source) features in deeper layers are
important for the translation task and that more shallow content features of
earlier layers are increasingly ignored. However, this can also be interpreted
negatively and could indicate that our simple attention mechanism is not able
to separate deeper features properly.

Comparing FATE to FADE, we find that FATE leads to a subtle increase in
training instability, resulting in slightly worse average performance over the
five runs per model. However, FATE also leads to our best models. Therefore,
we select the FATE block as the standard configuration for our model. The
deviation from the average values for all runs can be found in Table D.5 of
Appendix D. The slight increased instability suggests that the attention
mechanism of FATE can be further improved.

(a) Input

(b) 252$\times$252

(c) 352$\times$352

(d) 464$\times$464

(e) 512$\times$512

Figure 6.8: Qualitative ablation of crop sizes. For each crop size, results
are selected from the best model. Randomly sampled results can be found in
Figure D.10 of Appendix D. Best viewed in color. Figure 6.9: Quantitative
ablation of crop sizes.

Effect of global crop size. We successively increase the global crop size of
the generator and discriminators from 256$\times$256 to 512$\times$512 and
examine the effects on translation performance. As shown in Figure 6.6,
increasing the global crop size results in a better approximation of the
target domain style. However, increasing the global crop size also leads to an
increasing number of artifacts in the translated image. In Figure 6.9, we
report the score of various metrics with respect to the global crop size. The
commonly used metrics for measuring translation quality (IS, FID, and KID)
show that translation quality increases steadily up to a global crop size of
464$\times$464, after which the results become unstable. The cKVD metric also
shows an increase in average performance up to a crop size of 464$\times$464,
mainly because translation quality for the underrepresented person class
increases. This is intuitive since a larger crop size leads to a more frequent
appearance of underrepresented classes during training. Furthermore, the sKVD
metric shows a steady decline in consistency as the global crop size
increases. Therefore, we choose a tradeoff between approximation of the target
domain style, artifacts, and computational cost, and select 352$\times$352 as
the global crop size for our model.

(a) PFD$\rightarrow$Cityscapes

(b) Day$\rightarrow$Night

(c) Viper$\rightarrow$Cityscapes

(d) Clear$\rightarrow$Snowy

Figure 6.10: Additional qualitative results. Best viewed in color.

####  6.5 Conclusion

In this work, we have shown that content-based masking of the discriminator is
sufficient to significantly reduce content inconsistencies that arise in
unpaired image-to-image translation. Furthermore, artifacts caused by the
masking procedure can be significantly reduced by introducing a local
discriminator that utilizes a segmentation-based similarity sampling
technique. Moreover, our similarity sampling technique leads to a further
increase in performance when applied to global input crops. We have also shown
that our feature-based denormalization block is able to attend to specific
content features, such as features of shadows, but can slightly increase
training instability. In addition, we have proposed the cKVD metric to examine
translation quality at the class or category level. In our experiments, we
have found that these techniques lead to state-of-the-art performance on
photo-realistic sim-to-real transfer and the translation of weather. Although
our method performs well in Day$\rightarrow$Night translation, the remaining
limitations of our approach are especially evident in this task.

Limitations. We remark on limitations regarding the dataset, sampling, method,
and implementation. Probably the most significant limitations are the complex
public datasets currently available and in use, as they are not specifically
designed for unpaired translation. Collection strategies and datasets that
mitigate biases between source and target domains would be beneficial.
Furthermore, our sampling strategy only works on an image basis and could be
extended across the entire dataset to sample more significant pairs for
training. Although our method works for large crops, there is still a crop
size limit that must be taken into account when tuning the hyperparameters. In
addition, our method for mitigating content inconsistencies depends on the
segmentation model. In theory, the number of classes could be used to control
how fine-grained the content consistency should be, which leads to flexibility
but allows for errors depending on the segmentation quality. This can result
in artifacts such as glowing objects, as shown in Figure 6.11. Intra-class
inconsistencies that may arise from intra-class biases ignored by the loss,
such as small textures, represent another problem. Intra-class inconsistencies
are currently underexplored in unpaired image-to-image translation and are an
interesting direction for future research. Finally, we would like to point out
that the efficiency of our implementation could be further improved. Apart
from these limitations, our method achieves state-of-the-art performance in
complex translation tasks while mitigating inconsistencies through a masking
strategy that works by applying few tricks. Simple masking strategies have
proven to be very successful in other fields. Therefore, we believe that
masking strategies for unpaired image-to-image translation represent a
promising direction for further research.

Ethical and responsible use. Considering the limitations of current methods,
unpaired image-to-image translation methods should be trained and tested with
care, especially for safety-critical domains like autonomous driving. A major
concern is that it is often unclear or untested whether the transferred
content can still be considered consistent for subsequent tasks in the target
domain. Even though measures exist for content-consistent translation, they do
not allow for the explainability of what exactly is being transferred and
changed by the model on a fine-grained level. With our proposed cKVD metric we
contribute to this field by allowing class-specific translation measurements -
a direction that we hope is the right one. However, even if the content is
categorically consistent at a high (class) level, subcategories (like parts of
textures) may still be interchanged. At a lower level, content consistency and
style consistency are intertwined (e.g., a yellow stop sign). Another privacy
and security question is whether translation methods are (or will) be able to
(indirectly) project sensitive information from the target domain to the
translated images (e.g., exchange faces from simulation with faces of existing
persons during the translation). A controllable (class-level and in-class-
level) consistency method could help to resolve such issues.

(a) Glowing objects

(b) Intra-class inconsistency

(c) Minor hallucinations

(d) Class boundary artifiacts

Figure 6.11: Limitations. Best viewed in color.

## Clausula

### Conclusions

####  7.1 Conclusion and Discussion

This Ph.D. thesis addresses the field of unsupervised representation learning
from the perspectives of learning, evaluating, and transferring visual
representations. After an introduction to the field as a whole in chapter 1,
we have provided a high-level overview and an introduction of the subfields we
have studied and contributed to in sections 1.1 through 1.4. Subsequently, the
objectives and scope of this dissertation have been defined, along with
several research questions focused on the learning, evaluation, and transfer
of unsupervised visual representations. Following an outline of the thesis in
section 1.6, we have extensively discussed the related work with respect to
our contributions in chapter 2. In the proceeding chapters 3 to 6, we have
described our contributed backpropagation-free unsupervised representation
learning method, metrics for measuring objective function mismatches, CARLANE
Benchmark with a new unsupervised domain adaptation method, and content-
consistent unpaired image-to-image translation method. Each aforementioned
chapter consists of a description of the motivation for the proposed work, a
listing of the contributions with respect to the research questions, a method
description, the conducted experiments, and a conclusion. A summary of our
contributions is provided in the subsequent section 7.3.

Developments of unsupervised visual representation learning. In retrospect,
unsupervised learning of visual representations has made significant progress
in recent years. Therefore, we discuss our work as a small part of these
(positive) developments:

In unsupervised representation learning, performance gains for various visual
target tasks have been significant. Self-supervised pretext tasks have evolved
from data-indirect approaches such as rotation prediction towards more data-
direct prediction tasks such as contrastive learning, distillation, or feature
predictive distillation. Thus the trend towards the main paradigm of
unsupervised representation learning - reducing human intervention - shows
promising and steadily improving results. However, there is still
comparatively little focus on creating a multitude of unsupervised learning
rules apart from backpropagation. Nevertheless, several works - such as [186,
134] and ours [238] \- have contributed to the field of backpropagation-free
and/or biologically plausible (unsupervised) learning, leading to increasingly
deep models.

For the evaluation of unsupervised models, there has been an ongoing effort to
create novel methods for explaining the inner workings of unsupervised
representations and models. This has resulted in promising visualization
techniques and benchmarks. In recent years, the development of novel metrics
that do not rely on specific datasets and/or benchmarks has received little
attention but has not gone unnoticed either. Some works have led to several
insights into unsupervised learning setups by defining such new metrics, like
our metrics for measuring objective function mismatches [239], or by examining
existing metrics for unsupervised representation learning [93, 98].

For the transfer of unsupervised visual representations, unsupervised domain
adaptation is increasingly being applied to more complex tasks such as
semantic segmentation [52] or the CARLANE Benchmark [241]. However, with the
emergence of large models trained on vast amounts of cross-domain data, the
most basic approach to domain adaptation - simply training the model
concurrently on all these domains - may lead to surprising performance gains.
The problem of content consistency in the translation of unpaired images has
recently been recognized and addressed with less ill-posed GAN-based methods -
including [216, 251], and our own method [240] \- which have yielded promising
results. The evaluation of these methods in the context of content
inconsistency still lacks a metric that is widely adopted. Nevertheless, there
are ongoing efforts to define novel metrics, as can be seen in [216] or our
work [240].

Limitations of unsupervised visual representation learning. Now that we have
highlighted recent developments related to our work, we discuss some
insightful limitations that persist in these subfields. First, we would like
to emphasize limitations that apply to all of these subfields: In particular,
multimodality, availability of multimodal curated datasets, continuous
learning of models, the efficiency of models and training, and the
explainability of methods are still limited. Especially, the unavailability of
multidomain curated open-source datasets has been a relevant limitation with
regard to recent trends.

In backpropagation-free unsupervised representation learning, a major
limitation is the scalability of these methods to deeper models. In the case
of biologically plausible unsupervised learning, methods often rely on
longstanding neuroscientific insights without integrating the latest
neuroscientific findings. Furthermore, beyond the scientific aspects - such as
understanding the brain - it is still unclear whether biologically-plausible,
backpropagation-free approaches lead to significant practical benefits for
many tasks, as the trajectory of backpropagation-based learning still leads to
performance gains. As for the current state of the art of unsupervised
representation learning, there is still a performance gap compared to
supervised learning. However, this gap is continually closing. Although data-
indirect objectives with specific signals, such as rotation prediction, have
been succeeded by novel data-direct objectives, such as feature-predictive
distillation, these data-direct objectives do not account for the possible
plausible variations of masked features. Often there could be more than one
plausible solution for the predictive features of a masked region, which is
not taken into account by the utilized objectives. For large vision models
trained on carefully curated data, limitations lie in their trajectory: For
example, the utilized datasets do not cover all vision domains (or tasks), and
the current combination of models and training objectives requires additional
regularization to stabilize training. Furthermore, there is currently no
insight into whether these models exploit in-dataset statistics of the
datasets from which the curated dataset is built.

For the evaluation of unsupervised models, a representation is mostly measured
by transferring models to target tasks and by visualizing specific properties.
This is the only widely adopted consensus that defines good representations.
Other empirical measurements are barely adopted and underexplored. For
example, the robustness analysis of these models is limited to a few works and
lacks further studies. Moreover, to date, no measurement can reliably predict
the performance of models on a variety of target tasks.

For unsupervised domain adaptation, there is a lack of large multi-domain
datasets for complex tasks that strongly indicate the direct usability of
methods in the real word. Furthermore, stability, especially in the temporal
dimension of the adapted models, has not yet been sufficiently explored.
Another limitation is that only longstanding, but no recent unsupervised
auxiliary tasks have been explored in the context of unsupervised domain
adaptation.

In unpaired image-to-image translation, the unavailability of suitable bias-
reduced datasets is a major limitation that can have a strong impact on
performance. Given the benefits of real-time, high-resolution inference, there
is an especially strong need to improve the efficiency and memory utilization
of these models. Furthermore, a major concern is that it is often not clear
what exactly is being transferred and changed by the model and whether the
transferred content is still consistent. Even if the content is categorically
consistent at a high level, subcategories (like parts of textures) may still
be interchanged. At a lower level, content consistency and style consistency
are intertwined (e.g., a yellow stop sign).

Ethics of unsupervised visual representation learning. With the steady
integration of machine learning approaches into the society, ethical
considerations regarding these approaches are becoming ever more necessary.
These considerations apply especially to backbone models that are or will be
used for a multitude of tasks. As unsupervised representation learning methods
rapidly improve and the first large models trained on curated data from
multiple domains show promising results [198], analysis for fairness and
biases, such as analysis for geographic-, gender-, skintone-, and age-based
potentially harmful label associations, are rapidly gaining importance. In
addition, the environmental impact of such large models should be taken into
account. Here we believe that an open-source culture with releases of
datasets, implementations, and pre-trained models can reduce the environmental
impact since environmentally expensive model pretraining and dataset curation
are shared across the community. As models trained on vast curated datasets
containing billions of images proliferate, another increasingly relevant
ethical consideration is the potential encoding of personal information in
pre-trained models that could be indirectly obtained from their
representations of those models. The aforementioned considerations also apply
to the transfer of representation into other domains.

####  7.2 Future Perspectives on Unsupervised Representation Learning

Based on the discussed developments, limitations, and ethics in section 7.1,
current trends in unsupervised visual representation learning, and the
paradigm shift in natural language processing, we want to provide future
perspectives for unsupervised representation learning. In particular, large
unsupervised vision models, stably-trained on curated data, appear to be the
future of unsupervised representation learning. These models could encompass
or at least assist all the subfields discussed in this dissertation as well as
others. Carefully curated data and stable, scalable training objectives may be
critical to the development of these models. Going a step further, continual
unsupervised learning of carefully monitored backbones from curated data
streams could be a possible future. From an ethical perspective, aligning
these backbones with the right intent for the target task may become relevant,
which brings various scientific challenges, legal questions, and ethical
considerations. In addition, multimodal integration or training beyond the
vision domain could lead to more powerful models. A more specific future
perspective is the development of training objectives (based on
discriminators) that integrate plausible variability in masked feature
prediction. Another specific perspective is to replace the prediction head of
latent-conditioned joint-embedding predictive architectures with GAN-based or
diffusion-based models, which seems natural due to the latent conditioning.
These specific perspectives could complement the development of large vision
models trained on curated data. Future perspectives regarding backpropagation-
free unsupervised learning include continued examinations and developments
regarding the scalability of these models. Of particular interest appears to
be the successful integration of skip connections and the incorporation of the
advantages of image transformers into these models. For biologically plausible
unsupervised representational learning, the integration of the latest
neuroscience findings, such as new neuron formulations, in combination with
scaling efforts is promising.

For the evaluation of unsupervised visual representation learning, in addition
to continuing the current directions, a relevant direction is to further
investigate the temporal dimension of the training setup and its effects on
representations and models. This could also assist the process of creating
large vision models. Furthermore, with the rise of curated datasets, metrics
to prevent the model from exploiting in-database statistics (of the datasets
from which the curated dataset is built) may be relevant. For evaluations
regarding unpaired image-to-image translation, novel metrics to measure
content consistency and style consistency correctly on multiple consistency
levels could increase the trust in these methods.

For the unsupervised adaptation of visual representations, it is plausible
that the advent of large vision models, trained on curated data, could set a
new hard-to-beat state of the art and finally combine unsupervised
representation learning with unsupervised domain adaptation. However,
utilizing adaptation techniques for the alignment of large vision models
appears to be an interesting future research direction. For unpaired image-to-
image translation, adopting and investigating unpaired diffusion models and
large, carefully pre-trained (text-to-image) generation models in the context
of content consistency and style consistency is intriguing. A further
appealing direction could be the integration of large vision models into this
setup, for example, as encoders for an unpaired latent-to-latent translation.

####  7.3 Summary of Contributions

In this dissertation, we contribute to the field of unsupervised
representation learning by developing and evaluating methods to learn, adapt
and translate visual representations. We have designed a convolutional,
backpropagation-free method for unsupervised visual representation learning
with modules that can be stacked deeper than previous backpropagation-free
baselines. During the evaluation of our backpropagation-free models, we have
encountered objective function mismatches between our unsupervised pretext
task and the evaluated target tasks. This has led to the design of metrics for
measuring objective function mismatches between unsupervised pretext models
and (supervised) target models. With these metrics, we have investigated
several well-known unsupervised representation learning approaches and target
tasks and have found dependencies of the objective function mismatch with
respect to different aspects of the training setup and model. Furthermore, we
have highlighted the impact of the objective function mismatch on target task
performance.

A parallel line of research on an autonomous model vehicle has led to the
CARLANE Benchmark, the first 3-way sim-to-real domain adaptation benchmark for
2D lane detection. To achieve baseline performances for this benchmark, we
have combined well-known unsupervised domain adaptation methods with the
ultra-fast lane detection method and have performed evaluations on single-
source single-target and single-source multi-target unsupervised domain
adaptation. In addition, we have constructed a new unsupervised domain
adaptation method, which has reached state-of-the-art performance. We then
have focused on unpaired, one-sided domain adaptation of images and have
constructed a content-consistent unpaired image-to-image translation method.
Our method utilizes masked global and local discriminators to mitigate content
inconsistencies during the translation of images between biased datasets,
which are a common problem in the ill-posed setups of unpaired image-to-image
translation. In addition, we have proposed feature-attentive denormalization
to selectively fuse feature statistics into the generator stream, which allows
the model to ignore certain statistics of the source domain in different
layers. For the quantitative comparison, we have used well-known perceptual
metrics and have designed a new metric that incorporates content inconsistency
based on segmentation maps. Both quantitatively and qualitatively, our method
has outperformed the state of the art on the evaluated translation tasks.