# CARLANE: A Lane Detection Benchmark for Unsupervised Domain Adaptation from
Simulation to multiple Real-World Domains

Julian Gebele  ,1 Bonifaz Stuhr11footnotemark: 1 ,1,2 Johann
Haselberger11footnotemark: 1 ,1,3
1University of Applied Science Kempten
2Autonomous University of Barcelona
3Technische Universität Berlin
<EMAIL_ADDRESS>Equal contribution

###### Abstract

Unsupervised Domain Adaptation demonstrates great potential to mitigate domain
shifts by transferring models from labeled source domains to unlabeled target
domains. While Unsupervised Domain Adaptation has been applied to a wide
variety of complex vision tasks, only few works focus on lane detection for
autonomous driving. This can be attributed to the lack of publicly available
datasets. To facilitate research in these directions, we propose CARLANE, a
3-way sim-to-real domain adaptation benchmark for 2D lane detection. CARLANE
encompasses the single-target datasets MoLane and TuLane and the multi-target
dataset MuLane. These datasets are built from three different domains, which
cover diverse scenes and contain a total of 163K unique images, 118K of which
are annotated. In addition we evaluate and report systematic baselines,
including our own method, which builds upon Prototypical Cross-domain Self-
supervised Learning. We find that false positive and false negative rates of
the evaluated domain adaptation methods are high compared to those of fully
supervised baselines. This affirms the need for benchmarks such as CARLANE to
further strengthen research in Unsupervised Domain Adaptation for lane
detection. CARLANE, all evaluated models and the corresponding implementations
are publicly available at https://carlanebenchmark.github.io.

##  1 Introduction

Vision-based deep learning systems for autonomous driving have made
significant progress in the past years [1, 2, 3, 4, 5]. Recent state-of-the-
art methods achieve remarkable results on public, real-world benchmarks but
require labeled, large-scale datasets. Annotations for these datasets are
often hard to acquire, mainly due to the high expenses of labeling in terms of
cost, time, and difficulty. Instead, simulation environments for autonomous
driving, such as CARLA [6], can be utilized to generate abundant labeled
images automatically. However, models trained on data from simulation often
experience a significant performance drop in a different domain, i.e., the
real world, mainly due to the domain shift [7]. Unsupervised Domain Adaptation
(UDA) methods [8, 9, 10, 11, 12, 13, 14, 15] try to mitigate the domain shift
by transferring models from a fully-labeled source domain to an unlabeled
target domain. This eliminates the need for annotating images but assumes that
the target domain is accessible at training time. While UDA has been applied
to complex tasks for autonomous driving such as object detection [1, 16] and
semantic segmentation [17, 18], only few works focus on lane detection [19,
5]. This can be attributed to the lack of public UDA datasets for lane
detection.

| MoLane | TuLane
---|---|---
Source |  |  |  |
Target |  |  |  |
Figure 1: Images sampled from our CARLANE Benchmark.

To compensate for this data scarcity and encourage future research, we
introduce CARLANE, a sim-to-real domain adaptation benchmark for lane
detection. We use the CARLA simulator for data collection in the source domain
with a free-roaming waypoint-based agent and data from two distinct real-world
domains as target domains. This enables us to construct a benchmark that
consists of three datasets:

(1) MoLane focuses on abstract lane markings in the domain of a 1/8th Model
vehicle. We collect 80K labeled images from simulation as the source domain
and 44K unlabeled real-world images from several tracks with two lane markings
as the target domain. Further, we apply domain randomization as well as data
balancing. For evaluation, we annotate 2,000 validation and 1,000 test images
with our labeling tool.

(2) TuLane incorporates 24K balanced and domain-randomized images from
simulation as the source domain and the well-known TuSimple [20] dataset with
3,268 real-world images from U.S. highways with up to four labeled lanes as
the target domain. The target domain of MoLane is a real-world abstraction
from the target domain of TuLane, which may result in interesting insights
about UDA.

(3) MuLane is a balanced combination of MoLane and TuLane with two target
domains. For the source domain, we randomly sample 24K images from MoLane and
combine them with TuLane’s synthetic images. For the target domains, we
randomly sample 3,268 images from MoLane and combine them with TuSimple. This
allows us to investigate multi-target UDA for lane detection.

To establish baselines and investigate UDA on our benchmark, we evaluate
several adversarial discriminative methods, such as DANN [12], ADDA [13] and
SGADA [21]. Additionally, we propose SGPCS, which builds upon PCS [22] with a
pseudo labeling approach to achieve state-of-the-art performance.

Our contributions are three-fold: (1) We introduce CARLANE, a 3-way sim-to-
real benchmark, allowing single- and multi-target UDA. (2) We provide several
dataset tools, i.e., an agent to collect images with lane annotations in CARLA
and a labeling tool to annotate the real-world images manually. (3) We
evaluate several well-known UDA methods to establish baselines and discuss
results on both single- and multi-target UDA. To the best of our knowledge, we
are the first to adapt a lane detection model from simulation to multiple
real-world domains.

##  2 Related Work

###  2.1 Data Generation for Sim-to-Real Lane Detection

In recent years, much attention has been paid to lane detection benchmarks in
the real world, such as CULane [3], TuSimple [20], LLAMAS [23], and BDD100K
[24]. Despite the popularity of these benchmarks, there is few research that
focuses on sim-to-real lane detection datasets. Garnett et al. [4] propose a
method for generating synthetic images with 3D lane annotations in the open-
source engine blender. Their synthetic-3D-lanes dataset contains 300K train,
1,000 validation and 5,000 test images, while their real-world 3D-lanes
dataset consists of 85K images, which are annotated in a semi-manual manner.
Utilizing the data generation method from [4], Garnett et al. [19] collect 50K
labeled synthetic images to perform sim-to-real domain adaptation for 3D lane
detection. At this point, the source domain of the dataset is not publicly
available.

Recently, Hu et al. [5] investigated UDA techniques for 2D lane detection.
Their proposed data generation method relies on CARLA’s built-in agent to
automatically collect 16K synthetic images. However, the dataset is not
publicly available at this point. In comparison, our method leverages an
efficient and configurable waypoint-based agent. Furthermore, in contrast to
the aforementioned works, considering only single-source single-target UDA, we
additionally focus on multi-target UDA.

###  2.2 Unsupervised Domain Adaptation

Unsupervised Domain Adaptation has been extensively studied in recent years
[9]. In an early work, Ganin et al. [8] propose a gradient reversal layer
between the features extractor and a domain classifier to learn similar
feature distributions for distinct domains. Early discrepancy-based methods
employ a distance metric to measure the discrepancy of the source and target
domain [10, 25]. A prominent example is DAN [10] which uses maximum mean
discrepancies (MMD) [26, 27] to match embeddings of different domain
distributions. Recently, DSAN [11] builds upon DAN with local MMD and exploits
fine-grained features to align subdomains accurately.

Domain alignment can also be achieved through adversarial learning [28].
Adversarial discriminative methods such as DANN [12] or ADDA [13] employ a
domain classifier or discriminator, encouraging the feature extractor to
produce domain-invariant representations. While these methods mainly rely on
feature-level alignment, adversarial generative methods [29, 30] operate on
pixel-level.

In a recent trend, self-supervised learning methods are leveraged as auxiliary
tasks to improve domain adaptation effectiveness and to capture in-domain
semantic structures [14, 15, 31, 22, 32]. Furthermore, self-supervised
learning is utilized for cross-domain alignment as well, by matching class-
discriminative features [31, 33], task-discriminative features [34], class
prototypes [35, 22] or equivalent samples in the domains [36].

Furthermore, other recent works mitigate optimization inconsistencies by
minimizing the gradients discrepancy of the source samples and target samples
[37] or by applying a meta-learning scheme between the domain alignment and
the targeted classification task [38].

##  3 Data Generation

To construct our benchmark, we gather image data from a real 1/8th model
vehicle, and the CARLA simulator [6]. Ensuring the verification of results and
transferability to real driving scenarios, we extend our benchmark with the
TuSimple dataset [20]. This enables gradual testing, starting from simulation,
followed by model cars, and ending with full-scale real word experiments. Data
variety is achieved through domain randomization in all domains. However,
naively performing domain randomization might lead to an imbalanced dataset.
Therefore, similar driving scenarios are sampled across all domains, and a
bagging approach is utilized to uniformly collect lanes by their curvature
with respect to the camera position. We strictly follow TuSimple’s data format
[20] to maintain consistency across all our datasets.

###  3.1 Real-World Environment

As shown in Figure 2, we build six different 1/8th race tracks, where each
track is available in two different surface materials (dark and light gray).
We vary between dotted and solid lane markings, which are pure white and
$50\text{\,}\mathrm{mm}$ thick. The lanes are constantly
$750\text{\,}\mathrm{mm}$ wide, and the smallest inner radius is
$250\text{\,}\mathrm{mm}$. The track layouts are designed to roughly contain
the same proportion of straight and curved segments to obtain a balanced label
distribution. We construct these tracks in four locations with alternating
backgrounds and lighting conditions.

(a)

(b)  (c)

(d)

(e)

(f)

Figure 2: Overview of our track types for MoLane. (a) - (d) show the black
version of the training and validation tracks. These tracks are also
constructed using a light gray surface material. (e) and (f) depict our test
tracks.

###  3.2 Real-World Data Collection

Raw image data is recorded from a front-facing Stereolabs ZEDM camera with 30
FPS and a resolution of $1280\times 720$ pixels. A detailed description of the
1/8th car can be found in the Appendix. The vehicle is moved with a quasi-
constant velocity clockwise and counter-clockwise to cover both directions of
each track. All collected images from tracks (e) and (f) are used for the test
subset. In addition, we annotate lane markings with our labeling tool for
validation and testing, which is made publicly available.

###  3.3 Simulation Environment

We utilize the built-in APIs from CARLA to randomize multiple aspects of the
agent and environment, such as weather, daytime, ego vehicle position, camera
position, distractor vehicles, and world objects (i.a., walls, buildings, and
plants). Weather and daytime are varied systematically by adapting parameters
for cloud density, rain intensity, puddles, wetness, wind strength, fog
density, sun azimuth, and sun altitude. For further details, we refer to our
implementation. To occlude the lanes similar to real-world scenarios, up to
five neighbor vehicles are spawned randomly in the vicinity of the agent. We
consider five different CARLA maps in urban and highway environments (Town03,
Town04, Town05, Town06, and Town10) to collect our dataset, as the other
towns’ characteristics are not suitable for our task (i.a., mostly straight
lanes). In addition, we collect data from the same towns without world objects
to strengthen the focus on lane detection, similar to our model vehicle target
domain.

###  3.4 Simulation Data Agent

We implement an efficient agent based on waypoint navigation, which roams
randomly and reliably in the aforementioned map environments and collects
$1280\times 720$ images. In each step, the waypoint navigation stochastically
traverses the CARLA road map with a fixed lookahead distance of one meter. In
addition, we sample offset values $\Delta y_{k}$ from the center lane within
the range $\pm 1.20\text{\,}\mathrm{m}$. To avoid saturation at the lane
borders, which would occur with a sinusoidal function, we use the triangle
wave function:

$\Delta y_{k}=\frac{2m}{\pi}\arcsin(\sin(i_{k}))$ (1)

where $m$ is the maximal offset and $i_{k}$ is incremented by $0.08$ for each
simulation step $k$. Per frame, our agent moves to the next waypoint with an
increment of one meter, enabling the collection of highly diverse data in a
fast manner. We use a bagging approach for balancing, which allows us to
define lane classes based on their curvature.

##  4 The CARLANE Benchmark

The CARLANE Benchmark consists of three distinct sim-to-real datasets, which
we build from our three different domains. The details of the individual
subsets can be found in Table 1.

MoLane consists of images from CARLA and the real 1/8th model vehicle. For the
abstract real-world domain, we collect 46,843 images with our model vehicle,
of which 2,000 validation and 1,000 test images are labeled. For the source
domain, we use our simulation agent to gather 84,000 labeled images. To match
the label distributions between both domains, we define five lane classes
based on the relative angle $\beta$ of the agent to the center lane for our
bagging approach: strong left curve ($\beta\leq$$None$), soft left curve
($None$ $<\beta\leq$ $None$), straight ($None$ $<\beta<$ $None$), soft right
curve ($None$ $\leq\beta<$ $None$) and strong right curve ($None$$\leq\beta$).
In total, MoLane encompasses 130,843 images.

TuLane consists of images from CARLA, and a cleaned version of the TuSimple
dataset [20], which is licensed under the Apache License, Version 2.0. To
clean test set annotations, we utilize our labeling tool to ensure that the up
to four lanes closest to the car are correctly labeled. We adapt the bagging
classes to align the source dataset with TuSimple’s lane distribution: left
curve ($None$ $<\beta\leq$ $None$), straight ($None$ $<\beta<$ $None$) and
right curve ($None$ $\leq\beta<$ $None$).

MuLane is a multi-target UDA dataset and is a balanced mixture of images from
MoLane and TuLane. For MuLane’s entire training set and its source domain
validation and test set, we use all available images from TuLane and sample
the same amount of images from MoLane. We adopt the 1,000 test images from
MoLane’s target domain and sample 1,000 test images from TuSimple to form
MuLane’s test set. For the validation set, we use the 2,000 validation images
from MoLane and 2,000 of the remaining validation and test images of TuLane’s
target domain. In total, MuLane consists of 65,336 images.

To further analyze CARLANE, we visualize the ground truth lane distributions
in Figure 3. We observe that the lane distributions of source and target data
from our datasets are well aligned.

MoLane, TuLane, and MuLane are publicly available at
https://carlanebenchmark.github.io and licensed under the Apache License,
Version 2.0.

Table 1: Dataset overview. Unlabeled images denoted by *, partially labeled images denoted by **  Dataset | domain | total images | train | validation | test | lanes
---|---|---|---|---|---|---
MoLane | CARLA simulation | 84,000 | 80,000 | 4,000 | - |  $\leq$ 2
model vehicle | 46,843 | 43,843* | 2,000 | 1,000 |  $\leq$ 2
TuLane | CARLA simulation | 26,400 | 24,000 | 2,400 | - |  $\leq$ 4
TuSimple [20] | 6,408 | 3,268 | 358 | 2,782 |  $\leq$ 4
MuLane | CARLA simulation | 52,800 | 48,000 | 4,800 | - |  $\leq$ 4
model vehicle + TuSimple [20] | 12,536 | 6,536** | 4,000 | 2,000 |  $\leq$ 4

###  4.1 Dataset Format

For each dataset, we split training, validation, and test samples into source
and target subsets. Lane annotations are stored within a _.json_ file
containing the lanes’ y-values discretized by raw anchors, the lanes’
x-values, and the image file path following the data format of TuSimple[20].
Additionally, we adopt the method from [2] to generate _.png_ lane
segmentations and a _.txt_ file containing the linkage between the raw images
and their segmentation as well as the presence and absence of a lane.

###  4.2 Dataset Tasks

The main task of our datasets is UDA for lane detection, where the goal is to
predict lane annotations $Y_{t}\in\mathbb{R}^{R\times G\times N}$ given the
input image $X_{t}\in\mathbb{R}^{H\times W\times 3}$ from the unlabeled target
domain $\mathcal{D}_{\mathcal{T}}=\\{(X_{t})\\}_{t\in\mathcal{T}}$. $R$
defines the number of row anchors, $G$ the number of griding cells, and $N$
the number of lane annotations available in the dataset, where the definition
of $Y_{t}$ follows [20]. During training time, the images
$X_{s}\in\mathbb{R}^{H\times W\times 3}$, corresponding labels
$Y_{s}\in\mathbb{R}^{H\times W\times C}$ from the source domain
$\mathcal{D}_{\mathcal{S}}=\\{(X_{s},Y_{s})\\}_{s\in\mathcal{S}}$, and the
unlabeled target images $X_{t}$ are available. Additionally, MuLane focuses on
multi-target UDA, where
$\mathcal{D}_{\mathcal{T}}=\\{(X_{t_{1}})\cup(X_{t_{2}})\\}_{t_{1}\in\mathcal{T}_{1},t_{2}\in\mathcal{T}_{2}}$.

Although we focus on sim-to-real UDA, our datasets can be used for
unsupervised and semi-supervised tasks and partially for supervised learning
tasks. Furthermore, a real-to-real transfer can be performed between the
target domains of our datasets.

| MoLane | TuLane | MuLane
---|---|---|---
Source |  |  |
Target |  |  |
Figure 3: Lane annotation distributions of the three subsets of CARLANE. Since
the real-world training data of MoLane and MuLane is unlabeled, we utilize
their validation data for visualization.

##  5 Benchmark Experiments

We conduct experiments on our CARLANE Benchmark for several UDA methods from
the literature and our proposed method. Additionally, we train fully
supervised baselines on all domains.

###  5.1 Metrics

For evaluation, we use the following metrics:

(1) Lane Accuracy (LA) [2] is defined by $\textrm{LA}=\frac{p_{c}}{p_{y}}$,
where $p_{c}$ is the number of correctly predicted lane points and $p_{y}$ is
the number of ground truth lane points. Lane points are considered as correct
if their $L_{1}$ distance is smaller than the given threshold
$t_{pc}=\frac{20}{\cos(a_{yl})}$, where $a_{yl}$ is the angle of the
corresponding ground truth lane.

(2) False Positives (FP) and False Negatives (FN) [2]: To further determine
the error rate and to draw more emphasis on mispredicted or missing lanes, we
measure false positives with $\textrm{FP}=\frac{l_{f}}{l_{p}}$ and false
negatives with $\textrm{FN}=\frac{l_{m}}{l_{y}}$, where $l_{f}$ is the number
of mispredicted lanes, $l_{p}$ is the number of predicted lanes, $l_{m}$ is
the number of missing lanes and $l_{y}$ is the number of ground truth lanes.
Following [2], we classify lanes as mispredicted, if the $LA<85\%$.

###  5.2 Baselines

We use Ultra Fast Structure-aware Deep Lane Detection (UFLD) [2] as baseline
and strictly adopt its training scheme and hyperparameters. UFLD treats lane
detection as a row-based classification problem and utilizes the row anchors
defined by TuSimple [20]. To achieve a lower bound for the evaluated UDA
methods, we train UFLD as a supervised baseline on the source simulation data
(UFLD-SO). Furthermore, we train our baseline on the labeled real-world
training data for a surpassable fully-supervised performance in the target
domain (UFLD-TO). Since the training images from MoLane and MuLane have no
annotations, we train UFLD-TO in these cases on the labeled validation images
and validate our model on the entire test set.

###  5.3 Compared UDA Methods

We evaluate the following feature-level UDA methods on the CARLANE Benchmark
by adopting their default hyperparameters and tuning them accordingly. Each
model is initialized with the pre-trained feature encoder of our baseline
model (UFLD-SO). The optimized hyperparameters can be found in Table 2.

(1) DANN [12] is an adversarial discriminative method that utilizes a shared
feature encoder and a dense domain classifier connected via a gradient
reversal layer.

(2) ADDA [13] employs a feature encoder for each domain and a dense domain
discriminator. Following ADDA, we freeze the weights of the pre-trained
classifier of UFLD-SO to obtain final predictions.

(3) SGADA [21] builds upon ADDA and utilizes its predictions as pseudo labels
for the target training images. Since UFLD treats lane detection as a row-
based classification problem, we reformulate the pseudo label selection
mechanism. For each lane, we select the highest confidence value from the
griding cells of each row anchor. Based on their griding cell position, the
confidence values are divided into two cases: absent lane points and present
lane points. Thereby, the last griding cell represents absent lane points as
in [2]. For each case, we calculate the mean confidence over the corresponding
lanes. We then use the thresholds defined by SGADA to decide whether the
prediction is treated as a pseudo label.

(4) SGPCS (ours) builds upon PCS [22] and performs in-domain contrastive
learning and cross-domain self-supervised learning via cluster prototypes. Our
overall objective function comprises the in-domain and cross-domain loss from
PCS, the losses defined by UFLD, and our adopted pseudo loss from SGADA. We
adjust the momentum for memory bank feature updates to $0.5$ and use spherical
K-means [39] with $K=2,500$ to cluster them into prototypes.

Table 2: Optimized hyperparameters to achieve the reported results. $C$ denotes domain classifier parameters, $D$ denotes domain discriminator parameters, adv the adversarial loss from [13] and cls the classifier loss, sim the similarity loss and aux the auxiliary loss from [2]. Loss weights are set to $1.0$ unless stated otherwise. Method | Initial Learning Rate | Scheduler | Batch Size | Epochs | Losses | Other Changes
---|---|---|---|---|---|---
UFLD-SO | $4e^{-4}$ | Cosine Annealing | 4 | 150 | cls, sim, aux | -
DANN |  $1e^{-5}$, $C$: $1e^{-3}$ | $\frac{1e^{-5}}{(1+10p)^{0.75}}$ | 4 | 30 | cls, sim, aux, adv [12] |  $C$: 3 fc layers (1024-1024-2)
ADDA |  $1e^{-6}$, $D$: $1e^{-3}$ | Constant | 16 | 30 | map [13], adv [13] |  $D$: 3 fc layers (500-500-2)
SGADA |  $1e^{-6}$, $D$: $1e^{-3}$ | Constant | 16 | 15 | map [13], adv [13], pseudo: 0.25 | Pseudo label selection
SGPCS | $4e^{-4}$ | Cosine Annealing | 16 | 10 | in-domain [22], cross-domain [22] | -
cls, sim, aux, pseudo: 0.25
UFLD-TO | $4e^{-4}$ | Cosine Annealing | 4 | 300 | cls, sim, aux | -

###  5.4 Implementation Details

We implement all methods in PyTorch 1.8.1 and train them on a single machine
with four RTX 2080 Ti GPUs. Tuning all methods took a total amount of compute
of approximately 3.5 petaflop/s-days. The training times for each model range
from 4-13 days for UFLD baselines and 6-44 hours for domain adaption methods.
In addition, we found that applying output scaling on the last linear layer of
the model yields slightly better results. Therefore, we divide the models’
output by 0.5. Our implementation is publicly available at
https://carlanebenchmark.github.io.

###  5.5 Evaluation

Quantitative Evaluation. In Table 3 we report the results on MoLane, TuLane,
and MuLane across five different runs. We observe that UFLD-SO is able to
generalize to a certain extent to the target domain. This is mainly due to the
alignment of semantic structure from both domains. ADDA, SGADA, and our
proposed SGPCS manage to adapt the model to the target domain slightly and
consistently. However, DANN suffers from negative transfer [40] when trained
on MoLane and MuLane. The negative transfer of DANN for complex domain
adaptation tasks is also observed in other works [35, 41, 40, 42] and can be
explained by the source domain’s data distribution and the model complexity
[40]. In our case, the source domain contains labels not present in the target
domain, as shown in Figure 3, which is more pronounced in MoLane and MuLane.

We want to emphasize that with an accuracy gain of a maximum of 4.55% (SGPCS)
and high false positive and false negative rates, the domain adaptation
methods are not able to achieve comparable results to the supervised baselines
(UFLD-TO). Furthermore, we observe that false positive and false negative
rates increase significantly on MuLane, indicating that the multi-target
dataset forms the most challenging task. False positives and false negatives
represent wrongly detected and missing lanes which can lead to crucial impacts
on autonomous driving functions. These results affirm the need for the
proposed CARLANE Benchmark to further strengthen the research in UDA for lane
detection.

Table 3: Performance on the test set. Lane accuracy (LA), false positives (FP), and false negatives (FN) are reported in %. ResNet-18 | MoLane | TuLane | MuLane
---|---|---|---
LA | FP & FN | LA | FP | FN | LA | FP | FN
UFLD-SO | 89.39 | 25.25 | 87.43 | 34.21 | 23.48 | 88.02 | 50.24 | 26.08
DANN [12] | 87.65$\pm$0.48 | 29.97$\pm$1.21 | 88.74$\pm$0.32 | 32.71$\pm$0.52 | 21.64$\pm$0.65 | 86.01$\pm$0.67 | 55.33$\pm$1.22 | 36.30$\pm$1.90
ADDA [13] | 92.85$\pm$0.17 | 10.61$\pm$0.77 | 90.72$\pm$0.15 | 29.73$\pm$0.36 | 17.67$\pm$0.42 | 89.83$\pm$0.33 | 46.79$\pm$0.43 | 20.57$\pm$0.63
SGADA [21] | 93.82$\pm$0.10 |  7.13$\pm$0.22 |  91.70$\pm$0.13 |  28.42$\pm$0.34 |  16.10$\pm$0.43 | 90.71$\pm$0.10 |  45.13$\pm$0.32 |  17.26$\pm$0.36
SGPCS (ours) |  93.94$\pm$0.04 | 7.16$\pm$0.16 | 91.55$\pm$0.13 | 28.52$\pm$0.21 | 16.16$\pm$0.26 |  91.57$\pm$0.22 | 45.49$\pm$0.63 | 17.39$\pm$0.88
UFLD-TO | 97.35 | 0.50 | 94.97 | 18.05 | 3.84 | 96.57 | 34.06 | 2.49
ResNet-34 | LA | FP & FN | LA | FP | FN | LA | FP | FN
UFLD-SO | 90.35 | 22.25 | 89.42 | 32.35 | 21.19 | 89.17 | 48.86 | 23.67
DANN [12] | 90.91$\pm$0.42 | 19.73$\pm$1.51 | 91.06$\pm$0.14 | 30.17$\pm$0.20 | 18.54$\pm$0.25 | 88.76$\pm$0.22 | 48.93$\pm$0.47 | 24.16$\pm$0.89
ADDA [13] | 92.39$\pm$0.26 | 12.17$\pm$0.84 | 91.39$\pm$0.16 | 28.76$\pm$0.30 | 16.63$\pm$0.36 | 90.22$\pm$0.39 | 45.84$\pm$0.54 | 19.49$\pm$0.90
SGADA [21] | 93.31$\pm$0.10 | 9.41$\pm$0.16 | 92.04$\pm$0.09 | 28.18$\pm$0.20 | 15.99$\pm$0.24 |  91.63$\pm$0.03 |  44.18$\pm$0.12 |  16.23$\pm$0.16
SGPCS (ours) |  93.53$\pm$0.25 |  8.24$\pm$0.91 |  93.29$\pm$0.18 |  25.68$\pm$0.48 |  12.73$\pm$0.59 | 91.55$\pm$0.17 | 44.75$\pm$0.28 | 16.41$\pm$0.44
UFLD-TO | 97.21 | 0.30 | 94.43 | 20.74 | 7.20 | 96.54 | 33.76 | 2.03
UFLD-SO | DANN | ADDA | SGADA | SGPCS
---|---|---|---|---
|  |  |  |
Figure 4: t-SNE visualization of MuLane dataset. The source domain is marked
in blue, the real-world model vehicle target domain in red, and the TuSimple
domain in green.

Qualitative Evaluation. We use t-SNE [43] to visualize the features of the
features encoders for the source and target domains of MuLane in Figure 4.
t-SNE visualizations of MoLane and TuLane can be found in the Appendix. In
accordance with the quantitative results, we observe only a slight adaptation
of the source and target domains features for ADDA, SGADA, and SGPCS compared
to the supervised baseline UFLD-SO. Consequently, the examined well-known
domain adaptation methods have no significant effect on feature alignment. In
addition, we show results from the evaluated methods in Figure 5 and observe
that the models are able to predict target domain lane annotations in many
cases but are not able to achieve comparable results to the supervised
baseline (UFLD-TO).

In summary, we find quantitatively and qualitatively that the examined domain
adaptation methods do not significantly improve the performance of lane
detection and feature adaptation. For this reason, we believe that the
proposed benchmark could facilitate the exploration of new domain adaptation
methods to overcome these problems.

| MoLane | TuLane | MuLane
---|---|---|---
UFLD-SO |  |  |  |
DANN |  |  |  |
ADDA |  |  |  |
SGADA |  |  |  |
SGPCS |  |  |  |
UFLD-TO |  |  |  |
Figure 5: Qualitative results of target domain predictions. Ground truth lane
annotations are marked in blue, predictions in red.

##  6 Conclusion

We present CARLANE, the first UDA benchmark for lane detection. CARLANE was
recorded in three domains and consists of three datasets: the single-target
datasets MoLane and TuLane and the multi-target dataset MuLane, which is a
balanced combination of both. Based on the UFLD model, we conducted
experiments with different UDA methods on CARLANE and found that the selected
methods are able to adapt the model to target domains slightly and
consistently. However, none of the methods achieve comparable results to the
supervised baselines. The most significant performance differences are
noticeable in the high false positive and false negative rates of the UDA
methods compared to the target-only baselines, which is even more pronounced
in the MuLane multi-target task. These false-positive and false-negative rates
can negatively impact autonomous driving functions since they represent
misidentified and missing lanes. Furthermore, as shown in the t-SNE plots of
Figure 4, the examined well-known domain adaptation methods have no
significant effect on feature alignment. The current difficulties of the
examined UDA methods to adequately align the source and target domains confirm
the need for the proposed CARLANE benchmark. We believe that CARLANE eases the
development and comparison of UDA methods for lane detection. In addition, we
open-source all tools for dataset creation and labeling and hope that CARLANE
facilitates future research in these directions.

Limitations. One limitation of our work is that we only use a fixed set of
track elements within our 1/8th scaled environment. These track elements
represent only a limited number of distinct curve radii. Furthermore, neither
buildings nor traffic signs exist in MoLane’s model vehicle target domain.
Moreover, the full-scale real-world target domain of TuLane is derived from
TuSimple. TuSimple’s data was predominantly collected under good and medium
conditions and lacks variation in weather and time of day. In addition, we
want to emphasize that collecting data for autonomous driving is still an
ongoing effort and that datasets such as TuSimple do not cover all possible
real-world driving scenarios to ensure safe, practical use. For the
synthetically generated data, we limited ourselves to using existing CARLA
maps without defining new simulation environments. Despite these limitations,
CARLANE serves as a supportive dataset for further research in the field of
UDA.

Ethical and Responsible Use. Considering the limitations of our work, UDA
methods trained on TuLane and MuLane should be tested with care and under the
right conditions on a full-scale car. However, real-world testing with MoLane
in the model vehicle domain can be carried out in a safe and controlled
environment. Additionally, TuLane contains open-source images with unblurred
license plates and people. This data should be treated with respect and in
accordance with privacy policies. In general, our work contributes to the
research in the field of autonomous driving, in which a lot of unresolved
ethical and legal questions are still being discussed. The step-by-step
testing possibility across three domains makes it possible for our benchmark
to include an additional safety mechanism for real-world testing.

## References

  * [1] F. Munir, S. Azam, and M. Jeon, “SSTN: Self-Supervised Domain Adaptation Thermal Object Detection for Autonomous Driving,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 206–213, 2021.
  * [2] Z. Qin, H. Wang, and X. Li, “Ultra Fast Structure-aware Deep Lane Detection,” in The European Conference on Computer Vision (ECCV), 2020\.
  * [3] X. Pan, J. Shi, P. Luo, X. Wang, and X. Tang, “Spatial As Deep: Spatial CNN for Traffic Scene Understanding,” in AAAI Conference on Artificial Intelligence (AAAI), February 2018.
  * [4] N. Garnett, R. Cohen, T. Pe’er, R. Lahav, and D. Levi, “3D-LaneNet: End-to-End 3D Multiple Lane Detection,” in ICCV, pp. 1013 – 1021, 2019\.
  * [5] C. Hu, S. Hudson, M. Ethier, M. Al-Sharman, D. Rayside, and W. Melek, “Sim-to-Real Domain Adaptation for Lane Detection and Classification in Autonomous Driving,” 2022.
  * [6] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “CARLA: An Open Urban Driving Simulator,” 2017.
  * [7] K. Saenko, B. Kulis, M. Fritz, and T. Darrell, “Adapting visual category models to new domains,” in European conference on computer vision, pp. 213–226, Springer, 2010.
  * [8] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in International conference on machine learning, pp. 1180–1189, PMLR, 2015.
  * [9] G. Wilson and D. J. Cook, “A survey of unsupervised deep domain adaptation,” ACM Transactions on Intelligent Systems and Technology (TIST), vol. 11, no. 5, pp. 1–46, 2020.
  * [10] M. Long, Y. Cao, J. Wang, and M. Jordan, “Learning Transferable Features with Deep Adaptation Networks,” in Proceedings of the 32nd International Conference on Machine Learning, vol. 37, (Lille, France), pp. 97–105, PMLR, 07–09 Jul 2015.
  * [11] Y. Zhu, F. Zhuang, J. Wang, G. Ke, J. Chen, J. Bian, H. Xiong, and Q. He, “Deep Subdomain Adaptation Network for Image Classification,” IEEE Transactions on Neural Networks and Learning Systems, 2020.
  * [12] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. March, and V. Lempitsky, “Domain-Adversarial Training of Neural Networks,” Journal of Machine Learning Research, vol. 17, no. 59, pp. 1–35, 2016.
  * [13] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell, “Adversarial Discriminative Domain Adaptation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [14] J. Xu, L. Xiao, and A. M. López, “Self-Supervised Domain Adaptation for Computer Vision Tasks,” IEEE Access, vol. 7, pp. 156694–156706, 2019\.
  * [15] Y. Sun, E. Tzeng, T. Darrell, and A. A. Efros, “Unsupervised Domain Adaptation through Self-Supervision,” 2019.
  * [16] Q. Xu, Y. Zhou, W. Wang, C. R. Qi, and D. Anguelov, “SPG: Unsupervised Domain Adaptation for 3D Object Detection via Semantic Point Generation,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15426–15436, 2021.
  * [17] X. Wu, Z. Wu, H. Guo, L. Ju, and S. Wang, “DANNet: A One-Stage Domain Adaptation Network for Unsupervised Nighttime Semantic Segmentation,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
  * [18] S. Zhao, B. Li, X. Yue, Y. Gu, P. Xu, R. Hu, H. Chai, and K. Keutzer, “Multi-source Domain Adaptation for Semantic Segmentation,” in Advances in Neural Information Processing Systems, 2019.
  * [19] N. Garnett, R. Uziel, N. Efrat, and D. Levi, “Synthetic-to-Real Domain Adaptation for Lane Detection,” in ACCV, 2020.
  * [20] TuSimple, “TuSimple-benchmark.” https://github.com/TuSimple/tusimple-benchmark/tree/master/doc/lane_detection.  Accessed: 2021-11-16.
  * [21] I. B. Akkaya, F. Altinel, and U. Halici, “Self-Training Guided Adversarial Domain Adaptation for Thermal Imagery,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 4322–4331, June 2021.
  * [22] X. Yue, Z. Zheng, S. Zhang, Y. Gao, T. Darrell, K. Keutzer, and A. Sangiovanni-Vincentelli, “Prototypical Cross-domain Self-supervised Learning for Few-shot Unsupervised Domain Adaptation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
  * [23] K. Behrendt and R. Soussan, “Unsupervised Labeled Lane Markers Using Maps,” in Proceedings of the IEEE International Conference on Computer Vision, 2019.
  * [24] F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell, “BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [25] B. Sun, J. Feng, and K. Saenko, “Return of Frustratingly Easy Domain Adaptation,” in AAAI Conference on Artificial Intelligence, 2016.
  * [26] A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola, “A Kernel Method for the Two-Sample-Problem,” in Advances in Neural Information Processing Systems, vol. 19, pp. 513–520, MIT Press, 2007.
  * [27] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola, “A Kernel Two-Sample Test,” Journal of Machine Learning Research, vol. 13, no. 25, pp. 723–773, 2012.
  * [28] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Nets,” in Advances in Neural Information Processing Systems, vol. 27, Curran Associates, Inc., 2014.
  * [29] J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. A. Efros, and T. Darrell, “CyCADA: Cycle Consistent Adversarial Domain Adaptation,” in International Conference on Machine Learning (ICML), 2018.
  * [30] K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, and D. Krishnan, “Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 95–104, 2017.
  * [31] D. Kim, K. Saito, T.-H. Oh, B. A. Plummer, S. Sclaroff, and K. Saenko, “Cross-domain Self-supervised Learning for Domain Adaptation with Few Source Labels,” 2020.
  * [32] X. Xie, J. Chen, Y. Li, L. Shen, K. Ma, and Y. Zheng, “Self-supervised cyclegan for object-preserving image-to-image domain adaptation,” in European Conference on Computer Vision, pp. 498–513, Springer, 2020.
  * [33] H. Wang, T. Shen, W. Zhang, L.-Y. Duan, and T. Mei, “Classes matter: A fine-grained adversarial approach to cross-domain semantic segmentation,” in European conference on computer vision, pp. 642–659, Springer, 2020.
  * [34] G. Wei, C. Lan, W. Zeng, Z. Zhang, and Z. Chen, “Toalign: Task-oriented alignment for unsupervised domain adaptation,” Advances in Neural Information Processing Systems, vol. 34, pp. 13834–13846, 2021.
  * [35] K. Tanwisuth, X. Fan, H. Zheng, S. Zhang, H. Zhang, B. Chen, and M. Zhou, “A prototype-oriented framework for unsupervised domain adaptation,” Advances in Neural Information Processing Systems, vol. 34, pp. 17194–17208, 2021.
  * [36] Y. Zhao, L. Cai, et al., “Reducing the covariate shift by mirror samples in cross domain alignment,” Advances in Neural Information Processing Systems, vol. 34, pp. 9546–9558, 2021.
  * [37] Z. Du, J. Li, H. Su, L. Zhu, and K. Lu, “Cross-domain gradient discrepancy minimization for unsupervised domain adaptation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3937–3946, 2021.
  * [38] G. Wei, C. Lan, W. Zeng, and Z. Chen, “Metaalign: Coordinating domain alignment and classification for unsupervised domain adaptation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16643–16653, 2021.
  * [39] J. Johnson, M. Douze, and H. Jégou, “Billion-scale similarity search with GPUs,” IEEE Transactions on Big Data, vol. 7, no. 3, pp. 535–547, 2019.
  * [40] Z. Wang, Z. Dai, B. Póczos, and J. Carbonell, “Characterizing and avoiding negative transfer,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11293–11302, 2019.
  * [41] H. Fan, X. Chang, W. Zhang, Y. Cheng, Y. Sun, and M. Kankanhalli, “Self-supervised global-local structure modeling for point cloud domain adaptation with reliable voted pseudo labels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6377–6386, 2022.
  * [42] D. Kim, K. Saito, T.-H. Oh, B. A. Plummer, S. Sclaroff, and K. Saenko, “Cross-domain self-supervised learning for domain adaptation with few source labels,” arXiv preprint arXiv:2003.08264, 2020.
  * [43] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.,” Journal of machine learning research, vol. 9, no. 11, 2008.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes] See Section 6.

    2. (b)

Did you describe the limitations of your work? [Yes] See Section 6.

    3. (c)

Did you discuss any potential negative societal impacts of your work? [Yes]
See Section 6.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments (e.g. for benchmarks)…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
See Abstract, Section 5.4 and the supplemental material.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] See Section 4, Section 5, Section 5.4, Table 1,
Table 2 and the supplemental material.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes] Table 3.

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Section 5.4.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes] We cited
the TuSimple dataset [20].

    2. (b)

Did you mention the license of the assets? [Yes] See Section 4.

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [Yes]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [Yes] TuSimple is open-source and licensed under the
Apache License, Version 2.0 (January 2004).

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [Yes] See Section 6

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

##  Appendix A Appendix

###  A.1 Example Usage of the CARLANE Benchmark

A Jupyter Notebook with a tutorial to read the datasets for usage in PyTorch
can be found at https://carlanebenchmark.github.io.

###  A.2 Model Vehicle Description

In Figure 6, the self-built 1/8th model vehicle is shown, which we used to
gather the images for the 1/8th scaled target domain. A NVIDIA Jetson AGX is
the central computation unit powered by a separate Litionite Tanker Mini
25000mAh battery. For image collection, we utilize the software framework ROS
Melodic and a Stereolabs ZEDM stereo camera with an integrated IMU. The camera
is directly connected to the AGX and captures images with a resolution of
$1280\times 720$ pixels and a rate of $30$ FPS.

Figure 6: Picture of the 1/8th model vehicle we built to capture images in our
1/8th target domain.

###  A.3 Reproducibility of the Baselines

To ensure reproducibility, we strictly follow UFLD [2] and the corresponding
UDA method for model architecture and hyperparameters. Thereby, we utilize
UFLD as an encoder for the UDA method. We provide a detailed table of the
tuned hyperparameters, architecture changes, and objectives in the main text.
In addition, the trained weights of our baselines, their entire
implementation, and the configuration files of our baselines are made publicly
available at https://carlanebenchmark.github.io.

Initialization. We initialize convolutional layer weights with kaiming normal
and their biases with 0.0. Linear layer weights are initialized with normal
(mean = 0.0, std = 0.01), batch normalization weights and biases are
initialized with 1.0.

###  A.4 Additional Results

UFLD-SO | DANN | ADDA | SGADA | SGPCS
---|---|---|---|---
|  |  |  |
|  |  |  |
Figure 7: t-SNE visualizations of the MoLane dataset (top) and the TuLane
dataset (bottom). The source domain is marked in blue, the real-world model
vehicle target domain in red, and TuLane’s target domain in green.

t-SNE feature clustering. Figure 7 shows the t-SNE feature clustering of the
trained baselines for the MoLane and TuLane dataset, respectively. We observe
that few features of both domains spread over the entire plot for higher-
performing UDA methods. However, there are still large clusters of features
from one domain, indicating that the domain adaptation only occurred slightly.

Qualitative results. We randomly sample results from our baselines and show
them in Figures 9, 10, and 11. Compared to UFLD-SO, the UDA baselines ADDA,
SGADA, and SGPCS increase performance consistently. UFLD-TO samples show the
best results on the target domain.

###  A.5 Comparison to Related Work

Table 4: Comparison of CARLANE (ours) with datasets created by related work. Dataset | Year |  | Publicly
---
Available
Domains | Simulation | Resolution |  | Total
---
Images
Annotations
[4] | 2019 | ✗ | sim, real | blender | $480\times 360$ |  $391$K | 3D
[19] | 2020 | ✗ | sim, real | blender | $480\times 360$ |  $586$K | 3D
[5] | 2022 | ✗ | sim, real | Carla | $1280\times 720$ |  $23$K | 2D
ours | 2022 | ✓ | sim, real, scaled | Carla | $1280\times 720$ |  $163$K | 2D
Table 5: Comparison of applied variations for the collection of the synthetic datasets. Dataset |  Ego Vehicle |  Camera Position |  Lane Deviation |  Traffic |  Pedestrians |  World Objects |  Daytime |  Weather |  City |  Rural |  Highway |  Terrain |  Lane Topology |  Road Appearance
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
[4] | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ | ✗ | ✓ | ✓ | ✓
[19] | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ | ✗ | ✓ | ✓ | ✓
[5] | ✗ | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
ours | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓

In Table 4, we compare CARLANE with the datasets created by related work. The
main differentiators are that our dataset contains three distinct domains,
including a scaled model vehicle, and is publicly available. To further
compare our synthetic datasets with related work, the applied variations
during the data collection process are summarized in Table 5. Additionally, we
highlight noticeable differences in the visual quality of the simulation
engines in Figure 8. Scenes captured in Carla are more realistic and detailed.

[4] |  |  |
---|---|---|---
ours |  |  |
Figure 8: Visual comparison of simulation images from the custom blender simulation used in [4, 19] and the Carla simulation used by [5] and our work. We observe that scenes captured in Carla are more detailed and realistic. | MoLane | TuLane | MuLane
---|---|---|---
UFLD-SO |  |  |  |
DANN |  |  |  |
ADDA |  |  |  |
SGADA |  |  |  |
SGPCS |  |  |  |
UFLD-TO |  |  |  |
Figure 9: Qualitative results of target domain predictions. Images are randomly sampled. Ground truth lane annotations are marked in blue, predictions in red. | MoLane | TuLane | MuLane
---|---|---|---
UFLD-SO |  |  |  |
DANN |  |  |  |
ADDA |  |  |  |
SGADA |  |  |  |
SGPCS |  |  |  |
UFLD-TO |  |  |  |
Figure 10: Qualitative results of target domain predictions. Images are randomly sampled. Ground truth lane annotations are marked in blue, predictions in red. | MoLane | TuLane | MuLane
---|---|---|---
UFLD-SO |  |  |  |
DANN |  |  |  |
ADDA |  |  |  |
SGADA |  |  |  |
SGPCS |  |  |  |
UFLD-TO |  |  |  |
Figure 11: Qualitative results of target domain predictions. Images are
randomly sampled. Ground truth lane annotations are marked in blue,
predictions in red.

##  Appendix B Datasheet for the CARLANE Benchmark

Motivation For what purpose was the dataset created? Was there a specific task
in mind? Was there a specific gap that needed to be filled? Please provide a
description.

CARLANE was created to be the first publicly available single- and multi-
target Unsupervised Domain Adaptation (UDA) benchmark for lane detection to
facilitate future research in these directions. However, in a broader sense,
the datasets of CARLANE were also created for unsupervised and semi-supervised
learning and partially for supervised learning. Furthermore, a real-to-real
transfer can be performed between the target domains of our datasets.

Who created the dataset (e.g., which team, research group) and on behalf of
which entity (e.g., company, institution, organization)?

As released on June 17, 2022, the initial version of CARLANE was created by
Julian Gebele, Bonifaz Stuhr, and Johann Haselberger from the Institute for
Driver Assistance Systems and Connected Mobility (IFM). The IFM is a part of
the University of Applied Sciences Kempten. Furthermore, CARLANE was created
by Bonifaz Stuhr as part of his Ph.D. at the Autonomous University of
Barcelona (UAB) and by Johann Haselberger as part of his Ph.D. at the
Technische Universität Berlin (TU Berlin).

Who funded the creation of the dataset? If there is an associated grant,
please provide the name of the grantor and the grant name and number.

There is no specific grant for the creation of the CARLANE Benchmark. The
datasets were created as part of the work at the IFM and the University of
Applied Sciences Kempten.

Composition What do the instances that comprise the dataset represent (e.g.,
documents, photos, people, countries)? Are there multiple types of instances
(e.g., movies, users, and ratings; people and interactions between them; nodes
and edges)? Please provide a description.

The instances are drives on diverse roads in simulation, in an abstract 1/8th
real world, and in full-scale real-world scenarios, along with lane
annotations of the up to four nearest lanes to the vehicle.

How many instances are there in total (of each type, if appropriate)?

Table 6: Dataset overview. Unlabeled images denoted by *, partially labeled images denoted by **. Dataset | domain | total images | train | validation | test | lanes
---|---|---|---|---|---|---
MoLane | CARLA simulation | 84,000 | 80,000 | 4,000 | - |  $\leq$ 2
model vehicle | 46,843 | 43,843* | 2,000 | 1,000 |  $\leq$ 2
TuLane | CARLA simulation | 26,400 | 24,000 | 2,400 | - |  $\leq$ 4
TuSimple | 6,408 | 3,268 | 358 | 2,782 |  $\leq$ 4
MuLane | CARLA simulation | 52,800 | 48,000 | 4,800 | - |  $\leq$ 4
model vehicle + TuSimple | 12,536 | 6,536** | 4,000 | 2,000 |  $\leq$ 4

Table 6 shows the per-domain and per-subset breakdown of CARLANE instances.
TuSimple is available at https://github.com/TuSimple/tusimple-benchmark under
the Apache License Version 2.0, January 2004.

Does the dataset contain all possible instances or is it a sample (not
necessarily random) of instances from a larger set? If the dataset is a
sample, then what is the larger set? Is the sample representative of the
larger set (e.g., geographic coverage)? If so, please describe how this
representativeness was validated/verified. If it is not representative of the
larger set, please describe why not (e.g., to cover a more diverse range of
instances, because instances were withheld or unavailable).

The datasets of CARLANE contain samples of driving scenarios and lane
annotations encountered in simulation and the real world. The datasets are not
representative of all these driving scenarios, as the distribution of the
latter is highly dynamic and diverse. Instead, the motivation was to resemble
the variety and shifts of different domains in which such scenarios occur to
strengthen the systematic study of UDA methods for lane detection. Therefore,
CARLANE should be considered as an UDA benchmark with step-by-step testing
possibility across three domains, which makes it possible to include an
additional safety mechanism for real-world testing.

What data does each instance consist of? “Raw” data (e.g., unprocessed text or
images) or features? In either case, please provide a description.

Each labeled instance consists of the following components:

(1) A single $1280\times 720$ image from a driving scenario.

(2) A .json file entry for the corresponding subset containing lane
annotations following TuSimple. The lanes’ y-values discretized by 56 raw
anchors, the lanes’ x-values to 101 gridding cells, with the last gridding
cell representing the absence of a lane. The file path to the corresponding
image is also stored in the .json file.

(3) A .png file containing lane segmentations following UFLD
(https://github.com/cfzd/Ultra-Fast-Lane-Detection), where each of the four
lanes has a different label.

(4) A .txt file entry for the corresponding subset containing the linkage
between the raw image and its segmentation as well as the presence and absence
of a lane.

Each unlabeled instance consists of an $1280\times 720$ image from a driving
scenario and a .txt file entry for the corresponding subset.

Is there a label or target associated with each instance? If so, please
provide a description.

As described above, the labels per instance are discretized lane annotations
and lane segmentations.

Is any information missing from individual instances? If so, please provide a
description, explaining why this information is missing (e.g., because it was
unavailable). This does not include intentionally removed information, but
might include, e.g., redacted text.

Everything is included. No data is missing.

Are relationships between individual instances made explicit (e.g., users’
movie ratings, social network links)? If so, please describe how these
relationships are made explicit.

There are no relationships made explicit between instances. However, some
instances are part of the same drive and therefore have an implicit
relationship.

Are there recommended data splits(e.g., training, development/validation,
testing)? If so, please provide a description of these splits, explaining the
rationale behind them.

Each domain is split into training and validation subsets. Details are shown
in Table 6. The target domains for UDA additionally include test sets, which
were recorded from separate tracks (model vehicle) or driving scenarios
(TuSimple). Since UDA aims to adapt models to target domains, only the target
domains include a test set.

Are there any errors, sources of noise, or redundancies in the dataset? If so,
please provide a description.

CARLANE was recorded from different drives through simulation and real-world
domains. Therefore there are images captured from the same drive, which result
in similar scenarios for consecutive images. Target domain samples were
annotated by hand and may include human labeling errors. However, we double-
checked labels and cleaned TuSimple’s test set with our labeling tool.

Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g., websites, tweets, other datasets)? If it links to or
relies on external resources, a) are there guarantees that they will exist,
and remain constant, over time; b) are there official archival versions of the
complete dataset (i.e., including the external resources as they existed at
the time the dataset was created); c) are there any restrictions (e.g.,
licenses, fees) associated with any of the external resources that might apply
to a dataset consumer? Please provide descriptions of all external resources
and any restrictions associated with them, as well as links or other access
points, as appropriate.

CARLANE is entirely self-contained.

Does the dataset contain data that might be considered confidential (e.g.,
data that is protected by legal privilege or by doctor-patient
confidentiality, data that includes the content of individuals’ non-public
communications)? If so, please provide a description.

The full-scale real-world target domain contains open-source images with
unblurred license plates and people from the TuSimple dataset. This data
should be treated with respect and in accordance with privacy policies. The
other domains do not contain data that might be considered confidential since
there where recorded in simulations or a controlled 1/8th real-world
environment.

Does the dataset contain data that, if viewed directly, might be offensive,
insulting, threatening, or might otherwise cause anxiety? If so, please
describe why.

CARLANE includes driving scenarios; therefore, its datasets could cause
anxiety in people with driving anxiety.

Does the dataset identify any subpopulations (e.g., by age, gender)? If so,
please describe how these subpopulations are identified and provide a
description of their respective distributions within the dataset.

No.

Is it possible to identify individuals (i.e., one or more natural persons),
either directly or indirectly (i.e., in combination with other data) from the
dataset? If so, please describe how.

Yes, individuals could be identified in the full-scale real-world target
domain from TuSimple, since it contains unblurred license plates and people.
However, the remaining domains do not contain identifiable individuals.

Does the dataset contain data that might be considered sensitive in
anyway(e.g., data that reveals race or ethnic origins, sexual orientations,
religious beliefs, political opinions or union memberships, or locations;
financial or health data; biometric or genetic data; forms of government
identification, such as social security numbers; criminal history)? If so,
please provide a description.

The full-scale real-world target domain from TuSimple could implicitly reveal
sensitive information printed or put on the vehicles or people’s wearings.

Collection Process How was the data associated with each instance acquired?
Was the data directly observable (e.g., raw text, movie ratings), reported by
subjects (e.g., survey responses), or indirectly inferred/derived from other
data (e.g., part-of-speech tags, model-based guesses for age or language)? If
the data was reported by subjects or indirectly inferred/derived from other
data, was the data validated/verified? If so, please describe how.

The source domain images of driving scenarios and the corresponding lane
annotations were directly recorded from the simulation. Lanes were manually
labeled for the directly recorded real-world images. For the images collected
from the model vehicle, the authors annotated the data with a labeling tool
created for this task. The labeling tool is publicly available at
https://carlanebenchmark.github.io. The labeling tool is utilized to clean up
the annotations of the test set in the real-world domain. The authors do not
have information about the labeling process of the full-scale target domain
since its data is derived from the TuSimple dataset.

What mechanisms or procedures were used to collect the data (e.g., hardware
apparatuses or sensors, manual human curation, software programs, software
APIs)? How were these mechanisms or procedures validated?

The source domain data was collected using the CARLA simulator and its APIs
with a resolution of $1280\times 720$ pixels. The real-world 1/8th target
domain was collected with a Stereolabs ZEDM camera with 30 FPS and a
resolution of $1280\times 720$ pixels. The lane distributions were
additionally balanced with a bagging approach, and lanes were annotated with a
labeling tool. More information can be found in the corresponding paper and
the implementation. The implementation and all used tools are publicly
available at https://carlanebenchmark.github.io.

If the dataset is a sample from a larger set, what was the sampling strategy
(e.g., deterministic, probabilistic with specific sampling probabilities)?

Source domain dataset entries are sampled based on the relative angle $\beta$
of the agent to the center lane. For MoLane, five lane classes are defined for
the bagging approach: strong left curve ($\beta\leq$$None$), soft left curve
($None$ $<\beta\leq$ $None$), straight ($None$ $<\beta<$ $None$), soft right
curve ($None$ $\leq\beta<$ $None$) and strong right curve ($None$$\leq\beta$).

For TuLane, three lane classes are defined for the bagging approach: left
curve ($None$ $<\beta\leq$ $None$), straight ($None$ $<\beta<$ $None$) and
right curve ($None$ $\leq\beta<$ $None$).

Who was involved in the data collection process (e.g., students, crowdworkers,
contractors) and how were they compensated (e.g., how much were crowdworkers
paid)?

Only the authors were involved in the collection process. The authors do not
have information about the people involved in collecting the TuSimple dataset.

Over what timeframe was the data collected? Does this timeframe match the
creation timeframe of the data associated with the instances (e.g., recent
crawl of old news articles)? If not, please describe the timeframe in which
the data associated with the instances was created.

MoLane’s data was collected and annotated from June 2021 to August 2021. Data
for TuLane’s source domain was collected in February 2022.

Were any ethical review processes conducted (e.g., by an institutional review
board)? If so, please provide a description of these review processes,
including the outcomes, as well as a link or other access point to any
supporting documentation.

No ethical reviews have been conducted to date. However, an ethical review may
be conducted as part of the paper review process.

Preprocessing/cleaning/labeling Was any preprocessing/cleaning/labeling of the
data done (e.g., discretization or bucketing, tokenization, part-of-speech
tagging, SIFT feature extraction, removal of instances, processing of missing
values)? If so, please provide a description. If not, you may skip the
remaining questions in this section.

As described above, lane annotations were labeled or cleaned using a labeling
tool and sampled based on the relative angle $\beta$ of the agent to the
center lane.

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data
(e.g., to support unanticipated future uses)? If so, please provide a link or
other access point to the “raw” data.

No.

Is the software that was used to preprocess/clean/label the data available? If
so, please provide a link or other access point.

Yes, the software is available at https://carlanebenchmark.github.io.

Uses Has the dataset been used for any tasks already? If so, please provide a
description.

The datasets were used to create UDA baselines for the corresponding paper
presenting the CARLANE Benchmark.

Is there a repository that links to any or all papers or systems that use the
dataset? If so, please provide a link or other access point.

Yes, the baselines presented in the corresponding paper are available at
https://carlanebenchmark.github.io.

What(other) tasks could the dataset be used for?

In a broader sense, the datasets of CARLANE can also be used for unsupervised
and semi-supervised learning and partially for supervised learning.

Is there anything about the composition of the dataset or the way it was
collected and preprocessed/cleaned/labeled that might impact future uses? For
example, is there anything that a dataset consumer might need to know to avoid
uses that could result in unfair treatment of individuals or groups (e.g.,
stereotyping, quality of service issues) or other risks or harms (e.g., legal
risks, financial harms)? If so, please provide a description. Is there
anything a dataset consumer could do to mitigate these risks or harms?

Yes, TuLane and MuLane contain open-source images with unblurred license
plates and people. This data should be treated with respect and in accordance
with privacy policies. In general, CARLANE contributes to the research in the
field of autonomous driving, in which many unresolved ethical and legal
questions are still being discussed. The step-by-step testing possibility
across three domains makes it possible for our benchmark to include an
additional safety mechanism for real-world testing. This can help the consumer
to mitigate the risks and harms to some extent.

Are there tasks for which the dataset should not be used? If so, please
provide a description.

Since CARLANE focuses on UDA for lane detection and spans a limited number of
driving scenarios, consumers should not solely really on this dataset to train
models for fully autonomous driving.

Distribution Will the dataset be distributed to third parties outside of the
entity (e.g., company, institution, organization) on behalf of which the
dataset was created? If so, please provide a description.

Yes, CARLANE is publicly available on the internet for anyone interested in
using it.

How will the dataset will be distributed (e.g., tarball on website, API,
GitHub)? Does the dataset have a digital object identifier (DOI)?

CARLANE is distributed through kaggle at
https://www.kaggle.com/datasets/carlanebenchmark/carlane-benchmark

DOI: 10.34740/kaggle/dsv/3798459

When will the dataset be distributed?

The datasets have been available on kaggle since June 17, 2022.

Will the dataset be distributed under a copyright or other intellectual
property (IP) license, and/or under applicable terms o fuse (ToU)? If so,
please describe this license and/or ToU, and provide a link or other access
point to, or otherwise reproduce, any relevant licensing terms or ToU, as well
as any fees associated with these restrictions.

CARLANE is licensed under the Apache License Version 2.0, January 2004.

Have any third parties imposed IP-based or other restrictions on the data
associated with the instances? If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any relevant
licensing terms, as well as any fees associated with these restrictions.

TuSimple, which is used for TuLanes and MuLanes target domains, is licensed
under the Apache License Version 2.0, January 2004.

Do any export controls or other regulatory restrictions apply to the dataset
or to individual instances? If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any
supporting documentation.

Unknown to authors of the datasheet.

Maintenance Who will be supporting/hosting/maintaining the dataset?

CARLANE is hosted on kaggle and supported and maintained by the authors.

How can the owner/curator/manager of the dataset be contacted (e.g., email
address)?

The curators of the datasets can be contacted under
<EMAIL_ADDRESS>
Is there an erratum? If so, please provide a link or other access point.

No.

Will the dataset be updated (e.g., to correct labeling errors, add new
instances, delete instances)? If so, please describe how often, by whom, and
how updates will be communicated to dataset consumers (e.g., mailing list,
GitHub)?

New versions of CARLANE’s datasets will be shared and announced on our
homepage (https://carlanebenchmark.github.io) and at kaggle if corrections are
necessary.

Will older versions of the dataset continue to be supported/hosted/maintained?
If so, please describe how. If not, please describe how its obsolescence will
be communicated to dataset consumers.

Yes, we plan to support versioning of the datasets so that all the versions
are available to potential users. We maintain the history of versions via our
homepage (https://carlanebenchmark.github.io) and at kaggle. Each version will
have a unique DOI assigned.

If others want to extend/augment/build on/contribute to the dataset, is there
a mechanism for them to do so? If so, please provide a description. Will these
contributions be validated/verified? If so, please describe how. If not, why
not? Is there a process for communicating/distributing these contributions to
dataset consumers? If so, please provide a description.

Others can extend/augment/build on CARLANE with the support of the open-source
tools provided on our homepage. Besides these tools, there will be no
mechanism to validate or verify the extended datasets. However, others are
free to release their extension of the CARLANE Benchmark or its datasets under
the Apache License Version 2.0.