# COEFF-KANs: A Paradigm to Address the Electrolyte Field with KANs

Xinhe Li13, Zhuoying Feng13, Yezeng Chen23, Weichen Dai13, Zixu He1,
Yi Zhou13, Shuhong Jiao1

###### Abstract

To reduce the experimental validation workload for chemical researchers and
accelerate the design and optimization of high-energy-density lithium metal
batteries, we aim to leverage models to automatically predict Coulombic
Efficiency (CE) based on the composition of liquid electrolytes. There are
mainly two representative paradigms in existing methods: machine learning and
deep learning. However, the former requires intelligent input feature
selection and reliable computational methods, leading to error propagation
from feature estimation to model prediction, while the latter (e.g.
MultiModal-MoLFormer) faces challenges of poor predictive performance and
overfitting due to limited diversity in augmented data. To tackle these
issues, we propose a novel method COEFF (COlumbic EFficiency prediction via
Fine-tuned models), which consists of two stages: pre-training a chemical
general model and fine-tuning on downstream domain data. Firstly, we adopt the
publicly available MoLFormer model to obtain feature vectors for each solvent
and salt in the electrolyte. Then, we perform a weighted average of embeddings
for each token across all molecules, with weights determined by the respective
electrolyte component ratios. Finally, we input the obtained electrolyte
features into a Multi-layer Perceptron or Kolmogorov-Arnold Network to predict
CE. Experimental results on a real-world dataset demonstrate that our method
achieves SOTA for predicting CE compared to all baselines. Data and code used
in this work will be made publicly available after the paper is published.

## Introduction

The discovery and optimization of battery electrolytes are crucial in climate
and sustainable technology due to their potential to accelerate
decarbonization in all economic sectors (Agre, Molina, and Chu 2017; Tian et
al. 2020). Enhancing Coulombic Efficiency (CE) is vital for the adoption of
high-energy-density lithium metal batteries (Liu et al. 2023a; Yuan et al.
2021). Liquid electrolyte engineering is a promising strategy for improving
CE, but its complexity makes it challenging to predict and design electrolyte
performance (Aykol, Herring, and Anapolsky 2020). For example, high-throughput
screening accelerates the search for individual compounds but does not guide
the comprehensive design of new formulations (Abolhasani and Kumacheva 2023).
Fortunately, research has shown that modern energy storage devices typically
use liquid electrolytes composed of one or more organic solvents and salt
additives, and their formulations have a significant impact on CE (Jie et al.
2020; Yu et al. 2022). Thus, we can predict CE among a large number of
electrolyte formulations and screen out those with high CE to recommend to
chemical researchers, significantly reducing the workload of experimental
validation.

Figure 1: Overview of our method’s two-stage paradigm: pre-training a chemical
general model and fine-tuning on downstream domain data.

To deal with this challenge, existing works are mainly based on two
representative paradigms: machine learning (Kim et al. 2023; Narayanan
Krishnamoorthy et al. 2022) and deep learning methods (Zeng et al. 2024;
Soares et al. 2023; Ross et al. 2022; Sharma et al. 2023). The former
approaches predict the CE of electrolytes via a supervised learning model such
as regression, which takes selective features of components (e.g., redox
potentials, dielectric constants, ionic conductivity, and elemental
composition) as input. This process requires intelligent input feature
selection and reliable computational methods, which raises the critical issue
of error propagation from estimating input features to model predictions
(e.g., Kim et al. (2023) ignore the structural information of chemical
molecules). An exemplary work in the latter approaches is MultiModal-MoLFormer
(Soares et al. 2023), which leverages extensive chemical information learned
from unlabeled corpora in pre-training to predict the CE of electrolytes.
However, its predictive performance remains suboptimal, with a Root Mean
Square Error (RMSE) of only 0.195 on the test set. Furthermore, data
augmentation by shuffling the order of electrolyte components effectively
increased the dataset size from 147 to 27,266 samples, yet limited diversity
may lead to overfitting during training.

To address the above issues, we propose a novel electrolyte property
prediction method COEFF (COlumbic EFficiency prediction via Fine-tuned
models). As illustrated in Figure 1, COEFF is intensively designed for
learning the relationship between electrolyte composition and CE, which
consists of two stages: pre-training of chemical general models and fine-
tuning on downstream domain data. First, we adopt the publicly available
MoLFormer (Ross et al. 2022), which is constructed by pre-training an
efficient Transformer (Vaswani et al. 2017) encoder on a large unlabeled
chemical corpus to obtain embeddings at the molecular level. The input of
MoLFormer conforms to the Simplified Molecular Input Linear Entry
Specification (SMILES), which can represent both the elemental composition and
structural information of chemical molecules. For our application, we
independently input all solvents and salts in the electrolyte into MoLFormer
to extract the chemical features of each component. Then, we multiply the
token embeddings of all SMILES molecules by their corresponding ratios and sum
them to obtain representations at the formulation level. This is similar to
concatenating all embeddings along the token length dimension, followed by a
weighted average pooling layer, with the weights derived from the ratios of
the electrolyte components. Finally, we feed electrolyte embeddings into a
downstream model, including Multi-layer Perceptron (COEFF-MLP) or
Kolmogorov–Arnold Networks (COEFF-KAN) (Liu et al. 2024b), and the output is
the predicted value of CE. Our work demonstrates the potential of deep
learning in electrolyte design. By harnessing the power of language models, we
can accelerate the discovery and optimization of new formulations,
significantly reducing the experimental validation workload for chemical
researchers, and potentially revolutionizing various industries.

Experimental results on a real-world test set with 13 data points show that
our method (Min. 0.132 for COEFF-MLP and Min. 0.110 for COEFF-KAN) achieves
significant and consistent improvement in the RMSE metric for predicting CE
compared to all baselines, including machine learning (Min. 0.170) and deep
learning (Min. 0.195) methods. We will publish all source codes and datasets
of this work on GitHub for further research explorations.

In summary, the main contributions of this paper are threefold: (1) We focus
on the task of predicting CE based on electrolyte components and propose a
novel method COEFF, which consists of two stages: pre-training a chemical
general model and fine-tuning on downstream domain data. (2) To the best of
our knowledge, we are the first to introduce KAN networks into the task of
electrolyte property prediction and achieve better performance than MLP. (3)
We conduct extensive experiments on real-world datasets and show that COEFF
outperforms previous state-of-the-art works.

## Related Work

### Language Models for Chemistry

In recent years, the realm of large language models has seen significant
advancements, highlighting their remarkable ability to understand intricate
chemical language representations (Pan 2023; Jablonka et al. 2024; Castro
Nascimento and Pimentel 2023). Studies have shown that the two-stage paradigm
has great potential: first pre-training on large unlabeled datasets and then
fine-tuning on a specific downstream task (Schwaller et al. 2019; Ross et al.
2022; Moret et al. 2023). Leveraging self-supervised learning, these methods
successfully minimize reliance on labeled data and specific tasks. While pre-
trained language models perform well in predicting molecular properties (Liu
et al. 2023b; Chen and Bajorath 2023), their practical implementation on
further downstream tasks is still in its early stages and largely limited to
sequencing problems such as predicting protein sequences (Madani et al. 2023),
polymers (Chen, Tao, and Li 2021), and chemical reactions (Ranković and
Schwaller 2023). Moreover, prediction errors of most methods (Soares et al.
2023; Sharma et al. 2023; Kim et al. 2023) in the electrolyte field are larger
than experimental errors, making them difficult to achieve the goal of
practical application. We focus on the rarely studied task of CE prediction
and propose a novel method COEFF following the two-stage paradigm.

Figure 2: Overview of our proposed COEFF framework.

### Kolmogorov–Arnold Networks

Inspired by the Kolmogorov-Arnold representation theorem (Kolmogorov 1961),
researchers propose Kolmogorov-Arnold Networks (KANs) (Liu et al. 2024b) as
promising alternatives to MLPs. Unlike MLPs, KANs have learnable activation
functions on weights rather than fixed ones on neurons. Recent studies have
shown that in some specific domains, KANs have advantages in accuracy (Seydi
2024; Bresson et al. 2024; Kiamari, Kiamari, and Krishnamachari 2024),
interpretability (Li et al. 2024; Yang, Qin, and Yu 2024; Xu, Chen, and Wang
2024), continuous learning (Liu et al. 2024a), parameter efficiency (Abueidda,
Pantidis, and Mobasher 2024; Cheon 2024), etc. In the field of science, PIKAN
(Liu et al. 2024b; Shukla et al. 2024) combines physics-informed neural
networks (Raissi, Perdikaris, and Karniadakis 2019) and KANs to solve a 2D
Poison equation, and DeepOKAN (Abueidda, Pantidis, and Mobasher 2024) uses a
KAN operator network based on Gaussian radial basis functions to solve a 2D
orthotropic elasticity problem. In the field of image processing, U-KAN (Li et
al. 2024) redesigns the established U-Net (Ronneberger, Fischer, and Brox
2015) pipeline for medical image segmentation by integrating the dedicated KAN
layers on the tokenized intermediate representation, and KCN (Cheon 2024)
introduces KANs in various pre-trained convolutional neural network models for
remote sensing scene classification tasks. In the field of human-computer
interaction, iKAN (Liu et al. 2024a) is an incremental learning framework for
wearable sensor human activity recognition that tackles two challenges
simultaneously: catastrophic forgetting and non-uniform inputs. Inspired by
the above works, we apply KANs to the electrolyte field for the first time and
obtain better experimental results compared with MLPs.

## Methodology

### Preliminary

Our task is to predict the battery’s Coulombic Efficiency through the input of
electrolyte compositions. Suppose an electrolyte has $n$ SMILES molecules
$\mathcal{S}=\\{s_{1},s_{2},...,s_{n}\\}$ and corresponding molar ratios
$\mathcal{P}=\\{p_{1},p_{2},...,p_{n}\\}$. Utilizing a pre-trained chemical
language model, we can obtain a SMILES embedding set
$\mathcal{X}=\\{\mathbf{x_{1}},\mathbf{x_{2}},...,\mathbf{x_{n}}\\}$, where
each $\mathbf{x_{i}}$ represents the embedding of a single SMILES molecule.

We try two different network structures for the downstream task: (1) MLP aims
to replicate complex functional mappings via a series of nonlinear
transformations across multiple layers. The blocks can be formally represented
as:

$\mathrm{MLP}(\mathrm{\textbf{x}})=\left(\textbf{W}_{L-1}\circ\sigma\circ\textbf{W}_{L-2}\circ\ldots\circ\sigma\circ\textbf{W}_{0}\right)\mathrm{\textbf{x}}$
(1)

where $\textbf{W}_{i}$ denotes the weight matrix, $\sigma$ denotes the
activation function, and x denotes the model input. (2) KAN is inspired by the
Kolmogorov-Arnold representation theorem, and an L-layer KAN operates
analogously to an MLP, comprising a sequence of nested KAN layers. The blocks
can be formally represented as:

$\mathrm{KAN}(\mathrm{\textbf{x}})=\left(\mathbf{\Phi}_{L-1}\circ\mathbf{\Phi}_{L-2}\circ\ldots\circ\mathbf{\Phi}_{1}\circ\mathbf{\Phi}_{0}\right)\mathrm{\textbf{x}}$
(2)

where $\mathbf{\Phi}_{i}$ denotes the function matrix corresponding to the
$i^{\text{th}}$ KAN layer. For each KAN layer with $n_{\text{in}}$-dimensional
input and $n_{\text{out}}$-dimensional output, $\mathbf{\Phi}$ can be defined
as a matrix of 1D functions:

$\mathbf{\Phi}=\\{\phi_{q,p}\\}\quad
p=1,2,\ldots,n_{\text{in}},q=1,2,\ldots,n_{\text{out}}.$ (3)

### Overview of Method

Our method COEFF, illustrated in the Figure 2, is built upon an optimized
MoLFormer architecture, which is capable of capturing sufficiently precise
chemical and structural information solely from SMILES representations.

First, we employ MoLFormer, a pre-trained Transformer encoder, to generate
embeddings for chemical molecules. The dataset comprises a subset from 10% of
both the Zinc and PubChem datasets. We feed each SMILES, $s_{i}$, individually
into MoLFormer to obatain the token embeddings of all SMILES molecules
$\mathrm{x_{i}}$, a vector that represents the chemical features of each
component. After that, we scale $\mathrm{x_{i}}$ by their corresponding molar
ratios $p_{i}$ and subsequently aggregate these weighted embeddings through
summation to yield a feature vector that characterizes the entire electrolyte
embedding x. This is similar to concatenating all embeddings along the token
length dimension, followed by a weighted average pooling layer, with the
weights derived from the ratios of the electrolyte components. Finally, we
input the electrolyte embedding x into MLP or KAN to predict the CE of
electrolyte compositions.

### Data Processing

The dataset (Kim et al. 2023) confines attention to solvent volume and salt
molarity, emphasizing elemental composition’s role in CE. By approximating
weight and volume ratios, potential discrepancies are introduced. Thus,
adopting molar ratios for formulation enhances accuracy in simulating
experimental scenarios.

We converted electrolyte compositions into SMILESs with accompanying molar
ratios, better representing the relationship between electrolyte solvents and
salts, thereby enhancing the model’s capability to predict Logarithmic
Coulombic Efficiency (LCE).

### Pre-training of Chemical General Model

Our approach is based on MoLFormer, an advanced transformer-based model
commonly utilized for chemical language representations. MoLFormer is a large-
scale masked language model that processes inputs using a series of blocks
that alternate between self-attention and feed-forward connections.

MoLFormer with linear attention integrates rotary position encoding,
demonstrated to outperform absolute position encoding, enhancing the
representation of positional relationships in SMILES sequences without
significant computational overhead. The attention mechanism is defined as
follows:

$Attention_{m}(Q,K,V)=\frac{\sum_{n=1}^{N}\langle\phi(R_{m}q_{m}),\phi(R_{n}q_{n})\rangle
v_{n}}{\sum_{n=1}^{N}\langle\phi(R_{m}q_{m}),\phi(R_{n}q_{n})\rangle}$

where $\phi$ denotes the embedding function, $R$ the rotation matrix, and
$Q,K,V$ represent query, key, and value matrices respectively.

Using the capabilities of MoLFormer and enhancing it with the addition of
relative positional embeddings, our methodology presents an advanced and
efficient solution to anticipate the LCE of battery electrolytes, offering
valuable insight into a variety of chemical applications.

### Fine-tuning on Downstream Domain Data

We detail the SMILES’s tokenization and integration of MoLFormer-encoded
SMILES with their molar ratios.

Considering that 99.4 % of the SMILES sequences comprise no more than 202
tokens, to enhance operational efficiency, we impose a limitation on sequence
lengths, setting them within the range of 1 to 202 tokens.

Concatenate Input Disentangle Output (CIDO) MultiModal-MoLFormer is splicing
SMILES through <sep>as input to MoLFormer, which is against the format of
MoLFormer pre-training, so we innovate by preprocessing: concatenating
tokenised SMILES sequences before input and later disentangling post-encoding.
After encoding, the representation of each molecule is a shape tensor $B\times
L\times H$, where $B$ is the batch size, $L$ the token count per molecule, and
$H$ denotes the embedding dimensionality. This approach capitalises on
MoLFormer’s design while maintaining the integrity of distinct molecules.
design while maintaining the integrity of distinct molecular profiles.

We use two methods: COEFF-MLP and COEFF-KAN as our downstream network
structure to predict the CE of the electrolyte formulation. This configuration
processes the electrolyte embedding, x, to make the prediction, thereby
leveraging the model’s capacity to capture intricate relationships within the
molecular structures for enhanced efficiency forecasts. The number of layers
in the two methods is not fixed. Here we represent the representation obtained
by x after the i-th layer

$\mathrm{\textbf{x}_{i+1}}=\mathrm{MLP}_{i}(\mathrm{\textbf{x}_{i}}).$ (4)
$\mathrm{\textbf{x}_{i+1}}=\mathrm{KAN}_{i}(W_{i}\times\mathrm{\textbf{x}_{i}}\mid\Phi_{i}).$
(5)

## Experiments

### Experimental Setup

#### Datasets

To validate the efficacy of our method, we apply it to the Li/Cu half-cell-
based dataset (Kim et al. 2023), collected from some chemical literature.
Electrolyte formulations in this dataset consist of 2 to 7 components each,
described by SMILES. To numerically amplify the change in output with respect
to electrolyte variation, CE is converted to a logarithmic format (LCE,
defined as $\mathsf{-log(1-CE)}$). We remove an entry with a repeated ratio
but different measurement methods, and correct errors in some ratios and
molecular information. Thus, we obtain 149 electrolyte formulations for
training and 13 entries as a test set $\mathcal{T}_{\text{out}}$.
Additionally, we split the 149 data into training and test sets in a ratio of
7:3 to obtain $\mathcal{T}_{\text{in}}$. Box plots illustrated in Figure 3
provide a clear visualization of the distribution of LCE outputs for the
training data based on the count of formulation constituents.

Figure 3: Box plots depict the distribution of LCE outputs for training data
based on the formulation constituent count. The outer whiskers represent the
minimum and maximum values, the central line represents the median, and the
colored box represents the 25th to 75th percentile of the data. Data points
outside the outer whiskers are outliers observed in the data.

#### Baselines

We compare our method with some strong baselines, which are divided into two
groups: machine learning ($\dagger$) and deep learning ($\ddagger$)
approaches.

  * •

Kim et al. ($\dagger$) (Kim et al. 2023) use the elemental composition of
electrolytes as input features for the machine learning models (e.g., linear
regression, random forest, boosting, and bagging) to predict CE.

  * •

F-GCN ($\ddagger$) (Sharma et al. 2023) assembles multiple Graph Convolution
Networks (GCNs) in parallel to intuitively represent formulation constituents,
including no-TL F-GCN and TL F-GCN.

  * •

MoLFormer ($\ddagger$) (Ross et al. 2022) trains an efficient Transformer
encoder on SMILES sequences of 1.1 billion unlabeled molecules, using a linear
attention mechanism and rotational position embedding.

  * •

MultiModal-MoLFormer ($\ddagger$) (Soares et al. 2023) utilizes extensive
chemical information learned in pre-training to predict the performance of
electrolytes, allowing multiple molecules in SMILES format as input.

  * •

BART-SA ($\ddagger$) (Priyadarsini et al. 2024) obtains molecular
representations of individual electrolyte components from the pre-trained BART
model, then scales and sums to form a unified feature vector.

  * •

Uni-ELF ($\ddagger$) (Zeng et al. 2024) is a multi-level representation
learning framework that enhances electrolyte design through two-stage pre-
training.

#### Metrics

We use Root Mean Square Error (RMSE) as the metric that can be calculated by
the following formula:

$\text{RMSE}=\sqrt{\frac{1}{N}\sum_{i=1}^{N}(y_{i}-\hat{y}_{i})^{2}}$ (6)

where $N$ is the number of samples, $y_{i}$ and $\hat{y}_{i}$ are the
experimentally observed and model-predicted LCEs, respectively.

#### Hyperparameters and Training Details

For COEFF-MLP, we set the dimensions of MoLFormer embedding to 768, headers of
attention layers to 12, and the number of encoder layers to 12. The downstream
MLP consists of 2 linear layers with dropout 0.2 and ReLU (Agarap 2019)
activation functions. We optimize the parameters with AdamW (Loshchilov and
Hutter 2019) with batch size 4, learning rate $5\times 10^{-5}$, and epochs
100. For COEFF-KAN, we set 2 KANs and a linear layer in the downstream
network, while other hyperparameters are consistent with COEFF-MLP.

### Main Experimental Results

We conduct comparative experiments to evaluate the effectiveness of each
method in the predicting LCE task. Table 1 shows the RMSE results of all
models on $\mathcal{T}_{\text{in}}$ and $\mathcal{T}_{\text{out}}$ test sets.

Table 1: RMSE results of LCE prediction for different methods, where bold
indicates the best performance and underline denotes the second-best
performance. * stands for the ratio of training set to test set is 8:2, so it
is not included in the comparison.

Methods |  LCE
---|---
$\mathcal{T}_{\text{in}}$ | $\mathcal{T}_{\text{out}}$
Linear Regression | 0.585 | 0.170
Random Forest | 0.577 | -
Boosting | 0.587 | -
Bagging | 0.583 | -
no-TL F-GCN | - | 0.779
TL F-GCN | - | 0.390
MoLFormer | - | 0.213
MultiModal-MoLFormer | - | 0.195
BART-SA* | 0.148 | -
Uni-ELF | 0.184 | -
COEFF-MLP | 0.221 | 0.132
COEFF-KAN | 0.186 | 0.110

First, we observe that our method outperforms machine learning models on all
datasets. Especially on $\mathcal{T}_{\text{in}}$, the RMSE of COEFF-KAN is
0.399, 0.391, 0.401, and 0.397 lower than linear regression, random forest,
boosting, and bagging respectively. The performance of these approaches is
limited by whether the feature selection is appropriate and complete. For
example, Kim et al. (2023) use the elemental composition of liquid
electrolytes as input features. They successfully reveal that a reduction in
the solvent oxygen content is critical for superior CE but overlook structural
information of chemical molecules (e.g., geometric arrangements of elements in
isomers). This subtle distinction has been shown to affect electrolyte
properties and battery performance (Zhang and Viswanathan 2020, 2021).

Figure 4: Parity plots show predicted LCE values as scatterplots with respect
to the actual values.

Second, it is noticed that our method achieves state-of-the-art performance
among all deep learning models on $\mathcal{T}_{\text{out}}$. In particular,
the RMSE of COEFF-KAN is 0.280, 0.103, and 0.085 lower than F-GCN, MoLFormer,
and MultiModal-MoLFormer. Although our method, like most others, follows a
two-stage paradigm, it benefits from the accurate representation of chemical
molecules by MoLFormer, weighted average pooling at the formulation level, and
performance advantages of KANs in the scientific domain. Additionally, unlike
MultiModal-MoLFormer, which uses data augmentation to expand the training set
to 27,266, our model only requires fine-tuning with just over a hundred
samples to achieve more accurate predictions.

Third, experimental results demonstrate that introducing KANs into downstream
networks can better predict the LCE of electrolytes. Specifically, the RMSE of
COEFF-KAN is 0.030 and 0.022 lower than that of COEFF-MLP on
$\mathcal{T}_{\text{in}}$ and $\mathcal{T}_{\text{out}}$, respectively. We
speculate that there are two main reasons: (1) Better interpretability:
Compared to MLPs, design of KANs network structure makes the model’s behavior
easier to understand. By observing the external connection patterns, we can
have an insight into interactions between input features. The internal
activation functions help visualize how a single variable affects the output,
thereby increasing the transparency of models. (2) More efficient learning
with limited data size: Theoretically, KANs can achieve better fitting effects
with fewer samples, because they learn more complex and generalizable function
mappings through the synergy of trainable network structures and activation
functions.

Finally, our method is stable and effective for predicting LCE against unseen
data. As shown in Figure 4, most of the predicted data points for COEFF-MLP
and COEFF-KAN are close to the perfect prediction line and have similar
distributions. The generalization can make COEFF adapt to practical
application scenarios and help chemical researchers reduce the workload of
experimental verification.

### Ablation Experiments

In this section we will perform ablation experiments and analyses in terms of
CIDO and KAN respectively

CIDO. Table 2 reports the ablation with or without the method. In the LCE
prediction task, MLP and KAN use the CIDO method to improve performance on
$T_{out}$ by 0.261 and 0.278, respectively. The results show that using CIDO
can be effective for model prediction while saving some training time overhead
with the same machine and configuration.

Methods | $\mathcal{T}_{\text{out}}$ |  Time(s)
---|---|---
COEFF-MLP w/o CIDO | 0.393 | 715
COEFF-MLP w/ CIDO | 0.132 | 705
COEFF-KAN w/o CIDO | 0.388 | 741
COEFF-KAN w/ CIDO | 0.110 | 725
Table 2: RMSE results of LCE prediction at w/ and w/o CIDO method.

KAN. We used MLP with different depths and widths as a baseline to compare
with KAN. Both MLP and KAN were trained with 100 epochs using CIDO, and the
RMSE of the lowest-loss epoch of the validation set on the test set is plotted
as Figure 5. As can be seen from the figure, testing on the $T_{out}$, we find
that the optimal value of KAN is 17.42% better than that of MLP, thus proving
the effectiveness of the downstream network using KAN. At the same time, we
find that the results of KAN with different widths tend to stabilize with the
network depth RMSE, but MLP performs better at 2-3 layers of the network
architecture, and the performance is gradually unstable as the depth of the
network deepens.

Figure 5: RMSE results of LCE prediction at different network depths and
widths.

## Conclusion and Future Work

In this paper, we propose a novel method COEFF for accurately predicting the
LCE of electrolytes. There are two main contributions:

(1) We adopt a two-stage paradigm: pre-training a chemical general model and
fine-tuning it on domain data.

(2) We innovatively replace the downstream MLPs with KANs.

In the experiments, we show that our method significantly and consistently
outperforms state-of-the-art models on a real-world test set
$\mathcal{T}_{\text{out}}$.

In the future, we will explore the following directions:

(1) We will apply our model to more electrolyte properties (e.g.,
charge/discharge current and battery cycle life), more chemical fields, and
even material science subfields, as the two-stage paradigm is universal.

(2) We will fully utilize the interpretability of KANs to find the
quantitative relationship between electrolyte formulations and CE through
function fitting and visualization.

## References

  * Abolhasani and Kumacheva (2023) Abolhasani, M.; and Kumacheva, E. 2023.  The rise of self-driving labs in chemical and materials sciences.  _Nature Synthesis_ , 2(6): 483–492.
  * Abueidda, Pantidis, and Mobasher (2024) Abueidda, D. W.; Pantidis, P.; and Mobasher, M. E. 2024.  DeepOKAN: Deep Operator Network Based on Kolmogorov Arnold Networks for Mechanics Problems.  arXiv:2405.19143.
  * Agarap (2019) Agarap, A. F. 2019.  Deep Learning using Rectified Linear Units (ReLU).  arXiv:1803.08375.
  * Agre, Molina, and Chu (2017) Agre, P.; Molina, M.; and Chu, S. 2017.  The real climate debate.  _Nature_ , 550(7675): S62–S64.
  * Aykol, Herring, and Anapolsky (2020) Aykol, M.; Herring, P.; and Anapolsky, A. 2020.  Machine learning for continuous innovation in battery technologies.  _Nature Reviews Materials_ , 5(10): 725–727.
  * Bresson et al. (2024) Bresson, R.; Nikolentzos, G.; Panagopoulos, G.; Chatzianastasis, M.; Pang, J.; and Vazirgiannis, M. 2024.  KAGNNs: Kolmogorov-Arnold Networks meet Graph Learning.  arXiv:2406.18380.
  * Castro Nascimento and Pimentel (2023) Castro Nascimento, C. M.; and Pimentel, A. S. 2023.  Do large language models understand chemistry? a conversation with chatgpt.  _Journal of Chemical Information and Modeling_ , 63(6): 1649–1655.
  * Chen, Tao, and Li (2021) Chen, G.; Tao, L.; and Li, Y. 2021.  Predicting polymers’ glass transition temperature by a chemical language processing model.  _Polymers_ , 13(11): 1898.
  * Chen and Bajorath (2023) Chen, H.; and Bajorath, J. 2023.  Designing highly potent compounds using a chemical language model.  _Scientific Reports_ , 13(1): 7412.
  * Cheon (2024) Cheon, M. 2024.  Kolmogorov-Arnold Network for Satellite Image Classification in Remote Sensing.  arXiv:2406.00600.
  * Jablonka et al. (2024) Jablonka, K. M.; Schwaller, P.; Ortega-Guerrero, A.; and Smit, B. 2024.  Leveraging large language models for predictive chemistry.  _Nature Machine Intelligence_ , 6(2): 161–169.
  * Jie et al. (2020) Jie, Y.; Ren, X.; Cao, R.; Cai, W.; and Jiao, S. 2020.  Advanced liquid electrolytes for rechargeable Li metal batteries.  _Advanced Functional Materials_ , 30(25): 1910777.
  * Kiamari, Kiamari, and Krishnamachari (2024) Kiamari, M.; Kiamari, M.; and Krishnamachari, B. 2024.  GKAN: Graph Kolmogorov-Arnold Networks.  arXiv:2406.06470.
  * Kim et al. (2023) Kim, S. C.; Oyakhire, S. T.; Athanitis, C.; Wang, J.; Zhang, Z.; Zhang, W.; Boyle, D. T.; Kim, M. S.; Yu, Z.; Gao, X.; et al. 2023.  Data-driven electrolyte design for lithium metal anodes.  _Proceedings of the National Academy of Sciences_ , 120(10): e2214357120.
  * Kolmogorov (1961) Kolmogorov, A. N. 1961.  _On the representation of continuous functions of several variables by superpositions of continuous functions of a smaller number of variables_.  American Mathematical Society.
  * Li et al. (2024) Li, C.; Liu, X.; Li, W.; Wang, C.; Liu, H.; and Yuan, Y. 2024.  U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation.  arXiv:2406.02918.
  * Liu et al. (2023a) Liu, J.; Zhang, Y.; Zhou, J.; Wang, Z.; Zhu, P.; Cao, Y.; Zheng, Y.; Zhou, X.; Yan, C.; and Qian, T. 2023a.  Advances and prospects in improving the utilization efficiency of lithium for high energy density lithium batteries.  _Advanced Functional Materials_ , 33(34): 2302055.
  * Liu et al. (2024a) Liu, M.; Bian, S.; Zhou, B.; and Lukowicz, P. 2024a.  iKAN: Global Incremental Learning with KAN for Human Activity Recognition Across Heterogeneous Datasets.  arXiv:2406.01646.
  * Liu et al. (2023b) Liu, Y.; Zhang, R.; Li, T.; Jiang, J.; Ma, J.; and Wang, P. 2023b.  MolRoPE-BERT: An enhanced molecular representation with Rotary Position Embedding for molecular property prediction.  _Journal of Molecular Graphics and Modelling_ , 118: 108344.
  * Liu et al. (2024b) Liu, Z.; Wang, Y.; Vaidya, S.; Ruehle, F.; Halverson, J.; Soljačić, M.; Hou, T. Y.; and Tegmark, M. 2024b.  KAN: Kolmogorov-Arnold Networks.  arXiv:2404.19756.
  * Loshchilov and Hutter (2019) Loshchilov, I.; and Hutter, F. 2019.  Decoupled Weight Decay Regularization.  arXiv:1711.05101.
  * Madani et al. (2023) Madani, A.; Krause, B.; Greene, E. R.; Subramanian, S.; Mohr, B. P.; Holton, J. M.; Olmos, J. L.; Xiong, C.; Sun, Z. Z.; Socher, R.; et al. 2023.  Large language models generate functional protein sequences across diverse families.  _Nature Biotechnology_ , 41(8): 1099–1106.
  * Moret et al. (2023) Moret, M.; Pachon Angona, I.; Cotos, L.; Yan, S.; Atz, K.; Brunner, C.; Baumgartner, M.; Grisoni, F.; and Schneider, G. 2023.  Leveraging molecular structure and bioactivity with chemical language models for de novo drug design.  _Nature Communications_ , 14(1): 114.
  * Narayanan Krishnamoorthy et al. (2022) Narayanan Krishnamoorthy, A.; Wölke, C.; Diddens, D.; Maiti, M.; Mabrouk, Y.; Yan, P.; Grünebaum, M.; Winter, M.; Heuer, A.; and Cekic-Laskovic, I. 2022.  Data-Driven Analysis of High-Throughput Experiments on Liquid Battery Electrolyte Formulations: Unraveling the Impact of Composition on Conductivity.  _Chemistry-Methods_ , 2(9): e202200008.
  * Pan (2023) Pan, J. 2023.  Large language model for molecular chemistry.  _Nature Computational Science_ , 3(1): 5–5.
  * Priyadarsini et al. (2024) Priyadarsini, I.; Sharma, V.; Takeda, S.; Kishimoto, A.; Hamada, L.; and Shinohara, H. 2024.  Improving Performance Prediction of Electrolyte Formulations with Transformer-based Molecular Representation Model.  arXiv:2406.19792.
  * Raissi, Perdikaris, and Karniadakis (2019) Raissi, M.; Perdikaris, P.; and Karniadakis, G. E. 2019.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  _Journal of Computational physics_ , 378: 686–707.
  * Ranković and Schwaller (2023) Ranković, B.; and Schwaller, P. 2023.  BoChemian: Large language model embeddings for Bayesian optimization of chemical reactions.  In _NeurIPS_.
  * Ronneberger, Fischer, and Brox (2015) Ronneberger, O.; Fischer, P.; and Brox, T. 2015.  U-net: Convolutional networks for biomedical image segmentation.  In _MICCAI_.
  * Ross et al. (2022) Ross, J.; Belgodere, B.; Chenthamarakshan, V.; Padhi, I.; Mroueh, Y.; and Das, P. 2022.  Large-scale chemical language representations capture molecular structure and properties.  _Nature Machine Intelligence_ , 4(12): 1256–1264.
  * Schwaller et al. (2019) Schwaller, P.; Laino, T.; Gaudin, T.; Bolgar, P.; Hunter, C. A.; Bekas, C.; and Lee, A. A. 2019.  Molecular transformer: a model for uncertainty-calibrated chemical reaction prediction.  _ACS central science_ , 5(9): 1572–1583.
  * Seydi (2024) Seydi, S. T. 2024.  Unveiling the Power of Wavelets: A Wavelet-based Kolmogorov-Arnold Network for Hyperspectral Image Classification.  arXiv:2406.07869.
  * Sharma et al. (2023) Sharma, V.; Giammona, M.; Zubarev, D.; Tek, A.; Nugyuen, K.; Sundberg, L.; Congiu, D.; and La, Y.-H. 2023.  Formulation graphs for mapping structure-composition of battery electrolytes to device performance.  _Journal of Chemical Information and Modeling_ , 63(22): 6998–7010.
  * Shukla et al. (2024) Shukla, K.; Toscano, J. D.; Wang, Z.; Zou, Z.; and Karniadakis, G. E. 2024.  A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networks.  arXiv:2406.02917.
  * Soares et al. (2023) Soares, E.; Sharma, V.; Brazil, E. V.; Cerqueira, R.; and Na, Y.-H. 2023.  Capturing Formulation Design of Battery Electrolytes with Chemical Large Language Model.  In _NeurIPS_.
  * Tian et al. (2020) Tian, Y.; Zeng, G.; Rutt, A.; Shi, T.; Kim, H.; Wang, J.; Koettgen, J.; Sun, Y.; Ouyang, B.; Chen, T.; et al. 2020.  Promises and challenges of next-generation “beyond Li-ion” batteries for electric vehicles and grid decarbonization.  _Chemical reviews_ , 121(3): 1623–1669.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _NeurIPS_.
  * Xu, Chen, and Wang (2024) Xu, K.; Chen, L.; and Wang, S. 2024.  Kolmogorov-Arnold Networks for Time Series: Bridging Predictive Power and Interpretability.  arXiv:2406.02496.
  * Yang, Qin, and Yu (2024) Yang, S.; Qin, L.; and Yu, X. 2024.  Endowing Interpretability for Neural Cognitive Diagnosis by Efficient Kolmogorov-Arnold Networks.  arXiv:2405.14399.
  * Yu et al. (2022) Yu, Z.; Rudnicki, P. E.; Zhang, Z.; Huang, Z.; Celik, H.; Oyakhire, S. T.; Chen, Y.; Kong, X.; Kim, S. C.; Xiao, X.; et al. 2022.  Rational solvent molecule tuning for high-performance lithium metal battery electrolytes.  _Nature Energy_ , 7(1): 94–106.
  * Yuan et al. (2021) Yuan, S.; Kong, T.; Zhang, Y.; Dong, P.; Zhang, Y.; Dong, X.; Wang, Y.; and Xia, Y. 2021.  Advanced electrolyte design for high-energy-density Li-metal batteries under practical conditions.  _Angewandte Chemie_ , 133(49): 25828–25842.
  * Zeng et al. (2024) Zeng, B.; Chen, S.; Liu, X.; Chen, C.; Deng, B.; Wang, X.; Gao, Z.; Zhang, Y.; E, W.; and Zhang, L. 2024.  Uni-ELF: A Multi-Level Representation Learning Framework for Electrolyte Formulation Design.  arXiv:2407.06152.
  * Zhang and Viswanathan (2020) Zhang, Y.; and Viswanathan, V. 2020.  Not All fluorination is the same: Unique effects of fluorine functionalization of ethylene carbonate for tuning solid-electrolyte interphase in Li metal Batteries.  _Langmuir_ , 36(39): 11450–11466.
  * Zhang and Viswanathan (2021) Zhang, Y.; and Viswanathan, V. 2021.  Design rules for selecting fluorinated linear organic solvents for li metal batteries.  _The Journal of Physical Chemistry Letters_ , 12(24): 5821–5828.


# Perception Datasets for Anomaly Detection in
Autonomous Driving: A Survey

Daniel Bogdoll12, Svenja Uhlemeyer34, Kamil Kowol34, J. Marius Zöllner12  1FZI
Research Center for Information Technology, Germany  2Karlsruhe Institute of
Technology, Germany
3University of Wuppertal, Germany
4Interdisciplinary Center for Machine Learning and Data Analytics, Germany

###### Abstract

Deep neural networks (DNN) which are employed in perception systems for
autonomous driving require a huge amount of data to train on, as they must
reliably achieve high performance in all kinds of situations. However, these
DNN are usually restricted to a closed set of semantic classes available in
their training data, and are therefore unreliable when confronted with
previously unseen instances. Thus, multiple perception datasets have been
created for the evaluation of anomaly detection methods, which can be
categorized into three groups: real anomalies in real-world, synthetic
anomalies augmented into real-world and completely synthetic scenes. This
survey provides a structured and, to the best of our knowledge, complete
overview and comparison of perception datasets for anomaly detection in
autonomous driving. Each chapter provides information about tasks and ground
truth, context information, and licenses. Additionally, we discuss current
weaknesses and gaps in existing datasets to underline the importance of
developing further data.

###### Index Terms:

autonomous driving, perception, dataset, anomaly, outlier, out-of-
distribution, novelty, corner case

ML

    Machine Learning
CNN

    Convolutional Neural Network
DL

    Deep Learning
AD

    Autonomous Driving

##  I Introduction

When thinking about autonomous vehicles that move safely through traffic, it
is necessary to perceive the environment correctly in order to provide safe
driving. To ensure this, DNN must be extensively trained and tested with data
required to solve the task. In this context, numerous datasets have been
created for use in road traffic [1, 2, 3], most of which include daytime and
sunny weather and harmless everyday scenes. As numerous new datasets are
published each year [4], it is important to include anomalies, out-of-
distribution (OOD) instances, novelties, outlier, and corner cases, which
primarily describe what is unknown or unusual [5, 6, 7], to improve the
detection and eventually handling of safety-critical driving situations. To
deal with such circumstances, the field of anomaly detection is a highly
active research field [8, 9, 10, 11, 12, 13]. However, most public datasets
follow a closed world assumption [14] and offer no room to detect anomalies.

In this work, to the best of our knowledge, we offer a complete collection of
perception datasets with labeled anomalies in the domain of autonomous
driving. These datasets show a strong focus on object- and scene-level
anomalies, as described by Breitenstein et al. [5]. We visualize the number
and distribution of anomalies and provide insights and research gaps for
future work. We have included datasets, which

  * •

are public and available, as of 01 February 2023

  * •

provide sensor data from the ego-perspective, given that licenses of
potentially utilized datasets allow for that

  * •

include pixel- or point-wise anomaly labels, at least in the form of a small
validation set

Works, which we have excluded, include SiMOOD [15], as they only provide a
framework, but no raw data; TOR4D, Rare4D [16] and FS Web [17], as they are
not public; MUAD [18], DANGER-vKITTI, DANGER-vKITTI2 [19, 20], and FDP-set
[21], as they are not yet published; and WOS [22], as there are no anomaly
labels provided. We also excluded works which focus on adverse conditions,
such as WildDash [23, 24], ACDC [25] or Rain Augmentation [26], as they
classify entire scenes, which does not allow inferring the potential relevance
of anomalies.

Next to perception datasets, there are also trajectory datasets or frameworks
which include anomalies [27], such as R-U-MAAD [28], KING [29] or STRIVE [30].
While such anomalies are among the most challenging ones, those approaches do
not provide sensory perception data, but can only be executed in simulation,
which does not provide unambiguous visual environment representations.

Research Gap. While there are numerous new datasets in the field of autonomous
driving published each year [4], there are only very few works that focus on
dataset analysis in general and even less that focus on the field of anomaly
detection. While there exists a recent overview of anomaly detection methods
[8], there is a lack of structured knowledge related to datasets containing
anomalies, although such anomalies or corner cases are currently core limiting
factors for scaling autonomous vehicles.

Contribution. Our work aims to aid researchers in the field of anomaly
detection to gain an overview of all relevant datasets that include anomalies.
We provide clear selection criteria and point to specifically excluded
datasets for an even broader horizon. We provide detailed, structured
information and visualizations on 16 datasets in their historical order in
Section II. Our survey is the only one of its kind that provides a detailed
overview of currently available perception datasets for anomaly detection. In
Section III, we provide an extensive discussion on similarities, issues, and
research gaps. All code to recreate our visualizations is available on GitHub.

Dataset | Year | Sensors | Size (Test/Val) | Resolution | Anomaly Source | Temporal | #OOD Classes | Groundtruth |
---|---|---|---|---|---|---|---|---|---
Fishyscapes [17, 31, 32] |  |  |  |  |  |  |  |  |
FS Lost and Found | 2019 | Camera | 275 / 100 | $2048\times 1024$ | Recording | ✗ | 1 | Semantic Mask |
FS Static | 2019 | Camera | 1,000 / 30 | $2048\times 1024$ | Data Augmentation | ✗ | 1 | Semantic Mask |
CAOS [33, 34] |  |  |  |  |  |  |  |  |
StreetHazards | 2019 | Camera | 1,500 | $1280\times 720$ | Simulation | ✓ | 1 | Semantic Mask |
BDD-Anomaly | 2019 | Camera | 810 | $1280\times 720$ | Class Exclusion | ✗ | 3 | Semantic Mask |
SegmentMeIfYouCan [35, 36] |  |  |  |  |  |  |  |  |
RoadAnomaly21 | 2021 | Camera | 100 / 10 |  | $2048\times 1024$
---
$1280\times 720$
Web Sourcing | ✗ | 1 | Semantic Mask |
RoadObstacle21 | 2021 | Camera |  | 327 (+55) / 30
---
$1920\times 1080$ | Recording | ✓ | 1 | Semantic Mask |
CODA [37, 38] |  |  |  |  |  |  |  |  |
CODA-KITTI | 2022 | Camera, Lidar | 309 | $1242\times 1376$ | Void Classes | ✗ | 6 | Bounding Boxes |
CODA-nuScenes | 2022 | Camera, Lidar | 134 | $1600\times 900$ | Void Classes | ✗ | 17 | Bounding Boxes |
CODA-ONCE | 2022 | Camera, Lidar | 1,057 | $1920\times 1020$ | Automated OOD Proposal | ✗ | 32 | Bounding Boxes |
CODA2022-ONCE | 2022 | Camera, Lidar | 717 | $1355\times 720$ | Automated OOD Proposal | ✗ | 29 | Bounding Boxes |
CODA2022-SODA10M | 2022 | Camera | 4,167 |  | $1280\times 720$
---
$958\times 720$
Automated OOD Proposal | ✗ | 29 | Bounding Boxes |
Wuppertal OOD Tracking [22, 39, 40] |  |  |  |  |  |  |  |  |
Street Obstacle Sequences (SOS) | 2022 | Camera, Depth |  | 1,129
---
$1920\times 1080$ | Recording | ✓ | 13 | Instance Mask |
CARLA-WildLife (CWL) | 2022 | Camera, Depth | 1,210 | $1920\times 1080$ | Simulation | ✓ | 18 | Instance Mask |
Misc |  |  |  |  |  |  |  |  |
Lost and Found [41, 42] | 2016 | Stereo Cameras | 2,104 | $2048\times 1024$ | Recording | ✓ | 42 | Semantic Mask |
WD-Pascal [43, 44] | 2019 | Camera | 70 | $1920\times 1080$ | Data Augmentation | ✗ | 1 | Semantic Mask |
Vistas-NP [45, 46] | 2020 | Camera | 11,167 | Varying | Class Exclusion | ✗ | 4 | Semantic Mask |

Table I: Overview over all analyzed datasets, clustered by the benchmark in
which they were presented.

##  II Datasets

In autonomous driving, the detection of atypical and dangerous situations is
crucial for the safety of all road users. In order to improve the ability of
today’s models to handle such critical situations, datasets are required that
allow for targeted training and, more importantly, testing with such critical
situations. Therefore, various datasets have emerged in recent years, which we
describe in this section. As shown in Table I, we cluster datasets by their
benchmark and categorize them by their Anomaly Source:

Automated OOD Proposal. This approach allows for the utilization of large,
unlabeled datasets. Here, an automated proposal method is used to generate
first anomaly proposals. This can be done with any anomaly detection approach,
e.g., uncertainty, intermediate detections, geometric priors, or model
contradictions. Subsequently, human experts take care of false positives and
refine the proposals.

Misc Classes. Based on a labeled dataset, all regions which are either labeled
with void or misc can be examined further. These terms are often used
interchangeably and mostly refer to uncommon objects or irrelevant areas.
Human experts then relabel those classes as anomalies, if appropriate.

Class Exclusion. This approach is based on a labeled dataset. Hypothetical
anomalies are created by excluding frames with known classes from the train
and validation splits. A novel test split is created with these, treating the
selected classes as anomalies.

Web Sourcing. In this approach, human experts actively search for images that
include atypical classes. As a reference list for known classes, often
Cityscapes classes [47] are used.

Recording and Simulation. Here, anomalies are recorded through data collection
by driving in the real world [22, 42] or in the synthetic world [33, 48, 49].
Often, anomalies are also not included in the Cityscapes classes.

Data Augmentation. For this technique, any dataset can be used as a baseline.
By synthetic manipulation of scenes [50, 51], anomalies are pasted onto the
original image and can be labeled accordingly. As previously, anomalies are
typically not included in the Cityscapes classes.

In addition to distinguishing between different anomaly methods, we also
differentiate whether datasets provide single frames with anomalies or
scenarios [52] with a temporal context. In Figure 9, we show cumulated anomaly
masks for all datasets. These give an overview of the number of included
anomalies and in which regions of the images they can be found. In the
following chapters, we will introduce each dataset and provide details on the
anomalies, possible tasks, the general context, and license agreements.
Furthermore, we show examples for each dataset, where anomaly and void
instances are overlaid in orange and black, and outlined in green and red
respectively.

###  II-A Lost and Found

The Lost and Found dataset [41] was introduced in 2016 by Pinggera et al.,
being the first dataset with a focus on the detection of small road hazards,
as shown in Figure 1.

####  II-A1 Tasks and Ground truth

The provided stereo masks for the task of semantic segmentation allow for
pixel- and instance-level evaluation, as proposed by the authors. Their
instance-level approach is based on a 3D stixel representation, which is very
method-specific. As the dataset provides data from stereo cameras, geometric
methods can be applied. The anomalies include 42 individual object types that
can realistically be found in a street environment. The objects are
categorized into standard objects, random hazards, emotional hazards as
animals or toys, random non-hazards, and humans and include both static and
dynamic obstacles.

####  II-A2 Context

The data was collected in the greater Stuttgart area, Germany. It includes
irregular road surfaces, large object distances, and illumination changes
[41]. Typical environments include housing areas, parking lots, or industrial
areas [17].

####  II-A3 License

The dataset is “freely available to academic and non-academic entities for
non-commercial purposes” [42].

###  II-B Fishyscapes

The Fishyscapes (FS) benchmark [31] was introduced in 2019 by Blum et al. for
the evaluation of anomaly detection methods in semantic segmentation. While
most of the data is withheld for evaluation, the authors provide validation
sets for the different datasets FS Lost and Found and FS Static. A third FS
Web dataset is completely withheld. The first dataset is a subset of the Lost
and Found dataset [41]. The others are based on the Cityscapes [47] validation
data, overlayed with anomalous objects which are either extracted from the
generic Pascal VOC [53] dataset or crawled from the internet. The FS Static
validation frames are automatically generated from the Cityscapes dataset.

(a)

(b)

Figure 1: Lost and Found: Visualization of exemplary real anomaly types in
real-world scenes.

####  II-B1 Tasks and Ground truth

The FS datasets are designed for the task of semantic segmentation. FS Lost
and Found is enriched with fine-grained binary semantic masks. The refined
annotation of the background is shown by comparing Figure 2 with Figure 1.
Furthermore, sequences, where the anomalous objects are bicycles or children,
are filtered out, as they can be assigned to one of the Cityscapes classes.
For FS Static and FS Web, novel objects are blended into already annotated
scenes from Cityscapes, resulting in fully annotated semantic masks. The
anomalies extracted from the Pascal VOC dataset belong to the classes
_airplane, bird, boat, bottle, cat, chair, cow, dog, horse, sheep, sofa,_ and
_tvmonitor_.

####  II-B2 Context

As all images originate from the Cityscapes or the Lost and Found datasets,
they are recorded at daytime under clear weather conditions. For the augmented
Cityscapes data, this also entails that the street scenes include instances
from known classes, such as humans or other vehicles. Depending on the anomaly
type, the anomalies have a higher probability to appear either on the lower-
or the upper half.

####  II-B3 License

FS is licensed under the Apache License 2.0.

###  II-C CAOS

The Combined Anomalous Object Segmentation (CAOS) benchmark [33] was first
introduced in 2019 by Hendrycks et al. and includes the datasets StreetHazards
and BDD-Anomaly.

(a) FS Lost and Found

(b) FS Static

Figure 2: Fishyscapes: Samples from the val splits, showing real-world scenes
with real (left) and synthetic (right) anomalies.

StreetHazards is based on the CARLA simulation environment [54]. The training
set includes three towns from CARLA. A further town is reserved for the
validation set, and two additional ones are exclusive to the test set. BDD-
Anomaly, on the other hand, is based on the extensive BDD100K dataset [55],
where all instances from several classes were removed from the training and
validation sets and thus treated as anomalies in a novel test set.

####  II-C1 Tasks and Ground truth

Both datasets are designed for the task of semantic segmentation and were
aligned to provide both RGB image data and semantic ground truth in the same
resolution. The StreetHazards dataset provides a wide variety of scenarios. In
total, 250 different anomalies were inserted, taken from the Digimation Model
Bank Library and semantic ShapeNet. A complete list can be found in [33]. The
semantic masks are fully annotated with one additional anomaly class. For BDD-
Anomaly, the three classes motorcycle, train, and bicycle are treated as
anomalies. All frames from the original training and validation sets, which
include these classes, were moved to the novel test set, where they became
anomalies. An example is provided in Figure 3. As the BDD100K dataset provides
fully annotated semantic masks, they are also available for BDD-Anomaly, which
makes the anomalous classes distinguishable. As the validation sets for both
StreetHazards and BDD-Anomaly do not include anomalies, but are only provided
for the regular task of semantic segmentation, we excluded them from Table I.

####  II-C2 Context

For StreetHazards, the CARLA towns show slight variations but follow the same
theme. Different weather and daytime settings are included. While the dataset
provides scenarios, the ego-motion seems to have been performed manually, as
it is rather inconsistent. Most anomalies are not placed in regions with
relevance to the driving task. As BDD-Anomaly is based on BDD100K, it includes
varying sceneries, weather conditions, and times of the day.

####  II-C3 License

CAOS is provided under the MIT license [34]. For BDD100K, “permission to use,
copy, modify, and distribute this software and its documentation for
educational, research, and not-for-profit purposes” [56] is granted.

(a) StreetHazards

(b) BDD-Anomaly

Figure 3: CAOS: Simulated (left) and hypothetical (right) anomalies in the
StreetHazards and BDD-Anomaly test sets.

###  II-D WD-Pascal

WD-Pascal [43] is a small dataset published in 2019, where the WildDash (WD)
dataset [23] was augmented with animals from the PASCAL VOC 2007 dataset [57].

####  II-D1 Tasks and Ground truth

The dataset is produced for the task of semantic segmentation, but the data is
not provided explicitly. As part of the author’s code, it is assembled on the
fly and provided as a PyTorch dataset [44].

####  II-D2 Context

While the dataset is small, the variety remains relatively high due to the
WildDash dataset, as shown in Figure 4. The included animals are not always
complete and vary in size, which often leads to unrealistic augmentations.

####  II-D3 License

The WD-Pascal generation code is provided under a GPL-2.0 license [44]. The
necessary WildDash dataset comes with an extensive license agreement, where
only the intensity images are released under the CC BY-NC 4.0 license [58].
For the PASCAL VOC dataset, no license agreement is mentioned [59]. However,
some images are provided by Flickr, which introduce their own Terms of Use.

###  II-E Vistas-NP

Vistas-NP [45], introduced in 2020, is a large-scale anomaly dataset based on
the Mapillary Vistas dataset [60]. Similarly to BDD-Anomaly, they have
excluded classes from the train and validation splits, creating a novel test
split with hypothetical anomalies. With over 11,000 labeled frames, Vistas-NP
is the largest anomaly dataset to date.

####  II-E1 Tasks and Ground truth

The dataset is designed for the task of semantic segmentation, as visible in
Figure 5. The chosen anomaly classes differ from those in BDD-Anomaly to avoid
visual similarity of anomalous and non-anomalous classes, e.g., _train_ and
_bus_. Hence, a whole category is excluded, which includes all classes
associated with humans.

####  II-E2 Context

The underlying Mapillary Vistas dataset has a large variety. Compared to BDD-
Anomaly, where all images originate from the USA, images from multiple
countries are included. As a crowdsourcing approach is utilized, this is
reflected in a wide variety of resolutions.

####  II-E3 License

The “Vistas-NP dataset should be used under the same conditions as the
original dataset” [46], which is provided under the CC BY-NC-SA 4.0 license
[61].

###  II-F SegmentMeIfYouCan

The SegmentMeIfYouCan benchmark [35] was developed in 2021 by Chan et al.,
introducing two real-world datasets. A previous version of the RoadAnomaly21
dataset was already published in 2019 by Lis et al. [62]. The current version
was both refined and extended. It consists of images collected from the
internet, which show anomalous objects on or near the road. The RoadObstacle21
dataset was recorded by the authors and includes anomalous objects placed on
the road ahead. Similar to FS Lost and Found, which is also included in the
benchmark, these datasets only contain real anomalies.

(a)

(b)

Figure 4: WD-Pascal: Two examples of synthetically inserted anomalies into
real-world scenes.

####  II-F1 Tasks and Ground truth

Both datasets are designed for the task of semantic segmentation, the semantic
masks include binary anomaly labels. RoadAnomaly21 is designed for general
anomaly detection in full street scenes, whereas in RoadObstacle21, the road
is considered the region of interest, i.e., the _not anomaly_ class. Thus,
everything not included in this region is assigned to the _void_ class, which
is represented in Figure 6. The anomalies in RoadAnomaly21 can be categorized
into animals, e.g., _elephant, cow, horse_ , unknown vehicles, e.g.,
_airplane, boat trailer, tractor_ , and others, such as _tent, piano_ , or
_cones_. In RoadObstacle21, each object on the road ahead is considered an
obstacle. However, all obstacles in this dataset also fit the definition of
anomaly as objects which cannot be assigned to the Cityscapes classes.
Semantic masks are in both cases only published for small validation sets.

####  II-F2 Context

In RoadAnomaly21, images are collected from web resources and thus depict a
wide variety of environments and settings. All images are recorded during
daytime and in clear weather. The anomalies can appear anywhere in the image,
even in the sky. Therefore, they are not necessarily street hazards. The
images of RoadObstacle21 are recorded in Germany and Switzerland on seven
different road types, also during daytime and in clear weather. Additionally,
there are $55$ annotated frames by night and in snowy weather conditions.

####  II-F3 License

RoadObstacle21 is provided under the CC BY 4.0, RoadAnomaly21 under different
CC BY licenses.

###  II-G CODA

The CODA datasets, released in 2022, are the first ones that are based
primarily on datasets that include not only camera but also lidar data. They
are divided into the CODA Base and the CODA2022 subsets, which in turn consist
of different underlying datasets, namely KITTI [63], nuScenes [64], ONCE [65],
and SODA10M [66]. CODA Base was described in [37], while CODA2022 was a later
addition [38].

####  II-G1 Tasks and Ground truth

The CODA datasets are designed for object detection with ground truth anomaly
bounding boxes only in the image space, as Figure 7 shows. Common objects are
only labeled in the CODA2022 datasets. For the labeling of anomalies,
different techniques were used. For CODA-ONCE, unknown clusters from the lidar
space were mapped to the image space and those, that could not be classified
by an object detector, remained as anomaly proposals. In a second stage, a
manual process was applied to label the images. This includes proposal
refinement, false positive removal and manual additions. Additionally, pre-
labeling with CLIP [67] was used. Two types of anomalies were considered:
Risky objects, which might block the ego-vehicle, or novel objects, which do
not belong to a typical category. For CODA-KITTI and CODA-nuScenes, only the
second stage was applied, where uncommon classes from the existing labels were
used as proposals, e.g., the misc category in KITTI. The anomalies are grouped
into the categories vehicle, pedestrian, cyclist, animal, traffic facility,
obstruction, and misc. For CODA2022, a pre-labeling process based on FILIP
[68] was employed, followed by a manual process, as SODA10M is an unlabeled
dataset without lidar data. The dataset is in general larger and includes more
anomaly categories. The comparisons in Figure 7 and Figure 9 clearly show the
differences for the CODA-ONCE and CODA2022-ONCE datasets.

(a)

(b)

Figure 5: Vistas-NP: Examples showing humans as hypothetical anomalies in the
Vistas-NP test split.

####  II-G2 Context

As the CODA datasets are based on four different datasets, their variety is
rather high, including multiple countries, weather conditions, and times of
day. For most of the datasets, the majority of the anomalies appear on the
side of the road with little relation to the ego path. Only in
CODA2022-SODA10M, a different picture emerges, where more anomalies can be
found in the center of the images. This is due to the crowdsourcing approach
of the dataset.

####  II-G3 License

As ONCE, SODA10M and CODA are all related to Huawei, their images are included
in the datasets, while KITTI and nuScenes need to be downloaded separately.
The CODA dataset is provided under the CC BY-NC-SA 4.0 license [38]. KITTI is
available under the CC BY-NC-SA 3.0 license, and nuScenes under the CC BY-NC-
SA 4.0 license.

###  II-H Wuppertal OOD Tracking

These datasets were introduced in 2022 by Maag et al., enabling OOD detection
and tracking over video sequences. The Street Obstacle Sequences (SOS) dataset
contains annotated real-world scenes with real anomalies. The CARLA-WildLife
(CWL) dataset is a synthetic dataset similar to StreetHazards, where freely
available assets were inserted as anomalies. A third dataset, Wuppertal
Obstacle Sequences (WOS), consists of real-world, but unlabeled sequences.

####  II-H1 Tasks and Ground truth

SOS and CWL provide labels for the tasks of semantic segmentation, instance
segmentation and depth estimation. The datasets include semantic masks with
binary as well as class-specific anomaly labels. Analogously to
RoadObstacle21, the road represents the region of interest, thus, everything
besides the road is assigned to the _void_ class. Furthermore, both datasets
include instance and depth masks. For SOS, 1,129 out of the 8,994 total frames
were manually labeled. For CWL, also pixel-wise distance masks and fully
annotated semantic masks are available. The anomalies in SOS belong to $13$
anomaly types, e.g., _bag, umbrella_ or _toy_ , the anomalies in CWL to $18$
anomaly types, e.g., _dogs, pylons_ or _bags_.

(a) RoadAnomaly21

(b) RoadObstacle21

Figure 6: SegmentMeIfYouCan: Real-world examples from the RoadAnomaly21 test
and the RoadObstacle21 val splits.

####  II-H2 Context

All images, real and synthetic, were recorded during daytime in clear weather.
The static objects in SOS and CWL are placed in a way that they are mostly
enclosed by the road as shown in Figure 8. In contrast to StreetHazards, the
anomalies in CWL are all placed on the road ahead, which could cause safety-
critical street scenarios. Furthermore, they are more realistic and of higher
quality thanks to a newer version of the Unreal Engine. Besides the ego-
vehicle and the anomalies, there are no other vehicles or humans in the
scenes.

####  II-H3 License

All datasets are provided under CC BY 4.0 licenses. CARLA-WildLife was created
using the Unreal Engine along with CARLA [54], provided under the MIT license.
The assets from Unreal Engine 4.26, which were inserted as anomalies, are
provided under CC BY licenses.

(a) CODA-nuScenes

(b) CODA-ONCE

(c) CODA-KITTI

(d) CODA2022-SODA10M

(e) CODA2022-ONCE

Figure 7: CODA: Annotation of real-world anomalies from CODA Base and
CODA2022, based on different techniques.

##  III Discussion

We have presented a complete overview of 16 perception datasets in the field
of anomaly detection for autonomous driving. We presented many techniques to
define and generate anomalies, which come with certain challenges. In the
following, we provide an extensive discussion of these.

Definition of Normality. There is no clear definition of whether an object is
anomalous or not. However, a common approach is to define anomalies as _none-
of-the-known classes_ with respect to the $19$ Cityscapes evaluation classes.
Most anomaly techniques adhere to the definition of “Cityscapes as normality”:
For web sourcing, simulation, data augmentation and recording, the anomalous
objects are selected to fit into this definition. For the void class
approaches, the definition of anomalous objects depends on the respective
underlying dataset, as the _void_ or _misc_ category isn’t clearly defined,
either. Also for the automated OOD proposal technique, the definition of
normality strongly depends on the underlying classes of the employed
detector(s). Finally, for class exclusion, normality depends on the choice of
excluded classes. The anomalies labeled by this approach are usually not
anomalous with respect to Cityscapes and as such do not represent anomalies
which would be rare in the real world.

Realism. Especially for anomalies which were generated by Simulation or Data
Augmentation, the level of realism can vary strongly, as visible in Figure 9.
For example, anomalies in WD-Pascal and StreetHazards are often placed in
implausible locations or scaled in unrealistic ways. However, such instances
could appear in the real world, e.g., on billboards.

Sensor Data. Perception datasets with anomalies mostly provide camera data.
The CODA Base datasets are the only exception and also include lidar point
clouds; however, no anomaly labels are provided in 3D space. Crowdsourcing
approaches make it hard to transfer detection methods to the perception system
of an autonomous vehicle, as viewpoints vary strongly [69]. Finally,
representations of anomalies in camera data are not actionable for an
autonomous driving system, however, this is a general computer vision issue.

Regular Tasks. Often datasets only provide labels for the anomalies. In this
case only the task of anomaly detection is possible. However, it is generally
of interest to be able to detect anomalies while still performing well at
detecting or segmenting known classes.

(a) CARLA-WildLife

(b) Street Obstacle Sequences

Figure 8: Wuppertal OOD Tracking: Two examples showing simulated (left) and
real (right) anomalies.

(a) Lost and Found

(b) FS Lost and Found (val)

(c) FS Static (val)

(d) StreetHazards

(e) BDD-Anomaly

(f) WD-Pascal

(g) Vistas-NP

(h) RoadAnomaly21

(i) RoadObstacle21

(j) CODA-KITTI

(k) CODA-nuScenes

(l) CODA-ONCE

(m) CODA2022-ONCE

(n) CODA2022-SODA10M

(o) Street Obstacle Sequences

(p) CARLA-WildLife

Figure 9: Cumulated masks of all contained anomalies within the respective
datasets.

Domain Shift. The recording of anomalies in the real world is time-consuming,
as they are rarely present in ordinary street scenes and thus, have to be
selected and placed manually. Furthermore, anomalies that would lead to
dangerous driving situations cannot be captured. Consequently, anomaly
techniques such as data augmentation and simulation emerged to tackle these
issues. Simulation has the advantage of having full control. Thus, anomalies
certainly do not appear in the training data. However, a natural domain gap to
reality exists, so anomaly detection methods that perform well on synthetic
data are not implicitly reliable on real-world data. The same holds for data
augmentation, which mixes two domains, leading to unrealistic results. To
ensure that methods really detect the anomalous objects that are pasted into
the images, Fishyscapes pursues two strategies: Augmenting the Cityscapes
images and pasting in objects from known classes. The first strategy prevents
a method from only detecting pixels that differ from the non-augmented image,
the second indicates whether only the domain shift is identified.

Size. Datasets which include real-world scenes showing anomalies with respect
to Cityscapes, i.e., which are recorded or collected from web resources, are
usually very small. Lost and Found as the largest of these datasets only
provides coarse annotations, followed by the Street Obstacle Sequences
dataset, which however is highly redundant as the frames are extracted from
$20$ video sequences. These datasets are mainly for evaluation purposes and
not for training. They also provide a wide variety of different anomaly types,
which is beneficial for evaluating anomaly detection but is a hindrance for
further processing of these anomalies, e.g., in terms of image retrieval,
clustering and incremental learning. While Vistas-NP and CODA are comparably
large datasets, they are still not comparable to regular perception datasets
with hundreds of thousands of frames.

Similarity. Generating larger datasets as a combination of similar datasets
requires that those have a proper anomaly technique, definition of normality
and labeling policy. Such datasets include FS Lost and Found, RoadAnomaly21,
RoadObstacle21 and SOS for semantic segmentation. In particular, it is not
possible to combine other datasets in a meaningful way due to different
labeling policies.

With this overview of datasets in the field of anomaly detection and mentioned
challenges, we hope to contribute to larger, more diverse, or more specialized
datasets in the future.

## Acknowledgment

This work results from the projects AI Data Tooling (19A20001J, 19A20001E) and
AI Delta Learning (19A19013Q), funded by the German Federal Ministry for
Economic Affairs and Climate Action (BMWK).

## References

  * [1] J. Guo _et al._ , “Is it Safe to Drive? An Overview of Factors, Metrics, and Datasets for Driveability Assessment in Autonomous Driving,” _Transactions on Intelligent Transportation Systems_ , vol. 21, 2020.
  * [2] W. Liu _et al._ , “A Survey on Autonomous Driving Datasets,” in _International Conference on Dependable Systems and Their Applications (DSA)_ , 2021.
  * [3] D. Bogdoll _et al._ , “AD-Datasets: A Meta-Collection of Data Sets for Autonomous Driving,” in _International Conference on Vehicle Technology and Intelligent Transport Systems (VEHITS)_ , 2022.
  * [4] ——, “Impact, Attention, Influence: Early Assessment of Autonomous Driving Datasets,” _arXiv:2301.02200_ , 2023.
  * [5] J. Breitenstein _et al._ , “Systematization of Corner Cases for Visual Perception in Automated Driving,” in _Intelligent Vehicles Symposium (IV)_ , 2020.
  * [6] F. Heidecker _et al._ , “An Application-Driven Conceptualization of Corner Cases for Perception in Highly Automated Driving,” in _Intelligent Vehicles Symposium (IV)_ , 2021.
  * [7] D. Bogdoll _et al._ , “Description of Corner Cases in Automated Driving: Goals and Challenges,” in _International Conference on Computer Vision (ICCV) Workshop_ , 2021.
  * [8] ——, “Anomaly Detection in Autonomous Driving: A Survey,” in _Conference on Computer Vision and Pattern Recognition (CVPR) Workshop_ , 2022\.
  * [9] X. Du _et al._ , “Vos: Learning what you don’t know by virtual outlier synthesis,” in _International Conference on Learning Representations (ICML)_ , 2022.
  * [10] D. Bogdoll _et al._ , “Multimodal Detection of Unknown Objects on Roads for Autonomous Driving,” in _International Conference on Systems, Man and Cybernetics (SMC)_ , 2022.
  * [11] S. Uhlemeyer _et al._ , “Towards Unsupervised Open World Semantic Segmentation,” in _Conference on Uncertainty in Artificial Intelligence (UAI)_ , 2022.
  * [12] D. Bogdoll _et al._ , “Experiments on Anomaly Detection in Autonomous Driving by Forward-Backward Style Transfers,” in _International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME)_ , 2022.
  * [13] R. Chan _et al._ , _Deep Neural Networks and Data for Automated Driving_. Springer, 2022, ch. Detecting and Learning the Unknown in Semantic Segmentation.
  * [14] R. Reiter, _Logic and Data Bases_. Springer, 1978, ch. On Closed World Data Bases.
  * [15] R. Sena Ferreira _et al._ , “SiMOOD: Evolutionary Testing Simulation with Out-Of-Distribution Images,” in _Pacific Rim International Symposium on Dependable Computing (PRDC)_ , 2022.
  * [16] K. Wong _et al._ , “Identifying Unknown Instances for Autonomous Driving,” in _Conference on Robot Learning (CoRL)_ , 2020.
  * [17] H. Blum _et al._ , “Fishyscapes: A Benchmark for Safe Semantic Segmentation in Autonomous Driving,” in _International Conference on Computer Vision (ICCV) Workshop_ , 2019.
  * [18] G. Franchi _et al._ , “Muad: Multiple uncertainties for autonomous driving, a benchmark for multiple uncertainty types and tasks,” in _British Machine Vision Conference (BMVC)_ , 2022.
  * [19] S. Xu and L. H. Gilpin, “DANGER: A Framework of Danger-Aware Novel Dataset Generator Extension for Robustness Test of Machine Learning,” in _BayLearn Machine Learning Symposium_ , 2022.
  * [20] S. Xu _et al._ , “A Framework for Generating Dangerous Scenes for Testing Robustness,” in _NeurIPS Workshop_ , 2022.
  * [21] S. Lee _et al._ , “Fallen person detection for autonomous driving,” _Expert Systems with Applications_ , vol. 213, 2023.
  * [22] K. Maag _et al._ , “Two Video Data Sets for Tracking and Retrieval of Out of Distribution Objects,” in _Asian Conference on Computer Vision (ACCV)_ , 2023.
  * [23] O. Zendel _et al._ , “WildDash - Creating Hazard-Aware Benchmarks,” in _European Conference on Computer Vision (ECCV)_ , 2018.
  * [24] ——, “Unifying Panoptic Segmentation for Autonomous Driving,” in _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * [25] C. Sakaridis _et al._ , “ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding,” in _International Conference on Computer Vision (ICCV)_ , 2021.
  * [26] M. Tremblay _et al._ , “Rain Rendering for Evaluating and Improving Robustness to Bad Weather,” _International Journal of Computer Vision_ , vol. 129, 2021.
  * [27] K. Rösch _et al._ , “Space, Time, and Interaction: A Taxonomy of Corner Cases in Trajectory Datasets for Automated Driving,” in _Symposium Series on Computational Intelligence (SSCI)_ , 2022.
  * [28] J. Wiederer _et al._ , “A benchmark for unsupervised anomaly detection in multi-agent trajectories,” in _International Conference on Intelligent Transportation Systems (ITSC)_ , 2022.
  * [29] N. Hanselmann _et al._ , “KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients,” in _European Conference on Computer Vision (EECV)_ , 2022.
  * [30] D. Rempe _et al._ , “Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior,” in _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * [31] H. Blum _et al._ , “The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation,” _International Journal of Computer Vision (IJCV)_ , vol. 129, 2021.
  * [32] Boyang, Sun and Jiaxu, Xing and Blum, Hermann, “The Fishyscapes Benchmark - Dataset,” https://web.archive.org/web/20220728073515/https://fishyscapes.com/dataset, 2019, accessed: 2023-01-23.
  * [33] D. Hendrycks _et al._ , “Scaling Out-of-Distribution Detection for Real-World Settings,” in _International Conference on Machine Learning (ICML)_ , 2022.
  * [34] Hendrycks, Dan, “Scaling OOD Detection,” https://web.archive.org/web/20230123170629/https://github.com/hendrycks/anomaly-seg, 2022, accessed: 2023-01-23.
  * [35] R. Chan _et al._ , “SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation,” in _Conference on Neural Information Processing Systems (NeurIPS)_ , 2021.
  * [36] Segment Me If You Can, “Datasets,” https://web.archive.org/web/20230123165317/https://segmentmeifyoucan.com/datasets, 2023, accessed: 2023-01-23.
  * [37] K. Li _et al._ , “CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving,” in _European Conference on Computer Vision (ECCV)_ , 2022.
  * [38] Huawei, “CODA Data Usage,” https://web.archive.org/web/20230123164519/https://coda-dataset.github.io/documentation.html, 2022, accessed: 2023-01-23.
  * [39] Chan, Robin, “Video Data Sets for Tracking and Retrieval of Out of Distribution Objects,” https://web.archive.org/web/20230123164838/https://zenodo.org/communities/buw-ood-tracking/?page=1&size=20, 2022, accessed: 2023-01-23.
  * [40] K. Maag, “OOD-Tracking,” https://web.archive.org/web/20230125003841/https://github.com/kmaag/OOD-Tracking, 2022, accessed: 2023-01-25.
  * [41] P. Pinggera _et al._ , “Lost and Found: Detecting Small Road Hazards for Self-Driving Vehicles,” in _International Conference on Intelligent Robots and Systems (IROS)_ , 2016.
  * [42] S. Ramos _et al._ , “LostAndFoundDataset,” https://web.archive.org/web/20230123163224/http://wwwlehre.dhbw-stuttgart.de/~sgehrig/lostAndFoundDataset/index.html, 2016, accessed: 2023-01-23.
  * [43] P. Bevandić _et al._ , “Simultaneous Semantic Segmentation and Outlier Detection in Presence of Domain Shift,” in _German Conference on Pattern Recognition (GCPR)_ , 2019.
  * [44] P. Bevandic, “Simultaneous semantic segmentation and outlier detection,” https://web.archive.org/web/20220709140329/https://github.com/pb-brainiac/semseg_od, 2021, accessed: 2023-01-23.
  * [45] M. Grcić. _et al._ , “Dense Open-set Recognition with Synthetic Outliers Generated by Real NVP,” in _International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISAPP)_ , 2021\.
  * [46] M. Grcić, “Vistas-NP Dataset,” https://web.archive.org/web/20230123164057/https://github.com/matejgrcic/Vistas-NP, 2021, accessed: 2023-01-23.
  * [47] M. Cordts _et al._ , “The Cityscapes Dataset for Semantic Urban Scene Understanding,” _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [48] K. Kowol _et al._ , “A-Eye: Driving with the Eyes of AI for Corner Case Generation,” in _International Conference on Computer-Human Interaction Research and Applications (CHIRA)_ , 2022.
  * [49] D. Bogdoll _et al._ , “One Ontology to Rule Them All: Corner Case Scenarios for Autonomous Driving,” in _European Conference on Computer Vision (ECCV) Workshop_ , 2022.
  * [50] T. Koduri _et al._ , “AUREATE: An Augmented Reality Test Environment for Realistic Simulations,” in _World Congress Experience (WCX)_. SAE International, 2018.
  * [51] T. Genevois _et al._ , “Augmented Reality on LiDAR data: Going beyond Vehicle-in-the-Loop for Automotive Software Validation,” in _Intelligent Vehicles Symposium (IV)_ , 2022.
  * [52] S. Ulbrich _et al._ , “Defining and Substantiating the Terms Scene, Situation, and Scenario for Automated Driving,” in _International Conference on Intelligent Transportation Systems_ , 2015.
  * [53] M. Everingham _et al._ , “The pascal visual object classes (voc) challenge,” _International Journal of Computer Vision_ , vol. 88, 2009.
  * [54] A. Dosovitskiy _et al._ , “Carla: An open urban driving simulator,” in _Conference on Robot Learning (CoRL)_ , 2017.
  * [55] F. Yu _et al._ , “BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning,” _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [56] F. Yu, “BDD100K - License,” https://web.archive.org/web/20230124225455/https://doc.bdd100k.com/license.html, 2022, accessed: 2023-01-16.
  * [57] M. Everingham _et al._ , “The Pascal Visual Object Classes (VOC) Challenge,” _International Journal of Computer Vision_ , vol. 88, 2010.
  * [58] AIT, “Wilddash - License Agreements,” https://web.archive.org/web/20230126165451/https://wilddash.cc/license/wilddash, 2023, accessed: 2023-01-26.
  * [59] M. Everingham _et al._ , “The PASCAL Visual Object Classes Challenge 2007,” https://web.archive.org/web/20221107195921/http://host.robots.ox.ac.uk/pascal/VOC/voc2007/, 2008, accessed: 2023-01-26.
  * [60] G. Neuhold _et al._ , “The mapillary vistas dataset for semantic understanding of street scenes,” in _International Conference on Computer Vision (ICCV)_ , 2017.
  * [61] Mapillary, “Mapillary Vistas Dataset,” https://web.archive.org/web/20230126123653/https://www.mapillary.com/dataset/vistas, 2022, accessed: 2023-01-26.
  * [62] K. Lis _et al._ , “Detecting the unexpected via image resynthesis,” _International Conference on Computer Vision (ICCV)_ , 2019.
  * [63] A. Geiger _et al._ , “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite,” in _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2012.
  * [64] H. Caesar _et al._ , “nuScenes: A multimodal dataset for autonomous driving,” in _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [65] J. Mao _et al._ , “One Million Scenes for Autonomous Driving: ONCE Dataset,” in _NeurIPS Track on Datasets and Benchmarks_ , 2021.
  * [66] J. Han _et al._ , “SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Driving,” in _NeurIPS Track on Datasets and Benchmarks_ , 2021.
  * [67] A. Radford _et al._ , “Learning Transferable Visual Models From Natural Language Supervision,” in _International Conference on Machine Learning (ICML)_ , vol. 139, 2021.
  * [68] L. Yao _et al._ , “FILIP: Fine-grained Interactive Language-Image Pre-Training,” in _International Conference on Learning Representations (ICLR)_ , 2022.
  * [69] H. Reichert _et al._ , “Towards Sensor Data Abstraction of Autonomous Vehicle Perception Systems,” in _International Smart Cities Conference (ISC2)_ , 2021.