# Inspect, Understand, Overcome:
A Survey of Practical Methods
for AI Safety

Sebastian Houben  Fraunhofer Institute for Intelligent Analysis and
Information Systems  Stephanie Abrecht  Robert Bosch GmbH  Maram Akila
Fraunhofer Institute for Intelligent Analysis and Information Systems  Andreas
Bär  Technische Universität Braunschweig  Felix Brockherde  umlaut AG  Patrick
Feifel  Opel Automobile GmbH  Tim Fingscheidt  Technische Universität
Braunschweig  Sujan Sai Gannamaneni  Fraunhofer Institute for Intelligent
Analysis and Information Systems  Seyed Eghbal Ghobadi  Opel Automobile GmbH
Ahmed Hammam  Opel Automobile GmbH  Anselm Haselhoff  Hochschule Ruhr West
Felix Hauser  Karlsruhe Institute of Technology  Christian Heinzemann  Robert
Bosch GmbH  Marco Hoffmann  QualityMinds GmbH  Nikhil Kapoor  Volkswagen AG
Falk Kappel  ZF Friedrichshafen AG  Marvin Klingner  Technische Universität
Braunschweig  Jan Kronenberger  Hochschule Ruhr West  Fabian Küppers
Hochschule Ruhr West  Jonas Löhdefink  Technische Universität Braunschweig
Michael Mlynarski  QualityMinds GmbH  Michael Mock  Fraunhofer Institute for
Intelligent Analysis and Information Systems  Firas Mualla  ZF Friedrichshafen
AG  Svetlana Pavlitskaya  FZI Research Center for Information Technology
Maximilian Poretschkin  Fraunhofer Institute for Intelligent Analysis and
Information Systems  Alexander Pohl  QualityMinds GmbH  Varun Ravi-Kumar
Valeo S.A.  Julia Rosenzweig  Fraunhofer Institute for Intelligent Analysis
and Information Systems  Matthias Rottmann  University of Wuppertal  Stefan
Rüping  Fraunhofer Institute for Intelligent Analysis and Information Systems
Timo Sämann  Valeo S.A.  Jan David Schneider  Volkswagen AG  Elena Schulz
Fraunhofer Institute for Intelligent Analysis and Information Systems  Gesina
Schwalbe  Continental AG  Joachim Sicking  Fraunhofer Institute for
Intelligent Analysis and Information Systems  Toshika Srivastava  Audi AG
Serin Varghese  Volkswagen AG  Michael Weber  FZI Research Center for
Information Technology  Sebastian Wirkert  Bayerische Motorenwerke AG  Tim
Wirtz  Fraunhofer Institute for Intelligent Analysis and Information Systems
Matthias Woehrle  Robert Bosch GmbH

###### Abstract

The use of deep neural networks (DNNs) in safety-critical applications like
mobile health and autonomous driving is challenging due to numerous model-
inherent shortcomings. These shortcomings are diverse and range from a lack of
generalization over insufficient interpretability to problems with malicious
inputs. Cyber-physical systems employing DNNs are therefore likely to suffer
from _safety concerns_. In recent years, a zoo of state-of-the-art techniques
aiming to address these safety concerns has emerged. This work provides a
structured and broad overview of them. We first identify categories of
insufficiencies to then describe research activities aiming at their
detection, quantification, or mitigation. Our paper addresses both machine
learning experts and safety engineers: The former ones might profit from the
broad range of machine learning (ML) topics covered and discussions on
limitations of recent methods. The latter ones might gain insights into the
specifics of modern ML methods. We moreover hope that our contribution fuels
discussions on desiderata for ML systems and strategies on how to propel
existing approaches accordingly.

###### Contents

  1. 1 Introduction
  2. 2 Dataset Optimization
    1. 2.1 Outlier/Anomaly Detection
    2. 2.2 Active Learning
    3. 2.3 Domains
    4. 2.4 Augmentation
    5. 2.5 Corner Case Detection
  3. 3 Robust Training
    1. 3.1 Hyperparameter Optimization
    2. 3.2 Modification of Loss
    3. 3.3 Domain Generalization
  4. 4 Adversarial Attacks
    1. 4.1 Adversarial Attacks and Defenses
    2. 4.2 More Realistic Attacks
  5. 5 Interpretability
    1. 5.1 Visual Analytics
    2. 5.2 Intermediate Representations
    3. 5.3 Pixel Attribution
    4. 5.4 Interpretable Proxies
  6. 6 Uncertainty
    1. 6.1 Generative Models
    2. 6.2 Monte-Carlo Dropout
    3. 6.3 Bayesian Neural Networks
    4. 6.4 Uncertainty Metrics for DNNs in Frequentist Inference
    5. 6.5 Markov Random Fields
    6. 6.6 Confidence Calibration
  7. 7 Aggregation
    1. 7.1 Ensemble Methods
    2. 7.2 Temporal Consistency
  8. 8 Verification
    1. 8.1 Formal Testing
    2. 8.2 Black Box Methods
  9. 9 Architecture
    1. 9.1 Building Blocks
    2. 9.2 Multi-Task Networks
    3. 9.3 Neural Architecture Search
  10. 10 Model Compression
    1. 10.1 Pruning
    2. 10.2 Quantization
  11. 11 Discussion

##  1 Introduction

MMSebastian Houben1, Michael Mock1, Timo Sämann4, Gesina Schwalbe3, Joachim
Sicking1

In barely a decade, deep neural networks (DNNs) have revolutionized the field
of machine learning by reaching unprecedented, sometimes superhuman,
performances on a growing variety of tasks. Many of these neural models have
found their way into consumer applications like smart speakers, machine
translation engines or content feeds. However, in safety-critical systems,
where human life might be at risk, the use of recent DNNs is challenging as
various model-immanent insufficiencies are yet difficult to address.

This paper summarizes the promising lines of research in how to identify,
address, and at least partly mitigate these DNN insufficiencies. While some of
the reviewed works are theoretically grounded and foster the overall
understanding of training and predictive power of DNNs, others provide
practical tools to adapt their development, training or predictions. We refer
to any such method as a _safety mechanism_ if it addresses one or several
safety concerns in a feasible manner. Their effectiveness in mitigating safety
concerns is assessed by _safety metrics_ [OOAG19, CNH+18, SS20a, BGS+19]. As
most safety mechanisms target only a particular insufficiency, we conclude
that a _holistic safety argumentation_[SSH20, SS20a, BGS+19, WSRA20] for a
complex DNN-based systems will in many cases rely on a variety of safety
mechanisms.

We structure our review of these mechanisms as follows: Chapter 2 focuses on
_dataset optimization_ for network training and evaluation. It is motivated by
the well-known fact that, in comparison to humans, DNNs perform poorly on data
that is structurally different from training data. Apart from insufficient
generalization capabilities of these models, the data acquisition process and
distributional data shifts over time play vital roles. We survey potential
counter-measures, e.g., augmentation strategies and outlier detection
techniques.

Mechanisms that improve on _robustness_ are described in Chapters 3 and 4,
respectively. They deserve attention as DNNs are generally not resilient to
common perturbations and adversarial attacks.

Chapter 5 addresses incomprehensible network behavior and reviews mechanisms
that aim at _explainability_ , e.g., a more transparent functioning of DNNs.

Moreover, DNNs tend to overestimate their prediction confidence, especially on
unseen data. Straightforward ways to estimate prediction confidence yield
mostly unsatisfying results. Among others, this observation fuelled research
on more sophisticated _uncertainty estimations_ (see Chapter 6), _redundancy
mechanisms_ (see Chapter 7) and attempts to reach _formal verification_ as
addressed in Chapter 8.

At last, many safety-critical applications require not only accurate but also
near real-time decisions. This is covered by mechanisms on the DNN
_architectural level_ (see Chapter 9) and furthermore by _compression_ and
_quantization_ methods (see Chapter 10).

We conclude this review of mechanism categories with an outlook on the steps
to transfer a carefully arranged combination of safety mechanisms into an
actual holistic safety argumentation.

##  2 Dataset Optimization

MMMatthias Rottmann5

The performance of a trained model inherently relies on the nature of the
underlying dataset. For instance, a dataset with poor variability will hardly
result in a model ready for real-world applications. In order to approach the
latter, data selection processes such as corner case selection and active
learning are of utmost importance. These approaches can help to design
datasets that contain the most important information, while preventing the so
much desired information from getting lost in an ocean of data. For a given
dataset and active learning setups, data augmentation techniques are very
common aiming at extracting as much model performance out of the dataset as
possible.

On the other hand, safety arguments also require the analysis of how a model
behaves on out-of-distribution data, data that contains concepts the model has
not encountered during training. This is quite likely to happen as our world
is under constant change, in other words exposed to a constantly growing
domain shift. Therefore, these fields are lately gaining interest, also with
respect to perception in automated driving.

###  2.1 Outlier/Anomaly Detection

MMSujan Sai Gannamaneni1, Matthias Rottmann5

The terms anomaly, outlier and _out-of-distribution_ (OOD) data detection are
often used interchangeably in literature and refer to task of identifying data
samples that are not representative of training data distribution. Uncertainty
evaluation (cf. Chapter 6) is closely tied to this field as self-evaluation of
models is one of the active areas of research for OOD detection. In
particular, for image classification problems it has been reported that neural
networks often produce high confidence predictions on OOD data [NYC15, HG17].
The detection of such OOD inputs can either be tackled by post-processing
techniques that adjust the estimated confidence [LLS18, DT18] or by enforcing
low confidence on OOD samples during training [HAB19, HMD19]. Even guarantees
that neural networks produce low confidence predictions for OOD samples can be
provided under specific assumptions (cf. [MH20b]). More precisely, this work
utilizes Gaussian mixture models that, however, may suffer from high-
dimensional data and require strong assumptions on the distribution
parameters. Some approaches use generative models like _GANs_ [SSW+17, AAAB18]
and _autoencoders_ [ZP17] for outlier detection. The models are trained to
learn in-distribution data manifolds and will produce higher reconstruction
loss for outliers.

For OOD detection in semantic segmentation, only a few works have been
presented so far. Angus et al.[ACS19] present a comparative study of common
OOD detection methods, which mostly deal with image-level classification. In
addition, they provide a novel setup of relevant OOD datasets for this task.
Another work trains a fully convolutional binary classifier that distinguishes
image patches from a known set of classes from image patches stemming from an
unknown class [BKOŠ18]. The classifier output applied at every pixel will give
the per-pixel confidence value for an OOD object. Both of these works perform
at pixel level and without any sophisticated feature generation methods
specifically tailored for the detection of entire OOD instances. Up to now,
outlier detection has not been studied extensively for object detection tasks
based on benchmark object detection datasets. In [GBA+19], two CNNs are used
to perform object detection and binary classification (benign or anomaly) in a
sequential fashion, where the second CNN takes the localized object within the
image as input.

From a safety standpoint, detecting outliers or OOD samples is extremely
important and beneficial as training data cannot realistically be large enough
to capture all situations. Research in this area is heavily entwined with
progress in uncertainty estimation (cf. Chapter 6) and domain adaptation (cf.
Sec. 2.3). Extending research works to segmentation and object detection tasks
would be particularly significant for leveraging autonomous driving research.
In addition to safety, OOD detection can be beneficial in other aspects like
when using local expert models. For example, when using an expert model for
segmentation of urban driving scenes and another expert model for segmentation
of highway driving scenes, an OOD detector could act as trigger on which
models can be switched.

With respect to the approaches presented above, uncertainty-based and
generative model-based OOD detection methods are currently promising
directions of research. However, it remains an open question whether they can
unfold their potential well on segmentation and object detection tasks.

###  2.2 Active Learning

MMMatthias Rottmann5

It is widely known that, as a rule of thumb, for the training of any kind of
artificial neural network, an increase of training data leads to increased
performance. Obtaining labeled training data, however, is often very costly
and time consuming. _Active learning_ provides one possible remedy to this
problem: Instead of labeling every data point, active learning utilizes a
_query strategy_ to request labels from a teacher/an oracle which leverage the
model performance most. The survey paper by Settles [Set10] provides a broad
overview regarding query strategies for active learning methods. However,
except for _uncertainty sampling_ and _query by committee_ , most of them seem
to be infeasible in deep learning applications up to now. Hence, most of the
research activities in active deep learning focus on these two query
strategies, as we outline in the following.

It has been shown [GIG17, RKG18] for image classification that labels
corresponding to uncertain samples can leverage the networks’ performance
significantly and that a combination with semi-supervised learning is
promising. In both works, uncertainty of unlabeled samples is estimated via
Monte Carlo (MC) dropout inference. MC dropout inference and a chosen number
of training epochs are executed alternatingly, after performing MC dropout
inference, the unlabeled samples’ uncertainties are assessed by means of
sample-wise dispersion measures. Samples for which the DNN model is very
uncertain about its prediction are presented to an oracle and labeled.

With respect to object detection, a moderate number of active learning methods
has been introduced [BKD19, KLSL19, RUN18, DCG+19]. These approaches include
uncertainty sampling [BKD19, KLSL19] and query-by-committee methods [RUN18].
In [KLSL19, DCG+19], additional algorithmic features specifically tailored for
object detection networks are presented, i.e., separate treatment of the
localization and classification loss [KLSL19], as well as weak and strong
supervision schemes [DCG+19]. For semantic segmentation, an uncertainty-
sampling-based approach has been presented [MLG+18], which queries polygone
masks for image sections of a fixed size ($128\times 128$). Queries are
performed by means of accumulated entropy in combination with a cost
estimation for each candidate image section.

Recently, new methods for estimating the quality of a prediction [DT18,
RCH+18] as well as new uncertainty quantification approaches, e.g., gradient-
based ones [ORG18], have been proposed. It remains an open question whether
they are suitable for active learning. Since most of the conducted studies are
rather of academic nature, also their applicability to real-life data
acquisition is not yet demonstrated sufficiently. In particular, it is not
clear whether the proposed active learning schemes, including the label
acquistion, for instance in semantic segmentation, is suitable to be performed
by human labelers. Therefore, labeling acquisition with a common understanding
of the labelers’ convenience and suitability for active learning are a
promising direction for research and development.

###  2.3 Domains

MMJulia Rosenzweig1

The classical assumption in machine learning is that the training and testing
data sets are drawn from the same distribution, implying that the model is
deployed under the same conditions as it was trained under. However, as
[MTRA+12, JDCR12] mention, in real-world applications this assumption is often
violated in the sense that the training and the testing set stem from
different domains having different distributions. This poses difficulties for
statistical models and the performance will mostly degrade when they are
deployed on a domain $D^{\mathrm{test}}$, having a different distribution than
the training dataset (i.e., generalizing from the training to the testing
domain is not possible). This makes the study of domains not only relevant
from the machine learning perspective, but also from a safety point of view.

More formally, there are differing notions of a ’domain’ in literature. For
[Csu17b, MD18], a domain $\mathcal{D}=\\{\mathcal{X},P(X)\\}$ consists of a
feature space $\mathcal{X}\subset\mathbb{R}^{d}$ together with a marginal
probability distribution $P(X)$ with $X\in\mathcal{X}$. In [BCK+08, BDBC+10],
a domain is a pair consisting of a distribution over the inputs together with
a labeling function. However, instead of a sharp labeling function, it is also
widely accepted to define a (training) domain
$\mathcal{D}=\\{(x_{i},y_{i})\\}_{i=1}^{n}$ to consist of $n$ (labeled)
samples that are sampled from a joint distribution $P(x,y)$ (cf. [LCWJ18]).

The reasons for _distributional shift_ are diverse—as are the names to
indicate a shift. For example, if the rate of (class) images of interest is
different between training and testing set this can lead to a domain gap and,
e.g., result in differing overall error rates. Moreover, as [CLS+18] mentions,
changing weather conditions and camera setups in cars lead to a domain
mismatch in applications of autonomous driving. In biomedical image analysis,
different imaging protocols and diverse anatomical structures can hinder
generalization of trained models (cf. [DCO+19, KBL+17]). Common terms to
indicate distributional shift are _domain shift, dataset shift, covariate
shift, concept drift, domain divergence, data fracture, changing environments_
or _dataset bias_. References [Sto09, MTRA+12] provide an overview.

Methods and measures to overcome the problem of domain mismatch between one or
more (cf. [ZZW+18]) source domains and target domain(s) and the resulting poor
model performance are studied in the field of transfer learning and in
particular its subtopic domain adaptation (cf. [MD18]). For instance, adapting
a model that is trained on synthetically generated data to work on real data
is one of the core challenges, as can be seen [CLS+18, LZG+19, VJB+19].
Furthermore, detecting when samples are out-of-domain or out-of-distribution
is an active field of research (cf. [LLLS18] and the outlier/anomaly detection
in Sec. 2.1 as well as the topic of observers in the black-box methods in Sec.
8.2 for further reference). This is particularly relevant for machine learning
models that operate in the real world: If, e.g., an autonomous vehicle
encounters some situation that deviates strongly from what was seen during
training (e.g., due to some special event like a biking competition, carnival,
etc.) this can lead to wrong predictions and thereby potential safety issues
if not detected in time.

###  2.4 Augmentation

MMFalk Kappel13

Given the need for big amounts of data to train neural networks, one often
runs into a situation where data is lacking. This can lead to insufficient
generalization and an overfitting to the training data. An overview over
different techniques to tackle this challenge can be found in [KGC17]. One
approach to try and overcome this issue is the augmentation of data. It aims
at optimizing available data and increasing its amount, curating a dataset
that represents a wide variety of possible inputs during deployment.
Augmentation can as well be of help when having to work with a heavily
unbalanced dataset by creating more samples of underrepresented classes. A
broad survey on data augmentation is provided by [SK19]. They distinguish
between two general approaches to data augmentation with the first one being
data warping augmentations that focus on taking existing data and transforming
it in a way that does not effect labels. The other option are oversampling
augmentations, which create synthetic data that can be used to increase the
size of the dataset.
Examples of some of the most basic augmentations are flipping, cropping,
rotating, translating, shearing and zooming. These are affecting the geometric
properties of the image and are easily implemented [SK19]. The machine
learning toolkit Keras, for example, provides an easy way of applying them to
data using their ImageDataGenerator class [C+15]. Other simple methods include
adaptations in color space that affect properties such as lighting, contrast
and tints, which are common variations within image data. Filters can be used
to control increased blur or sharpness [SK19]. In [ZZK+17] random erasing is
introduced as a method with similar effect as cropping, aiming at gaining
robustness against occlusions. An example for mixing images together as an
augmentation technique can be found in [Ino18].
The abovementioned methods have in common that they work on the input data but
there are different approaches that make use of deep learning for
augmentation. An example for making augmentations in feature space using
autoencoders can be found in [DT17]. They use the representation generated by
the encoder and generate new samples by interpolation and extrapolation
between existing samples of a class. The lack of interpretability of
augmentations in feature space in combination with the tendency to perform
worse than augmentations in image space present open challenges for those
types of augmentations [SK19, WGSM16]. Adversarial training is another method
that can be used for augmentation. The goal of adversarial training is to
discover cases that would lead to wrong predictions. That means the augmented
images won’t necessarily represent samples that could occur during deployment
but that can help in achieving more robust decision boundaries [SK19]. An
example of such an approach can be found in [LCPB18]. Generative modelling can
be used to generate synthetic samples that enlarge the dataset in a useful way
with GANs, variational autoencoders and the combination of both are important
tools in this area [SK19]. Examples for data augmentation in medical context
using a CycleGAN [ZPIE17] can be found in [SYPS19] and using a progressively
growing GAN [KALL17] in [BCG+18]. Next to neural style transfer [GEB15] that
can be used to change the style of an image to a target style, AutoAugment
[CZM+19] and population based augmentation [HLS+19] are two more interesting
publications. In the latter two, the idea is to search a predefined search
space of augmentations to gather the best selection.

###  2.5 Corner Case Detection

MMAlexander Pohl16, Marco Hoffmann16, Michael Mlynarski16, Timo Sämann4

Ensuring that AI-based applications behave correctly and predictably even in
unexpected or rare situations is a major concern that gains importance
especially in safety-critical applications such as autonomous driving. In the
pursuit of more robust AI corner cases play an important role.

The meaning of the term corner case varies in the literature. Some consider
mere erroneous or incorrect behavior as corner cases [ZHML19, TPJR18, PCYJ17].
For example, in [BBLF19] corner cases are referred to as situations in which
an object detector fails to detect relevant objects at relevant locations.
Others characterize corner cases mainly as rare combinations of input
parameter values [HDHH20, KKB18]. This project adopts the first definition:
Inputs that result in unexpected or incorrect behaviour of the AI function are
defined as corner cases.

Contingent on the hardware, the AI architecture and the training data, the
search space of corner cases quickly becomes incomprehensibly large. While
manual creation of corner cases (e.g., constructing or re-enacting scenarios)
might be more controllable, approaches that scale better and allow for a
broader and more systematic search for corner cases require extensive
automation.

One approach to automatic corner case detection is based on transforming the
input data. The _DeepTest_ framework [TPJR18] uses three types of image
transformations: linear, affine and convolutional transformations. In addition
to these transformations, metamorphic relations help detect undesirable
behaviors of deep learning systems. They allow changing the input while
asserting some characteristics of the result [XHM+11]. For example, changing
the contrast of input frames should not affect the steering angle of a car
[TPJR18]. Input-output pairs that violate those metamorphic relations can be
considered as corner cases.

Among other things, the white-box testing framework _DeepXplore_ [PCYJ17]
applies a method called _gradient ascent_ to find corner cases (cf. Sec. 8.1).
In the experimental evaluation of the framework, three variants of deep
learning architectures were used to classify the same input image. The input
image was then changed according to the gradient ascent of an objective
function that reflected the difference in the resulting class probabilities of
the three model variants. When the changed (now artificial) input resulted in
different class label predictions by the model variants, the input was
considered as a corner case.

In [BBLF19], corner cases are detected on video sequences by comparing
predicted with actual frames. The detector has three components: The first
component, semantic segmentation, is used to detect and locate objects in the
input frame. As the second component, an image predictor trained on frame
sequences predicts the actual frame based on the sequence preceding that
frame. An error is determined by comparing the actual with the predicted
(i.e., expected) frame, following the idea that only situations that are
unexpected for AI-based perception functions may be potentially dangerous and
therefore a corner case. Both the segmentation and the prediction error are
then fed into the third component of the detector, which determines a corner
case score that reflects the extent to which unexpected relevant objects are
at relevant locations.

In [HDHH20], a corner case detector based on simulations in a _Carla_
environment [DRC+17] is presented. In the simulated world, AI agents control
the vehicles. During simulations, state information of both the environment
and the AI agents are fed into the corner case detector. While the environment
provides the real vehicle states, the AI agents provide estimated and
perceived state information. Both sources are then compared to detect
conflicts (e.g., collisions). These conflicts are recorded for analysis.

Several ways of automatically generating and detecting corner cases exist.
However, corner case detection is a task with challenges of its own: Depending
on the operational domain including its boundaries, the space of possible
inputs can be very large. Also, some types of corner cases are specific to the
AI architecture, e.g., the network type or the network layout used. Thus,
corner case detection has to assume a holistic point of view on both model and
input, adding further complexity and reducing transferability of previous
insights.

Although it can be argued that rarity does not necessarily characterize corner
cases, rare input data might have the potential of challenging the AI
functionality (cf. Sec. 2.1). Another research direction could investigate
whether structuring the input space in a way suitable for the AI functionality
supports the detection of corner cases. Provided that the operational domain
is conceptualized as an ontology, ontology-based testing [BMM18] may support
automatic detection. A properly adapted generator may specifically select
promising combinations of extreme parameter values and, thus, provide valuable
input for synthetic test data generation.

##  3 Robust Training

MMNikhil Kapoor7

Recent works [AW18, HD19, RSFD16, ETTS19, BRW18, BHSFs19, FF15] have shown
that state-of-the-art deep neural networks (DNNs) performing a wide variety of
computer vision tasks such as image classification [KSH12, HZRS15, MGR+18],
object detection [Gir15, RDGF15, HGDG17] and semantic segmentation [CPSA17,
ZSR+19, WSC+19, LBS+19] are not robust to small changes in the input.

Robustness of neural networks is an active and open research field that can be
considered highly relevant for achieving safety in autonomous driving.
Currently, most of the research is directed towards either improving
adversarial robustness [SZS+14] (robustness against carefully designed
perturbations that aim at causing misclassifications with high confidence), or
improving corruption robustness [HD19] (robustness against commonly occurring
augmentations such as weather changes, addition of Gaussian noise, photometric
changes, etc.). While adversarial robustness might be more of a security issue
than a safety issue, corruption robustness, on the other hand, can be
considered highly safety-relevant.

Equipped with these definitions, we broadly term _robust training_ here as
methods or mechanisms that aim at improving either adversarial or corruption
robustness of a DNN, by incorporating modifications into the architecture or
into the training mechanism itself.

###  3.1 Hyperparameter Optimization

MMSeyed Eghbal Ghobadi8, Patrick Feifel8

The final performance of a neural network depends highly on the learning
process. The process includes the actual optimization and may additionally
introduce training methods such as dropout, regularization, or parametrization
of a multi-task loss.

These methods adapt their behavior for predefined parameters. Hence, their
optimal configuration is a priori unknown. We refer to them as
_hyperparameters_. Important hyperparameters comprise, for instance, the
initial learning rate, steps for learning rate reduction, learning rate decay,
momentum, batch size, dropout rate and number of iterations. Their
configuration has to be determined according to the architecture and task of
the CNN [HKV19]. The search of an optimal hyperparameter configuration is
called hyperparameter optimization (HO).

HO is usually described as an optimization problem [HKV19]. Thereby, the
combined configuration space is defined as
$\boldsymbol{\Lambda}=\lambda_{1}\times\lambda_{2}\times\cdots\lambda_{N}$,
according to each domain $\lambda_{n}$. Their individual spaces can be
continuous, discrete, categorical or binary.

Hence, we aim to find an optimal hyperparameter configuration
$\lambda^{\star}$ by minimizing an objective function
$\mathcal{O}\left(\right)$, which evaluates a trained model $\mathcal{M}$ on
the validation dataset $\mathcal{D}^{\textrm{val}}$ with the loss
$\mathcal{L}$:

$\lambda^{\star}=\text{arg
min}_{\lambda\in\boldsymbol{\Lambda}}\;\mathcal{O}\left(\mathcal{L},\mathcal{M}_{\lambda},\mathcal{D}^{\textrm{train}},\mathcal{D}^{\textrm{val}}\right)$
(1)

This problem statement is widely regarded in traditional machine learning and
primarily based on Bayesian optimization (BO) in combination with Gaussian
processes. However, a straightforward application to deep neural networks
encounters problems due to a _lack of scalability, flexibility and robustness_
[ZCY+19], [FKH18].

To exploit the benefits of BO, many authors proposed different combinations
with other approaches. Hyperband [LJD+17] in combination with BO (BOHB)
[FKH18] frames the optimization as “… a pure exploration non-stochastic
infinite-armed bandit problem …”. The method of BO for iterative learning
(BOIL) [NSO19] internalizes iteratively collected information about the
learning curve and the learning algorithm itself. The authors of [WTPFW19]
introduce the trace-aware knowledge gradient (taKG) as an acquisition function
for BO (BO-taKG) which “leverages both trace information and multiple fidelity
controls”. Thereby BOIL and BO-taKG achieve state-of-research performance
regarding CNNs outperforming Hyperband.

Other approaches such as the orthogonal array tuning method (OATM) [ZCY+19] or
HO by reinforcement learning (Hyp-RL) [JGST19] turn away from the Bayesian
approaches and offer new research directions.

Finally, the insight that many authors include kernel sizes and number of
kernels and layers in their hyperparameter configuration should be emphasized.
More work should be spent on the distinct integration of HO in the performance
estimation strategy of neural architecture search (cf. Sec. 9.3).

###  3.2 Modification of Loss

MMNikhil Kapoor7

There exist many approaches that aim at directly modifying the loss function
with an objective of improving either adversarial or corruption robustness
[Sea20, PLZ19, XCKW19, IS19, TP18, HK18, WSC+19]. One of the earliest
approaches for improving corruption robustness was introduced by Zheng et
al.[ZSLG16] called _stability training_ , where they introduce a
regularization term that penalizes the network prediction to a clean and an
augmented image. However, their approach does not scale to many augmentations
at the same time. Janocha et al.[JC17] then introduced a detailed analysis on
the influence of multiple loss functions to model performance as well as
robustness and suggested that expectation-based losses tend to work better
with noisy data and squared-hinge losses tend to work better for clean data.
Other well-known approaches are mainly based on variations of data
augmentation [CZM+19, CZSL19, ZCG+19, LYP+19], which can be computationally
quite expensive.

In contrast to corruption robustness, there exist many more approaches based
on adversarial examples. We highlight some of the most interesting and
relevant ones here. Mustafa et al.[Hao19] proposes to add a loss term that
maximally separates class-wise feature map representations, hence increasing
the distance from data points to the corresponding decision boundaries.
Similarly, Pang et al.[PXD+20] proposed the Max-Mahalanobis center (MMC) loss
to learn more structured representations and induce high-density regions in
the feature space. Chen et al.[CBLR18] proposed a variation of the well-known
cross entropy (CE) loss that not only maximizes the model probabilities of the
correct class, but in addition, also minimizes model probabilities of
incorrect classes. Cisse et al.[CBG+17] constraints the Lipschitz constant of
different layers to be less than one which restricts the error propagation
introduced by adversarial perturbations to a DNN. Dezfooli et al.[MDFUF19]
proposed to minimize the curvature of the loss surface locally around data
points. They emphasize that there exists a strong correlation between locally
small curvature and correspondingly high adversarial robustness.

All of these methods highlighted above are evaluated mostly for image
classification tasks on smaller datasets, namely CIFAR-10 [Ale09], CIFAR-100
[Ale09], SVHN [Yuv11], and only sometimes on ImageNet [KSH12]. Very few
approaches have been tested rigorously on complex safety-relevant tasks such
as _object detection_ and _semantic segmentation_ , etc. Moreover, methods
that improve adversarial robustness are only tested on a small subset of
attack types under differing attack specifications. This makes comparing
multiple methods difficult.

In addition, methods that improve corruption robustness are evaluated over a
standard data set of various corruption types which may or may not be relevant
to its application domain. In order to assess multiple methods for their
effect on safety-related aspects, a thorough robustness evaluation methodology
is needed, which is largely missing in the current literature. This evaluation
would need to take into account relevant disturbances/corruption types present
in the real world (application domain) and had to assess robustness towards
such changes in a rigorous manner. Without such an evaluation, we run the risk
of being overconfident in our network, thereby harming safety.

###  3.3 Domain Generalization

MMFiras Mualla13

Domain generalization (DG) can be seen as an extreme case of _domain
adaptation_ (DA). The latter is a type of transfer learning, where the source
and target tasks are the same (e.g., shared class labels) but the source and
target domains are different (e.g., another image acquisition protocol or a
different background) [Csu17a, WYKN20]. The DA can be either supervised (SDA),
where there is little available labeled data in the target domain, or
unsupervised (UDA), where data in the target domain is not labeled. The DG
goes one step further by assuming that the target domain is entirely unknown.
Thus, it seeks to solve the train-test domain shift in general. While DA is
already an established line of research in the machine learning community, DG
is relatively new [MBS13], though with an extensive list of papers in the last
few years.

Probably, the first intuitive solution that one may think of to implement DG
is neutralizing the domain-specific features. It was shown in [WHLX19] that
the gray-level co-occurrence matrices (GLCM) tend to perform poorly in
semantic classification (e.g., digit recognition) but yield good accuracy in
textural classification compared to other feature sets such as SURF and LBP.
DG was thus implemented by decorrelating the model’s decision from the GLCM
features of the input image even without the need of domain labels.

Besides the aforementioned intensity-based statistics of an input image, it is
known that characterizing image style can be done based on the correlations
between the filter responses of a DNN layer [GEB16] (neural style transfer).
In [SMK20], the training images are enriched with stylized versions, where a
style is defined either by an external style (e.g., cartoon or art) or by an
image from another domain. Here, DG is addressed as a _data augmentation_
problem.

Some approaches [LTG+18, MH20a] try to learn generalizable latent
representations by a kind of adversarial training. This is done by a generator
or an encoder, which is trained to generate a hidden feature space that
maximizes the error of a domain discriminator but at the same time minimizes
the classification error of the task of concern. Another flavor of adversarial
training can be seen in [LPWK18], where an adversarial autoencoder [MSJ+16] is
trained to generate features, which a discriminator cannot distinguish from
random samples drawn from a prior Laplace distribution. This regularization
prevents the hidden space from overfitting to the source domains, in a similar
spirit to how variational autoencoders do not leave gaps in the latent space.
In [MH20a], it is argued that the domain labels needed in such approaches are
not always well-defined or easily available. Therefore they assume unknown
latent domains which are learned by clustering in a space similar to the
style-transfer features mentioned above. The pseudo labels resulting from
clustering are then used in the adversarial training.

Autoencoders have been employed for DG not only in an adversarial setup, but
also in the sense of _multi-task learning_ nets [Car97], where the
classification task in such nets is replaced by a reconstruction one. In
[GKZB15], an autoencoder is trained to reconstruct not only the input image
but also the corresponding images in the other domains.

In the core of both DA and DG we are confronted with a distribution matching
problem. However, estimating the probability density in high-dimensional
spaces is intractable. The density-based metrics such as Kullback-Leibler
divergence are thus not directly applicable. In statistics, the so-called
_two-samples tests_ are usually employed to measure the distance between two
distributions in a point-wise manner, i.e., without density estimation. For
deep learning applications, these metrics need not only to be point-wise but
also differentiable. The two-samples tests were approached in the machine
learning literature using (differentiable) K-NNs [DK17], classifier two-
samples tests (C2ST) [LO17], or based on the theory of kernel methods
[SGSS07]. More specifically, the _maximum mean discrepancy_ (MMD) [GBR+06,
GBR+12], which belongs to the kernel methods, is widely used for DA [GKZ14,
LZWJ17, YDL+17, YLW+20] but also for DG [LPWK18]. Using the MMD, the distance
between two samples is estimated based on pairwise kernel evaluations, e.g.,
the radial basis function (RBF) kernel.

While the DG approaches generalize to domains from which zero shots are
available, the so-called _zero shot learning_ (ZSL) approaches generalize to
tasks (e.g., new classes in the same source domains) for which zero shots are
available. Typically, the input in ZSL is mapped to a semantic vector per
class instead of a simple class label. This can be, for instance, a vector of
visual attributes [LNH14] or a word embedding of the class name [KXG17]. A
task (with zero shots at training time) can be then described by a vector in
this space. In [MARC20], there is an attempt to combine ZSL and DG in the same
framework in order to generalize to new domains as well as new tasks, which is
also referred to as _heterogeneous domain generalization_.

Note that most discussed approaches for DG require non-standard handling,
i.e., modifications to models, data, and/or the optimization procedure. This
issue poses a serious challenge as it limits the practical applicability of
these approaches. There is a line of research which tries to address this
point by linking DG to other machine learning paradigms, especially the model-
agnostic meta-learning (MAML) [FAL17] algorithm, in an attempt to apply DG in
a model-agnostic way. Loosely speaking, a model can be exposed to simulated
train-test domain shift by training on a small _support set_ to minimize the
classification error on a small _validation set_. This can be seen as an
instance of a _few shot learning_ (FSL) problem [WYKN20]. Moreover, the
procedure can be repeated on other (but related) FSL tasks (e.g., different
classes) in what is known as _episodic training_. The model transfers its
knowledge from one task to another task and learns how to learn fast for new
tasks. This can be thus seen as a _meta-learning_ objective [HAMS20] (in a FSL
setup). Since the goal of DG is to adapt to new domains rather than new tasks,
several model-agnostic approaches [LYSH18, LZY+19, BSC18, DdCKG19] try to
recast this procedure in a DG setup.

##  4 Adversarial Attacks

MMAndreas Bär15

Over the last few years, deep neural networks (DNNs) consistently showed
state-of-the-art performance across several vision-related tasks. While their
superior performance on clean data is indisputable, they show a lack of
robustness to certain input patterns, denoted as _adversarial examples_
[SZS+14]. In general, an algorithm for creating adversarial examples is
referred to as an _adversarial attack_ and aims at fooling an underlying DNN,
such that the output changes in a desired and malicious way. This can be
carried out without any knowledge about the DNN to be attacked (black-box
attack) [MDFF16, PMG+17], or with full knowledge about the parameters,
architecture, or even training data of the respective DNN (white-box attack)
[GSS15, CW17a, MMS+18]. While initially being applied on simple classification
tasks, some approaches aim at finding more realistic attacks [TVRG19, JLS+19],
which particularly pose a threat to safety-critical applications, such as DNN-
based environment perception systems in autonomous vehicles. Altogether, this
motivated the research in finding ways of defending against such adversarial
attacks [GSS15, MDFUF19, XZZ+19, GRCvdM18]. In this section, we introduce the
current state of research regarding adversarial attacks in general, more
realistic adversarial attacks closely related to the task of environment
perception for autonomous driving, and strategies for detecting or defending
adversarial attacks. We conclude each section by clarifying current challenges
and research directions.

###  4.1 Adversarial Attacks and Defenses

MMAndreas Bär15, Seyed Eghbal Ghobadi8, Ahmed Hammam8

The term _adversarial example_ was first introduced by Szegedy et al.[SZS+14].
From there on, many researchers tried to find new ways of crafting adversarial
examples more effectively. Here, the fast gradient sign method (FGSM) [GSS15],
DeepFool [MDFF16], least-likely class method (LLCM) [KGB17a, KGB17b], C&W
[CW17b], momentum iterative fast gradient sign method (MI-FGSM) [DLP+18], and
projected gradient descent (PGD) [MMS+18] are a few of the most famous attacks
so far. In general, these attacks can be executed in an iterative fashion,
where the underlying adversarial perturbation is usually bounded by some norm
and is following additional optimization criteria, e.g., minimizing the number
of changed pixels.

The mentioned attacks can be further categorized as image-specific attacks,
where for each image a new perturbation needs to be computed. On the other
hand, image-agnostic attacks aim at finding a perturbation, which is able to
fool an underlying DNN on a set of images. Such a perturbation is also
referred to as a _universal adversarial perturbation_ (UAP). Here, the
respective algorithm UAP [MDFFF17], fast feature fool (FFF) [MGB17], and prior
driven uncertainty approximation (PD-UA) [LJL+19] are a few honorable
mentions. Although the creation process of a universal adversarial
perturbation typically relies on a white-box setting, they show a high
_transferability_ across models [HBMF20]. This allows black-box attacks, where
one model is used to create a universal adversarial perturbation, and another
model is being attacked with the beforehand-created perturbation. Another way
of designing black-box attacks is to create a surrogate DNN, which mimics the
respective DNN to be attacked and thus can be used in the process of
adversarial example creation [PMG+17]. On the contrary, some research has been
done to create completely incoherent images (based on evolutionary algorithms
or gradient ascent) to fool an underlying DNN [NYC15]. Different from that,
another line of work has been proposed to alter only some pixels in images to
attack a respective model. Here [NK16] and [NK17] have used optimization
approaches to perturb some pixels in images to produce targeted attacks,
aiming at a specific class output, or non-targeted attacks, aiming at
outputting a class different from the network output or the ground truth. This
can be extended up to finding one pixel in the image to be exclusively
perturbed to generate adversarial images [SVS19, NK16]. The authors of [BF17,
PKGB18, SBMC17] proposed to train generative models to generate adversarial
examples. Given an input image and the target label, a generative model is
trained to produce adversarial examples for DNNs. However, while the produced
adversarial examples look rather unrealistic to a human, they are able to
completely deceive a DNN.

The existence of adversarial examples not only motivated research in finding
new attacks, but also in finding _defense strategies_ to effectively defend
these attacks. Especially for safety-critical applications, such as DNN-based
environment perception for autonomous driving, the existence of adversarial
examples needs to be handled accordingly. Similar to adversarial attacks, one
can categorize defense strategies into two types: _model-specific_ defense
strategies and _model-agnostic_ defense strategies. The former refers to
defense strategies, where the model of interest is modified in certain ways.
The modification can be done on the architecture, training procedure, training
data, or model weights. On the other hand, model-agnostic defense strategies
consider the model to be a black box. Here, only the input or the output is
accessible. Some well-known model-specific defense strategies include
adversarial training [GSS15, MMS+18], the inclusion of robustness-oriented
loss functions during training [CLC+19, MDFUF19, KYL+20], removing adversarial
patterns in features by denoising layers [HRF19, MKH+19, XWvdM+19], and
redundant teacher-student frameworks [BHSFs19, BKV+20]. The majority of model-
agnostic defense strategies primarily focuses on various kinds of (gradient
masking) pre-processing strategies [BFW+19, GRCvdM18, GR19, JWCF19, LLL+19,
RSFM19, TCBZ19]. The idea is to remove the adversary from the respective
image, such that the image is transformed from the adversarial space back into
the clean space.

Nonetheless, Athalye et al.[ACW18] showed that gradient masking alone is not a
sufficient criterion for a reliable defense strategy. In addition, detection
and out-of-distribution techniques have also been proposed as model-agnostic
defense strategies against adversarial attacks. Here, the Mahalanobis distance
[LLLS18] or area under the receiver operating characteristic curve (AUROC) and
area under the precision-recall curve (AUPR) [HG17] are used to detect
adversarial examples. The authors of [HG17, LLLS17, HMCKBF17] on the other
hand proposed to train networks to detect, whether the input image is out-of-
distribution or not.

Moreover, Feinman et al.[FCSG17] proved that adversarial attacks usually
produce high uncertainty on the output of the DNN. As a consequence, they
proposed to use the dropout technique to estimate uncertainty on the output to
identify a possible adversarial attack.

Regarding adversarial attacks, the majority of the listed attacks are designed
for image classification. Only a few adversarial attacks consider tasks that
are closely related to autonomous driving, such as bounding box detection,
semantic segmentation, instance segmentation, or even panoptic segmentation.
Also, the majority of the adversarial attacks rely on a white-box setting,
which is usually not present for a potential attacker. Especially universal
adversarial perturbations have to be considered as a real threat due to their
high model transferability. Generally speaking, the existence of adversarial
examples has not been thoroughly studied yet. An analytical interpretation is
still missing, but could help in designing more mature defense strategies.

Regarding defense strategies, adversarial training is still considered as one
of the most effective ways of increasing the robustness of a DNN. Nonetheless,
while adversarial training is indeed effective, it is rather inefficient in
terms of training time. In addition, model-agnostic defenses should be favored
as once being designed, they can be easily transferred to different models.
Moreover, as most model-agnostic defense strategies rely on gradient-masking
and it has been shown that gradient-masking is not a sufficient property for a
defense strategy, new ways of designing model-agnostic defenses should be
taken into account. Furthermore, out-of-distribution and adversarial attacks
detection or even correction methods have been a new trend for identifying
attacks. However, as the environment perception system of an autonomous
driving vehicle could rely on various information sources, including LiDAR,
optical flow, or depth from a stereo camera, techniques of information fusion
should be further investigated to mitigate or even eliminate the effect of
adversarial examples.

###  4.2 More Realistic Attacks

MMSvetlana Pavlitskaya14

We consider the following two categories of realistic adversarial attacks: (1)
image-level attacks, which not only fool a neural network but also pose a
provable threat to autonomous vehicles, and (2) attacks which have been
applied in a real world or in a simulation environment, such as car learning
to act (CARLA) [DRC+17].

Some notable examples in the first category of attacks include attacks on
semantic segmentation [HMCKBF17] or person detection [TVRG19].

In the second group of approaches, the attacks are specifically designed to
survive real world distortions, including different distances, weather and
lighting conditions, as well as camera angles. For this, adversarial
perturbations are usually concentrated in a specific image area, called
_adversarial patch_. Crafting an adversarial patch involves specifying a patch
region in each training image, applying transformations to the patch, and
iteratively changing the pixel values within this region to maximize the
network prediction error. The latter step typically relies on an algorithm,
proposed for standard adversarial attacks, which aim at crafting invisible
perturbations while misleading neural networks, e.g., C&W [CW17b], Jacobian-
based saliency map attack (JSMA) [PMG+17], and PGD [MMS+18].

The first printable adversarial patch for image classification was described
by Brown et al.[BMR+17]. Expectation over transformations (EOT) [AEIK17] is
one of the influential updates to the original algorithm—it permits to
robustify patch-based attacks to distortions and affine transformations.
Localized and visible adversarial noise (LaVAN) [KZG18] is a further method to
generate much smaller patches (up to 2% of the pixels in the image). In
general, fooling image classification with a patch is a comparatively simple
task, because adversarial noise can mimic an instance of another class and
thus lower the prediction probability for a true class.

Recently, patch-based attacks for a more tricky task of object detection have
been described [LYL+18, TVRG19]. Also, Lee and Kolter [LK19] generate a patch
using PGD [MMS+18], followed by EOT applied to the patch. With this approach,
all detections in an image can be successfully suppressed, even without any
overlap of a patch with bounding boxes. Furthermore, several approaches for
generating an adversarial T-shirt have been proposed, including [XZL+19,
WLDG19].

DeepBillboard [ZLZ+18] is the first attempt to attack end-to-end driving
models with adversarial patches. The authors propose to generate a single
patch for a sequence of input images to mislead four steering models,
including DAVE-2 in a drive-by scenario.

Apart from physical feasibility, inconspicuousness is crucial for a realistic
attack. Whereas adversarial patches usually look like regions of noise,
several works have explored attacks with an inconspicuous patch. In
particular, Eykholt et al.[EEF+18] demonstrate the vulnerability of road sign
classification to the adversarial perturbations in the form of only black and
white stickers. In [BHG+19], an end-to-end driving model is attacked in CARLA
by painting of black lines on the road. Also, Kong and Liu [KL19] use a
generative adversarial network to get a realistic billboard to attack an end-
to-end driving model in a drive-by scenario. In [DMW+20], a method to hide
visible adversarial perturbations with customized styles is proposed, which
leads to adversarial traffic signs that look unsuspicious to a human.

Current research mostly focuses on attacking image-based perception of an
autonomous vehicle. Adversarial vulnerability of further components of an
autonomous vehicle, e.g., LiDAR-based perception, optical flow and depth
estimation, has only recently gained attention. Furthermore, most attacks
consider only a single component of an autonomous driving pipeline, the
question whether the existing attacks are able to propagate to further
pipeline stages has not been studied yet. The first work in this direction
[JLS+19] describes an attack on object detection and tracking. The evaluation
is, however, limited to a few clips, where no experiments in the real world
have been performed. Overall, the research on realistic adversarial attacks,
especially combined with physical tests, is currently in the starting phase.

##  5 Interpretability

MMFelix Brockherde10

Neural networks are, by their nature, black boxes and therefore intrinsically
hard to interpret [Tay06]. Due to their unrivaled performance, they still
remain first choice for advanced systems even in many safety-critical areas,
such as level 4 automated driving. This is why the research community has
invested considerable effort to unhinge the black-box character and make deep
neural networks more transparent.

We can observe three strategies that provide different view points towards
this goal in the state of the art. First is the most direct approach of
opening up the black box and looking at intermediate representations (Sec.
5.2). Being able to interpret individual layers of the system facilitates
interpretation of the whole. The second approach tries to provide
interpretability by explaining the network’s decisions with pixel attributions
(Sec. 5.3). Aggregated explanations of decision can then lead to
interpretability of the system itself. Third is the idea of approximating the
network with interpretable proxies to benefit from the deep neural networks
performance while allowing interpretation via surrogate models (Sec. 5.4).
Underlying all aspects here is the area of visual analytics (Sec. 5.1).

There exists earlier research in the medical domain to help human experts
understand and convince them of machine learning decisions [CLG+15]. Legal
requirements in the finance industry gave rise to interpretable systems that
can justify their decisions. An additional driver for interpretability
research was the concern for Clever Hans predictors [LWB+19].

###  5.1 Visual Analytics

MMElena Schulz1

Traditional data science has developed a huge tool set of automated analysis
processes conducted by computers, which are applied to problems that are well-
defined in the sense that the dimensionality of input and output as well as
the size of the data set they rely on is manageable. For those problems that
in comparison are more complex, the automation of the analysis process is
limited and/or might not lead to the desired outcome. This is especially the
case with unstructured data like image or video data in which the underlying
information cannot directly be expressed by numbers. Rather, it needs to be
transformed to some structured form to enable computers to perform some task
of analysis. Additionally, with an ever increasing amount of various types of
data being collected, this “information overload” cannot solely be analyzed by
automatic methods [KAF+08, KMT10].

_Visual analytics_ addresses this challenge as “the science of analytical
reasoning facilitated by interactive visual interfaces” [CT05]. Visual
analytics therefore does not only focus on either computationally processing
data or visualizing results but coupling both tightly with interactive
techniques. Thus, it enables an integration of the human expert into the
iterative visual analytics process: Through visual understanding and human
reasoning, the knowledge of the human expert can be incorporated to
effectively refine the analysis. This is of particular importance, where a
stringent safety argumentation for complex models is required. With the help
of visual analytics, the line of argumentation can be built upon arguments
that are understandable for humans. To include the human analyst efficiently
into this process, a possible guideline is the _visual analytics mantra_ by
Keim: “Analyze first, show the important, zoom, filter and analyze further,
details on demand” [KAF+08]111Extending the original _visualization mantra_ by
Shneiderman “Overview first, filter and zoom, details on demand”[Shn96]..

The core concepts of visual analytics therefore rely on well-designed
interactive visualizations, which support the analyst in the tasks of, e.g.,
reviewing, understanding, comparing and inferring not only the initial
phenomenon or data but also the computational model and its results itself
with the goal of enhancing the analytical process.

Driven by various fields of application, visual analytics is a
multidisciplinary field with a wide variety of task-oriented development and
research. As follows, recent work has been done in several areas: depending on
the task, there exist different pipeline approaches to create whole _visual
analytics systems_ [WZM+16]; the injection of human expert knowledge into the
process of determining trends and patterns from data is the focus of
_predictive visual analytics_ [LCM+17, LGH+17]; enabling the human to explore
_high-dimensional data_ [LMW+17] interactively and visually (e.g., via
dimensionality reduction [SZS+17]) is a major technique to enhance the
understandability of complex models (e.g., neural networks); the iterative
improvement and the understanding of machine learning models is addressed by
using interactive visualizations in the field of _general machine learning_
[LWLZ17] or the other way round: using machine learning to improve
visualizations and guidance based on user interactions [ERT+17]. Even more
focused on the loop of simultaneously developing and refining machine learning
models is the area of _interactive machine learning_ , where the topics of
_interface design_ [DK18] and the _importance of users_ [ACKK14, SSZ+17] are
discussed. One of the current research directions is using visual analytics in
the area of _deep learning_ [GTC+18, HKPC18, CL18]. However, due to the
interdisciplinarity of visual analytics, there are still open directions and
ongoing research opportunities.

Especially in the domain of neural networks and deep learning, visual
analytics is a relatively new approach in tackling the challenge of
_explainability_ and _interpretability_ of those often called _black boxes_.
To enable the human to better interact with the models, research is done in
enhancing the _understandability_ of complex deep learning models and their
outputs with the use of proper visualizations. Other research directions
attempt to achieve improving the _trustability_ of the models, giving the
opportunity to inspect, diagnose and refine the model. Further, possible areas
for research are _online training processes_ and the development of
_interactive systems_ covering the whole process of training, enhancing and
monitoring machine learning models. Here, the approach of _mixed guidance_ ,
where system-initiated guidance is combined with user-initiated guidance, is
discussed among the visual analytics community as well. Another challenge and
open question is creating ways of _comparing models_ to examine which model
yields a better performance, given specific situations and selecting or
combining the best models with the goal of increasing performance and overall
safety.

###  5.2 Intermediate Representations

MMFelix Hauser11, Jan Kronenberger9, Seyed Eghbal Ghobadi8

In general, representation learning [BCV13] aims to extract lower dimensional
features in latent space from higher dimensional inputs. These features are
then used as an effective representation for regression, classification,
object detection and other machine learning tasks. Preferably, latent features
should be disentangled, meaning that they represent separate factors found in
the data that are statistically independent. Due to their importance in
machine learning, finding meaningful intermediate representations has long
been a primary research goal. Disentangled representations can be interpreted
more easily by humans and can for example be used to explain the reasoning of
neural networks [HDR18].

Among the longer known methods for extracting disentangled representations are
principal component analysis (PCA) [FP78, JC16], independent component
analysis [HO00], and nonnegative matrix factorization [BBL+07]. PCA is highly
sensitive to outliers and noise in the data. Therefore, more robust algorithms
were proposed. In [SBS12] already a small neural network was used as an
encoder and the algorithm proposed in [FXY12] can deal with high-dimensional
data. Some robust PCA algorithms are provided with analytical performance
guarantees [XCS10, RA17, RL19].

A popular method for representation learning with deep networks is the
variational autoencoder (VAE) [KW14]. An important generalization of the
method is the $\beta$-VAE variant [HMP+17], which improved the disentanglement
capability [FAA18]. Later analysis added to the theoretical understanding of
$\beta$-VAE [BHP+18, SZYP19, KP20]. Compared to standard autoencoders, VAEs
map inputs to a distribution, instead of mapping them to a fixed vector. This
allows for additional regularization of the training to avoid overfitting and
ensure good representations. In $\beta$-VAEs the trade-off between
reconstruction quality and disentanglement can be fine-tuned by the
hyperparameter $\beta$.

Different regularization schemes have been suggested to improve the VAE
method. Among them are Wasserstein autoencoders [TBGS19, XW19], attribute
regularization [PL20] and relational regularization [XLH+20]. Recently, a
connection between VAEs and nonlinear independent component analysis was
established [KKMH20] and then expanded [SRK20].

Besides VAEs, deep generative adversarial networks can be used to construct
latent features [SLY15, CDH+16, MSJ+16]. Other works suggest centroid encoders
[GK20] or conditional learning of Gaussian distributions [SYZ+20] as
alternatives to VAEs. In [KWG+18] concept activation vectors are defined as
being orthogonal to the decision boundary of a classifier. Apart from deep
learning, entirely new architectures, such as capsule networks [SFH17], might
be used to disassemble inputs.

While many different approaches for disentangling exist, the feasibility of
the task is not clear yet and a better theoretical understanding is needed.
The disentangling performance is hard to quantify, which is only feasible with
information about the latent ground truth [EW18]. Models that overly rely on
single directions, single neurons in fully connected networks or single
feature maps in CNNs, have the tendency to overfit [MBRB18]. According to
[LBL+19], unsupervised learning does not produce good disentangling and even
small latent spaces do not reduce the sample complexity for simple tasks. This
is in direct contrast to newer findings that show a decreased sample
complexity for more complex visual downstream tasks [vLSB20]. So far, it is
unclear if disentangling improves the performance of machine learning tasks.

In order to be interpretable, latent disentangled representations need to be
aligned with human understandable concepts. In [EIS+19] training with
adversarial examples was used and the learned representations were shown to be
more aligned with human perception. For explainable AI, disentangling alone
might not be enough to generate interpretable output and additional
regularization could be needed.

###  5.3 Pixel Attribution

MMStephanie Abrecht2, Felix Brockherde10, Toshika Srivastava12

The non-linearity and complexity of DNNs allow them to solve perception
problems, like detecting a pedestrian, that cannot be specified in detail. At
the same time, the automatic extraction of features given in an input image
and the mapping to the respective prediction is counterintuitive and
incomprehensible for humans, which makes it hard to argue safety for a neural
network-based perception task. Feature importance techniques are currently
predominantly used to diagnose the causes of incorrect model behaviors
[BXS+20]. So-called _attribution maps_ are a visual technique to express the
relationship between relevant pixels in the input image and the network’s
prediction. Regions in an image that contain relevant features are highlighted
accordingly. Attribution approaches mostly map to one of three categories.

Gradient-based and activation-based approaches (such as [SDV+16, SVZ14,
SGSK17, AS17, STK+17, BBM+15, SDBR14, MBB+15] amongst others) rely on the
gradient of the prediction with respect to the input. Regions that were most
relevant for the prediction are highlighted. Activation-based approaches
relate the feature maps of the last convolutional layer to output classes.

Perturbation-based approaches [FV17, ZF13, ZCAW17, HMK+19] suggest
manipulating the input. If the prediction changes significantly, the input may
hold a possible explanation at least.

While gradient-based approaches are oftentimes faster in computation,
perturbation-based approaches are much easier to interpret.

As many studies have shown [MS17, AGM+18], there is still a lot of research to
be done before attribution methods are able to robustly provide explanations
for model predictions, in particular erroneous behavior. One key difficulty is
the lack of an agreed-upon definition of a good attribution map including
important properties. Even between humans, it is hard to agree on what a good
explanation is due to its subjective nature. This lack of ground truth makes
it hard or even impossible to quantitatively evaluate an explanation method.
Instead, this evaluation is done only implicitly. One typical way to do this
is the axiomatic approach. Here a set of desiderata of an attribution method
are defined, on which different attribution methods are then evaluated.
Alternatively, different attribution methods may be compared by perturbing the
input features starting with the ones deemed most important and measuring the
drop in accuracy of the perturbed models. The best method will result into the
greatest overall loss in accuracy as the number of inputs are omitted
[ACÖG17]. Moreover, for gradient-based methods it is hard to assess if an
unexpected attribution is caused by a poorly performing network or a poorly
performing attribution method [FV17]. How to cope with negative evidence,
i.e., the object was predicted because a contrary clue in the input image was
missing, is an open research question. Additionally, most methods were shown
on classification tasks until now. It remains to be seen how they can be
transferred to object detection and semantic segmentation tasks. In the case
of perturbation-based methods, the high computation time and single-image
analysis inhibit wide-spread application.

###  5.4 Interpretable Proxies

MMGesina Schwalbe3

Neural networks are capable of capturing complicated logical (cor)relations.
However, this knowledge is encoded on a _sub-symbolic_ level in the form of
learned weights and biases, meaning that the reasoning behind the processing
chain cannot be directly read out or interpreted by humans [CPT01]. To explain
the sub-symbolic processing, one can either use attribution methods (cf. Sec.
5.3), or lift this sub-symbolic representation to a _symbolic_ one [GBY+18],
meaning a more interpretable one. Interpretable proxies or surrogate models
try to achieve the latter: The DNN behavior is approximated by a model that
uses symbolic knowledge representations. Symbolic representations can be
linear models like LIME [RSG16] (proportionality), decision trees (if-then-
chains) [GBY+18], or loose sets of logical rules. Logical connectors can
simply be AND and OR but also more general ones like at-least-M-of-N [CPT01].
The _expressiveness_ of an approach refers to the logic that is used: Boolean-
only versus first-order logic, and binary versus fuzzy logic truth values
[TAGD98]. Other than attribution methods (cf. Sec. 5.3), these representations
can capture combinations of features and (spatial) relations of objects and
attributes. As an example consider “eyes are closed” as explanation for
“person asleep”: Attribution methods only could mark the location of the eyes,
dismissing the relations of the attributes [RSS18]. All mentioned surrogate
model types (linear, set of rules) require interpretable input features in
order to be interpretable themselves. These features must either be directly
obtained from the DNN input or (intermediate) output, or automatically be
extracted from the DNN representation. Examples for extraction are the super-
pixeling used in LIME for input feature detection, or concept activation
vectors [KWG+18] for DNN representation decoding.

Quality criteria and goals for interpretable proxies are [TAGD98]: _accuracy_
of the standalone surrogate model on unseen examples, _fidelity_ of the
approximation by the proxy, _consistency_ with respect to different training
sessions, and _comprehensibility_ measured by the complexity of the rule set
(number of rules, number of hierarchical dependencies). The criteria are
usually in conflict and need to be balanced: Better accuracy may require a
more complex, thus less expressive sets of rules.

Approaches for interpretable proxies differ in the validity range of the
representations: Some aim for surrogates that are only valid _locally_ around
specific samples, like in LIME [RSG16] or in [RSS18] via inductive logic
programming. Other approaches try to more _globally_ approximate aspects of
the model behavior. Another categorization is defined by whether full access
(_white-box_), some access (_gray-box_), or no access (_black-box_) to the DNN
internals is needed. One can further differentiate between _post-hoc_
approaches that are applied to a trained model, and approaches that try to
integrate or _enforce symbolic representations_ during training. Post-hoc
methods cover the wide field of rule extraction techniques for DNNs. The
reader may refer to [Hai16, AK12]. Most white- and gray-box methods try to
turn the DNN connections into if-then rules that are then simplified, like
done in DeepRED [ZLMJ16]. A black-box example is validity interval analysis
[Thr95], which refines or generalizes rules on input intervals, either
starting from one sample or a general set of rules. Enforcement of symbolic
representations can be achieved by enforcing an output structure that provides
insights to the decision logic, such as textual explanations, or a rich output
structure allowing investigation of correlations [XLZ+18]. An older discipline
for enforcing symbolic representations is the field of neural-symbolic
learning [SSZ19]. The idea is based on a hybrid learning cycle in which a
symbolic learner and a DNN iteratively update each other via rule insertion
and extraction.

The comprehensibility of global surrogate models suffers from the complexity
and size of concurrent DNNs. Thus, stronger rule simplification methods are
required [Hai16]. The alternative direction of local approximations mostly
concentrates on linear models instead of more expressive rules [Thr95, RSS18].
Furthermore, balancing of the quality objectives is hard since available
indicators for interpretability may not be ideal. And lastly, applicability is
heavily infringed by the requirement of interpretable input features. These
are usually not readily available from input (often pixel-level) or DNN
output. Supervised extraction approaches vary in their fidelity, and
unsupervised ones do not guarantee to yield meaningful or interpretable
results, respectively, such as the super-pixel clusters of LIME.

##  6 Uncertainty

MMMichael Mock1

Uncertainty refers to the view that a neural network is not conceived as a
deterministic function but as a probabilistic function or estimator,
delivering a random distribution for each input point. Ideally, the mean value
of the distribution should be as close as possible to the ground truth value
of the function being approximated by the neural network and the uncertainty
of the neural network refers to its variance when being considered as a random
variable, thus allowing to derive a confidence with respect to the mean value.
Regarding safety, the variance may lead to estimations about the confidence
associated with a specific network output and opens the option for discarding
network outputs with insufficient confidence.

There are roughly two broad approaches for training neural networks as
probabilistic functions: Parametric approaches [KG17] and Bayesian neural
networks on the one hand, such as [BCKW15], where the transitions along the
network edges are modeled as probability distributions, and ensemble-based
approaches on the other hand [LPB17, SOF+19], where multiple networks are
trained and considered as samples of a common output distribution. Apart from
training as probabilistic function, uncertainty measures have been derived
from single, standard neural networks by post-processing on the trained
network logits, leading for example to calibration measures (cf. e.g.,
[SOF+19]).

###  6.1 Generative Models

MMSebastian Wirkert6, Tim Wirtz1

_Generative models_ belong to the class of unsupervised machine learning
models. From a theoretical perspective, these are particularly interesting,
because they offer a way to analyze and model the density of data. Given a
finite data set $\mathcal{D}$ independently distributed according to some
distribution $p(x)$, generative models aim to estimate or enable sampling from
the underlying density $p(x)$ in a model $q_{\theta}(x)$. The resulting model
can be used for data indexing [Wes04], data retrieval [ML11], for visual
recognition [KSH12], speech recognition and generation [HDY+12], language
processing [KM03, CE17] and robotics [T+02]. Following [OE18], we can group
generative models into two main classes:

  * •

Cost function-based models such as autoencoder [KW14, Doe16], deep belief
networks [Hin09] and generative adversarial networks [Goo16, RMC15, GPAM+14].

  * •

Energy-based models [LCH+06, SH09], where the joint probability density is
modeled by an energy function.

Beside these _deep learning_ approaches, generative models have been studied
in machine learning in general for quite some time (cf. [Wer78, JMS96, Fry77,
Sil86, Gra18, Sco15, She04]). A very prominent example of generative networks
are Gaussian processes [Ras03, Mac98, WR96, WB98] and their deep learning
extensions [DL13, BHLHL+16] as generative models.

An example of a generative model being employed for image segmentation
uncertainty estimation is the probabilistic U-Net [KRPM+18]. Here a
variational autoencoder (VAE) conditioned on the image is trained to model
uncertainties. Samples from the VAE are fed into a segmentation U-Net which
can thus give different results for the same image. This was tested in context
of medical images, where inter-rater disagreements lead to uncertain
segmentation results and Cityscapes segmentation. For the Cityscapes
segmentation the investigated use case was label ambiguity (e.g., is a BMW X7
a car or a van) using artificially created, controlled ambiguities. Results
showed that the probabilistic U-Net could reproduce the segmentation ambiguity
modes more reliably than competing methods such as a dropout U-Net which is
based on techniques elaborated in the next section.

###  6.2 Monte-Carlo Dropout

MMJoachim Sicking1

A widely used technique to estimate model uncertainty is Monte-Carlo (MC)
dropout [GG16], that offers a Bayesian motivation, conceptual simplicity and
scalability to application-size networks. This combination distinguishes MC
dropout from competing Bayesian neural network (BNN) approximations (like
[BCKW15],[RBB18], see Sec. 6.3). However, these approaches and MC dropout
share the same goal: to equip neural networks with a _self-assessment_
mechanism that detects unknown input concepts and thus potential model
insufficiencies.

On a technical level, MC dropout assumes prior distributions on network
activations, usually independent and identically distributed (i.i.d.)
Bernoulli distributions. Model training with iteratively drawn Bernoulli
samples, the so-called _dropout masks_ , then yields a data-conditioned
posterior distribution within the chosen parametric family. It is interesting
to note that this training scheme was used earlier—independent from an
uncertainty context—for better model generalization [SHK+14]. At inference,
sampling provides estimates of the input-dependent output distributions. The
spread of these distributions is then interpreted as the prediction
uncertainty that originates from limited knowledge of model parameters.
Borrowing ‘frequentist’ terms, MC dropout can be considered as an implicit
network ensemble, i.e., as a set of networks that share (most of) their
parameters.

In practice, MC dropout requires only a minor modification of the optimization
objective during training and multiple, trivially parallelizable forward
passes during inference. The loss modification is largely agnostic to network
architecture and does not cause substantial overhead. This is in contrast to
the sampling-based inference that increases the computational effort
massively—by estimated factors of 20-100 compared to networks without MC
dropout. A common practice is therefore the use of _last-layer dropout_
[SOF+19] that reduces computational overhead to estimated factors of 2-10.
Alternatively, analytical moment propagation allows sampling-free MC-dropout
inference at the price of additional approximations (e.g., [PFC+19]). Further
extensions of MC dropout target the integration of data-inherent (aleatoric)
uncertainty [KG17] and tuned performance by learning layer-specific dropout
rate using concrete relaxations [GHK17].

The quality of MC-dropout uncertainties is typically evaluated using negative
log-likelihood (NLL), expected calibration error (ECE) and its variants (cf.
Sec. 6.6) and by considering correlations between uncertainty estimates and
model errors (e.g., AUSE [ICG+18]). Moreover, it is common to study how useful
uncertainty estimates are for solving auxiliary tasks like out-of-distribution
classification [LPB17] or robustness w.r.t. adversarial attacks.

MC dropout is a working horse of safe ML, being used with various networks and
for a multitude of applications (e.g., [BFS18]). However, several authors
pointed out shortcomings and limitations of the method: MC dropout bears the
risk of over-confident false predictions ([Osb16]), offers less diverse
uncertainty estimates compared to (the equally simple and scalable) deep
ensembles ([LPB17], see Sec. 7.1) and provides only rudimentary approximations
of true posteriors.

Relaxing these modelling assumptions and strengthening the Bayesian motivation
of MC dropout is therefore an important research avenue. Further directions
for future work are the development of _semantic uncertainty mechanisms_
(e.g., [KRPM+18]), improved local uncertainty calibrations and a better
understanding of the outlined sampling-free schemes to uncertainty estimation.

###  6.3 Bayesian Neural Networks

MMMaram Akila1

As the name suggests, Bayesian neural networks (BNNs) are inspired by a
Bayesian interpretation of probability (for an introduction cf. [Mac03]). In
essence, it rests on Bayes’ theorem,

$p(x|y)p(y)=p(x,y)=p(y|x)p(x)\quad\Rightarrow\quad
p(x|y)=\frac{p(y|x)p(x)}{p(y)}\,,$ (2)

stating that the conditional probability density function (PDF) $p(x|y)$ for
$x$ given $y$ may be expressed in terms of the inverted conditional PDF
$p(y|x)$. For machine learning, where one intends to make predictions $y$ for
unknown $x$ given some training data $\mathcal{D}$, this can be reformulated
into

$y=\operatorname{NN}(x|W)\quad\text{with}\quad
p(W|\mathcal{D})=\frac{p(\mathcal{D}|W)p(W)}{p(\mathcal{D})}\,.$ (3)

Therein NN denotes a conventional (deep) neural network (DNN) with model
parameters $W$, e.g., the set of weights and biases. In contrast to a regular
DNN, the weights are given in terms of a probability distribution
$p(W|\mathcal{D})$ turning also the output $y$ of a BNN into a distribution.
This allows to study the mean $\mu=\langle y^{1}\rangle$ of the DNN for a
given $x$ as well as higher moments of the distribution, typically the
resulting variance $\sigma^{2}=\langle(y-\mu)^{2}\rangle$ is of interest,
where

$\left\langle
y^{k}\right\rangle=\int\\!\operatorname{NN}(x|W)^{k}p(W|\mathcal{D})\,\mathrm{d}W\,.$
(4)

While $\mu$ yields the output of the network, the variance $\sigma^{2}$ is a
measure for the uncertainty of the model for the prediction at the given
point. Central to this approach is the probability of the data given the
model, here denoted by $p(\mathcal{D}|W)$, as it is the key component
connecting model and training data. Typically, the prior distribution
$p(\mathcal{D})$ is “ignored” as it only appears as a normalization constant
within the averages, see (4). In the cases where the data $\mathcal{D}$ is
itself a distribution due to inherent uncertainty, i.e., presence of aleatoric
risk [KG17], such a concept seems natural. However, Bayesian approaches are
also applicable for all other cases. In those, loosely speaking, the
likelihood of $W$ is determined via the chosen loss function (for the
connection between the two concepts cf. [Bis06]).

On this general level, Bayesian approaches are broadly accepted and also find
use for many other model classes besides neural networks. However, the loss
surfaces of DNNs are known for their high dimensionality and strong non-
convexity. Typically, there are abundant parameter combinations $W$ that lead
to (almost) equally good approximations to the training data $\mathcal{D}$
with respect to a chosen loss. This makes an evaluation of $p(W|\mathcal{D})$
for DNNs close to impossible in full generality. At least no (exact) solutions
for this case exist at the moment.

Finding suitable approximations to the posterior distribution
$p(W|\mathcal{D})$ is an ongoing challenge for the construction of BNNs. At
this point we only summarize two major research directions in the field. One
approach is to assume that the distribution factorizes. While the full
solution would be a joined distribution implying correlations between
different weights (etc.), possibly even across layers, this approximation
takes each element $w_{i}$ of $W$ to be independent form the others. Although
this is a strong assumption, it is often made, in this case parameters for the
respective distributions of each element can be learned via training (cf.
[BCKW15]). The second class of approaches focuses on the region of the loss
surface around the minimum chosen for the DNN. As discussed, the loss relates
to the likelihood and quantities such as the curvature at the minimum,
therefore directly connected to the distribution of $W$. Unfortunately,
already using this type of quantities requires further approximations [RBB18].
Alternatively, the convergence of the training process may be altered to
sample networks close to the minimum [WT11]. While this approach contains
information about correlations among the $w_{i}$, it is usually restricted to
a specific minimum. For a non-Bayesian ansatz taking into account several
minima, see deep ensembles in Sec. 7.1. BNNs also touch other concepts such as
MC dropout (cf. Sec. 6.2 or [GG16]), or prior networks, which are based on a
Bayesian interpretation but use conventional DNNs with an additional (learned)
$\sigma$ output [MG18].

###  6.4 Uncertainty Metrics for DNNs in Frequentist Inference

MMMatthias Rottmann5

Classical uncertainty quantification methods in frequentist inference are
mostly based on the outputs of statistical models. Their uncertainty is
quantified and assessed for instance via dispersion measures in classification
(such as entropy, probability margin or variation ratio), or confidence
intervals in regression. However, the nature of DNN architectures [RDGF15,
CPK+17] and the cutting edge applications tackled by those (e.g., semantic
segmentation, cf. [COR+16]) open the way towards more elaborate uncertainty
quantification methods. Besides the mentioned classical approaches,
intermediate feature representations within a DNN (cf. [OSM20, CZYS19]) or
gradients according to self-affirmation that represent re-learning stress (see
[ORG18]) reveal additional information. In addition, in case of semantic
segmentation, the geometry of a prediction may give access to further
information, cf. [RCH+18, RS19, MRG19]. By the computation of statistics of
those quantities as well as low-dimensional representations thereof, we obtain
more elaborate uncertainty quantification methods specifically designed for
DNNs that can help us to detect misclassifications and out-of-distribution
objects (cf. [HG17]).

Features gripped during a fordward pass of a data point $x$ through a DNN $f$
can be considered layer-wise, i.e., $f^{(\ell)}(x)$ after the $\ell$-th layer.
These can be translated into a handful of quantities per layer [OSM20] or
further processed by another DNN that aims that detecting errors [CZYS19].
While in particular [OSM20] presents a proof of concept on small scale
classification problems, their applicability to large scale datasets and
problems such as semantic segmentation and object detection remain open.

The development for gradient-based uncertainty quantification methods [ORG18]
is guided by one central question: If the present prediction was true, how
much re-learning would this require. The corresponding hypothesis is that
wrong predictions would be more in conflict with the knowledge encoded in the
deep neural network than correct ones, therefore causing increased re-learning
stress. Given a predicted class

$\hat{y}=\mathop{\mathrm{arg\,max}}_{y}f_{y}(x)$ (5)

we compute the gradient of layer $\ell$ corresponding to the predicted label.
That is, given a loss function $\mathcal{L}$, we compute

$\nabla_{w_{\ell}}\mathcal{L}(\hat{y},x,w)$ (6)

via backpropagation. The obtained quantities can be treated similarly to the
case of forward pass features. While this concept seems to be prohibitively
expensive for semantic segmentation (at least when calculating gradients for
each pixel of $\hat{y}$), its applicability to object detection might be
feasible, in particular with respect to offline applications. Gradients are
also of special interest in active learning with query by _expected model
change_ (cf. [Set10]).

In the context of semantic segmentation, geometrical information on segments
shapes as well as neighborhood relations of predicted segments can be taken
into account along side with dispersion measures. It has been demonstrated
[RCH+18, RS19, MRG19] that the detection of errors in an in-distribution
setting strongly benefits from geometrical information. Recently, this has
also been considered in scenarios under moderate domain shift [ORF20].
However, its applicability to out-of-distribution examples and to other
sensors than the camera remains subject to further research.

###  6.5 Markov Random Fields

MMSeyed Eghbal Ghobadi8, Ahmed Hammam8

Although deep neural networks are currently the state of the art for almost
all computer vision tasks, Markov random fields (MRF) remain one of the
fundamental techniques used for many computer vision tasks, specifically image
segmentation [KK11],[LWZ09]. MRFs hold its power in the essence of being able
to model dependencies between pixels in an image. With the use of energy
functions, MRFs integrate pixels into models relating between unary and
pairwise pixels together [WKP13]. Given the model, MRFs are used to infer the
optimal configuration yielding the lowest energy using mainly maximum a
posteriori (MAP) techniques. Several MAP inference approaches are used to
yield the optimal configuration such as graph cuts [KRBT08] and belief
propagation algorithms [FZ10]. However, as with neural networks, MAP inference
techniques result in deterministic point estimates of the optimal
configuration without any sense of uncertainty in the output. To obtain
uncertainties on results from MRFs, most of the work is directed towards
modelling MRFs with Gaussian distributions. Getting uncertainties from MRFs
with Gaussian distributions is possible by two typical methods: Either
approximate models are inferred to the posterior, from which sampling is easy
or the variances can be estimated analytically, or approximate sampling from
the posterior is used. Approximate models include those inferred using
variational Bayesian (VB) methods, like mean-field approximations, and using
Gaussian process (GP) models enforcing a simplified prior model [LUAD16],
[Bis06]. Examples of approximate sampling methods include traditional Markov
chain Monte Carlo (MCMC) methods like Gibbs sampling [GG84]. Some recent
theoretical advances propose the perturb-and-MAP framework and a Gumbel
perturbation model (GPM) [PY11],[HMJ13] to exactly sample from MRF
distributions. Another line of work has also been proposed, where MAP
inference techniques are used to estimate the probability of the network
output. With the use of graph cuts, [KT08] try to estimate uncertainty using
the min-marginals associated with the label assignments of a random field.
Here, the work by Kohli and Torr [KT08] was extended to show how this approach
can be extended to techniques other than graph cuts [TA12] or compute
uncertainties on multi-label marginal distributions [STP17].

A current research direction is the incorporation of MRFs with deep neural
networks, along with providing uncertainties on the output [SU15, CPK+17].
This can also be extended to other forms of neural networks such as recurrent
neural networks to provide uncertainties on segmentation of streams of videos
with extending dependencies of pixels to previous frames [LLL+17], [ZJRP+15].

###  6.6 Confidence Calibration

MMFabian Küppers9, Anselm Haselhoff9

Neural network classifiers output a label $\hat{Y}\in\mathcal{Y}$ on a given
input $X\in\mathcal{X}$ with an associated confidence $\hat{P}$. This
confidence can be interpreted as a probability of correctness that the
predicted label matches the ground truth label $Y\in\mathcal{Y}$. Therefore,
these probabilities should reflect the ”self-confidence” of the system. If the
empirical accuracy for any confidence level matches the predicted confidence,
a model is called _well calibrated_. Therefore, a _classification_ model is
perfectly calibrated if

$\displaystyle\underbrace{\mathbb{P}(\hat{Y}=Y|\hat{P}=p)}_{\text{accuracy
given }p}=\underbrace{p}_{\text{confidence}}\quad\forall p\in[0,1]$ (7)

is fullfilled [GPSW17]. For example, assume 100 predictions with confidence
values of 0.9. We call the model well calibrated if 90 out of these 100
predictions are actually correct. However, recent work has shown that modern
neural networks tend to be too overconfident in their predictions [GPSW17].
The deviation of a model to the perfect calibration can be measured by the
expected calibration error (ECE) [NCH15]. It is possible to recalibrate models
as a post-processing step after classification. One way to get a calibration
mapping is to group all predictions into several bins by their confidence.
Using such a binning scheme, it is possible to compute the empirical accuracy
for certain confidence levels, as it is known for a long time already in
reconstructing confidence outputs for Viterbi decoding [HR90]. Common methods
are histogram binning [ZE01], isotonic regression [ZE02] or more advanced
methods like Bayesian binning into quantiles (BBQ) [NCH15] and ensembles of
near-isotonic regression (ENIR) [NC16]. Another way to get a calibration
mapping is to use scaling methods based on logistic regression like Platt
scaling [Pla99], temperature scaling [GPSW17] and beta calibration [KSFF17].

In the setting of probabilistic regression, a model is calibrated if, e.g.,
95% of the true target values are below or equal to a credible level of 95%
(so called _quantile-calibrated regression_) [GBR07, KFE18, SDKF19]. A
regression model is usually calibrated by fine-tuning its predicted CDF in a
post-processing step to match the empirical frequency. Common approaches
utilize isotonic regression [KFE18], logistic and beta calibration [SKF18], as
well as Gaussian process models [SKF18, SDKF19] to build a calibration
mapping. In contrast to quantile-calibrated regression, [SDKF19] have recently
introduced the concept of distribution calibration, where calibration is
applied on a distribution level and naturally leads to calibrated quantiles.

Recent work has shown that miscalibration in the scope of _object detection_
also depends on the position and scale of a detected object [KKSH20]. The
additional box regression output is denoted by $\hat{R}$ with $J$ as the size
of the used box encoding. Furthermore, if we have no knowledge about all
anchors of a model (which is a common case in many applications), it is not
possible to determine the accuracy. Therefore, Küppers et al. [KKSH20] use the
precision as a surrogate for accuracy and propose that an _object detection
model_ is perfectly calibrated if

$\displaystyle\underbrace{\mathbb{P}(M=1|\hat{P}=p,\hat{Y}=y,\hat{R}=r)}_{\text{precision
given }p,y,r}=\underbrace{p}_{\text{confidence}}\quad\forall
p\in[0,1],y\in\mathcal{Y},r\in\mathbb{R}^{J}$ (8)

is fulfilled, where $M=1$ denotes a correct prediction that matches a ground-
truth object with a chosen IoU threshold and $M=0$ denotes a mismatch,
respectively. The authors propose the detection-expected calibration error
(D-ECE) as the extension of the ECE to object detection tasks in order to
measure miscalibration also by means of the position and scale of detected
objects. Other approaches try to fine-tune the regression output in order to
obtain more reliable object proposals [JLM+18, RTG+19] or to add a
regularization term to the training objective such that training yields models
that are both well-performing and well-calibrated [PTC+17, SSH19].

##  7 Aggregation

MMMaram Akila1

From a high-level perspective, a _neural network_ is based on processing
inputs and coming to some output conclusion, e.g., mapping incoming image data
onto class labels. Aggregation or collection of non-independent information on
either the input or output side of this network function can be used as a tool
to leverage its performance and reliability. Starting with the input, any
additional “dimension” to add data can be of use. For example, in the context
of autonomous vehicles this might be input from any further sensor measuring
the same scene as the original one, e.g., stereo cameras or LiDAR. Combining
those sensor sets for prediction is commonly referred to as _sensor fusion_
[CBSW19]. Staying with the example, the scene will be monitored consecutively
providing a whole (temporally ordered) stream of input information. This may
be used either by adjusting the network for this kind of input [KLX+17] or in
terms of a post-processing step, in which the predictions are aggregated by
some measure of temporal consistency.

Another more implicit form of aggregation is training the neural network on
several “independent” tasks, e.g., segmentation and depth regression. Although
the individual task is executed on the same input, the overall performance can
still benefit from the correlation among all given tasks. We refer to the
discussion on _multi-task networks_ in Sec. 9.2. By extension, solving the
same task in multiple different ways, can be beneficial for performance and
provide a measure of redundancy. In this survey, we focus on single-task
systems and discuss _ensemble methods_ in the next section and the use of
_temporal consistency_ in the one thereafter.

###  7.1 Ensemble Methods

MMJoachim Sicking1

Training a neural network is optimizing its parameters to fit a given training
data set. The commonly used gradient-based optimization schemes cause
convergence in a ‘nearby’ local minimum. As the loss landscapes of neural
networks are notoriously non-convex [CHM+15], various locally optimal model
parameter sets exist. These local optima differ in the degree of optimality
(“deepness”), qualitative characteristics (“optimal for different parts of the
training data”) and their generalizability to unseen data (commonly referred
to by the geometrical terms of “sharpness” and “flatness” of minima [KMN+16]).

A single trained network corresponds to one local minimum of such a loss
landscape and thus captures only a small part of a potentially diverse set of
solutions. _Network ensembles_ are collections of models and therefore better
suited to reflect this multi-modality. Various modelling choices shape a loss
landscape: the selected model class and its meta-parameters (like architecture
and layer width), the training data and the optimization objective.
Accordingly, approaches to diversify ensemble components range from
combinations of different model classes over varying training data (bagging)
to methods that train and weight ensemble components to make up for the flaws
of other ensemble members (boosting) [Bis06].

Given the millions of parameters of application-size networks, ensembles of
NNs are resource-demanding w.r.t. computational load, storage and runtime
during training and inference. This complexity increases linearly with
ensemble size for naïve ensembling. Several approaches were put forward to
reduce some dimensions of this complexity: _snapshot ensembles_ [HLP+17]
require only one model optimization with a cyclical learning-rate
schedule—leading to an optimized training runtime. The resulting training
trajectory passes through several local minima. The corresponding models
compose the ensemble. On the contrary, _model distillation_ [HVD15a] tackles
runtime at inference. They ‘squeeze’ a NN ensemble into a single model that is
optimized to capture the gist of the model set. However, such a compression
goes along with reduced performance compared to the original ensemble.

Several hybrids of single model and model ensemble exist: Multi-head networks
[AJD18] share a backbone network that provides inputs to multiple prediction
networks. Another variant are mixture-of-expert models that utilize a gating
network to assign inputs to specialized expert networks [SMM+17]. Multi-task
networks (cf. Sec. 9.2) and Bayesian approximations of NNs (cf. Sec. 6.3 and
Sec. 6.2) can be seen as implicit ensembles.

NN ensembles (or deep ensembles) are not only used to boost model quality.
They pose the frequentist’s approach to estimating NN uncertainties and are
state-of-the-art in this regard [LPB17, SOF+19]. The emergent field of
federated learning is concerned with the integration of decentrally trained
ensemble components [MMR+16] and safety-relevant applications of ensembling
range from autonomous driving [Zha12] to medical diagnostics [RRMH17]. Taking
this safe-ML perspective, promising research directions comprise a more
principled and efficient composition of model ensembles, e.g., by application-
driven diversification, as well as improved techniques to miniaturize
ensembles, e.g., by gaining a better understanding of methods like
distillation. In the long run, better designed, more powerful learning systems
might partially reduce the need for combining weaker models in a network
ensemble.

###  7.2 Temporal Consistency

MMTimo Sämann4

The focus of previous DNN development for semantic segmentation has been on
single image prediction. This means that the final and intermediate results of
the DNN are discarded after each image. However, the application of a computer
vision model often involves the processing of images in a sequence, i.e.,
there is a temporal consistency in the image content between consecutive
frames (for a metric, cf., e.g., [VBB+20]). This consistency has been
exploited in previous work to increase quality and reduce computing effort.
Furthermore, this approach offers the potential to improve the robustness of
DNN prediction by incorporating this consistency as a-priori knowledge into
DNN development. The relevant work in the field of video prediction can be
divided in two major approaches:

  1. 1.

DNNs are specially designed for video prediction. This usually requires
training from scratch and the availability of training data in a sequence.

  2. 2.

A transformation from single prediction DNNs to video prediction DNNs takes
place. Usually no training is required, i.e., the existing weights of the
model can be used unaltered.

The first set of approaches often involves _conditional random fields_ (CRF)
and its variants. CRFs are known for their use as postprocessing step in the
prediction of semantic segmentation, in which their parameters are learned
separately or jointly with the DNN [ZJRP+15]. Another way to use
spatiotemporal features is to include 3D convolutions, which add an additional
dimension to the conventional 2D convolutional layer. Tran et al.[TBF+15] use
3D convolution layers for video recognition tasks such as action and object
recognition. One further approach to use spatial and temporal characteristics
of the input data is to integrate _long short-term memory_ (LSTM) [HS97], a
variant of the _recurrent neural network_ (RNN). Fayyaz et al.[FSS+16]
integrate LSTM layers between the encoder and decoder of their convolutional
neural network for semantic segmentation. The significantly higher GPU memory
requirements and computational effort are a disadvantage of this method. More
recently, Nilsson and Sminchisescu [NS18] deployed _gated recurrent units_ ,
which generally requires significantly less memory. An approach to improve
temporal consistency of automatic speech recognition outputs is known as a
posterior-in-posterior-out (PIPO) LSTM “sequence enhancer”, a postfilter which
could be applicable to video processing as well [LSF19]. A disadvantage of the
described methods is that sequential data for training must be available,
which may be limited or show a lack of diversity.

The second class of approaches has the advantage that it is model-independent
most of the time. Shelhamer et al.[SRHD16] found that the deep feature maps
within the network change only slightly with temporal changes in video
content. Accordingly, [GJG17] calculate the optical flow of the input images
from time steps $t_{0}$ and $t_{-1}$ and convert it into the so-called
_transform flow_ which is used to transform the feature maps of the time step
$t_{-1}$ so that an aligned representation to the feature map $t_{0}$ is
achieved. Sämann et al.[SAMG19] use a confidence-based combination of feature
maps from previous time steps based on the calculated optical flow.

##  8 Verification

MMGesina Schwalbe3

Verification and validation is an integral part of the safety assurance for
any safety critical systems. As of the functional safety standard for
automotive systems [3218], _verification_ means to determine whether given
requirements are met [3218, 3.180], such as performance goals. _Validation_ on
the other side tries to assess whether the given requirements are sufficient
and adequate to guarantee safety [3218, 3.148], e.g., whether certain types of
failures or interactions simply were overlooked. The latter is usually
achieved via extensive testing in real operation conditions of the integrated
product. This differs from the notion of validation used in the machine
learning community in which it usually refers to simple performance tests on a
selected dataset. In this section, we want to concentrate on general
verification aspects for deep neural networks.

Verification as in the safety domain encompasses (manual) inspection and
analysis activities, and testing. However, the contribution of single
processing steps within a neural network to the final behavior can hardly be
assessed manually (compare to the problem of interpretability in Sec. 5).
Therefore, we here will concentrate on different approaches to verification
testing. Section 8.1 covers approaches to systematic test data selection.
While the suggested methods assume full access to the model internals for
coverage measurement, this is not in all cases available. Therefore, Sec. 8.2
highlights assessment approaches that consider the DNN as a black-box
component. Also, the topic of verification activity during operation with the
help of observers is discussed.

###  8.1 Formal Testing

MMChristian Heinzemann2, Gesina Schwalbe3, Matthias Woehrle2

Formal testing refers to testing methods that include formalized and formally
verifiable steps, e.g., for test data acquisition, or for verification in the
local vicinity of test samples. For image data, local testing around valid
samples is usually more practical than fully formal verification: (Safety)
properties are not expected to hold on the complete input space but only on
the much smaller unknown lower-dimensional manifold of real images [WGH19].
Sources of such samples can be real ones or generated ones using an input
space formalization or a trained generative model.

Coverage criteria for the data samples are commonly used for two purposes: (a)
deciding when to stop testing or (b) identifying missing tests. For CNNs,
there are at least three different approaches towards coverage: (1) approaches
that establish coverage based on a model with semantic features of the input
space [GHHW20], (2) approaches trying to semantically cover the latent feature
space of neural network or a proxy network (e.g., an autoencoder) [SS20a], and
(3) approaches trying to cover neurons and their interactions, inspired by
classical software white-box analysis [PCYJ17, SHK+19].

Typical types of properties to verify are simple test performance, local
stability (robustness), a specific structure of the latent spaces like
embedding of semantic concepts [SS20b], and more complex logical constraints
on inputs and outputs, which can be used for testing when fuzzified [RDG18].
Most of these properties require in-depth semantic information about the DNN
inner workings, which is often only available via interpreting intermediate
representations [KWG+18], or interpretable proxies / surrogates (cf. Sec.
5.4), which do not guarantee fidelity.

There exist different testing and formal verification methods from classical
software engineering that have already been applied to CNNs. _Differential
testing_ as used by DeepXPlore [PCYJ17] trains $n$ different CNNs for the same
task using independent data sets and compares the individual prediction
results on a test set. This allows to identify inconsistencies between the
CNNs but no common weak spots. _Data augmentation_ techniques start from a
given data set and generate additional transformed data. _Generic data
augmentation_ for images like rotations and translation are state-of-the-art
for training but may also be used for testing. _Concolic testing_ approaches
incrementally grow test suites with respect to a coverage model to finally
achieve completeness. Sun et al.[SWR+18] use an adversarial input model based
on some norm (cf. Sec. 4.1), e.g., an $L_{p}$-norm, for generating additional
images around a given image using concolic testing. _Fuzzing_ generates new
test data constrained by an input model and tries to identify interesting test
cases, e.g., by optimizing white-box coverage mentioned above [OOAG19].
Fuzzing techniques may also be combined with the differential testing approach
discussed above [GJZ+18]. In all these cases it needs to be ensured that the
image as well as its meta-data remain valid for testing after transformation.
Finally, _proving methods_ surveyed by Liu et al.[LAL+19] try to formally
prove properties on a trained neural network, e.g., based on satisfiablity
modulo theories (SMT). These approaches require a formal characterization of
an input space and the property to be checked, which is hard for non-trivial
properties like contents of an image.

Existing formal testing approaches can be quite costly to integrate into
testing workflows (cf. [WGH19]): Differential testing and data augmentation
require several inferences per initial test sample; concolic and fuzzy testing
apply an optimization to each given test sample, while convergence towards the
coverage goals is not guaranteed; also, the iterative approaches need tight
integration into the testing workflow; and lastly, proving methods usually
have to balance computational efficiency against the precision or completeness
of the result [LAL+19]. Another challenge of formal testing is that machine
learning applications usually solve problems for which no (formal)
specification is possible. This makes it hard to find useful requirements for
testing [ZHML19] and properties that can be formally verified. Even partial
requirements such as specification of useful input perturbations, specified
corner cases, and valuable coverage goals are typically difficult to identify
[SS20a, BTLF20].

###  8.2 Black Box Methods

MMJonas Löhdefink15, Julia Rosenzweig1

In machine learning literature, neural networks are often referred to as black
boxes due to the fact that their internal operations and their decision making
are not completely understood [SZT17], hinting at a lack of interpretability
and transparency. However, in this survey we consider a black box to be a
machine learning model to which we only have oracle (query) access [PMG+17,
TZJ+16]. That means we can query the model to get input-output pairs, but we
do not have access to the specific architecture (or weights, in case of neural
networks). As [OAFS18] describes, black boxes are increasingly wide-spread,
e.g., healthcare, autonomous driving or ML as a service in general, due to
proprietary, privacy or security reasons.

As deploying black boxes gains popularity, so do methods that aim to extract
internal information such as architecture and parameters or to find out,
whether a sample belongs to the training dataset. These include _model
extraction attacks_ [KTP+20, TZJ+16], _membership inference attacks_ [SSSS17],
general attempts to reverse-engineer the model [OAFS18] or to attack it
adversarially [PMG+17]. Protection and counter-measures are also actively
researched: [KMAM18] proposes a warning system that estimates how much
information an attacker could have gained from queries. The authors of
[ABC+18] use watermarks for models to prevent illegal re-distribution and to
identify intellectual property.

Many papers in these fields make use of so-called _surrogate_ , _avatar_ or
_proxy models_ that are trained on input-output pairs of the black box. In
case the black-box output is available in soft form (e.g., logits),
distillation as first proposed by [HVD15b] can be applied to train the
surrogate (student) model. Then, any white-box analysis can be performed on
the surrogates (cf. e.g., [PMG+17]) to craft adversarial attacks targeted at
the black box. More generally, (local) surrogates as for example in [RSG16]
can be used to (locally) explain its decision-making. Moreover, these
techniques are also of interest if one wants to compare or test black-box
models (cf. Sec. 8.1, formal verification). This is the case, among others, in
ML marketplaces, where you wish to buy a pre-trained model [ABC+18], or if you
want to verify or audit that a third-party black-box model obeys regulatory
rules (cf. [CH19]).

Another topic of active research are so-called _observers_. The concept of
observers is to evaluate the interface of a black-box module to determine if
it behaves as expected within a given set of parameters. The approaches can be
divided into _model-explaining_ and _anomaly-detecting_ observers. First,
model explanation methods answer the question of which input characteristic is
responsible for changes at the output. The observer is able to alter the
inputs for this purpose. If the input of the model under test evolves only
slightly but the output changes drastically, this can be a signal that the
neural network is mislead, which is also strongly related to adversarial
examples (cf. Chapter 4). Hence, the reason for changes in the classification
result via the input can be very important. In order to figure out in which
region of an input image the main reason for the classification is located,
[FV17] “delete” information from the image by replacing regions with generated
patches until the output changes. This replaced region is likely responsible
for the decision of the neural network. Building upon this, [UEKH19] adapt the
approach to medical images and generate “deleted” regions by a variational
autoencoder (VAE). Second, anomaly-detecting observers register input and
output anomalies, either examining input and output independently or as an
input-output pair, and predict the black-box performance in the current
situation. In contrast to model-explaining approaches, this set of approaches
has high potential to be used in an online scenario since it does not need to
modify the model input. The maximum mean discrepancy (MMD) [BGR+06] measures
the domain gap between two data distributions independently from the
application and can be used to raise a warning if input or output
distributions during inference deviate too strongly from their respective
training distributions. By use of a GAN-based autoencoder [LFK+20] perform a
domain shift estimation using neural networks in conjunction with the
Wasserstein distance as domain mismatch metric. This metric can also be
evaluated by use of a casual time-variant aggregation of distributions during
inference time.

##  9 Architecture

MMMichael Weber14

In order to solve a specific task, the architecture of a CNN and its building
blocks play a significant role. Since the early days of using CNNs in image
processing, when they were applied to handwriting recognition [LBD+89] and the
later breakthrough in general image classification [KSH12], the architecture
of the networks has changed radically. Did the term of _deep learning_ for
these first convolutional neural networks imply a depth of approximately four
layers, their depth increased significantly during the last years and new
techniques had to be developed to successfully train and utilize these
networks [HZRS16]. In this context, new activation functions [RZL18] as well
as new loss functions [LGG+17] have been designed and new optimization
algorithms [KB15] were investigated.

With regard to the layer architecture, the initially alternating repetition of
convolution and pooling layers as well as their characteristics have changed
significantly. The convolution layers made the transition from a few layers
with often large filters to many layers with small filters. A further trend
was then the definition of entire modules, which were used repeatedly within
the overall architecture as so-called _network in network_ [LCY14].

In areas such as autonomous driving, there is also a strong interest in the
simultaneous execution of different tasks within one single convolutional
neural network architecture. This kind of architecture is called _multi-task
learning_ (MTL) [Car97] and can be utilized in order to save computational
resources and at the same time to increase performance of each task [KTMFs20].
Within such multi-task networks, usually one shared feature extraction part is
followed by one separate so-called head per task [TWZ+18].

In each of these architectures, manual design using expert knowledge plays a
major role. The role of the expert is the crucial point here. In recent years,
however, there have also been great efforts to automate the process of finding
architectures for networks or, in the best case, to learn them. This is known
under the name _neural architecture search_ (NAS).

###  9.1 Building Blocks

MMMichael Weber14

Designing a convolutional neural network typically includes a number of design
choices. The general architecture usually contains a number of convolutional
and pooling layers which are arranged in a certain pattern. Convolutional
layers are commonly followed by a non-linear activation function. The learning
process is based on a loss function which determines the current error and an
optimization function that propagates the error back to the single convolution
layers and its learnable parameters.

When CNNs became state of the art in computer vision [KSH12], they were
usually built using a few alternating convolutional and pooling layers having
a few fully connected layers in the end. It turned out that better results are
achieved with deeper networks and so the number of layers increased [SZ15]
over the years. To deal with these deeper networks, new architectures had to
be developed. In a first step, to reduce the number of parameters, the
convolutional layers with partly large filter kernels were replaced by several
layers with small $3\times 3$ kernels. Today, most architectures are based on
the _network in network_ principle [LCY14], where more complex modules are
used repeatedly. Examples of such modules are the _inception module_ from
GoogleNet [SWY+15] or the _residual block_ from ResNet [HZRS16]. While the
inception module consists of multiple parallel strings of layers, the residual
blocks are based on the _highway network_ [SGS15], which means that they can
bypass the original information and the layers in between are just learning
residuals. With ResNeXt [XGD+17] and Inception-ResNet [SIVA17] there already
exist two networks that combine both approaches. For most tasks, it turned out
that replacing the fully connected layers by convolutional layers is much more
convenient making the networks fully convolutional [LSD15]. These so-called
_fully convolutional networks_ (FCN) are no longer bound to fixed input
dimensions. Note that with the availability of convolutional long short-term
memory (ConvLSTM) structures also fully convolutional recurrent neural
networks (FCRNs) became available for fully scalable sequence-based tasks
[SCW+15, SDF+20].

Inside the CNNs, the _rectified linear unit_ (ReLU) has been the most
frequently used activation function for a long time. However, since this
function suffers from problems related to the mapping of all negative values
to zero like the vanishing gradient problem, new functions have been
introduced in recent years. Examples are the _exponential linear unit_ (ELU),
_swish_ [RZL18] and the _non-parametric linearly scaled hyperbolic tangent_
(LiSHT) [RMDC19]. In order to be able to train a network consisting of these
different building blocks, the loss function is the most crucial part. This
function is responsible for how and what the network ultimately learns and how
exactly the training data is applied during the training process to make the
network train faster or perform better. So the different classes can be
weighted in a classification network with fixed values or so-called
$\alpha$-balancing according to their probability of occurrence. Another
interesting approach is weighting training examples according to their
easiness for the current network [LGG+17], [WFZ19]. For multi-task learning
also weighting tasks based on their uncertainty [KGC18] or gradients [CBLR18]
can be done as further explained in Sec. 9.2. A closer look on how a
modification of the loss function might affect safety-related aspects is given
in Sec. 3.2.

###  9.2 Multi-Task Networks

MMMarvin Klingner15, Varun Ravi-Kumar4, Timo Sämann4, Gesina Schwalbe3

_Multi-task learning_ (MTL) in the context of neural networks describes the
process of optimizing several tasks simultaneously by learning a unified
feature representation [Car97, GHL+20, RBV18, KBFs20] and coupling the task-
specific loss contributions, thereby enforcing cross-task consistency [CPMA19,
LYW+19, KTMFs20].

Unified feature representation is usually implemented by sharing the
parameters of the initial layers inside the encoder (also called feature
extractor). It not only improves the single tasks by more generalized learned
features but also reduces the demand for computational resources at inference.
Not an entirely new network has to be added for each task but only a task-
specific decoder head. It is essential to consider the growing amount of
visual perception tasks in autonomous driving, e.g., depth estimation,
semantic segmentation, motion segmentation, and object detection. While the
parameter sharing can be soft, as in _cross stitch_ [MSGH16] and _sluice
networks_ [RBAS17], or hard [TWZ+18, Kok17], meaning ultimately sharing the
parameters, the latter is usually preferred due to its straightforward
implementation and lower computational complexity during training and
inference.

Compared to implicitly coupling tasks via a shared feature representation,
there are often more direct ways to optimize the tasks inside cross-task
losses jointly. It is only made possible as, during MTL, there are network
predictions for several tasks, which can be enforced to be consistent. As an
example, sharp depth edges should only be at class boundaries of semantic
segmentation predictions. Often both approaches to MTL are applied
simultaneously [CLLW19, YZS+18] to improve a neural network’s performance as
well as to reduce its computational complexity at inference.

While the theoretical expectations for MTL are quite clear, it is often
challenging to find a good weighting strategy for all the different loss
contributions as there is no theoretical basis on which one could choose such
a weighting with early approaches either involving heuristics or extensive
hyperparameter tuning. The easiest way to balance the tasks is to use uniform
weight across all tasks. However, the losses from different tasks usually have
different scales, and uniformly averaging them suppresses the gradient from
tasks with smaller losses. Addressing these problems, Kendall et al. [KGC18]
propose to weigh the loss functions by the _homoscedastic uncertainty_ of each
task. One does not need to tune the weighting parameters of the loss functions
by hand, but they are adapted automatically during the training process.
Concurrently Chen et al. [CBLR18] propose _GradNorm_ , which does not
explicitly weigh the loss functions of different tasks but automatically
adapts the gradient magnitudes coming from the task-specific network parts on
the backward pass. Liu et al. [LJD19] proposed dynamic weight average (DWA),
which uses an average of task losses over time to weigh the task losses.

###  9.3 Neural Architecture Search

MMPatrick Feifel8, Seyed Eghbal Ghobadi8

In the previous sections we saw manually engineered modifications of existing
CNN architectures proposed by ResNet [HZRS16] or Inception [SWY+15]. They are
results of human design and showed their ability to improve performance.
ResNet introduces a _skip connection_ in building blocks and Inception makes
use of its specific _inception module_. Hereby, the intervention by an expert
is crucial. The approach of _neural architecture search_ (NAS) aims to
automate this time-consuming and manual design of neural network
architectures.

NAS is closely related to hyperparameter optimization (HO), which is described
in Sec. 3.1. Originally, both tasks were solved simultaneously. Consequently,
the kernel size or number of filters were seen as additional hyperparamters.
Nowadays, the distinction between HO and NAS should be stressed. The
concatenation of complex building blocks or modules cannot be accurately
described with single parameters. This simplification is no longer suitable.

To describe the NAS process, the authors of [EMH19b] define three steps: (1)
definition of search space, (2) search strategy and (3) performance estimation
strategy.

The majority of search strategies take advantage of the _NASNet search space_
[ZVSL18] which arranges various operations, e.g., convolution, pooling within
a single cell. However, other spaces based on a chain or multi-branch
structure are possible [EMH19b]. The search strategy comprises advanced
methods from sequential model-based optimization (SMBO) [LZN+18], Bayesian
optimization [KNS+18], evolutionary algorithms [RAHL19, EMH19a], reinforcement
learning [ZVSL18, PGZ+18] and gradient descent [LSY19, SDW+19]. Finally, the
performance estimation describes approximation techniques due to the
impracticability of multiple evaluation runs. For a comprehensive survey
regarding the NAS process we refer to [EMH19b].

Recent research has shown that reinforcement learning approaches such as
NASNet-A [ZVSL18] and ENAS [PGZ+18] are partly outperformed by evolutionary
algorithms, e.g., AmoebaNet [RAHL19] and gradient-based approaches, e.g.,
DARTS [LSY19].

Each of these approaches focuses on different optimization aspects. Gradient-
based methods are applied to a continuous search space and offer faster
optimization. On the contrary, the evolutionary approach LEMONADE [EMH19a]