11institutetext:  University of Bremen, Department of Mathematics and Computer
Science, Germany
<EMAIL_ADDRESS>22institutetext: DTU Compute,
Technical University of Denmark, Kongens Lyngby, Denmark
<EMAIL_ADDRESS>Autonomy Assurance International Programme,
University of York, Deramore Lane, York YO10-5GH, United Kingdom

# Probabilistic Risk Assessment of an Obstacle Detection System for GoA 4
Freight Trains

Mario Gleirscher 1133 0000-0002-9445-6863 Anne E. Haxthausen 22
0000-0001-7349-8872 Jan Peleska Partially funded by the German Ministry of
Economics, Grant Agreement 20X1908E.11 0000-0003-3667-9775

###### Abstract

In this paper, a quantitative risk assessment approach is discussed for the
design of an obstacle detection function for low-speed freight trains with
grade of automation (GoA) 4. In this 5-step approach, starting with single
detection channels and ending with a three-out-of-three (3oo3) model
constructed of three independent dual-channel modules and a voter, a
probabilistic assessment is exemplified, using a combination of statistical
methods and parametric stochastic model checking. It is illustrated that,
under certain not unreasonable assumptions, the resulting hazard rate becomes
acceptable for specific application settings. The statistical approach for
assessing the residual risk of misclassifications in convolutional neural
networks and conventional image processing software suggests that high
confidence can be placed into the safety-critical obstacle detection function,
even though its implementation involves realistic machine learning
uncertainties.

###### Keywords:

Autonomous train control, Safety certification, Neural network-based object
detection, Probabilistic risk assessment, Fault tree analysis

##  1 Introduction

#### Motivation and Background

Autonomous transportation systems, their technical feasibility, safety and
security are currently in the main focus of both academic research and
industrial developments. This has been caused by both the significant progress
made in academia regarding the enabling technologies – in particular,
artificial intelligence (AI) – and the attractive business cases enabled by
driverless road vehicles, trains, and aircrafts.

A major obstacle preventing the immediate deployment of autonomous
transportation systems in their designated operational environments is their
safety assessment. The latter poses several technical challenges [19, 18, 2],
in particular, the trustworthiness of AI-based methods involving machine
learning (such as obstacle recognition on roads and railway tracks), as well
as standardisation challenges: in the railway and aviation domains, approved
standards for the certification of safety-critical autonomous trains or
aircrafts are still unavailable.

The standardisation situation is more advanced in the automotive domain, where
a stack of standards involving ISO 26262 [17], ISO 21448 [16], and ANSI/UL
4600 [27] have been approved by the US-American Department of Transportation
for the certification of autonomous road vehicles.111See
https://www.youtube.com/watch?v=xCIjxiVO48Q.

The standard ANSI/UL 4600 is of particular interest, since its authors
emphasise that it is applicable to operational safety assurance on system
level for all types of autonomous products [27, Section 1.2.1], potentially
with a preceding system-specific revision of the checklists proposed in the
standard. In a previous publication, Peleska et al. [21] have suggested a
particular control system architecture for autonomous trains and performed a
qualitative evaluation according to ANSI/UL 4600. This work resulted in the
assessment that a system-level certification based on this standard is
feasible for the class of autonomous metro trains and freight trains, running
in an open operational environment, as can be expected in European railway
networks today.

It should be noted that autonomous trains in closed environments (platform
screen doors, secured tracks, as provided by underground metro trains or
airport people movers, where the problem of unexpected obstacles can be
neglected) already exist since decades [14]. The current challenge consists in
integrating autonomous train operation safely into the “normal” open
operational environments of today’s European railway networks.

#### Objectives and Contributions

This paper complements our previous work [21] with respect to probabilistic
risk assessment and associated verification methods for an autonomous train
control system architecture with the highest grade of automation GoA 4 (no
train engine driver or other personnel present): for a real-world
certification, it is necessary to add a probabilistic risk analysis to the
qualitative evaluation. We specialise here on autonomous low-speed freight
trains travelling across railway networks, the latter providing movement
authorities via interlocking systems and radio block centres. For this type of
train, the most important safety-relevant AI-based sub-system is the _obstacle
detection (OD) module_ , consisting of sensors and perceptors. As explained in
the previous work [21], the reaction to obstacle indications from OD and the
state transitions between fully autonomous GoA4 mode and degraded modes due to
sensor and perceptor failures can be specified, designed, implemented,
verified, validated, and certified with conventional methods, typically
according to standards like EN 50128, EN 50129 [8, 11]. The OD function can be
evaluated according to ANSI/UL 4600, as explained in [21].

Note that while OD has been extensively researched in the automotive domain
[7], the results obtained there cannot be directly transferred to railways
considered in this paper: the two domains require OD for different distances,
and the detection criteria differ, because trains require the obstacle
association with their tracks. We consider the following methodological
aspects and risk analysis results to be the main contributions in this paper.

  1. 1.

We propose a new verification method for (deep) neural networks (NN)
performing classification tasks such as obstacle detection. This method allows
to determine the residual probability ${p_{\text{\Lightning}}}$ for a
systematic classification error in the trained NN. Increasing the training
effort, this method allows us to reduce ${p_{\text{\Lightning}}}$ to an
acceptable value.

  2. 2.

We use a redundant design for the obstacle detection (OD) function introduced
in our previous work [21] that allows to reduce the probability of a detection
failure, due to stochastic independence between redundant channels. To show
stochastic independence, a new statistical method is proposed.

  3. 3.

We use parametric stochastic model checking to obtain probabilistic results
for the hazard rate of the OD function. The parametric approach allows us to
leave some values undefined, so that their influence on the hazard rate
becomes visible, and the concrete risk values can be calculated at a later
point, when reliable concrete parameter values are available.

  4. 4.

The probabilistic analysis shows that, using a redundant 3oo3 design where
each of the three sub-systems consists of a dual-channel module, the OD
function is already certifiable today with an acceptable hazard rate of less
than $10^{-7}/h$ for low-speed autonomous freight trains, even if only camera-
based sensors and perceptors are used.222The requirement for low speed (less
or equal 120km/h) is based on the fact that no reliable failure probabilities
for camera-based obstacle detection modules have been published for trains
with higher velocities [23]. Further reduction of the hazard rate can be
achieved by using additional fail-stop sensors/perceptors based on different
technologies (such as radar and LiDAR), and apply sensor/perceptor fusion over
the results of the non-failing units.

To the best of our knowledge, our contribution is the first to apply this
combination of statistical tests and stochastic model checking to the field of
risk analysis for concrete designs of autonomous train control systems.

#### Overview

The redundant design for the OD sensor/perceptor function proposed in our
previous work [21] is presented again in Section 2. In Section 3, the risk
modelling objectives and the applicable tolerable hazard rate are discussed.
In Section 4, the statistical test strategies and the concept of risk
modelling and probabilistic analysis by means of parametric stochastic model
checking are described. The results of the probabilistic analysis are
presented. Section 5 contains a conclusion. Below, we give references to
related work where appropriate.

##  2 Fail-Safe Design of Obstacle Detection Modules

As described in our previous work [21], the OD function cannot be validated
according to the existing EN 5012x standards, since the latter do not consider
AI-based functions whose behaviour depends on machine learning techniques,
such as (deep) neural networks (NN). Consequently, the safety of the intended
functionality not only depends on the software implementation (of neural
networks), but also on the choice of training data and validation data sets
[16].

Figure 1: 2oo2 design pattern for OD module or similar sense/perceive
functions

The standard ANSI/UL 4600 provides guidance on how the safety of the intended
functionality should be demonstrated in a certification undertaking. From the
performance data available (see discussion in Section 4.6), however, we
conclude that non-redundant sensor/perceptor components relying on machine
learning and neural networks alone cannot achieve the tolerable hazard rates
for safety integrity level SIL-3 discussed in Section 3 below.

Therefore, we suggest a redundant channel design according to the 2oo2
pattern333In the literature, the term “N-out-of-M” is used with different
meanings. In this paper, NooM means that N consistent results produced by
$M\geq N$ channels are needed to be accepted by the voter. Otherwise, the
system falls to the safe side. (see Fig. 1), where two stochastically
independent sensor/perceptor implementations provide data to a voter, and the
voter decides “to the safe side”: for obstacle detection, for example, the
voter would decide ‘obstacle present’ as soon as one channel indicates this.
For distance estimates delivered by both channels in an ‘obstacle present’
situation without disagreement, the voter selects the shorter distance.
Moreover, the voter signals a perceptor error to the kernel, if channels
disagree over a longer time period. To obtain stochastic independence between
channels, we advocate that one channel should be implemented by conventional
image processing methods, without the use of AI. Alternatively, two
differently trained NNs can be used. In any case, the stochastic independence,
needs to be verified by a statistical test, as described in Section 4.5 below.

In the remainder of this paper, the channel using conventional image
evaluation techniques is denote by Channel-$\mathsf{c}$, and the channel using
a NN-based perceptor by Channel-$\mathsf{n}$, as indicated in Fig. 1.

##  3 Risk Assessment Objective and Tolerable Hazard Rate

The top-level hazard to be analysed for OD is

$\mathbf{H}_{\mathbf{OD}}$

OD signals ‘NO OBSTACLE’ to the kernel though an obstacle is present.

We call this situation specified by $\mathbf{H}_{\mathbf{OD}}$ a _false
negative_ produced by OD. The objective of the risk modelling approach and the
associated model evaluation by stochastic model checking is to answer the
following risk analysis question.

> Taking into account the OD design described above: is the resulting hazard
> rate of $\mathbf{H}_{\mathbf{OD}}$ less than the tolerable hazard rate for
> collisions between trains and obstacles?

The _tolerable hazard rate (THR)_ for the obstacle detection (OD) module in a
freight train to produce a false negative (i.e. fail to the unsafe side) is

$\mathbf{THR}_{\mathbf{OD}}=10^{-7}/h\quad\text{\sl-- the tolerable hazard
rate for obstacle detection}$ (1)

according to the discussion by Rangra et al. [22]. This is the THR associated
with SIL-3, and it is justified by the fact that a collision between a freight
train and an obstacle does not endanger as many humans, as would be the case
for a passenger train. This assessment has been confirmed by the research
project ATO-RISK [5], where a more detailed investigation of an adequate SIL
classification for OD has been made. The project also advocates SIL-3 as the
strictest safety integrity level, but additionally elaborates technical and
operational boundary conditions where an even weaker SIL might be acceptable.
These THR-related investigations have not yet been introduced into the current
EN 5012x standards [9, 10, 8, 11], since the latter do not consider automation
grades GoA 3 or GoA 4 yet. Also, the new standard ANSI/UL 4600 does not
provide quantitative SIL-related requirements. It can be expected from these
analyses [22, 5], however, that the “official” THRs, when published in new
revisions of the railway standards, will not be stricter than SIL-3 with
$\mathbf{THR}_{\text{OD}}$ as specified in Equation (1).

##  4 Probabilistic Risk Assessment Approach

The objective of the risk assessment approach described in this section is to
determine a trustworthy _hazard rate_ ($\mathbf{HR}_{\mathbf{OD}}$) for the OD
module and discuss the boundary conditions ensuring that the hazard rate is
less or equal to the tolerable hazard rate:
$\mathbf{HR}_{\mathbf{OD}}\leq\mathbf{THR}_{\mathbf{OD}}$.

###  4.1 Strategy Overview

Figure 2: Overview of the probabilistic risk assessment and assurance approach

The assurance strategy for the OD function comprises the following steps (Fig.
2). (1) An initial functional hazard analysis is performed for the 2oo2 OD
module by means of a fault tree analysis. This fault tree serves to check the
completeness of the following bottom-up steps for risk assessment for one 2oo2
module. (2) The NN-based OD Channel-$\mathsf{n}$ (see Fig. 1) is verified by
means of statistical tests to estimate the residual probability
${p_{\text{\Lightning}}}^{\mathsf{n}}$ for systematic misclassifications that
may be produced by this channel (Step 2$\mathsf{n}$ below). For OD
Channel-$\mathsf{c}$ based on conventional image processing, a similar, but
simpler procedure can be applied; this is described in Step 2$\mathsf{c}$. (3)
The stochastic independence between the two channels is demonstrated by means
of another statistical test. (4) A continuous-time Markov model is created for
the 2oo2 OD module, and a probabilistic risk analysis is performed by means of
parametric stochastic model checking, taking the 2oo2 design into account.
From this Markov model, the failure rate of the 2oo2 OD module is determined
by means of stochastic model checking. (5) With three stochastically
independent OD modules and another voter, a sensor/perceptor fusion can be
achieved, resulting in a failure rate that conforms to the tolerable hazard
rate $\mathbf{THR}_{\mathbf{OD}}$. These steps are now described in detail.

###  4.2 Step 1. Functional Hazard Analysis for OD Module

The fault tree (FT) in Fig. 3 serves as the basis for constructing the
failure-related aspects and the associated mitigations in the model of the
2oo2 OD module. We explain the most important aspects of the FT here. The
remaining elements of Fig. 3 should be clear from the context and the comments
displayed in the figure. The top-level hazard event $\mathbf{H}_{\mathbf{OD}}$
is the occurrence of a false negative (OD signals ‘no obstacle’ to the kernel,
though an obstacle is present).

Figure 3: Fault tree of the 2oo2-OD module for the top-level event
$\mathbf{H}_{\mathbf{OD}}$

In all sub-components of the OD module (voters, sensors, perceptors,
communication links, power supplies), we can assume that no systematic HW, SW
or firmware failures are still present, because we require that the software
is developed according to SIL-4. Therefore, the remaining failure
possibilities to consider are (a) transient HW faults, (b) terminal HW
failures, (c) systematic residual perceptor failures to detect obstacles.

The left-hand side of the FT consider cases where the two channels deliver
contradictory results, but the voter fails to handle the contradiction
appropriately, due to a transient fault. Undetected sensor faults (transient
or terminal) in one channel can arise from HW faults or environmental
conditions (fog, snow, sandstorms). Undetected perceptor faults can arise from
HW faults or residual failures to detect certain types of obstacles.

A simultaneous channel fault (middle box on level 2) leading to
$\mathbf{H}_{\mathbf{OD}}$ could be caused by simultaneous sensor failures or
by simultaneous perceptor faults. The former hazard is mitigated by the sensor
capabilities to detect its own degradations, the stochastic independence of HW
failures (due to the redundant HW design), and by the stochastic independence
of the redundant perceptors, as described in Step 3 below. The latter hazard
is mitigated by reducing the probability for systematic perceptor faults
through the tests performed in Step 2 and by the stochastic independence of
both perceptors demonstrated in Step 3, reducing the probability of a
simultaneous random false negative.

###  4.3 Step 2$\mathsf{n}$. Testing for Systematic Classification Errors:
Channel-$\mathsf{n}$

Equivalence Classes and Their Identification – Channel $\mathsf{n}$ In the
real operational environment, an infinite variety of concrete obstacles could
occur. Therefore, it is desirable to partition their (finite, but still very
large number of) pixel image representations into _input equivalence classes_.
For convolutional neural networks (CNN) typically used for image
classification, it was assumed until recently that such classes could not be
determined by exact calculation or at least by numerical approximation. This
has changed during the last years [12, 3, 4], and we follow the approach of
Benfenati and Marta [3, 4] for this purpose: the authors explain how to
approximate the classification function of a deep NN by means of
differentiable mappings $\Lambda_{i}$ between differentiable manifolds
$M_{i}$:

$M_{0}\stackrel{{\scriptstyle\Lambda_{1}}}{{\longrightarrow}}M_{1}\stackrel{{\scriptstyle\Lambda_{2}}}{{\longrightarrow}}M_{2}\dots
M_{n-1}\stackrel{{\scriptstyle\Lambda_{n}}}{{\longrightarrow}}M_{n}$

Manifold $M_{0}$ represents the set of possible input images, and
$M_{1},\dots,M_{n-1}$ the intermediate hidden layers of the CNN. For our
purposes, $M_{n}$ is a one-dimensional output manifold that can be mapped to
$[0,1]$, such that all $z\in[0,1)$ represent classification result “no
obstacle”, while the $z=1$ represents “obstacle present”. The maps
$\Lambda_{1},\dots,\Lambda_{n-1}$ represent the inter-layer mappings of the
CNN. Some of these are smooth (e.g. the filter applications), others can be
approximated by smooth alternatives. The map
$\Lambda_{n}:M_{n-1}\longrightarrow M_{n}$ is a smooth approximation444For
example, by means of the Gaussian-error-linear unit (GELU). of the ReLU
(rectified linear unit) activation function, typically used in a CNN.

Using the Euclidean metric $g_{n}$ on $M_{n}$, repetitive pullbacks of $g_{n}$
through $\Lambda_{n},\dots,\Lambda_{1}$ introduce a degenerate Riemannian
metric $g_{0}$ on $M_{0}$: using $\Lambda$ to denote the composite map
$\Lambda_{n}\circ\dots\circ\Lambda_{1}:M_{0}\longrightarrow M_{n}$, the
distance from $p$ to $p^{\prime}$ in $M_{0}$ is simply
$|\Lambda(p)-\Lambda(p^{\prime})|$.

Given an image $p\in M_{0}$ that is classified by the CNN as “obstacle”, so
that $\Lambda(p)=1$, all images $p^{\prime}$ that can be reached from $p$ on a
null curve, that is, a piecewise smooth curve of length null in the degenerate
metric $g_{0}$, are also classified by the CNN as obstacles.555The length of
differentiable curve $\gamma$ in $M_{0}$ is obtained by integrating over the
length of its tangent vectors is some curve parametrisation, say,
$\gamma(t),t\in[0,1]$ [6]. The length of a tangent vector
$v=\stackrel{{\scriptstyle\cdot}}{{\gamma}}(t)$ is obtained by calculating
$\sqrt{g_{0}(v,v)}$: the metric $g_{0}$ on $M_{0}$ induces a bilinear form
(also denoted by $g_{0}$ on the tangent space at $\gamma(t)$. For degenerate
metrics $g_{0}$, non-zero tangent vectors can have zero length, since
$g_{0}(v,v)=0$. The _obstacle image space_ ${\cal
O}=\Lambda^{-1}(\\{1\\})\subseteq M_{0}$ of all images classified by the CNN
as obstacles, however, is not null-connected: for some image points
$p,p^{\prime\prime}$ that are both classified as obstacles, every piecewise
smooth curve connecting $p$ and $p^{\prime\prime}$ traverses one or more
regions of points mapped to “no obstacle”. Each sub-manifold of ${\cal O}$
consisting of pairwise null-connectible image points represents an equivalence
class of the CNN.

Statistical Test Based on Coupon Collector’s Problem Consider the $\ell$
equivalence classes $c_{1},\dots,c_{\ell}\subseteq{\cal O}$ representing null-
connected image sets to be classified as obstacles by the trained NN
implementing the perceptor of Channel $\mathsf{n}$. In an ideal perceptor,
every real-world obstacle would produce an image $p\in M_{0}$ fitting into
some class $c_{i}$, that is, $p\in c_{i}$. We are interested in an estimate
for the residual error probability ${p_{\text{\Lightning}}}^{\mathsf{n}}$ for
the existence of a further subset of “undetetected” images
$u_{\ell+1}\subseteq M_{0}\setminus{\cal O}$ representing obstacles in the
real word, but classified as “no obstacle” by the NN, because they are not
contained in $\bigcup_{i=1}^{\ell}c_{i}={\cal O}$.

Recall the naive statistical approach to estimate
${p_{\text{\Lightning}}}^{\mathsf{n}}$: one could apply the Monte Carlo method
by taking $n$ independent image samples $\\{p_{1},\dots,p_{n}\\}$ representing
obstacles and determine
$\hat{P}_{\text{\Lightning},n}=\frac{n_{\text{\Lightning}}}{n}$, where
$n_{\text{\Lightning}}$ denotes the number of false negatives obtained by the
NN on the sample $\\{p_{1},\dots,p_{n}\\}$. Then
$\hat{P}_{\text{\Lightning},n}$ converges to ${p_{\text{\Lightning}}}$ for
$n\rightarrow\infty$ with probability 1. This approach is unsuitable for our
purposes, since the sample size $n$ must be very large for trustworthy
estimation of small residual error probabilities
${p_{\text{\Lightning}}}^{\mathsf{n}}$.666Weijing Shi et al. [24] state that a
misclassification probability of ${p_{\text{\Lightning}}}\approx 10^{-12}$
would require a sample size of $n\approx 10^{13}$.

As a more promising alternative approach, we therefore suggest to obtain an
estimate for ${p_{\text{\Lightning}}}^{\mathsf{n}}$ by means of statistical
tests based on a generalised variant of the _Coupon Collector’s Problem (CCP)_
[13]. This CCP variant considers $\ell$ different types of coupons in an urn,
such that drawing a coupon of type $i\in\\{1,\dots,\ell\\}$ from the urn with
replacement has probability $p_{i}$. The CCP considers the random variable $X$
denoting the number of draws necessary to obtain a coupon of each type at
least once. The expected value of $X$ is calculated by [13]

$E(X)=\int_{0}^{\infty}\big{(}1-\prod_{i=1}^{\ell}(1-e^{-p_{i}x})\big{)}\mathrm{d}x\;.$
(2)

For the application of the CCP in the context of this article, we assume the
availability of a large database $D$ of ‘obstacle-on-track’ sample images
representing the urn in the CCP. We assume that there exists a random
selection mechanism for $D$, such that the selected samples are stochastically
independent from each other, concerning their ontology classification. The
obstacle images from $D$ take the role of the CCP-samples to be drawn from the
urn. If the image sample fits into equivalence class $c_{i},\
i\in\\{1,\dots,\ell\\}$, this corresponds to the CCP coupon being of type $i$.

For a verification run $V_{k}$, we draw sample images from $D$ until every
known equivalence class $c_{1},\dots,c_{\ell}$ has been covered by at least
one sample. If a run $V_{k}$ fails because a substantial number of samples did
not fit into any class, the training of the neural network is repeated with an
extended set of samples, and a new collection of equivalence classes
$c_{1}^{\prime},\dots c_{\ell^{\prime}}^{\prime}$ is determined, as described
above. Then the verification runs are repeated.

While $E(X)$ gives us an idea of the number of samples needed to cover every
equivalence class $c_{i}$ at least once, we need a (higher) value of samples
$\overline{S}$ required to discover all members with sufficiently high
probability. Hence we are looking for an $\overline{S}\in\mathbb{N}$ such that
the probability $\tau=\mathsf{P}(X<\overline{S})$ is close to 1. Adapting the
estimation approach suggested by Hauer et al. [15] to our problem, the
probability $\mathsf{P}(X<S)$ for any $S\in\mathbb{N}$ can be calculated by
using a large number of verification runs $V_{k},\ k=1,\dots,m$ and count the
occurrences $occ(i),\ i\in\mathbb{N}$ of verification runs in
$\\{V_{1},\dots,V_{m}\\}$, where all equivalence classes
$c_{1},\dots,c_{\ell}$ have been covered after $i$ samples. Note that
$occ(i)=0$ for $i<\ell$, because we need at least $\ell$ images to cover that
many classes. Then $\mathsf{P}(X<S)$ can be estimated by

$\mathsf{P}(X<S)=\frac{1}{m}\sum_{i=1}^{S}occ(i)\;,$ (3)

so we select $\overline{S}$ as the smallest $S$ such that the value of
$\mathsf{P}(X<S)$ calculated by Equation (3) is greater or equal $\tau$. The
number $m$ of verification sets $V_{k}$ to be used in Equation (3) determines
the confidence that we can have in the estimate for
$\mathsf{P}(X<\overline{S})$; minimal values for $m$ achieving a given
confidence level can be calculated as described by Hauer et al. [15].

Assume that the NN implementing the perceptor of Channel $\mathsf{n}$ has
$\ell$ equivalence classes, as described above. Assume further that the
verification runs $V_{k},\ k=1,\dots,m$ have been performed successfully and
resulted in probability estimates $p_{1},\dots,p_{\ell}$ for an obstacle to
fall into class $c_{1},\dots,c_{\ell}$. Now we test the hypothesis that the
successful verification runs have overlooked an obstacle type that does not
fit into any of the identified classes $c_{1},\dots,c_{\ell}$, but is
associated with a subset $u\subseteq M_{0}\setminus{\cal O}$ of obstacle
images leading to false negatives. Assume further that the occurrence
probability for such an obstacle is $p_{u}$. Then we have to re-scale the
probability estimates $p_{1},\dots,p_{\ell}$ to
$p_{i}^{\prime}=(1-p_{u})p_{i}$, so that
$\big{(}\sum_{i=1}^{\ell}p_{i}^{\prime}\big{)}+p_{u}=1.$ According to Equation
(2), the expected number of samples needed for covering the $(\ell+1)$ classes
is

$E(X)=\int_{0}^{\infty}\Big{(}1-\big{(}\prod_{i=1}^{\ell}(1-e^{-p_{i}x})\big{)}\cdot(1-e^{-p_{u}x})\Big{)}\mathrm{d}x\;.$

Now we apply Equation (3) for this extended hypothetical partition
$\\{c_{1},\dots,c_{\ell},u\\}$, to estimate the number $m_{\text{new}}$ of
verification runs $V_{k}$ to be performed in order to get at least one sample
for each partition element, including $u$, again with a high confidence level
and the same probability
$\mathsf{P}(X_{\text{new}}<\overline{S}_{\text{new}})=\tau$. Then the
verification runs are extended to $V_{1},\dots,V_{m_{\text{new}}}$. If this
extended suite of verification runs does not reveal the existence of such a
partition element $u$, we can conclude with the given confidence level that
the original set of classes $c_{1},\dots,c_{\ell}$ implemented by the NN is
complete.

###  4.4 Step 2$\mathsf{c}$. Testing for Systematic Classification Errors:
Channel-$\mathsf{c}$

Equivalence Classes and Their Identification For the perceptor of Channel c,
an input equivalence class consists of a set of images covering the same path
in the perceptor software control flow graph, so that they all end up with the
same classification result.

Statistical Tests The statistical tests regarding the probability
${p_{\text{\Lightning}}}^{\mathsf{c}}$ of systematic residual classification
errors in Channel-$\mathsf{c}$ can be performed in analogy to Step
2$\mathsf{n}$, but now the equivalence classes are identified by software
control flow paths instead of null-connected sub-manifolds of the obstacle
image space ${\cal O}$.

###  4.5 Step 3. Stochastic Independence Between the two Channels

On hardware-level, stochastic independence between the two OD channels is
justified by redundancy and segregation arguments: the channels use different
cameras, and the perceptors are deployed on different processor boards with
separate power supplies and wiring, both for electrical current and
communication links between sensors, perceptors, and voter. There are no
communication or synchronisation links between the channels.

The remaining common cause failure of the two channels that cannot be avoided
is given by adverse weather conditions (like fog, sand storms, or snow)
corrupting the camera images. This can be detected by the sensors themselves
by identifying consecutive images as identical without discernible shapes
(fog) or as white noise (sand storm, snow). We can expect that at least one of
the two channels detects this condition and raises a fault that will cause the
voter to signal ‘OD failure’ to the kernel. This will lead to an emergency
stop of the train. Consequently, we are only interested in stochastic
independence of the two perceptors in absence of this detectable common cause
failure.

As discussed for the fault tree model of Step 1, the only remaining potential
cause for stochastic dependence would be that the two perceptors evaluate
images “in a similar way”. To demonstrate the absence of such a dependency, we
apply the method of Sun et al. [25] for explaining the reasons for
classification results: the method provides an algorithm for identifying a
subset of image pixels that were essential for obtaining the classification
result. For the demonstration of stochastic independence, we define two bit
matrix-valued random variables $R_{i},\ i=\mathsf{c},\mathsf{n}$. Variable
$R_{i}$ encodes these explanations obtained by Channel $\mathsf{c}$ and
Channel $\mathsf{n}$, respectively, as a pixel matrix, where only the
essential pixels are represented by non-zero values.

While performing the verification runs $V_{k}$ of Step 2c and Step 2n, the
sequence of matrix values $R_{\mathsf{c}},R_{\mathsf{n}}$ obtained from the
images of $V_{k}$ are determined (both channels need to run the same
verifications $V_{k}$ in the same order, so that the same sequence of images
is used over all runs $V_{k}$). Then the stochastic independence between
$R_{\mathsf{c}}$ and $R_{\mathsf{n}}$ can be tested by means of the widely-
used $\chi^{2}$-test. If this test indicates a stochastic dependence between
perceptors $\mathsf{c}$ and $\mathsf{n}$, then the NN has to be retrained with
a different data set, or another structure of the NN (for example, another
layering) needs to be chosen.

The main result obtained from the stochastic independence of $R_{\mathsf{c}}$
and $R_{\mathsf{n}}$ is that false negative misclassifications occur at the
two channels in a stochastically independent way. More formally, let $X_{i},\
i=\mathsf{c},\mathsf{n}$ be two Boolean random variables with interpretation
“$X_{i}=\text{true}$ if and only if a false negative misclassification occurs
in the perceptor of Channel $i$”. Then, with
$a,b\in\\{\text{true},\text{false}\\}$, stochastic independence allows us to
calculate

$\mathsf{P}(X_{\mathsf{c}}=a\land
X_{\mathsf{n}}=b)=\mathsf{P}(X_{\mathsf{c}}=a)\cdot\mathsf{P}(X_{\mathsf{n}}=b)\;.$

In particular, the probability of a simultaneous misclassification in both
channels (case $a=\text{true}~{}\wedge~{}b=\text{true}$) can be calculated as
the product of the misclassification probabilities of each channel.

###  4.6 Step 4. Determining $\mathbf{HR}_{\mathbf{OD}}$ for the 2oo2-OD
Module

We now quantify the probability of an $\mathbf{H}_{\mathbf{OD}}$ event for a
single module demand. Recall that $\mathbf{H}_{\mathbf{OD}}$ means an obstacle
is present within OD range or the OD module is provided with degraded data,
but neither is detected by the module (i.e., $r=\mathrm{no}$) and the module’s
voter component misses to raise an error flag (i.e., $f=\mathrm{false}$) that
could be considered by the automatic train protection.

Figure 4: SysML activity chart describing the data processing in the OD module

We model the two channels ($i\in\\{\mathsf{c,n}\\}$) and the voter of the OD
module (Fig. 1) using a SysML activity chart with a Petri net interpretation
as shown in Fig. 4. For each processing cycle (i.e., when new tokens are
placed at the beginning of each line), both channels perform a _sense_ and a
_perceive_ action with the data $d\in D$ flowing (i.e., carried with the
tokens) from the environment into both channels and from the top to the
bottom. For illustration, we use $D=0..2$, with $d=0$ for “obstacle absent”,
$d=1$ for “obstacle present”, and $d=2$ for “degraded inputs” (e.g., dense
fog, covered sensors). The _environment_ part enables a conditional risk
assessment of the OD module based on the stochastic _generation_ of inputs
from $D$. In our assessment, the environment only generates $d\in\\{1,2\\}$.

We use a continuoustime Markov chain (CTMC) as stochastic model for the OD
module. Given variables $V$, a CTMC is a tuple
$\mathcal{M}=(S,s_{0},\mathbf{R},L)$ with state space $S\in
2^{V\to\mathbb{N}}$, initial state $s_{0}\in S$, transition rate matrix
$\mathbf{R}\colon S\times S\to\mathbb{R}_{\geq 0}$, and labelling function
$L\colon S\to 2^{\mathit{AP}}$ for a set $\mathit{AP}$ of atomic propositions.
Properties to be checked of $\mathcal{M}$ can be specified in continuous
stochastic logic (CSL). For example, the expression
$\mathcal{M},s\models\mathsf{P}_{>p}[\mathsf{F}\phi]$ is satisfied if and only
if the CSL formula $\mathsf{P}_{>p}[\mathsf{F}\phi]$ is true in $\mathcal{M}$
and $s\in S$, that is, if the probability ($\mathsf{P}$) of eventually
($\mathsf{F}$) reaching some state $s^{\prime}$ satisfying $\phi$ from $s$ in
$\mathcal{M}$ is greater than $p$. If $\phi$ is a propositional formula, its
satisfaction in $s\in S$ ($s\models\phi$) is checked using the atomic
propositions from $L(s)$. More details about CSL model checking, for example,
with the Prism model checker, can be obtained from [20].

To work with CTMCs, we translate the activity chart from Fig. 4 into a
probabilistic guarded command777Such commands are of the form
$[a]g\to\lambda_{1}\colon u_{11}\&u_{12}\dots+\dots+\lambda_{n}\colon
u_{nm}\dots$ with an action $a$, a guard $g$, and probabilistic multiple-
assignments $u_{ij}$ applied with rate $\lambda_{i}$. program (Listing 5).
From this program, a probabilistic model checker can derive a CTMC
$\mathcal{M}$ that formalises the semantics of the activity chart, allowing
the processing in the two channels to be non-deterministically concurrent,888
Each of the timed synchronised interleavings of the four sequential components
in Fig. 4 carries information about the _expected time of occurrence_ of
events and, thus, the accumulated expected duration of a particular
interleaving. This allows one to derive timed termination probabilities and
rates of the processing cycle. finally synchronising on the _vote_ action.
This type of concurrency enables us to make assumptions about the processing
speed in the two channels independently and flexibly.

The Listing 5 shows fragments of the program describing one channel, its
processing stages, and the voter component. The state space $S$ of the
associated $\mathcal{M}$ is defined via a stage counter ($s_{i}\in 0..5$),
data flow variables ($\mathit{sen}_{i}$, $\mathit{per}_{i}$,
$\mathit{com}_{i}:D$) for each channel, variables for the input data $d:D$,
the result $r:D$, and a Boolean failure flag $f$. We use the initial state
$s_{0}(v)=0$ for $v\neq f$ and $s_{0}(f)=\mathrm{false}$. The transition rate
matrix $\mathbf{R}$ is defined indirectly via probabilistic updates: For each
update $u$ (e.g., a fault) of an action $a$ (e.g.,
$\mathtt{Perceive}_{\textsf{n}}^{o}$), we provide a rate
$\lambda_{a,u}=p_{u}\cdot\lambda_{a}$, where $p_{u}$ is the probability of
seeing update $u(s)$ if an action $a$ is performed in state $s$ and
$\lambda_{a}$ is the average speed, frequency, or rate at which action $a$ in
$s$ is completed. We can either provide a compound rate $\lambda_{a,u}$ or
separate parameters $p_{u}$ and $\lambda_{a}$. For example, for the
$\mathtt{Perceive}_{\textsf{n}}^{o}$ action (i.e., NN-based perception, given
the sensor forwards a picture with an obstacle, line 4), we consider a single
failure mode (line 5) with probability ${p_{\text{\Lightning}}}^{\mathsf{n}}$
(estimated in Sect. 4.3) multiplied with a perception speed estimate
$\lambda_{\mathsf{pn}}$.

As described in Sect. 2, the output at the end of each processing cycle is a
tuple $(r,f)$ with the voting result $r$ and the status of the failure flag
$f$. Under normal operation, $r$ contains either the concurring result of both
channels or an error to the _safe side_ (i.e.,
$\max_{i\in\\{\mathsf{c,n}\\}}\\{\mathit{com}_{i}\\}$) in case of
contradictory channel results. For example, if one channel reports an obstacle
and the other does not, the nominal voter would forward “obstacle present” and
raise a flag.

Figure 5: Probabilistic program fragment showing parts of the NN channel and
the voter. The influence on some of the FT events from Fig. 3 is indicated.

For the model, we need to provide probability and speed estimates of the
channel- and stage-specific faults. For example, we use
${p_{\text{\Lightning}}}^{\mathsf{n}}$ and $\lambda_{\mathsf{pn}}$ for the
probability of an NN-perceptor fault $\mathit{SP}_{\mathsf{n}}$ and the
speed999Speed estimates can be set to 1 for a CTMC where estimates are
unavailable and relative speed and performance does not play a role in the
risk assessment. of the associated fault-prone action
$\mathtt{Perceive}_{\mathsf{n}}^{o}$. Analogously, we provide
${p_{\text{\Lightning}}}^{\mathsf{c}}$ and $\lambda_{\mathsf{pc}}$ for the
conventional perceptor, ${p_{\text{\Lightning}}}^{v}$ and
$\lambda_{\mathsf{v}}$ for $V_{\mathsf{r}}$, and, similarly, for the other
events defined in the fault tree (e.g., $\mathit{SP}_{\mathsf{r}}$,
$\mathit{SP}_{\mathsf{s}}$, $\mathit{SC}$, $\mathit{SS}_{\mathsf{di}}$,
$\mathit{SS}_{\mathsf{sh}}$; Fig. 3). Based on these parameters, the CTMC
allows us to quantify time-independent probabilities of intermediate and top-
level events in the fault tree, for example, $\mathit{UC}$, $S$, and, in
particular, the probability $\mathsf{P}[\mathit{FN}]$ of the top-level event
$\mathbf{H}_{\mathbf{OD}}$, that is, a false negative under the condition that
either an obstacle or degraded data is present.

To make our assessment independent of a particular
${p_{\text{\Lightning}}}^{\mathsf{n}}$ and
${p_{\text{\Lightning}}}^{\mathsf{c}}$, we perform a parametric CTMC analysis
that yields a function
$\mathsf{P}[\mathit{FN}]({p_{\text{\Lightning}}}^{\mathsf{n}},{p_{\text{\Lightning}}}^{\mathsf{c}})$.
Consider the parametric CTMC
$\mathcal{M}({p_{\text{\Lightning}}}^{\mathsf{n}},{p_{\text{\Lightning}}}^{\mathsf{c}})=(S,s_{0},\mathbf{R}({p_{\text{\Lightning}}}^{\mathsf{n}},{p_{\text{\Lightning}}}^{\mathsf{c}}),L)$
derived from Listing 5. By $S_{\mathsf{od}}=\\{s\in S\mid
s_{\mathsf{c}}=s_{\mathsf{n}}\land s_{\mathsf{c}}=1\land(d=1\lor d=2)\\}$, we
select only those _intermediate states_ where the OD module is provided with
either a present obstacle ($d=1$) or degraded data ($d=2$) at its sensing
stage ($s_{\mathsf{c}}=1$). According to the fault tree (Fig. 3), we select
_final states_ with the predicate

$\displaystyle\mathrm{fin}$
$\displaystyle\equiv\;\big{(}(s_{\mathsf{c}}=s_{\mathsf{n}}\land
s_{\mathsf{c}}=5\land\neg f)$ at final stage $s_{i}=5$ with muted flag
($V_{\mathrm{r}}$), $\displaystyle\land\;((com_{\mathsf{c}}\neq
com_{\mathsf{n}})$ observe contradictory results ($UC$), or a
$\displaystyle\lor(com_{\mathsf{c}}=com_{\mathsf{n}}\land r\neq d))\big{)}$
simultaneous channel or voter fault ($S,V_{r}$).

These are all states at the final processing stage ($s_{i}=5$) that correspond
to either $\mathit{UC}$ or $S$ in the fault tree and, hence,
$\mathbf{H}_{\mathbf{OD}}$. Then, we compute
$\mathsf{P}[\mathit{FN}]({p_{\text{\Lightning}}}^{\mathsf{n}},{p_{\text{\Lightning}}}^{\mathsf{c}})$
by quantifying ($\mathsf{P}_{=?}[\cdot]$) and accumulating
($\sum_{S_{0}}\cdot$) the conditional probabilities of the unbounded
reachability ($\mathsf{F}\;\mathrm{fin}$) of a final state in
$S_{\mathsf{f}}=\\{s\in S\mid s\models\mathrm{fin}\\}$ from some intermediate
state $s\in S_{\mathsf{od}}$. The corresponding formula is

$\displaystyle\mathbf{H}_{\mathbf{OD}}({p_{\text{\Lightning}}}^{\mathsf{n}},{p_{\text{\Lightning}}}^{\mathsf{c}})=\mathsf{P}[\mathit{FN}]({p_{\text{\Lightning}}}^{\mathsf{n}},{p_{\text{\Lightning}}}^{\mathsf{c}})$
(4) $\displaystyle=\sum_{s\in
S_{\mathsf{od}}}\big{(}\underbrace{\mathcal{M}({p_{\text{\Lightning}}}^{\mathsf{n}},{p_{\text{\Lightning}}}^{\mathsf{c}}),s_{0}\models\mathsf{P}_{=?}[\mathsf{F}\,s]}_{\text{probability
of reaching $s$ from
$s_{0}$}}\big{)}\cdot\big{(}\underbrace{\mathcal{M}({p_{\text{\Lightning}}}^{\mathsf{n}},{p_{\text{\Lightning}}}^{\mathsf{c}}),s\models\mathsf{P}_{=?}[\mathsf{F}\,\mathrm{fin}]}_{\text{probability
of reaching $\mathrm{fin}$ from $s$}}\big{)}\;.$

Note that the CSL quantification operator $\mathsf{P}_{=?}$ used inside the
sum operator transforms the satisfaction relation $\models$ into a real-valued
function.

Shown in Fig. 6a, one OD module has a residual probability for an undetected
false negative in range
$\mathsf{P}[\mathit{FN}]({p_{\text{\Lightning}}}^{\mathsf{n}},{p_{\text{\Lightning}}}^{\mathsf{c}})\in[0.0016,0.005]$,
depending on the residual misclassification probability
${p_{\text{\Lightning}}}^{\mathsf{n}},{p_{\text{\Lightning}}}^{\mathsf{c}}\in[0.02,0.1]$.
Reports on failure probabilities of image classification based on both
conventional image evaluation and trained neural networks indicate that, as of
today, neither ${p_{\text{\Lightning}}}^{\mathsf{n}}$ nor
${p_{\text{\Lightning}}}^{\mathsf{c}}$ will be below $0.02$ [23, 1]. For
example, assuming
${p_{\text{\Lightning}}}^{\mathsf{n}}={p_{\text{\Lightning}}}^{\mathsf{c}}=0.04$,
one OD module alone will have a hazard rate of approximately
$\lambda_{\mathsf{od}}\cdot 0.0016=6\cdot 10^{-5}$ with
$\lambda_{\mathsf{od}}=2/24h^{-1}$ denoting the frequency of obstacle
occurrences or degraded data. While this does not yet conform to
$\mathbf{THR}_{\mathbf{OD}}$ specified in Equation (1) (see also the
parameter-dependent hazard rates in Fig. 6a), it allows to apply sensor fusion
to create a composite OD system respecting $\mathbf{THR}_{\mathbf{OD}}$.

(a)
$\mathsf{P}[\mathit{FN}]({p_{\text{\Lightning}}}^{\mathsf{n}},{p_{\text{\Lightning}}}^{\mathsf{c}})$

(b)
$\mathbf{HR}_{\mathbf{OD}}({p_{\text{\Lightning}}}^{\mathsf{n}},{p_{\text{\Lightning}}}^{\mathsf{c}})$

Figure 6: The functions in (a) and (b) result from computing the symbolic
solution of the right-hand side of Eq. (4) using the parametric CTMC
$\mathcal{M}({p_{\text{\Lightning}}}^{\mathsf{n}},{p_{\text{\Lightning}}}^{\mathsf{c}})$.

###  4.7 Step 5. Determining $\mathbf{HR}_{\mathbf{OD}}$ for the Fused 3oo3 OD
System

We create a 3oo3 sensor fusion system, using three stochastically independent
(that is, differently trained and with different image recognition software)
OD modules with 2-channel structure as described above: a 3oo3 voter raises an
error leading immediately to braking the train, as soon as an “obstacle/no
obstacle” indication is no longer given unanimously by the three OD modules.
This means that single and double faults are immediately detected and result
in immediate fault negation by going into a safe state. As explained in the
previous paragraph, each module has a failure rate that is smaller than
$2\cdot 10^{-4}h^{-1}$. Therefore, applying the rule [11, B.3.5.2, 5)] of EN
50129, the detection of triple faults for such a system is not required.

Assuming that all three OD modules have a probability for producing a false
negative that is less or equal to
$\mathsf{P}[\mathit{FN}]({p_{\text{\Lightning}}}^{\mathsf{n}},{p_{\text{\Lightning}}}^{\mathsf{c}})$,
the hazard rate for a safety-critical false negative produced by this 3oo3 OD
system (Fig. 6b) is

$\mathbf{HR}_{\mathbf{OD}}({p_{\text{\Lightning}}}^{\mathsf{n}},{p_{\text{\Lightning}}}^{\mathsf{c}})=\lambda_{\mathsf{od}}\cdot\big{(}\mathsf{P}[\mathit{FN}]({p_{\text{\Lightning}}}^{\mathsf{n}},{p_{\text{\Lightning}}}^{\mathsf{c}})\big{)}^{3}\;.$
(5)

With $\mathsf{P}[\mathit{FN}](0.04,0.04)=0.0016$ as discussed above, this
ensures

$\mathbf{HR}_{\mathbf{OD}}(0.04,0.04)=\frac{2}{24}\cdot 0.0016^{3}=3.413\cdot
10^{-10}<\mathbf{THR}_{\mathbf{OD}}=10^{-7}\;.$

##  5 Conclusion

We have presented a 5-step approach to probabilistic risk assessment for
autonomous freight trains in open environments with automated obstacle
detection. This approach is based on a preceding qualitative evaluation of the
assurance steps required to enable a certification according to the standard
ANSI/UL 4600. The risk figures obtained indicate that autonomous freight
trains based on the train control system architecture advocated here can
achieve adequate safety with obstacle detection based on camera images alone,
provided that at least three independent 2oo2 OD modules are fused into an
integrated 3oo3 OD detector. The costs to achieve this can be expressed in the
number of statistical tests to be performed in order to guarantee these upper
risk bounds. Moreover, our example illustrates that, under realistic
assumptions, the failure probability of an OD module corresponds to the
product of the classification error probabilities; sensor and voter faults
play no significant role in the overall assessment.

The statistical testing strategy described here requires considerable effort,
since several verification runs $\\{V_{1},\dots,V_{m_{\text{new}}}\\}$ are
involved and have to be repeated if too many false negatives require a new
training phase. To avoid the latter, it is advisable to verify first that the
trained NN is _free of adversarial examples_ : in our case, these are images
$p,p^{\prime}$ that are close to each other in some metric conforming to the
human understanding of image similarity (e.g. two similar vehicles standing on
the track at a level crossing), where $p$ is correctly classified as an
obstacle, but $p^{\prime}$ is not. A highly effective testing method for
detecting adversarial examples has been suggested by Sun et al. [26]. It is
based on a novel structural coverage metric for CNN, that is analogous to the
MC/DC coverage in software testing. A detailed verification cost evaluation
will be considered in a future contribution.

It is important to note that the introduction of redundancy (e.g. 2oo2) to
achieve fail-safe designs, as described in EN 50129 [11, B.3.1], is only
admissible for random HW faults according to this standard. The occurrence of
residual HW design failures, SW failures, or failures due to imperfect machine
learning processes is not taken into account. For HW design and SW (including
the implementation of NN software) developed and verified according to SIL-4,
the assumption that safety-critical residual failures exist can be also
neglected for the context of this paper. The probability of a residual
systematic failure in a trained NN, however, needs to be taken into account.
Therefore, a certification of the OD module in an autonomous freight train
cannot be performed on the basis of the current EN 5012x standards alone.
Instead, ANSI/UL 4600 needs to be used: according to this new standard for
autonomous control systems, the failure model is allowed to take systematic
residual failures caused by imperfect machine learning into account.

## References

  * [1] Akai, N., Hirayama, T., Murase, H.: Experimental stability analysis of neural networks in classification problems with confidence sets for persistence diagrams. Neural Networks 143, 42–51 (2021), https://doi.org/10.1016/j.neunet.2021.05.007
  * [2] Araujo, H., Mousavi, M.R., Varshosaz, M.: Testing, validation, and verification of robotic and autonomous systems: A systematic review. ACM Trans. Softw. Eng. Methodol. (may 2022), https://doi.org/10.1145/3542945, just Accepted
  * [3] Benfenati, A., Marta, A.: A singular riemannian geometry approach to deep neural networks i. theoretical foundations. Neural Networks 158, 331–343 (2023), https://doi.org/10.1016/j.neunet.2022.11.022
  * [4] Benfenati, A., Marta, A.: A singular riemannian geometry approach to deep neural networks II. reconstruction of 1-d equivalence classes. Neural Networks 158, 344–358 (2023), https://doi.org/10.1016/j.neunet.2022.11.026
  * [5] Braband, J., Lindner, L., Rexin, F.: Risk analyses for obstacle detection in automatic driving. SIGNAL+DRAHT 115(3), 12–20 (2023)
  * [6] Carmo, M.d.: Differential Geometry of Curves and Surfaces. Dover Books on Mathematics, Dover Publications, 2nd edn. (2016)
  * [7] Caro, M., Tabani, H., Abella, J., Moll, F., Morancho, E., Canal, R., Altet, J., Calomarde, A., Cazorla, F.J., Rubio, A., Fontova, P., Fornt, J.: An automotive case study on the limits of approximation for object detection. Journal of Systems Architecture 138, 102872 (2023), https://www.sciencedirect.com/science/article/pii/S1383762123000516
  * [8] CENELEC: EN 50128:2011 Railway applications - Communication, signalling and processing systems - Software for railway control and protection systems (2011)
  * [9] CENELEC: EN 50126-1 Railway Applications - The Specification and Demonstration of Reliability, Availability, Maintainability and Safety (RAMS) \- Part 1: Generic RAMS Process (2017)
  * [10] CENELEC: EN 50126-2 Railway Applications - The Specification and Demonstration of Reliability, Availability, Maintainability and Safety (RAMS) \- Part 2: Systems Approach to Safety (2017)
  * [11] CENELEC: Railway applications - Communication, signalling and processing systems - Safety related electronic systems for signalling (2018)
  * [12] Cheng, C.H., Huang, C.H., Yasuoka, H.: Quantitative Projection Coverage for Testing ML-enabled Autonomous Systems. In: Lahiri, S.K., Wang, C. (eds.) Automated Technology for Verification and Analysis. pp. 126–142. Lecture Notes in Computer Science, Springer International Publishing, Cham (2018)
  * [13] Flajolet, P., Gardy, D., Thimonier, L.: Birthday paradox, coupon collectors, caching algorithms and self-organizing search. Discrete Applied Mathematics 39(3), 207–229 (1992), https://www.sciencedirect.com/science/article/pii/0166218X9290177C
  * [14] Flammini, F., Donato, L.D., Fantechi, A., Vittorini, V.: A vision of intelligent train control. In: Dutilleul, S.C., Haxthausen, A.E., Lecomte, T. (eds.) Reliability, Safety, and Security of Railway Systems. Modelling, Analysis, Verification, and Certification - 4th International Conference, RSSRail 2022, Paris, France, June 1-2, 2022, Proceedings. Lecture Notes in Computer Science, vol. 13294, pp. 192–208. Springer (2022), https://doi.org/10.1007/978-3-031-05814-1_14
  * [15] Hauer, F., Schmidt, T., Holzmüller, B., Pretschner, A.: Did we test all scenarios for automated and autonomous driving systems? In: 2019 IEEE Intelligent Transportation Systems Conference, ITSC 2019, Auckland, New Zealand, October 27-30, 2019. pp. 2950–2955. IEEE (2019), https://doi.org/10.1109/ITSC.2019.8917326
  * [16] ISO: ISO/DIS 21448: Road vehicles — Safety of the intended functionality. European Committee for Electronic Standardization (2021), iCS: 43.040.10, Draft International Standard
  * [17] ISO/DIS 26262-4: Road vehicles – functional safety – part 4: Product development: system level. Tech. rep., International Organization for Standardization (2009)
  * [18] Kalra, N., Paddock, S.M.: Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability? RAND Corporation, Santa Monica, CA (2016)
  * [19] Koopman, P., Wagner, M.D.: Autonomous vehicle safety: An interdisciplinary challenge. IEEE Intell. Transp. Syst. Mag. 9(1), 90–96 (2017), https://doi.org/10.1109/MITS.2016.2583491
  * [20] Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: Verification of probabilistic real-time systems. In: Gopalakrishnan, G., Qadeer, S. (eds.) Computer Aided Verification (CAV), 23rd Int. Conf. LNCS, vol. 6806, pp. 585–591. Springer (2011)
  * [21] Peleska, J., Haxthausen, A.E., Lecomte, T.: Standardisation considerations for autonomous train control. In: Margaria, T., Steffen, B. (eds.) Leveraging Applications of Formal Methods, Verification and Validation. Practice - 11th International Symposium, ISoLA 2022, Rhodes, Greece, October 22-30, 2022, Proceedings, Part IV. Lecture Notes in Computer Science, vol. 13704, pp. 286–307. Springer (2022), https://doi.org/10.1007/978-3-031-19762-8_22
  * [22] Rangra, S., Sallak, M., Schön, W., Belmonte, F.: Risk and safety analysis of main line autonomous train operation: Context, challenges and solutions. In: Lambda Mu 21 “ Maîtrise des risques et transformation numérique : opportunités et menaces ”. Reims, France (Oct 2018), https://hal.archives-ouvertes.fr/hal-02073235
  * [23] Ristić-Durrant, D., Franke, M., Michels, K.: A Review of Vision-Based On-Board Obstacle Detection and Distance Estimation in Railways. Sensors (Basel, Switzerland) 21(10), 3452 (May 2021), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8156009/
  * [24] Shi, W., Alawieh, M.B., Li, X., Yu, H., Aréchiga, N., Tomatsu, N.: Efficient statistical validation of machine learning systems for autonomous driving. In: Liu, F. (ed.) Proceedings of the 35th International Conference on Computer-Aided Design, ICCAD 2016, Austin, TX, USA, November 7-10, 2016. p. 36. ACM (2016), https://doi.org/10.1145/2966986.2980077
  * [25] Sun, Y., Chockler, H., Huang, X., Kroening, D.: Explaining image classifiers using statistical fault localization. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J. (eds.) Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII. Lecture Notes in Computer Science, vol. 12373, pp. 391–406. Springer (2020), https://doi.org/10.1007/978-3-030-58604-1_24
  * [26] Sun, Y., Wu, M., Ruan, W., Huang, X., Kwiatkowska, M., Kroening, D.: Concolic testing for deep neural networks. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. pp. 109–119. ASE ’18, Association for Computing Machinery, New York, NY, USA (Sep 2018), https://doi.org/10.1145/3238147.3238172
  * [27] Underwriters Laboratories Inc.: ANSI/UL 4600-2020 Standard for Evaluation of Autonomous Products – First Edition. Underwriters Laboratories Inc., 333 Pfingsten Road, Northbrook, Illinois 60062-2096, 847.272.8800 (April 2020)