00footnotetext: AMS 2010 subject classifications: Primary 60F05,82B30,60G60.

# Normal approximation for fire incident simulation using permanental Cox
processes

Dawud Thongtha and Nathakhun Wiroonsri††footnotemark:
Mathematics and Statistics with Applications Research Group,
Department of Mathematics, King Mongkut’s University of Technology Thonburi
Research partially supported by the TSRI Fundamental Fund 2020 (Grant Number:
64A306000047).

###### Abstract

Estimating the number of natural disasters benefits the insurance industry in
terms of risk management. However, the estimation process is complicated due
to the fact that there are many factors affecting the number of such
incidents. In this work, we propose a Normal approximation technique for
associated point processes for estimating the number of natural disasters
under the following two assumptions: 1) the incident counts in any two
distinct areas are positively associated and 2) the association between these
counts in two distinct areas decays exponentially with respect to distance
outside some small local neighborhood. Under the stated assumptions, we extend
previous results for the Normal approximation technique for associated point
processes, i.e., the establishment of non-asymptotic $L^{1}$ bounds for the
functionals of these processes [28]. Then we apply this new result to
permanental Cox processes that are known to be positively associated. Finally,
we apply our Normal approximation results for permanental Cox processes to
Thailand’s fire data from 2007 to 2020, which was collected by the Geo-
Informatics and Space Technology Development Agency of Thailand.

Keywords: Correlation inequality, Cox process, Local dependence, Random
fields, Natural disaster, Positive association

JEL: C020 Mathematical Methods

##  1 Introduction

Probability and statistical models have been widely used in most business
sectors, including the insurance business. They are major tools for estimating
loss, claim severity, claim count and claim probabilities, as these factors
significantly affect their business operations. Since claim occurrences are
used in determining policy premiums and are usually not predictable, claim
simulation and prediction are some of the main tools for handling the
situation. Simulation and prediction techniques for the insurance industry
have attracted researchers’ attention for quite some time (see [14], [1] and
[12] and references therein).

The loss can be estimated as the product of the frequency of risk events and
some overall measure of severity. Therefore, approximating the number of
occurrences of risk events and claim probabilities is key to successful loss
estimation. These related topics have been studied and developed in various
directions. For instance, the works [17] and [16] improved statistical and
stochastic methods for estimating the probability and frequency of flood
events. Machine learning techniques have also been applied to estimate claims
and reserves (see [23], [29] and [5], for example).

A point process (sometimes called a random point field) is one of the most
powerful tools in probability. These processes, which have been widely
developed from their root, namely, the Poisson point process, have been
applied in several areas, including insurance (see[30], [27] and [2]).
Intuitively, a point process is an appropriate tool for modeling the
occurrences of risk events and insurance claims, as it explains random natural
phenomenon across space and time. Many researchers have proposed different
approaches based on point processes for modeling natural disasters, especially
fires. In 2007, in [25], it was shown that a simple point process outperforms
the Burning Index in predicting wildfire incidents in Los Angeles County. In
2011, the work [30] extended the model in [25] by considering relevant
covariates, such as historical spatial burn patterns and wind direction. Also,
in 2015, the work [27] used a spatial point process to study the risk
associated with insurance customers using geographical information systems.

Some point processes, such as the classic Poisson point process, have the
property that the intensities in the relevant areas are independent. However,
when considering natural disasters or insurance claims, a dependency structure
must be introduced. In 2009, the work [15] proposed a copula approach to
finding the joint probability distribution for hydrological variables and then
using it to study the spatial dependence in extreme river flows and
precipitation. When exact distributions are unknown, approximate distributions
can be used to estimate probabilities of risk events. For example, the work
[16] applied a theoretical result related to an approximation of a conditional
distribution in [13] to develop a new method for estimating the probability of
widespread flood events. In our work, we use a Normal approximation for
associated point processes to estimate the total number of fire incidents.
When approximating distributions, the major issue is dependency. Stein’s
method ([26]) is one of the best tools for handling dependency. This well-
known method for approximating limiting distributions has the additional
benefit of providing non-asymptotic bounds for these distributions . In this
work, we use the concept of local dependency, which can be handled by Stein’s
method, as shown in [7].

This work is divided into two parts. First, we extend the theoretical results
in [28] to provide non-asymptotic $L^{1}$ bounds for the functionals of
associated point processes under more general assumptions. These assumptions
are motivated by the nature of natural disasters, as discussed later in this
section. We also apply our main theoretical results to the permanental Cox
process detailed in Section 2, which is known to be positively associated. In
the second part, we simulate Thailand’s fire incidents using permanental Cox
processes based on the satellite data on Thailand ‘s fire occurrences
collected by the Geo-Informatics and Space Technology Development Agency
(GISTDA) from 2007 to 2020. Finally, we check our Normal approximation results
from the first part using our simulation results in the second part.

Stein’s method has been used to obtain bounds for point processes under
various assumptions and Poisson processes (see [3], [4], [8], [9] and [10] for
examples). However, association properties did not appear in the context of
Normal approximations for point processes until the work [22] proved a central
limit theorem for their functionals under certain conditions and the work [28]
obtained rates of convergence using Stein’s method under a stronger
assumption.

Next, we state our assumptions regarding natural disasters. Based on the
literature, natural disaster incidents in a particular area seem to affect
such incidents in another area. For instance, according to [30], wind may
cause fire to extend to nearby areas . Thus, we are inspired to claim that for
some natural disasters, such as fires, there is some positive relation between
the incident counts for two distinct areas. Moreover, we also claim that the
further apart the two areas are, the weaker the relation should be. Therefore,
we make the following two assumptions in our work:

  1. (A1)

The chances of incidents in any two areas are positively associated. Thus,
incident counts in any two distinct areas are positively associated.

  2. (A2)

The association between the incident counts in two distinct areas decays
exponentially with respect to the distance between the areas. However, the
decay is assumed to begin outside some small local neighborhood.

Though our assumptions are quite intuitive, to the best of our knowledge, we
are unaware of any other works that use a dependent structure with respect to
location to model natural disaster incidents.

The remainder of this work is organized as follows. We provide some necessary
background and definitions in Section 2. Then the theoretical results are
stated and proved in Section 3. Section 4 is devoted to the simulation of
fires in Thailand using the dataset from GISTDA and checking our normal
approximation results based on this simulation. Finally, a conclusion is
provided in Section 5.

##  2 Some background and definitions

We devote this section to a review of the theoretical background of point
processes, permanental Cox processes and Stein’s method.

###  2.1 Point and permanental Cox processes

A point process is a collection of random points on some mathematical space,
such as Euclidean space $\mathbb{R}^{d}$. As mentioned earlier, point
processes are popular tools in the insurance industry (see [2], [27] and
[30]).

Let $X\subset\mathbb{R}^{d}$ with $d\in\mathbb{N}$ be a point process. For
$\text{card}(A)$, the cardinality of $A\subset\mathbb{R}^{d}$, let

$\displaystyle N(A)=\text{card}(X\cap A).$

The point process $X$ is said to be simple if $N(\\{{\bf a}\\})\in\\{0,1\\}$
a.s. for all ${\bf a}\in\mathbb{R}^{d}$. Also, the process $X$ is said to be
locally finite if it takes values in

$\displaystyle\Omega=\left\\{x\subset\mathbb{R}^{d}:\text{card}(x\cap
B)<\infty,\forall\text{ bounded }B\subset\mathbb{R}^{d}\right\\}.$

The process $X$ is said to be negatively associated if for all coordinate-wise
increasing functions $\psi:\mathbb{N}^{k}\rightarrow\mathbb{R}$ and
$\phi:\mathbb{N}^{l}\rightarrow\mathbb{R}$ and for all families of pairwise
disjoint Borel sets $\\{A_{i}\mid 1\leq i\leq k\\}$ and $\\{B_{j}\mid 1\leq
j\leq l\\}$ such that

$\displaystyle(\displaystyle\cup_{i}A_{i})\cap(\displaystyle\cup_{j}B_{j})=\emptyset,$
(1)

we have

$\mathbb{E}\left[\psi\left(N(A_{1}),\ldots,N(A_{k})\right)\phi\left(N(B_{1}),\ldots,N(B_{l})\right)\right]\\\
\leq\mathbb{E}\left[\psi\left(N(A_{1}),\ldots,N(A_{k})\right)\right]\mathbb{E}\left[\phi\left(N(B_{1}),\ldots,N(B_{l})\right)\right].$

Similarly, the point process is said to be positively associated if the above
inequality is reversed and the families of Borel sets are not necessarily
assumed to satisfy (1). In addition, a point process is said to be associated
if it is either negatively or positively associated.

Next, we state the definition of the nth order intensity functions of point
processes with respect to Lebesgue measure. Let $n\in\mathbb{N}$ and
$X\in\Omega$. If there exists a non-negative function
$\rho_{n}:\left(\mathbb{R}^{d}\right)^{n}\rightarrow\mathbb{R}$ such that

$\displaystyle\mathbb{E}\left[\sum_{\begin{subarray}{c}{\bf x}_{1},\ldots,{\bf
x}_{n}\in X\\\ \text{all distinct}\end{subarray}}f({\bf x}_{1},\ldots,{\bf
x}_{n})\right]=\int_{\left(\mathbb{R}^{d}\right)^{n}}f({\bf x}_{1},\ldots,{\bf
x}_{n})\rho_{n}({\bf x}_{1},\ldots,{\bf x}_{n})d{\bf x}_{1}\ldots d{\bf
x}_{n}$

for all locally integrable functions
$f:\left(\mathbb{R}^{d}\right)^{n}\rightarrow\mathbb{R}$, then $\rho_{n}$ is
called the nth order intensity function of $X$. Now, for ${\bf x},{\bf
y}\in\mathbb{R}^{d}$, let

$\displaystyle D({\bf x},{\bf y})=\rho_{2}({\bf x},{\bf y})-\rho_{1}({\bf
x})\rho_{1}({\bf y}).$ (2)

It follows that

$\displaystyle\mathrm{Cov}(N(A),N(B))=\int_{A\times B}D({\bf x},{\bf y})d{\bf
x}d{\bf y}.$

Cox processes are well-known point processes. They are considered to be
generalizations of the Poisson point processes for which the intensity is a
random measure. A permanental Cox process [19] is the Cox process with
intensity functions

$\displaystyle\rho_{n}({\bf x}_{1},\ldots,{\bf
x}_{n})=\mathbb{E}\prod_{i=1}^{n}\Lambda({\bf x}_{i}),$ (3)

where $\Lambda$ is a random measure defined as

$\displaystyle\Lambda({\bf x}_{i})=Y_{1}^{2}({\bf
x}_{i})+\cdots+Y_{l}^{2}({\bf x}_{i}),$ (4)

where the $Y_{1},\ldots,Y_{l}$ are $l$ independent, zero-mean, real-valued
Gaussian random fields with covariance function $C$.

In this work, we consider the case where the Gaussian random fields are
stationary; hence, $C$ only depends on $r=|{\bf x}_{i}-{\bf x}_{j}|_{\infty}$,
where $|\cdot|_{\infty}$ denotes the vector max norm. Also, we consider just
the specific case when

$\displaystyle C(r)=\kappa_{c}e^{-\lambda_{c}r},$ (5)

where $\kappa_{c}$ and $\lambda_{c}$ are constants. This process has been
used, for instance, in [20], [31] and [21]. It is known to be positively
associated (see [11] and [18]).

###  2.2 Stein’s bound for local dependent random variables

Stein’s method, introduced by Charles Stein [26] in 1972, is a widely known
technique for finding non-asymptotic bounds for approximations of probability
distributions. It was motivated by the idea that $W$ has the standard Normal
distribution if and only if

$\displaystyle\mathbb{E}Wf(W)=\mathbb{E}f^{\prime}(W)$

for all absolutely continuous functions $f$ with
$\mathbb{E}|f^{\prime}(W)|<\infty$. This identity leads to the differential
equation

$\displaystyle h(w)-\mathbb{E}h(Z)=f^{\prime}_{h}(w)-wf_{h}(w),$ (6)

where $Z$ is a standard Normal random variable and $h$ is a test function. If
$h\in\mathcal{H}$ for some Borel set $\mathcal{H}$ and we replace $w$ by a
random variable $W$, the error in the distributional approximation of $W$ by
$Z$ on $\mathcal{H}$ can be bounded by obtaining the non-asymptotic bound of
the expectation of the supremum of the right-hand side of (6) over
$\mathcal{H}$. In general, doing so is much simpler than computing the left-
hand side of (6) directly. Taking $h\in\mathcal{H}_{1}$, where
$\mathcal{H}_{1}=\\{h\mid|h(x)-h(y)|\leq|x-y|\\}$, we obtain

$\displaystyle
d_{1}(\mathcal{L}(W),\mathcal{L}(Z))=\sup_{h\in\mathcal{H}_{1}}|\mathbb{E}h(W)-\mathbb{E}h(Z)|.$

This distance is known as $L^{1}$ or the Wasserstein distance. Stein’s method
has been used in various applications, and it is one of the best ways to
handle dependent situations (see [6] and [24]). One of the classic dependent
cases, handled by Stein’s method, is the local dependent structure introduced
in [7]. A collection of random variable $X_{1},\ldots,X_{n}$ has dependency
neighborhoods $N_{i}\subseteq\\{1,2,\ldots,n\\},i=1,2,\ldots,n$ if $X_{i}$ is
independent of $X_{j}$ for all $j\notin N_{i}-\\{i\\}$. Next, we state a
version of the local dependence bound that appeared in the note [24].

######  Theorem 2.1 ([24])

Let $X_{1},\ldots,X_{n}$ be random variables such that
$\mathbb{E}[X_{i}^{4}]<\infty$, $\mathbb{E}X_{i}=0$,
$\sigma^{2}=\mathrm{Var}(\sum_{i}X_{i})$, and define $W=\sum_{i}X_{i}/\sigma$.
Let the collection $(X_{1},\ldots,X_{n})$ have dependency neighborhoods
$N_{i}$, $i=1,\ldots,n$, and also define $D=\max_{1\leq i\leq n}|N_{i}|$. Then
for $Z$, a standard Normal random variable, we have

$\displaystyle d_{1}\big{(}{\cal L}(W),{\cal
L}(Z)\big{)}\leq\frac{D^{2}}{\sigma^{3}}\sum_{i=1}^{n}\mathbb{E}|X_{i}|^{3}+\frac{\sqrt{28}D^{3/2}}{\sqrt{\pi}\sigma^{2}}\sqrt{\sum_{i=1}^{n}\mathbb{E}[X_{i}^{4}]}.$

We apply Theorem 2.1 to obtain our main results in the next section.

##  3 Main results

In this section, we state and prove our main result, which is extended from
[28] and then apply it to permanental Cox Processes. In this work, we relax
the assumption in [28] that the covariance of the second-order intensity
decays exponentially and assume that it decays exponentially outside of some
local neighborhood. Let $X\in\Omega$ be an associated point process and

$\displaystyle Y_{\bf i}=f_{\bf i}(X\cap C_{\bf i})-\mathbb{E}f_{\bf i}(X\cap
C_{\bf i}),{\bf i}\in\mathbb{Z}^{d},$ (7)

where the $f_{\bf i}:\Omega\rightarrow\mathbb{R}$ are real-valued measurable
functions and the $C_{\bf i}$ are defined as $d$-dimensional unit cubes
centered at ${\bf i}$. Note that the union of $C_{\bf i}$ forms a covering of
$\mathbb{R}^{d}$.

We let ${\bf 1}\in\mathbb{Z}^{d}$ denote the vector with all components $1$,
and write inequalities such as ${\bf a}<{\bf b}$ for vectors ${\bf a},{\bf
b}\in\mathbb{R}^{d}$ when they hold component-wise. In this work, we consider

$\displaystyle S_{{\bf k}}^{n}=\sum_{{\bf i}\in B_{{\bf k}}^{n}}Y_{{\bf
i}}\quad\mbox{where}\quad B_{{\bf k}}^{n}=\left\\{{\bf
i}\in\mathbb{Z}^{d}:{\bf k}\leq{\bf i}<{\bf k}+n{\bf 1}\right\\}.$ (8)

The work [22] obtained a CLT for the sum above with $B_{{\bf k}}^{n}$ replaced
by any sequence of strictly increasing finite domains of $\mathbb{Z}^{d}$. Let
$C_{\bf i}$ be any $d$-dimensional cube centered at $x_{\bf i}=R\cdot{\bf i}$
with fixed $R>0$ and with fixed side length $s\geq R$. We state our main
result for $R=s=1$. We then add a remark after the theorem to discuss how to
generalize this result. In the following, for $k\in\mathbb{N}$, we denote
$\left\|X\right\|_{k}=\left\\{\mathbb{E}|X|^{k}\right\\}^{\frac{1}{k}}$.

###### Theorem 3.1

For $d\in\mathbb{N}$, let $X$ be a locally finite simple associated point
process on $\mathbb{R}^{d}$. For ${\bf k}\in\mathbb{Z}^{d}$, let $S_{{\bf
k}}^{n}$ be as in (8), with $Y_{\bf i}$ given in (7), $R=s=1$ and
$\sigma_{n,{\bf k}}^{2}=\mathrm{Var}(S_{{\bf k}}^{n})$. Assume that the
following conditions are satisfied:

  1. (a)

The first two intensity functions of $X$ are well defined;

  2. (b)

$\sup_{{\bf i}\in\mathbb{Z}^{d}}\left\|Y_{\bf i}\right\|_{4}=M<\infty$;

  3. (c)

$\sup_{|{\bf x}-{\bf y}|_{\infty}\geq r}D({\bf x},{\bf y})\leq\kappa
e^{-\lambda r}$ for some $\kappa,\lambda>0$ and
$r>Kn^{\frac{2}{3}\left(\frac{4d-1-1/d}{4d+2}\right)}$ with $K>0$;

  4. (d)

$\sigma_{n,{\bf k}}^{2}\geq\gamma n^{d}$ for some $\gamma>0$.

Then, for the standard Normal random variable $Z$,

$\displaystyle d_{1}\left(\mathcal{L}\left(\frac{S_{{\bf
k}}^{n}}{\sigma_{n,{\bf k}}}\right),\mathcal{L}(Z)\right)$ $\displaystyle\leq$
$\displaystyle\left(\frac{\sqrt{28}M^{2}(2K)^{3d/2}}{\gamma\sqrt{\pi}}+C_{1,d,M,\kappa,\gamma}\right)\frac{1}{n^{d/(4d+2)}}+\frac{M^{3}(2K)^{2d}}{\gamma^{3/2}n^{d/6-1/(6d+3)}}$
$\displaystyle\hskip
5.0pt+\frac{C_{2,d,M,\kappa,\gamma}n^{d(4d+1)/(6d+3)}}{\exp\left(\theta_{d,M,\kappa,\gamma}n^{d/(4d+2)}\right)}+\frac{C_{3,d,M,\kappa,\gamma}n^{7d/6}}{\exp\left(2\theta_{d,M,\kappa,\gamma}n^{d/(4d+2)}\right)},$

where $d_{1}$ is the $L^{1}$ distance,

$\displaystyle\theta_{d,M,\kappa,\gamma}=\frac{\lambda}{3}\left(\frac{\sqrt{2\gamma}\kappa^{1/3}\left((4\mu_{\lambda}+2\nu_{\lambda})^{d}-\left(2\nu_{\lambda}\right)^{d}\right)}{18^{d+1}\sqrt{\pi}dM}\right)^{1/(2d+1)},$
$\displaystyle C_{1,d,M,\kappa,\gamma}=\left(\frac{9\cdot
36^{d}M^{4d+3}\left((4\mu_{\lambda}+2\nu_{\lambda})^{d}-\left(2\nu_{\lambda}\right)^{d}\right)^{2d}}{\gamma^{2d+(3/2)}\pi^{d}}\right)^{1/(2d+1)}$
$\displaystyle\hskip
100.0pt\times\left(\frac{1}{(2d)^{2d/(2d+1)}}+2(2d)^{1/(2d+1)}\right),$
$\displaystyle C_{2,d,M,\kappa,\gamma}=\frac{3\cdot
6^{d}\kappa^{1/3}M^{2}\theta_{d,M,\kappa,\gamma}^{4d/3}}{\sqrt{\pi}\gamma},\ \
\ C_{3,d,M,\kappa,\gamma}=\frac{2^{d+1}\kappa^{2/3}M}{\sqrt{\gamma}},$

and

$\displaystyle\mu_{\lambda}=\frac{e^{\frac{2\lambda}{r}}}{\left(e^{\frac{\lambda}{r}}-1\right)^{2}},\
\ \
\nu_{\lambda}=\frac{e^{\lambda/r}}{\left(e^{\frac{\lambda}{r}}-1\right)^{2}}.$

Proof: We prove this theorem by applying Theorem 2.1 stated above and Theorem
3.1 in [28], handling local dependence and non-local dependence separately.
First, we address local dependence only by assuming that $\mathrm{Cov}(Y_{\bf
i},Y_{\bf j})=0$ for $|{\bf i}-{\bf
j}|_{\infty}>Kn^{\frac{2}{3}\left(\frac{4d-1-1/d}{4d+2}\right)}$. Invoking
Theorem 2.1 with
$D=(2K)^{d}n^{\frac{2}{3}\left(\frac{4d^{2}-d-1}{4d+2}\right)}$, Assumption
(b) that $\mathbb{E}|Y_{\bf j}|^{3}\leq M^{3}$ and $\mathbb{E}[Y_{\bf
j}^{4}]\leq M^{4}$ and using Assumption (d), we have that the $L^{1}$ distance
is bounded by

$\displaystyle\frac{\sqrt{28}M^{2}(2K)^{3d/2}}{\gamma\sqrt{\pi}n^{d/(4d+2)}}+\frac{M^{3}(2K)^{2d}}{\gamma^{3/2}n^{d/6-1/(6d+3)}}.$
(9)

We now assume that Assumption (c) is true for all $r>0$. Invoking Theorem 3.1
in [28], we have that the $L^{1}$ distance is bounded by

$\displaystyle\frac{C_{1,d,M,\kappa,\gamma}}{n^{d/(4d+2)}}+\frac{C_{2,d,M,\kappa,\gamma}n^{d(4d+1)/(6d+3)}}{\exp\left(\theta_{d,M,\kappa,\gamma}n^{d/(4d+2)}\right)}+\frac{C_{3,d,M,\kappa,\gamma}n^{7d/6}}{\exp\left(2\theta_{d,M,\kappa,\gamma}n^{d/(4d+2)}\right)}$
(10)

The distance $d_{1}\big{(}{\cal L}(W_{{\bf k}}^{n}),{\cal L}(Z)\big{)}$ is
bounded by the sum of (9) and (10) because the local neighborhood covariance
terms and the remaining covariance terms have been handled by (9) and (10),
respectively. $\Box$

###### Remark 3.2

  1. 1.

The associated assumption and Assumption (c) are motivated by (A1) and (A2) in
the introduction, respectively.

  2. 2.

The size of the local neighborhood is $O\left(n^{d/(4d+2)}\right)$, which is
flexible and can be increasing in $n$.

  3. 3.

The above theorem can be extended to the case where $R$ and $S$ are greater
than one by following the same proof. The bound will end up with a larger
constant.

Note that we added the assumption that (c) is true for all $r>0$ in the above
proof after the local covariance terms had been handled. Although doing so may
make the constant larger, the rate of convergence is unaffected.

Next, we apply the above theorem to the permanental Cox process on
$\mathbb{R}^{d}$, which is known to be positively associated. In this theorem,
we consider a function $f:\Omega\rightarrow\mathbb{R}$ defined by

$\displaystyle f(Y)=\sum_{S\subset Y}g(S)\mathbf{1}_{|S|=p},$ (11)

where $g$ is a bounded function supported on sets $S$ having exactly $p$
elements such that $g(S)=0$ when $diam(S)>\tau$ for some fixed $\tau>0$ and
$p\in\mathbb{N}$. Here, we denote $diam(S)=\sup_{x,y\in
S}\left|x-y\right|_{\infty}$. Also, we focus specifically on the case that
$p=1$. We note here that if $p=1$ and $g(S)=1$, $f(X\cap\Lambda_{n})$ is the
number of points in $X\cap\Lambda_{n}$ or $N(\Lambda_{n})$, where
$\Lambda_{n}\subset\mathbb{R}^{d}$. The result is stated as follows:

###### Theorem 3.3

Let $n,k\in\mathbb{N}$ and $X$ be a permanental Cox process with intensity
functions as in (3) and $\Lambda({\bf x}_{i})$ as in (4), where the $Y_{i}$,
$i=1,\cdots,k$ are independent, mean-zero, real-valued Gaussian random fields.
Let $f$ be defined as in (11) with $g$ be bounded and $g(S)=0$ when
$diam(S)>\tau$ for some fixed $\tau>0$ and $p=1$. Letting
$\sigma_{n}^{2}=\mathrm{Var}(f(X\cap\Lambda_{n}))$ with
$\Lambda_{n}\subset\mathbb{R}^{d}$ be such that $|\Lambda_{n}|=n^{d}$, assume
that

$\displaystyle\sup_{|{\bf x}-{\bf y}|_{\infty}\geq
r}\mathrm{Cov}\left(\Lambda({\bf x}),\Lambda({\bf y})\right)\leq O(e^{-\lambda
r}),$ (12)

for some $\lambda>0$ and $r>Kn^{d/(4d+2)}$ with $K>0$, and

$\displaystyle\liminf_{n}\sigma_{n}^{2}/n^{d}>0.$ (13)

Then for
$W_{n}=(f(X\cap\Lambda_{n})-\mathbb{E}f(X\cap\Lambda_{n}))/\sigma_{n}$, there
exists $C>0$ such that

$\displaystyle
d_{1}\left(\mathcal{L}(W_{n}),\mathcal{L}(Z)\right)\leq\frac{C}{n^{d/(4d+2)}},$
(14)

where $Z$ is a standard Normal random variable.

Proof: To prove the theorem, we follow the same argument as that used in the
proof of Theorem 4.3 in [28]. It requires the use of Theorem 3.1, where the
assumptions (12) and (13) are needed. Because Theorem 4.3 of [28] is for
determinantal point processes, the variance condition used in its proof cannot
be used here; thus, the proofs differ at this point. Therefore, it is
sufficient to show that

$\displaystyle\mathrm{Var}\left(\sum_{S\subset
X\cap\Lambda_{n}}g(S)\mathbf{1}_{|S|=p}\right)=O(\Lambda_{n}),$

which can be proved by adapting the proof of Lemma B.6 in [22] for the case
where $X$ is a permanental Cox process. It is sufficient to verify that the
term (B.4) in that proof is bounded for $k=0,\ldots,p$. For the term $k=0$, we
show that the intensity $\rho_{2p}$ is bounded as follows:

$\displaystyle|\rho_{2p}({\bf x}_{1},\cdots,{\bf x}_{2p})|$ $\displaystyle=$
$\displaystyle\mathbb{E}\prod_{j=1}^{2p}\Lambda({\bf
x}_{j})=\mathbb{E}\prod_{j=1}^{2p}\left(Y^{2}_{1}({\bf
x}_{j})+\cdots+Y^{2}_{l}({\bf x}_{j})\right)$ $\displaystyle=$
$\displaystyle\sum_{1\leq i_{1},\ldots,i_{2p}\leq
l}\mathbb{E}\left[Y^{2}_{i_{1}}({\bf x}_{1})Y^{2}_{i_{2}}({\bf x}_{2})\cdots
Y^{2}_{i_{2p}}({\bf x}_{2p})\right]$ $\displaystyle\leq$
$\displaystyle\sum_{1\leq i_{1},\ldots,i_{2p}\leq
l}\left(\mathbb{E}\left[Y^{2p}_{i_{1}}({\bf
x}_{1})\right]\mathbb{E}\left[Y^{2p}_{i_{2}}({\bf
x}_{2})\right]\cdots\mathbb{E}\left[Y^{2p}_{i_{2p}}({\bf
x}_{2p})\right]\right)^{1/p}$ $\displaystyle\leq$ $\displaystyle
l^{2p}\left(M(2p-1)!!\right)^{2},$

where $M=\max_{1\leq i\leq l,{\bf x}\in\mathbb{R}^{d}}\sigma^{2}_{i}({\bf
x})$, with $\sigma^{2}_{i}({\bf x})=\mathrm{Var}(Y_{i}({\bf x}))$. Note that
we have used Hölder’s inequality and the fact that $Y_{i}$ is Normal and its
central moment is

$\mathbb{E}|Y_{i}({\bf x})|^{2p}=\sigma^{2p}_{i}({\bf x})(2p-1)!!$.

As we only consider the case $p=1$, the term $k=1$ is obvious. Thus, the order
of the variance is verified. $\Box$

###### Remark 3.4

We remark here that in the proof of Theorem 3.3, we set $M=\max_{1\leq i\leq
l,{\bf x}\in\mathbb{R}^{d}}\sigma_{i}^{2}({\bf x})$, which could be large.
However, in our application, we later set the $\sigma_{i}^{2}({\bf x})$ to be
equal for all $i$ and $\sigma_{i}^{2}({\bf x})=m({\bf x})/l$, where $m({\bf
x})$ is the maximum number of fires in area ${\bf x}$, as derived from the
historical data, which is assumed to be bounded.

##  4 Application to the Thai fire dataset

In this section, we use our main results to simulate Thailand’s fires via
permanental Cox processes. We use the GISTDA Thai fire dataset, collected from
2007 to 2020 by satellite. The dataset consists of the latitudes and
longitudes of all fire incident locations. Recall that we claim that the fire
counts from two distinct areas should be positively correlated. Moreover, we
claim that their covariance begins to decay exponentially outside some small
local neighborhood. We split this section into three subsections. We first
explore the dataset and check to see that Assumptions (A1) and (A2) are not
contradicted by the data in the first subsection. The second subsection is
devoted to applying our main results to fire simulations. In the last
subsection, we evaluate the effect of varying the decay parameter for the
permanental Cox processes.

###  4.1 Exploring the dataset

In this subsection, we first explore the dataset and verify that it does not
contradict the assumptions in Theorem 3.3. We follow the process outlined in
Figure 1. Figure 2 shows the total number of fires each year in Thailand,
where the mean and the standard deviation were 27,048 and 6,980.94,
respectively. Figure 3 shows the locations of fires in 2007, 2014 and 2020,
respectively, from left to right.

Fires data 2007 - 2020Verifying the assumptions (Theorem 3.3)Descriptive
statisticsAssumption (12)Assumption (13) The variance orderLocal
neighborhoodExponential decay

Figure 1: Exploring the GISTDA fire dataset Figure 2: The total number of
yearly fires for the period 2007-2020 Figure 3: The actual fire locations in
2007, 2014 and 2020

We now examine whether the fire dataset follows assumptions (12) and (13). As
the actual intensity function is unknown and cannot be obtained from the
dataset, we can only check the covariances between fire counts in two distinct
areas separated by a distance $r$. We first check to see if the covariance
does not decay in a small local neighborhood. Then we further check to see if
the covariance decays exponentially as the distance between areas increases.
We also show the variances of the fire counts corresponding to $n$, with $n$
going from 10 to 645, where $645$ is the greatest distance possible in
Thailand (1 unit equals to 0.01 degrees of latitude or longitude). Note that
the total area of Thailand is 513,120 square kilometers, which equals
$716^{2}$. Since 1 degree is about 111 kilometers, the total area is
approximately $6.45^{2}$ square degrees, which is equivalent to $645^{2}$
square units when 1 unit is 0.01 degrees. Therefore, the largest $n$ possible
in Thailand is 645.

As the entire country is too large to be considered a small local
neighborhood, we specifically check four provinces from four regions,
including Bangkok, Chaing Mai, Kanchanaburi and Khon Kaen. Note that we did
not select a province from Southern Thailand, as the number of fires in this
region was too low. Figure 4 shows no sign of exponential decay in the
covariance pattern for these provinces when the unit distance is 0.002 degrees
($\sim 222$ meters). Figure 5 shows the covariance decay pattern based on
distance for all of Thailand. We obtain an exponential decay rate of 0.15 when
we use a unit distance of 0.01 degrees ($\sim 1.11$ kilometers). Figure 6
shows the values of $\sigma_{n}^{2}/n^{2}$ for $n$ from 10 to 645. Obviously,
assumption 13 is not contradicted.

Figure 4: Covariance decay patterns for Bangkok, Chaing Mai, Kanchanaburi and
Khon Kaen with a unit distance of 0.002 degrees Figure 5: Covariance decay
pattern for Thailand with a unit distance of 0.01 degrees Figure 6: Variance
of the total number of fires divided by $n^{2}$

###  4.2 Simulation of Thai fires using the permanental Cox process

We devote this subsection to simulating fires in Thailand via the process
outlined in Figure 7.

Fires data 2007 - 2020Estimating permanental Cox process parameters
$\sigma^{2},l$ in (4)Estimating covariance decaying parameters $\lambda_{c}$
and $K_{c}$ in (5)Simulating Fires corresponding to the estimated
parametersTesting normality of the simulated firesVerifying assumptions (12)
and (13) for the simulated firesComputing the $L_{1}$ distance between the
number of simulated fires and a standard normalComparing the rate of
convergence to the bound in (14)

Figure 7: Simulation of Thailand’s fires

We first model Thailand’s fires using permanental Cox processes. Note that
when we refer to the area ${\bf x}$, we are referring to a unit square that
has ${\bf x}$ at the top left corner, where ${\bf x}\subset\mathbb{R}^{2}$ and
1 unit equals 0.01 degrees of latitude or longitude. We estimate the variances
of $Y_{j}({\bf x}_{i})$ and $l$ from

$\displaystyle\Lambda({\bf x}_{i})=Y_{1}^{2}({\bf
x}_{i})+\cdots+Y_{l}^{2}({\bf x}_{i})$

by using the first two sample moments. We set the means of the data and
process to be equal and seek the closest variances. Writing
$\sigma_{j}^{2}({\bf x}_{i})=\mathrm{Var}(Y_{j}({\bf x}_{i}))$ for each ${\bf
x}_{i}$ and using the fact that the $Y_{j}({\bf x}_{i})$ for $j=1,2,\ldots,l$
are zero-mean, independent Gaussian random variables, we set
$\sum_{j=1}^{l}\sigma_{j}^{2}({\bf x}_{i})$ equal to the sample mean of area
${\bf x}_{i}$ and $2\sum_{j=1}^{l}\sigma_{j}^{4}({\bf x}_{i})$ equal to the
sample variance of area ${\bf x}_{i}$. We then set $\sigma_{j}^{2}({\bf
x}_{i})=\sigma^{2}({\bf x}_{i})$ for all $j$ for simplicity and solve for the
closest $\sigma^{2}({\bf x}_{i})$ and $l$.

Denoting $m({\bf x}_{i})$ and $v({\bf x}_{i})$ as the mean and variance of the
total fires in the unit cube at position ${\bf x}_{i}$ from the dataset, we
have

$\displaystyle\sigma^{2}({\bf x}_{i})=\frac{m({\bf x}_{i})}{l}\text{ \ for all
\ }i.$

Then setting

$\displaystyle 2l\sigma^{4}({\bf
x}_{i})=\sum_{j=1}^{l}\mathrm{Var}(Y^{2}_{j}({\bf x}_{i}))=v({\bf x}_{i}),$

and substituting $\sigma^{2}({\bf x}_{i})$, we have

$\displaystyle l=\frac{2m^{2}({\bf x}_{i})}{v({\bf x}_{i})}.$

For simplicity, we set $l=\max\left(1,\left\lfloor\frac{2m^{2}({\bf
x}_{i})}{v({\bf x}_{i})}\right\rfloor\right)$. Although the $l$’s differ for
different ${\bf x}_{i}$, we can choose the largest $l$. For a ${\bf x}_{i}$
with a smaller $l$, we set $\sigma^{2}({\bf x}_{i})$ equal to zero for all the
remaining terms.

To estimate the covariance function of the Gaussian processes from (5), we
need estimates of $\kappa_{c}$ and $\lambda_{c}$. Plugging in $r=0$, we have
$\kappa_{c}=\sigma^{2}({\bf x}_{i})$; thus, it suffices to estimate
$\lambda_{c}$. We seek a $\lambda_{c}$ from a fine grid from 0 to 0.5 that
results in the covariance decay rate closest to the 0.15 from the real
dataset. Note that the decay rate of 0.15 is computed from the number of
fires, whereas $\lambda_{c}$ is the decay rate of the covariance function for
the Gaussian process. Therefore, it is not possible to set $\lambda_{c}=0.15$
directly.

By following the procedure above, we find the largest $l$, i.e., $l=8$, and
closest $\lambda_{c}(0.1$). Figure 8 shows an example of a simulation with
parameters estimated from the procedure above and 6 iterations.

Figure 8: This figure shows fire count simulation results using the estimated
parameters and six iterations. Here, the simulation’s mean and standard
deviation are 26,873 and 1078.15, respectively.

Next, we simulate fire counts for the whole country for 20 iterations and
check that the covariance decays exponentially with distance. Since the number
of areas in the entire country at a unit distance of 0.01 degrees from another
area is extremely large, 20 is a reasonable number of iterations for the
simulation. Figure 9 shows the covariance plot and the decay rate of 0.15 from
the real dataset.

Figure 9: The black dots show the covariances between the simulated fire
counts for two distinct unit areas according to the distance between the
areas. The red line shows exponential decay at a rate of 0.15.

Then we simulate fires across the whole country for 500 iterations to check
that the variance is on the order of $n^{2}$ and to compare the results with
the Normal approximation from Theorem 3.3. As in Subsection 4.1, we take $n$
from 10 to 645, where $645$ units is the greatest distance possible in
Thailand (1 unit equals 0.01 degrees of latitude or longitude). We
preliminarily check the normality of the simulated fire counts using the
Shapiro–Wilk test. Figure 10 shows the Shapiro–Wilk p-values based on $n$,
while the red horizontal line indicates the 0.05 significance level. We can
see that when $n$ is large, the null hypothesis that the total number of fires
is Normally distributed is not rejected.

Figure 10: The Shapiro–Wilk p-values for the fire counts

We end this section by checking the variance assumption in assumption (13) and
compare the simulation’s $L^{1}$ distance to the bound in (14). Figure 11
shows a random pattern for the variances of fire counts divided by $n^{2}$,
which agrees with assumption (13). Figure 12 shows the $L^{1}$ distances
between the standardized fire counts from our simulation and a standard Normal
distribution . We also plot the rate of convergence of $n^{d/(4d+2)}$ from
Theorem 3.3. The rate from our simulation tends to follow the rate from our
main bound.

Figure 11: The variances of fire counts divided by $n^{2}$ Figure 12: The
$L^{1}$ distances between the standardized fire counts and a standard Normal.
distrbution

###  4.3 The effect of changing the covariance decay rate

The main assumptions that we made in this work is that the fire counts for two
distinct areas are positively associated and their covariance decays
exponentially with respect to their distance from each other. Nevertheless,
the decay rate may vary from country to country or area to area and may also
change over time. Therefore, in this section, we are interested in examining
the Normal approximation results for different covariance decay rates through
the parameter $\lambda_{c}$ from (5).

Figure 13 shows that the $L^{1}$ distances between the standardized fire
counts from our simulation and a standard Normal distribution for areas
ranging in size from 9 to 645 square degrees when
$\lambda_{c}=1,0.2,0.1,0.05,0.017$. The black line shows the rate of
convergence of $n^{d/(4d+2)}$ from Theorem 3.3. Intuitively, the distribution
should be closer to the Normal distribution when $\lambda_{c}$ is larger, as a
larger value implies that the fire counts in the areas are closer to
independence. This reasoning agrees with the simulation results shown in
Figure 13.

Figure 13: Normal approximations for different values of $\lambda_{c}$

##  5 Discussion and conclusion

In this work, we break our contribution into two parts: developing theories
and simulating Thailand fire counts. For the first part, we generalize the
normal approximation of a functional of the associated point processes in [28]
by relaxing the assumption that the covariance decays exponentially everywhere
to exempting local neighborhoods. Then we apply the main result (Theorem 3.1)
to permanental Cox processes, which are known to be positively associated.
Unlike the main theorem involving hypercubes of size $n^{d}$ in
$\mathbb{R}^{d}$, Theorem 3.3 is flexible in that it covers Normally
approximating a functional of permanental Cox processes in an area of any
shape of size $n^{d}$, which allows us to apply this result to the fire counts
in Thailand.

For the simulation, we use a permanental Cox process to simulate fire counts
in Thailand. We assume that the fire counts for two distinct areas are
positively correlated. Moreover, we assume that their covariance decays
exponentially outside of some small local neighborhood. We use the GISTDA
Thailand fire dataset collected by Thailand’s satellites from 2007 to 2020 to
estimate the parameters for the process. The $L^{1}$ distance between the
total number of fires from our simulated fire counts and the Normal
distribution agrees with the rate from the bound in Theorem 3.3. By varying
the covariance parameter in (5), we can vary the $L^{1}$ distances from our
simulated results. Nevertheless, Normality still holds, and the rates do not
contradict the one in our main theorem.

In real-world applications, the new approach proposed in this research may be
used to model any natural disaster incidents or model claims by area for
property insurance firms. Thus, it benefits policymakers both in government
and the private sector in terms of managing risk. Moreover, future researchers
can use our idea to simulate natural disaster incidents or insurance claims
with some other point processes that are not necessarily well-known but
satisfy the assumptions of Theorem 3.1.

## Acknowledgment

The authors would like to thank the Geo-Informatics and Space Technology
Development Agency (GISTDA) for use of the fire dataset.

## Declarations

Funding: This work was supported financially by the TSRI Fundamental Fund 2020
(Grant Number: 64A306000047).

Conflicts of interest/Competing interests: No potential conflicts of interest
were reported by the authors.

Availability of data and material: This dataset is under the license of the
Geo-Informatics and Space Technology Development Agency.

Code availability: Available upon request.

IRB approval: Not applicable.

## References

  * [1] Abledu, G. K., Dadey, E. and Kobina , A., Probability modeling and simulation of insurance claims in Ghana, Global Journal of Commerce & Management Perspective, 3(5) (2014) 41–49.
  * [2] Albrecher, H., Araujo-Acuna, J. C. and Beirlant, J., Fitting non-stationary Cox process: an application to fire insurance data, North American Actuarial Journal, 0(0) (2020) 1–28.
  * [3] Barbour, A. D., Stein’s method and Poisson process convergence, Journal of Applied Probability, 25 (1988) 175–184.
  * [4] Barbour, A. D. and Brown, T. C., Stein’s method and point process approximation, Stochastic Processes and their Applications, 43(1) (1992) 9–31.
  * [5] Bärtl, M. and Krummaker, S., Prediction of claims in export credit finance: a comparison of four machine learning techniques, Risks, 8(22) (2020) 1–29.
  * [6] Chen, L.Y.H., Goldstein, L. and Shao, Q.M., Normal Approximation by Stein’s Method, Springer, New York, 2011.
  * [7] Chen, L.Y.H. and Shao, Q.M., Normal approximation under local dependence, Annals of Probability, 32 (2004) 1985–2028.
  * [8] Chen, L.Y.H. and Xia, A., Stein’s method, Palm theory and Poisson process approximation, Annals of Probability, 32(3) (2004) 2545–2569.
  * [9] Chen, L.Y.H. and Xia, A., Poisson process approximation: from Palm theory to Stein’s method, IMS Lecture Notes-Monograph Series: Time Series and Related Topics, 52 (2006) 236–244.
  * [10] Chen, L.Y.H. and Xia, A., Poisson process approximation for dependent superposition of point processes, Bernoulli, 17(2) (2011) 530–544.
  * [11] Eisenbaum, N., Characterization of positively correlated squared Gaussian processes, The Annals of Probability, 42(2) (2014) 559–575.
  * [12] Gabrielli, A. and Wüthrich, M. V., An individual claims history simulation machine, Risks, 6(29) (2018) 1–33.
  * [13] Heffernan, J.E. and Resnick, S.I., Limit laws for random vectors with an extreme component, Annals of Applied Probability, 17 (2007) 537–571.
  * [14] Jessen A.H., Mikosch T. and Samorodnitsky G., Prediction of outstanding payments in a Poisson cluster model, Scandinavian Actuarial Journal, 3(2011) 214–237.
  * [15] Keef, C., Svensson C. and Tawn, J.A., Spatial dependence in extreme river flows and precipitation for Great Britain, Journal of Hydrology, 378 (2009) 240–252.
  * [16] Keef, C., Tawn, J. A. and Lamb, R., Estimating the probability of widespread flood events, Environmetrics, 24 (2013) 13–21.
  * [17] Lamb, R., Rainfall-runoff modelling for flood frequency estimation, Encyclopedia of Hydrological Sciences, Anderson MG (eds). John Wiley & Sons(2005) 1923–1954.
  * [18] Last, G., Szekli, R., Yogeshwaran, D., Some remarks on associated random fields, random measures and point processes, ALEA Latin American Journal of Probability and Mathematical Statistics, 17 (2020) 355–374.
  * [19] McCullagh, P., Møller, J., The permanental process, Advances in Applied Probability, 38(4) (2006) 873–888.
  * [20] McCullagh, P., Yang, J., Stochastic classification models, International Congress of Mathematicians, 3 (2006) 669–686.
  * [21] Mustafa, H. A., Ekti, A. R., Shakir, M. Z., Imran, M. A., Tafazolli, R., Intracell interference characterization and cluster interference for D2D communication, IEEE Transactions on Vehicular Technology, 67(9) (2018) 8536–8548.
  * [22] Poinas, A., Delyon, B. and Lavancier, F., Mixing properties and central limit theorem for associated point processes, Bernoulli, 25(3) (2019) 1724–1754.
  * [23] Quan, Z. and Valdez, E.A., Predictive analytics of insurance claims using multivariate decision trees, Dependence Modeling, 6 (2018) 377–407.
  * [24] Ross, N. Fundamentals of Stein’s method. Probability Surveys, 8 (2011) 210–293.
  * [25] Schoenberg, F.P., Chang, C., Keeley, J., Pompa, J., Woods, J. and Xu, H., A critical assessment of the burning index in Los Angeles County, California, International Journal of Wildland Fire, 16(4) (2007) 473–483.
  * [26] Stein, C., A bound for the error in the normal approximation to the distribution of a sum of dependent random variables. Proceedings of the Sixth Berkeley Symposium on Mathematical Statistics and Probability University of California Press, 2 (1972) 210–293.
  * [27] Törnqvist, G., Modelling insurance claims with spatial point process: an applied case-control study to improve the use of geographical information in insurance pricing, Master thesis, Umeå University, 2015
  * [28] Wiroonsri, N., Normal approximation for associated point processes via Stein’s method with applications to determinantal point processes, Journal of Mathematical Analysis and Applications, 480(1) (2019) 123396.
  * [29] Wüthrich M. V., Machine learning in individual claims reserving, Scandinavian Actuarial Journal, (2018) 1–16.
  * [30] Xu, H. and Schoenberg F.P., Point process modeling of wildfire hazard in Los Angeles County, California, The Annals of Applied Statistics, 5(2011) 684–704.
  * [31] Yang, J., Miescke, K., McCullagh, P., Classification based on a permanental process with cyclic approximation, Biometrika, 99(4) (2012) 775–786.


(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext:  Zenseact, Gothenburg, Sweden 22institutetext: Linköping
Univeristy, Linköping, Sweden 33institutetext: Lund University, Lund, Sweden
44institutetext: Delft University of Technology, Delft, Netherlands

# NeuroNCAP: Photorealistic Closed-loop Safety Testing for Autonomous Driving

William Ljungbergh **1122 Adam Tonderski **1133 Joakim Johnander 11 Holger
Caesar 44 Kalle Åström 33 Michael Felsberg 22 Christoffer Petersson 11

###### Abstract

We present a versatile NeRF-based simulator for testing autonomous driving
(AD) software systems, designed with a focus on sensor-realistic closed-loop
evaluation and the creation of safety-critical scenarios. The simulator learns
from sequences of real-world driving sensor data and enables reconfigurations
and renderings of new, unseen scenarios. In this work, we use our simulator to
test the responses of AD models to safety-critical scenarios inspired by the
European New Car Assessment Programme (Euro NCAP). Our evaluation reveals
that, while state-of-the-art end-to-end planners excel in nominal driving
scenarios in an open-loop setting, they exhibit critical flaws when navigating
our safety-critical scenarios in a closed-loop setting. This highlights the
need for advancements in the safety and real-world usability of end-to-end
planners. By publicly releasing our simulator and scenarios as an easy-to-run
evaluation suite, we invite the research community to explore, refine, and
validate their AD models in controlled, yet highly configurable and
challenging sensor-realistic environments.

###### Keywords:

Autonomous driving Closed-loop simulation Trajectory planning Neural rendering

11footnotetext: Denotes equal contribution.

https://github.com/atonderski/neuro-ncap

Figure 1: The core idea in NeuroNCAP is to leverage NeRFs to realistically
simulate many safety-critical scenarios from a sequence of real-world data.
Here we show the original scenario, followed by examples of our three types of
collision scenarios: stationary, frontal, and side. The inserted safety-
critical actor has been highlighted for illustration purposes. We can generate
hundreds of unique scenarios from each log by selecting different actors,
jittering their trajectories, and choosing different starting conditions for
the ego vehicle. Note that scenarios are not pre-generated, but rather
obtained by iteratively generating new images, computing a plan, and acting
upon said plan.

##  1 Introduction

Recent work on autonomous driving (AD) [22, 23] suggests designing and
training a holistic neural network for mapping sensor inputs directly to a
planned trajectory. Compared to prior work that used modular software stacks,
engineered interfaces between modules, or handcrafted rules, this end-to-end
approach has several advantages. First, as the driving behavior is learned,
the predicted trajectories are expected to resemble how a typical human driver
would act. Second, the approach is scalable in the sense that more data leads
to more robust as well as generalizable driving performance [4, 23] and in the
sense that there is no need to manually design intermediate interfaces or cost
functions. The neural network may be divided into modules, but the interfaces
between them are learned in order to mitigate information loss.

Hu _et al_. [22] demonstrated that their end-to-end planner, UniAD, performed
well on the popular nuScenes [5] planning benchmark. This is an open-loop
benchmark, where the tested planner never influences the driving. Instead, the
plans are compared to the trajectory taken by the vehicle during data
collection and a score is computed based on the similarity between the two.
Codevilla _et al_. [8], as well as Dauner _et al_. [11], shed some doubt about
the correlation between such an open-loop score and the actual driving
performance. This begs the question, how would state-of-the-art end-to-end
planners fare if their predicted policy would be acted upon? Unlike regular
planners that can be evaluated in a closed-loop manner using straightforward
object-level simulations, end-to-end planners require complex sensor
simulations to accurately predict their behavior in real-world scenarios. This
introduces significant challenges due to the complexity and computational
demands of high-fidelity sensor simulation. Moreover, the nuScenes benchmark
contains normal driving scenarios, in which no collisions occur. It is unclear
how state-of-the-art end-to-end planners would perform in safety-critical
scenarios, where a crash is likely unless swift corrective action is
undertaken.

In this work, we subject state-of-the-art end-to-end planners to closed-loop
evaluation in safety-critical scenarios. Given sensor data, planners predict a
plan. The plan is then executed under the constraints of a vehicle model in
order to propagate the state of the ego-vehicle forward in time. Given the new
state, we use recent advances in neural rendering – NeRFs – to resolve the
problem of generating realistic sensor data. These three steps are then
repeated until either a crash occurs or we deem the scenario to be over. By
executing the predicted plan, we aim to reduce the gap between model
evaluation and deployment. It is possible that the neural renderer exacerbates
the gap, if the rendered images are not sufficiently realistic. We quantify to
what extent the renderer affects the gap by analyzing the intermediary
perception outputs that are available in state-of-the-art (SotA) end-to-end
planners, such as UniAD [22] and VAD [23]. Moreover, we ensure that replacing
the real sensor data with rendered sensor data during open-loop evaluation
does not negatively affect the open-loop performance.

To generate safety-critical scenarios, we take inspiration from the European
New Car Assessment Protocol (Euro NCAP) for collision avoidance [14]. This
protocol comprises several scenario types that have been identified as safety-
critical. These scenario types are rare, but are likely to lead to a collision
unless the planner properly deals with them. We craft scenarios by altering
recordings of scenes from the nuScenes dataset [5]. We evaluate the driving
quality by whether there is a crash, and at what velocity that crash occurs.
Our benchmark should be viewed as a necessary but not sufficient condition for
high quality driving. To summarize, our contributions are as follows:

  1. 1.

We release an open source framework for photorealistic closed-loop simulation
for autonomous driving.

  2. 2.

We construct safety-critical scenarios, inspired by the industry standard Euro
NCAP, that cannot safely be collected in the real world.

  3. 3.

Using the simulator and our scenarios, we design a novel evaluation protocol
that focuses on collisions rather than displacement metrics.

  4. 4.

We show that two SotA end-to-end planners fail severely in our safety-critical
scenarios despite accurately perceiving the environment.

##  2 Related Work

End-to-end driving models: The autonomous driving task has traditionally been
divided into different modules – e.g., perception, prediction, and planning –
that are constructed individually [22, 23, 29]. Hu _et al_. [22] argue that
this division comes with a number of disadvantages: information loss across
modules, error accumulation, and feature misalignment. Jiang _et al_. [23]
emphasize that a planning module might need access to the semantic information
of the sensor data that would not be present in a hand-crafted interface.
These two works proceed to argue in favor of end-to-end planning. The
pioneering work of Pomerlau _et al_. [31] proposes such a planner, where a
single neural network is trained to map sensor input to an output trajectory.
Decades of neural network advancements sparked new interest in end-to-end
planning [7, 4, 9, 10, 32]. The black-box nature of these planners, however,
makes them difficult to optimize and their results hard to interpret [23]. Hu
_et al_. [22] and Jiang _et al_. [23] propose two end-to-end neural network
planners with intermediary outputs, corresponding to those of a modular
approach. Their planners are divided into modules, but the module interfaces
are learned, consisting of deep feature vectors.

Open-loop evaluation of end-to-end planners: Pomerleau _et al_. [31] evaluated
their driving model by letting it drive a real-world test-vehicle. Such a
setup makes large-scale testing costly and results may be hard to reproduce.
Recent works on end-to-end planning [22, 23] instead evaluate in _open loop_ ,
where models predict a plan based on recorded sensor data. The predicted plans
are never executed and instead, the actions are fixed to whatever was
recorded. This setup has also been used in works on object-level planning [15,
26, 12], which assume perfect perception and feed both a map of the static
environment as well as tracks of dynamic objects into the model. Such open-
loop evaluation constitutes a gap between evaluation and real-world
deployment. Moreover, performance is usually measured as the distance between
the predicted plan and the trajectory driven by the vehicle in the recording
[22, 23, 15, 26, 12]. While an error of zero corresponds to human-level
driving, it is not necessarily true that a lower error is better. This can be
realized by considering a scenario where two different trajectories are
equally good. These issues were studied by Codevilla _et al_. [8] and they
found that open-loop evaluation is not necessarily correlated with actual
driving quality. Dauner _et al_. [11] draw similar conclusions.

Closed-loop evaluation and simulation: Given the aforementioned issues of
open-loop evaluation, closed-loop simulation becomes attractive. Several
object-level simulators have been proposed [25, 2, 6, 17]. These simulators do
not generate sensor data, however, which makes it impossible to test end-to-
end planners in closed loop. A number of hand-crafted graphical simulators
have been proposed [13, 34, 35]. The challenge for such simulators is twofold:
it is difficult to create realistic-looking images and it is hard to create
graphical assets that capture the variety of the real world. Work on world
models [38, 18] demonstrate that the future of a scene – e.g., an Atari game –
can be predicted in latent space and that vectors in latent space can be
decoded into sensor input. Hu _et al_. [20] build a world-model from a large-
scale real-world automotive dataset. Amini _et al_. [3] propose VISTA, in
which novel views can be synthesized around the local trajectory by
unprojecting the closest image via predicted depth, and reprojecting. Yang _et
al_. [41] propose to use neural radiance fields (NeRF) to create
photorealistic sensor input of a scene. The method was subsequently improved
by Tonderski _et al_. [36] with more accurate sensor modeling and higher
rendering quality, particularly for the 360-degree setting considered here.

New Car Assessment Programs: The New Car Assessment Program (NCAP) was
introduced in 1979 by the U.S. Department of Transportation’s National Highway
Safety Administration in order to provide consumers with information on the
relative safety potential of automobiles [19]. NCAP crash-tested vehicles and
scored vehicles based on the probability of serious injury. A similar European
protocol was proposed in 1996, the European New Car Assessment Programme (Euro
NCAP). In 2009, Euro NCAP was overhauled in order to also include tests of
emerging crash avoidance systems [37]. Initially, this included electronic
stability control and speed assistance systems, but this was later extended to
include additional systems, such as autonomous emergency braking [37] and
autonomous emergency steering [14]. In this work, we take inspiration from the
Euro NCAP automatic collision avoidance assessment protocol [14]. This
protocol provides scenarios in which a crash will occur unless action is
taken. To obtain a full score, the vehicle needs to brake or steer to avoid
the accident. Partial scores are awarded if the impact velocity is
sufficiently reduced.

##  3 Method

Our end-to-end planning evaluation protocol comprises a closed-loop-simulator
(see Section 3.1) and a collision-focused evaluation protocol (see Section
3.2).

###  3.1 Closed-loop Simulator

Figure 2: Our closed-loop simulation engine comprises four parts. First, given
a driving log, a neural renderer (NeRF) provides photo-realistic images given
the ego-vehicle state. Second, an AD model (e.g., the end-to-end planner UniAD
[22]) uses these to predict a future ego-trajectory. Third, a controller
estimates acceleration and steering signals. Finally, a vehicle model
propagates the ego-vehicle state one step into the future. This process is
then iterated to achieve closed-loop simulation. Blue indicates simulator,
green indicates AD system.

Our closed-loop simulator repeatedly performs four steps. First, high-quality
camera input is rendered given the state and camera calibration of the ego
vehicle. The renderer is constructed from a log of a driving vehicle. Second,
the end-to-end planner predicts a future ego-vehicle trajectory given the
rendered camera input and the ego-vehicle state. Third, a controller converts
the planned trajectory to a set of control inputs. Fourth, a vehicle model
propagates the ego-state forward in time given the control inputs. This
procedure is illustrated in Figure 2. Next, we elaborate on each of the four
steps.

Neural renderer: In order to simulate novel sensor data, we adopt a neural
renderer [28]. NeRFs learn an implicit representation of the 3D environment
from a log of collected real-world data. Once trained, NeRFs can render
sensor-realistic novel views from said scene. Recent advances add the ability
to also edit the dynamic objects in the scene by changing their corresponding
3D bounding boxes [30]. Specifically, actors can be removed, added, or set to
follow novel trajectories, which in our case enables the creation of safety-
critical scenarios. For example, to simulate a rare but critical safety
scenario, a vehicle that is originally moving in an adjacent lane can be
positioned to be stationary and in the same lane as the ego-vehicle. This
novel situation necessitates the ego-vehicle to either brake or execute a
precise overtaking maneuver.

Two things should be noted. First, the recently proposed NeuRAD [36] also
supports the rendering of LiDAR data. However, as state-of-the-art end-to-end
planners consume only camera data, we focus only on camera data in this work.
Second, as we show in our experiments, the domain gap introduced by modern
NeRFs compared to real data is sufficiently small for the perception parts of
end-to-end planners to still function with high performance. However, we
expect this gap to be reduced further with future developments in neural
rendering.

AD model: Recent works on end-to-end planning [22, 23, 21] describe a system
that consumes (i) raw sensor data; (ii) the ego-vehicle state; and (iii) a
high-level plan to predict a planned trajectory. The planned trajectory
comprises waypoints at some frequency and with some time horizon. It should be
noted that while we primarily aim to analyze state-of-the-art end-to-end
planners, this module can be replaced with any type of planner, e.g., a
modular detector-tracker-planner pipeline.

Controller: In order to apply the vehicle model, the waypoints need to be
converted to a sequence of control signals, corresponding to a sequence of
steering angle ($\delta$) and acceleration ($a$) commands. Following Caesar
_et al_. [24], we achieve this with a linear quadratic regulator (LQR). Note
that while we only analyze planners that output waypoints, the planner could
instead directly output a sequence of control signals.

Vehicle model: Given a set of control signals, generated from the planned
trajectory, the vehicle state is propagated through time. To this end, we
follow prior closed-loop simulators [6, 24] and adopt a discrete version of
the kinematic bicycle model [33]. It can formally be described as

$S=\begin{pmatrix}x\\\ y\\\ \theta\\\
v\end{pmatrix},\quad\frac{dS}{dt}=\begin{pmatrix}v\cos{\theta}\\\
v\sin{\theta}\\\ \frac{v\tan{\delta}}{L}\\\ a\end{pmatrix}\enspace.$ (1)

The state $S$ is composed of $x$, $y$, $\theta$, and $v$, where $x$ and $y$ is
the longitudinal and lateral position; $\theta$ the rotation; and $v$ the
speed of the vehicle. Furthermore, $L$ is the wheelbase of the vehicle, and
$\delta$ and $a$ are the control signals. We adopt control signal limits and
the wheelbase based on the vehicle that was used when collecting the data.
Note that $x$, $y$, and $\theta$ live in a global coordinate system, whereas
$v$, $L$, $\delta$, and $a$ are values.

###  3.2 Evaluation

In contrast to common evaluation practices – _i.e_., averaging performance
across large-scale datasets – we instead focus our evaluation on a small set
of carefully designed safety-critical scenarios. These scenarios have been
crafted such that any model that cannot successfully handle all of them,
should be considered unsafe. We have taken inspiration from the industry
standard Euro NCAP testing [14] (see Section 2) and define three types of
scenarios, each characterized by the behavior of the actor that we are about
to collide with: _stationary_ , _frontal_ , and _side_. Following the Euro
NCAP nomenclature, we refer to this actor as the _target actor_. The aim is to
control the ego-vehicle to avoid a collision with the target actor or at least
decrease the collision velocity.

For each scenario type, we create multiple scenarios. Each scenario is based
on data collected from around 20 seconds of real-world driving. The ego-
vehicle and target actor states are initialized such that if current speeds
and steering angles are maintained, a collision will occur at approximately 4
seconds into the future. All non-stationary actors are removed from the scene
and we randomly select one of these to be the target actor, taking into
consideration whether the actor has been observed sufficiently closely, and
under the necessary angles, to produce realistic renderings. As our renderer
is limited to rigid actors, we exclude pedestrians from this selection.
Finally, we randomly jitter the position, rotation, and velocity of the target
actor within scenario-specific intervals. During evaluation, we run each
scenario for a large number of runs (with a fixed random seed) and compute
average results. Next, we describe the characteristics of each type of
scenario.

(a) Stationary scenario.

(b) Frontal scenario.

(c) Side scenario.

Figure 3: Different scenario types used in the NeuroNCAP evaluation protocol.
The planner is allowed perceptual input $t_{pre}$ seconds before the test
starts in order to build temporal context. Once the test starts, at
$t=t_{start}$, there are multiple actions that can lead to a successfully
completed scenario e.g., harsh breaking or a steering maneuver. To increase
the robustness of the test and allow for multiple runs, we introduce small
random perturbations to the target actor. Note that this is an illustration
and not the actual output of our renderer.

Stationary: This is a relatively simple type of scenario where a stationary
target actor has been placed in the ego-lane. The target vehicle can be placed
with an arbitrary rotation, but will remain stationary throughout the
scenario. This means that the ego-vehicle can either commit to a harsh break
or a steering maneuver to avoid a crash. See Fig. 3(a) for an illustration.

Frontal: The frontal scenarios comprise a target actor that is driving in the
opposite direction and has drifted over into the ego-lane on a collision path
with the ego-vehicle. Thus, the ego-vehicle cannot avoid crashing by breaking,
only reducing the impact speed. To completely avoid a collision, the ego-
vehicle must instead perform a steering maneuver. See Fig. 3(b) for an
illustration.

Side: The side-collision scenarios feature a target actor crossing our lane
from a perpendicular direction. If the current velocity of the ego-vehicle is
maintained, there will be a side-collision. The ego-vehicle can avoid
collision either by braking for the oncoming target actor, or by conducting a
slight steering maneuver while speeding past the target actor. See Fig. 3(c)
for an illustration.

NeuroNCAP score: For each scenario, a score is computed. A full score is
achieved only by completely avoiding collision. Partial scores are awarded by
successfully reducing the impact velocity. In spirit of the 5-star Euro NCAP
rating system[14] we compute the NeuroNCAP score (NNS) as

$\text{NNS}=\begin{cases}5.0&\text{if no collision}\\\
4.0\cdot\text{max}(0,1-v_{i}/v_{r})&\text{otherwise}\end{cases}\enspace,$ (2)

where $v_{i}$ is the impact speed as the magnitude of relative velocity
between ego-vehicle and colliding actor, and $v_{r}$ is the reference impact
speed that would occur if no action is performed. In other words, the score
corresponds to a 5-star rating if collision is entirely avoided, and otherwise
the rating is linearly decreased from four to zero stars at (or exceeding) the
reference impact speed.

##  4 Experiments

First, we start by outlining the details of our experiments in Sec. 4.1. Next,
we show the quantitative results from our NeuroNCAP evaluation in Sec. 4.2 and
some qualitative examples in Sec. 4.3. Last, we show results from our real-to-
sim study in Sec. 4.4, building more confidence in the results from the
NeuroNCAP evaluation.

###  4.1 Experimental Setting

Dataset: While there are many datasets targeting autonomous driving [1, 27,
39, 40, 16], nuScenes [5] has received the most widespread adaptation for end-
to-end planning. It features urban environments with highly interactive
scenarios, making it suitable for our safety-critical scenario generation.
Thanks to its widespread adaptation it also allows us to use official
implementations and network weights of the models we evaluate. NuScenes is
divided into 1000 sequences, out of which 150 are reserved for validation.
From these 150, we choose 14 diverse sequences – deemed to be suitable based
on the behavior of agents present in the scene – to serve as the basis for our
safety-critical scenarios.

Scenarios: Each scenario is designed by hand, considering which actors are
suitable for the given sequence, the most reasonable collision trajectories,
as well as defining allowed ranges for the different kinds of randomization.
During evaluation we run each scenario 100 times (with fixed random seed) and
average the results. Not all sequences can be used for all types of scenarios,
as for instance we cannot simulate a realistic side collision on a single
straight road. We therefore select suitable sequences for each scenario type.
For more details, and qualitative examples of each scenario, we refer to the
supplementary material.

Neural renderer: As our renderer, we opt to use NeuRAD [36], a SotA neural
renderer developed specifically for autonomous driving and verified to work
well with nuScenes. As we wish to maximize the reconstruction quality, we use
the larger configuration (NeuRAD-L), and train for 100k steps with default
hyperparameters. As pose information in nuScenes is limited to the bird’s eye
view plane, we employ pose optimization to recover the missing information.
Finally, we adopt actor flipping along the symmetry axis [41] to enable
realistic rendering of actors from all viewpoints.

AD models: We evaluate two current SotA end-to-end driving models, namely
UniAD [22] and VAD [23], according to our proposed evaluation protocol. In
both cases, we make use of the pre-trained weights made available by the
authors, trained on the same dataset, without any alterations to the
configuration of said models. Both of these models consume 360° camera input,
along with can-bus signals and a high-level command: right, left, or straight,
and output a sequence of future waypoints up to 3 seconds into the future.
While this is shorter than the initial time-to-collision (TTC) in our
scenarios, it is not an issue as the evasive maneuver can, and should, begin
before the final waypoint intersects the current actor position. Additionally,
our scenarios are designed to be quite lenient, so that a plan at TTC < 3s can
still successfully avoid collision.

One major difference between these two models is that UniAD applies a
collision-avoidance optimization post-processing step to their predicted
trajectory. The optimization is performed using a classical solver with a
cost-function based on predicted occupancy and the non-optimized output
trajectory. This optimization was shown to drastically decrease the collision-
rate when evaluated in open loop, and we can now study it in the more
interesting closed-loop setting. To enable more directly comparable analysis,
we implement the same collision avoidance optimization for VAD. However, as
VAD does not directly predict future occupancy, we rasterize their predicted
future objects and use this as the future occupancy. Note that this approach
possibly overestimates occupancy, as all future modes are treated as equally
likely.

For comparison we implement a naïve baseline method based on the perception
outputs of UniAD/VAD. The planning logic is simply a constant velocity model
unless we observe an object in a corridor in front of the ego-vehicle, in
which case we perform a braking maneuver. The corridor is defined as $\pm 2$
meters in the lateral direction and ranging from 0 to $2v_{ego}$ meters in the
longitudinal direction, _i.e_. we brake if we have TTC < 2s with an object in
front of us.

###  4.2 NeuroNCAP Results

We evaluate VAD [23] and UniAD [22], as well as the naïve baseline, on our
safety-critical scenarios. We also evaluate both methods with and without
perception-based trajectory post-processing. We report the NeuroNCAP score (2)
and collision rate per scenario type in Tab. 1. Note that the collision rate
is not averaged over time, but is defined as the ratio of scenarios that
passed without any collisions.

Surprisingly, we find that the plan predicted directly by the network, _i.e_.
without post-processing, is extremely unsafe and crashes most of the time,
even in the simple stationary scenarios. For reference, the naïve baseline
achieves an almost perfect score in the stationary setting, showing both that
the perception of these models is not at fault, and that very simple logic can
avoid collision. Trajectory post-processing further confirms this, reducing
the collision rate dramatically in the stationary setting. Side and frontal
scenarios are more difficult to handle with this rule-based logic, and the
baseline crashes 100% of the time, albeit with a lower impact speed (thus
scoring higher). Surprisingly, the end-to-end methods again show almost no
reaction to the impending collision, with 98-99% collision rate in frontal
scenarios. Trajectory post-processing improves safety somewhat, but is not
nearly as effective as in the stationary setting.

We believe that these results highlight a drastic flaw in the design or
training of current end-to-end autonomous driving systems. Reducing the
contradictions between the predicted plan and the auxiliary outputs is a
promising area of improvement for future end-to-end planners. Notably, VAD
actually attempts to address this by using multiple loss terms that directly
encourage the model to output a plan that is consistent with its perception
and prediction outputs. However, as our experiments show, this alignment step
does not generalize well, at least not to this type of safety-critical
scenario.

|  | NeuroNCAP Score $\uparrow$ | Collision rate (%) $\downarrow$
---|---|---|---
Model | Post-proc. | Avg. | Stat. | Frontal | Side | Avg. | Stat. | Frontal | Side
Base-U | - | 2.65 | 4.72 | 1.80 | 1.43 | 69.90 | 9.60 | 100.00 | 100.00
Base-V | - | 2.67 | 4.82 | 1.85 | 1.32 | 68.70 | 6.00 | 100.00 | 100.00
UniAD | x | 0.73 | 0.84 | 0.10 | 1.26 | 88.60 | 87.80 | 98.40 | 79.60
VAD† | x | 0.66 | 0.47 | 0.04 | 1.45 | 92.50 | 96.20 | 99.60 | 81.60
UniAD† | ✓ | 1.84 | 3.54 | 0.66 | 1.33 | 68.70 | 34.80 | 92.40 | 78.80
VAD | ✓ | 2.75 | 3.77 | 1.44 | 3.05 | 50.70 | 28.70 | 73.60 | 49.80
Table 1: NeuroNCAP evaluation results. End-to-end planners fail in novel,
critical scenarios. Trajectory post-processing, as proposed in UniAD, helps
significantly. The naïve baseline uses the perception of either UniAD (U) or
VAD (V) to determine braking. †Corresponds to the model’s original setting.

###  4.3 Qualitative Results

We augment the quantitative analysis with rendered front-camera images from
each scenario type in Fig. 4, with overlaid projections of the planned
trajectories. Fig. 4(a) depicts a successful avoidance maneuver, while also
highlighting our ability to render complex entities such as a motorcyclist.
However, without post-processing, the planners are prone to seemingly ignoring
the safety-critical event, as seen in Fig. 4(b).

To further examine this issue, Fig. 5 presents the perception and planning
outputs across various scenarios, demonstrating that UniAD, without post-
processing, frequently plans hazardous trajectories despite robust perception
capabilities (left). This indicates that the high collision rate is not due to
a real-to-sim gap induced by our renderer, but rather due to the planner not
handling the covariate shift between the training data and our safety-critical
scenarios. When post-processing is enabled (right), trajectories are adjusted
to avoid collisions according to the model’s internal perception and
prediction outputs. This can result in a few different outcomes. In some
cases, such as in Fig. 5(a), this is able to completely prevent a collision by
slowing down and/or steering around the target. However, the optimization does
not adequately consider the extent of the ego-vehicle, sometimes causing the
adjustments to be too small, resulting in glancing collisions as can be seen
in Fig. 5(b). Finally, due to not considering the trajectory holistically, and
only optimizing each waypoint individually, the result is sometimes
catastrophic, as seen in Fig. 5(c) and Fig. 4(c). Here, the waypoints are
repelled from the object in the opposite direction, actually causing the
vehicle to steer into the target actor and accelerate right before impact.

(a) stationary (UniAD)

(b) side (VAD)

(c) frontal (UniAD)

Figure 4: Qualitative examples of three NeuroNCAP scenarios, with projected
planning output (green, before controller) and the actual designed future
trajectory of the target actor (blue). In some cases the planner reacts
successfully (a), does not react at all (b), or attempts to avoid collision
but fails (c). Our simulator can accurately render complex actors (a), but
sometimes exhibits unrealistic artifacts for very close objects (b) and (c).

(a) Successful avoidance

(b) Insufficient margin

(c) Catastrophic failure

Figure 5: UniAD perception and planning output for three different scenarios,
with (right) and without (left) trajectory post-processing. Highlighting
unsafe planning despite strong perception, as well as strengths and weaknesses
of post-processing. The plot features ground truth objects (grey) and
predicted objects (class dependent color), and their predicted future
trajectories. Moreover, we show the ego-vehicle (black), its planned
trajectory (black) and the reference trajectory it is steering towards (red).

###  4.4 Simulation Gap Study

One important – if not the most important – concern of doing testing in
simulation is to what degree the results transfer to the real world.
Therefore, we extensively measure the real-to-sim gap across three crucial
facets of the driving system – perception, prediction, and planning – in both
open and closed-loop settings.

Replay real-to-sim (open-loop): The authors of NeuRAD [36] show that the
perception gap between rendered and real images is very small, at least in
terms of 3D object detection and relative depth estimation. However, this
study was only partially performed on nuScenes, and crucially, did not
consider planning metrics, which is what is most relevant in this work.
Therefore, we analyze the open-loop real-to-sim gap of VAD and UniAD on the
full sequences that our scenarios are based on. In Tab. 2, we report the
standard planning metrics Average Displacement Error (ADE) and Collision Rate
(CR) metrics, computed over 1, 2, and 3 seconds into the future. We also
present the perception real-to-sim gap in terms of NDS, utilizing the
auxiliary detection outputs from each model. Our analysis is limited to VAD’s
maximum range of $\pm 30$m and excludes pedestrians as they are not utilized
in any of our scenarios (and explicitly not modeled by our renderer).

The displacement error is very similar between original and rendered images,
indicating a minimal real-to-sim gap in the open-loop setting. In terms of
collision rate, both VAD and UniAD show similar or lower collision rates on
rendered images. In terms of detection performance, UniAD shows practically
identical performance on real and simulated data, whereas VAD has a gap of
roughly 3 points in terms of NDS. However, we argue that this is within
acceptable margins, especially as most evaluated objects are farther away and
thus less visible than our inserted target actors.

|  | ADE @ T (m) $\downarrow$ | CR @ T (%) $\downarrow$ | Detection $\uparrow$
---|---|---|---|---
Model | Modality | 1.00s | 2.0s | 3.0s | 1.00s | 2.0s | 3.0s |  NDS∗
UniAD | original | 0.44 | 0.75 | 1.16 | 0.00 | 0.12 | 0.21 | 0.490
UniAD | simulated | 0.47 | 0.80 | 1.24 | 0.00 | 0.12 | 0.24 | 0.489
VAD | original | 0.43 | 0.71 | 1.01 | 0.00 | 0.08 | 0.11 | 0.449
VAD | simulated | 0.44 | 0.76 | 1.16 | 0.00 | 0.00 | 0.08 | 0.413
Table 2: Real-to-sim evaluation in open loop on our 14 nuScenes sequences,
using typical planning and detection metrics. ∗Computed over VAD’s range for
fair comparison between models.

Scenario real-to-sim (closed-loop): The previous study was performed on the
original sequences, where ground truth is readily available for perception,
prediction, and planning. However, we also study the behavior of the models in
our closed-loop scenarios, where we have edited the actors and can observe the
scene from novel views. Particularly, we aim to verify that the model is able
to accurately perceive and predict the motion of the inserted actor that is
causing the criticality of the scenario. Thus, we evaluate metrics related to
the estimation of these actors, considering recall at present (0s) for
perception and recall at future times for prediction. In calculating recall,
we define a true positive as a detection or future prediction with either a
non-zero overlap or a center distance smaller than 2 meter with the target
actor. This approach is deliberately lenient to distinguish inherent
perception uncertainty in 3D estimation from failures induced by rendering
artifacts. We compute and average these metrics for all our scenarios and
present them on a per-range-to-actor basis in Tab. 3.

|  | Recall @ 0s $\uparrow$ | Recall @ 1s$\uparrow$ | Recall @ 2s$\uparrow$ | Recall @ 3s $\uparrow$
---|---|---|---|---|---
Scenario | Model | 5-15m | 15-25m | 25-35m | 5-15m | 15-25m | 25-35m | 5-15m | 15-25m | 25-35m | 5-15m | 15-25m | 25-35m
Stationary | UniAD | 0.97 | 0.98 | 0.89 | 0.94 | 0.95 | 0.84 | 0.94 | 0.93 | 0.76 | 0.94 | 0.90 | 0.69
VAD | 1.00 | 0.96 | 0.70 | 0.97 | 0.87 | 0.64 | 0.93 | 0.82 | 0.60 | 0.91 | 0.80 | 0.57
Frontal | UniAD | 0.83 | 0.97 | 0.90 | 0.82 | 0.96 | 0.90 | 0.80 | 0.93 | 0.83 | 0.77 | 0.91 | 0.73
VAD | 0.96 | 0.97 | 0.70 | 0.91 | 0.93 | 0.69 | 0.83 | 0.86 | 0.63 | 0.65 | 0.69 | 0.54
Side | UniAD | 0.92 | 0.96 | 0.65 | 0.92 | 0.96 | 0.65 | 0.90 | 0.95 | 0.62 | 0.72 | 0.63 | 0.56
VAD | 0.90 | 0.64 | 0.44 | 0.87 | 0.64 | 0.40 | 0.82 | 0.62 | 0.38 | 0.56 | 0.51 | 0.36

Table 3: Target actor recall in different ranges and at different times in the
future. Note that the model consistently has a good understanding of the
target actors’ dynamics as shown by the high recall during future prediction.

The results in Tab. 3 indicate that the model in most cases has a good
understanding of the target actor, its current state, and its potential future
states. With high recall scores (>80%) in the most safety-critical ranges
(5-25m to the target-actor), the planner should be able to plan a safe
collision avoidance maneuver. Note that VAD has a range of $\pm 30$ meters in
the longitudinal direction and $\pm 15$ meters in the lateral direction. This
can be seen in the overall decreased recall rate in the 25-35m range, as well
as in the 15-25 meter range in the side scenarios.

##  5 Limitations

We see the following limitations. First, the neural renderer is limited in the
scenes and scenarios, _e.g_., no rain, it is able to accurately render.
Moreover, large deviations in ego-vehicle trajectory and very close objects
lead to visual artifacts (see Fig. 4). Second, we adopt a simplified vehicle
model, which does not model, _e.g_., delays, friction, or suspension. Further,
we do not consider road surface aspects such as bumps, potholes, gravel, etc.
Third, we have adopted a single controller for all models, even though they
are tightly coupled. Our evaluation protocol allows for submitting AD models
that directly output control signals. Fourth, the neural renderer is unable to
deal with deformable objects, such as pedestrians. We hope that further
advances in neural rendering will lift this restriction and enable a new set
of safety-critical scenarios focusing on vulnerable road users. Fifth, the
target actor follows a predetermined trajectory, without dynamically reacting
to the ego-vehicle. While this follows the Euro NCAP setting, we believe that
future scenarios with multiple actors would require reactive behavior.

##  6 Conclusion

In conclusion, our simulation environment offers a novel approach for
evaluating the safety of autonomous driving models, drawing on real-world
sensor data and Euro NCAP-inspired safety protocols. Through the NeuroNCAP
framework, which includes stationary, frontal, and side collision scenarios,
we have exposed significant vulnerabilities in current SotA planners. These
findings not only underline the urgent need for advancements in the safety of
end-to-end planners but also suggest promising paths for future research. By
making our evaluation suite openly available to the wider research community,
we aim to catalyze progress towards safer autonomous driving. Looking ahead,
we anticipate evolving the suite to tackle a wider range of scenarios,
integrating more refined vehicle models, and employing advanced neural
rendering techniques, thereby setting new benchmarks for safety evaluation.

## Acknowledgements

We thank Georg Hess, Carl Lindström, and Adam Lilja for fruitful discussions.
This work was partially supported by the Wallenberg AI, Autonomous Systems and
Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.
Computational resources were provided by NAISS at NSC Berzelius, partially
funded by the Swedish Research Council, grant agreement no. 2022-06725.

## References

  * [1] Alibeigi, M., Ljungbergh, W., Tonderski, A., Hess, G., Lilja, A., Lindström, C., Motorniuk, D., Fu, J., Widahl, J., Petersson, C.: Zenseact open dataset: A large-scale and diverse multimodal dataset for autonomous driving. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 20178–20188 (2023)
  * [2] Althoff, M., Koschi, M., Manzinger, S.: Commonroad: Composable benchmarks for motion planning on roads. In: 2017 IEEE Intelligent Vehicles Symposium (IV). pp. 719–726. IEEE (2017)
  * [3] Amini, A., Gilitschenski, I., Phillips, J., Moseyko, J., Banerjee, R., Karaman, S., Rus, D.: Learning robust control policies for end-to-end autonomous driving from data-driven simulation. IEEE Robotics and Automation Letters 5(2), 1143–1150 (2020)
  * [4] Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L.D., Monfort, M., Muller, U., Zhang, J., et al.: End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316 (2016)
  * [5] Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., Beijbom, O.: nuScenes: A multimodal dataset for autonomous driving. In: CVPR. pp. 11621–11631 (2020)
  * [6] Caesar, H., Kabzan, J., Tan, K.S., Fong, W.K., Wolff, E., Lang, A., Fletcher, L., Beijbom, O., Omari, S.: nuPlan: A closed-loop ml-based planning benchmark for autonomous vehicles. In: Computer Vision and Pattern Recognition (CVPR) ADP3 workshop (2021)
  * [7] Chen, C., Seff, A., Kornhauser, A., Xiao, J.: Deepdriving: Learning affordance for direct perception in autonomous driving. In: Proceedings of the IEEE international conference on computer vision. pp. 2722–2730 (2015)
  * [8] Codevilla, F., Lopez, A.M., Koltun, V., Dosovitskiy, A.: On offline evaluation of vision-based driving models. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 236–251 (2018)
  * [9] Codevilla, F., Müller, M., López, A., Koltun, V., Dosovitskiy, A.: End-to-end driving via conditional imitation learning. In: 2018 IEEE international conference on robotics and automation (ICRA). pp. 4693–4700. IEEE (2018)
  * [10] Codevilla, F., Santana, E., López, A.M., Gaidon, A.: Exploring the limitations of behavior cloning for autonomous driving. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9329–9338 (2019)
  * [11] Dauner, D., Hallgarten, M., Geiger, A., Chitta, K.: Parting with misconceptions about learning-based vehicle motion planning. In: 7th Annual Conference on Robot Learning (2023), https://openreview.net/forum?id=o82EXEK5hu6
  * [12] Deo, N., Wolff, E., Beijbom, O.: Multimodal trajectory prediction conditioned on lane-graph traversals. In: Conference on Robot Learning. pp. 203–212. PMLR (2022)
  * [13] Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: Carla: An open urban driving simulator. In: Conference on robot learning. pp. 1–16. PMLR (2017)
  * [14] EuroNCAP: Assessment protocol – safety assist - collision avoidance (2023), https://www.euroncap.com/media/79866/euro-ncap-assessment-protocol-sa-collision-avoidance-v104.pdf
  * [15] Gao, J., Sun, C., Zhao, H., Shen, Y., Anguelov, D., Li, C., Schmid, C.: Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11525–11533 (2020)
  * [16] Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the kitti vision benchmark suite. In: 2012 IEEE conference on computer vision and pattern recognition. pp. 3354–3361. IEEE (2012)
  * [17] Gulino, C., Fu, J., Luo, W., Tucker, G., Bronstein, E., Lu, Y., Harb, J., Pan, X., Wang, Y., Chen, X., et al.: Waymax: An accelerated, data-driven simulator for large-scale autonomous driving research. Advances in Neural Information Processing Systems 36 (2024)
  * [18] Hafner, D., Lillicrap, T., Ba, J., Norouzi, M.: Dream to control: Learning behaviors by latent imagination. In: International Conference on Learning Representations (2019)
  * [19] Hershman, L.L.: The us new car assessment program (ncap): Past, present and future (2001)
  * [20] Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., Corrado, G.: Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080 (2023)
  * [21] Hu, S., Chen, L., Wu, P., Li, H., Yan, J., Tao, D.: St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. In: European Conference on Computer Vision. pp. 533–549. Springer (2022)
  * [22] Hu, Y., Yang, J., Chen, L., Li, K., Sima, C., Zhu, X., Chai, S., Du, S., Lin, T., Wang, W., et al.: Planning-oriented autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17853–17862 (2023)
  * [23] Jiang, B., Chen, S., Xu, Q., Liao, B., Chen, J., Zhou, H., Zhang, Q., Liu, W., Huang, C., Wang, X.: Vad: Vectorized scene representation for efficient autonomous driving. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8340–8350 (2023)
  * [24] Karnchanachari, N., Geromichalos, D., Tan, K.S., Li, N., Eriksen, C., Yaghoubi, S., Mehdipour, N., Bernasconi, G., Fong, W.K., Guo, Y., Caesar, H.: Towards learning-based planning: The nuPlan benchmark for real-world autonomous driving. In: International Conference on Robotics and Automation (ICRA) (2024)
  * [25] Krajzewicz, D.: Traffic simulation with sumo–simulation of urban mobility. Fundamentals of traffic simulation pp. 269–293 (2010)
  * [26] Liang, M., Yang, B., Hu, R., Chen, Y., Liao, R., Feng, S., Urtasun, R.: Learning lane graph representations for motion forecasting. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. pp. 541–556. Springer (2020)
  * [27] Mei, J., Zhu, A.Z., Yan, X., Yan, H., Qiao, S., Chen, L.C., Kretzschmar, H.: Waymo open dataset: Panoramic video panoptic segmentation. In: European Conference on Computer Vision. pp. 53–72. Springer (2022)
  * [28] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65(1), 99–106 (2021)
  * [29] Mobileye: Mobileye under the hood (2024), https://www.mobileye.com/ces-2024/
  * [30] Ost, J., Mannan, F., Thuerey, N., Knodt, J., Heide, F.: Neural scene graphs for dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2856–2865 (2021)
  * [31] Pomerleau, D.A.: Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems 1 (1988)
  * [32] Prakash, A., Chitta, K., Geiger, A.: Multi-modal fusion transformer for end-to-end autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7077–7087 (2021)
  * [33] Rajamani, R.: Vehicle dynamics and control. Springer Science & Business Media (2011)
  * [34] Shah, S., Dey, D., Lovett, C., Kapoor, A.: Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In: Field and Service Robotics: Results of the 11th International Conference. pp. 621–635. Springer (2018)
  * [35] Son, T.D., Bhave, A., Van der Auweraer, H.: Simulation-based testing framework for autonomous driving development. In: 2019 IEEE International Conference on Mechatronics (ICM). vol. 1, pp. 576–583. IEEE (2019)
  * [36] Tonderski, A., Lindström, C., Hess, G., Ljungbergh, W., Svensson, L., Petersson, C.: Neurad: Neural rendering for autonomous driving. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)
  * [37] Van Ratingen, M., Williams, A., Anders, L., Seeck, A., Castaing, P., Kolke, R., Adriaenssens, G., Miller, A.: The european new car assessment programme: A historical review. Chinese journal of traumatology 19(02), 63–69 (2016)
  * [38] Watter, M., Springenberg, J., Boedecker, J., Riedmiller, M.: Embed to control: A locally linear latent dynamics model for control from raw images. Advances in neural information processing systems 28 (2015)
  * [39] Wilson, B., Qi, W., Agarwal, T., Lambert, J., Singh, J., Khandelwal, S., Pan, B., Kumar, R., Hartnett, A., Pontes, J.K., Ramanan, D., Carr, P., Hays, J.: Argoverse 2: Next generation datasets for self-driving perception and forecasting. In: Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks 2021) (2021)
  * [40] Xiao, P., Shao, Z., Hao, S., Zhang, Z., Chai, X., Jiao, J., Li, Z., Wu, J., Sun, K., Jiang, K., et al.: Pandaset: Advanced sensor suite dataset for autonomous driving. In: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC). pp. 3095–3101. IEEE (2021)
  * [41] Yang, Z., Chen, Y., Wang, J., Manivasagam, S., Ma, W.C., Yang, A.J., Urtasun, R.: Unisim: A neural closed-loop sensor simulator. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1389–1399 (2023)