# Skewness as a test of dark energy perturbations

Raquel Emy Fazolo<EMAIL_ADDRESS>PPGCosmo, Universidade Federal do
Espírito Santo, 29075-910, Vitória, ES, Brazil  Luca Amendola
<EMAIL_ADDRESS>Institute of Theoretical Physics,
Philosophenweg 16, Heidelberg University, 69120, Heidelberg, Germany  Hermano
Velten<EMAIL_ADDRESS>Departamento de Física, Universidade
Federal de Ouro Preto (UFOP), Campus Morro do Cruzeiro, 35400-000, Ouro Preto-
MG Brazil

###### Abstract

We investigate the role played by dark energy perturbations in the skewness
$S_{3}$ of large-scale matter distribution. We consider a two-fluid universe
composed by matter and dark energy, with perturbations in both components, and
we estimate numerically the skewness of the matter density field as a function
of the dark energy parameters. We characterize today’s $S_{3}$ value for
quintessence and phantom dark energy cosmologies as well as its dependence on
the matter density parameter $\Omega_{m0}$ and the dark energy sound speed
$c^{2}_{s}$ with accurate numerical fitting. These fits can be used to test
cosmology against future high quality data on large scale structure.

Cosmology, dark energy, large scale structure, cosmological perturbations

##  I Introduction

Observations show that today’s universe is composed mostly of dark energy
($\approx 70\%$), a substance responsible for the current accelerated
expansion which also, under normal circumstances, slows down the clustering of
the universe’s matter content. As well known, the simplest form of dark energy
is a constant $\Lambda$ in Einstein’s general relativity acting only at
expanding background level. A fluid model for dark energy is however also a
valid description allowing for a time evolution of its energy density. The
implementation of the fluid formalism demands the definition of an equation of
state parameter (EoS), $w=P/\rho$, defined as the ratio of the fluid’s
pressure $P$ and its energy density $\rho$. For the dark energy case, even
though a time evolving EoS parameter provides a vast collection of possible
dynamical scenarios, a constant $w$ is enough to produce a rich phenomenology
of dark energy models.

In a first approximation, the late-time universe can be described as a
homogeneous and isotropic expanding background filled with matter and dark
energy. Aspects of large scale structure are studied mainly via the definition
of the matter density contrast $\delta\rho_{m}$ and dark energy perturbations
are usually neglected or are shown not to be able to contribute to the linear
regime of structure formation [1]. The latter assumption is formally obeyed by
a cosmological constant since it does not cluster and, to a good
approximation, by time-evolving dark energy models with speed of sound close
to unity [2, 3, 4, 5, 6, 7, 8]. On the other hand, dark energy perturbations
in nonadiabatic dark energy models are severely constrained by observations
[9].

While the role played by dark energy perturbations at first order has been
widely studied in the literature [10, 11, 12, 13, 14], dark energy effects at
higher clustering perturbative orders are still poorly explored: for example,
and more specifically for the goal of this work, the effects of dark energy
perturbation on the skewness ($S_{3}$) of the matter density. The skewness,
defined as the normalized third order moment of the counts-in-cells
statistics, can be used to evaluate asymmetric features in the probability
distribution function (PDF) of the matter clustering field. A finite skewness
indicates the existence of a non-vanishing balance between clustering and
voids in the large scale structure distribution. Indeed, it is well known that
the cosmological gravitational clustering process yields to a skewed large
scale structure [15, 16]. In a fully matter-dominated universe (the Einstein-
de Sitter model), one finds the well-known result $S_{3}\simeq 34/7\simeq
4.857$ [17, 16], to be corrected for finite-volume effects.

In previous work [18], by following the treatment used in Ref. [19], dark
energy effects on $S_{3}$ have been studied by modelling the entire cosmic
substratum, i.e., matter and dark energy, as a single fluid. By encoding dark
energy effects into an effective equation of state and an effective speed of
sound for the total matter fluid, a significant increase of the skewness value
($S_{3}\sim 15$) was found for any dynamical dark energy model while skewness
for $\Lambda$CDM model remains close to $34/7$. This result, while still
employing a simplistic treatment, has showed that the skewness $S_{3}$ is a
promising test to identify cosmic signatures of dark energy perturbations.
While being an interesting cosmological observable, the actual $S_{3}$ value
is still poorly constrained by the current observations [20] but in the near
future, with surveys like Euclid [21, 22], a reliable map the three-
dimensional matter distribution of the universe can provide an accurate
measurement of $S_{3}$. On the other hand, large scale N-body simulations can
be used to investigate skewness as done recently in Ref. [23].

Our aim with this work is to continue the search for dark energy signatures on
the skewness of matter distribution. Now, going beyond the analysis performed
in Ref. [18], we fully take into account fluctuations in both cosmological
fluids using the perturbed equations up to second order for a universe
composed by matter and dark energy. The dark energy density contrast acts as
an independent source of the matter fluctuations.

The main product of this paper is embodied in a series of numerical fitting
formulas for the $S_{3}$ value calculated as a function of $\Omega_{m0}$ and
$w_{de}$, with different choices of the dark energy sound speed $c_{s}^{2}$.
These fits might be useful when forecoming high-precision data from large
scale structure surveys will be able to measure $S_{3}$ to a percent
precision. We expand over previous results in several ways: a) we include the
dark energy perturbations; b) we simultaneously fit for $\Omega_{m0}$ and
$w_{de}$; c) we adopt different values for $c_{s}^{2}$. The main equations of
our work are presented in the next section. In the third section we provide
the numerical analysis and the fitting formulas for $S_{3}$. We conclude in
the final section.

##  II Background and Perturbed Dynamics

Since we focus on the late time cosmological dynamics (where the radiation
contribution is negligible), we consider a non-interacting two-fluid model
composed by pressureless matter $P_{m}=0$ (subscript $m$) and dark energy with
pressure $P_{de}=w_{de}\rho_{de}$ (subscript $de$). In a general-relativistic
based description of the gravitational interaction sourced by matter and dark
energy, Einstein’s equations take the form

$R_{\mu\nu}-\frac{1}{2}g_{\mu\nu}\mathcal{R}=8\pi
G\tensor{T}{{}_{(m)\mu\nu}}+8\pi G\tensor{T}{{}_{(de)\mu\nu}},$ (1)

being $R_{\mu\nu}$ the Ricci tensor and $\mathcal{R}=g^{\mu\nu}R_{\mu\nu}$ the
Ricci scalar. Both components are described by a perfect fluid energy momentum
tensor. Using the background expansion rate of a flat, homogeneous and
isotropic universe - the Friedmann-Lemaître-Robertson-Walker (FLRW) metric -
one finds

$\frac{H^{2}}{H^{2}_{0}}=\Omega_{m0}a^{-3}+\Omega_{de0}\,e^{-3\int
da\frac{1+w_{de}}{a}}.$ (2)

where $H=\dot{a}/a$ is the Hubble expansion rate parameter, a dot represents
the derivative with respect to cosmic time, and the scale factor is normalized
to the present value $a_{0}=1$. Since we assume a flat universe, we have
$\Omega_{de0}=1-\Omega_{m0}$.

Let us now focus on the cosmological perturbation analysis needed to
calculated the skewness of matter distribution. Since on sub-horizon scales
Newtonian physics describes fairly well the cosmological expansion and its
perturbations, we restrict our formalism to a post-Newtonian approach, or neo-
Newtonian, as e.g. in [24, 25]. Then, within the fluid limit, the cosmological
dynamics can be recast in the form of a hydrodynamical system with velocity
field given by the Hubble’s law $\vec{u}=H\vec{r}$. In this formalism energy
conservation is expressed in terms of the following continuity equation [24]

$\left(\frac{\partial\rho}{\partial
t}\right)_{r}+\vec{\nabla}_{r}\cdot(\rho\vec{u})+P\vec{\nabla}_{r}\cdot\vec{u}=0.$
(3)

The momentum conservation of a fluid moving under the influence of the
gravitational field $\phi$ is represented by the Euler equation

$\left(\frac{\partial\vec{u}}{\partial
t}\right)_{r}+(\vec{u}\cdot\vec{\nabla}_{r})\vec{u}=-\vec{\nabla}_{r}\phi-(\rho+P)^{-1}\vec{\nabla}_{r}P.$
(4)

It is worth noting that we are going to neglect shear and vorticity in the
Euler equation, a widely-employed approximation that allows our resulting set
of equations to be closed.

Each individual fluid obeys equations (3) and (4). In the case of a $N$-fluid
system, the entire dynamics will be given by $2N$ equations sourced by the
function $\phi$, i.e., $2N+1$ variables. Thus, the system of equations will be
closed by considering the Poisson equation (in the neo-Newtonian cosmology)
relating the gravitational field to the contribution of the components such
that

$\nabla^{2}\phi=4\pi G(\rho+3P).$ (5)

For a system composed by $N=2$ components, i.e. matter and dark energy, we end
up with a set of 5 equations. This post-Newtonian approach allows one to
obtain the relativistic Friedmann Eq. (2) with the identification
$\vec{u}=H\vec{r}$.

Now, in order to assess the statistical moments of the matter density field,
we introduce small perturbations around the background quantities, i.e.,

$\displaystyle\rho$ $\displaystyle=\rho_{0}+\delta\rho,$ (6) $\displaystyle P$
$\displaystyle=P_{0}+\delta P,$ $\displaystyle\phi$
$\displaystyle=\phi_{0}+\varphi,$ $\displaystyle\vec{u}$
$\displaystyle=\vec{u}_{0}+\vec{v}.$

It is also convenient to adopt the comoving coordinates $\vec{x}$ such that

$\displaystyle\vec{r}=a\vec{x},\qquad\vec{\nabla}_{x}=a\vec{\nabla}_{r},$
$\displaystyle\left(\frac{\partial f(\vec{x},t)}{\partial
t}\right)_{r}=\left(\frac{\partial f}{\partial
t}\right)_{x}-\frac{\dot{a}}{a}(\vec{x}\cdot\vec{\nabla}_{x})f,$
$\displaystyle\vec{u}=\dot{a}\vec{x}+\vec{v}(\vec{x},t).$

Focusing on the scalar modes of the cosmological perturbations that are
necessary to calculate the skewness of the matter density field one has to
keep terms up to second order in the perturbed expansion. We proceed by
isolating the divergence of the velocity field term in the perturbed
continuity Eq. (3) and inserting it into Eq. (4). With this procedure we find
the following equation for the density contrast $\delta=\delta\rho/\rho$

$\displaystyle\delta^{\prime\prime}$
$\displaystyle+\delta^{\prime}\left[3(c_{s}^{2}-w)+\left(1+\frac{\mathcal{H}^{\prime}}{\mathcal{H}}\right)-\frac{w^{\prime}}{1+w+(1+c_{s}^{2})\delta}\right]-{\delta^{\prime}}^{2}\left[\frac{4/3+c_{s}^{2}}{1+w+(1+c_{s}^{2})\delta}\right]$
$\displaystyle+\delta\delta^{\prime}\left[\frac{(w-c_{s}^{2})(5+3c_{s}^{2})-{c^{\prime}}_{s}^{2}}{1+w+(1+c_{s}^{2})\delta}\right]+\delta^{2}\left[\frac{3(w-c_{s}^{2}){c^{\prime}}_{s}^{2}+3(w-c_{s}^{2})^{2}}{1+w+(1+c_{s}^{2})\delta}\right]$
$\displaystyle+3\delta\left[({c}_{s}^{2\prime}-w^{\prime})+\left(1+\frac{\mathcal{H}^{\prime}}{\mathcal{H}}\right)(c_{s}^{2}-w)+\frac{w^{\prime}(w-c_{s}^{2})}{1+w+(1+c_{s}^{2})\delta}\right]$
$\displaystyle+\frac{1+w+(1+c_{s}^{2})\delta}{\mathcal{H}^{2}}\nabla^{2}\varphi=\frac{1+w+(1+c_{s}^{2})\delta}{\mathcal{H}^{2}}\nabla\left[\frac{\nabla(c_{s}^{2}\delta)}{1+w+(1+c_{s}^{2})\delta}\right],$
(7)

where the prime represents a derivative with respect to $N=\ln(a)$ and
$\mathcal{H}=a^{\prime}/a$. Eq. (II) represents the most general equation for
the evolution of the density contrast of a fluid with effective speed of sound
given by $c^{2}_{s}=\delta P/\delta\rho$ sourced by a gravitational potential
$\varphi$.

Let us now apply Eq. (II) to the matter fluid. In the case of a pressureless
component both the equation of state parameter and the speed of sound vanish
i.e., $w_{m}=c_{s(m)}^{2}=0$, and therefore Eq. (II) reduces to the well-known
equation

$\delta_{m}^{\prime\prime}+\delta_{m}^{\prime}\left(1+\frac{\mathcal{H}^{\prime}}{\mathcal{H}}\right)-\delta_{m}^{2^{\prime}}\left(\frac{4/3}{1+\delta_{m}}\right)+(1+\delta_{m})\frac{\nabla^{2}\varphi}{\mathcal{H}^{2}}=0.$
(8)

The contribution of the gravitational potential $\varphi$ is provided by the
perturbed Poisson equation. If the universe is filled with both pressureless
matter and dark energy, the perturbed Poisson equation reads

$\nabla^{2}\varphi=-\frac{3}{2}\Omega_{m}\mathcal{H}^{2}\delta_{m}-\frac{3}{2}\Omega_{de}\mathcal{H}^{2}\left[1+3c_{s(de)}^{2}\right]\delta_{de}.$
(9)

Hereafter we use the simplified notation $c_{s(de)}^{2}\equiv c_{de}^{2}$ for
the effective dark energy speed of sound. We will treat $c_{de}^{2}$ as a
constant free parameter in our analysis. At this point one can notice that all
dark energy perturbations effects on the gravitational potential can be
eliminated by setting $c_{de}^{2}=-1/3$. However, of course, this choice does
not represent a realistic dark energy fluid.

In order to decouple Eq. (8) into first order and second order contributions,
we expand the matter density contrast in terms of the growth functions $D_{i}$
[15, 16]

$\delta_{m}=\sum_{i=1}^{\infty}\delta_{i}=\sum_{i=1}^{\infty}\frac{D_{i}(\eta)}{i!}\delta_{0}^{i}\simeq
D_{1}\delta_{0}+\frac{D_{2}}{2}\delta_{0}^{2}+\mathcal{O}(\delta_{0}^{3})\,,$
(10)

where $\delta_{0}$ is the density contrast at some initial epoch. By
neglecting $\mathcal{O}(\delta_{0}^{3})$ terms in the above sum the resulting
dynamics is given by two equations for the evolution of the first order
($D_{1}$) and second order $(D_{2})$ growth functions. They read,
respectively,

$\displaystyle
D_{1}^{\prime\prime}+D_{1}^{\prime}\left(1+\frac{\mathcal{H}^{\prime}}{\mathcal{H}}\right)-\frac{3}{2}D_{1}\Omega_{m}-\frac{3}{2}\Omega_{de}\delta_{de}(1+3c_{de}^{2})$
$\displaystyle=$ $\displaystyle 0,$ (11) $\displaystyle
D_{2}^{\prime\prime}+{D^{\prime}}_{2}\left(1+\frac{\mathcal{H}^{\prime}}{\mathcal{H}}\right)-\frac{8}{3}{D}_{1}^{2\prime}-\frac{3}{2}D_{2}\Omega_{m}-3D_{1}^{2}\Omega_{m}-3D_{1}\Omega_{de}\delta_{de}(1+3c^{2}_{de})$
$\displaystyle=$ $\displaystyle 0\,.$ (12)

The dark energy perturbation terms can be easily identified in the above
equations. They can be switched off by setting $\delta_{de}=0$. Notice that we
kept $\delta_{de}$ at first order because dark energy is expected to be much
less clustered than matter.

With these equations we can finally calculate the skewness $S_{3}$, defined as
[16]

$S_{3}=\frac{3D_{2}}{D^{2}_{1}}.$ (13)

We now calculate $S_{3}$ in the case the matter density field is sourced by
the dark energy fluctuations $\delta_{de}$. The evolution of the latter is
obtained by adapting Eq. (II) to the dark energy fluid. Since Eqs. (11) and
(12) are sourced by the first order dark energy density perturbation
$\delta_{de}$, we also have to provide its linear equation in Fourier space

$\displaystyle\delta_{de}^{\prime\prime}$
$\displaystyle+\delta_{de}^{\prime}\left[3(c_{de}^{2}-w_{de})+\left(1+\frac{\mathcal{H}^{\prime}}{\mathcal{H}}\right)-\frac{w_{de}^{\prime}}{1+w_{de}}\right]$
$\displaystyle+3\delta_{de}\left[(c_{de}^{2\prime}-w_{de}^{\prime})+\left(1+\frac{\mathcal{H}^{\prime}}{\mathcal{H}}\right)(c_{de}^{2}-w_{de})-\frac{w_{de}^{\prime}(c^{2}_{de}-w_{de})}{1+w_{de}}\right]$
$\displaystyle-(1+w_{de})\frac{3}{2}\left[\Omega_{m}D_{1}+\Omega_{de}(1+3c_{de}^{2})\delta_{de}\right]+\frac{c_{de}^{2}k^{2}}{\mathcal{H}^{2}}\delta_{de}=0.$
(14)

When dark energy perturbations are included, a dependence on the wavenumber
$k$ appears in Eq. (II). We have checked however that $S_{3}$ is very weakly
dependent on the $k$-value for the range of scales of cosmological interest,
with changes by at most $10^{-5}$. Therefore we neglect the $k$-dependence and
adopt for all results the scale $k=0.1$ $h/$Mpc, close to the range of scales
effectively observed in large-scale surveys.

The dark energy speed of sound $c^{2}_{de}$ will be taken as an additional
free parameter. Hereafter we will adopt the following values

  * •

$c^{2}_{de}=0$, a situation in which dark energy perturbation follows the
behavior of the pressureless-like clustering fluid;

  * •

$c^{2}_{de}=1$, the effective speed of sound of a scalar field component;

  * •

$c^{2}_{de}=1/3$, as indicated recently by the analysis of Ref. [26].

The analysis of [26] refers to a model different from the one of this work,
but suggests the possibility of a dark energy model with a preferred sound
speed different from zero or unity, so we adopt this value for qualitative
purposes. Finally, besides giving accurate fits for these values, we also
include $c^{2}_{de}$ as a third parameter along with $\Omega_{m0}$ and
$w_{de}$.

##  III Results and numerical fits

An expansion for the skewness value around the Einstein-De-Sitter universe
($\Omega_{m}=1$ and $\Omega_{\Lambda}=0$) is often employed in the literature.
A widely known result is provided by Bernardeau et al. (2002) [16] where a
weak dependence of the skewness on the $\Omega_{m0}$ parameter
$S_{3}=34/7+6/7(\Omega^{-0.03}_{m0}-1)$ has been found for the case where
$\Lambda=0$. The main goal of this paper is to extend this analysis by
including the effects of dark energy fluctuations as a function of the matter
density parameter $\Omega_{m0}$, the dark energy equation of state $w_{de}$,
and its squared sound speed $c^{2}_{de}$.

We begin our analysis by showing in Fig. 1 how the skewness evolves in time
for a $\Lambda$CDM cosmology with parameters $\Omega_{m0}=0.3,w_{de}=-1$ (blue
line) and for the Einstein-de Sitter (EdS) model (black line). As expected,
the non-Gaussianity increases with time reaching asymptotically a constant
value for both cases. One can clearly see the manifestation of dark energy
around redshift 1. We also show models with dark energy perturbations for
$w_{de}=-0.8$ and $w_{de}=-1.2$.

Figure 1: Skewness evolution as a function of redshift. The colors represent
the following cases: blue for a $\Lambda$CDM case i.e., $\Omega_{m0}=0.3$,
$w_{de}=-1$; black for EdS case i.e., $\Omega_{m0}=1$ and $c^{2}_{de}=0$;
orange for the quintessence case with $\Omega_{m0}=0.3$, $w_{de}=-0.8$ and
$c^{2}_{de}=1$ and finally the dashed magenta for a phantom case with
$\Omega_{m0}=0.3$, $w_{de}=-1.2$ and $c^{2}_{de}=1$, the last two includes
dark energy perturbations. The black line and the dashed magenta are almost
indistinguishable.

Next, we compare $S_{3}$ with and without dark energy perturbations. We solve
numerically Eq. (11), Eq. (12), and Eq. (II) varying the parameters
$\Omega_{m0}$ and $w_{0}$. We choose $a_{i}=1/(1+z_{i})$, being $z_{i}\simeq
300000$ such as when $t\rightarrow 0$ we have EdS values [15], our initial
condition and also assume $D_{1}(a_{i})\propto a$ and $D_{2}(a_{i})\propto
a^{2}$. We show in Fig. 2 today’s skewness value $S_{3}$ as a function of
$\Omega_{m0}$ for various values of $w_{de}$ and $c^{2}_{de}$. On the left
plot of Fig.2 we notice that for the quintessence cases ($w_{de}=-0.9$ in the
figure) the inclusion of dark energy perturbations has a relatively large
impact. We find that dark energy perturbations always lower the skewness with
respect to $\delta_{de}=0$. As for phantom cases ($w_{de}=-1.2$ in plot), we
find almost no change. This seems to indicate that the skewness can be a much
better tool to test quintessence models than for models in the phantom regime.
Also the speed of sound for the dark energy fluid affects the skewness in
different ways depending on whether dark energy is phantom [27] or
quintessence [28] as also shown in Fig. 2. Overall, we see that the typical
change in $S_{3}$ from model to model is of the order of 1 or 2%.

In order to provide an expression for $S_{3}$ as a function of the
cosmological parameters, we have performed phenomenological fits to the
numerical results. To assess the quality of the fits, we adopt two measures,
the maximum relative deviation (MRD) and the average standard relative
deviation (ASRD), defined respectively as the following percentages

$\displaystyle{\rm MRD}=$ $\displaystyle 100\,{\rm Max}\Big{\lvert}\frac{{\rm
Fit}(i)-{\rm Data}(i)}{{\rm Fit}(i)}\Big{\rvert},$ $\displaystyle{\rm ASRD}=$
$\displaystyle 100\sqrt{\frac{1}{N}\sum\Big{(}\frac{{\rm Fit}(i)-{\rm
Data}(i)}{{\rm Fit}(i)}\Big{)^{2}}},$ (15)

where ${\rm Data}(i)$ represents the numerical outcomes of our system of
equations for a given set parameter values $i$ and ${\rm Fit}(i)$ is the
fitted function.

Figure 2: Dependence of the skewness on the density parameter $\Omega_{m0}$.
In the left panel the dot-dashed lines represent $c^{2}_{de}=0$, solid lines
$c^{2}_{de}=1$ and dashed lines is the case without dark energy perturbations
($\delta_{de}=0$). The colors refers to different $w_{de}$ values: blue for
$w_{de}=-1$, magenta for $w_{de}=-1.2$ and orange for $w_{de}=-0.9$. On the
right side we show the skewness for $\Lambda\neq 0$ by Bernardeau (1994) [15]
(dashed blue) compared to our result (dashed black) for the $\Lambda$CDM case
($w_{de}=-1$, $\Omega_{m0}=0.3$ (black dot) and no dark energy perturbations
$\delta_{de}=0$). The black line is for the EdS skewness value of
$34/7=4.857$. The fit Eq.(16) is represented by the red cross symbols. We also
show the skewness when dark energy perturbations are included, for
$w_{de}=-0.9$ (orange) $w_{de}=-1.2$ (magenta), both with $c^{2}_{de}=1$. The
inset in the right panel shows the full behavior of the $w_{de}=-0.9$ case.

In Fig. 2 (right side) we show that the skewness calculated for a $\Lambda$CDM
($\delta_{de}=0$) case is almost identical to Bernadeau’s results in [15] for
$\Lambda\neq 0$. A fit to our results is provided as the following equation:

$S_{3}=4.857-0.053(\Omega_{m0}^{0.129}-1).$ (16)

For this fit we obtain the following values quality measurements: MRD=$0.01\%$
and ASRD$=0.002\%$ compared to the results of the equations itself. We also
plot in the same figure the EdS reference value $34/7=4.857$ as the horizontal
black line and repeat the cases for dynamical dark energy models for
quintessence $w_{de}=-0.9$ and phantom $w_{de}=-1.2$ cosmologies including
dark energy perturbations, both with $c_{de}^{2}=1$ as in the left panel.

As already mentioned, the skewness depends much more strongly on $w_{de}$ in
the regime $w_{de}>-1$ than below it (i.e., in the phantom regime), as shown
more clearly in Fig. 3.

Figure 3: Dependence of $S_{3}$ on the dark energy equation of state $w_{de}$.
The colored curves refers to different values of the matter density parameter,
namely $\Omega_{m0}=0.2$ (red), $\Omega_{m0}=0.3$ (blue) and $\Omega_{m0}=0.4$
(purple). The black line sets the Einstein-de Sitter value $34/7$. We assumed
$c^{2}_{de}=1$ for all curves.

Our goal now is to obtain a general fitting formula for $S_{3}$ that includes
the dependence on both $\Omega_{m0}$ and $w_{de}$ for various values of
$c^{2}_{de}$. We propose the following parametrization, defined by six
constants $a,b,..f$:

$S_{3}({\rm
fit})=a+b(\Omega_{m0}^{c}-1)+d|(1+w_{de})|^{e}+f(1+\Omega_{m0}w_{de}).$ (17)

By setting $d=e=f=0$ one obtains the previous formula (16). Also, the constant
$S_{3}$ value of the EdS model is immediately retrieved in the limit
$\Omega_{m0}=1$ and $w_{de}=-1$. We aim at an average (ASRD) and maximum (MRD)
precision much better than 1% across the entire parametric range.

Tables 1 (phantom regime only) and 2 (quintessence regime only) report the
values for the fitting free parameters $a,b,...,f$ and their respective MRD
and ASRD values for some values of $c_{de}^{2}$. Although the ASRD is always
much smaller than 1%, the MRD is larger for the quintessence regime, but we
find that the deviation mostly occurs near the boundary regions of
$\Omega_{m0}$. So we also present in Tab. 3 the results for a smaller range of
$\Omega_{m0}$ ($0.15\leq\Omega_{m0}\leq 0.45$), obtaining improved fits.

In order to visualise the accuracy of the fitting formula Eq.(17) with the
parameters of Table(2), we show in Fig.4 the comparison between the
numerically generated data and the fit. Here we have chosen to display the
quintessence case; since this is the case with largest MRD and ASRD values,
all other parameter configurations produce a better fit.

Phantom ($-1.25\leq w_{de}\leq-1$) | a | b | c | d | e | f | MRD% | ASRD%
---|---|---|---|---|---|---|---|---
$c_{de}^{2}=0$ | 4.861 | -0.052 | 0.908 | -0.023 | 0.444 | -0.042 | 0.06 | 0.01
$c_{de}^{2}=1$ | 4.860 | -0.067 | 0.899 | -0.030 | 0.552 | -0.053 | 0.08 | 0.01
$c_{de}^{2}=1/3$ | 4.860 | -0.059 | 0.903 | -0.026 | 0.500 | -0.047 | 0.07 | 0.01
Table 1: Fit values for Eq. (17) for the phantom regime $-1.25\leq w_{de}\leq-1$. Quintessence ($-1\leq w_{de}\leq-0.8$) | a | b | c | d | e | f | MRD% | ASRD%
---|---|---|---|---|---|---|---|---
$c_{de}^{2}=0$ | 4.852 | 0.334 | 1.076 | -2.534 | 1.924 | 0.364 | 1.12 | 0.06
$c_{de}^{2}=1$ | 4.861 | 0.437 | 1.138 | -2.948 | 1.699 | 0.468 | 2.23 | 0.14
$c_{de}^{2}=1/3$ | 4.856 | 0.405 | 1.109 | -2.841 | 1.791 | 0.437 | 1.72 | 0.11
Table 2: Fit values for Eq. (17) for the quintessence regime $-1\leq w_{de}\leq-0.8$. | Quintessence ($-1\leq w_{de}\leq-0.8$)
---
($0.15\leq\Omega_{m0}\leq 0.45$)
a | b | c | d | e | f | MRD% | ASRD%
$c_{de}^{2}=0$ | 4.853 | 0.244 | 0.986 | -2.290 | 1.905 | 0.261 | 0.07 | 0.02
$c_{de}^{2}=1$ | 4.863 | 0.237 | 0.981 | -2.321 | 1.605 | 0.240 | 0.11 | 0.03
$c_{de}^{2}=1/3$ | 4.858 | 0.250 | 0.982 | -2.362 | 1.729 | 0.260 | 0.09 | 0.02
Table 3: Fit values for Eq. (17) for the quintessence regime $-1\leq w_{de}\leq-0.8$ but made for a smaller range of $\Omega_{m0}$. | a | b | c | d | e | f | g | MRD% | ASRD%
---|---|---|---|---|---|---|---|---|---
Phantom ($-1.25\leq w_{de}\leq-1$) | 4.860 | -0.061 | 0.903 | -0.027 | 0.510 | -0.049 | 0.000 | 0.01 | 0.06
Quintessence ($-1\leq w_{de}\leq-0.8$) | 4.844 | 0.408 | 1.115 | -2.854 | 1.794 | 0.439 | -0.028 | 0.21 | 0.19
Table 4: Fit values for Eq. (18) for the phantom and quintessence regimes.

We also propose a general fitting formula with $c_{de}^{2}$ as a variable [29,
30, 31]. The results are presented in Table(4) and Fig.(5):

$S_{3}({\rm
fit}\\_c_{de}^{2})=a+b(\Omega_{m0}^{c}-1)+d|(1+w_{de})|^{e}+f(1+\Omega_{m0}w_{de})+g(c_{de}^{2}+w_{de}).$
(18)

We can see from Table (1) that $c_{de}^{2}$ has a very little effect in the
phantom regime. For example for $w_{de}=-1.10$ and $\Omega_{m0}=0.3$, one has
a slight increase of only $0.01\%$ in the $S_{3}$ value when $c_{de}^{2}$
varies from $0$ to $1$. The impact is larger in the quintessence regime Table
(2): for instance, for $w_{de}=-0.90$ and $\Omega_{m0}=0.3$, when $c_{de}^{2}$
varies from $0$ to $1$, we have a decrease of $0.5\%$ in the $S_{3}$ value.

Figure 4: Variation of skewness with $\Omega_{m0}$ and $w_{de}$. On the left
panels we fix $\Omega_{m0}=0.3$; on the right panels we fix $w_{de}=-0.9$. The
red line represents the fit from Eq. (17) and Table 2, and the black dots the
data. Here we have the case for $c_{de}^{2}=0$ (top row) and $c_{de}^{2}=1$
(lower row). Figure 5: Variation of skewness with $c^{2}_{de}$. The red line
represents the fit Eq.(18) and Table 4 for $\Omega_{m0}=0.3$ and
$w_{de}=-0.9$; the black dots are the data.

Apart from the theoretical analysis performed so far, it is also important to
have contact with observed quantities. However, the value of $S_{3}$ for
biased tracers is actually expected to be shifted by a factor and a constant
that depend on the linear and non-linear bias parameters [32], plus a
correction that depends on the cell window function. One could compare the
data with $S_{3}$ and find the best fit values of the bias parameters, but
here we limit ourselves to plotting $S_{3}$ against the data to have a general
idea of the trend.

In Fig. 6 we show our results against the observed values from Ref. [20] from
the CFHTLS-Wide survey. In this analysis the skewness is calculated using a
counts-in-cells technique for several intervals of redshift. Here we managed
to reproduce a larger range of theoretical results than in [20] using a well-
behaved (i.e., less oscillating than the top-hat) window function, namely

$W({kR})=\frac{\sqrt{2}e^{-\frac{{kR}}{\sqrt{2}}}}{{kR}}\sin\left(\frac{{kR}}{\sqrt{2}}\right),$
(19)

where $W(kR)$ is the Fourier transform of the smoothed top-hat spherically
symmetric region with characteristic radius $R$ defined as
$W(r)=1/(1+(r/R)^{4})$ normalized to unity. Then, the following expression is
used [20, 16, 33, 34]

$S_{3}=\frac{34}{7}+\gamma_{1}.$ (20)

where $\gamma_{1}=d\ln\sigma^{2}(R)/d\ln R$ is the count-in-cells correction
[15, 20] and the variance of the density field in real space is given by

$\sigma^{2}(R)=\frac{1}{2\pi^{2}}\int P(k,z)\,W^{2}(kR)\,k^{2}\,dk.$ (21)

where $P(k,z)$ is a $\Lambda$CDM power spectrum at redshift $z$. We can see
that the general trend is consistent with the data, but clearly the precision
is still far from allowing a robust constraint on the cosmological and the
bias parameters.

Figure 6: Skewness compared to observational data from [20]. The blue line
represents the theoretical plot using our set of equations with
$\Omega_{m0}=0.27$, $w_{de}=-1$, $c^{2}_{de}=0$ and $H_{0}=71$ km/s/Mpc that
was also used in [20]. The orange line represents quintessence $w_{de}=-0.8$
and $c^{2}_{de}=1$. For the phantom case $w_{de}=-1.2$ and $c^{2}_{de}=1$, the
curve is almost identical to the blue line. The theoretical curve is almost
independent of $z$; we plot here the $z=0.9$ case.

##  IV Conclusions

This work presents numerical estimates of $S_{3}$ (the third moment of the
matter density field) when including dark energy perturbations while varying
$\Omega_{m0},w_{de}$ and $c^{2}_{de}$. Whereas a cosmological constant affects
the matter clustering features only via the background expansion, a dynamical
dark energy field with its associated perturbations can lead to noticeable
imprints on the large scale structure. This means that a measure of $S_{3}$
within a precision of 1% can put new constraints on cosmological parameters.

In a previous work, [18], the universe content was modeled as a effective
single fluid (an admixture of dark energy and matter), in which the dark
energy contributions have been considered via the total (the single fluid
realization) equation of state parameter and sound speed. In contrast, in this
work we consider the universe as composed by two separated fluids, both of
which being perturbed. As expected, we obtain now values for today’s $S_{3}$
much closer to the $\Lambda$CDM case (a factor of three smaller than in [18]),
in agreement therefore with standard results presented in the literature, e.g.
[15] and [20].

We found clear differences in the trend for quintessence ($w_{de}>-1$) and
phantom models ($w_{de}<-1$), being $S_{3}$ more sensitive to the dark energy
equation of state in the quintessence regime. This allows in principle to use
the skewness to test cosmological models with $w_{de}>-1$.

We provided several fitting formulas accurate to within (or better than) 0.1%
on average for $S_{3}$ as a function of the cosmological parameters
$\Omega_{m0},w_{de}$ and $c^{2}_{de}$. These fits may be useful to compare the
cosmological models to large scale structure at higher orders.

In future work, we plan to extend this analysis to include more cosmological
parameters and modified gravitational models, as e.g. in [35, 36].

## Acknowledgments

Raquel Fazolo thanks CNPq and FAPES for financial support. Hermano Velten
thanks CAPES/CNPq and Proppi/UFOP for partial finantial support. LA
acknowledges financial support from DFG project 456622116.

## References

  * [1] Luca Amendola.  Linear and non-linear perturbations in dark energy models.  Phys. Rev. D, 69:103524, 2004.
  * [2] L. R. Abramo, R. C. Batista, L. Liberato, and R. Rosenfeld.  Structure formation in the presence of dark energy perturbations.  JCAP, 11:012, 2007.
  * [3] L. R. Abramo, R. C. Batista, L. Liberato, and R. Rosenfeld.  Physical approximations for the nonlinear evolution of perturbations in inhomogeneous dark energy scenarios.  Phys. Rev. D, 79:023516, 2009.
  * [4] D. F. Mota and C. van de Bruck.  On the Spherical collapse model in dark energy cosmologies.  Astron. Astrophys., 421:71–81, 2004.
  * [5] Ronaldo C. Batista and Valerio Marra.  Clustering dark energy and halo abundances.  JCAP, 11:048, 2017.
  * [6] Matteo Fasiello and Zvonimir Vlah.  Nonlinear fields in generalized cosmologies.  Phys. Rev. D, 94(6):063516, 2016.
  * [7] Emiliano Sefusatti and Filippo Vernizzi.  Cosmological structure formation with clustering quintessence.  JCAP, 03:047, 2011.
  * [8] Takao Endo, Atsushi J. Nishizawa, and Kiyotomo Ichiki.  Effect of dark energy perturbation on cosmic voids formation.  Mon. Not. Roy. Astron. Soc., 478(4):5230–5239, 2018.
  * [9] Winfried Zimdahl, Júlio Fabris, Hermano Velten, and Ramón Herrera.  On (non-)dynamical dark energy.  Phys. Dark Univ., 30:100681, 2020.
  * [10] Manvendra Pratap Rajvanshi and J. S. Bagla.  Nonlinear spherical perturbations in Quintessence Models of Dark Energy.  JCAP, 06:018, 2018.  [Erratum: JCAP 03, E01 (2020)].
  * [11] F. Pace, C. Fedeli, L. Moscardini, and M. Bartelmann.  Structure formation in cosmologies with oscillating dark energy.  Mon. Not. Roy. Astron. Soc., 422:1186–1202, 2012.
  * [12] L. Raul Abramo, Ronaldo C. Batista, and Rogerio Rosenfeld.  The signature of dark energy perturbations in galaxy cluster surveys.  JCAP, 07:040, 2009.
  * [13] A. Mehrabi, S. Basilakos, and F. Pace.  How clustering dark energy affects matter perturbations.  Mon. Not. Roy. Astron. Soc., 452(3):2930–2939, 2015.
  * [14] Francesco Pace, Sven Meyer, and Matthias Bartelmann.  On the implementation of the spherical collapse model for dark energy models.  JCAP, 10:040, 2017.
  * [15] Francis Bernardeau.  Skewness and Kurtosis in large scale cosmic fields.  Astrophys. J., 433:1, 1994.
  * [16] F. Bernardeau, S. Colombi, E. Gaztanaga, and R. Scoccimarro.  Large scale structure of the universe and cosmological perturbation theory.  Phys. Rept., 367:1–248, 2002.
  * [17] P. J. E. Peebles.  The large-scale structure of the universe.  1980\.
  * [18] Hermano Velten and Raquel Emy Fazolo.  Skewness of matter distribution in clustering dark energy cosmologies.  Phys. Rev. D, 101(2):023518, 2020.
  * [19] R. R. R. Reis, M. Makler, and Ioav Waga.  Skewness as a test for quartessence.  Phys. Rev. D, 69:101301, 2004.
  * [20] M. Wolk, H. J. McCracken, S. Colombi, J. N. Fry, M. Kilbinger, P. Hudelot, Y. Mellier, and O. Ilbert.  Evolution of hierarchical clustering in the CFHTLS-Wide since z~1.  Mon. Not. Roy. Astron. Soc., 435:2, 2013.
  * [21] Luca Amendola et al.  Cosmology and fundamental physics with the Euclid satellite.  Living Rev. Rel., 21(1):2, 2018.
  * [22] R. Laureijs et al.  Euclid Definition Study Report.  10 2011.
  * [23] Jaan Einasto, Anatoly Klypin, Gert Hütsi, L. J. Liivamägi, and Maret Einasto.  Evolution of skewness and kurtosis of cosmic density fields.  Astron. Astrophys., 652:A94, 2021.
  * [24] J. A. S. Lima, V. Zanchin, and Robert H. Brandenberger.  On the Newtonian cosmology equations with pressure.  Mon. Not. Roy. Astron. Soc., 291:L1–L4, 1997.
  * [25] R. R. R. Reis.  Domain of validity of the evolution of perturbations in Newtonian cosmology with pressure.  Phys. Rev. D, 67:087301, 2003.  [Erratum: Phys.Rev.D 68, 089901 (2003)].
  * [26] Adam Moss, Edmund Copeland, Steven Bamford, and Thomas Clarke.  A model-independent reconstruction of dark energy to very high redshift.  9 2021.
  * [27] R. R. Caldwell.  A Phantom menace?  Phys. Lett. B, 545:23–29, 2002.
  * [28] K. Benabed and F. Bernardeau.  Testing quintessence models with large scale structure growth.  Phys. Rev. D, 64:083501, 2001.
  * [29] Olga Sergijenko and Bohdan Novosyadlyj.  Sound speed of scalar field dark energy: weak effects and large uncertainties.  Phys. Rev. D, 91(8):083007, 2015.
  * [30] Tobias Basse, Ole Eggers Bjaelde, and Yvonne Y. Y. Wong.  Spherical collapse of dark energy with an arbitrary sound speed.  JCAP, 10:038, 2011.
  * [31] Paolo Creminelli, Guido D’Amico, Jorge Norena, Leonardo Senatore, and Filippo Vernizzi.  Spherical collapse in quintessence models with zero speed of sound.  JCAP, 03:027, 2010.
  * [32] J. N. Fry and Enrique Gaztanaga.  Biasing and Hierarchical Statistics in Large-Scale Structure.  Astrophys. J. , 413:447, August 1993.
  * [33] F. Bernardeau, R. Juszkiewicz, A. Dekel, and F. R. Bouchet.  Omega from the skewness of the cosmic velocity divergence.  Mon. Not. Roy. Astron. Soc., 274:20–26, 1995.
  * [34] R. Juszkiewicz, F. R. Bouchet, and S. Colombi.  Skewness induced by gravity.  Astrophys. J. Lett., 412:L9, 1993.
  * [35] Luca Amendola and Claudia Quercellini.  Skewness as a Test of the Equivalence Principle.  Phys. Rev. Lett. , 92(18):181102, May 2004.
  * [36] Takayuki Tatekawa and Shinji Tsujikawa.  Second-order matter density perturbations and skewness in scalar-tensor modified gravity models.  JCAP, 09:009, 2008.


# Situation Monitor: Diversity-Driven Zero-Shot Out-of-Distribution Detection
using Budding Ensemble Architecture for Object Detection

Syed Sha Qutub  Intel Labs, Munich, Germany  Technical University of Munich,
Munich, Germany  Michael Paulitsch  Intel Labs, Munich, Germany  Kay-Ulrich
Scholl  Intel Labs, Munich, Germany  Neslihan Kose Cihangir  Intel Labs,
Munich, Germany  Korbinian Hagn  Intel Labs, Munich, Germany  Fabian Oboril
Intel Labs, Munich, Germany  Gereon Hinz  Technical University of Munich,
Munich, Germany  Alois Knoll  Technical University of Munich, Munich, Germany

###### Abstract

We introduce Situation Monitor, a novel zero-shot Out-of-Distribution (OOD)
detection approach for transformer-based object detection models to enhance
reliability in safety-critical machine learning applications such as
autonomous driving. The Situation Monitor utilizes the Diversity-based Budding
Ensemble Architecture (DBEA) and increases the OOD performance by integrating
a diversity loss into the training process on top of the budding ensemble
architecture, detecting Far-OOD samples and minimizing false positives on
Near-OOD samples. Moreover, utilizing the resulting DBEA increases the model’s
OOD performance and improves the calibration of confidence scores,
particularly concerning the intersection over union of the detected objects.
The DBEA model achieves these advancements with a 14% reduction in trainable
parameters compared to the vanilla model. This signifies a substantial
improvement in efficiency without compromising the model’s ability to detect
OOD instances and calibrate the confidence scores accurately.

##  1 Introduction

In machine learning, models must exhibit effective generalization capabilities
that require adaptability beyond their training data. This adaptability is
crucial for ensuring the effective performance of models in real-life
situations populated with diverse objects. In real-world applications,
particularly in safety-critical scenarios such as autonomous driving or
medical diagnosis, the capability to detect OOD instances is decisive for
ensuring the robust performance of machine learning models. OOD instances
refer to situations where the model encounters data patterns or objects that
differ significantly from what it was exposed to during training. Detecting
OOD instances becomes particularly challenging when models are expected to
generalize effectively across diverse and unpredictable situations [22]. This
adaptability is essential for ensuring the reliability and safety of machine
learning models in dynamic and complex environments.

Figure 1: Out-of-Distribution definition, Dotted line represents the decision
boundary of an OOD detection model that generalizes effectively.

To address this classification challenge, this study explores the two types of
OOD conditions, as depicted in Fig. 1: Near-OOD and Far-OOD [9]. In the
context of a Near-OOD dataset, there is a notable resemblance in features and
characteristics to the training dataset, also referred to as in-distribution
(IN) data. The Near-OOD dataset may resemble datasets obtained from diverse
acquisition sensors, e.g., when evaluating a model trained with the autonomous
driving dataset KITTI [10], other autonomous driving datasets like BDD100K
[23], Cityscapes [6] (2D bounding boxes from CityPersons [25]) or Lyft [11]
(2D bounding boxes computed from 3D bounding boxes) can be categorized as
Near-OOD. The Far-OOD dataset introduces a different paradigm, where the
dataset is entirely dissimilar to the In-Distribution (IN) dataset, surpassing
the characteristics of Near-OOD conditions. Given the previous example, the
CoCo [15] dataset would suffice for this Far-OOD condition. The OOD module
integrated within the Deep Neural Networks (DNNs) for a specific application
should avoid miss-classifying Near-OOD instances as OOD while correctly
flagging instances significantly different from the training dataset, i.e.,
Far-OOD, as the OOD cases.

With the transition from traditional Convolution Neural Networks (CNNs) to
transformer-based models due to their remarkable ability to capture long-range
dependencies and contextual information [19], a shift and redesign of OOD
methods designed initially for CNNs is in progress.

Therefore, we showcase the OOD detection performance of our proposed Situation
Monitor derived by leveraging the deviations of the ensemble predictions of
the DBEA model integrated into the DINO-DETR [24] transformer-based vision
model. DBEA is derived from the budding ensemble architecture proposed by
[18]. The tandem loss function is hereby extended to incorporate a diversity
loss function. Given that vision-based transformers commonly feature encoder-
decoder structures. As a final stage of fully connected layers, it is
generally applicable to integrate the Situation Monitor into various other
vision transformer models without loss of generality.

In this work, we propose to:

  * •

Create the DBEA model by incorporating diversity-based loss into the training
process of BEA.

  * •

Define the Situation Monitor to detect Far-OOD samples and suppress false
positives on Near-OOD samples.

  * •

Minimize the overhead of the transformer model with the Situation Monitor
compared to baseline models.

Through comprehensive ablation study experiments, we evaluate the performance
of DBEA in comparison to multiple sample-free OOD detection baselines. These
baselines were specifically trained on well-established datasets such as KITTI
and BDD100K, serving as benchmarks for our analysis.

##  2 Related Work

Researchers are continuously exploring innovative methodologies and refining
existing techniques to bolster the capabilities of object detectors,
particularly in handling out-of-distribution situations. Classifying Deep
Neural Networks (DNNs) into deterministic and probabilistic networks provides
a foundational understanding.

Deterministic Networks vs Probabilistic Networks: Deterministic networks in
DNNs generate consistent outputs for a given input in a deterministic manner
[8]. However, these networks cannot model prediction uncertainty. As a result,
confidence scores associated with their predictions become crucial for
measuring uncertainty, serving as valuable indicators for OOD detection [3].
In contrast, probabilistic DNNs explicitly model uncertainty in their
predictions by outputting probability distributions over possible outcomes
[8]. This explicit modelling benefits OOD detection across a wide range of
applications [1]. While OOD detection using probabilistic networks can be
computationally demanding, researchers have devised various techniques for
deterministic networks, primarily focusing on post-hoc methods [21, 13, 14].

Unified Frameworks for OOD Detection: In addition to post-hoc methods, a few
unified frameworks incorporate OOD detection seamlessly into the primary task
of the DNN. These frameworks utilize zero-shot or few-shot learning strategies
to improve OOD detection capabilities [4, 7].

The findings from [12], indicating that wider networks with similar
architecture learn more similar features, are integrated into the subsequent
work of the sample-free uncertainty estimation method BEA [18]. BEA notably
exhibited significant improvements in OOD detection, primarily focusing on
anchor-based CNN models for object detection. It is worth noting that most of
the related research is connected to CNN-based DNNs, and there is limited
exploration into adapting and extending these techniques for transformer-based
object detection models.

Sample-Free Uncertainty Estimation Method: Notably, BEA [18], a sample-free
uncertainty estimation method, demonstrated considerable performance
enhancements in OOD detection. This method primarily focused on anchor-based
CNN models for object detection. Similarly, Gaussian-Yolov3 [5] introduced
Gaussian parameters to exploit variances and reduce false positives by
calibrating them to the IoU.

In summary, researchers are continually exploring new methodologies and
refining existing techniques to enhance the capability of object detectors in
handling out-of-distribution situations. This includes methods focused on both
deterministic and probabilistic networks, as well as unified frameworks that
seamlessly incorporate OOD detection into the primary task of the DNN.
Additionally, there is potential for adapting and extending these techniques
for transformer-based object detection models. However, most of this research
is associated with CNN-based DNNs, and there is limited exploration into
adapting and extending these techniques for transformer-based object detection
models.

##  3 Problem Statement

Figure 2: The primary aim of the Situation Monitor is to distinguish between
known and unknown situations. For instance, a model trained on datasets such
as KITTI for automotive scenarios is categorized as a Near-Out-of-Distribution
(Near-OOD) situation. Consequently, encountering an indoor scenario would be
labelled as Far-Out-of-Distribution (Far-OOD) situation by the model.

In the closed-world assumption, the training dataset ($\mathcal{D}$) and
testing dataset ($\mathcal{T}$) is from in-distribution dataset $\mathcal{I}$,
i.e., $\mathcal{D}$ $\subset$ $\mathcal{I}$. Therefore, the samples from the
testing dataset is $\mathcal{S(T)}=\mathcal{S(T^{\mathcal{I}})}$. However, in
open-world settings and practical, real-world scenarios, samples are also
drawn from OOD data. Therefore, the OOD samples $\mathcal{O}$ are composed of
both Near-OOD ($\mathcal{O^{\textit{near}}}$) and Far-OOD
($\mathcal{O^{\textit{far}}}$), i.e. $\mathcal{O}$ =
$\mathcal{O^{\textit{near}}}$ \+ $\mathcal{O^{\textit{far}}}$. Similarly, in
the open world setting, the testing data $\mathcal{T}$ consists of known
situations and classes originating from Near-OOD $\mathcal{O^{\textit{near}}}$
and unknown situation and unknown classes to the model originating from Far-
OOD $\mathcal{O^{\textit{far}}}$ i.e.,
$\mathcal{T}=\mathcal{T^{\mathcal{I}}}+\mathcal{T}^{\mathcal{O}^{near}}+\mathcal{T}^{\mathcal{O}^{far}}$
and accordingly the samples from the testing data is
$\mathcal{S(T)}=\mathcal{S(T^{\mathcal{I}})}+\mathcal{S}(\mathcal{T}^{\mathcal{O}^{near}})+\mathcal{S}(\mathcal{T}^{\mathcal{O}^{far}})$.

A model trained with $\mathcal{D}$ is not only required to detect objects from
$\mathcal{S(T^{\mathcal{I}})}$ but it is also required to generalize well to
$\mathcal{S}(\mathcal{T}^{\mathcal{O}^{near}})$. Therefore, a OOD detection
module should not flag $\mathcal{S}(\mathcal{T}^{\mathcal{O}^{near}})$ as an
OOD, rather only classify the samples from
$\mathcal{S}(\mathcal{T}^{\mathcal{O}^{far}})$ as an OOD sample.

As shown in Fig. 2, the primary purpose of the Situation Monitor is to
distinguish between known and unknown situations. Leveraging the remarkable
generalization capability of deep learning models [17], the monitor adeptly
classifies Near-OOD situations as known situations. This classification is
grounded in the understanding that Near-OOD instances share significant
similarities with the In-Distribution (IN) dataset, aligning with the inherent
generalization prowess of deep learning models. Consequently, the Situation
Monitor is pivotal in identifying situations based on their familiarity with
the model’s learned context. By accurately discerning known and unknown
situations, the monitor empowers the model to make informed decisions in real-
world applications, even when encountering instances beyond its training data.
This enhances the model’s reliability and performance in diverse and dynamic
environments. Overall, the Situation Monitor plays a crucial role in ensuring
the effectiveness of deep learning models across various situations.

In summary, our goal is to train a transformer-based object detection model on
a training set $\mathcal{D}$ and without introducing the samples from
$\mathcal{O}$, the Situation Monitor (OOD detection module) of the model
should have ability to classify the samples from
$\mathcal{S}(\mathcal{T}^{\mathcal{O}^{far}})$ as an OOD situation and on
contrary should not flag the $\mathcal{S}(\mathcal{T}^{\mathcal{O}^{near}})$
samples as an OOD situation.

##  4 Our Approach

###  4.1 DBEA: Diversity based Budding Ensemble Architecture

The recently introduced Budding Ensemble Architecture (BEA) [18] represents a
sample-free methodology for detecting OOD instances, showcasing notable
performance advancements. Consequently, we employ the Budding Ensemble
Architecture approach in the design of our Situation Monitor.

Within the Budding Ensemble Architecture (BEA) framework, a unified backbone
and duplicated detectors replace the conventional ensemble setup. This
alteration enhances confidence score calibration, diminishes uncertainty
errors, and introduces an overlooked advantage: superior OOD detection
compared to other state-of-the-art sample-free methods. The Tandem loss
function ($\mathcal{L_{\mathbf{tandem}}}$) introduced in BEA, devised
initially for YOLOV3 [20] and SSD [16] was primarily tailored for anchor-based
object detection models. Therefore, the $\mathcal{L_{\mathbf{tandem}}}$
function to be applied to the transformer model requires a series of
modifications and adaptations to integrate with this novel ensemble approach
seamlessly.

Figure 4: Within the BEA model, tandem detectors undergo further refinement
with diversity loss, aiming to enhance the distinction between $\alpha$ and
$\beta$ detectors. This leads to the development of Diversity-based BEA.
Figure 5: Adapted architecture diagram from DINO-DETR [24] for DBEA-DINO-DETR,
illustrating the replication of final layers. For a comprehensive
understanding of DINO-DETR, please refer to Figure 2 in [24]. In BEA, it is
proposed to duplicate the final layers as $\alpha$ and $\beta$ to create
tandem detectors, which are subsequently utilized in the situation monitoring
model.

We base our Situation Monitor design on the ResNet-based DINO-DETR transformer
model. It typically comprises a CNN-based backbone (ResNet), self-attention
layer-based encoders, and decoders, followed by regression layers for
classification and bounding box detection. We leverage the insights from BEA
to tailor the DINO-DETR object detection model for our Situation Monitor.
Specifically, we introduce diversity-based tandem detection layers, as
depicted in Fig. LABEL:fig:abstract_arch_b. In the architectural integration
of BEA into DINO-DETR, the 3 layer feed-forward neural network layers (FFN)
forming the final regression layers immediately following the decoder are
duplicated, resulting in two detectors, $\mathbf{\alpha}$ and $\mathbf{\beta}$
collectively called as tandem detectors. During inference, confidence scores
and bounding boxes are computed from the mean of the tandem detectors.

Despite the duplication of layers, as indicated in the BEA paper [18], no
advantages were observed for the Situation Monitor during the training of the
DINO-DETR model. This lack of advantage stems from the tandem layers learning
similar representations. Therefore, the incorporation of the tandem loss
function, $\mathcal{L_{\mathbf{tandem}}}$, alongside the original base loss
function, $\mathcal{L_{\mbox{base}}}$, becomes crucial. The adaptation of BEA
with the diversity-based $\mathcal{L_{\mathbf{tandem}}}$ function is called
Diversity-based Budding Ensemble Architecture (DBEA). This serves a dual
purpose: creating an effective Situation Monitor and enhancing the calibration
of confidence scores.

The adapted tandem loss is now defined as follows:

$\displaystyle\mathfrak{L_{\mbox{ta}}(\phi})$
$\displaystyle=\mathbbm{1}_{\textit{i}}^{\mbox{obj}}\sqrt{\left(\phi^{\alpha}_{i}-\phi^{\beta}_{i}\right)^{2}},$
(1) $\displaystyle\mathcal{L_{\mbox{ta}}}$
$\displaystyle=\mathfrak{L_{\mbox{ta}}}(\bm{x})+\mathfrak{L_{\mbox{ta}}}(\bm{y})+\mathfrak{L_{\mbox{ta}}}(\bm{w})+\mathfrak{L_{\mbox{ta}}}(\bm{h})$

$\displaystyle\mathfrak{L_{\mbox{tq}}(\phi})$
$\displaystyle=\mathbbm{1}_{\textit{i}}^{\mbox{noobj}}\frac{1}{\sqrt{\left(\phi^{\alpha}_{i}-\phi^{\beta}_{i}\right)^{2}}},$
(2) $\displaystyle\mathcal{L_{\mbox{tq}}}$
$\displaystyle=\mathfrak{L_{\mbox{tq}}}(\bm{x})+\mathfrak{L_{\mbox{tq}}}(\bm{y})+\mathfrak{L_{\mbox{tq}}}(\bm{w})+\mathfrak{L_{\mbox{tq}}}(\bm{h})$

$\displaystyle\mathcal{L_{\mbox{tandem}}}$
$\displaystyle=\lambda_{ta}\mathcal{L_{\mbox{ta}}}+\lambda_{tq}\mathcal{L_{\mbox{tq}}}$
(3)

where $\mathbbm{1}_{\textit{i}}$ denotes whether the object prediction
overlaps with the ground-truth.The variables $\bm{x}$ and $\bm{y}$ signify the
predicted centre points of the bounding box, while $\bm{w}$ and $\bm{h}$
represent the predicted height and width of the object, respectively. The
$\mathcal{L_{\mathbf{tandem}}}$ operates on positive predictions and negative
predictions, which is possible due to access to ground truth during training.
The Tandem-Aiding loss function $\mathcal{L_{\mathbf{ta}}}$ diminishes the
errors associated with the positive predictions between $\mathbf{\alpha}$ and
$\mathbf{\beta}$ detector. Similarly, the Tandem-Quelling loss function
$\mathcal{L_{\mathbf{tq}}}$ amplifies the errors related to negative
predictions between $\mathbf{\alpha}$ and $\mathbf{\beta}$ detectors.
Therefore promoting agreement and disagreement concerning positive and
negative predictions.

Within the BEA paper, the $\mathcal{L_{\mbox{ta}}}$ and
$\mathcal{L_{\mbox{tq}}}$ loss functions are applied independently to specific
components of both classification and bounding box regression outputs. In the
context of the transformer model, the $\mathcal{L_{\mathbf{tandem}}}$ is
exclusively applied to the bounding box regression layers as shown in Eq. (1)
and (2). This selective application is adopted because introducing this loss
to the classification layer decreases the performance of the Situation
Monitor. To support $\mathcal{L_{\mbox{tandem}}}$ loss function for better
calibration of the confidence scores, we introduce diversity at classification
regression layer output between the tandem layers as shown in Fig. 4. To this
end, we use similarity score as a diversity measure to ensure that the tandem
detectors capture distinct representations from the decoders. Specifically, we
use the cosine similarity score to introduce the diversity as depicted in Eq.
(4).

$\displaystyle
Cosine\\_Similarity(\alpha,\beta)=\frac{\alpha.\beta}{||\alpha||.||\beta||};\alpha,\beta\in
Z$ (4)
$\displaystyle\mathcal{L_{\mbox{diversity}}}=\frac{1}{n}\sum_{i=1}^{n}Cosine\\_Similarity(\mathfrak{\alpha}_{i},\beta_{i})$

where $n$ represents total number of predictions and $Z$ represents the
classification logits denoted as $Z=(z_{1},z_{2},..z_{k})$, $z_{i}$ is the
unnormalized score for class $i$.

Incorporating diversity introduces a dynamic range of classification outputs
at the tandem layers. This diversity is instrumental in fostering a broader
spectrum of values, thereby providing the tandem loss function with a larger
space to operate upon. Specifically, the tandem loss function leverages this
diversity to mitigate errors associated with positive predictions effectively.
By encouraging divergence in classification values, the model can better
discern and refine its understanding of positive instances. Simultaneously,
this diversity amplifies errors in the context of negative predictions. To
this end, the diversity and tandem-based loss function is constructed by
incorporating both $\mathcal{L_{\mathbf{tandem}}}$ and
$\mathcal{L_{\mathbf{diversity}}}$ into the base loss function of the DINO-
DETR model, denoted as $\mathcal{L_{\mathbf{base}}}$. This integration is
illustrated in Equation (5).

$\displaystyle\mathcal{L_{\mbox{dbea}}}$
$\displaystyle=\mathcal{L_{\mbox{base}}}+\mathcal{L_{\mbox{tandem}}}+\lambda_{div}\mathcal{L_{\mbox{diversity}}}$
(5)

The training hyper-parameters of the DBEA-DINO-DETR remain unchanged except
the $\lambda$ parameters introduced in Eq. (3) and (5).

###  4.2 Situation Monitor

The Situation Monitor is a component of the BEA-DINO-DETR that serves as an
Out-of-Distribution (OOD) detection module. It identifies Far-OOD situations
by analyzing disparities in the predictions of tandem layer bounding boxes.
The transformer model undergoes end-to-end training through a zero-shot
approach, which, in this case, means an explicit OOD dataset is not shown
during the training process. This methodology distinguishes explicit
situations by highlighting the errors in variance between the tandem layer
predictions, specifically in Far-OOD situations. The
$\mathcal{L_{\mbox{dbea}}}$ loss function, incorporating cosine diversity
during training, compels tandem detection layers to consider objects from
diverse feature maps and perspectives.

The introduction of diversity loss ($\mathcal{L_{\mbox{diversity}}}$) during
training prompts a change in perspective, compelling tandem detection layers
to examine objects from diverse feature maps and viewpoints. Driven by the
$\mathcal{L_{\mbox{tandem}}}$ loss function, tandem layers generate distinct
predictions for bounding box centre points (Fig. 6). In instances where the
set $\mathcal{T}$ is a subset of $\mathcal{I}$ within the specific category
$\mathcal{O^{\textit{near}}}$, it is observed that the predicted widths and
heights tend to be closely aligned with no large variances. Similarly, we
should expect large variances when $\mathcal{O^{\textit{far}}}$ is
encountered. Therefore, to capture Far-OOD situations effectively at the image
level, a methodology is applied to heighten the variances of predicted centre
points ($\bm{x}$ and $\bm{y}$) and heights and widths ($\bm{w}$ and $\bm{h}$).
These bounding box-related attributes are referred to as $b$ in the following
equation.

Figure 6: Situation Monitor: The variance between $\alpha_{DBEA}$ and
$\beta_{DBEA}$ detector predictions is interpreted as the prediction
uncertainty $\mathcal{U}_{SM}$. A high uncertainty means Far-OOD, whereas low
uncertainty means Near-OOD or in-distribution.

$\displaystyle
xy_{var}=\sqrt{\sum_{i\in(x,y)}{(b^{\alpha}_{i}-b^{\beta}_{i}})^{2}},$ (6)
$\displaystyle xy_{centered\\_var}=xy_{var}*\mu(xy_{var}),$ $\displaystyle
wh_{var}=\sum_{i\in(w,h)}{(b^{\alpha}_{i}-b^{\beta}_{i}})^{2},$ $\displaystyle
wh_{centered\\_var}=wh_{var}*\mu(wh_{var}),$
$\displaystyle\mathcal{U}_{SM}=\mu(\sqrt{xy_{centered\\_var}}*wh_{centered\\_var})$

To heighten sensitivity to deviations and pinpoint errors at an image-specific
level, these variances undergo additional processing as shown in Eq. (6). This
involves centring variances by multiplication with their respective means. The
final OOD value, denoted as $\mathcal{U}_{SM}$, is computed as the mean of
these variances at the image level, as depicted in Eq. (6). This approach
effectively addresses global trends across the entire dataset while localizing
errors to individual images. By doing so, anomalies unique to each image are
identified, enhancing performance in capturing Far-OOD situations.

##  5 Evaluation Metrics for Situation Monitor

The impact of adding $\mathcal{L_{\mbox{tandem}}}$ and
$\mathcal{L_{\mbox{diversity}}}$ to the $\mathcal{L_{\mbox{base}}}$ loss
function is evaluated using mean average precision (mAP) metrics. The
calibration is evaluated using Pearson correlation (PCorr) between confidence
scores and the intersection over union of objects and the ground truths.
Pearson correlation is evaluated on two sets PCorr-all and PCorr-tp which is
correlation on all the predictions and separately on true positive samples.
The Situation Monitor in a DBEA model uses $\mathcal{U}_{SM}$ values and it is
assessed using four standard metrics to evaluate it’s performance in detecting
OOD situations. Whereas, the vanilla baseline model’s OOD detection is
evaluated using their confidence scores. In this study, we do not assess the
object detection performance (mAP/AP) on Near-OOD datasets. Instead, we simply
demonstrate that the Situation Monitor does not identify Near-OOD samples as
OOD.

  * •

AUROC calculates the area under the receiver operating characteristic curve, a
metric utilized to assess the performance of OOD detection. The OOD
($\mathcal{T}(\mathcal{T}^{\mathcal{O}^{far}})$) samples are considered as
positive samples. A higher AUROC value indicates superior performance.

  * •

AUPR(In/Out) is the area under the receiver operating characteristic curve and
is a key metric for evaluating OOD detection performance. It assesses how well
a model distinguishes positive instances, with
$\mathcal{T}(\mathcal{T}^{\mathcal{O}^{far}})$ samples considered as
positives. AUPR comprises of AUPR-In and AUPR-Out. It considers the in-
distribution ($\mathcal{T}(\mathcal{T}^{\mathcal{I}})$) samples as either
positive or negative respectively. A higher AUPR value indicates superior
model performance.

  * •

FPR@95 expressed as FPR (false positive rate) at a fixed TPR (true positive
rate) point represents the rate of falsely identified positive instances among
all negative instances when the true positive rate is held at a specific
percentage which in this case at 95%. The lower value indicates superior
performance.

  * •

DE@95 is the detection error at 95% TPR quantifies the detection error (miss-
classification probability) when TPR is set at 95%. A lower value indicates
superior performance.

##  6 Experiments

In this section, we conduct an ablation study to analyze the impact of each
component. We then compare our method with the previous state-of-the-art
sample-free OOD detection approach.

###  6.1 Experiment Setup

The Situation Monitor (SM) is integrated into the DBEA-DINO-DETR object
detection model, allowing end-to-end training without freezing the backbone or
any particular layer. This ensures tandem detectors learn based on the
$\mathcal{L_{\mbox{dbea}}}$ loss function. Evaluations are conducted on the
KITTI [10] and BDD100K [23] datasets, widely used computer vision datasets for
autonomous driving situations. These datasets are divided into training,
validation, and testing sets with ratios of 7.5:1:1.5 and 8.5:0.75:0.75,
respectively. Eight out of nine usable classes are evaluated for the KITTI
dataset, while all ten classes are considered for the BDD100K dataset.
Additionally, CoCo’s evaluation dataset is utilized. A consistent $416\times
416$ input image size is used for training across all models. For YoloV3 [20]
and SSD [16] models, SGD optimizer is employed with a learning rate of 0.001,
momentum of 0.95, and weight decay of 0.001, trained for 300 epochs with
batch-size of 28. DINO-DETR-based models utilize 900 queries, generating 900
predictions per image, with the top 100 predictions considered.
ResNet-50-based transformer models use an AdamW optimizer with a weight decay
of 0.0001 and a learning rate of 0.0002, trained for 50 epochs with a batch
size of 5. Both the baseline and our approach share identical training
hyperparameters, ensuring consistency.

Figure 7: Analysis of the impact of varying parameter
$\mathcal{L_{\mbox{div}}}$ in the ablation study.

###  6.2 Ablation Study on $\lambda_{div}$, $\lambda_{ta}$ and $\lambda_{tq}$
on KITTI trained DBEA-DINO-DETR

In this ablation study section, multiple DBEA-DINO-DETR models are trained on
the KITTI dataset wherein the parameters outlined in equations 5 and 3 are
varied. The primary objective is systematically showcasing and analysing each
component’s distinctive impact. In all ablation experiments, the parameter
$\lambda_{ta}$ is consistently maintained at either one when necessary or at
zero. This is because $\mathcal{L_{\mbox{ta}}}$ is designed to minimize errors
in positive predictions, and an increase in the $\lambda_{ta}$ factor beyond
$\lambda_{tq}$ adversely affects calibration and the fundamental object
detection capability of the model.

Figure 8: Analysis of the impact of varying parameter
$\mathcal{L_{\mbox{tandem}}}$ in the ablation study.

Model |  mAP (%) $\mathbf{\uparrow}$ | AP50 (%) $\mathbf{\uparrow}$ |  PCorr(all) (%) $\mathbf{\uparrow}$ |  PCorr(tp) (%) $\mathbf{\uparrow}$ |  | Out-of-distribution detection on COCO dataset
---
| AUROC
---
(%) $\mathbf{\uparrow}$
| AUPR (In/Out)
---
(%) $\mathbf{\uparrow}$
| FPR@95
---
(%) $\mathbf{\downarrow}$
| DE@95
---
(%) $\mathbf{\downarrow}$
KITTI trained |  |  |  |  |  |  |
Vanilla-YOLOv3 | 45.3 | 87.4 | 80.3 | 45.5 | 28.5 | 63.4/17.2 | 95.4 | 74.4
BEA-YOLOv3 [18] | 54.8 | 89.3 | 80.8 | 45.6 | 91.7 | 90.5/92.5 | 33.6 | 18.7
Vanilla-SSD | 62.6 | 88.6 | 76.2 | 56.2 | 35.7 | 44.3/37.6 | 96.5 | 55.3
BEA-SSD [18] | 63.1 | 89.6 | 74.5 | 54.4 | 91.6 | 92.6/90.7 | 57.6 | 30.6
Vanilla-DINO-DETR | 72.9 | 95.2 | 78.2 | 49.2 | 41.4 | 42.6/45.0 | 96.1 | 49.8
BEA-DINO-DETR | 73.8 | 95.8 | 79.1 | 52.0 | 92.4 | 93.1/94.4 | 15.1 | 9.6
DBEA-DINO-DETR (ours) | 74.6 | 95.8 | 79.6 | 54.2 | 98.3 | 98.5/98.3 | 10.3 | 7.6
BDD100K trained |  |  |  |  |  |  |
Vanilla-YOLOv3 | 25.0 | 53.7 | 69.3 | 38.7 | 22.0 | 27.4/43.6 | 98.7 | 40.9
BEA-YOLOv3 [18] | 27.6 | 58.1 | 69.0 | 36.0 | 96.5 | 97.5/91.8 | 12.3 | 8.5
Vanilla-DINO-DETR | 47.3 | 84.0 | 71.1 | 37.4 | 25.3 | 36.5/35.9 | 99.8 | 49.6
DBEA-DINO-DETR (ours) | 47.9 | 84.6 | 78.9 | 48.3 | 99.6 | 99.7/99.6 | 1.2 | 2.2

Table 1: Comparison of the performance of our sample-free method with that of
previous state-of-the-art sample-free method. Our method is trained on the
KITTI and BDD100K datasets, and the out-of-distribution detection evaluation
is conducted on the COCO dataset instances. Figure 9: Analysis of the impact
of varying parameter $\mathcal{L_{\mbox{tq}}}$ in the ablation study.

Fig. 7 illustrates the impact of introducing $\mathcal{L_{\mbox{diversity}}}$
with varied $\lambda_{div}$ parameters on the Situation Monitor performance
and its OOD detection. The radar plots presented in this study will include
the outcomes of a baseline model (the ”vanilla model”) and reference optimal
values against which the metrics will be compared. Without enabling diversity,
the performance on the Far-OOD CoCo dataset experiences a significant decline
compared to its enabled counterpart. The optimal configuration is identified
when the $\lambda_{div}$ parameter is set to $40$, resulting in elevated
Average Precision metrics, as well as improved AUROC and AUPR OOD metrics,
while concurrently maintaining lower FPR@95 and detection error (DE@95)
values. A subsequent increase of $\lambda_{div}$ from $40$ to $80$ brings
about enhancements across all metrics, with only a minor impact on detection
error and a slight decrease in FPR@95. Hence, choosing $40$ as the parameter
proves to be the most effective, achieving a balance across all metrics. The
selection of $\lambda_{div}$, incremented by a factor of 2, is influenced by
the consideration that both the classification and GIOU loss [2] components of
$\mathcal{L_{\mbox{base}}}$ are scaled by a factor of 2, making factors of 2
more effective to the effectiveness of $\lambda_{div}$.

Similarly, the influence of $\lambda_{tq}$ is illustrated in Fig. 8, with
$\lambda_{ta}$ and $\lambda_{div}$ held at 1 and 40, respectively. This
specific radar plot highlights that setting $\lambda_{tq}$ to 10 produces the
most well-balanced results compared to other factors. While the transformer
model exhibits commendable calibration when trained with a $\lambda_{tq}$
factor of 100, there is a slight decline in the Situation Monitor OOD
performance.

Fig. 9 illustrates the results of comprehensive ablation experiments
encompassing all three factors: $\lambda_{div}$, $\lambda_{ta}$, and
$\lambda_{tq}$. In the absence of diversity and tandem loss, the model
exhibits comparable Average Precision (AP) performance but experiences a
notable decline in the FPR@95 and DE@95 metrics. Notably, when the model is
trained with specific parameters, namely $\lambda_{div}=40$, $\lambda_{ta}=1$,
and $\lambda_{tq}=10$, it achieves optimal performance, manifesting in
heightened accuracy and superior OOD detection capabilities. The
configuration, which results in lower FPR@95 and DE@95 metrics, indicates
enhanced OOD detection performance. The improved threshold associated with
these parameter values contributes to superior classification, leading to
better OOD detection.

###  6.3 Benchmark Results

Tab. 1 comprehensively compares baseline models (CNN and transformer-based)
with state-of-the-art sample-free OOD detection methods. Using various metrics
discussed in Section 5, we assess the Situation Monitor’s overall object
detection and OOD capabilities integrated into the DBEA-DINO-DETR model.
Yolov3, SSD and DINO baseline models are trained using original
hyperparameters, and the reported results in Tab. 1 reflect their peak
performance. Applying our approach to the DINO-DETR model resulted in the
DBEA-DINO-DETR model. Training it on datasets KITTI and BDD100K showcases
enhanced detection accuracy and improved correlation between predicted
confidence scores and intersection over union. On the BDD100K dataset, our
DBEA-based DINO-DETR model demonstrates improved correlation in overall and
true positive predictions. Additionally, there is a substantial enhancement in
the Out-of-Distribution (OOD) performance for detecting CoCo images.
Specifically, the OOD detection performance for CoCo images is higher when the
model is trained with BDD100K than the KITTI model. This highlights our
approach’s effectiveness in advancing object detection and OOD performance,
especially on larger datasets. Illustrated in Fig. LABEL:fig:AUROC_hist_a, the
Situation Monitor adeptly identifies $\mathcal{O^{\textit{far}}}$ OOD samples.
Furthermore, the Situation Monitor demonstrates strong generalization to
$\mathcal{O^{\textit{near}}}$ datasets, evident in overlapping histograms of
$\mathcal{U}_{SM}$ values Fig. LABEL:fig:AUROC_hist_b, LABEL:fig:AUROC_hist_c
and LABEL:fig:AUROC_hist_d. In contrast to the Situation Monitor introduced in
this study, the OOD values of BEA-based CNN models [18] tended to miss-
classify even $\mathcal{O^{\textit{near}}}$ as $\mathcal{O^{\textit{far}}}$ in
their OOD detection.

###  6.4 Overhead Analysis

Given the computational intensity of transformers relative to the CNNs, the
DBEA adaptation of transformers incurs additional costs due to the duplication
of final regression layers. To mitigate this overhead, we limit the feed-
forward channels in both the encoder and decoder layers (N) of DBEA-DINO-DETR
from 2048 to 1024, offsetting the overhead of duplicating the final regression
layers to create the tandem layers. The Vanilla model trained on the KITTI
dataset with N=2048 has a size of 48M, while our DBEA-DINO-DETR is only 42M,
representing a 14% reduction compared to the Vanilla model. We also observed
that there is little but negligible advantage in maintaining the same number
of feed-forward channels of 2048 as the vanilla model. The results of the
Situation Monitor presented in this section are based on the DBEA model
trained with 1024 feed-forward channels.

##  7 Conclusion

In summary, this paper introduces a Situation Monitor driven by zero-shot
learning, which integrates a novel Diversity-based Budding Ensemble
Architecture (DBEA) loss function. The incorporation of the DBEA loss function
empowers the object detection model to not only identify Far-OOD samples but
also to generalize over similar Near-OOD instances effectively. This prevents
miss-classification as OOD, enhancing the model’s adaptability and robustness.
Through an extensive ablation study with the parameters of DBEA’s loss
functions, we show significant improvement in detection accuracy and superior
OOD detection outcomes compared to baseline models and other existing methods.
Additionally, our research demonstrates the scalability of the DBEA-based
model, validated through successful training with KITTI and BDD100K datasets.
The Situation Monitor is suitable for safety-critical applications, being 14%
less computationally intensive than the baseline DINO-DETR model.

## Acknowledgements

This work was partially funded by the Federal Ministry for Economic Affairs
and Energy of Germany as part of the research project SafeWahr (Grant Number:
19A21026C).

## References

  * Ahuja et al. [2019] Nilesh A Ahuja, Ibrahima Ndiour, Trushant Kalyanpur, and Omesh Tickoo.  Probabilistic modeling of deep features for out-of-distribution and adversarial detection.  _arXiv preprint arXiv:1909.11786_ , 2019.
  * Araujo et al. [2019] Alexandre B. Araujo, Allan G. Oliveira, and Claudia R. Jung.  Giou loss for object detection.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , pages 4894–4902, 2019.
  * Berger et al. [2021] Christoph Berger, Magdalini Paschali, Ben Glocker, and Konstantinos Kamnitsas.  Confidence-based out-of-distribution detection: a comparative study and analysis.  In _Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Perinatal Imaging, Placental and Preterm Image Analysis: 3rd International Workshop, UNSURE 2021, and 6th International Workshop, PIPPI 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, October 1, 2021, Proceedings 3_ , pages 122–132. Springer, 2021.
  * Chen et al. [2020] Xingyu Chen, Xuguang Lan, Fuchun Sun, and Nanning Zheng.  A boundary based out-of-distribution classifier for generalized zero-shot learning.  In _European conference on computer vision_ , pages 572–588. Springer, 2020.
  * Choi et al. [2019] Jiwoong Choi, Dayoung Chun, Hyun Kim, and Hyuk-Jae Lee.  Gaussian yolov3: An accurate and fast object detector using localization uncertainty for autonomous driving.  In _Proceedings of the IEEE/CVF International conference on computer vision_ , pages 502–511, 2019.
  * Cordts et al. [2016] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3213–3223, 2016.
  * Esmaeilpour et al. [2022] Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu.  Zero-shot out-of-distribution detection based on the pre-trained model clip.  In _Proceedings of the AAAI conference on artificial intelligence_ , pages 6568–6576, 2022.
  * Feng et al. [2021] Di Feng, Ali Harakeh, Steven L Waslander, and Klaus Dietmayer.  A review and comparative study on probabilistic object detection in autonomous driving.  _IEEE Transactions on Intelligent Transportation Systems_ , 23(8):9961–9980, 2021.
  * Fort et al. [2021] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan.  Exploring the limits of out-of-distribution detection.  _Advances in Neural Information Processing Systems_ , 34:7068–7081, 2021.
  * Geiger et al. [2012] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In _2012 IEEE conference on computer vision and pattern recognition_ , pages 3354–3361. IEEE, 2012.
  * Kesten et al. [2019] R. Kesten, M. Usman, J. Houston, T. Pandya, K. Nadhamuni, A. Ferreira, M. Yuan, B. Low, A. Jain, P. Ondruska, S. Omari, S. Shah, A. Kulkarni, A. Kazakova, C. Tao, L. Platinsky, W. Jiang, and V. Shet.  Woven planet perception dataset 2020.  https://woven.toyota/en/perception-dataset, 2019.
  * Kornblith et al. [2019] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton.  Similarity of neural network representations revisited.  In _International conference on machine learning_ , pages 3519–3529. PMLR, 2019.
  * Kuleshov et al. [2018] Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon.  Accurate uncertainties for deep learning using calibrated regression.  In _International conference on machine learning_ , pages 2796–2804. PMLR, 2018.
  * Liang et al. [2017] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant.  Enhancing the reliability of out-of-distribution image detection in neural networks.  _arXiv preprint arXiv:1706.02690_ , 2017.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Liu et al. [2016] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14_ , pages 21–37. Springer, 2016.
  * Pinto et al. [2022] Francesco Pinto, Philip HS Torr, and Puneet K. Dokania.  An impartial take to the cnn vs transformer robustness contest.  In _European Conference on Computer Vision_ , pages 466–480. Springer, 2022.
  * Qutub et al. [2023] Syed Sha Qutub, Neslihan Kose, Rafael Rosales, Michael Paulitsch, Korbinian Hagn, Florian Geissler, Yang Peng, Gereon Hinz, and Alois Knoll.  Bea: Revisiting anchor-based object detection dnn using budding ensemble architecture.  2023\.
  * Raghu et al. [2021] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy.  Do vision transformers see like convolutional neural networks?  _Advances in Neural Information Processing Systems_ , 34:12116–12128, 2021.
  * Redmon and Farhadi [2018] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  _arXiv preprint arXiv:1804.02767_ , 2018.
  * Wilson et al. [2023] Samuel Wilson, Tobias Fischer, Feras Dayoub, Dimity Miller, and Niko Sünderhauf.  Safe: Sensitivity-aware features for out-of-distribution object detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 23565–23576, 2023.
  * Yang et al. [2021] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu.  Generalized out-of-distribution detection: A survey.  _arXiv preprint arXiv:2110.11334_ , 2021.
  * Yu et al. [2020] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell.  Bdd100k: A diverse driving dataset for heterogeneous multitask learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2636–2645, 2020.
  * Zhang et al. [2022] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum.  Dino: Detr with improved denoising anchor boxes for end-to-end object detection.  _arXiv preprint arXiv:2203.03605_ , 2022.
  * Zhang et al. [2017] Shanshan Zhang, Rodrigo Benenson, and Bernt Schiele.  Citypersons: A diverse dataset for pedestrian detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3213–3221, 2017.