#  $\pi$ phase ambiguity of cosmic birefringence

Fumihiro Naokawa  Research Center for the Early Universe, The University of
Tokyo, Bunkyo-ku, Tokyo 113-0033, Japan  Department of Physics, Graduate
School of Science, The University of Tokyo, Bunkyo-ku, Tokyo 113-0033, Japan
Toshiya Namikawa  Center for Data-Driven Discovery, Kavli IPMU (WPI), UTIAS,
The University of Tokyo, Kashiwa 277-8583, Japan  Kai Murai  Department of
Physics, Tohoku University, Sendai, Miyagi 980-8578, Japan  Ippei Obata  Kavli
IPMU (WPI), UTIAS, The University of Tokyo, Kashiwa 277-8583, Japan  Kohei
Kamada  School of Fundamental Physics and Mathematical Sciences, Hangzhou
Institute for Advanced Study, University of Chinese Academy of Sciences (HIAS-
UCAS), Hangzhou 310024, China  International Centre for Theoretical Physics
Asia-Pacific (ICTP-AP), Beijing/Hangzhou, China  Research Center for the Early
Universe, The University of Tokyo, Bunkyo-ku, Tokyo 113-0033, Japan

###### Abstract

We point out that the rotation angle $\beta$ of cosmic birefringence, which is
a recently reported parity-violating signal in the cosmic microwave background
(CMB), has a phase ambiguity of $n\pi\,(n\in\mathbb{Z})$. This ambiguity has a
significant impact on the interpretation of the origin of cosmic
birefringence. Assuming an axion-like particle as the origin of cosmic
birefringence, this ambiguity can be partly broken by the anisotropic cosmic
birefringence and the shape of the CMB angular power spectra. We also discuss
constraints on $\beta$ from existing experimental results.

††preprint: RESCEU-11/24††preprint: TU-1232

Introduction.— Recent analyses of cosmic microwave background (CMB)
polarization data have found a tantalizing hint of cosmic birefringence [1]—a
phenomenon where light traveling through space experiences a rotation of its
polarization plane—with a statistical significance of $3.6\,\sigma$ [2, 3, 4,
5, 6]. This parity-violating signature is a smoking gun of new physics beyond
the $\Lambda$ Cold Dark Matter Model and the Standard Model of particle
physics [7].

Cosmic birefringence can be attributed to the presence of a pseudoscalar field
of “axion-like” particle (ALP). In general, an ALP can have the Chern-Simons
coupling with the photon as $\mathcal{L}\supset-g\phi
F_{\mu\nu}\tilde{F}^{\mu\nu}/4$, where $g$ is a coupling constant, $\phi$ is
the axion-like field, $F_{\mu\nu}$ is the electromagnetic tensor, and
$\tilde{F}^{\mu\nu}$ is its dual. Through this coupling, the ALP can cause the
rotation of the polarization plane. In particular, the time evolution of the
homogeneous axion-like field, $\bar{\phi}(t)$, induces isotropic cosmic
birefringence. The total rotation angle is given by

$\beta=\frac{g}{2}\int^{t_{0}}_{t_{*}}\mathrm{d}t\,\frac{\mathrm{d}\bar{\phi}(t)}{\mathrm{d}t}=\frac{g}{2}\left[\bar{\phi}(t_{0})-\bar{\phi}(t_{*})\right]\equiv\frac{g}{2}\Delta\bar{\phi},$
(1)

where $t_{0}$ is the present time, and $t_{*}$ is the last scattering time of
CMB photons. Therefore, measuring $\beta$ with high precision results in
precisely determining the model parameters and dynamics of the ALP [8, 9, 10,
11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26].

In the cosmic birefringence analyses, we have estimated $\beta$ based on the
relationship between the CMB angular power spectra in the presence of cosmic
birefringence and those without cosmic birefringence as we describe below (see
also [27, 28]). To derive the relationship, we first express an observed CMB
polarization in the presence of cosmic birefringence as

$P^{o}(\hat{\bm{n}})=e^{2{i\mkern 1.0mu}\beta}P(\hat{\bm{n}})\,,$ (2)

where $P\equiv Q+iU$ with $Q$ and $U$ denoting the Stokes $Q$ and $U$
parameters of linear polarization, respectively, $\hat{\bm{n}}$ is a line-of-
sight unit vector, and $\beta$ is a rotation angle of the polarization plane.
The superscript $o$ denotes the observed quantities at present, and $P$ on the
right-hand side is the intrinsic polarization. In CMB analysis, we decompose
CMB polarization into the parity-even $E$-modes and parity-odd $B$-modes as
[29, 30]

$E_{lm}\pm{i\mkern 1.0mu}B_{lm}=-\int\\!\\!\,{\rm d}^{2}\hat{\bm{n}}\,\,_{\pm
2}Y^{*}_{lm}(\hat{\bm{n}})[Q\pm{i\mkern 1.0mu}U](\hat{\bm{n}})\,,$ (3)

where ${}_{\pm 2}Y_{lm}(\hat{\bm{n}})$ is the spin-2 spherical harmonics. In
the presence of isotropic cosmic birefringence, they are modified as

$\displaystyle E^{o}_{lm}$
$\displaystyle=E_{lm}\cos(2\beta)-B_{lm}\sin(2\beta)\,,$ (4) $\displaystyle
B^{o}_{lm}$ $\displaystyle=E_{lm}\sin(2\beta)+B_{lm}\cos(2\beta)\,,$ (5)

while the coefficient of a spherical harmonics expansion of temperature
fluctuations,$~{}\Theta_{lm}=\int\\!\\!\,{\rm
d}^{2}\hat{\bm{n}}\,\,Y^{*}_{lm}(\hat{\bm{n}})\Theta(\hat{\bm{n}})~{}$, is not
modified by the rotation. The observed angular power spectra relevant to
cosmic birefringence analysis are given by

$\displaystyle C_{l}^{\Theta E,o}$ $\displaystyle=C_{l}^{\Theta
E}\cos(2\beta)\,,$ (6) $\displaystyle C_{l}^{\Theta B,o}$
$\displaystyle=C_{l}^{\Theta E}\sin(2\beta)\,,$ (7) $\displaystyle
C_{l}^{EE,o}$
$\displaystyle=C_{l}^{EE}\cos^{2}(2\beta)+C_{l}^{BB}\sin^{2}(2\beta)\,,$ (8)
$\displaystyle C_{l}^{BB,o}$
$\displaystyle=C_{l}^{EE}\sin^{2}(2\beta)+C_{l}^{BB}\cos^{2}(2\beta)\,,$ (9)
$\displaystyle C_{l}^{EB,o}$
$\displaystyle=\frac{1}{2}(C_{l}^{EE}-C_{l}^{BB})\sin(4\beta)\,.$ (10)

Here we have assumed no intrinsic $EB$ and $\Theta B$ correlations. From the
last three equations, we obtain the relationship between $\beta$ and the
observed CMB polarization spectra as [31]

$\tan(4\beta)=\frac{2C_{l}^{EB,o}}{C_{l}^{EE,o}-C_{l}^{BB,o}}.$ (11)

Therefore, we can estimate $\beta$ from the above equation, and previous works
have reported $\beta\sim 0.3$ deg [2, 3, 4, 5, 6].

One important effect that has been overlooked in previous work is the
possibility of multiple rotations. All observed power spectra in Eqs. (6)–(10)
are invariant under $\beta\to\beta\pm\pi$, and the general solution of $\beta$
is given by

$\beta=\frac{1}{4}\arctan\left(\frac{2C_{l}^{EB,o}}{C_{l}^{EE,o}-C_{l}^{BB,o}}\right)+n\pi~{}~{}~{}(n\in\mathbb{Z}).$
(12)

The integer parameter $n$ is a convenient index for the number of rotations.
This $n\pi$ phase ambiguity reflects that Stokes parameters are spin-2
quantities. Eq. (12) is valid regardless of the origin of cosmic birefringence
as long as all CMB photons experience a uniform birefringence angle $\beta$.

In this work, we show how to constrain $n$ from CMB observables. If the
rotation angle is constant in time, we cannot constrain $n$ from the CMB
angular power spectrum alone. From a phenomenological point of view, however,
depending on a certain ALP model as the origin of cosmic birefringence, we can
limit the allowed range of $\beta$. From Eq. (1), larger $|\beta|$ means
larger $g|\Delta\bar{\phi}|$, while upper limits of $|\Delta\bar{\phi}|$ (more
precisely, the density parameter of ALP, $\Omega_{\phi}$) and $g$ are set [8]
by the cosmological [32, 33] and astrophysical observations [34],
respectively. Assuming a mass potential model $V(\phi)=m_{\phi}^{2}\phi^{2}/2$
for the ALP, for example, the upper limit is given by $|\beta|\lesssim
10^{7}~{}\mathrm{deg}~{}(|n|\lesssim 10^{5})$ from the condition to explain
$\beta\sim 0.3$ obtained in Ref. [8]. In the following, we show that CMB
observations can also tightly constrain $n$.

Constraint from anisotropic cosmic birefringence.— The rotation angle of
cosmic birefringence has anisotropic component,
$\beta(\hat{\bm{n}})=\beta+\alpha(\hat{\bm{n}})$, if the axion-like field has
fluctuations:

$\displaystyle\alpha(\hat{\bm{n}})\equiv-\frac{g\delta\phi(\chi_{*}\hat{\bm{n}},\eta_{*})}{2}=-\frac{g\bar{\phi}_{\rm
ini}}{2}\frac{\delta\phi(\chi_{*}\hat{\bm{n}},\eta_{*})}{\bar{\phi}_{\rm
ini}}\,,$ (13)

where we have split the ALP as
$\phi({\bm{x}},\eta)=\bar{\phi}(t)+\delta\phi({\bm{x}},\eta)$, $\chi_{*}$ is
the comoving distance to the last scattering surface,
$\bar{\phi}_{\mathrm{ini}}=\bar{\phi}(t_{*})$, and
$\eta_{*}=\eta_{0}-\chi_{*}$ is the conformal time at the last scattering with
$\eta_{0}$ being the present conformal time.

As $n$ increases, $g\bar{\phi}_{\rm ini}/2$ increases, and so does the
amplitude of the anisotropic birefringence. On the other hand, multiple CMB
observations have placed upper bounds on the amplitude of the anisotropic
cosmic birefringence. This upper limit gives a constraint on $n$.

The fluctuations of the axion-like field inevitably acquire non-zero values
from the metric perturbations, if the background field exists [35, 36, 37].
For a background solution of $\bar{\phi}$ determined by $m_{\phi}$ and
$\bar{\phi}_{\mathrm{ini}}$, we solve the evolution equations of the perturbed
pseudoscalar fields normalized as
$\delta\varphi\equiv\delta\phi/\bar{\phi}_{\rm ini}$ [35, 36]:

$\displaystyle\delta\varphi^{\prime\prime}$
$\displaystyle+2\mathcal{H}\delta\varphi^{\prime}+\left(k^{2}+a^{2}m_{\phi}^{2}\right)\delta\varphi$
$\displaystyle=\bar{\varphi}^{\prime}(3\Phi^{\prime}+\Psi^{\prime})-2a^{2}m_{\phi}\bar{\varphi}\Psi\,,$
(14)

where ′ denotes the time derivative with respect to the conformal time $\eta$,
$a$ is the scale factor, $k$ is the comoving Fourier wavenumber of the
perturbations, $\mathcal{H}=a^{\prime}/a$,
$\bar{\varphi}=\bar{\phi}/\bar{\phi}_{\rm ini}$, and $\Phi$ and $\Psi$ are the
gravitational potential in the conformal Newtonian gauge (see Ref. [35] for
their definitions) and are obtained from CAMB [38]. The adiabatic initial
condition is given by

$\displaystyle\delta\varphi$
$\displaystyle=\frac{1}{2}\frac{\bar{\varphi}^{\prime}}{\mathcal{H}}\Psi\,,$
(15) $\displaystyle\delta\varphi^{\prime}$
$\displaystyle=-\frac{\bar{\varphi}^{\prime}}{2}\Psi-\frac{1}{2}\frac{a^{2}m_{\phi}\bar{\varphi}}{\mathcal{H}}\Psi\,.$
(16)

We define the transfer function as

$\displaystyle\delta\varphi(k,\eta)\equiv T_{\delta\varphi}(k,\eta){\cal
R}_{\rm ini}(k)\,,$ (17)

where $\mathcal{R}_{\rm ini}$ is the primordial comoving curvature
perturbation, which determines $\Phi$ and $\Psi$. The angular power spectrum
of the anisotropic birefringence from recombination is given by [37, 39]:

$\displaystyle C_{L}^{\alpha\alpha}=\left(\frac{g\bar{\phi}_{\rm
ini}}{2}\right)^{2}4\pi\int\\!\\!\,{\rm d}(\ln k)\,\,\mathcal{P}_{\cal
R}(k)[j_{L}(k\chi_{*})T_{\delta\varphi}(k,\eta_{*})]^{2}\,,$ (18)

with $j_{L}(k\chi)$ being the $L$-th spherical Bessel function and ${\cal
P}_{\cal R}(k)=A_{\rm s}(k/k_{*})^{n_{\rm s}-1}$, where $A_{\rm s}$ is the
primordial scalar amplitude, $n_{s}$ is the spectral index, and the pivot
scale is chosen as $k_{*}=0.05$ Mpc-1.

CMB experiments have placed upper bounds on the anisotropic cosmic
birefringence from the recombination, assuming that $\delta\phi$ is
approximately constant during the recombination [40, 41, 42, 43]. We thus
restrict the mass range to $m_{\phi}\lesssim 10^{-28}\,$eV. The transfer
function is obtained numerically by solving Eq. (14). We compute the angular
power spectrum with the Planck PR3 best-fit values of TT,TE,EE+lowE+lensing in
Table 2 of Ref. [33].

Figure 1: Angular power spectrum of anisotropic cosmic birefringence from the
ALP (mass potential model) with varying $m_{\phi}$. We assume
$g\bar{\phi}_{\rm ini}/2=0.34$ deg and $n=0$. The shaded regions are excluded
by the measurements of $C_{L}^{\alpha\alpha}$ by Planck (green) [42], ACTPol
(blue) [40], and SPTpol (red) [41].

Figure 1 shows the angular power spectrum of the anisotropic cosmic
birefringence with $g\bar{\phi}_{\rm ini}/2=0.34$ deg and $n=0$. If $n\geq 1$,
the power spectrum approximately scales as
$C_{L,n}^{\alpha\alpha}/C_{L,n=0}^{\alpha\alpha}\simeq((g\bar{\phi}_{\mathrm{ini}})_{n}/(g\bar{\phi}_{\mathrm{ini}})_{n=0})^{2}\sim(180n\,{\rm
deg}/0.34\,{\rm deg})^{2}\sim 10^{5}n^{2}$. We also show the latest bounds on
the amplitude of the scale-invariant anisotropic birefringence power spectrum,
$L(L+1)C_{L}^{\alpha\alpha}/2\pi\lesssim 10^{-3}$ deg2 at $2\,\sigma$. Using
the scaling relation, and the bounds at $L\lesssim 10^{2}$, we find that
$n\lesssim 10$ ($m_{\phi}=10^{-28}$ eV), $n\lesssim 10^{3}$
($m_{\phi}=10^{-29}$ eV), $n\lesssim 10^{5}$ ($m_{\phi}=10^{-30}$ eV), and
$n\lesssim 10^{7}$ ($m_{\phi}=10^{-31}$ eV).

Constraint from power spectrum tomography.— When the constant rotation
approximation is invalid, we can no longer use Eq. (10) to calculate
$C_{l}^{EB}$. In that case, we have to solve the Boltzmann equation of CMB
polarization including the cosmic birefringence effect [44, 45, 46, 47]:

${}_{\pm 2}\Delta^{\prime}_{P}+{i\mkern 1.0mu}q\mu{}_{\pm 2}\Delta_{P}$
$\displaystyle\qquad=an_{\rm e}\sigma_{T}\left[-{}_{\pm
2}\Delta_{P}+\sqrt{\frac{6\pi}{5}}{}_{\pm 2}Y_{20}(\mu)\Pi(\eta,q)\right]$
$\displaystyle\qquad\qquad\pm{i\mkern 1.0mu}g\phi^{\prime}{}_{\pm
2}\Delta_{P}\,,$ (19)

where $n_{e}$ is the electron number density, $\sigma_{T}$ is the Thomson
scattering cross section, and $\Pi$ is the polarization source term [29]. The
last term represents the effect of birefringence. The equation of motion of
the homogeneous axion-like field $\bar{\phi}$ is given by

$\bar{\phi}^{\prime\prime}+2\frac{a^{\prime}}{a}\bar{\phi}^{\prime}+a^{2}m_{\phi}^{2}\bar{\phi}=0\,.$
(20)

As shown in Fig. 2, the ALP dynamics through cosmic history depends on
$m_{\phi}$. After solving Eqs. ($\pi$ phase ambiguity of cosmic birefringence)
and (20), we perform a spin-2 spherical harmonics expansion on ${}_{\pm
2}\Delta_{P}$ as

${}_{\pm 2}\Delta_{P}(\eta_{0},q,\mu)$ $\displaystyle\equiv-\sum_{l}{i\mkern
1.0mu}^{-l}\sqrt{4\pi(2l+1)}$
$\displaystyle\quad\times[\Delta_{E,l}(q)\pm{i\mkern
1.0mu}\Delta_{B,l}(q)]{}_{\pm 2}Y_{l0}(\mu)\,.$ (21)

From these coefficients, we obtain the angular power spectrum as

$C_{l}^{XY}=4\pi\int\\!\\!\,{\rm d}(\ln q)\,\,\mathcal{P}_{\cal
R}(q)\Delta_{X,l}(q)\Delta_{Y,l}(q)\,,$ (22)

where $X$ and $Y$ denote $E$-mode or $B$-mode. We obtain the angular power
spectra using a Boltzmann solver, birefCLASS [15, 48, 49].

Reference [15] revealed that the shape of $C_{l}^{EB}$ changes depending on
the ALP dynamics. When $\bar{\phi}(t)$ substantially evolves during the
recombination epoch (like $m_{\phi}=10^{-28}~{}\mathrm{eV}$ in Fig. 2), the
peaks of $C_{l}^{EB}$ at high-$l$ are shifted. This is because the
birefringence angle of each photon varies depending on when it was released.

When $n\neq 0$, this effect is amplified because $\beta$ is larger by $n\pi$
for the same amount of $\Delta\bar{\phi}$. In Fig. 3, we show $C_{l}^{EB}$ for
$m_{\phi}=10^{-29}\,\mathrm{eV}$ and various $n$. As expected, the peak
positions of $C_{l}^{EB}$ are shifted when $n\not=0$. On the other hand, we
may adjust the overall amplitude by tuning $\beta$ marginally. In the case of
$m_{\phi}=10^{-30}\,\mathrm{eV}$, such effect is weaker, and for
$m_{\phi}=10^{-31}\,\mathrm{eV}$ shown in Fig. 4, we can no longer find any
changes of $C_{l}^{EB}$ other than the overall amplitude. Therefore, for
$m_{\phi}=10^{-31}\,\mathrm{eV}$, all $n$ in the figure are not excluded by
the shift of the peaks of $C_{l}^{EB}$.

However, even for such light mass $C_{l}^{EB}$ at low-$l$ has an interesting
behavior. Figure 5 shows the enlarged view of Fig. 4, focusing on low-$l$
($l\lesssim 30$). The amplitudes of $C_{l}^{EB}$ are approximately zero for
all $n$ except for $1\leq n\lesssim 4$. This is interpreted as follows. For
these $n$, the oscillation of $\bar{\phi}$ during reionization is enhanced,
resulting in the increased amplitude of $C_{l}^{EB}$ at $l\sim 3$. When
$n\gtrsim 10$, however, the corresponding birefringence angle becomes much
larger than $\pi$, and $C_{l}^{EB}$ is washed out due to the superposition of
randomly-oriented photons’ polarizations emitted at the reionization epoch.

The washout effect also appears in $C_{l}^{EE}$. Figure 6 shows $C_{l}^{EE}$
for $m_{\phi}=10^{-31}~{}\mathrm{eV}$, focusing on the reionization bump at
$l\lesssim 10$ [50] with the error bars from the Planck observation [51]. We
find that the reionization bump is suppressed with increasing $n$ and almost
vanishes for $n\gtrsim 10$. This behavior is also due to the washout effect by
the amplified birefringence during reionization.

$C_{l}^{EE}$ without the reionization bump are not excluded significantly with
Planck [51], but future full-sky observations may determine its presence or
absence (also for $C_{l}^{EB}$). In other words, using the above low-$l$
features, we can explore the value of $n$ for relatively light ALP mass. Here,
we only discussed the case of $m_{\phi}=10^{-31}~{}\mathrm{eV}$, but we also
find a similar trend at low-$l$ for $m_{\phi}=10^{-30}~{}\mathrm{eV}$ and
$10^{-32}~{}\mathrm{eV}$.

Figure 2: ALP dynamics for
$10^{-32}~{}\mathrm{eV}<m_{\phi}<10^{-28}~{}\mathrm{eV}$. Axion-like field
changes around recombination for $m_{\phi}\sim 10^{-28}~{}\mathrm{eV}$ and
reionization for $m_{\phi}\sim 10^{-31}~{}\mathrm{eV}$ Figure 3: EB power
spectrum for $m_{\phi}=10^{-29}~{}\mathrm{eV}$ for $n=0$ (blue), $n=2$
(orange), $n=8$ (green), and $n=32$ (red). Figure 4: Same as Fig. 3 but for
$m_{\phi}=10^{-31}~{}\mathrm{eV}$ Figure 5: Low-$l$ feature of EB power
spectrum for $m_{\phi}=10^{-31}~{}\mathrm{eV}$ with varying $n$. While for
$n=0$, there is not a reionization bump, there are bumps for some small
nonzero $n$. Figure 6: Low-$l$ feature of EE power spectrum for
$m_{\phi}=10^{-31}~{}\mathrm{eV}$. The purple points with error bars are the
latest results ($l\leq 10$) by the final Planck Data Release [51].

Summary and Discussion.— We reexamined the possible rotation angle $\beta$ of
cosmic birefringence including the phase ambiguity, which has been overlooked
in previous studies. Due to this ambiguity, $\beta$ is not determined up to an
integer parameter $n$ (Eq. (12)). Assuming that cosmic birefringence is
induced by an axion-like field with a mass potential
$V(\phi)=m_{\phi}^{2}\phi^{2}/2$, we explored the ways to constrain $n$. We
found two major constraints on $n$: one from anisotropic cosmic birefringence
and the other from power spectrum tomography. The former is more sensitive to
heavier ALP like $m_{\phi}\sim 10^{-28}~{}\mathrm{eV}$, while the latter
limits $n$ for $m_{\phi}\sim\mathcal{O}(10^{-31})$ eV using the reionization
bump of $C_{l}^{EE}$ as well as $C_{l}^{EB}$. We conclude that there are a
large number of overlooked solutions of $\beta$ for recent reports of cosmic
birefringence, and we can test the possibility of $n\neq 0$ by the near future
observations like Simons Observatory [52], LiteBIRD [53], and CMB-S4 [54].

The calculation of cosmic birefringence employs an approximation,
$|g_{\phi}\dot{\bar{\phi}}\lambda_{\rm CMB}|\ll 1$, where $\lambda_{\rm CMB}$
is the wave length of the CMB photon. In CMB experiments, we usually use
frequency bands around $100\,$GHz corresponding to $\lambda_{\rm CMB}\sim
1\,$mm. This paper assumes ALPs whose field values change in a cosmological
time scale, i.e.,
$g_{\phi}\dot{\bar{\phi}}\sim\Delta(g_{\phi}\bar{\phi})/\Delta\eta$ with
$\Delta\eta\gtrsim 1\,$Mpc $\sim 10^{25}\,$mm. Our analysis shows that
$\Delta(g_{\phi}\bar{\phi})\lesssim 10^{10}$. Thus, the approximation still
holds even if we allow the maximum value of $g_{\phi}\bar{\phi}_{\rm ini}$
obtained in this paper.

We have focused on ALP-induced cosmic birefringence with the mass potential
model. Similar features in the power spectra will also appear in other models
of cosmic birefringence, including the early dark energy case [20]. Thus, when
interpreting the results from Ref. [23], we need to account for cases where
$n\not=0$. In general, a tiny polarization rotation angle that produces
negligible modifications to the power spectra is enhanced by increasing $n$,
and the power spectra could be significantly modified for a nonzero $n$.

The constraint on $n$ degenerates with the CMB optical depth $\tau$ when we
use the reionization bump of $C_{l}^{EE}$ for
$m_{\phi}=10^{-30}\,\text{--}\,10^{-32}$ eV. A larger $\tau$ produces a larger
amplitude of the reionization bump while a larger $n$ attenuates it.
Therefore, when we use $C_{l}^{EE}$ to determine $n$, we have to determine
$\tau$ in other ways. For example, the results of recent astronomical
observations of high-z galaxies like JWST provide a powerful way to determine
$\tau$ independently. Recent results disfavor larger $\tau$ [55, 56].
Performing parameter fitting, setting both $\tau$ and $n$ as free parameters,
is left for future work.

###### Acknowledgements.

We are grateful to Eiichiro Komatsu, Maresuke Shiraishi, and Masami Ouchi for
fruitful discussions on this paper. This work was supported in part by JSPS
KAKENHI Grant Numbers JP24KJ0668 (F. N.), JP20H05859 and JP22K03682 (T. N.),
JP23KJ0088 and JP24K17039 (K. M.), JP19K14702 (I. O.), and JP23K17687 (K. K.),
and the National Natural Science Foundation of China (NSFC) under Grant No.
12347103 (K. K.). F. N. thanks the Forefront Physics and Mathematics Program
to Drive Transformation (FoPM), a World-leading Innovative Graduate Study
(WINGS) Program, the University of Tokyo. Part of this work uses resources of
the National Energy Research Scientific Computing Center (NERSC). The Kavli
IPMU is supported by World Premier International Research Center Initiative
(WPI Initiative), MEXT, Japan.

## References

  * Komatsu [2022] E. Komatsu, Nature Rev. Phys. 4, 452 (2022), arXiv:2202.13919 [astro-ph.CO] .
  * Minami and Komatsu [2020] Y. Minami and E. Komatsu, Phys. Rev. Lett. 125, 221301 (2020), arXiv:2011.11254 [astro-ph.CO] .
  * Diego-Palazuelos  _et al._ [2022] P. Diego-Palazuelos _et al._ , Phys. Rev. Lett. 128, 091302 (2022), arXiv:2201.07682 [astro-ph.CO] .
  * Eskilt [2022] J. R. Eskilt, Astron. Astrophys. 662, A10 (2022), arXiv:2201.13347 [astro-ph.CO] .
  * Eskilt and Komatsu [2022] J. R. Eskilt and E. Komatsu, Phys. Rev. D 106, 063503 (2022), arXiv:2205.13962 [astro-ph.CO] .
  * Eskilt  _et al._ [2023a] J. R. Eskilt _et al._ (Cosmoglobe), Astron. Astrophys. 679, A144 (2023a), arXiv:2305.02268 [astro-ph.CO] .
  * Nakai  _et al._ [2024] Y. Nakai, R. Namba, I. Obata, Y.-C. Qiu, and R. Saito, JHEP 01, 057 (2024), arXiv:2310.09152 [astro-ph.CO] .
  * Fujita  _et al._ [2021] T. Fujita, K. Murai, H. Nakatsuka, and S. Tsujikawa, Phys. Rev. D 103, 043509 (2021), arXiv:2011.11894 [astro-ph.CO] .
  * Takahashi and Yin [2021] F. Takahashi and W. Yin, JCAP 04, 007 (2021), arXiv:2012.11576 [hep-ph] .
  * Fung  _et al._ [2021] L. W. H. Fung, L. Li, T. Liu, H. N. Luu, Y.-C. Qiu, and S. H. H. Tye, JCAP 08, 057 (2021), arXiv:2102.11257 [hep-ph] .
  * Nakagawa  _et al._ [2021] S. Nakagawa, F. Takahashi, and M. Yamada, Phys. Rev. Lett. 127, 181103 (2021), arXiv:2103.08153 [hep-ph] .
  * Jain  _et al._ [2021] M. Jain, A. J. Long, and M. A. Amin, JCAP 05, 055 (2021), arXiv:2103.10962 [astro-ph.CO] .
  * Choi  _et al._ [2021] G. Choi, W. Lin, L. Visinelli, and T. T. Yanagida, Phys. Rev. D 104, L101302 (2021), arXiv:2106.12602 [hep-ph] .
  * Obata [2022] I. Obata, JCAP 09, 062 (2022), arXiv:2108.02150 [astro-ph.CO] .
  * Nakatsuka  _et al._ [2022] H. Nakatsuka, T. Namikawa, and E. Komatsu, Phys. Rev. D 105, 123509 (2022), arXiv:2203.08560 [astro-ph.CO] .
  * Lin and Yanagida [2023] W. Lin and T. T. Yanagida, Phys. Rev. D 107, L021302 (2023), arXiv:2208.06843 [hep-ph] .
  * Gasparotto and Obata [2022] S. Gasparotto and I. Obata, JCAP 08, 025 (2022), arXiv:2203.09409 [astro-ph.CO] .
  * Lee  _et al._ [2022] N. Lee, S. C. Hotinli, and M. Kamionkowski, Phys. Rev. D 106, 083518 (2022), arXiv:2207.05687 [astro-ph.CO] .
  * Jain  _et al._ [2022] M. Jain, R. Hagimoto, A. J. Long, and M. A. Amin, JCAP 10, 090 (2022), arXiv:2208.08391 [astro-ph.CO] .
  * Murai  _et al._ [2023] K. Murai, F. Naokawa, T. Namikawa, and E. Komatsu, Phys. Rev. D 107, L041302 (2023), arXiv:2209.07804 [astro-ph.CO] .
  * Gonzalez  _et al._ [2023] D. Gonzalez, N. Kitajima, F. Takahashi, and W. Yin, Phys. Lett. B 843, 137990 (2023), arXiv:2211.06849 [hep-ph] .
  * Qiu  _et al._ [2023] Y.-C. Qiu, J.-W. Wang, and T. T. Yanagida, Phys. Rev. Lett. 131, 071802 (2023), arXiv:2301.02345 [hep-ph] .
  * Eskilt  _et al._ [2023b] J. R. Eskilt, L. Herold, E. Komatsu, K. Murai, T. Namikawa, and F. Naokawa, Phys. Rev. Lett. 131, 121001 (2023b), arXiv:2303.15369 [astro-ph.CO] .
  * Namikawa and Obata [2023] T. Namikawa and I. Obata, Phys. Rev. D 108, 083510 (2023), arXiv:2306.08875 [astro-ph.CO] .
  * Gasparotto and Sfakianakis [2023] S. Gasparotto and E. I. Sfakianakis, JCAP 11, 017 (2023), arXiv:2306.16355 [astro-ph.CO] .
  * Ferreira  _et al._ [2024] R. Z. Ferreira, S. Gasparotto, T. Hiramatsu, I. Obata, and O. Pujolas, JCAP 05, 066 (2024), arXiv:2312.14104 [hep-ph] .
  * Lue  _et al._ [1999] A. Lue, L.-M. Wang, and M. Kamionkowski, Phys. Rev. Lett. 83, 1506 (1999), arXiv:astro-ph/9812088 .
  * Feng  _et al._ [2006] B. Feng, M. Li, J.-Q. Xia, X. Chen, and X. Zhang, Phys. Rev. Lett. 96, 221302 (2006), arXiv:astro-ph/0601095 .
  * Zaldarriaga and Seljak [1997] M. Zaldarriaga and U. Seljak, Phys. Rev. D 55, 1830 (1997), astro-ph/9609170 .
  * Kamionkowski  _et al._ [1997] M. Kamionkowski, A. Kosowsky, and A. Stebbins, Phys. Rev. D 55, 7368 (1997), arXiv:astro-ph/9611125 .
  * Zhao  _et al._ [2015] G.-B. Zhao, Y. Wang, J.-Q. Xia, M. Li, and X. Zhang, JCAP 07, 032 (2015), arXiv:1504.04507 [astro-ph.CO] .
  * Hlozek  _et al._ [2015] R. Hlozek, D. Grin, D. J. E. Marsh, and P. G. Ferreira, Phys. Rev. D 91, 103512 (2015), arXiv:1410.2896 [astro-ph.CO] .
  * Aghanim  _et al._ [2020] N. Aghanim _et al._ (Planck), Astron. Astrophys. 641, A6 (2020), [Erratum: Astron.Astrophys. 652, C4 (2021)], arXiv:1807.06209 [astro-ph.CO] .
  * Berg  _et al._ [2017] M. Berg, J. P. Conlon, F. Day, N. Jennings, S. Krippendorf, A. J. Powell, and M. Rummel, Astrophys. J. 847, 101 (2017), arXiv:1605.01043 [astro-ph.HE] .
  * Caldwell  _et al._ [2011] R. R. Caldwell, V. Gluscevic, and M. Kamionkowski, Phys. Rev. D 84, 043504 (2011), arXiv:1104.1634 [astro-ph.CO] .
  * Capparelli  _et al._ [2020] L. M. Capparelli, R. R. Caldwell, and A. Melchiorri, Phys. Rev. D 101, 123529 (2020), arXiv:1909.04621 [astro-ph.CO] .
  * Greco  _et al._ [2023] A. Greco, N. Bartolo, and A. Gruppuso, J. Cosmol. Astropart. Phys. 05, 026 (2023), arXiv:2211.06380 [astro-ph.CO] .
  * Lewis  _et al._ [2000] A. Lewis, A. Challinor, and A. Lasenby, Astrophys. J. 538, 473 (2000), arXiv:astro-ph/9911177 .
  * Greco  _et al._ [2024] A. Greco, N. Bartolo, and A. Gruppuso, (2024), arXiv:2401.07079 [astro-ph.CO] .
  * Namikawa  _et al._ [2020] T. Namikawa _et al._ , Phys. Rev. D 101, 083527 (2020), arXiv:2001.10465 [astro-ph.CO] .
  * Bianchini  _et al._ [2020] F. Bianchini _et al._ (SPT), Phys. Rev. D 102, 083504 (2020), arXiv:2006.08061 [astro-ph.CO] .
  * Bortolami  _et al._ [2022] M. Bortolami, M. Billi, A. Gruppuso, P. Natoli, and L. Pagano, J. Cosmol. Astropart. Phys. 09, 075 (2022), arXiv:2206.01635 [astro-ph.CO] .
  * Zagatti  _et al._ [2024] G. Zagatti, M. Bortolami, A. Gruppuso, P. Natoli, L. Pagano, and G. Fabbian, JCAP 05, 034 (2024), arXiv:2401.11973 [astro-ph.CO] .
  * Liu  _et al._ [2006] G.-C. Liu, S. Lee, and K.-W. Ng, Phys. Rev. Lett. 97, 161303 (2006), arXiv:astro-ph/0606248 .
  * Finelli and Galaverni [2009] F. Finelli and M. Galaverni, Phys. Rev. D 79, 063002 (2009), 0802.4210 .
  * Gubitosi  _et al._ [2014] G. Gubitosi, M. Martinelli, and L. Pagano, J. Cosmol. Astropart. Phys. 12, 020 (2014), arXiv:1410.1799 [astro-ph.CO] .
  * Lee  _et al._ [2016] S. Lee, G.-C. Liu, and K.-W. Ng, The Universe 4, 29 (2016), arXiv:1912.12903 [astro-ph.CO] .
  * Naokawa and Namikawa [2023] F. Naokawa and T. Namikawa, Phys. Rev. D 108, 063525 (2023), arXiv:2305.13976 [astro-ph.CO] .
  * Lesgourgues [2011] J. Lesgourgues, (2011), arXiv:1104.2932 [astro-ph.IM] .
  * Zaldarriaga [1997] M. Zaldarriaga, Phys. Rev. D 55, 1822 (1997), arXiv:astro-ph/9608050 .
  * Tristram  _et al._ [2024] M. Tristram _et al._ , Astron. Astrophys. 682, A37 (2024), arXiv:2309.10034 [astro-ph.CO] .
  * Ade  _et al._ [2019] P. Ade _et al._ (Simons Observatory), JCAP 02, 056 (2019), arXiv:1808.07445 [astro-ph.CO] .
  * Allys  _et al._ [2023] E. Allys _et al._ (LiteBIRD), PTEP 2023, 042F01 (2023), arXiv:2202.02773 [astro-ph.IM] .
  * Abazajian  _et al._ [2022] K. Abazajian _et al._ (CMB-S4), Astrophys. J. 926, 54 (2022), arXiv:2008.12619 [astro-ph.CO] .
  * Nakane  _et al._ [2024] M. Nakane _et al._ , Astrophys. J. 967, 28 (2024), arXiv:2312.06804 [astro-ph.GA] .
  * Umeda  _et al._ [2023] H. Umeda _et al._ , (2023), arXiv:2306.00487 [astro-ph.GA] .


a three-dimensional set. Assume that there exist $N,\varepsilon,C>0$ such that
for every $x,x^{\prime}\in D,0\leq r,r^{\prime}\leq 1$,

$\mathbb{E}\left[\left|B(x,r)-B(x^{\prime},r^{\prime})\right|^{N}\right]\leq
C\left\\{(r\vee
r^{\prime})\left\|(x,r)-(x^{\prime},r^{\prime})\right\|\right\\}^{3+\varepsilon}.$

Then there is a modification $\tilde{B}$ of $B$ satisfying: for every
configuration $\omega$ and for all $\alpha\in(0,\varepsilon/N)$, there exists
$C_{\alpha}(\omega)\in(0,\infty)$ such that for all $x,x^{\prime}\in D,0\leq
r,r^{\prime}\leq 1$,

$\left|\tilde{B}(x,r)-\tilde{B}(x^{\prime},r^{\prime})\right|\leq
C_{\alpha}(\omega)\left\\{(r\vee
r^{\prime})\left\|(x,r)-(x^{\prime},r^{\prime})\right\|\right\\}^{\alpha}.$

###### Proof.

In this proof we will denote by $\mathcal{D}_{2}^{n}$ (resp.
$\mathcal{D}_{1}^{n}$) the set of two-dimensional (resp. one-dimensional)
dyadic points of generation $n$ lying in $D$ (resp. $[0,1]$). We will write
$\mathcal{D}_{2}=\cup_{n\geq 1}\mathcal{D}_{2}^{n}$ and
$\mathcal{D}_{1}=\cup_{n\geq 1}\mathcal{D}_{1}^{n}$. Take
$\alpha<\varepsilon/N$. If
$x\in\mathcal{D}_{2}^{n},r\in\mathcal{D}_{1}^{n},r^{\prime}=r+2^{-n}$, we have

$\mathbb{P}\left(\left|B(x,r)-B(x,r^{\prime})\right|\geq(r\vee
r^{\prime})^{\alpha}2^{-n\alpha}\right)\leq C\left((r\vee
r^{\prime})2^{-n}\right)^{3+\varepsilon-\alpha N}\leq
C2^{-n(3+\varepsilon-\alpha N)}.$

By a union bound and because $\varepsilon-\alpha N>0$, we get that

$\sum_{n\geq 1}\mathbb{P}\left(\exists
x\in\mathcal{D}_{2}^{n},r\in\mathcal{D}_{1}^{n},\left|B(x,r)-B(x,r+2^{-n})\right|\geq(r+2^{-n})^{\alpha}2^{-n\alpha}\right)$

is finite. By Borel-Cantelli lemma, it implies that

$\sup\left\\{(r+2^{-n})^{-\alpha}2^{n\alpha}\left|B(x,r)-B(x,r+2^{-n})\right|,n\geq
1,x\in\mathcal{D}_{2}^{n},r\in\mathcal{D}_{1}^{n}\right\\}$

is almost surely finite (the quantity above is clearly finite for a finite
number of $n$). We can proceed in the same way for the first component to show
that

$\sup\left\\{\frac{\left|B(x,r)-B(x^{\prime},r^{\prime})\right|}{\left\\{(r\vee
r^{\prime})\left\|(x,r)-(x^{\prime},r^{\prime})\right\|\right\\}^{\alpha}},\begin{array}[]{l}n\geq
1,x,x^{\prime}\in\mathcal{D}_{2}^{n},\\\
r,r^{\prime}\in\mathcal{D}_{1}^{n},\\\
\left\|(x,r)-(x^{\prime},r^{\prime})\right\|_{L^{1}}=2^{-n}\end{array}\right\\}$

is almost surely finite. We denote this quantity by $K_{\alpha}$. Now consider
$(x,r),(x,r^{\prime})\in\mathcal{D}_{2}\times\mathcal{D}_{1}$. Assume for
instance that $r<r^{\prime}$. Consider $p\geq 1$ s.t. $2^{-p}\leq
r^{\prime}-r\leq 2^{-p+1}$ and $k$ the smallest integer such that $r\leq
k2^{-p}$. Let $\ell\geq 0$ be s.t.
$(x,r),(x,r^{\prime})\in\mathcal{D}_{2}^{p+\ell}\times\mathcal{D}_{1}^{p+\ell}$.
We can write $r$ and $r^{\prime}$ as

$r=k2^{-p}-\varepsilon_{1}2^{-p-1}-\dots-\varepsilon_{\ell}2^{-p-\ell}\mathrm{~{}and~{}}r^{\prime}=k2^{-p}+\varepsilon^{\prime}_{0}2^{-k}+\varepsilon^{\prime}_{1}2^{-p-1}+\dots+\varepsilon^{\prime}_{\ell}2^{-p-\ell}$

with $\varepsilon_{i},\varepsilon^{\prime}_{i}\in\\{0,1\\}$. Define for all
$i=0,\dots,\ell$,

$r_{i}=k2^{-p}-\varepsilon_{1}2^{-p-1}-\dots-\varepsilon_{i}2^{-p-i}\mathrm{~{}and~{}}r^{\prime}_{i}=k2^{-p}+\varepsilon^{\prime}_{0}2^{-k}+\varepsilon^{\prime}_{1}2^{-p-1}+\dots+\varepsilon^{\prime}_{i}2^{-p-i}.$

In particular, $r_{i},r^{\prime}_{i}\in\mathcal{D}_{1}^{p+i}$ and
$r_{i}-r_{i+1},r^{\prime}_{i+1}-r^{\prime}_{i}\in\\{0,2^{-p-i-1}\\}$. Notice
that for all $i=0\dots\ell-1,r_{i}\vee r^{\prime}_{i}\leq r^{\prime}$.
Similarly, there exists a sequence $(x_{i},i=0\dots 2\ell)$ with $x_{2\ell}=x$
such that

$\displaystyle\forall
i=0\dots\ell-1,x_{2i},x_{2i+1}\in\mathcal{D}_{2}^{p+i},\left\|x_{2i}-x_{2i+1}\right\|_{L^{1}}\in\\{0,2^{-p-i}\\}.$

We can bound from above $\left|B(x,r)-B(x_{0},r_{0})\right|$ by

$\displaystyle\sum_{i=0}^{\ell-1}\left|B(x_{2i+2},r_{i+1})-B(x_{2i+1},r_{i+1})\right|+\left|B(x_{2i+1},r_{i+1})-B(x_{2i},r_{i+1})\right|+\left|B(x_{2i},r_{i+1})-B(x_{2i},r_{i})\right|$
$\displaystyle\leq
3K_{\alpha}(r^{\prime})^{\alpha}\sum_{i=0}^{\ell-1}2^{-(p+i+1)\alpha}\leq
3\frac{2^{-\alpha}}{1-2^{-\alpha}}K_{\alpha}(r^{\prime})^{\alpha}2^{-p\alpha}.$

The same bound holds true for
$\left|B(x,r^{\prime})-B(x_{0},r^{\prime}_{0})\right|$ and

$\displaystyle\left|B(x,r)-B(x,r^{\prime})\right|$ $\displaystyle\leq
6\frac{2^{-\alpha}}{1-2^{-\alpha}}K_{\alpha}(r^{\prime})^{\alpha}2^{-p\alpha}+\left|B(x_{0},r_{0},s_{0})-B(x_{0},r^{\prime}_{0},s_{0})\right|$
$\displaystyle\leq\left(1+6\frac{2^{-\alpha}}{1-2^{-\alpha}}\right)K_{\alpha}(r^{\prime})^{\alpha}2^{-p\alpha}\leq\left(1+6\frac{2^{-\alpha}}{1-2^{-\alpha}}\right)K_{\alpha}(r^{\prime})^{\alpha}(r^{\prime}-r)^{\alpha}.$

A similar treatment can be applied when $x$ varies as well. If we define for
all $(x,r)\in\mathcal{D}_{2}\times\mathcal{D}_{1}$,

$\tilde{B}(x,r)=\left\\{\begin{array}[]{ll}B(x,r)&\mathrm{if~{}}K_{\alpha}<\infty\\\
0&\mathrm{otherwise}\end{array}\right.,$

and for all $(x,r)\in D\times[0,1]$,

$\tilde{B}(x,r)=\lim_{\begin{subarray}{c}(x^{\prime},r^{\prime})\to(x,r)\\\
(x^{\prime},r^{\prime})\in\mathcal{D}_{2}\times\mathcal{D}_{1}\end{subarray}}\tilde{B}(x^{\prime},r^{\prime}),$

then $\tilde{B}$ satisfies the requirements of the lemma. ∎

##  Appendix B A tail estimate for $|\mu^{*}(Q(0,1))|$

In this appendix, we give some details on the estimate (1.8) from [LSZ17].
Recall that we denote by $\mu^{*}$ the imaginary chaos associated to the whole
plane GFF $\Gamma^{*}$. The main result of [LSZ17] is the existence of a
constant $c^{**}(\beta)\in\mathbb{R}$ such that

$\mathbb{E}\left|\mu^{*}([0,1]^{2})\right|^{2N}=N^{\beta^{2}N/2}e^{c^{**}(\beta)N+o(N)},\qquad\text{as}\quad
N\to\infty.$ (B.1)

See [LSZ17, Corollary 1.3]. Let us define the constant $c^{*}(\beta)>0$ by

$c^{*}(\beta)=\frac{\beta^{2}}{2}e^{-1-2c^{**}(\beta)/\beta^{2}},\qquad
c^{**}(\beta)=-\frac{\beta^{2}}{2}\Big{(}1+\log\frac{2c^{*}(\beta)}{\beta^{2}}\Big{)}.$
(B.2)

In [LSZ17, Corollary A.2], the authors argue that (B.1) yields the following
tail estimate:

$\lim_{t\to\infty}t^{-4/\beta^{2}}\log\mathbb{P}\left(|\mu^{*}(Q(0,1))|>t\right)=-c^{*}(\beta).$
(B.3)

The upper bound follows directly from (B.1) by optimising some Markov’s
inequality. The lower bound however necessitates more care and the proof in
[LSZ17] actually seems to contain a small gap. Indeed, two random variables
$X$ and $Y$ are considered (with the notations therein $X=Y_{\delta}-C_{2}$
and $Y=|\int_{D}e^{i\beta h(x)}dx|^{2}$). These variables satisfy for all
$k\geq 1$, $\mathbb{E}[X^{k}]\leq\mathbb{E}[Y^{k}]$. It is then inferred that
$X$ is stochastically dominated by $Y$ in the sense that
$\mathbb{P}(X>t)\leq\mathbb{P}(Y>t)$ for all $t$. This implication is however
erroneous in general as one can see with the example $X=1/3$ a.s. and $Y$ a
Bernoulli random variable with $\mathbb{P}(Y=1)=1-\mathbb{P}(Y=0)=2/3$. Since
several main results of the current paper crucially use this tail estimate, we
give an alternative elementary way of obtaining (B.3) from (B.1).

Let us also mention that we have corrected a typo in [LSZ17] where (B.3) is
stated with $t^{2/\beta^{2}}$ rather than $t^{4/\beta^{2}}$. Indeed, with
their notations, the $2k$-th moment of the absolute value of the imaginary
chaos is equal to $Z_{k,\beta^{2}}$ rather than $Z_{k,2\beta^{2}}$, contrary
to what is stated in [LSZ17, Lemma A.1]. Now, at the critical value
$\beta=\sqrt{2}$, the tail (B.3) has a Gaussian behaviour. This is consistent
with the fact that the imaginary chaos becomes a white noise at
$\beta=\sqrt{2}$ (see [LRV15, JSW20]).

We now explain our proof of the lower bound of (B.3). Let $\nu$ be the law of
$\left|\mu^{*}(Q(0,1))\right|$, $\alpha=4/\beta^{2}$ and $c^{*}=c^{*}(\beta)$
be the constant defined in (B.2). As mentioned, by a direct application of
Markov’s inequality, for all $d^{*}<c^{*}$, if $t$ is large enough
$\nu[t,\infty)\leq e^{-d^{*}t^{\alpha}}$. We want to show the reverse
inequality, i.e. that

$\liminf_{t\to\infty}\frac{\log\nu[t,\infty)}{t^{\alpha}}\geq-c^{*}.$ (B.4)

For all $\varepsilon>0$ small enough and $d^{*}<c^{*}$, letting $k(t)$ be the
unique even integer in $[d^{*}\alpha t^{\alpha},d^{*}\alpha t^{\alpha}+2)$, we
first claim that

$\left|\int_{0}^{\infty}\frac{x^{k(t)}}{t^{k(t)}}\nu(dx)-\int_{(1-\varepsilon)^{1/\alpha}t}^{(1+\varepsilon)^{1/\alpha}t}\frac{x^{k(t)}}{t^{k(t)}}\nu(dx)\right|\leq
e^{-(1+o(1))d^{*}(1+\varepsilon^{2}/4)t^{\alpha}},$ (B.5)

where $o(1)\to 0$ as $t\to\infty$.

###### Proof of (B.5).

Since $-\log(1+u)+1+u=1+u^{2}/2+O(u^{3})$ as $u\to 0$, there exists $u_{0}>0$
s.t. for all $u\in(-u_{0},u_{0})$, $-\log(1+u)+1+u\geq 1+u^{2}/4$. We now fix
$\varepsilon\in(0,u_{0})$ and $d^{*}<c^{*}$. There exists $t_{0}>0$ large
enough such that for all $t\geq t_{0}$,
$\nu[t,\infty)\leq\int_{t}^{\infty}\frac{1}{Z}e^{-d^{*}x^{\alpha}}dx$ where
$Z$ is the renormalising constant $Z=\int_{0}^{\infty}e^{-d^{*}x^{\alpha}}dx$.
We thus have the stochastic domination

$\nu(dx)\prec\frac{1}{Z}e^{-d^{*}x^{\alpha}}dx,\qquad\text{for}\quad x\geq
t_{0}.$ (B.6)

We decompose the left hand side of (B.5) into several parts depending on
whether $x\leq t_{0}$, $x\in[t_{0},(1-\varepsilon)^{1/\alpha}t]$ or
$x\geq(1+\varepsilon)^{1/\alpha}t$. For the first part, there exists $c>0$
depending on $\alpha,d^{*},t_{0}$ such that

$\int_{0}^{t_{0}}\frac{x^{k(t)}}{t^{k(t)}}\nu(dx)\leq
t_{0}^{k(t)}t^{-k(t)}\leq e^{-ct^{\alpha}\log t}.$

Noticing that $x\mapsto x^{k(t)}e^{-d^{*}x^{\alpha}}$ is increasing on
$[0,(\frac{k(t)}{d^{*}\alpha})^{1/\alpha}]$ and then decreasing, we also have:

$\displaystyle\int_{t_{0}}^{(1-\varepsilon)^{1/\alpha}t}\frac{x^{k(t)}}{t^{k(t)}}\nu(dx)$
$\displaystyle\leq\frac{1}{Z}\int_{t_{0}}^{(1-\varepsilon)^{1/\alpha}t}\frac{x^{k(t)}}{t^{k(t)}}e^{-d^{*}x^{\alpha}}dx\leq\frac{1}{Z}(1-\varepsilon)^{1/\alpha}t(1-\varepsilon)^{k(t)/\alpha}e^{-d^{*}(1-\varepsilon)t^{\alpha}}$
$\displaystyle=e^{-(1+o(1))\left(-d^{*}\log(1-\varepsilon)+d^{*}(1-\varepsilon)\right)t^{\alpha}}\leq
e^{-(1+o(1))d^{*}(1+\varepsilon^{2}/4)t^{\alpha}}$

because $\varepsilon<u_{0}$. Similarly (essentially change
$\varepsilon\leftarrow-\varepsilon$ in the computations above)

$\int_{(1+\varepsilon)^{1/\alpha}t}^{t^{2}}\frac{x^{k(t)}}{t^{k(t)}}\nu(dx)\leq
e^{-(1+o(1))d^{*}(1+\varepsilon^{2}/4)t^{\alpha}}.$

The integral from $t^{2}$ to $\infty$ is even smaller and putting things
together, we have obtained (B.5). ∎

Let $\varepsilon>0$ be small and $d^{*}<c^{*}$ close enough to $c^{*}$ so that
$\varepsilon^{2}/4>\log(c^{*}/d^{*}).$ By (B.1),

$\displaystyle\int_{0}^{\infty}\frac{x^{k(t)}}{t^{k(t)}}\nu(dx)$
$\displaystyle=t^{-k(t)}\exp\left(\frac{1}{\alpha}k(t)\log\frac{k(t)}{2}-\frac{1}{\alpha}\left(1+\log\frac{\alpha
c^{*}}{2}\right)k(t)+o(k)\right)$
$\displaystyle=\exp\left(-(1+o(1))d^{*}\left(1+\log\frac{c^{*}}{d^{*}}\right)t^{\alpha}\right).$

Because we have chosen $\varepsilon$ and $d^{*}$ so that
$\varepsilon^{2}/4>\log(c^{*}/d^{*})$, combining the above estimate with (B.5)
leads to

$\int_{(1-\varepsilon)^{1/\alpha}t}^{(1+\varepsilon)^{1/\alpha}t}\frac{x^{k(t)}}{t^{k(t)}}\nu(dx)\geq\exp\left(-(1+o(1))d^{*}\left(1+\log\frac{c^{*}}{d^{*}}\right)t^{\alpha}\right).$

In particular,

$\displaystyle\nu\left[(1-\varepsilon)^{1/\alpha}t,\infty\right)$
$\displaystyle\geq(1+\varepsilon)^{-k(t)/\alpha}\int_{(1-\varepsilon)^{1/\alpha}t}^{(1+\varepsilon)^{1/\alpha}t}\frac{x^{k(t)}}{t^{k(t)}}\nu(dx)$
$\displaystyle\geq\exp\left(-(1+o(1))d^{*}\left(1+\log\frac{c^{*}}{d^{*}(1-\varepsilon)}\right)t^{\alpha}\right).$

We have thus shown that

$\liminf_{t\to\infty}\frac{\log\nu[t,\infty)}{t^{\alpha}}\geq-\frac{d^{*}}{1-\varepsilon}\left(1+\log\frac{c^{*}}{d^{*}(1-\varepsilon)}\right).$

By letting $d^{*}\to c^{*}$ and then $\varepsilon\to 0$, we obtain (B.4) as
wanted.

## References

  * [AJ21] Juhan Aru and Janne Junnila.  Reconstructing the base field from imaginary multiplicative chaos.  Bulletin of the London Mathematical Society, 53(3):861–870, 2021\.
  * [AJJ22] Juhan Aru, Antoine Jego, and Janne Junnila.  Density of imaginary multiplicative chaos via Malliavin calculus.  Probability Theory and Related Fields, 184(3-4):749–803, 2022.
  * [BJ10] Julien Barral and Xiong Jin.  Multifractal analysis of complex random cascades.  Communications in Mathematical Physics, 297(1):129–168, Jul 2010\.
  * [BJM10a] Julien Barral, Xiong Jin, and Benoît Mandelbrot.  Convergence of complex multiplicative cascades.  Ann. Appl. Probab., 20(4):1219–1252, 08 2010.
  * [BJM10b] Julien Barral, Xiong Jin, and Benoît Mandelbrot.  Uniform convergence for complex $[0,1]$-martingales.  Ann. Appl. Probab., 20(4):1205–1218, 08 2010.
  * [DES93] Bernard Derrida, Martin R Evans, and Eugene R Speer.  Mean field theory of directed polymers with random complex weights.  Communications in mathematical physics, 156:221–244, 1993.
  * [DNPV12] Eleonora Di Nezza, Giampiero Palatucci, and Enrico Valdinoci.  Hitchhiker’s guide to the fractional Sobolev spaces.  Bulletin des sciences mathématiques, 136(5):521–573, 2012.
  * [GKR23] Colin Guillarmou, Antti Kupiainen, and Rémi Rhodes.  Compactified imaginary Liouville theory, 2023.
  * [HMP10] Xiaoyu Hu, Jason Miller, and Yuval Peres.  Thick points of the Gaussian free field.  Ann. Probab., 38(2):896–926, 03 2010.
  * [Jaf00] Stéphane Jaffard.  On the Frisch-Parisi conjecture.  Journal de Mathématiques Pures et Appliquées, 79(6):525 – 552, 2000.
  * [JSV19] Janne Junnila, Eero Saksman, and Lauri Viitasaari.  On the regularity of complex multiplicative chaos, 2019.
  * [JSW20] Janne Junnila, Eero Saksman, and Christian Webb.  Imaginary multiplicative chaos: moments, regularity and connections to the Ising model.  Ann. Appl. Probab., 30(5):2099–2164, 2020.
  * [LRV15] Hubert Lacoin, Rémi Rhodes, and Vincent Vargas.  Complex Gaussian Multiplicative Chaos.  Communications in Mathematical Physics, 337(2):569–632, Jul 2015\.
  * [LSZ17] Thomas Leblé, Sylvia Serfaty, and Ofer Zeitouni.  Large Deviations for the Two-Dimensional Two-Component Plasma.  Communications in Mathematical Physics, 350(1):301–360, Feb 2017\.
  * [Mey93] Yves Meyer.  Wavelets and Operators, volume 1 of Cambridge Studies in Advanced Mathematics.  Cambridge University Press, 1993.
  * [MP10] Peter Mörters and Yuval Peres.  Brownian Motion.  Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2010.
  * [MS17] Jason Miller and Scott Sheffield.  Imaginary geometry IV: interior rays, whole-plane reversibility, and space-filling trees.  Probability Theory and Related Fields, 169(3):729–869, Dec 2017\.
  * [RV14] Rémi Rhodes and Vincent Vargas.  Gaussian multiplicative chaos and applications: A review.  Probab. Surveys, 11:315–392, 2014.
  * [SV06] Daniel W. Stroock and S. R. Srinivasa Varadhan.  Multidimensional diffusion processes.  Classics in Mathematics. Springer-Verlag, Berlin, 2006.  Reprint of the 1997 edition.
  * [Tri83] Hans Triebel.  Theory of function spaces.  Birkhäuser Basel, 1983.
  * [Ver10] Mark Veraar.  Regularity of Gaussian white noise on the d-dimensional torus.  Banach Center Publications, 95, 10 2010.
  * [VH14] Ramon Van Handel.  Probability in high dimension.  Lecture Notes (Princeton University), 2014.
  * [Zak81] Moshe Zakai.  Some Classes of Two-Parameter Martingales.  The Annals of Probability, 9(2):255 – 265, 1981.


# Sim-on-Wheels: Physical World in the Loop Simulation for Self-Driving

Yuan Shen∗, Bhargav Chandaka∗, Zhi-hao Lin, Albert Zhai, Hang Cui, David
Forsyth and Shenlong Wang * denotes equal contributions. The authors are with
the University of Illinois at Urbana-Champaign. Email: {yshen47, bhargav9,
cl121, azhai2, hangcui3, daf<EMAIL_ADDRESS>
###### Abstract

We present Sim-on-Wheels, a safe, realistic, and vehicle-in-loop framework to
test autonomous vehicles’ performance in the real world under safety-critical
scenarios. Sim-on-wheels runs on a self-driving vehicle operating in the
physical world. It creates virtual traffic participants with risky behaviors
and seamlessly inserts the virtual events into images perceived from the
physical world in real-time. The manipulated images are fed into autonomy,
allowing the self-driving vehicle to react to such virtual events. The full
pipeline runs on the actual vehicle and interacts with the physical world, but
the safety-critical events it sees are virtual. Sim-on-Wheels is safe,
interactive, realistic, and easy to use. The experiments demonstrate the
potential of Sim-on-Wheels to facilitate the process of testing autonomous
driving in challenging real-world scenes with high fidelity and low risk.
Additional results and open-sourced code are available on our project page
here: https://sim-on-wheels.github.io/.

##  I Introduction

Evaluating how a self-driving car performs in dangerous scenarios is hard.
Pure real-world evaluations create situations that are dangerous to
participants, while pure simulation evaluations may simulate various scenarios
inaccurately, such as cases in which the vehicle has extreme control inputs.
This paper describes a mixed method, Sim-on-Wheels. In Sim-on-Wheels, we run
actual autonomy stack on real cars, but create scenarios by inserting people
and objects into the sensor feed in real-time. This means we can evaluate the
autonomy stack in scenarios known to be dangerous to pedestrians without
risking harm because the pedestrians are simulated. Furthermore, we apply the
control inputs to a real vehicle. If the autonomy could cause an uncontrolled
skid, we will be able to measure that. Fig. 1 illustrates our evaluation
pipeline and Tab. I compares it to previous methods. In contrast to previous
approaches, Sim-on-Wheels is simultaneously safe, interactive, realistic, and
easy to use.

| Realistic Sensor | Real-world Physics | Closed-loop | Safe | Convenient
---|---|---|---|---|---
Off-policy datasets [1, 2] | ✓ |  |  | ✓ | ✓
Real-world: road test [3, 4, 5] | ✓ | ✓ | ✓ |  |
Real-world: test track [6, 7] | ✓ | ✓ | ✓ |  ✓ – |
Simulation: CG-based [8, 9, 10] |  |  | ✓ | ✓ | ✓
Simulation: data-driven [11, 12, 13, 14, 15] |  ✓ – |  | ✓ | ✓ | ✓
Sim-on-Wheels (ours) | ✓ | ✓ | ✓ | ✓ | ✓
TABLE I: A reliable self-driving vehicle evaluation framework requires
providing realistic sensors, realistic physics, and closed-loop interaction,
all while being safe and easy to use. We situate past frameworks along these
five dimensions and discuss them in Section II.

There is no current consensus on evaluation protocols for autonomous vehicles.
Safety evaluation is typically through a combination of real-world road-tests,
off-policy data collection, and computer simulation. Real-world testing is a
resource-intensive and risky process, and testing in some scenarios is
unethical (because there is a strong chance of hurting a participant).
Annoyingly, these are the cases where evaluation is particularly important.
Off-policy data can be an effective tool for training and evaluating
perception algorithms, but does not yield a closed-loop evaluation of the
safety of the entire autonomy stack. Computer simulation is safe and scalable,
but is not currently reliable in extreme physical and mechanical situations.
Sim-on-Wheels is a mashup of real-world road tests (so we can observe true
vehicle behavior) and computer simulation (so we don’t have to risk harm to
participants).

Figure 1:  Sim-on-Wheels Pipeline. In Sim-on-Wheels’ evaluation paradigm,
vehicle autonomy is evaluated on images that are perceived in the real world
but transmitted to the onboard simulator and manipulated in real-time to show
important and dangerous traffic scenarios. The autonomy is asked to react to
the manipulated sensory input as if the scenario actually happened. Onboard
evaluation can be conducted in real-time to verify the safety and
effectiveness of the autonomy.

An ideal self-driving evaluation environment should be safe, realistic, and
closed-loop. Achieving safe evaluation is challenging, because one should be
evaluating dangerous scenarios but experiments that pose a risk to life are
unethical. Realistic evaluation is essential – we need to be sure that
evaluation predictions reflect real-life behavior. Finally, closed-loop
evaluation is essential because we must evaluate interactions between the
environment and the end-to-end perception to action behavior of the
controller. Sim-on-Wheels is intrinsically safe and closed-loop. We show that
Sim-on-Wheels results are realistic by both evaluating the realism of the
inserted objects and by comparing conclusions on real and Sim-on-Wheels
scenarios (Section IV-D). We use Sim-on-Wheels to “spoof” a total of 40
variations of safety-critical scenarios using two different autonomous vehicle
pipelines (Section IV-C). With the capability of testing scenarios configured
at system limit, our Sim-on-Wheel framework reveals our modular agent is more
cautious than our end-to-end learned agent in terms of obstacle avoidance,
achieving a lower collision rate but taking longer to reach the goals.

##  II Related Work

Self-driving autonomy: There are two paradigms for self-driving autonomy: (i)
modular [16, 17] and (ii) end-to-end [18]. A modular stack has multiple sub-
tasks in a pipeline framework, including localization [19, 20], perception
[21, 22, 23], planning [24], and control [16, 25, 26]. Attractions include
interpretability, modularity, and versatility [18], but tuning the pipeline
can be challenging, and errors can propagate. End-to-end autonomy directly
maps sensor input to planner or controller commands [27, 28, 29]. End-to-end
methods are usually easier to develop, but tend to be difficult to interpret,
so it is hard to diagnose errors, establish safety guarantees, and incorporate
traffic rules [30]. Recent advances in end-to-end learnable pipeline autonomy
have demonstrated promising results by combining the strengths of both
paradigms [28, 31]. Sim-on-Wheels is designed to be agnostic to the
modular/end-to-end choice. We evaluate both kinds of autonomy stack in Section
IV-C.

Self-driving evaluation: Evaluation practice is shaped by an important tension
between safety and accuracy (accurate evaluation requires assessment of
dangerous scenarios, posing risks to life). One strategy is to use off-policy
datasets [1, 2], which are safe and convenient. But because they do not close
the perception-action loop, such evaluations cannot accurately assess a full
autonomy stack. Another is to use real-world road tests [3, 4, 5]. These are
expensive and pose large risks to safety [32], and so are necessarily limited
in scope. Road tests on test tracks [6, 7] are somewhat safer than actual road
tests, but are expensive to set up and necessarily provide relatively little
environment diversity.

Yet another is to use a simulator, which is safe and convenient. Simulated
sensor inputs (as in [8, 9, 10]) face a sim2real gap, despite significant
literature on improving the realism of simulation (e.g. data-driven simulation
in [11, 12, 14, 15]; dynamic models in [33, 34, 35] ; environments in [13, 36,
37, 38]). It is extremely difficult to be sure that a simulator captures all
relevant physical modeling. This is particularly important in dangerous
scenarios, where one expects extreme control inputs and odd physics may become
important. For example, reverted rubber hydroplaning is an effect where very
aggressive braking causes tire rubber to break down and capture a surface
water film that breaks contact with the road; this and similar effects can
make an important contribution to whether a stack is safe, but may not appear
in simulators. Though this could be dealt with by adding modeling capacity to
simulators, it remains difficult to know what to add and when to stop. In
contrast, Sim-on-Wheels uses a real vehicle (and so relies on nature for these
effects) but simulates dangerous scenarios (and so does not endanger
participants).

Vehicle-in-the-loop simulation Sim-on-Wheels generally is a vehicle in the
loop simulation, because it incorporates the entire vehicle into the test.
Early such methods use a simulated driving environment [39, 40, 41, 42], with
attendant sim2real problems. MiRE [43] improves realism by using a body
tracking system to map a human into the scene to act as a pedestrian, but the
environment is far from realistic. AR on LiDAR [44] inserts objects in a
perceptually realistic manner into LiDAR point clouds (but not RGB images).
WIL [45] is a general framework for integrating simulated sensor inputs and
real inputs, but does not attend to rendering realism. In contrast, Sim-on-
Wheels provides realistic rendering aimed at specific, safety-critical
scenarios.

|  |  |  |
---|---|---|---|---
Static Obstacle | Jaywalker | Jaywalker w/ Occlusion | Traffic light violation | 3D assets

Figure 2: Illustration of Testing Scenarios and 3D Assets. Left: Our scenarios
depict common pre-collision events such as road obstacles, jaywalking,
jaywalking with occlusions, and traffic light violations. These scenarios are
represented as customizable and reproducible spatial-temporal waypoint
trajectories for all actors, with triggers, and can be easily expanded upon.
Right: We show a subset of our 3D assets, generated from real-world 3D scans
using an iPhone equipped with LiDAR.

##  III Simulation in the Physical World

Sim-on-Wheels operates by inserting actors, objects, and their animations into
the camera stream observed by a controller for a physical autonomous vehicle
platform (the “ego-vehicle”) moving in a real test space. Fig. 1 depicts the
entire pipeline of our framework. There are three main components. Authoring:
one must first author a driving scenario to be evaluated (which is likely to
be safety critical), including defining appropriate real-world waypoints and
animation sequences for virtual actors/objects, and determining the planned
path for the ego-vehicle to follow (Section III-A). Insertion rendering: a
real-time procedure takes raw RGB-D images, composites the simulated events
into the image stream, and re-publishes the composite images to the agent.
Sufficiently realistic insertion rendering means the ego-vehicle should react
in real-time to the inserted objects as if they were truly present (Section
III-B). Evaluation: metrics such as time to goal, stopping distance, and
collision rate are computed onboard to evaluate the effectiveness and safety
of different autonomy agents (Section III-C).

###  III-A Authoring Safety-Critical Scenarios

We choose safety-critical pre-crash scenarios based on the NHTSA pre-crash
event report [46]. Our scenarios encompass common traffic events such as
static obstacles, jaywalking, jaywalking with occlusions, and traffic light
violations. They are modeled as spatial-temporal waypoint trajectories for all
actors, allowing for full reproducibility of each scenario. The testing is
conducted to mimic both rural and urban environments, either in a straight
road segment or a four-way intersection. A selection of the scenarios is
depicted in Fig. 2.

Authoring involves selecting from a rich collection of 3D assets, including
artist-designed assets from SketchFab [47] and in-house created assets
reconstructed using an iPhone and multi-view reconstruction software [48]. For
each scenario, the human actors are animated by Mixamo [49] with realistic and
diverse human animations, such as walking and running.

The evaluation procedure involves triggering each scenario as the ego-vehicle
reaches the trigger zone at a certain speed range. A crash will occur if the
vehicle fails to conduct any evasive action. The hyper-parameters of each
scenario can be adjusted to control the level of difficulty, including the
type and trajectory of static objects for the static obstacle scenarios, the
type and trajectory of traffic light runners and trigger distance for the
intersection scenarios, and the walking speed, type and the number of
jaywalkers and trigger distance for the jaywalking scenarios, etc. Our
scenario bank can be easily expanded to cover additional safety-critical
events. One unique advantage of Sim-on-Wheels is that it enables setting
aggressive hyper-parameters without any risk for physical harm to any vehicles
or pedestrians, and the results of the evaluations provide a comprehensive
understanding of the performance limits of each autonomy stack. Another is
that the effect of (say) actor motion or dress on outcomes can be assessed by
evaluating the same scenario for different instances of each actor.

###  III-B Insertion Rendering

Insertion rendering involves producing realistic frames of a scene by
inserting assets (image fragments; 3D models; etc.) into a target image
(variants in [50, 51, 52, 53, 54]). Realistic frames can be produced very fast
if difficulties presented by lighting, shadows, and geometrical consistency
can be managed. Once the scenario is triggered, we render the simulated
scenarios and compose them into images in real time. This requires the
insertion rendering to be realistic, efficient, and geometrically consistent.
To achieve this, we adopt a real-time OpenGL-based rasterization pipeline
[55].

We first place the object accurately in the predefined world coordinate, and
the camera pose is acquired in real-time through an RTK-INS localization
module. The lighting consists of the skybox and the sunlight; parameters are
inferred from real-time weather, GPS, and the time of day.

The rendering process is then conducted using a customized physical-based
rendering (PBR) shader that follows the split-sum shading model, as described
by the equation:
$L(\mathbf{x},\bm{\omega}_{o})=L_{a}+L_{s}(\bm{\omega}_{s})f_{r}(\mathbf{x},\bm{\omega}_{o},\bm{\omega}_{s})(\bm{\omega}_{s}\cdot\mathbf{n})$
, where $x$ is the observed point, $\bm{\omega}_{o}$ is the outgoing ray and
$\bm{\omega}_{s}$ is the incoming ray. $L(\mathbf{x},\bm{\omega}_{o})$ is
observed radiance; $L_{a}$ is ambient sky color and $L_{s}$ is the directional
sunlight; $f_{r}$ is the Cook-Torrence reflection model [56]:
$f_{r}=k_{d}f_{d}+k_{s}f_{s}$, $f_{d}$ is diffuse reflection under sunlight,
and $f_{s}$ describes specular reflection, which accesses the base color,
roughness, and metallic textures of the object to compute specular reflection.
The resulting output, as shown in Fig. 3, exhibits a visually appealing
surface appearance.

Shadows cast by the inserted objects contribute to the perceived realism. In
our framework, a two-pass shadow mapping procedure is applied [57]. The first
pass renders a depth buffer from the lighting source to the visible surface,
and the second pass renders per-view depth from the camera perspective. The
inconsistent depth between the two identifies shadowed areas, and Poisson
sampling is used to reduce aliasing effects. In addition, occlusion reasoning
is conducted by comparing the rendered depth and perceived depth from the
stereo cameras, ensuring the correct depth ordering between the foreground
objects and the background scene. Finally, the rendered objects are composited
into the perceived image through alpha-channel blending.

spacingssReal |  |  |  |  |  |
---|---|---|---|---|---|---
spacinSimulation |  |  |  |  |  |

Figure 3: Rendering Quality. We evaluate the quality of insertion rendering by
comparing reconstructed 3D humans/objects in Sim-on-Wheels with their real-
world counterparts under the same pose. The results demonstrate that our real-
time insertion rendering can produce realistic, high-fidelity appearances, and
cast-shadows. Tab. IV quantitatively measures the sim2real gap.

###  III-C Evaluation

The performance of an autonomous driving stack is being evaluated in a
simulated safety-critical scenario using recorded behaviors. We adopt the
evaluation metrics from the CARLA platform [8], which include the collision
rate and the trip completion time. The collision rate represents the
percentage of scenarios where the ego-vehicle experiences at least one
collision, which is determined by checking for overlap between the oriented
bounding box of the vehicle and other virtual objects. In addition to ensuring
safety, we aim for our autonomous vehicle to be as efficient as possible,
which is reflected in the goal-reaching time. During each run, the goal-
reaching time is measured until the vehicle is within 5m of a static obstacle,
or in other scenarios, within 1.5m of the end of its planned path. In any
scenario, if the vehicle is not able to reach its goal, we penalize that run
by recording its time metric as 100s. Furthermore, in order to account for
real-world uncertainties, we report the mean collision rate and goal-reaching
time over multiple runs under different hyper-parameters for one safety-
critical scenario.

Static Obstacle | Jaywalker | Jaywalker with Occlusion | Traffic Light Violation
---|---|---|---
|  |  |
|  |  |
Imitation Learning Agent | Imitation Learning Agent | Imitation Learning Agent | Imitation Learning Agent
Reached Goal: No; Collision: No | Reached Goal:  Yes; Collision: Yes | Reached Goal:  Yes; Collision: No | Reached Goal:  Yes; Collision: No
|  |  |
|  |  |
Modular Agent | Modular Agent | Modular Agent | Modular Agent
Reached Goal:  Yes; Collision: No | Reached Goal:  Yes; Collision: No | Reached Goal:  No; Collision: Yes | Reached Goal:  Yes; Collision: No

Figure 4: Qualitative driving results. : We show bird’s-eye view layouts of
four different scenarios across both agents in ego-vehicle coordinates
(meters). The images are captured at the first braking point before
approaching an obstacle. We can see that the modular agent tends to be more
cautious and takes longer to reach the goal, while the imitation learner
drives smoothly but brakes too early/late in certain scenarios.

| Static Obstacle | Traffic Light Violation | Jaywalker | Jaywalker with Occlusion
---|---|---|---|---
Agent Type |  Collision Rate $\downarrow$ |  Time to Obstacle $\downarrow$ |  Collision Rate $\downarrow$ |  Time To Goal $\downarrow$ |  Collision Rate $\downarrow$ |  Time To Goal $\downarrow$ |  Collision Rate $\downarrow$ |  Time To Goal $\downarrow$
Modular Agent | 0.0 | 53.33 | 0.0 | 86.02 | 0.125 | 34.35 | 0.96 | 37.81
Imitation Learning Agent | 0.33 | 27.78 | 0.33 | 22.90 | 0.875 | 19.58 | 0.58 | 29.41

TABLE II: Sim-on-Wheels Benchmark Results: These metrics are computed as
described in Section III-C. All time metrics above are measured in seconds.
Findings: The results show that the modular pipeline is better overall at
avoiding collisions, but the imitation learning pipeline reaches the goal
faster.

##  IV Experiments

The goal of the experiments in this section is to address the following three
crucial questions: (1) Can the Sim-on-Wheels framework, as proposed, be
utilized as a rigorous and comprehensive benchmark for evaluating the
performance of various autonomous stacks? (2) Can we empirically validate the
authenticity of our simulation? (3) To what extent does the onboard simulation
result in an increase in latency?

In this section, we first provide an overview of the hardware platform and the
test track used for our experiments. We then benchmark the performance of two
self-driving agents in various safety-critical scenarios using the Sim-on-
Wheels framework and conduct a comprehensive analysis of their performance. To
quantify the reality gap between the simulation and the real world, we conduct
an empirical analysis of the gaps in sensory data and their impact on the
perception and action output of the onboard autonomy. Finally, we provide
analysis and discussions of our framework.

###  IV-A Real-World Testbed

All of our experiments are carried out on the Polaris GEM e2, a street-legal,
two-seater electric vehicle with a top speed of 25 mph. The sensor stack of
the vehicle includes a Velodyne-16 lidar, a Novatel RTK GNSS+INS unit, a ZED 2
Stereo Camera, and a Delphi ESR 2.5 Radar. The vehicle also supports drive-by-
wire through the PACMod kit, enabling steering, acceleration, and braking
through software. Our experiments utilize the AStuff Spectra 2 [58], an
industrial-grade edge computing platform equipped with an Nvidia A4000 GPU.
This computer is connected to a built-in monitor located on the passenger side
dashboard, facilitating the ease of running and debugging software directly
in-vehicle.

The experiments were carried out in a shared testing track facility, where the
testbed area was secured using cones and traffic tape to restrict public
access and guarantee the safety of all individuals involved. At all times, a
designated safety driver was present behind the wheel of the vehicle, while
another team member was stationed outside the vehicle as a lookout, prepared
to respond in the event of an emergency.

###  IV-B Evaluated Autonomous Agents

We subject two distinct autonomous agents for evaluation using the Sim-on-
Wheels framework: (1) a modular autonomy stack, and (2) an end-to-end
imitation learning stack.

####  IV-B1 Modular autonomy

Our modular autonomy pipeline takes as input the RGB-D image stream and a
coarse planned path. It is composed of four components: detection, tracking,
motion prediction, and rule-based longitudinal planning.

Obstacle detection is split into two modules: static obstacles and dynamic
traffic participants. Both rely on pretrained segmentation models without any
fine-tuning. Static obstacles are detected by applying a pretrained foreground
segmentation model [59]. Dynamic obstacles are detected by applying a
pretrained instance segmentation model [60] to extract instance masks for
pedestrians and cars. Each instance mask is converted to a 3D position by
taking the median $x$ and $y$ coordinates from the associated depth.

At the tracking stage, greedy matching [61] is performed to associate the
latest detected object and existing tracks based on a bird’s eye view. We then
estimate the state (velocity and position in bird’s eye view) through a linear
motion model. Using these velocities and the ego-car’s current speed and
planned trajectory, we predict the positions of each entity at every
$0.2\text{\,}\mathrm{s}$ step up to $10\text{\,}\mathrm{s}$ into the future
and identify potential collisions. At each future time step, we search for
collisions within a fixed collision radius ($3\text{\,}\mathrm{m}$) and travel
distance threshold ($5\text{\,}\mathrm{m}$). This threshold can be increased
to compensate for latency. If a collision is found, we output a desired speed
of zero. Otherwise, we output $2\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$.

####  IV-B2 Imitation learning (IL) agent

We also train an end-to-end neural controller agent using behavior cloning.
The network takes as input the latest eight frames of speed, position, and RGB
image information, which are separated by 0.2 seconds. The controller outputs
a continuous brake command to be executed 0.2s in the future, accounting for
the latency of the simulation and autonomy pipeline.

The network follows a spatial-temporal recurrent neural network architecture
[62, 63]. We use an FCN [64] backbone pre-trained on instance segmentation on
the COCO dataset [65] to encode object-level information. The visual features
are concatenated with other meta-inputs and fed into a gated recurrent unit
(GRU) to incorporate temporal information. The decoder is a multi-layer
perceptron outputting a brake value.

To train the network, diverse real-world data is collected from human driving
where the driver brakes for static objects and jaywalking pedestrians. The
training data contains 195 sequences of static-object braking and
approximately 200 sequences of dynamic jaywalking. The network learns to
minimize the difference between human driving and its own actions using a
$L_{1}$ regression loss, with the Adam optimizer and a learning rate of 5e-4.
Weighted sampling is applied to focus on the relevant samples (e.g. snippets
right before brakes), and data augmentation techniques, such as random color
jittering and random cropping, are used to increase robustness. We also apply
a dropout rate of 0.8 during training.

####  IV-B3 Vehicle controller

The output from both the modular agent and the IL agent is sent to the same
vehicle controller to produce the final vehicle command. The longitudinal
direction is controlled by a proportional-integral (PI) speed controller [66].
Meanwhile, the lateral direction is controlled by the Stanley controller
utilizing a bicycle model [67].

####  IV-B4 Human driver reference

We also benchmark human driver performance within our Sim-on-Wheel system as a
reference. The human driver operates the vehicle, navigating it along the
planned path and avoiding collisions within simulated scenarios by monitoring
real-time augmented video feeds displayed on the GEM vehicle’s screen. To
prevent the human driver from prematurely reacting to the expectation of an
obstacle, we randomized the spawn point of the moving agent, forcing the
driver to react on the fly.

###  IV-C Autonomy Benchmark Results

Tab. II reports the performance metrics of our driving agents using the Sim-
on-Wheels evaluation framework. The qualitative examples of the representative
scenarios are depicted in Fig. 4. Our analysis, based on the evaluation
results and log replays, highlights a few key findings.

Our results indicate that the modular agent generally takes longer to reach
the destination due to false positive detections of obstacles, leading to
intermittent braking. Nevertheless, it is capable of safely reaching the goal
in most cases involving static objects and standard jaywalkers. However, the
jaywalker-with-occlusion scenario presents a challenge for the modular agent,
as there is limited reaction time between the first appearance of the
jaywalker and a potential collision. This results in the failure of the agent
to brake in time. Furthermore, the challenging textures of the occluding walls
causes the perception module to miss detections.

In contrast, our results indicate that the imitation learning agent does not
experience intermittent braking and achieves a shorter time to reach the goal.
Furthermore, it performs acceptably well in the intersection scenario with
vehicles, which was not part of the training data. However, the agent tends to
react late, resulting in higher jaywalker collisions. This may be due to
latency differences during training and onboard deployment. It is important to
note that such drawbacks were not frequently observed during offline
validation, highlighting the importance of real-world vehicle-in-the-loop
testing.

Note that none of the agents, including the human driver (Tab. III), achieve a
zero collision rate across all the scenarios, which highlights the difficulty
of our designed safety-critical scenarios. It is worth mentioning that many of
the test cases, particularly the jaywalking ones, are impractical to test in
the real world due to safety concerns and can only be physically evaluated
with Sim-on-Wheels.

|
---|---

Figure 5: Human Driving Failure Case: The one episode in which the human
driver collided was the variation of Jaywalker with Occlusion scenario that
involved multiple fast jaywalkers. The left image is captured at the braking
point.

| Jaywalker with Occlusion
---|---
Agent Type |  Collision Rate $\downarrow$ |  Time To Goal $\downarrow$
Human | 0.08 | 31.79

TABLE III: Human Driver Benchmark on Sim-on-Wheels. These metrics are computed
as described in Section III-C

Sensor | Perception | Action
---|---|---
PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  Outlier-$\%\downarrow$ | $\text{mIoU}\uparrow$ | $\text{Average Endpoint Offset (m)}\downarrow$ | $\Delta\text{Predicted Brake in }L_{1}\downarrow$
21.8 | 0.833 | 0.108 | 4.9 | 0.839 |  1.09 $\pm$ 0.49 |  0.18 $\pm$ 0.08

TABLE IV: Reality Gap. The reality gap is assessed using three metrics: 1)
sensor image fidelity between reality and simulation, 2) mIoU of perception
algorithm output, and 3) similarity of final actions (brake values). Results
indicate a small reality gap for Sim-on-Wheels, validating its efficacy and
reliability as an evaluation framework.

###  IV-D Reality Gap Analysis

In Fig 3, we qualitatively assess our insertion rendering by comparing real
vs. simulated results. The image pairs appear quite similar overall,
demonstrating the robust realism of the framework. However, some minor
differences do exist, such as incomplete 3D reconstructed shapes, slight
differences in shaded color, and variations in sunlight intensity, shadow
shape, and cloud patterns due to the two images being taken in a windy outdoor
environment at different times.

Tab. IV reports the quantitative measure using the peak signal-to-noise ratio
(PSNR), structural similarity index (SSIM) [68], and a learned perceptual
similarity metric (LPIPS) [69]. The mean absolute error (MAE) of each pixel is
calculated and the percentage of outlier pixels with errors over a threshold
of $25.5$ under the RGB intensity range $(0,255)$ were reported. As indicated
in Tab. IV, the results show that the virtual object insertions exhibit high
fidelity and realism with a low percentage of outlier pixels.

We also evaluate the impact of the reality gap on the performance of our
autonomy pipelines. Firstly, we measure the mean Intersection over Union
(mIoU) between the static obstacle segmentation network’s outputs on real and
simulated images. Although the silhouettes of the obstacles match closely, the
mIoU is not perfect, which is likely due to the sensitivity of the network to
slight variations in color and the background, such as cloud movement. This
limitation is a result of the time required to physically arrange obstacles.

Additionally, we evaluate the action-level reality gap by comparing the
agent’s behavior in real and simulated scenarios. For this, we set the initial
vehicle position, planned path, obstacle type, and position to be identical in
both scenarios, with only the obstacle being different (real object vs its
digital twin). Our experiments show that the modular agent can stop without
collision in both real and Sim-on-Wheels experiments. However, we do observe a
small discrepancy in its actual trajectory, indicating that the reality gap is
not fully closed yet. This may also be due to other factors, such as changes
in background illumination as mentioned above.

###  IV-E Generalization to other environments

|
---|---
Original | Sim-on-Wheels

Figure 6: Sim-on-Wheels on KITTI-360: Two jaywalkers on a street are simulated
on top of KITTI-360 camera data.

We can easily adapt to new environments as long as the scenario is compatible
with the surroundings. As one example, we generalize to KITTI-360 data [70] as
shown Fig. 6.

| Sim-on-Wheels | Autonomy | ROS | Total | Relative %
---|---|---|---|---|---
Offline | 43.5ms | - | - | - | -
Online | 63.5ms | - | 212.8ms | 276.3ms | 22.98
Online + IL | 71.5ms | 102.5ms | 214.4ms | 388.4ms | 18.41
Online + Modular | 98.4ms | 129.5ms | 227.3ms | 455.2ms | 21.62

TABLE V: System Runtime Breakdown. IL stands for imitation learning agent, and
modular for modular agent.

###  IV-F Runtime Analysis

The runtime performance of the Sim-on-Wheels system was evaluated on an
onboard computer equipped with a single Intel Xeon(R) E-2278G CPU @3.40GHz
$\times$16 and a single Nvidia RTX A4000 GPU. The results are presented in
Tab. V. The standalone rendering component of the system processes pre-
recorded sensor data with a frame rate of 23 FPS. Upon integration with ROS,
the frame rate is impacted by the consumption of system resources by hardware
drivers, communication modules, the autonomy agent, and the controller.
Nonetheless, the system is able to publish rendered images at a minimum frame
rate of 10 FPS, even when the autonomy agents run concurrently with the Sim-
on-Wheels.

Our results show that the end-to-end latency from the camera capture time to
vehicle command increases by 18-21% with the integration of Sim-on-Wheels,
which could be mitigated by equipping the vehicle with two onboard GPUs.
Additionally, both autonomy agents can predict future states and actions to
compensate for such latency. Tab. V also reveals that the raw ZED2 stereo
camera streaming, decoding, and post-processing consumes a substantial amount
of time. Despite using separate callback threads to process incoming ROS
messages in parallel, a delay still persists due to high-bitrate camera
streaming. We plan to further investigate a faster sensor-computer interface
to address this issue.

##  V Conclusion

We propose Sim-on-Wheels, a vehicle-in-the-loop framework for evaluating the
performance of autonomous vehicles in real-world scenarios in a safe and
realistic manner. To the best of our knowledge, Sim-on-Wheels is the first
framework of its kind to support the integration of simulation and real-world
testing practices for the safe and realistic evaluation of autonomous
vehicles. Our results demonstrate the versatility and reliability of Sim-on-
Wheels as a framework for evaluating various agents. To further support the
research and development of autonomous driving, we will open-source Sim-on-
Wheels to the community and establish a safe, closed-loop, end-to-end, real-
world benchmark.

## Acknowledgments

This project has been funded by the Amazon Research Award, Nvidia Hardware
Grants, the Insper-Illinois Innovative Grant, and the NCSA Faculty Fellow. We
are grateful for the support of the Center for Autonomy and John M. Hart. We
thank Wei-Chiu Ma and Jason Ren for proofreading.

## References

  * [1] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in _CVPR_ , 2012.
  * [2] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine _et al._ , “Scalability in perception for autonomous driving: Waymo open dataset,” in _CVPR_ , 2020.
  * [3] D. A. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,” _NeurIPS_ , 1988.
  * [4] K. D. Kusano, K. Beatty, S. Schnelle, F. Favaro, C. Crary, and T. Victor, “Collision avoidance testing of the waymo automated driving system,” _arXiv preprint arXiv:2212.08148_ , 2022.
  * [5] “Cruise Driverless Rides,” https://getcruise.com/rides. Accessed at 2023-02-04.
  * [6] “TRC - Transportation Research Center,” https://www.trcpg.com. Accessed at 2023-02-04.
  * [7] “Illinois Autonomous and Connected Track,” https://ict.illinois.edu/i-act/testing-arena. Accessed at 2023-02-04.
  * [8] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: An open urban driving simulator,” in _CoRL_ , 2017.
  * [9] “NVIDIA DRIVE Sim,” https://developer.nvidia.com/drive/drive-constellation. Accessed at 2023-02-04.
  * [10] S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: High-fidelity visual and physical simulation for autonomous vehicles,” in _Field and Service Robotics: Results of the 11th International Conference_ , 2018.
  * [11] S. Manivasagam, S. Wang, K. Wong, W. Zeng, M. Sazanovich, S. Tan, B. Yang, W.-C. Ma, and R. Urtasun, “Lidarsim: Realistic lidar simulation by leveraging the real world,” in _CVPR_ , 2020.
  * [12] Y. Chen, F. Rong, S. Duggal, S. Wang, X. Yan, S. Manivasagam, S. Xue, E. Yumer, and R. Urtasun, “Geosim: Realistic video simulation via geometry-aware composition for self-driving,” in _CVPR_ , 2021.
  * [13] S. Tan, K. Wong, S. Wang, S. Manivasagam, M. Ren, and R. Urtasun, “Scenegen: Learning to generate realistic traffic scenes,” in _CVPR_ , 2021.
  * [14] A. Amini, T.-H. Wang, I. Gilitschenski, W. Schwarting, Z. Liu, S. Han, S. Karaman, and D. Rus, “Vista 2.0: An open, data-driven simulator for multimodal sensing and policy learning for autonomous vehicles,” in _ICRA_ , 2022.
  * [15] J. Wang, S. Manivasagam, Y. Chen, Z. Yang, I. A. Bârsan, A. J. Yang, W.-C. Ma, and R. Urtasun, “Cadsim: Robust and scalable in-the-wild 3d reconstruction for controllable sensor simulation,” in _CoRL_ , 2022.
  * [16] S. Thrun, M. Montemerlo, H. Dahlkamp, D. Stavens, A. Aron, J. Diebel, P. Fong, J. Gale, M. Halpenny, G. Hoffmann _et al._ , “Stanley: The robot that won the darpa grand challenge,” _Journal of field Robotics_ , 2006.
  * [17] J. Daudelin, G. Jing, T. Tosun, M. Yim, H. Kress-Gazit, and M. Campbell, “An integrated system for perception-driven autonomy with modular robots,” _Science Robotics_ , 2018.
  * [18] A. Tampuu, T. Matiisen, M. Semikin, D. Fishman, and N. Muhammad, “A survey of end-to-end driving: Architectures and training methods,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * [19] J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt _et al._ , “Towards fully autonomous driving: Systems and algorithms,” in _IV_ , 2011.
  * [20] I. A. Barsan, S. Wang, A. Pokrovsky, and R. Urtasun, “Learning to localize using a lidar intensity map,” in _CoRL_ , 2018.
  * [21] X. Chen, H. Ma, J. Wan, B. Li, and T. Xia, “Multi-view 3d object detection network for autonomous driving,” in _CVPR_ , 2017.
  * [22] Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Y. Qiao, and J. Dai, “Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers,” in _ECCV_ , 2022.
  * [23] M. Liang, B. Yang, S. Wang, and R. Urtasun, “Deep continuous fusion for multi-sensor 3d object detection,” in _ECCV_ , 2018.
  * [24] A. Kelly and B. Nagy, “Reactive nonholonomic trajectory generation via parametric optimal control,” _IJRR_ , 2003.
  * [25] E. F. Camacho and C. B. Alba, _Model predictive control_. Springer science & business media, 2013.
  * [26] M. A. Johnson and M. H. Moradi, _PID control_. Springer, 2005.
  * [27] U. Muller, J. Ben, E. Cosatto, B. Flepp, and Y. Cun, “Off-road obstacle avoidance through end-to-end learning,” _NeurIPS_ , 2005.
  * [28] W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang, S. Casas, and R. Urtasun, “End-to-end interpretable neural motion planner,” in _CVPR_ , 2019.
  * [29] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang _et al._ , “End to end learning for self-driving cars,” _arXiv preprint arXiv:1604.07316_ , 2016.
  * [30] M. Bojarski, P. Yeres, A. Choromanska, K. Choromanski, B. Firner, L. Jackel, and U. Muller, “Explaining how a deep neural network trained with end-to-end learning steers a car,” _arXiv preprint arXiv:1704.07911_ , 2017.
  * [31] W. Zeng, S. Wang, R. Liao, Y. Chen, B. Yang, and R. Urtasun, “Dsdnet: Deep structured self-driving network,” in _ECCV_ , 2020.
  * [32] N. T. S. Board, “Collision between vehicle controlled by developmental automated driving system and pedestrian,” _NTSB Highway Accident Report_ , 2019.
  * [33] S. Höfer, K. Bekris, A. Handa, J. C. Gamboa, M. Mozifian, F. Golemo, C. Atkeson, D. Fox, K. Goldberg, J. Leonard _et al._ , “Sim2real in robotics and automation: Applications and challenges,” _IEEE transactions on automation science and engineering_ , 2021.
  * [34] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-real transfer of robotic control with dynamics randomization,” in _ICRA_ , 2018.
  * [35] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray _et al._ , “Learning dexterous in-hand manipulation,” _IJRR_ , 2020.
  * [36] S. Suo, S. Regalado, S. Casas, and R. Urtasun, “Trafficsim: Learning to simulate realistic multi-agent behaviors,” in _CVPR_ , 2021.
  * [37] L. Feng, Q. Li, Z. Peng, S. Tan, and B. Zhou, “Trafficgen: Learning to generate diverse and realistic traffic scenarios,” _arXiv preprint arXiv:2210.06609_ , 2022.
  * [38] Q. Sun, X. Huang, B. C. Williams, and H. Zhao, “Intersim: Interactive traffic simulation via explicit relation modeling,” in _IROS_ , 2022.
  * [39] T. Bokc, M. Maurer, and G. Farber, “Validation of the vehicle in the loop (vil); a milestone for the simulation of driver assistance systems,” in _IV_ , 2007.
  * [40] A. Albers and T. Düser, “Implementation of a vehicle-in-the-loop development and validation platform,” in _FISITA World automotive congress_ , 2010.
  * [41] M. F. Drechsler, V. Sharma, F. Reway, C. Schütz, and W. Huber, “Dynamic vehicle-in-the-loop: A novel method for testing automated driving functions,” _SAE International Journal of Connected and Automated Vehicles_ , 2022.
  * [42] W. Hoenig, C. Milanes, L. Scaria, T. Phan, M. Bolas, and N. Ayanian, “Mixed reality for robotics,” in _IROS_ , 2015.
  * [43] M. Funk Drechsler, J. B. Peintner, G. Seifert, W. Huber, and A. Riener, “Mixed reality environment for testing automated vehicle and pedestrian interaction,” in _13th International Conference on Automotive User Interfaces and Interactive Vehicular Applications_ , 2021.
  * [44] T. Genevois, J.-B. Horel, A. Renzaglia, and C. Laugier, “Augmented reality on lidar data: Going beyond vehicle-in-the-loop for automotive software validation,” in _IV_ , 2022.
  * [45] C. Hildebrandt and S. Elbaum, “World-in-the-loop simulation for autonomous systems validation,” in _ICRA_ , 2021.
  * [46] W. G. Najm, J. D. Smith, M. Yanagisawa _et al._ , “Pre-crash scenario typology for crash avoidance research,” United States. National Highway Traffic Safety Administration, Tech. Rep., 2007.
  * [47] “Sketchfab search,” https://sketchfab.com/search?q=bus&type=models. Accessed at 2023-02-04.
  * [48] “Realityscan,” https://www.capturingreality.com/. Accessed at 2023-02-04.
  * [49] Adobe, “Mixamo,” https://www.mixamo.com. Accessed at 2023-02-04.
  * [50] J.-F. Lalonde, D. Hoiem, A. A. Efros, C. Rother, J. Winn, and A. Criminisi, “Photo clip art,” _TOG_ , 2007.
  * [51] Z. Liao, K. Karsch, H. Zhang, and D. Forsyth, “An Approximate Shading Model with Detail Decomposition for Object Relighting,” _IJCV_ , 2018.
  * [52] K. Karsch, V. Hedau, D. Forsyth, and D. Hoiem, “Rendering synthetic objects into legacy photographs,” in _SIGGRAPH Asia_ , 2011.
  * [53] K. Karsch, K. Sunkavalli, S. Hadap, N. Carr, H. Jin, R. Fonte, M. Sittig, and D. Forsyth, “Automatic scene inference for 3d object compositing,” _ACM Trans. Graph._ , 2014.
  * [54] Z. Liao, A. Farhadi, Y. Wang, I. Endres, and D. Forsyth, “Building a dictionary of image fragments,” in _CVPR_ , 2012.
  * [55] S. Dombi. Moderngl, high performance python bindings for opengl 3.3+. https://github.com/moderngl/moderngl. Accessed at 2023-02-04.
  * [56] R. L. Cook and K. E. Torrance, “A reflectance model for computer graphics,” _ToG_ , 1982.
  * [57] L. Williams, “Casting curved shadows on curved surfaces,” in _Proceedings of the 5th annual conference on Computer graphics and interactive techniques_ , 1978.
  * [58] “Astuff spectra 2,” https://autonomoustuff.com/products/astuff-spectra-2. Accessed at 2023-02-04.
  * [59] J. Xu, Z. Xiong, and S. P. Bhattacharyya, “Pidnet: A real-time semantic segmentation network inspired from pid controller,” _arXiv preprint arXiv:2206.02066_ , 2022.
  * [60] T. Cheng, X. Wang, S. Chen, W. Zhang, Q. Zhang, C. Huang, Z. Zhang, and W. Liu, “Sparse instance activation for real-time instance segmentation,” in _CVPR_ , 2022.
  * [61] M. Dyer, A. Frieze, and B. Pittel, “The average performance of the greedy matching algorithm,” _The Annals of Applied Probability_ , 1993.
  * [62] Y. Xia, J. Kim, J. Canny, K. Zipser, T. Canas-Bajo, and D. Whitney, “Periphery-fovea multi-resolution driving model guided by human attention,” in _WACV_ , 2020.
  * [63] R. L. Abduljabbar, H. Dia, P.-W. Tsai, and S. Liyanage, “Short-term traffic forecasting: An lstm network for spatial-temporal speed prediction,” _Future Transportation_ , 2021.
  * [64] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _CVPR_ , 2015.
  * [65] T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, P. Doll’a r, and C. L. Zitnick, “Microsoft COCO: common objects in context,” _CoRR_ , 2014.
  * [66] L. Zheng, “A practical guide to tune of proportional and integral (pi) like fuzzy controllers,” in _[1992 Proceedings] IEEE International Conference on Fuzzy Systems_ , 1992.
  * [67] P. Polack, F. Altché, B. d’Andréa Novel, and A. de La Fortelle, “The kinematic bicycle model: A consistent model for planning feasible trajectories for autonomous vehicles?” in _IV_ , 2017.
  * [68] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _TIP_ , 2004\.
  * [69] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in _CVPR_ , 2018\.
  * [70] Y. Liao, J. Xie, and A. Geiger, “KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d,” _Pattern Analysis and Machine Intelligence (PAMI)_ , 2022.