# Beating the 3 dB Limit for Intracavity Squeezing
and Its Application to Nondemolition Qubit Readout

Wei Qin  Theoretical Quantum Physics Laboratory, RIKEN Cluster for Pioneering
Research, Wako-shi, Saitama 351-0198, Japan  Adam Miranowicz  Theoretical
Quantum Physics Laboratory, RIKEN Cluster for Pioneering Research, Wako-shi,
Saitama 351-0198, Japan  Institute of Spintronics and Quantum Information,
Faculty of Physics, Adam Mickiewicz University, 61-614 Poznań, Poland  Franco
Nori  Theoretical Quantum Physics Laboratory, RIKEN Cluster for Pioneering
Research, Wako-shi, Saitama 351-0198, Japan  RIKEN Center for Quantum
Computing, Wako-shi, Saitama 351-0198, Japan  Department of Physics, The
University of Michigan, Ann Arbor, Michigan 48109-1040, USA

###### Abstract

While the squeezing of a propagating field can, in principle, be made
arbitrarily strong, the cavity-field squeezing is subject to the well-known
$3$ dB limit, and thus has limited applications. Here, we propose the use of a
fully quantum degenerate parametric amplifier (DPA) to beat this squeezing
limit. Specifically, we show that by simply applying a two-tone driving to the
signal mode, the pump mode can, counterintuitively, be driven by the photon
loss of the signal mode into a squeezed steady state with, in principle, an
arbitrarily high degree of squeezing. Furthermore, we demonstrate that this
intracavity squeezing can increase the signal-to-noise ratio of longitudinal
qubit readout exponentially with the degree of squeezing. Correspondingly, an
improvement of the measurement error by many orders of magnitude can be
achieved even for modest parameters. In stark contrast, using intracavity
squeezing of the semiclassical DPA cannot practically increase the signal-to-
noise ratio and thus improve the measurement error. Our results extend the
range of applications of DPAs and open up new opportunities for modern quantum
technologies.

_Introduction.—_ Squeezed states of light Drummond and Ficek (2004) form a
fundamental building block in modern quantum technologies ranging from quantum
metrology Schnabel (2017); Lawrie  _et al._ (2019) to quantum information
processing Braunstein and van Loock (2005); Weedbrook  _et al._ (2012). In
particular, squeezing of a propagating field can, in principle, be made
arbitrarily strong, due to destructive interference between the reflected
input field and the transmitted cavity field; e.g., the squeezing of up to 15
dB has been experimentally achieved Vahlbruch  _et al._ (2016). Such a
propagating-field squeezing has been widely used for, e.g., gravitational-wave
detection et al. (LIGO Scientific Collaboration); Aasi  _et al._ (2013); Grote
_et al._ (2013), mechanical cooling Asjad  _et al._ (2016); Clark  _et al._
(2017), nondemolition qubit readout Barzanjeh  _et al._ (2014); Didier  _et
al._ (2015a, b); Govia and Clerk (2017), and even demonstrating quantum
supremacy Zhong  _et al._ (2020, 2021). However, these applications inherently
suffer from transmission and injection losses, which are a major obstacle to
using extremely fragile squeezed states. To address this problem, exploiting
intracavity squeezing (i.e., squeezing of a cavity field) offers a promising
route.

Figure 1: (a) Schematic of our proposal with a fully quantum DPA. We use two
cavities to represent the pump mode $\hat{a}_{p}$ (frequency $\omega_{p}$,
loss rate $\kappa_{p}$) and the signal mode $\hat{a}_{s}$ (frequency
$\omega_{s}$, loss rate $\kappa_{s}$). The single-photon parametric coupling
between them has a strength $g$. A driving tone at frequency $\omega_{d}$ is
applied to the pump mode and, simultaneously, the signal mode is driven by the
other two tones at frequencies $\omega_{\pm}^{\rm d}$. (b) Time evolution of
the squeezing parameter $\xi^{2}_{p}$ for $G_{+}/G_{-}=0.5$ and $0.7$. We
assumed that $\Delta_{s}=100g$, $\Delta_{p}=0.1\Delta_{s}$, $\Omega_{\rm
2pd}=0.05\Delta_{s}$, $\kappa_{s}=100\kappa_{p}=0.4g$, and $G_{-}=g_{0}$.
Curves are the effective predictions, while symbols are the exact results. (c)
Steady-state squeezing parameter
$\left(\xi_{p}^{2}\right)_{\mathchoice{\raisebox{1.2pt}{$\displaystyle\rm
ss$}}{\raisebox{1.2pt}{$\textstyle\rm ss$}}{\raisebox{1.2pt}{$\scriptstyle\rm
ss$}}{\raisebox{1.2pt}{$\scriptscriptstyle\rm ss$}}}$ versus the cooperativity
$\mathcal{C}$ for $\kappa_{s}=100\kappa_{p}$, and for $G_{+}/G_{-}=0.8$,
$0.9$, and $0.99$. In (b) and (c), the gray shaded areas refer to the regime
below the $3$ dB limit.

To date, intracavity squeezing has been applied, e.g., to cool mechanical
resonators Huang and Agarwal (2009); Asjad  _et al._ (2019); Lau and Clerk
(2020), to enhance light-matter interactions Lü  _et al._ (2015); Zeytinoğlu
_et al._ (2017); Qin  _et al._ (2018); Leroux  _et al._ (2018); Ge  _et al._
(2019); Li  _et al._ (2020); Burd  _et al._ (2021); Tang  _et al._ (2022), to
improve high-precision measurements Zagoskin  _et al._ (2008); Peano  _et al._
(2015); Eddins  _et al._ (2019), and to generate nonclassical states Krippner
_et al._ (1994); Munro and Reid (1995); Groszkowski  _et al._ (2020); Chen
_et al._ (2021); Qin  _et al._ (2021). Despite such developments, the range
and quality of applications of intracavity squeezing are still largely limited
by the fact that quantum noise of a cavity field cannot be reduced below one-
half of the zero-point fluctuations in the steady state Drummond  _et al._
(1981); Milburn and Walls (1981); Collett and Gardiner (1984), i.e., the 3 dB
limit. However, how to beat this limit has so far remained challenging,
although for more complicated mechanical oscillators, the steady-state
squeezing beyond 3 dB has been widely demonstrated both theoretically Kronwald
_et al._ (2013); Woolley and Clerk (2014); Kustura  _et al._ (2022) and
experimentally Szorkovszky  _et al._ (2013); Lei  _et al._ (2016). The reason
for the 3 dB limit of intracavity squeezing is the cavity photon loss, which
is always present, destroys the essence of squeezing, i.e., two-photon
correlations. In this manuscript, we show that, if such a photon loss is
exploited as a resource, a strong steady-state intracavity squeezing can be
achieved.

In our approach, we consider a fully quantum DPA, where both pump and signal
modes are quantized. We show that a strong photon loss of the signal mode can
steer the pump mode into a squeezed steady state, with a noise level reduced
far beyond $3$ dB. In this way, an arbitrarily strong steady-state squeezing
of the pump mode can, in principle, be achieved. Note that optical experiments
performed in the 1990s (see, e.g., Paschotta  _et al._ (1994); Ralph  _et al._
(1995)) demonstrated bright squeezing of the pump mode (i.e., the second-
harmonic mode) by driving the signal mode (i.e., the fundamental mode).
However, it was achieved for output squeezing only, not for intracavity
squeezing.

To beat the $3$ dB limit of intracavity squeezing, a theoretical approach,
which requires a fast modulation of the coupling between the cavity and its
environment, has been proposed Didier  _et al._ (2014); and very recently, an
experimental demonstration with three microwave modes coupled via a specific
Josephson ring modulator was reported in Ref. Dassonneville  _et al._ (2021).
In contrast, our approach relies only on common degenerate parametric
amplification processes, and therefore is more compatible with current quantum
technologies based on parametric amplification. More remarkably, we show that
only a two-tone driving, if applied to the signal mode, can result in a strong
steady-state squeezing for the pump mode. This is rather counterintuitive;
indeed, common sense suggests that, as mentioned above, the steady-state
intracavity squeezing of a DPA is usually limited to $3$ dB. We note that
quantum intracavity noise reduction can also be realized via squeezing of
photon-number fluctuations, corresponding to the sub-Poissonian photon-number
statistics or photon antibunching (see, e.g., the early predictions Ritsch
(1990); Teich and Saleh (1988) and very recent demonstrations of 3 dB
squeezing like in Carroll  _et al._ (2021)).

Fast and high-fidelity nondemolition qubit readout is a prerequisite for
quantum error correction Schindler  _et al._ (2011); Kelly  _et al._ (2015)
and fault-tolerant quantum computation Raussendorf and Harrington (2007);
Gambetta  _et al._ (2017). Using squeezed light to improve such a readout is a
long-standing goal Barzanjeh  _et al._ (2014); Didier  _et al._ (2015b, a);
Blais  _et al._ (2021). However, the simplest strategy, i.e., dispersive qubit
readout Blais  _et al._ (2021); Krantz  _et al._ (2019), induces a qubit-
state-dependent rotation of squeezing, such that the amplified noise in the
antisqueezed quadrature is introduced into the signal quadrature, ultimately
limiting the improvement of the signal-to-noise ratio (SNR). Thus, related
experimental demonstrations in this context have remained elusive. Until
recently, an improvement, enabled by injecting squeezed light into a cavity,
was realized Eddins  _et al._ (2018) for longitudinal qubit readout Didier
_et al._ (2015b); Blais  _et al._ (2021); Gu  _et al._ (2017); Touzard  _et
al._ (2019); Ikonen  _et al._ (2019), which can enable much shorter
measurement times than the dispersive readout. However, due to transmission
and injection losses, more than half of the amount of squeezing is lost, and
consequently the reported SNR is increased only by $\simeq 25\%$.

Here, we propose to apply our strong intracavity squeezing to longitudinal
qubit readout, thus avoiding transmission and injection losses. We demonstrate
that the SNR can be increased exponentially, and the measurement error is
improved by many orders of magnitude for modest parameters. In sharp contrast,
intracavity squeezing of the semiclassical DPA cannot significantly improve
the SNR during a practically feasible measurement time, even though squeezing
of the output field is very strong. Our main results are summarized in Table I
in sup .

_Physical model.—_ A fully quantum DPA, as shown in Fig. 1(a), consists of a
pump mode $\hat{a}_{p}$ and a signal mode $\hat{a}_{s}$, which are coupled
through a single-photon parametric coupling of strength $g$. We assume that
the pump mode is driven by a tone of frequency $\omega_{d}$ and amplitude
$\mathcal{E}_{d}$, and, additionally, the signal mode is subject to a two-tone
driving of frequencies $\omega_{\pm}^{\rm d}$ and amplitudes
$\mathcal{E}_{\pm}$. The corresponding Hamiltonian in a frame rotating at
$\omega_{d}$ is $\hat{H}=\hat{H}_{0}+\hat{H}_{\rm 2td}$, with

$\displaystyle\hat{H}_{0}=\;$
$\displaystyle\Delta_{p}\hat{a}_{p}^{\dagger}\hat{a}_{p}+\Delta_{s}\hat{a}_{s}^{\dagger}\hat{a}_{s}$
$\displaystyle+g\left(\hat{a}_{s}^{2}\hat{a}_{p}^{\dagger}+{\rm
H.c.}\right)+\left(\mathcal{E}_{d}\hat{a}^{\dagger}_{p}+{\rm H.c.}\right),$
(1) $\displaystyle\hat{H}_{\rm 2td}=\;$ $\displaystyle\Omega_{\rm
2td}\left(t\right)\hat{a}_{s}^{\dagger}+{\rm H.c.},$ (2)

where $\Delta_{p}=\omega_{p}-\omega_{d}$,
$\Delta_{s}=\omega_{s}-\omega_{d}/2$, $\Omega_{\rm
2td}\left(t\right)=\mathcal{E}_{-}\exp\left(-i\omega_{-}t\right)+\mathcal{E}_{+}\exp\left(-i\omega_{+}t\right)$.
Here, $\omega_{p}$, $\omega_{s}$ are the resonance frequencies of the pump and
signal modes, and $\omega_{\pm}=\omega_{\pm}^{\rm d}-\omega_{d}/2$. We
describe photon losses with the Lindblad dissipator
$\mathcal{L}\left(\hat{o}\right)\hat{\rho}=\hat{o}\hat{\rho}\hat{o}^{\dagger}-\frac{1}{2}\left(\hat{o}^{\dagger}\hat{o}\hat{\rho}+\hat{\rho}\hat{o}^{\dagger}\hat{o}\right)$,
so that the system dynamics is determined by the master equation
$\dot{\hat{\rho}}=-i\left[\hat{H},\hat{\rho}\right]+\kappa_{p}\mathcal{L}\left(\hat{a}_{p}\right)\hat{\rho}+\kappa_{s}\mathcal{L}\left(\hat{a}_{s}\right)\hat{\rho}$,
where $\kappa_{p}$ and $\kappa_{s}$ are the photon-loss rates. Upon
introducing the displacement transformation
$\hat{a}_{p}\rightarrow\hat{a}_{p}+\alpha_{p}^{\rm d}$, where $\alpha_{p}^{\rm
d}=\mathcal{E}_{d}/\left(i\kappa_{p}/2-\Delta_{p}\right)$, the Hamiltonian
$\hat{H}_{0}$ becomes
$\hat{H}_{0}=\Delta_{p}\hat{a}_{p}^{\dagger}\hat{a}_{p}+\hat{H}_{\rm
2pd}+\hat{V}$. Here,

$\displaystyle\hat{H}_{\rm 2pd}=\;$
$\displaystyle\Delta_{s}\hat{a}_{s}^{\dagger}\hat{a}_{s}+\Omega_{\rm
2pd}\left(\hat{a}_{s}^{2}+{\rm H.c.}\right),$ (3) $\displaystyle\hat{V}=\;$
$\displaystyle g\left(\hat{a}_{s}^{2}\hat{a}_{p}^{\dagger}+{\rm H.c.}\right),$
(4)

where $\Omega_{\rm 2pd}=g\alpha_{p}^{\rm d}$ can be viewed as the strength of
a two-photon driving of the mode $\hat{a}_{s}$. We have assumed, for
simplicity, that $\alpha_{p}^{\rm d}$ is real.

Since the single-photon coupling $g$ is usually weak, the most studied regime
of the DPA is for $\alpha_{p}^{\rm d}\gg 1$. It is then standard to drop
$\hat{V}$, leaving only $\hat{H}_{\rm 2pd}$. In this case, the pump mode is
treated as a classical field, and the DPA is referred to as semiclassical. For
such a semiclassical DPA, the signal mode cannot be squeezed above 3 dB, even
with nonlinear corrections arising from the coupling $\hat{V}$ Milburn and
Walls (1981); Chaturvedi  _et al._ (2002); sup . The reason for this moderate
squeezing is the photon loss of the signal mode. That is, the leakage of
single photons of some correlated photon pairs injected by the two-photon
driving $\Omega_{\rm 2pd}$ causes a partial loss of two-photon correlations,
and thus of intracavity squeezing. However, as demonstrated below, the photon
loss of the signal mode, when turned from a noise source into a resource via
reservoir engineering, can steer a quantized pump mode into a squeezed steady
state. More importantly, this photon loss can strongly suppress the
detrimental effect of the photon loss of the pump mode on squeezing,
ultimately leading to a strong steady-state intracavity squeezing.

_Squeezing far beyond 3 dB.—_ Recently, it has been shown experimentally that
the available single-photon coupling $g$ can range from tens of kHz to tens of
MHz Leghtas  _et al._ (2015); Touzard  _et al._ (2018); Lescanne  _et al._
(2020); Chang  _et al._ (2020); Vrajitoarea  _et al._ (2020); Guo  _et al._
(2016); Bruch  _et al._ (2019); Wang  _et al._ (2021); Nehra  _et al._ (2022).
These advances allow one to consider the effect of the coupling $\hat{V}$,
e.g., two-photon loss Leghtas  _et al._ (2015); Touzard  _et al._ (2018);
Lescanne  _et al._ (2020); Everitt  _et al._ (2014); Mirrahimi  _et al._
(2014); Sun  _et al._ (2019a, b). We here focus on the case of $\Delta_{s}\neq
0$, and introduce a signal Bogoliubov mode,
$\hat{\beta}_{s}=\hat{a}_{s}\cosh\left(r_{s}\right)+\hat{a}_{s}^{\dagger}\sinh\left(r_{s}\right)$,
with $\tanh\left(2r_{s}\right)=2\Omega_{\rm 2pd}/\Delta_{s}$. The Hamiltonian
$\hat{H}_{\rm 2pd}$ is then diagonalized, yielding $\hat{H}_{\rm
2pd}=\Lambda_{s}\hat{\beta}_{s}^{\dagger}\hat{\beta}_{s}$, where
$\Lambda_{s}=\sqrt{\Delta_{s}^{2}-4\Omega_{\rm 2pd}^{2}}$. Likewise, the
coupling $\hat{V}$ and the two-tone driving $\hat{H}_{\rm 2td}$ become

$\displaystyle\hat{V}=\;$ $\displaystyle
g_{0}\hat{\beta}_{s}^{\dagger}\hat{\beta}_{s}\left(\hat{a}_{p}+\hat{a}_{p}^{\dagger}\right)+\hat{R}_{1}+\hat{R}_{1}^{\dagger},$
(5) $\displaystyle\hat{H}_{\rm 2td}=\;$ $\displaystyle\Omega_{\rm
2td}\left(t\right)\cosh\left(r_{s}\right)\hat{\beta}_{s}^{\dagger}+\hat{R}_{2}+{\rm
H.c.},$ (6)

where
$\hat{R}_{1}=g\left[\cosh^{2}\left(r_{s}\right)\hat{\beta}_{s}^{2}+\sinh^{2}\left(r_{s}\right)\hat{\beta}_{s}^{\dagger
2}\right]\hat{a}_{p}^{\dagger}$, $\hat{R}_{2}=-\Omega_{\rm
2td}\left(t\right)\sinh\left(r_{s}\right)\hat{\beta}_{s}$, and
$g_{0}=-g\sinh\left(2r_{s}\right)$. We further assume the limit
$\left\\{g,\Omega_{\rm 2pd},\Delta_{p}\right\\}\ll\Delta_{s}$, such that
$r_{s}\ll 1$, and both $\hat{R}_{1}$ and $\hat{R}_{2}$ can be dropped as high-
frequency components (see sup ), yielding

$\displaystyle\hat{V}\simeq\;$ $\displaystyle
g_{0}\hat{\beta}_{s}^{\dagger}\hat{\beta}_{s}\left(\hat{a}_{p}+\hat{a}_{p}^{\dagger}\right),$
(7) $\displaystyle\hat{H}_{\rm 2td}\simeq\;$
$\displaystyle\cosh\left(r_{s}\right)\Omega_{\rm
2td}\left(t\right)\hat{\beta}_{s}^{\dagger}+{\rm H.c.}$ (8)

Equations (7), (8) are reminiscent of the two-tone driven radiation-pressure
interaction in cavity optomechanics Aspelmeyer  _et al._ (2014). With such an
interaction, the cavity photon loss can stabilize a strong squeezing of
mechanical motion Kronwald  _et al._ (2013); Woolley and Clerk (2014); Wollman
_et al._ (2015); Pirkkalainen  _et al._ (2015); Lei  _et al._ (2016);
Ockeloen-Korppi  _et al._ (2018). Here, we harness a similar mechanism, and
assume that $\omega_{\pm}=\Lambda_{s}\pm\Delta_{p}$, so that the mode
$\hat{\beta}_{s}$ is coupled to a pump Bogoliubov mode,
$\hat{\beta}_{p}=\hat{a}_{p}\cosh\left(r_{p}\right)+\hat{a}_{p}^{\dagger}\sinh\left(r_{p}\right)$,
through the effective Hamiltonian sup ,

$\displaystyle\hat{H}_{\rm
eff}=\mathcal{G}\left(\hat{\beta}_{p}\hat{\beta}_{s}^{\dagger}+\hat{\beta}_{p}^{\dagger}\hat{\beta}_{s}\right).$
(9)

Here, $\tanh(r_{p})=G_{+}/G_{-}$ and $\mathcal{G}=\sqrt{G_{-}^{2}-G_{+}^{2}}$.
We have defined $G_{\pm}=g_{0}\alpha^{\pm}_{s}$, where $\alpha_{s}^{\pm}$
(given in sup ) are the field amplitudes of the mode $\hat{\beta}_{s}$ induced
by the two-tone driving $\Omega_{\rm 2td}$, and for simplicity both have been
assumed to be real.

Furthermore, we have
$\mathcal{L}\left(\hat{a}_{s}\right)\hat{\rho}\simeq\mathcal{L}\left(\hat{\beta}_{s}\right)\hat{\rho}$
for $r_{s}\ll 1$, and the system dynamics can thus be described with the
effective master equation

$\dot{\hat{\rho}}=-i\left[\hat{H}_{\rm
eff},\hat{\rho}\right]+\kappa_{p}\mathcal{L}\left(\hat{a}_{p}\right)\hat{\rho}+\kappa_{s}\mathcal{L}(\hat{\beta}_{s})\hat{\rho}.$
(10)

It is seen that for a large $\kappa_{s}$, the photon loss of the mode
$\hat{\beta}_{s}$ can cool the mode $\hat{\beta}_{p}$ into the ground state,
corresponding to the squeezed vacuum state of the mode $\hat{a}_{p}$, which
can theoretically have an arbitrary degree of squeezing. Such a squeezed
steady state is unique, and can be reached from any state of the mode
$\hat{a}_{p}$. The reason is that any state of the mode $\hat{a}_{p}$ can be
expressed in terms of the ground and excited states of the mode
$\hat{\beta}_{p}$, but of these, all the excited states are depopulated by the
photon loss of the mode $\hat{\beta}_{s}$ in the steady state. This initial-
state independence enables the detrimental effect of the photon loss of the
mode $\hat{a}_{p}$ on squeezing to be strongly suppressed as long as
$\kappa_{s}\gg\kappa_{p}$ (see sup  for more details), consequently, forming a
strong steady-state squeezing for the mode $\hat{a}_{p}$. During the formation
of this squeezing, any odd photon-number state of the mode $\hat{a}_{p}$ is
reached by two different transitions, which are induced by the two-tone
driving $\Omega_{\rm 2td}$. Achieving a desired steady-state squeezing, i.e.,
a superposition of only even photon-number states, requires destructive
interference between these two transitions to cancel out the population of all
the odd photon-number states.

Figure 2: (a) SNR improvement, i.e., ${\rm SNR}_{z}^{\rm fq}/{\rm
SNR}_{z}^{\rm std}$, versus the degree $r_{p}$ of intracavity squeezing for
different values of the DPA cooperativity: $\mathcal{C}=5$, $10$, $30$, and
$\infty$. An exponential improvement can be obtained for
$\mathcal{C}\gg\exp\left(2r_{p}\right)/4$. (b) SNR and (c) measurement error
versus the measurement time. The solid and dashed curves correspond to the
longitudinal readout using intracavity squeezing of the fully quantum
($r_{p}=2$, $\mathcal{C}=5$) and semiclassical ($r_{\rm out}^{\rm sc}=2$)
DPAs, respectively, while the dash-dotted curves are results of the standard
longitudinal readout with no squeezing. The green shaded area represents the
experimentally most interesting regime. In (b) and (c), all the parameters are
the same except that $\chi_{z}=\kappa$ in (c). (d) SNR (left axis) and
measurement error (right axis) versus $\mathcal{C}$ in the fully quantum-DPA
case for $\tau=1/\kappa$. Other parameters are the same as in (c).

To quantify the degree of squeezing, we use the squeezing parameter Ma  _et
al._ (2011),

$\xi^{2}_{p}=1+2\left(\mbox{$\langle\hat{a}_{p}^{\dagger}\hat{a}_{p}\rangle$}-\left|\mbox{$\langle\hat{a}_{p}\hat{a}_{p}\rangle$}\right|\right).$
(11)

Its time evolution is plotted in Fig. 1(b). Specifically, we compare the
effective and exact results, and show an excellent agreement between them.
Therefore, the effective master equation in Eq. (10) can be used to predict
some larger squeezing by deriving the steady-state squeezing parameter,

$\displaystyle\left(\xi_{p}^{2}\right)_{\mathchoice{\raisebox{1.2pt}{$\displaystyle\rm
ss$}}{\raisebox{1.2pt}{$\textstyle\rm ss$}}{\raisebox{1.2pt}{$\scriptstyle\rm
ss$}}{\raisebox{1.2pt}{$\scriptscriptstyle\rm
ss$}}}=\frac{1+4\mathcal{C}\exp\left(-2r_{p}\right)}{1+4\mathcal{C}},$ (12)

where $\mathcal{C}=\mathcal{G}^{2}/\left(\kappa_{s}\kappa_{p}\right)$ is the
cooperativity of the DPA. In Fig. 1(c),
$\left(\xi_{p}^{2}\right)_{\mathchoice{\raisebox{1.2pt}{$\displaystyle\rm
ss$}}{\raisebox{1.2pt}{$\textstyle\rm ss$}}{\raisebox{1.2pt}{$\scriptstyle\rm
ss$}}{\raisebox{1.2pt}{$\scriptscriptstyle\rm ss$}}}$ is plotted versus
$\mathcal{C}$. For realistic parameters of $\kappa_{s}=100\kappa_{p}$, we find
that a modest ratio $G_{+}/G_{-}$ can keep
$\left(\xi_{p}^{2}\right)_{\mathchoice{\raisebox{1.2pt}{$\displaystyle\rm
ss$}}{\raisebox{1.2pt}{$\textstyle\rm ss$}}{\raisebox{1.2pt}{$\scriptstyle\rm
ss$}}{\raisebox{1.2pt}{$\scriptscriptstyle\rm ss$}}}$ above $3$ dB even for
$\mathcal{C}\simeq 0.4$. Moreover,
$\left(\xi_{p}^{2}\right)_{\mathchoice{\raisebox{1.2pt}{$\displaystyle\rm
ss$}}{\raisebox{1.2pt}{$\textstyle\rm ss$}}{\raisebox{1.2pt}{$\scriptstyle\rm
ss$}}{\raisebox{1.2pt}{$\scriptscriptstyle\rm ss$}}}$ increases as
$\mathcal{C}$, and ultimately reaches its maximum value,

$\left(\xi_{p}^{2}\right)_{\mathchoice{\raisebox{1.2pt}{$\displaystyle\rm
ss$}}{\raisebox{1.2pt}{$\textstyle\rm ss$}}{\raisebox{1.2pt}{$\scriptstyle\rm
ss$}}{\raisebox{1.2pt}{$\scriptscriptstyle\rm
ss$}}}^{\mathchoice{\raisebox{-1.2pt}{$\displaystyle\rm
max$}}{\raisebox{-1.2pt}{$\textstyle\rm
max$}}{\raisebox{-1.2pt}{$\scriptstyle\rm
max$}}{\raisebox{-1.2pt}{$\scriptscriptstyle\rm
max$}}}=\exp\left(-2r_{p}\right)=\frac{1-G_{+}/G_{-}}{1+G_{+}/G_{-}}.$ (13)

For example, with $G_{+}/G_{-}=0.99$, we predict a maximum squeezing of
$\left(\xi_{p}^{2}\right)_{\mathchoice{\raisebox{1.2pt}{$\displaystyle\rm
ss$}}{\raisebox{1.2pt}{$\textstyle\rm ss$}}{\raisebox{1.2pt}{$\scriptstyle\rm
ss$}}{\raisebox{1.2pt}{$\scriptscriptstyle\rm
ss$}}}^{\mathchoice{\raisebox{-1.2pt}{$\displaystyle\rm
max$}}{\raisebox{-1.2pt}{$\textstyle\rm
max$}}{\raisebox{-1.2pt}{$\scriptstyle\rm
max$}}{\raisebox{-1.2pt}{$\scriptscriptstyle\rm max$}}}\simeq 23$ dB. Thus by
increasing the ratio $G_{+}/G_{-}$ to $\lesssim 1$, we can, in principle, make
intracavity squeezing arbitrarily strong. This is a counterintuitive result
from the usually accepted point of view: the steady-state intracavity
squeezing of a DPA is fundamentally limited to $3$ dB.

_Enhanced longitudinal qubit readout.—_ As an application, we below show that
our intracavity squeezing in a fully quantum DPA can exponentially improve the
SNR of longitudinal qubit readout. In sup , we also analyze the longitudinal
readout using intracavity squeezing of a semiclassical DPA. However, we
demonstrate that this semiclassical-DPA intracavity squeezing cannot enable a
practically useful increase in the SNR, even with a strong squeezing of the
output field.

To begin, we consider the Hamiltonian

$\hat{H}_{z}^{\rm fq}=\hat{H}_{\rm
eff}+\chi_{z}\hat{\sigma}_{z}\left(\hat{a}_{p}e^{-i\phi_{z}}+\hat{a}^{\dagger}_{p}e^{i\phi_{z}}\right),$
(14)

where $\hat{\sigma}_{z}$ is the Pauli matrix of the qubit. The first term is
used to generate intracavity squeezing, while the second term accounts for the
longitudinal qubit-field coupling of strength $\chi_{z}$ and phase $\phi_{z}$.
Possible experimental implementations of $\hat{H}_{z}^{\rm fq}$ are discussed
in sup . Since the photon loss of the mode $\hat{\beta}_{s}$ is strong, we
adiabatically eliminate the mode $\hat{\beta}_{s}$ to obtain the following
equation of motion for the mode $\hat{a}_{p}$,

$\dot{\hat{a}}_{p}=-ie^{i\phi_{z}}\chi_{z}\hat{\sigma}_{z}-\frac{\kappa}{2}\hat{a}_{p}-\sqrt{\kappa}\hat{\mathcal{A}}_{\rm
in}\left(t\right),$ (15)

where $\kappa=\kappa_{p}^{\rm ad}+\kappa_{p}$ is the overall photon loss rate.
Here, $\kappa_{p}^{\rm ad}=4\mathcal{G}^{2}/\kappa_{s}$ is the rate of the
adiabatic photon loss. Moreover, we have defined the overall input noise as
$\hat{\mathcal{A}}_{\rm in}\left(t\right)=\left[\sqrt{\kappa_{p}^{\rm
ad}}\hat{a}_{p,{\rm in}}^{\rm
ad}\left(t\right)+\sqrt{\kappa_{p}}\hat{a}_{p,{\rm
in}}\left(t\right)\right]/\sqrt{\kappa}$. It involves two uncorrelated noise
operators, $\hat{a}_{p,{\rm in}}^{\rm ad}\left(t\right)$ and $\hat{a}_{p,{\rm
in}}\left(t\right)$. The former represents the adiabatic noise arising from
the photon loss of the mode $\hat{\beta}_{s}$, and is given by
$i\hat{a}_{p,{\rm in}}^{\rm ad}\left(t\right)=\hat{\beta}_{s,{\rm
in}}\left(t\right)\cosh\left(r_{p}\right)+\hat{\beta}_{s,{\rm
in}}^{\dagger}\left(t\right)\sinh\left(r_{p}\right)$, where
$\hat{\beta}_{s,{\rm in}}\left(t\right)$ is the noise operator of the mode
$\hat{\beta}_{s}$. As seen in Eq. (10), $\hat{\beta}_{s,{\rm
in}}\left(t\right)$ can be considered as the vacuum noise, and therefore
$\hat{a}_{p,{\rm in}}^{\rm ad}\left(t\right)$ corresponds to the squeezed
vacuum noise of the mode $\hat{a}_{p}$. Moreover, the operator
$\hat{a}_{p,{\rm in}}\left(t\right)$ represents the vacuum noise inducing the
natural photon loss of the mode $\hat{a}_{p}$. Note that $\hat{a}_{p}$ in Eq.
(15) is a field operator displaced by an amount $\alpha_{p}^{\rm d}$, but the
side effect of this displacement on the qubit readout is negligible as a high-
frequency effect sup .

The longitudinal coupling maps the qubit state onto the output quadrature,
$\hat{\mathcal{Z}}_{\rm out}\left(t\right)=\hat{\mathcal{A}}_{\rm
out}\left(t\right)e^{-i\phi_{h}}+\hat{\mathcal{A}}_{\rm
out}^{\dagger}\left(t\right)e^{i\phi_{h}}$, which is measured by a homodyne
setup with a detection angle $\phi_{h}$. Here, $\hat{\mathcal{A}}_{\rm
out}\left(t\right)=\hat{\mathcal{A}}_{\rm
in}\left(t\right)+\sqrt{\kappa}\hat{a}_{p}\left(t\right)$ is the overall
output field. An essential parameter quantifying the homodyne detection is the
SNR, which is evaluated using the operator
$\hat{M}=\sqrt{\kappa}\int_{0}^{\tau}dt\,\hat{\mathcal{Z}}_{\rm
out}\left(t\right)$, with $\tau$ the measurement time, and is defined as

${\rm
SNR}=\left|\mbox{$\langle\hat{M}\rangle$}_{\uparrow}-\mbox{$\langle\hat{M}\rangle$}_{\downarrow}\right|\left(\mbox{$\langle\hat{M}_{N}^{2}\rangle$}_{\uparrow}+\mbox{$\langle\hat{M}_{N}^{2}\rangle$}_{\downarrow}\right)^{-1/2},$
(16)

where $\hat{M}_{N}=\hat{M}-\mbox{$\langle\hat{M}\rangle$}$ characterizes the
measurement noise, and $\\{\uparrow,\downarrow\\}$ refers to the qubit state.
The SNR of the readout using our fully quantum-DPA intracavity squeezing is
then given by

${\rm SNR}_{z}^{\rm
fq}=\sqrt{\frac{1+4\mathcal{C}}{1+4\mathcal{C}\exp\left(-2r_{p}\right)}}{\rm
SNR}_{z}^{\rm std},$ (17)

where ${\rm SNR}_{z}^{\rm
std}=8\chi_{z}\tau\left[1-2\left(1-e^{-\kappa\tau/2}\right)/\kappa\tau\right]/\sqrt{2\kappa\tau}$
refers to the SNR of the standard longitudinal readout with no squeezing.
Equation (17) shows a distinct improvement in the SNR, as in Fig. 2(a). Such
an improvement increases as the cooperativity $\mathcal{C}$, which can, in
principle, be made arbitrarily large. Furthermore, as long as
$\mathcal{C}\gg\exp\left(2r_{p}\right)/4$, we have

$\displaystyle{\rm SNR}_{z}^{\rm fq}\simeq\exp\left(r_{p}\right){\rm
SNR}_{z}^{\rm std},$ (18)

an exponential improvement in the SNR.

More importantly, the SNR improvement in Eqs. (17), (18) holds for any
measurement time. The reason is that the degree of squeezing of the
measurement noise equals the degree of intracavity squeezing, i.e.,
$\mbox{$\langle\hat{M}_{N}^{2}\rangle$}/\kappa\tau=\left(\xi_{p}^{2}\right)_{\mathchoice{\raisebox{1.2pt}{$\displaystyle\rm
ss$}}{\raisebox{1.2pt}{$\textstyle\rm ss$}}{\raisebox{1.2pt}{$\scriptstyle\rm
ss$}}{\raisebox{1.2pt}{$\scriptscriptstyle\rm ss$}}}$, and is independent of
the measurement time. This is in stark contrast to the case of using the
semiclassical-DPA intracavity squeezing, where, as discussed in sup , the
degree of squeezing of the measurement noise increases from the initial value
zero, as the measurement time increases, and consequently a large increase in
the SNR needs an extremely long measurement time. Assuming realistic
parameters of $r_{p}=2$ ($\simeq 17$ dB) and $\mathcal{C}=5$, our approach
gives an approximately fourfold improvement for any measurement time, as
illustrated in Fig. 2(a). However, when using the semiclassical-DPA
intracavity squeezing, there is almost no improvement for the short-time
measurement of most interest in experiments, even though the output-field
squeezing, characterized by the parameter $r_{\rm out}^{\rm
sc}=\ln\left[(\kappa_{s}+4\Omega_{\rm 2pd})/(\kappa_{s}-4\Omega_{\rm
2pd})\right]$, is strong sup .

In Figs. 2(b), 2(c), we plot the SNR and the measurement error,
$\epsilon_{m}=1-\mathcal{F}_{m}$, for the longitudinal readout using the fully
quantum- and semiclassical-DPA intracavity squeezing, and also for the
standard longitudinal readout with no squeezing. Here,
$\mathcal{F}_{m}=\frac{1}{2}\left[1+{\rm erf}\left({\rm SNR}/2\right)\right]$
is the measurement fidelity. Choosing $r_{p}=2$, and
$\chi_{z}=\kappa=2\pi\times 3$ MHz for our approach, a short measurement time
of $\tau=1/\kappa\simeq 53$ ns gives ${\rm SNR}^{\rm fq}_{z}\simeq 4.7$ for
$\mathcal{C}=5$. This corresponds to a measurement error of
$\epsilon_{m}\simeq 4.4\times 10^{-4}$. When $\mathcal{C}$ increases, as in
Fig. 2(d), ${\rm SNR}^{\rm fq}_{z}$ can further increase to a maximum of
$\simeq 8.9$, and the measurement error rapidly decreases, reaching a minimum
of $\simeq 1.5\times 10^{-10}$. However, at the same measurement time, both
the standard longitudinal readout with no squeezing and the case of using the
semiclassical-DPA intracavity squeezing enable a much lower SNR, i.e., ${\rm
SNR}^{\rm std}_{z}\simeq{\rm SNR}^{\rm sc}_{z}\simeq 1.1$, and,
correspondingly, a measurement error of $\simeq 0.22$, which is many orders of
magnitude larger.

_Conclusions.—_ We have introduced a method of how to exploit a fully quantum
DPA to beat the $3$ dB limit of intracavity squeezing. We have demonstrated
that an arbitrary steady-state squeezing can, in principle, be achieved for
the pump mode, by simply applying a two-tone driving to the signal mode. This
counterintuitive intracavity squeezing can exponentially increase the SNR of
longitudinal qubit readout, and improve the measurement error by many orders
of magnitude. In contrast, the semiclassical-DPA intracavity squeezing cannot
enable a useful increase in the SNR, due to the impractical requirement of a
long measurement time. Our proposal is valid for both microwave and optical
cavities, but we believe that it is easier to implement it with microwaves in
quantum circuits. The resulting intracavity squeezing is equivalent to an
externally generated and injected squeezing but without transmission and
injection losses. Thus, this intracavity squeezing, as a powerful alternative
to that external squeezing, could find many quantum applications in addition
to the qubit readout, and further excite more interest to exploit the
potential of DPAs for modern quantum technologies.

###### Acknowledgements.

W.Q. was supported in part by the Incentive Research Project of RIKEN. A.M.
was supported by the Polish National Science Centre (NCN) under the Maestro
Grant No. DEC-2019/34/A/ST2/00081. F.N. is supported in part by: Nippon
Telegraph and Telephone Corporation (NTT) Research, the Japan Science and
Technology Agency (JST) [via the Quantum Leap Flagship Program (Q-LEAP), and
the Moonshot R&D Grant Number JPMJMS2061], the Japan Society for the Promotion
of Science (JSPS) [via the Grants-in-Aid for Scientific Research (KAKENHI)
Grant No. JP20H00134], the Army Research Office (ARO) (Grant No.
W911NF-18-1-0358), the Asian Office of Aerospace Research and Development
(AOARD) (via Grant No. FA2386-20-1-4069), and the Foundational Questions
Institute Fund (FQXi) via Grant No. FQXi-IAF19-06.

## References

  * Drummond and Ficek (2004) P. D. Drummond and Z. Ficek,  _Quantum Squeezing_ (Springer, Berlin, 2004).
  * Schnabel (2017) R. Schnabel, “Squeezed states of light and their applications in laser interferometers,” Phys. Rep. 684, 1–51 (2017).
  * Lawrie  _et al._ (2019) B. J. Lawrie, P. D. Lett, A. M. Marino, and R. C. Pooser, “Quantum sensing with squeezed light,” ACS Photonics 6, 1307–1318 (2019).
  * Braunstein and van Loock (2005) S. L. Braunstein and P. van Loock, “Quantum information with continuous variables,” Rev. Mod. Phys. 77, 513–577 (2005).
  * Weedbrook  _et al._ (2012) C. Weedbrook, S. Pirandola, R. García-Patrón, N. J. Cerf, T. C. Ralph, J. H. Shapiro, and S. Lloyd, “Gaussian quantum information,” Rev. Mod. Phys. 84, 621–669 (2012).
  * Vahlbruch  _et al._ (2016) H. Vahlbruch, M. Mehmet, K. Danzmann, and R. Schnabel, “Detection of 15 dB Squeezed States of Light and Their Application for the Absolute Calibration of Photoelectric Quantum Efficiency,” Phys. Rev. Lett. 117, 110801 (2016).
  * et al. (LIGO Scientific Collaboration) J. Abadie et al. (LIGO Scientific Collaboration), “A gravitational wave observatory operating beyond the quantum shot-noise limit,” Nat. Phys. 7, 962–965 (2011).
  * Aasi  _et al._ (2013) J. Aasi _et al._ , “Enhanced sensitivity of the LIGO gravitational wave detector by using squeezed states of light,” Nat. Photonics 7, 613–619 (2013).
  * Grote  _et al._ (2013) H. Grote, K. Danzmann, K. L. Dooley, R. Schnabel, J. Slutsky, and H. Vahlbruch, “First Long-Term Application of Squeezed States of Light in a Gravitational-Wave Observatory,” Phys. Rev. Lett. 110, 181101 (2013).
  * Asjad  _et al._ (2016) M. Asjad, S. Zippilli, and D. Vitali, “Suppression of Stokes scattering and improved optomechanical cooling with squeezed light,” Phys. Rev. A 94, 051801(R) (2016).
  * Clark  _et al._ (2017) J. B. Clark, F. Lecocq, R. W. Simmonds, J. Aumentado, and J. D. Teufel, “Sideband cooling beyond the quantum backaction limit with squeezed light,” Nature (London) 541, 191–195 (2017).
  * Barzanjeh  _et al._ (2014) Sh. Barzanjeh, D. P. DiVincenzo, and B. M. Terhal, “Dispersive qubit measurement by interferometry with parametric amplifiers,” Phys. Rev. B 90, 134515 (2014).
  * Didier  _et al._ (2015a) N. Didier, A. Kamal, W. D. Oliver, A. Blais, and A. A. Clerk, “Heisenberg-Limited Qubit Read-Out with Two-Mode Squeezed Light,” Phys. Rev. Lett. 115, 093604 (2015a).
  * Didier  _et al._ (2015b) N. Didier, J. Bourassa, and A. Blais, “Fast Quantum Nondemolition Readout by Parametric Modulation of Longitudinal Qubit-Oscillator Interaction,” Phys. Rev. Lett. 115, 203601 (2015b).
  * Govia and Clerk (2017) L. C. G. Govia and A. A. Clerk, “Enhanced qubit readout using locally generated squeezing and inbuilt Purcell-decay suppression,” New J. Phys. 19, 023044 (2017).
  * Zhong  _et al._ (2020) H.-S. Zhong _et al._ , “Quantum computational advantage using photons,” Science 370, 1460–1463 (2020).
  * Zhong  _et al._ (2021) H.-S. Zhong _et al._ , “Phase-Programmable Gaussian Boson Sampling Using Stimulated Squeezed Light,” Phys. Rev. Lett. 127, 180502 (2021).
  * Huang and Agarwal (2009) S. Huang and G. S. Agarwal, “Enhancement of cavity cooling of a micromechanical mirror using parametric interactions,” Phys. Rev. A 79, 013821 (2009).
  * Asjad  _et al._ (2019) M. Asjad, N. E. Abari, S. Zippilli, and D. Vitali, “Optomechanical cooling with intracavity squeezed light,” Opt. Express 27, 32427–32444 (2019).
  * Lau and Clerk (2020) H.-K. Lau and A. A. Clerk, “Ground-State Cooling and High-Fidelity Quantum Transduction via Parametrically Driven Bad-Cavity Optomechanics,” Phys. Rev. Lett. 124, 103602 (2020).
  * Lü  _et al._ (2015) X.-Y. Lü, Y. Wu, J. R. Johansson, H. Jing, J. Zhang, and F. Nori, “Squeezed Optomechanics with Phase-Matched Amplification and Dissipation,” Phys. Rev. Lett. 114, 093602 (2015).
  * Zeytinoğlu  _et al._ (2017) S. Zeytinoğlu, A. İmamoğlu, and S. Huber, “Engineering Matter Interactions Using Squeezed Vacuum,” Phys. Rev. X 7, 021041 (2017).
  * Qin  _et al._ (2018) W. Qin, A. Miranowicz, P.-B. Li, X.-Y. Lü, J. Q. You, and F. Nori, “Exponentially Enhanced Light-Matter Interaction, Cooperativities, and Steady-State Entanglement Using Parametric Amplification,” Phys. Rev. Lett. 120, 093601 (2018).
  * Leroux  _et al._ (2018) C. Leroux, L. C. G. Govia, and A. A. Clerk, “Enhancing Cavity Quantum Electrodynamics via Antisqueezing: Synthetic Ultrastrong Coupling,” Phys. Rev. Lett. 120, 093602 (2018).
  * Ge  _et al._ (2019) W. Ge, B. C. Sawyer, J. W. Britton, K. Jacobs, J. J. Bollinger, and M. Foss-Feig, “Trapped Ion Quantum Information Processing with Squeezed Phonons,” Phys. Rev. Lett. 122, 030501 (2019).
  * Li  _et al._ (2020) P.-B. Li, Y. Zhou, W.-B. Gao, and F. Nori, “Enhancing Spin-Phonon and Spin-Spin Interactions Using Linear Resources in a Hybrid Quantum System,” Phys. Rev. Lett. 125, 153602 (2020).
  * Burd  _et al._ (2021) S. C. Burd, R. Srinivas, H. M. Knaack, W. Ge, A. C. Wilson, D. J. Wineland, D. Leibfried, J. J. Bollinger, D. T. C. Allcock, and D. H. Slichter, “Quantum amplification of boson-mediated interactions,” Nat. Phys. 17, 898–902 (2021).
  * Tang  _et al._ (2022) L. Tang, J. Tang, M. Chen, F. Nori, M. Xiao, and K. Xia, “Quantum Squeezing Induced Optical Nonreciprocity,” Phys. Rev. Lett. 128, 083604 (2022).
  * Zagoskin  _et al._ (2008) A. M. Zagoskin _et al._ , “Controlled Generation of Squeezed States of Microwave Radiation in a Superconducting Resonant Circuit,” Phys. Rev. Lett. 101, 253602 (2008).
  * Peano  _et al._ (2015) V. Peano, H. G. L. Schwefel, Ch. Marquardt, and F. Marquardt, “Intracavity Squeezing Can Enhance Quantum-Limited Optomechanical Position Detection through Deamplification,” Phys. Rev. Lett. 115, 243603 (2015).
  * Eddins  _et al._ (2019) A. Eddins, J. M. Kreikebaum, D. M. Toyli, E. M. Levenson-Falk, A. Dove, W. P. Livingston, B. A. Levitan, L. C. G. Govia, A. A. Clerk, and I. Siddiqi, “High-Efficiency Measurement of an Artificial Atom Embedded in a Parametric Amplifier,” Phys. Rev. X 9, 011004 (2019).
  * Krippner  _et al._ (1994) L. Krippner, W. J. Munro, and M. D. Reid, “Transient macroscopic quantum superposition states in degenerate parametric oscillation: Calculations in the large-quantum-noise limit using the positive P representation,” Phys. Rev. A 50, 4330–4338 (1994).
  * Munro and Reid (1995) W. J. Munro and M. D. Reid, “Transient macroscopic quantum superposition states in degenerate parametric oscillation using squeezed reservoir fields,” Phys. Rev. A 52, 2388–2391 (1995).
  * Groszkowski  _et al._ (2020) P. Groszkowski, H.-K. Lau, C. Leroux, L. C. G. Govia, and A. A. Clerk, “Heisenberg-Limited Spin Squeezing via Bosonic Parametric Driving,” Phys. Rev. Lett. 125, 203601 (2020).
  * Chen  _et al._ (2021) Y.-H. Chen _et al._ , “Shortcuts to Adiabaticity for the Quantum Rabi Model: Efficient Generation of Giant Entangled Cat States via Parametric Amplification,” Phys. Rev. Lett. 126, 023602 (2021).
  * Qin  _et al._ (2021) W. Qin, A. Miranowicz, H. Jing, and F. Nori, “Generating Long-Lived Macroscopically Distinct Superposition States in Atomic Ensembles,” Phys. Rev. Lett. 127, 093602 (2021).
  * Drummond  _et al._ (1981) P. D. Drummond, K. J. McNeil, and D. F. Walls, “Non-equilibrium Transitions in Sub/second Harmonic Generation,” Opt. Acta 28, 211–225 (1981).
  * Milburn and Walls (1981) G. Milburn and D. F. Walls, “Production of squeezed states in a degenerate parametric amplifier,” Opt. Commun. 39, 401–404 (1981).
  * Collett and Gardiner (1984) M. J. Collett and C. W. Gardiner, “Squeezing of intracavity and traveling-wave light fields produced in parametric amplification,” Phys. Rev. A 30, 1386–1391 (1984).
  * Kronwald  _et al._ (2013) A. Kronwald, F. Marquardt, and A. A. Clerk, “Arbitrarily large steady-state bosonic squeezing via dissipation,” Phys. Rev. A 88, 063833 (2013).
  * Woolley and Clerk (2014) M. J. Woolley and A. A. Clerk, “Two-mode squeezed states in cavity optomechanics via engineering of a single reservoir,” Phys. Rev. A 89, 063805 (2014).
  * Kustura  _et al._ (2022) K. Kustura, C. Gonzalez-Ballestero, A. R. Sommer, N. Meyer, R. Quidant, and O. Romero-Isart, “Mechanical Squeezing via Unstable Dynamics in a Microcavity,” Phys. Rev. Lett. 128, 143601 (2022).
  * Szorkovszky  _et al._ (2013) A. Szorkovszky, G. A. Brawley, A. C. Doherty, and W. P. Bowen, “Strong Thermomechanical Squeezing via Weak Measurement,” Phys. Rev. Lett. 110, 184301 (2013).
  * Lei  _et al._ (2016) C. U. Lei, A. J. Weinstein, J. Suh, E. E. Wollman, A. Kronwald, F. Marquardt, A. A. Clerk, and K. C. Schwab, “Quantum Nondemolition Measurement of a Quantum Squeezed State Beyond the 3 dB limit,” Phys. Rev. Lett. 117, 100801 (2016).
  * Paschotta  _et al._ (1994) R. Paschotta, M. Collett, P. Kürz, K. Fiedler, H. A. Bachor, and J. Mlynek, “Bright squeezed light from a singly resonant frequency doubler,” Phys. Rev. Lett. 72, 3807–3810 (1994).
  * Ralph  _et al._ (1995) T. C. Ralph, M. S. Taubman, A. G. White, D. E. McClelland, and H. A. Bachor, “Squeezed light from second-harmonic generation: experiment versus theory,” Opt. Lett. 20, 1316–1318 (1995).
  * Didier  _et al._ (2014) N. Didier, F. Qassemi, and A. Blais, “Perfect squeezing by damping modulation in circuit quantum electrodynamics,” Phys. Rev. A 89, 013820 (2014).
  * Dassonneville  _et al._ (2021) R. Dassonneville, R. Assouly, T. Peronnin, A.A. Clerk, A. Bienfait, and B. Huard, “Dissipative Stabilization of Squeezing Beyond 3 dB in a Microwave Mode,” PRX Quantum 2, 020323 (2021).
  * Ritsch (1990) H. Ritsch, “Quantum noise reduction in lasers with nonlinear absorbers,” Quantum Opt.: J. Euro. Opt. Soc. B 2, 189–203 (1990).
  * Teich and Saleh (1988) M. C. Teich and B. E. A. Saleh, “Photon Bunching and Antibunching,” Prog. Opt. 26, 1–104 (1988).
  * Carroll  _et al._ (2021) M. A. Carroll, G. D’Alessandro, G. L. Lippi, G.-L. Oppo, and F. Papoff, “Photon-number squeezing in nano- and microlasers,” Appl. Phys. Lett. 119, 101102 (2021).
  * Schindler  _et al._ (2011) P. Schindler, J. T. Barreiro, T. Monz, V. Nebendahl, D. Nigg, M. Chwalla, M. Hennrich, and R. Blatt, “Experimental repetitive quantum error correction,” Science 332, 1059–1061 (2011).
  * Kelly  _et al._ (2015) J. Kelly _et al._ , “State preservation by repetitive error detection in a superconducting quantum circuit,” Nature 519, 66–69 (2015).
  * Raussendorf and Harrington (2007) R. Raussendorf and J. Harrington, “Fault-Tolerant Quantum Computation with High Threshold in Two Dimensions,” Phys. Rev. Lett. 98, 190504 (2007).
  * Gambetta  _et al._ (2017) J. M. Gambetta, J. M. Chow, and M. Steffen, “Building logical qubits in a superconducting quantum computing system,” npj Quantum Inf. 3, 2 (2017).
  * Blais  _et al._ (2021) A. Blais, A. L. Grimsmo, S. M. Girvin, and A. Wallraff, “Circuit quantum electrodynamics,” Rev. Mod. Phys. 93, 025005 (2021).
  * Krantz  _et al._ (2019) P. Krantz, M. Kjaergaard, F. Yan, T. P. Orlando, S. Gustavsson, and W. D. Oliver, “A quantum engineer’s guide to superconducting qubits,” Appl. Phys. Rev. 6, 021318 (2019).
  * Eddins  _et al._ (2018) A. Eddins, S. Schreppler, D. M. Toyli, L. S. Martin, S. Hacohen-Gourgy, L. C. G. Govia, H. Ribeiro, A. A. Clerk, and I. Siddiqi, “Stroboscopic Qubit Measurement with Squeezed Illumination,” Phys. Rev. Lett. 120, 040505 (2018).
  * Gu  _et al._ (2017) X. Gu, A. F. Kockum, A. Miranowicz, Y.-x. Liu, and F. Nori, “Microwave photonics with superconducting quantum circuits,” Phys. Rep. 718-719, 1–102 (2017).
  * Touzard  _et al._ (2019) S. Touzard, A. Kou, N. E. Frattini, V. V. Sivak, S. Puri, A. Grimm, L. Frunzio, S. Shankar, and M. H. Devoret, “Gated Conditional Displacement Readout of Superconducting Qubits,” Phys. Rev. Lett. 122, 080502 (2019).
  * Ikonen  _et al._ (2019) J. Ikonen, J. Goetz, J. Ilves, A. Keränen, A. M. Gunyho, M. Partanen, K. Y. Tan, D. Hazra, L. Grönberg, V. Vesterinen, S. Simbierowicz, J. Hassel, and M. Möttönen, “Qubit Measurement by Multichannel Driving,” Phys. Rev. Lett. 122, 080503 (2019).
  * (62) See Supplementary Material, which includes Ref. Gamel and James (2010); Gardiner and Collett (1985); Wang  _et al._ (2019); Strandberg  _et al._ (2021); Lu  _et al._ (2021); Liu  _et al._ (2006); Kerman (2013); Billangeon  _et al._ (2015a, b); Richer and DiVincenzo (2016); Richer  _et al._ (2017); Stassi and Nori (2018); You and Nori (2003); Blais  _et al._ (2007); Dassonneville  _et al._ (2020), at http://xxx for a table containing a summary of our main results, and also for more technical details about the 3 dB squeezing limit, derivation of the effective Hamiltonian and its interpretation in the laboratory frame, physical mechanism of beating the 3 dB limit, analysis of longitudinal qubit readout using the semiclassical-DPA and fully-quantum-DPA intracavtiy squeezing, and discussion of a possible implementation of the intracavity-squeezing-enhanced longitudinal qubit readout.
  * Chaturvedi  _et al._ (2002) S. Chaturvedi, K. Dechoum, and P. D. Drummond, “Limits to squeezing in the degenerate optical parametric oscillator,” Phys. Rev. A 65, 033805 (2002).
  * Leghtas  _et al._ (2015) Z. Leghtas _et al._ , “Confining the state of light to a quantum manifold by engineered two-photon loss,” Science 347, 853 (2015).
  * Touzard  _et al._ (2018) S. Touzard _et al._ , “Coherent Oscillations inside a Quantum Manifold Stabilized by Dissipation,” Phys. Rev. X 8, 021005 (2018).
  * Lescanne  _et al._ (2020) R. Lescanne, M. Villiers, T. Peronnin, A. Sarlette, M. Delbecq, B. Huard, T. Kontos, M. Mirrahimi, and Z. Leghtas, “Exponential suppression of bit-flips in a qubit encoded in an oscillator,” Nat. Phys. 16, 509 (2020).
  * Chang  _et al._ (2020) C. W. S. Chang, C. Sabín, P. Forn-Díaz, F. Quijandría, A. M. Vadiraj, I. Nsanzineza, G. Johansson, and C. M. Wilson, “Observation of three-photon spontaneous parametric down-conversion in a superconducting parametric cavity,” Phys. Rev. X 10, 011011 (2020).
  * Vrajitoarea  _et al._ (2020) A. Vrajitoarea, Z. Huang, P. Groszkowski, J. Koch, and A. A Houck, “Quantum control of an oscillator using a stimulated Josephson nonlinearity,” Nat. Phys. 16, 211 (2020).
  * Guo  _et al._ (2016) X. Guo, C.-L. Zou, and H. X. Tang, “Second-harmonic generation in aluminum nitride microrings with 2500%/W conversion efficiency,” Optica 3, 1126–1131 (2016).
  * Bruch  _et al._ (2019) A. W. Bruch, X. Liu, J. B. Surya, C.-L. Zou, and H. X. Tang, “On-chip ${\chi}^{(2)}$ microring optical parametric oscillator,” Optica 6, 1361–1366 (2019).
  * Wang  _et al._ (2021) J.-Q. Wang, Y.-H. Yang, M. Li, X.-X. Hu, J. B. Surya, X.-B. Xu, C.-H. Dong, G.-C. Guo, H. X. Tang, and C.-L. Zou, “Efficient Frequency Conversion in a Degenerate ${\chi}^{(2)}$ Microresonator,” Phys. Rev. Lett. 126, 133601 (2021).
  * Nehra  _et al._ (2022) R. Nehra, R. Sekine, L. Ledezma, Q. Guo, R. M. Gray, A. Roy, and A. Marandi, “Few-cycle vacuum squeezing in nanophotonics,” arXiv preprint arXiv:2201.06768 (2022).
  * Everitt  _et al._ (2014) M. J. Everitt, T. P. Spiller, G. J Milburn, R. D. Wilson, and A. M. Zagoskin, “Engineering dissipative channels for realizing Schrödinger cats in SQUIDs,” Front. ICT 1, 1 (2014).
  * Mirrahimi  _et al._ (2014) M. Mirrahimi, Z. Leghtas, V. V. Albert, S. Touzard, R. J. Schoelkopf, L. Jiang, and M. H. Devoret, “Dynamically protected cat-qubits: a new paradigm for universal quantum computation,” New J. Phys. 16, 045014 (2014).
  * Sun  _et al._ (2019a) F.-X. Sun, Q. He, Q. Gong, R. Y. Teh, M. D. Reid, and P. D. Drummond, “Schrödinger cat states and steady states in subharmonic generation with Kerr nonlinearities,” Phys. Rev. A 100, 033827 (2019a).
  * Sun  _et al._ (2019b) F.-X. Sun, Q. He, Q. Gong, R. Y. Teh, M. D. Reid, and P. D. Drummond, “Discrete time symmetry breaking in quantum circuits: exact solutions and tunneling,” New J. Phys. 21, 093035 (2019b).
  * Aspelmeyer  _et al._ (2014) M. Aspelmeyer, T. J. Kippenberg, and F. Marquardt, “Cavity optomechanics,” Rev. Mod. Phys. 86, 1391 (2014).
  * Wollman  _et al._ (2015) E. E. Wollman, C. U. Lei, A. J. Weinstein, J. Suh, A. Kronwald, F. Marquardt, A. A. Clerk, and K. C. Schwab, “Quantum squeezing of motion in a mechanical resonator,” Science 349, 952–955 (2015).
  * Pirkkalainen  _et al._ (2015) J.-M. Pirkkalainen, E. Damskägg, M. Brandt, F. Massel, and M. A. Sillanpää, “Squeezing of Quantum Noise of Motion in a Micromechanical Resonator,” Phys. Rev. Lett. 115, 243601 (2015).
  * Ockeloen-Korppi  _et al._ (2018) C. F. Ockeloen-Korppi, E. Damskägg, J.-M. Pirkkalainen, M. Asjad, A. A. Clerk, F. Massel, M. J. Woolley, and M. A. Sillanpää, “Stabilized entanglement of massive mechanical oscillators,” Nature (London) 556, 478 (2018).
  * Ma  _et al._ (2011) J. Ma, X. Wang, C.-P. Sun, and F. Nori, “Quantum spin squeezing,” Phys. Rep. 509, 89 (2011).
  * Gamel and James (2010) O. Gamel and D. F. V. James, “Time-averaged quantum dynamics and the validity of the effective Hamiltonian model,” Phys. Rev. A 82, 052106 (2010).
  * Gardiner and Collett (1985) C. W. Gardiner and M. J. Collett, “Input and output in damped quantum systems: Quantum stochastic differential equations and the master equation,” Phys. Rev. A 31, 3761–3774 (1985).
  * Wang  _et al._ (2019) X. Wang, A. Miranowicz, and F. Nori, “Ideal Quantum Nondemolition Readout of a Flux Qubit without Purcell Limitations,” Phys. Rev. Applied 12, 064037 (2019).
  * Strandberg  _et al._ (2021) I. Strandberg, G. Johansson, and F. Quijandría, “Wigner negativity in the steady-state output of a Kerr parametric oscillator,” Phys. Rev. Research 3, 023041 (2021).
  * Lu  _et al._ (2021) Y. Lu, I. Strandberg, F. Quijandría, G. Johansson, S. Gasparinetti, and P. Delsing, “Propagating Wigner-Negative States Generated from the Steady-State Emission of a Superconducting Qubit,” Phys. Rev. Lett. 126, 253602 (2021).
  * Liu  _et al._ (2006) Y.-x. Liu _et al._ , “Controllable Coupling between Flux Qubits,” Phys. Rev. Lett. 96, 067003 (2006).
  * Kerman (2013) A. J. Kerman, “Quantum information processing using quasiclassical electromagnetic interactions between qubits and electrical resonators,” New J. Phys. 15, 123011 (2013).
  * Billangeon  _et al._ (2015a) P.-M. Billangeon, J. S. Tsai, and Y. Nakamura, “Circuit-QED-based scalable architectures for quantum information processing with superconducting qubits,” Phys. Rev. B 91, 094517 (2015a).
  * Billangeon  _et al._ (2015b) P.-M. Billangeon, J. S. Tsai, and Y. Nakamura, “Scalable architecture for quantum information processing with superconducting flux qubits based on purely longitudinal interactions,” Phys. Rev. B 92, 020509(R) (2015b).
  * Richer and DiVincenzo (2016) S. Richer and D. DiVincenzo, “Circuit design implementing longitudinal coupling: A scalable scheme for superconducting qubits,” Phys. Rev. B 93, 134501 (2016).
  * Richer  _et al._ (2017) S. Richer, N. Maleeva, S. T. Skacel, I. M. Pop, and D. DiVincenzo, “Inductively shunted transmon qubit with tunable transverse and longitudinal coupling,” Phys. Rev. B 96, 174520 (2017).
  * Stassi and Nori (2018) R. Stassi and F. Nori, “Long-lasting quantum memories: Extending the coherence time of superconducting artificial atoms in the ultrastrong-coupling regime,” Phys. Rev. A 97, 033823 (2018).
  * You and Nori (2003) J. Q. You and F. Nori, “Quantum information processing with superconducting qubits in a microwave field,” Phys. Rev. B 68, 064509 (2003).
  * Blais  _et al._ (2007) A. Blais, J. Gambetta, A. Wallraff, D. I. Schuster, S. M. Girvin, M. H. Devoret, and R. J. Schoelkopf, “Quantum-information processing with circuit quantum electrodynamics,” Phys. Rev. A 75, 032329 (2007).
  * Dassonneville  _et al._ (2020) R. Dassonneville _et al._ , “Fast High-Fidelity Quantum Nondemolition Qubit Readout via a Nonperturbative Cross-Kerr Coupling,” Phys. Rev. X 10, 011045 (2020).


# Enhancing End-to-End Autonomous Driving with
Latent World Model

Yingyan Li1,2 Lue Fan1,2 Jiawei He1,2 Yuqi Wang1,2 Yuntao Chen3 Zhaoxiang
Zhang${}^{1,2,3}\textsuperscript{\Letter}$ Tieniu Tan1,2

1 Institute of Automation, Chinese Academy of Sciences
2 University of Chinese Academy of Sciences
3 CAIR, HKISI, CAS
Email<EMAIL_ADDRESS>
Code: https://github.com/BraveGroup/LAW

###### Abstract

End-to-end autonomous driving has garnered widespread attention. Current end-
to-end approaches largely rely on the supervision from perception tasks such
as detection, tracking, and map segmentation to aid in learning scene
representations. However, these methods require extensive annotations,
hindering the data scalability. To address this challenge, we propose a novel
self-supervised method to enhance end-to-end driving without the need for
costly labels. Specifically, our framework LAW uses a LAtent World model to
predict future latent features based on the predicted ego actions and the
latent feature of the current frame. The predicted latent features are
supervised by the actually observed features in the future. This supervision
jointly optimizes the latent feature learning and action prediction, which
greatly enhances the driving performance. As a result, our approach achieves
state-of-the-art performance in both open-loop and closed-loop benchmarks
without costly annotations.

##  1 Introduction

End-to-end autonomous driving [15, 22, 30, 40, 14] is increasingly recognized
for its potential advantages over traditional methods. The traditional
planners cannot access the original sensor data. This leads to information
loss and error accumulation [15, 22]. In contrast, end-to-end planners process
sensor data to directly output planning decisions, which is shown as a
promising area for further exploration.

Most end-to-end autonomous driving methods [15, 22, 14, 30], though operating
in an end-to-end fashion, leverage a variety of auxiliary tasks such as
detection, tracking, and map segmentation. These auxiliary tasks help the
model learn better scene representations. However, they require a large amount
of manual annotations, which is quite expensive and limits the data
scalability. In contrast, a few end-to-end methods [35, 4, 40] do not adopt
perception tasks and only learn from recorded driving videos and trajectories.
These approaches can leverage a large amount of available data, making it a
promising direction. However, using only limited guidance from trajectories
makes it difficult for the network to learn effective scene representations
and achieve optimal driving performance.

To address this issue, we enhance end-to-end driving through self-supervised
learning, as illustrated in Fig. 1. Traditional self-supervised methods [10,
6] in imaging typically concentrate on static, single-frame images. However,
autonomous driving involves a dynamic series of inputs, making it essential to
use temporal data effectively. A key skill in driving is predicting future
conditions based on the current surroundings. Inspired by this, we propose a
self-supervised task aimed at forecasting latent features. Specifically, a
latent world model is developed to forecast future states based on the current
states and ego actions, where the states are represented as the latent scene
features within the network. During training, we extract the latent feature of
the future frame to supervise the predicted latent feature from the latent
world model. As a result, we jointly optimize the latent feature learning and
trajectory prediction of the current frame.

Moreover, we establish a simple yet strong planner to extract view-wise latent
features and serve as the testbed of the proposed latent world model. Unlike
previous methods, this planner does not incorporate ad-hoc modules and
perception-related branches, making it more straightforward to understand the
inner workings of the latent world model. Given this planner and the latent
world model, we have side products. Since the latent world model is capable of
predicting future view latent features, we can skip the feature extraction
process of some views in the future frame and use the predicted futures of
these views as a substitution. By skipping the feature extraction for certain
views, we enhance the efficiency of the entire pipeline. To determine which
views should be substituted, we propose a view selection strategy. Combined
with view latent substitution, this strategy significantly speeds up the whole
pipeline with minimal performance loss.

In summary, our main contributions are as follows:

  * •

We propose a LAtent World model for self-supervised learning that enhances the
training of end-to-end autonomous driving framework.

  * •

Based on the latent world model, we further propose a view selection strategy,
which greatly accelerates the pipeline while incurring minimal performance
loss.

  * •

Our framework LAW achieves state-of-the-art results on both open-loop and
closed-loop benchmarks without manual annotations.

Figure 1:  The comparison between the previous auxiliary tasks and our latent
prediction task. While previous works in (a) rely on auxiliary perception
tasks with extensive annotations, we aim to enhance the end-to-end driving
model through the latent world model in (b). During training, we obtain the
latent feature from the future frame to jointly supervise the latent feature
and predicted trajectory of the current frame. Seg.: Segmentation.

##  2 Related Works

###  2.1 End-to-End Autonomous Driving

We divide end-to-end autonomous driving methods [15, 22, 31, 35] into two
categories, explicit methods and implicit methods, depending on whether
performing traditional perception tasks.

Explicit end-to-end methods [2, 30, 19, 34] perform multiple perception tasks
simultaneously, such as detection [24, 17], tracking [45, 37], map
segmentation [15, 22] and occupancy prediction [38, 18]. As a pioneering work,
P3 [32] employs a differentiable semantic occupancy representation as a cost
factor in the motion planning process. Following this, ST-P3 [14] introduces a
spatial-temporal feature learning approach to generate more representative
features for perception, prediction, and planning tasks concurrently. Then,
many works [15, 30, 20, 19] focus on performing detection and BEV map
segmentation tasks based on the BEV feature map. As a representative, UniAD
[15] integrates multiple modules, including tracking and motion prediction, to
support goal-driven planning. VAD [22] explores vectorized scene
representation for planning purposes.

Implicit end-to-end methods [35, 4, 43, 40] present a promising direction as
they avoid utilizing a large number of perception annotations. Early implicit
end-to-end methods [43, 35] primarily relied on reinforcement learning. For
instance, MaRLn [35] designed a reinforcement learning algorithm based on
implicit affordances, while LBC [4] trained a reinforcement learning expert
using privileged (ground-truth perception) information. Using trajectory data
generated by the reinforcement learning expert, TCP [40] combined a trajectory
waypoint branch with a direct control branch, achieving good performance.
However, implicit end-to-end methods often suffer from inadequate scene
representation capabilities. Our work aims to address this issue through
latent prediction.

###  2.2 World Model in Autonomous Driving

Existing world models in autonomous driving can be categorized into two types:
image-based world models and occupancy-based world models. Image-based world
models [11, 39, 12] aim to enrich the autonomous driving dataset through
generative approaches. GAIA-1 [12] is a generative world model that utilizes
video, text, and action inputs to create realistic driving scenarios. MILE
[11] produces urban driving videos by leveraging 3D geometry as an inductive
bias. Drive-WM [39] utilizes a diffusion model to predict future images and
plans based on these predicted images. Copilot4D [42] tokenizes sensor
observations with VQVAE [36] and then predicts the future via discrete
diffusion. Another category involves occupancy-based world models [44, 29].
OccWorld [44] and DriveWorld [29] use the world model to predict the
occupancy, which requires occupancy annotations. On the contrary, our proposed
latent world model requires no manual annotations.

##  3 Preliminary

End-to-End Autonomous Driving In the task of end-to-end autonomous driving,
the objective is to estimate the future trajectory of the ego vehicle in the
form of waypoints. Formally, let
$\mathbf{I}_{t}=\\{\mathbf{I}_{t}^{1},\mathbf{I}_{t}^{2},\ldots,\mathbf{I}_{t}^{N}\\}$
be the set of $N$ surrounding multi-view images captured at time step $t$. We
expect the model to predict a sequence of waypoints
$\mathbf{W}_{t}=\\{\mathbf{w}_{t}^{1},\mathbf{w}_{t}^{2},\ldots,\mathbf{w}_{t}^{M}\\}$,
where each waypoint $\mathbf{w}_{t}^{i}=(x_{t}^{i},y_{t}^{i})$ represents the
predicted BEV position of the ego vehicle at time step $t+i$. $M$ represents
the number of future positions of the ego vehicle that the model aims to
predict.

World Model In autonomous driving tasks, a world model aims to predict future
states based on the current state and actions. To be specific, let
$\mathbf{\hat{F}}_{t}$ represent the features extracted from the current frame
at time step $t$,
$\mathbf{W}_{t}=\\{\mathbf{w}_{t}^{1},\mathbf{w}_{t}^{2},\ldots,\mathbf{w}_{t}^{M}\\}$
denote the sequence of planned waypoints by the planner, the world model
predicts features $\mathbf{\hat{F}}_{t+1}$ of the future frame using
$\mathbf{\hat{F}}_{t}$ and $\mathbf{W}_{t}$.

##  4 Method

The overall methodology is divided into three parts. First, we develop a
strong and general end-to-end planner in Sec. 4.1 to extract latent111The
terms ”latent” and ”latent feature” convey the same meaning.. Next, based on
the end-to-end planner, we introduce a world model to predict latent in Sec.
4.2. Finally, the predicted latents can substitute for some unimportant
latents so we propose a view selection approach in Sec. 4.3.

Figure 2: The overall framework. Initailly, we develop an end-to-end driving
framework to extract view latents and predict waypoints. Then, we predict the
view latents of the next frame via the latent world model. The predicted view
latent is supervised by the observed view latent of the next frame. Obs.:
Observed, Pred.: Predicted.

###  4.1 End-to-End Planner with Latent Extraction

To extract effective latent feature, we introduce a general and strong end-to-
end planner. Initially, $N$-view images are processed through an image
backbone to extract their respective feature representations. Following PETR
[25], we generate 3D position embeddings for these image features. These
position embeddings are integrated with the image features to uniquely
identify each view. The enriched image features are denoted as
$\mathbf{F}=\\{\mathbf{f}^{1},\mathbf{f}^{2},\ldots,\mathbf{f}^{N}\\}$.

Then, we employ a view attention mechanism to compress $\mathbf{F}$ into
observed view latent $\mathbf{V}$. Here, we use the term "observed" to
distinguish this view latent from others discussed later in the paper. To be
specific, for $N$ views, there are $N$ learnable view queries
$\mathbf{Q}_{view}=\\{\mathbf{q}_{view}^{1},\mathbf{q}_{view}^{2},\ldots,\mathbf{q}_{view}^{N}\\}$.
Each view query $\mathbf{q}^{i}_{view}$ undergoes a cross-attention with its
corresponding image feature $\mathbf{f}^{i}$, resulting in $N$ observed view
latent $\mathbf{V}=\\{\mathbf{v}^{1},\mathbf{v}^{2},\ldots,\mathbf{v}^{N}\\}$,
where

$\mathbf{v}^{i}=\text{CrossAttention}(\mathbf{q}_{view}^{i},\mathbf{f}^{i},\mathbf{f}^{i}).$
(1)

$\mathbf{f}^{i}$ serves as the key and value of the cross attention. Next, we
perform the temporal aggregation to the observed view latent. The observed
view latent $\mathbf{V}$ is enhanced by a historical view latent $\mathbf{H}$,
which is generated from the previous frame (the details will be discussed in
Sec. 4.2). In this way, we have

$\mathbf{E}=\mathbf{V}+\mathbf{H},$ (2)

where $\mathbf{E}$ is named enhanced view latent. Given $\mathbf{E}$, we
develop a waypoint decoder to decode waypoints. This module uses waypoint
queries to extract relevant information from $\mathbf{E}$. To be specific, we
initialize $M$ waypoint queries,
$\mathbf{Q}_{wp}=\\{\mathbf{q}_{wp}^{1},\mathbf{q}_{wp}^{2},\ldots,\mathbf{q}_{wp}^{M}\\}$,
where each query is a learnable embedding. These waypoint queries interact
with $\mathbf{E}$ through a cross-attention mechanism. The updated waypoint
queries are then passed through an MLP head to output the waypoints
$\mathbf{W}=\\{\mathbf{w}^{1},\mathbf{w}^{2},\ldots,\mathbf{w}^{M}\\}$, which
is formulated as:

$\mathbf{w}^{j}=\text{MLP}(\text{CrossAttention}(\mathbf{q}^{j}_{wp},\mathbf{E},\mathbf{E})).$
(3)

During training, we use the L1 loss to measure the discrepancy between the
predicted waypoints and the ground truth waypoints as:

$\mathcal{L}_{\text{waypoint}}=\sum_{j=1}^{M}\|\mathbf{w}_{t}^{j}-\mathbf{w}_{t}^{j,\text{GT}}\|_{1},$
(4)

The proposed end-to-end planner extracts the latent simply and effectively,
which serves as a good testbed of the latent world model.

###  4.2 World Model for Latent Prediction

In this section, we utilize the latent world model to predict the view latents
of the future frame. To begin with, we generate action-based view latents
based on the enhanced view latents $\mathbf{E}_{t}$ and predicted waypoints
$\mathbf{W}_{t}$. Specifically, let
$\mathbf{E}_{t}=\\{\mathbf{e}_{t}^{1},\mathbf{e}_{t}^{2},\ldots,\mathbf{e}_{t}^{N}\\}$,
we convert
$\mathbf{W}_{t}=\\{\mathbf{w}^{1}_{t},\mathbf{w}^{2}_{t},\ldots,\mathbf{w}^{M}_{t}\\}$
into a one-dimensional vector $\widetilde{\mathbf{w}}_{t}\in\mathbb{R}^{2M}$.
We then concatenate $\mathbf{e}_{t}^{i}$ and $\widetilde{\mathbf{w}}_{t}$
along the feature channel dimension. The concatenated vector is transformed by
an MLP to form $\mathbf{a}_{t}^{i}$, which matches the feature channel
dimension of $\mathbf{e}_{t}^{i}$. Formally, the action-based view latent of
the $i$-th view is denoted as:

$\mathbf{a}_{t}^{i}=\text{MLP}([\mathbf{e}_{t}^{i},\widetilde{\mathbf{w}}_{t}]),$
(5)

where $[\cdot,\cdot]$ denotes the concatenating operation. The overall action-
based view latent is
$\mathbf{A}_{t}=\\{\mathbf{a}_{t}^{1},\mathbf{a}_{t}^{2},\ldots,\mathbf{a}_{t}^{N}\\}$.
Subsequently, given $\mathbf{A}_{t}$, we obtain the predicted view latent
$\mathbf{P}_{t+1}$ of the frame $t+1$ by the latent world model:

$\mathbf{P}_{t+1}=\text{LatentWorldModel}(\mathbf{A}_{t}).$ (6)

The network architecture of the latent world model is a transformer decoder,
which consists of two blocks. Each block contains a self-attention and FFN
module. The self-attention is performed in the view dimension. During
training, we use the end-to-end planner to extract the observed view latent
$\mathbf{V}_{t+1}$ of frame $t+1$. $\mathbf{V}_{t+1}$ serves as the
supervision of $\mathbf{P}_{t+1}$ using an L2 loss function:

$\mathcal{L}_{\text{latent}}=\sum_{i=1}^{N}\|\mathbf{p}_{t+1}^{i}-\mathbf{v}_{t+1}^{i}\|_{2},$
(7)

where
$\mathbf{P}_{t+1}=\\{\mathbf{p}_{t+1}^{1},\ldots,\mathbf{p}_{t+1}^{N}\\}$ and
$\mathbf{V}_{t+1}=\\{\mathbf{v}_{t+1}^{1},\ldots,\mathbf{v}_{t+1}^{N}\\}$.

Besides, given $\mathbf{A}_{t}$, we encode the temporal information into the
history view latent $\mathbf{H}_{t+1}$. $\mathbf{H}_{t+1}$ is used to enhance
the observed view latent $\mathbf{V}_{t+1}$ through Eq. (2). To be specific,
we conduct self-attention on $\mathbf{A}_{t}$ in the view dimension, obtaining

$\mathbf{H}_{t+1}=\text{SelfAttention}(\mathbf{A}_{t}).$ (8)

$\mathbf{H}_{t+1}$ and $\mathbf{P}_{t+1}$ serve distinct functions.
$\mathbf{H}_{t+1}$ aims to encode temporal information as a residual, whereas
$\mathbf{P}_{t+1}$ is designed to predict the view latent of the future frame.
In addition, $\mathbf{P}_{t+1}$ serves as a good substitute for the observed
view latent of future frames, which inspires us to propose the concept of view
selection with latent substitution.

###  4.3 View Selection via Latent Substitution

We propose a view selection approach thanks to the effective view latent
predicted by the world model. Taking multi-view videos as input, this approach
dynamically selects some informative views to extract features. The other
views are not processed and their corresponding view latents are substituted
by the predicted view latent from the world model. As shown in Fig. 3, this
section consists of three components. First, given several potential view
selection strategies, the Selection Reward Prediction component predicts the
rewards of these strategies and chooses the strategy with the highest reward.
Then, the Planner with Selected Views predicts the trajectory given the
selected views. During training, we propose a Selection Reward Labeling
module, which assigns a reward label to each selection strategy.

Selection Reward Prediction As illustrated in Fig. 3(a) and (b), we introduce
a reward prediction module designed to estimate the reward associated with
each selection strategy. The reward quantitatively reflects the effectiveness
of the planning outcomes obtained using each strategy. In detail, we define
$K$ selection queries. These $K$ selection queries correspond to $K$ potential
selection strategies. Each selection query is a learnable embedding. Each
strategy selects specific views for processing while discarding the rest.
Then, we update the selection queries by performing cross-attention between
the queries and the view latent $\mathbf{P}_{t+1}$ predicted by the world
model. The updated selection queries are fed into an MLP head to predict the
rewards. Given these rewards, we choose the strategy with the highest
predicted reward. The strategy is chosen at the frame $t$ and the views
selected by this strategy serve as the input of the Planner with Selected
Views at frame $t+1$.

Figure 3: The pipeline of end-to-end driving with view selection. Given the
predicted view latents, (a) predict rewards for the selection queries defined
in (b), where each query corresponds to a selection strategy. Then, we adopt
the selection strategy with the best rewards to select views, which serves as
the input of the planner in (c). During training, we propose a selection
reward labeling module in (d) to label each selection strategy. Pred.:
Predicted, Obs.: Observed. L2 Dist.: L2 Distance.

Planner with Selected Views This planner takes selected views as input to
produce waypoints as Fig. 3(c) shows. It shares the same weights as the
planner in Sec. 4.1. To be specific, Let $N$ represent the total number of
views. Define $S\subseteq\\{1,2,\ldots,N\\}$ as the set of indices for views
that are selected. $\bar{S}$ is denoted as the complement of $S$. Given
$\mathbf{v}_{i}\in\mathbf{E}_{t+1}$ and
$\hat{\mathbf{v}}_{j}\in\mathbf{P}_{t+1}$, the combined view latents
$\mathbf{v}_{combined}$ are formulated as:

$\mathbf{v}_{combined}=\bigoplus_{i\in
S}\mathbf{v}_{i}\oplus\bigoplus_{j\in\bar{S}}\hat{\mathbf{v}}_{j},$ (9)

where $\bigoplus$ denotes the concatenation operation over the latents in the
view dimension. Then, the combined view latents are passed through the same
pipeline in Sec. 4.1 to predict trajectories. Based on the Planner with
Selected Views, we propose a selection reward labeling module to label the
selection strategies.

Selection Reward Labeling We introduce the reward labeling approach as Fig
3(d) shows. Specifically, for the $k$-th strategy, the corresponding selected
views are fed into the Planner with Selected Views to predict waypoints
$\hat{\mathbf{w}}_{k}$. The reward label $\hat{d}_{k}$ of $k$-th strategy is
defined as the L2 distance between the predicted waypoints
$\hat{\mathbf{w}}_{k}$ and the ground truth waypoints $\hat{\mathbf{w}}^{GT}$
as:

$\hat{d}_{k}=-\|\hat{\mathbf{w}}_{k}-\hat{\mathbf{w}}^{GT}\|_{2}.$ (10)

The larger $\hat{d}_{k}$ is, the closer $\hat{\mathbf{w}}_{k}$ are to the
ground truth waypoints $\hat{\mathbf{w}}^{GT}$. During training, we use the L1
loss to learn the rewards, formulated as
$\mathcal{L}_{\text{reward}}=\sum_{k=1}^{K}||d_{k}-\hat{d}_{k}||_{1},$

In summary, the total loss of our framework is:

$\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{waypoint}}+\mathcal{L}_{\text{latent}}+\mathcal{L}_{\text{reward}},$
(11)

where $\mathcal{L}_{\text{reward}}$ is an optional loss depending on whether
using the view selection approach. The weight of each loss is discussed in the
implementation details.

##  5 Experiments

###  5.1 Setup

Open-loop Benchmark The open-loop benchmark uses recorded video streams of
expert drivers along with the corresponding trajectories of the ego vehicle.
We conduct our experiments on the nuScenes dataset [1], which comprises 1,000
driving scenes. In line with previous works [14, 16, 22], we employ
Displacement Error (DE) and Collision Rate (CR) to comprehensively evaluate
the planning performance. The Displacement Error measures the L2 distance
between the predicted trajectory and the GT trajectory. The Collision Rate
quantifies the rate of collisions that occur with other objects when following
the predicted trajectory.

Closed-loop Benchmark Closed-loop evaluation is essential to autonomous
driving as it constantly updates the sensor inputs based on the driving
actions. The training dataset is collected from the CARLA [9] simulator
(version 0.9.10.1) using the teacher model Roach [43] following [40, 20],
resulting in 189K frames. We use the widely-used Town05 Long benchmark [20,
33, 11] to assess the closed-loop driving performance. We use the official
metrics: Route Completion (RC) represents the percentage of the route
completed by the autonomous agent. Infraction Score (IS) quantifies the number
of infractions as well as violations of traffic rules. A higher Infraction
Score indicates better adherence to safe driving practices. Driving Score (DS)
is the primary metric used to evaluate overall performance. It is calculated
as the product of Route Completion and Infraction Score.

Table 1:  Comparison with State-of-the-art methods on the open-loop nuScenes
[1] Benchmark. FPS of ST-P3, VAD and LAW are tested on the NVIDIA Geforce RTX
3090 GPU. FPS of UniAD is tested on the NVIDIA Tesla A100 GPU. ${\ddagger}:$
LiDAR-based methods. We do not use any historical ego status information.

Method | L2 (m) $\downarrow$ | Collision (%) $\downarrow$ | Latency (ms) |
---|---|---|---|---
1s | 2s | 3s | Avg. | 1s | 2s | 3s | Avg. | FPS
NMP‡ [41] | - | - | 2.31 | - | - | - | 1.92 | - | - | -
SA-NMP‡ [41] | - | - | 2.05 | - | - | - | 1.59 | - | - | -
FF‡ [13] | 0.55 | 1.20 | 2.54 | 1.43 | 0.06 | 0.17 | 1.07 | 0.43 | - | -
EO‡ [23] | 0.67 | 1.36 | 2.78 | 1.60 | 0.04 | 0.09 | 0.88 | 0.33 | - | -
ST-P3 [14] | 1.33 | 2.11 | 2.90 | 2.11 | 0.23 | 0.62 | 1.27 | 0.71 | 628.3 | 1.6
UniAD [15] | 0.48 | 0.96 | 1.65 | 1.03 | 0.05 | 0.17 | 0.71 | 0.31 | 555.6 | 1.8
VAD [22] | 0.41 | 0.70 | 1.05 | 0.72 | 0.07 | 0.17 | 0.41 | 0.22 | 224.3 | 4.5
LAW | 0.26 | 0.57 | 1.01 | 0.61 | 0.14 | 0.21 | 0.54 | 0.30 | 51.2 | 19.5

Table 2: Performance on closed-loop Town05 Long benchmark on CARLA. Expert:
Imitation learning from the driving trajectories of a privileged expert. Seg.:
Semantic Segmentation. Map.: BEV Map Segmentation. Dep.: Depth Estimation.
Det.: 3D Object Detection. _Latent Prediction_ : our proposed self-supervised
task.

Method | Supervision | DS$\uparrow$ | RC$\uparrow$ | IS$\uparrow$
---|---|---|---|---
CILRS [7] | Expert | 7.8$\pm$0.3 | 10.3$\pm$0.0 | 0.75$\pm$0.05
LBC [5] | Expert | 12.3$\pm$2.0 | 31.9$\pm$2.2 | 0.66$\pm$0.02
Transfuser [30] | Expert, Dep., Seg., Map., Det. | 31.0$\pm$3.6 | 47.5$\pm$5.3 | 0.77$\pm$0.04
Roach [43] | Expert | 41.6$\pm$1.8 | 96.4$\pm$2.1 | 0.43$\pm$0.03
LAV [3] | Expert, Seg., Map., Det. | 46.5$\pm$2.3 | 69.8$\pm$2.3 | 0.73$\pm$0.02
TCP [40] | Expert | 57.2$\pm$1.5 | 80.4$\pm$1.5 | 0.73$\pm$0.02
MILE [11] | Expert, Map., Det. | 61.1$\pm$3.2 | 97.4$\pm$0.8 | 0.63$\pm$0.03
ThinkTwice [21] | Expert, Dep., Seg., Det. | 65.0$\pm$1.7 | 95.5$\pm$2.0 | 0.69$\pm$0.05
DriveAdapter [20] | Expert, Map., Det. | 65.9$\pm$- | 94.4$\pm$- | 0.72$\pm$-
Interfuser [33] | Expert, Map., Det. | 68.3$\pm$1.9 | 95.0$\pm$2.9 | -
LAW | Expert, _Latent Prediction_ | 70.1$\pm$2.6 | 97.8$\pm 0.9$ | 0.72$\pm$0.03

Implementation Details The default configuration of LAW does _not_ include
view selection unless specified. For the open-loop benchmark, we use Swin-
Transformer-Tiny [26] (Swin-T) as the backbone. The input image is resized to
800 $\times$ 320\. We employ a Cosine Annealing [27] learning rate schedule
with an initial learning rate of 5e-5. AdamW [28] optimizer is utilized with a
weight decay of 0.01 and we train the model with batch size 8 for 12 epochs on
8 RTX 3090 GPUs. The weight of the waypoint loss and latent prediction loss
are set to 1.0. As for the planner with selected views, we finetune it with
the reward loss based on the LAW. We set the initial learning rate to 5e-6 and
train for an additional 6 epochs. The weight of the reward loss is set to 1.0.
For the closed-loop benchmark. And we use ResNet-34 as the backbone following
[40] for a fair comparison. We use the TCP head [40] following [20]. The size
of the input image is 900 $\times$ 256\. The optimizer is Adam. The learning
rate is set to 1e-4 and the weight decay is 1e-7. We train the model with
batch size 128 for 60 epochs. The learning rate is reduced by a factor of 2
after 30 epochs.

###  5.2 Comparison with State-of-the-art Methods

For the open-loop benchmark, we compare LAW with several state-of-the-art
methods, including UniAD [15], VAD [22] on the nuScenes dataset. The results
are summarized in Table 1. LAW outperforms UniAD and VAD in terms of the
average L2 displacement error over 1s, 2s, and 3s prediction horizons.
Moreover, our method achieves remarkable real-time performance with a latency
of 30.9 ms, highlighting the efficiency of our approach. For the closed-loop
benchmark, as shown in Table 2, our proposed method outperforms all existing
methods. Notably, our approach surpasses previous leading methods such as
ThinkTwice [21] and DriveAdapter [20], which incorporate extensive supervision
from depth estimation, semantic segmentation, and map segmentation.

###  5.3 Ablation Study

Latent Prediction

Table 3: Ablation study on latent prediction. The latent world model receives
two types of inputs: view latents and predicted trajectory. No input refers to
not utilizing the world model. Agg.: Aggregation. Pred.: Predicted. Traj.:
Trajectory.

Temporal Agg. | Input of World Model | L2 (m) $\downarrow$ | Collision (%) $\downarrow$
---|---|---|---
View Latent | Pred. Traj. | 1s | 2s | 3s | Avg. | 1s | 2s | 3s | Avg.
- | - | - | 0.44 | 0.95 | 1.65 | 1.01 | 0.27 | 0.57 | 1.32 | 0.72
$\checkmark$ | - | - | 0.32 | 0.67 | 1.14 | 0.71 | 0.20 | 0.30 | 0.73 | 0.41
$\checkmark$ | $\checkmark$ | - | 0.30 | 0.64 | 1.12 | 0.68 | 0.18 | 0.27 | 0.66 | 0.37
$\checkmark$ | $\checkmark$ | $\checkmark$ | 0.26 | 0.57 | 1.01 | 0.61 | 0.14 | 0.21 | 0.54 | 0.30

In this ablation study, we investigate the effectiveness of the latent
prediction. The results are presented in Table 3. Initially, we only use the
view latent as input of the world model, which means omitting the predicted
trajectory component.

Table 4: Ablation study on latent prediction on Town05 Long benchmark. Latent Prediction | DS$\uparrow$ | RC$\uparrow$ | IS$\uparrow$
---|---|---|---
$\times$ | 67.9$\pm$2.1 | 98.6$\pm$0.8 | 0.68$\pm$0.02
$\checkmark$ | 70.1$\pm$2.6 | 97.8$\pm 0.9$ | 0.72$\pm$0.03

As shown in the third row of Table 3, this approach results in a slight
performance improvement compared with the model without the latent prediction.
When we include the predicted trajectory as part of the input (fourth row of
Table 3), performance is significantly enhanced. It shows that an accurate
prediction of future latents requires the incorporation of driving actions,
highlighting the rationality of using the latent world model. Additionally, we
provide an ablation study on the latent prediction in the closed-loop setting,
as depicted in Table 4. Notably, we observed substantial improvements in the
Infraction Score. This indicates that the capability to predict future
scenarios effectively aids in mitigating potential collisions.

Network Architecture of Latent World Model To validate the impact of the
network architecture of the latent world model, we conduct experiments as
shown in Table 5. Firstly, it is evident that a single-layer neural network,
represented as Linear Projection, is not adequate for fulfilling the functions
of the world model, resulting in significantly degraded performance. The two-
layer MLP shows considerable improvement in performance. However, it lacks the
capability to facilitate interactions among latents from different views.
Therefore, we use the transformer decoder as our default network architecture,
which achieves the best results among the tested architectures. This suggests
that for any particular view, incorporating information from multiple adjacent
views can enhance the prediction of its future latent.

Table 5: Ablation study on different network architecture of the latent world
model. Linear Projection means a single-layer network. Arch.: Architecture.

World Model Arch. | L2 (m) $\downarrow$ | Collision (%) $\downarrow$
---|---|---
1s | 2s | 3s | Avg. | 1s | 2s | 3s | Avg.
Linear Projection | 0.31 | 0.65 | 1.14 | 0.70 | 0.26 | 0.34 | 0.66 | 0.42
Two-layer MLP | 0.27 | 0.58 | 1.07 | 0.64 | 0.17 | 0.23 | 0.59 | 0.33
Transformer Decoder | 0.26 | 0.57 | 1.01 | 0.61 | 0.14 | 0.21 | 0.54 | 0.30

The Time Horizon of Latent World Model In this experiment, the world model
predicts latent features at three distinct future time horizons: 0.5 seconds,
1.5 seconds, and 3.0 seconds. This corresponds to the first, third, and sixth
future frames from the current frame, given that keyframes occur every 0.5
seconds in the nuScenes dataset. The results, displayed in Table 6, show that
the model achieves the best performance at the 1.5-second horizon. The reasons
are as follows. The 0.5-second interval typically presents scenes with minimal
changes, providing insufficient dynamic content to improve feature learning.
In contrast, the 3.0-second interval increases the complexity of the
prediction task, which hinders better feature learning. This conclusion aligns
with observations from MAE [10], where both excessively low and high mask
ratios negatively impact the ability of the network.

Table 6: Ablation study on the different time horizons for latent prediction.
The world model is capable of predicting latent at various future time
horizons from the present moment.

Time Horizon | L2 (m) $\downarrow$ | Collision (%) $\downarrow$
---|---|---
1s | 2s | 3s | Avg. | 1s | 2s | 3s | Avg.
0.5s | 0.26 | 0.57 | 1.01 | 0.61 | 0.14 | 0.21 | 0.54 | 0.30
1.5s | 0.26 | 0.54 | 0.93 | 0.58 | 0.14 | 0.17 | 0.45 | 0.25
3.0s | 0.28 | 0.59 | 1.01 | 0.63 | 0.13 | 0.20 | 0.48 | 0.27

View Selection To ablate the effectiveness of our view selection approach, we
conduct the experiments shown in Table 7. We train our model with the view
selection module and then test it with several strategies: 1) front view and a
random view, 2) front view and a view selected by our view selection module,
3) front view and a view selected based on the rewards label in Sec. 4.3. This
reward label is generated with the help of GT trajectory and this experiment
serves as the upper bound. 4) all six views. The reason for fixing the front
view will be discussed in the appendix A.1. The results demonstrate that the
selection made by our view selection module significantly outperforms random
selections and closely approaches the upper bound set by the GT.

Table 7: Ablation study on the view selection approach. We use the model
trained with the selection reward prediction module for inference. GT view: We
adopt the view selection strategy with the reward label instead of with the
predicted reward.

Selected views | L2 (m) $\downarrow$ | Collision (%) $\downarrow$ | Latency (ms) $\downarrow$
---|---|---|---
1s | 2s | 3s | Avg. | 1s | 2s | 3s | Avg.
Front + a random view | 0.36 | 0.73 | 1.23 | 0.77 | 0.16 | 0.27 | 0.78 | 0.40 | 30.9
Front + predicted view | 0.30 | 0.64 | 1.10 | 0.68 | 0.16 | 0.25 | 0.72 | 0.38 | 30.9
Front + GT view | 0.28 | 0.56 | 0.97 | 0.60 | 0.15 | 0.22 | 0.61 | 0.33 | 30.9
Six views | 0.26 | 0.57 | 1.01 | 0.61 | 0.14 | 0.21 | 0.54 | 0.30 | 51.2

##  6 Conclusion and Limitation

In conclusion, this paper introduces a novel self-supervised approach using
the latent world model. This approach enhances the learning of scene
representations in end-to-end autonomous driving systems without costly
annotations. Although our method has demonstrated promising outcomes on
current benchmarks, it is constrained by the limited volume of data utilized.
In future work, we aim to enhance the scalability of our approach by applying
it to larger and more diverse datasets. Leveraging large-scale data, we intend
to employ the latent world model for pretraining.

## References

  * Caesar et al. [2020] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In _CVPR_ , 2020.
  * Casas et al. [2021] Sergio Casas, Abbas Sadat, and Raquel Urtasun.  Mp3: A unified model to map, perceive, predict and plan.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14403–14412, 2021.
  * Chen and Krähenbühl [2022] Dian Chen and Philipp Krähenbühl.  Learning from all vehicles.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 17222–17231, 2022.
  * Chen et al. [2020a] Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Learning by cheating.  In _Conference on Robot Learning_ , pages 66–75. PMLR, 2020a.
  * Chen et al. [2020b] Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Learning by cheating.  In _CoRL_ , pages 66–75. PMLR, 2020b.
  * Chen et al. [2020c] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020c.
  * Codevilla et al. [2019] Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon.  Exploring the limitations of behavior cloning for autonomous driving.  In _ICCV_ , 2019.
  * Contributors [2020] MMDetection3D Contributors.  MMDetection3D: OpenMMLab Next-generation Platform for General 3D Object Detection.  https://github.com/open-mmlab/mmdetection3d, 2020.
  * Dosovitskiy et al. [2017] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun.  Carla: An open urban driving simulator.  In _CoRL_ , 2017.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _CVPR_ , pages 16000–16009, 2022.
  * Hu et al. [2022a] Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zak Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton.  Model-based imitation learning for urban driving.  _NeurIPS_ , 2022a.
  * Hu et al. [2023a] Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado.  Gaia-1: A generative world model for autonomous driving.  _arXiv preprint arXiv:2309.17080_ , 2023a.
  * Hu et al. [2021] Peiyun Hu, Aaron Huang, John Dolan, David Held, and Deva Ramanan.  Safe local motion planning with self-supervised freespace forecasting.  In _CVPR_ , 2021.
  * Hu et al. [2022b] Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao.  St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning.  In _ECCV_ , 2022b.
  * Hu et al. [2022c] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al.  Goal-oriented autonomous driving.  _arXiv preprint arXiv:2212.10156_ , 2022c.
  * Hu et al. [2023b] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li.  Planning-oriented autonomous driving.  In _CVPR_ , 2023b.
  * Huang et al. [2021] Junjie Huang, Guan Huang, Zheng Zhu, and Dalong Du.  Bevdet: High-performance multi-camera 3d object detection in bird-eye-view.  _arXiv preprint arXiv:2112.11790_ , 2021.
  * Huang et al. [2023] Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu.  Tri-perspective view for vision-based 3d semantic occupancy prediction.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9223–9232, 2023.
  * Jaeger et al. [2023] Bernhard Jaeger, Kashyap Chitta, and Andreas Geiger.  Hidden biases of end-to-end driving models.  In _ICCV_ , 2023.
  * Jia et al. [2023a] Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langechuan Liu, and Hongyang Li.  Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving.  In _ICCV_ , 2023a.
  * Jia et al. [2023b] Xiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie, Conghui He, Junchi Yan, and Hongyang Li.  Think twice before driving: towards scalable decoders for end-to-end autonomous driving.  In _CVPR_ , 2023b.
  * Jiang et al. [2023] Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang.  Vad: Vectorized scene representation for efficient autonomous driving.  In _ICCV_ , 2023.
  * Khurana et al. [2022] Tarasha Khurana, Peiyun Hu, Achal Dave, Jason Ziglar, David Held, and Deva Ramanan.  Differentiable raycasting for self-supervised occupancy forecasting.  In _ECCV_ , 2022.
  * Li et al. [2022] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai.  Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.  _arXiv preprint arXiv:2203.17270_ , 2022.
  * Liu et al. [2022] Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun.  Petr: Position embedding transformation for multi-view 3d object detection.  In _European Conference on Computer Vision_ , pages 531–548. Springer, 2022.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * Loshchilov and Hutter [2016] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  _arXiv preprint arXiv:1608.03983_ , 2016.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Min et al. [2024] Chen Min, Dawei Zhao, Liang Xiao, Jian Zhao, Xinli Xu, Zheng Zhu, Lei Jin, Jianshu Li, Yulan Guo, Junliang Xing, et al.  Driveworld: 4d pre-trained scene understanding via world models for autonomous driving.  _arXiv preprint arXiv:2405.04390_ , 2024.
  * Prakash et al. [2021] Aditya Prakash, Kashyap Chitta, and Andreas Geiger.  Multi-modal fusion transformer for end-to-end autonomous driving.  In _CVPR_ , 2021.
  * Renz et al. [2022] Katrin Renz, Kashyap Chitta, Otniel-Bogdan Mercea, A Koepke, Zeynep Akata, and Andreas Geiger.  Plant: Explainable planning transformers via object-level representations.  _arXiv preprint arXiv:2210.14222_ , 2022.
  * Sadat et al. [2020] Abbas Sadat, Sergio Casas, Mengye Ren, Xinyu Wu, Pranaab Dhawan, and Raquel Urtasun.  Perceive, predict, and plan: Safe motion planning through interpretable semantic representations.  In _ECCV_ , 2020.
  * Shao et al. [2022] Hao Shao, Letian Wang, RuoBing Chen, Hongsheng Li, and Yu Liu.  Safety-enhanced autonomous driving using interpretable sensor fusion transformer.  _CoRL_ , 2022.
  * Shao et al. [2023] Hao Shao, Letian Wang, Ruobing Chen, Steven L Waslander, Hongsheng Li, and Yu Liu.  ReasonNet: End-to-End Driving with Temporal and Global Reasoning.  In _CVPR_ , pages 13723–13733, 2023.
  * Toromanoff et al. [2020] Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde.  End-to-end model-free reinforcement learning for urban driving using implicit affordances.  In _CVPR_ , 2020.
  * Van Den Oord et al. [2017] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2021] Qitai Wang, Yuntao Chen, Ziqi Pang, Naiyan Wang, and Zhaoxiang Zhang.  Immortal tracker: Tracklet never dies.  _arXiv preprint arXiv:2111.13672_ , 2021.
  * Wang et al. [2023a] Yuqi Wang, Yuntao Chen, Xingyu Liao, Lue Fan, and Zhaoxiang Zhang.  Panoocc: Unified occupancy representation for camera-based 3d panoptic segmentation.  _arXiv preprint arXiv:2306.10013_ , 2023a.
  * Wang et al. [2023b] Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang.  Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving.  _arXiv preprint arXiv:2311.17918_ , 2023b.
  * Wu et al. [2022] Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao.  Trajectory-guided control prediction for end-to-end autonomous driving: a simple yet strong baseline.  _NeurIPS_ , 2022.
  * Zeng et al. [2019] Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, and Raquel Urtasun.  End-to-end interpretable neural motion planner.  In _CVPR_ , 2019.
  * Zhang et al. [2023] Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, and Raquel Urtasun.  Learning unsupervised world models for autonomous driving via discrete diffusion.  _arXiv preprint arXiv:2311.01017_ , 2023.
  * Zhang et al. [2021] Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool.  End-to-end urban driving by imitating a reinforcement learning coach.  In _ICCV_ , 2021.
  * Zheng et al. [2023] Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu.  Occworld: Learning a 3d occupancy world model for autonomous driving.  _arXiv preprint arXiv:2311.16038_ , 2023.
  * Zhou et al. [2020] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Tracking objects as points.  In _European conference on computer vision_ , pages 474–490. Springer, 2020.

##  Appendix A Appendix

###  A.1 More Analysis of View Selection

In our view selection approach, we always use the front view and dynamically
choose one additional view from the other five views. The reason for doing
this is driven by the following experiments.

Fixing Front View Helps This section presents the experimental justification
for always choosing the front camera view as one of the input views.
Specifically, given the trained end-to-end planner without reward loss, we
conduct two view-selection strategies on it. The first strategy always uses
the front camera and selects one random view from the remaining five cameras.
The second strategy randomly chooses two cameras. The results are shown in
Table 8. This experiment shows that the strategy of fixing the front view and
randomly selecting an additional view outperforms the random selection of two
views. This superiority can be attributed to the fact that the front view can
provide more crucial information for the planning task, especially in forward-
driving scenarios.

Table 8: The importance of front view. Selected Views | L2 (m) $\downarrow$ | Collision (%) $\downarrow$
---|---|---
1s | 2s | 3s | Avg. | 1s | 2s | 3s | Avg.
Two random views | 0.39 | 0.79 | 1.32 | 0.83 | 0.15 | 0.28 | 0.91 | 0.45
Front + a random view | 0.36 | 0.74 | 1.24 | 0.78 | 0.16 | 0.28 | 0.82 | 0.42

Different Selection Strategy

Figure 4: (a) Comparison between different selection strategies. (b) Detailed
latency analysis.

Reducing the number of _fixed cameras_ can also alleviate the computational
burden. Therefore, it is natural to question how the view selection approach
compares to a configuration utilizing a reduced number of fixed cameras. To
investigate this, we carry out experiments as illustrated in Fig. 4 (a). The
specific settings of the experiments are as follows. We set up four groups of
experiments with the number of views used being 1, 2, 4, and 6. When using
only one view, we always use the front camera. For two views, the fixed-view
model is trained and tested using the front and back cameras, while the
dynamic-view model (i.e., our method) fixes the front camera and then selects
the most informative view from the remaining five cameras. For four views, the
fixed-view model uses the front, front-left, front-right, and back cameras,
while the dynamic-view model fixes the front camera and chooses three
additional cameras from the remaining five. Finally, for six views, we use all
available cameras. The results demonstrate that our dynamic selection method
consistently outperforms fixed-view settings with the same number of views.
This indicates that our method is capable of selecting the informative views
from multiple options.

Latency Breakdown Our Planner with Selected Views attains an impressive speed
of 32.4 FPS. Here we present the detailed latency of each module. We test it
on the NVIDIA Geforce RTX 3090 GPU with batch size 1. The code is based on the
mmdetection3d [8]. The specific latency associated with each module in our
model is detailed in Fig. 4 (b). As illustrated in the figure, the backbone
constitutes the majority of the model’s latency. Reducing the number of input
views leads to a linear decrease in the cost of the backbone. Since we only
have two views passing through the backbone, our view-selection method
substantially boosts the efficiency.

View Selection Visualization

Figure 5: The visualization of view selection. We outline the selected views
in orange boxes.

As depicted in Fig. 5, we present a visualization analysis of four typical
cases. From these visualizations, we derive two key insights: 1) Our method
has a preference for views with visually salient objects. As demonstrated in
cases (a), (b), and (c), the model tends to select views that feature groups
of people, vehicles with the potential to cut in, or vehicles at risk of a
rear-end collision. This preference arises because such objects have a
significant impact on driving behavior. Through reward learning, our model can
identify the view that is most important at a certain moment, by the clues
offered by these salient objects. 2) Our model learns prior knowledge of
specific scenarios. Human drivers have prior knowledge about the world, such
as the expectation of pedestrians suddenly appearing at crosswalks, which
necessitates slower driving. Our view latent reconstruction module can also
learn similar priors. For instance, in case (d), the view selection model,
aided by the reconstruction module, reasonably focuses on the crosswalk.