Email<EMAIL_ADDRESS>
# Periodic Korteweg-de Vries soliton potentials generate magnetic field
strength with exact quasisymmetry

W. Sengupta  [  Department of Astrophysical Sciences, Princeton University,
Princeton, NJ, 08543  Princeton Plasma Physics Laboratory, Princeton, NJ,
08540  N. Nikulsin  Department of Astrophysical Sciences, Princeton
University, Princeton, NJ, 08543  Princeton Plasma Physics Laboratory,
Princeton, NJ, 08540  E.J.Paul  Department of Astrophysical Sciences,
Princeton University, Princeton, NJ, 08543  Princeton Plasma Physics
Laboratory, Princeton, NJ, 08540  S. Buller  Institute for Research in
Electronics and Applied Physics, University of Maryland, College Park,
Maryland 20742, USA  R.Nies  Department of Astrophysical Sciences, Princeton
University, Princeton, NJ, 08543  Princeton Plasma Physics Laboratory,
Princeton, NJ, 08540  S.R.Hudson  Princeton Plasma Physics Laboratory,
Princeton, NJ, 08540  A. Bhattacharjee  Department of Astrophysical Sciences,
Princeton University, Princeton, NJ, 08543  Princeton Plasma Physics
Laboratory, Princeton, NJ, 08540

###### Abstract

Quasisymmetry (QS) is a hidden symmetry of the magnetic field strength,
$|\mathbf{B}|$, that confines charged particles effectively in a nonsymmetric
toroidal plasma equilibrium. Recent numerical breakthroughs have shown that
excellent QS can be realized in a toroidal volume. Here, we show that the
hidden symmetry of QS has a deep connection to the underlying symmetry that
makes solitons possible. In particular, we demonstrate that a class of
quasisymmetric $|\mathbf{B}|$ is described by a periodic finite-gap soliton
potential of the well-known Korteweg-de Vries (KdV) equation. Our exact and
non-perturbative method drastically reduces the number of independent
$|\mathbf{B}|$ parameters on a magnetic flux surface to just three, which
could make stellarator optimization schemes significantly more efficient.
Furthermore, we deduce an upper bound on the maximum toroidal volume that can
be quasisymmetric. In the neighborhood of the outermost surface,
$|\mathbf{B}|$ approaches the form of the 1-soliton reflectionless potential.

††preprint: AIP/123-QED

Physical systems possessing hidden symmetries (HS) attract serious attention
Charalambous, Dubovsky, and Ivanov (2021); Lee  _et al._ (2021); Liu and
Tegmark (2022); Landreman and Paul (2022); Moser (1979); Leach and
Andriopoulos (2008) because of the far-reaching impact of the symmetry on the
physical properties of the systems. HS manifests only in special coordinates,
which are constructed, along with the invariants, as part of the solution to
the problem. In the presence of HS, the partial differential equations (PDEs)
governing classical dynamical fields often reduce to integrable PDEs. (Moser,
1979; Ovsienko and Khesin, 1987; Leach and Andriopoulos, 2008) An example of
the latter is the well-known Korteweg-de Vries (KdV) equation with soliton
solutions.Gardner  _et al._ (1967); Novikov  _et al._ (1984)

An analogous and well-studiedLewis Jr (1967); Leach and Andriopoulos (2008);
Gjaja and Bhattacharjee (1992) mechanical example of HS occurs in the one-
dimensional time-dependent harmonic oscillator problem, which can be shown to
admit the so-called Ermakov-Lewis invariant (ELI). The ELI is the action
variable for the harmonic oscillator. The symmetry that leads to the ELI is
nontrivial Leach and Andriopoulos (2008) and related to that underlying the
KdV.Brezhnev (2008); Leach and Andriopoulos (2008) Furthermore, for soliton
(or reflectionless) potentials, the adiabatic invariants obtained using the
eikonal (Wentzel-Kramers-Brillouin or WKB) approximation become exact
invariants. (Gjaja and Bhattacharjee, 1992; Berry and Howls, 1990)

In plasma physics, a hidden symmetry called quasisymmetry (QS), (Boozer, 1983;
Helander, 2014) forces the magnetic field strength, $B=|\mathbf{B}|$, to be
symmetric,(Boozer, 1983; Rodriguez, 2022) whereas B itself is fully three-
dimensional (3D). Several practical reasons exist for building 3D toroidal
magnetic confinement fusion devices called stellarators with quasisymmetric
$B$. The HS in $B$ makes the bounce action (second adiabatic invariant),
$J_{||}$, independent of the field-line label,(Helander, 2014) which
guarantees excellent particle confinement on a torus. The 3D nature of B
allows stellarators to avoid intrinsic difficulties (such as powerful
transients) associated with large plasma currents. (Helander, 2014)

It is currently unknown whether QS can be achieved exactly in a toroidal
volume for a magnetic field that satisfies the constraint of force-balance in
ideal magnetohydrodynamics (MHD).Freidberg (2014)(Burby, Kallinikos, and
MacKay, 2020; Helander, 2014; Plunk and Helander, 2018; Garren and Boozer,
1991a, b) Analytical results (Garren and Boozer, 1991a, b; Plunk and Helander,
2018; Landreman and Sengupta, 2019) seem to indicate that in the presence of
scalar plasma pressure, the resulting 3D nonlinear system of PDEs is
overdetermined and does not have solutions in a toroidal volume.(Garren and
Boozer, 1991a, b; Landreman and Sengupta, 2019; Constantin, Drivas, and
Ginsberg, 2021; Rodríguez, Helander, and Bhattacharjee, 2020; Rodríguez,
Sengupta, and Bhattacharjee, 2022) In particular, the near-axis expansion
(NAE),(Garren and Boozer, 1991a, b; Landreman and Sengupta, 2018) an
asymptotic expansion in the distance from the magnetic axis, is typically
divergent beyond the second-order. On the other hand, designs with a precise
level of QS and MHD force balance have been obtained through large-scale
numerical optimization, hinting at the possible existence of special solutions
to the overdetermined problem.

Further analytical insight into these unique configurations that approximately
solve the overdetermination problem is needed. In the special case of
isodynamic magnetic fields, which satisfy a constraint more restrictive but
related to QS,(Helander, 2014; Schief, 2003) the soliton solutions(Schief,
2003) of the vortex filament equations play a significant role. However, the
overdetermination problem for QS is more complicated than that for isodynamic
fields. Hence we focus on the functional form of QS magnetic field strength
without enforcing a force-balance condition. Our motivation here is to study
the possible connections between QS and other HS with an invariant, such as
the ELI,Leach and Andriopoulos (2008); Gjaja and Bhattacharjee (1992)
analogous to the field-line label independence of $J_{||}$.

In this Letter, starting from the basic definitions and properties of QS, we
show non-perturbatively that QS admits a class of analytic $B$ that is a
periodic solution(Novikov  _et al._ , 1984; Kamchatnov, 2000) of the KdV
equation. The exact analytic solution for $B$ is valid in a toroidal volume
and is consistent with the NAE predictions close to the magnetic axis. The
connection with periodic KdV solitons allows us to describe $B$ on a magnetic
flux surface with only three functions of the magnetic flux, related to the
so-called “spectral parameter”(Novikov  _et al._ , 1984) of the KdV. We show
that these functions are closely related to the spectrum of $B$ in Boozer
coordinates Helander (2014) where the HS is manifest, thereby highlighting the
low dimensionality of the Boozer spectrum. Furthermore, our analysis shows
that the periodicity of $B$ is maintained only when these functions are
distinct. Consequently, we can estimate the region of validity of volumetric
quasisymmetry based on where the two spectral parameters intersect (shown
below). Remarkably, near the point of intersection, the connection length
(period of $B$) begins to diverge, and $B$ approaches the form of a
reflectionless potential (a single soliton that decays at infinity). We
demonstrate that our analytic theory can describe a vacuum equilibrium with
precise quasi-axisymmetry (QA)Landreman and Paul (2022) and many other
numerically-optimized quasisymmetric configurations.

We begin by reviewing some of the known basic properties of QS. We enforce QS
through the condition (Freidberg, 2014)

$\displaystyle\frac{\partial B}{\partial\ell}=f(B;\psi),$ (1)

which is equivalent to the “triple-product” form of QS.(Rodriguez, 2022;
Rodríguez, Helander, and Bhattacharjee, 2020) Moreover, due to the toroidal
setting, $B$ is subject to the following periodic boundary condition
(Helander, 2014)

$\displaystyle B(\psi,\alpha,\ell)=B(\psi,\alpha,\ell+L(\psi)).$ (2)

Here, $(\psi,\alpha,\ell)$ is the standard Clebsch coordinate system such that
$\bm{B}=\bm{\nabla}\psi\times\bm{\nabla}\alpha$,Helander (2014) with flux
label $\psi$, field-line label $\alpha$, and $\ell$, the arclength along the
magnetic field. Since $\alpha$ is a multi-valued function, we impose suitable
toroidal cuts. The period of $B$, $L(\psi)$, is called the connection length
and is a quantity of great physical interest. In terms of the translation
operator $T$ such that $Tf(\ell)=f(\ell+L)$, the periodic boundary condition
can be written $TB=B$. In addition, (1) implies Freidberg (2014) that

$\displaystyle\frac{\partial B}{\partial\alpha}+H(\psi,\alpha)\frac{\partial
B}{\partial\ell}=0,$ (3)

where $H(\psi,\alpha)$ is independent of $\ell$ and depends on the definition
of the origin of $\ell$. There is a considerable degree of freedom in choosing
$H$. For example, by redefining $\ell\mapsto\ell-\int d\alpha H(\psi,\alpha)$,
we can reduce (3) to $\partial_{\alpha}B=0$.(Freidberg, 2014) Furthermore,
because of (1), any integral along the magnetic field line between two points
of equal values of $B=B_{b}$ satisfies

$\displaystyle\partial_{\alpha}\int_{B\leq B_{b}}d\ell\;F(B;\psi)=0.$ (4)

Omnigeneous magnetic fields, Helander (2014) in contrast with quasisymmetric
ones, satisfy (4) but not (2).

We now address the central problem of this Letter. We shall consider the
quasisymmetric system (1), (2), and (3) as a PDE system on a constant flux
surface $\psi$. We will define a time-like variable, $t\propto\int
H(\psi,\alpha)d\alpha$, and a space-like variable, $x\propto l$, where both
proportionality factors depend only on $\psi$. The fundamental problem is to
find periodic time-dependent $B(\ell,t;\psi)$ that support time-independent
integrals that obey (4). In particular, we shall focus on PDEs with exact ELI.

In our search for a special $B$ that has an exact invariant, we are led to the
ELI (or equivalently Courant-Snyder Courant and Snyder (1958)) theory. Leach
and Andriopoulos (2008); Lewis Jr (1967); Gjaja and Bhattacharjee (1992) A
harmonic oscillator with a variable frequency,
$\omega^{2}(\ell,t)\equiv\mathcal{E}-B(\ell,t),$ possesses the nontrivial
exact ELI, $I(\mathcal{E})$. The oscillator equation (Brezhnev, 2008; Gjaja
and Bhattacharjee, 1992) is nothing but the time-independent Schrodinger
equation (SE)

$\displaystyle\mathcal{L}\Psi=\mathcal{E}\Psi,\quad\mathcal{L}\equiv(-\partial_{\ell}^{2}+B(\ell,t;\psi)),$
(5)

with $B$ as a potential and $\mathcal{E}$, a real constant. The exact solution
to the SE (5) is of the form $\Psi_{\pm}=\rho\exp{i\chi_{\pm}}$, where
$\chi_{\pm}=\pm\sqrt{2I}\int d\ell/\rho^{2}$, and $\rho$ satisfies the
nonlinear Ermakov-Pinney Leach and Andriopoulos (2008); Brezhnev (2008)
equation

$\displaystyle(\partial_{\ell}^{2}+(\mathcal{E}-B))\rho=2I(\mathcal{E})/\rho^{3}.$
(6)

Moreover, since $B$ is periodic in $\ell$, the Floquet-Bloch theorem Magnus
and Winkler (2013); Novikov  _et al._ (1984); Kamchatnov (2000) guarantees
that there exists simultaneous eigenfunctions, $\Psi_{\pm}$, such that

$\displaystyle\mathcal{L}\Psi_{\pm}=\mathcal{E}\Psi_{\pm},\quad
T\Psi_{\pm}=e^{\pm i\mathfrak{p}L}\Psi_{\pm},$ (7)

where $\mathcal{E}$ is now called the spectral parameter. The Floquet
multiplier, $\mathfrak{p}(\mathcal{E})$, also known as the quasimomentum,
Novikov  _et al._ (1984) is given by $\pm\mathfrak{p}(\mathcal{E})L=\oint
d\ell\chi_{R}$, where $\chi_{R}$ is the real part of $\chi_{\pm}$. Thus
$\mathfrak{p}(\mathcal{E})$ is an exact invariant and is proportional to
$\sqrt{2I(\mathcal{E})}$.

The SE (5) enjoys a special connection with soliton theory since it is also
one of the Lax pairs,Gardner  _et al._ (1967); Lax (1968); Novikov  _et al._
(1984) for the KdV equation, (Kamchatnov, 2000; Novikov  _et al._ , 1984)

$\displaystyle\frac{\partial B}{\partial t}+6B\frac{\partial B}{\partial
x}+\frac{\partial^{3}B}{\partial x^{3}}=0.$ (8)

The fundamental idea behind Lax pairs Lax (1968); Tao (2009) is that most
integrable nonlinear systems can be obtained as a consistency condition of the
overdetermined linear system

$\displaystyle L\Psi=\mathcal{E}\Psi,\quad\partial_{t}\Psi=P\Psi,$ (9)

where $L,P$ are linear operators. The isospectral
($\partial_{t}\mathcal{E}=0$) consistency condition of the system (9), given
by the Lax equation,

$\displaystyle\partial_{t}L=[P,L],$ (10)

then leads to an integrable nonlinear PDE. The choice of the Schrodinger
operator as a Lax pair, i.e., $L=\mathcal{L}$, together with suitable choices
for $P$, leads to the KdV hierarchy.Novikov  _et al._ (1984) Since the
evolution is isospectral, the quasimomentum $\mathfrak{p}(\mathcal{E})$
remains independent of time if $B(\ell,t,\psi)$ evolves under KdV. Moreover,
the Lax equation guarantees an infinite number of conserved quantities of the
form (4). We note that the periodicity condition, together with the Lax
equation, imposes severe restrictions Gesztesy and Weikard (1996); Dubrovin,
Krichever, and Novikov (1990) on the choice of $P,L$. Consequently, the
Floquet-Bloch spectrum can be shown to have only a finite number of
gaps.Novikov  _et al._ (1984); Brezhnev (2008)

With these results from soliton theory in mind, let us now turn to the QS
problem. As discussed earlier, a special finite-gap potential $B$, which
solves KdV, is guaranteed to have invariants (4) and the ELI, $I$. However, to
gain physical insight, we shall now consider the WKB limit of the SE (5) for
$\Psi_{\pm}$. In the WKB approach, we obtain
$\chi_{\pm}\approx\pm\sqrt{\mathcal{E}-B}$. Froman (1979), from which it
follows that $\pm\mathfrak{p}L=\oint d\ell\chi_{R}$ is the adiabatic invariant
$J_{||}$

$\displaystyle J_{||}=\oint\sqrt{\mathcal{E}-B}\;d\ell.$ (11)

Here we have interpreted $\mathcal{E}$ to be related to the particle energy.
Since only the real part of $\chi_{\pm}$ is involved, integration is performed
in the region $\mathcal{E}-B\geq 0$. Therefore, both passing and trapped
orbits Helander (2014); Froman (1979) are accounted for. From the QS condition
(4), it then follows that $J_{||}$, and in general,
$\mathfrak{p}(\mathcal{E})$ given by (7), are time-independent. Furthermore,
for passing orbits, an alternative approach based on a large $\mathcal{E}$
limit Novikov  _et al._ (1984) shows that $\mathfrak{p}(\mathcal{E})$
generates the infinite hierarchy of time-independent KdV integrals that are
polynomials in $B$ and its derivatives, and satisfy (4).

Noting the relation between the KdV and QS invariants, we now consider the
consistency between the QS conditions (1)-(3) and the KdV equations. The
structure of equation (3) suggests that the search for a quasisymmetric $B$
can be further restricted to traveling wave solutions of the KdV equation,
i.e., $B=B(\ell-\int d\alpha H(\psi,\alpha))$. The equation obtained from the
KdV equation by first moving to the traveling wave frame and integrating twice
Kamchatnov (2000) is indeed of the form (1) with the function $f(B;\psi)$
being a cubic function of $B$, that is,

$\displaystyle\left(\frac{\partial
B}{\partial\ell}\right)^{2}=\mathcal{D}(\psi)(B_{M}-B)(B-B_{m})(B-B_{X}),$
(12)

where the roots of the cubic polynomial, $B_{i}=\\{B_{M},B_{m},B_{X}\\}$, are
functions of $\psi$, and $\mathcal{D}(\psi)$ is a proportionality factor. The
exact solution of (12) subject to the periodicity constraint (2) can be
expressed in terms of elliptic functions.(Kamchatnov, 2000; Novikov  _et al._
, 1984) Thus, we have shown that $B$ given by a periodic traveling wave
solution of the KdV equation meets the QS requirements (1)-(4).

Note that the QS constraints by themselves are not sufficient to restrict the
form of the arbitrary function $f(B;\psi)$ in the $\partial B/\partial\ell$
equation (1) to a simple cubic in (12). However, if we make the connection
between the QS invariants and the ELI, the cubic relation follows from
periodicity (2) and the isospectral (Lax) condition (10).

To obtain the proportionality factors between $(x,t)$ and $(l,\alpha)$
explicitly, we now proceed from (12) and derive the KdV equation.
Differentiating (12) twice with $\ell$ and using the $\partial_{\alpha}B$
equation (3) we get (8), the KdV equation for $B$. Here, the coordinates
$(x,t)$ are defined in terms of $(\alpha,\ell)$ as

$\displaystyle\frac{x}{\ell}=\sqrt{\frac{\mathcal{D}}{2}},\;\;\frac{t}{\int
d\alpha H}=\frac{\sqrt{\mathcal{D}/2}}{2(B_{M}+B_{m}+B_{X})}.$ (13)

The $\partial_{\alpha}B$ relation (3), now takes the following form of a 1D
traveling wave equation with speed $c(\psi)$,

$\displaystyle\frac{\partial B}{\partial t}+c(\psi)\frac{\partial B}{\partial
x}=0,\quad c(\psi)=2(B_{M}+B_{m}+B_{X}).$ (14)

$B$ is then obtained from the cnoidal solution (Kamchatnov, 2000) of KdV,

$\displaystyle B$
$\displaystyle=B_{m}+(B_{M}-B_{m})\text{cn}^{2}\left(\sqrt{\frac{B_{M}-B_{X}}{2}}(x-ct),m\right),$
$\displaystyle m$ $\displaystyle=\frac{B_{M}-B_{m}}{B_{M}-B_{X}},\quad 0\leq
m\leq 1.$ (15)

Here, $\text{cn}(u,m)$ is the Jacobi cosine elliptic function with modulus
$m$. $B$ varies between $B_{m}$ and $B_{M}$. The period of $B$, $L(\psi)$,
diverges logarithmically as $m\to 1$. One would expect this behavior near a
separatrix, where periodicity along a field line is lost because the field
line cannot cross the X-point. In particular, it is interesting to note that
in the $m\to 1$ limit, $B$ takes the form of the infinite KdV soliton
(reflectionless potential).

The connection with KdV significantly reduces the dimensionality of this class
of quasisymmetric $B$. We first note that on any flux surface $\psi$, we have
four flux functions that determine $B$, namely $\mathcal{D}$ and
$B_{M},B_{m},B_{X}$, the three real roots of the cubic polynomial in (12).
However, as shown in Appendix A $\mathcal{D}$ can be expressed in terms of the
rotational transform $\iota$ and the roots. Therefore, the connection with KdV
points to a hidden low-dimensionality of the description of $B$.

The functions $B_{M}$ and $B_{m}$ denote the values of $B$ on a flux surface
at the maxima and minima, and $B_{X}$ is a third root of the polynomial. The
behavior of $B$ in Boozer coordinates is well known (Garren and Boozer, 1991a,
b; Landreman and Sengupta, 2018) near the magnetic axis and is consistent with
(12). On the axis $B$ is a constant, $B_{0}$, and varies as $B\approx
B_{0}(1+\overline{\eta}\sqrt{2\psi}\cos\vartheta)+2\psi(B_{20}+B_{22}\cos
2\vartheta)$ near the axis. Here, $\vartheta$ is the well-known Boozer
angle.(Landreman and Sengupta, 2018) The constant $\overline{\eta}$ dominates
the maximum variation of $B$ on a flux surface, with the $O(2\psi)$ term being
only a small second-order correction. Within the NAE framework, $B_{X}$ and
$\mathcal{D}$ are constant throughout the volume, with
$\mathcal{D}\propto((\iota-N/M))^{2}$. The NAE predicts

$\displaystyle\frac{B_{M}}{B_{0}},\frac{B_{m}}{B_{0}}=1\pm\overline{\eta}\sqrt{2\psi},\quad\frac{B_{X}}{B_{0}}=\frac{4B_{22}/B_{0}+\overline{\eta}^{2}}{4B_{22}/B_{0}+2\overline{\eta}^{2}}.$
(16)

Evolution of $B$ under KdV guarantees the well-known property that these roots
are constants on a flux surface. The limiting surface where $m\to 1$
corresponding to $B_{X}\to B_{m}$ is the separatrix. We can use (16) to
approximately predict its location.

We shall now demonstrate numerically that a large class of quasisymmetric $B$
satisfies the integrated form of KdV as given in (12). As demonstrated in Fig.
1(a), $(dB/dl)^{2}$, when plotted as a function of $B$, has the form of a
cubic polynomial in the precise QA equilibrium. Landreman and Paul (2022) The
scatter is due to imperfect QS. On the other hand, Fig. 1(b) shows that
increasing the polynomial order beyond three does not lead to an observable
decrease in error (defined as $1-r^{2}$, where $r^{2}$ is the coefficient of
determination), as the fit error is dominated by QS error. Thus, Fig. 1
supports equation (12) as a model for a device with perfect QS. A similar
analysis for other configurations, which tend to have more QS error, is shown
in Appendix B. (Note that $s$ is the normalized toroidal flux in all figures.)

Figure 1: The value of $(dB/d\ell)^{2}$ on the outermost flux surface in the
precise QA equilibrium Landreman and Paul (2022) is plotted as a function of
$B$, alongside a cubic polynomial fit in (a). In (b), the errors of quadratic,
cubic, and quartic fits are compared on each flux surface, demonstrating that
a cubic ansatz (as predicted by KdV) is sufficient.

In Fig. 2, we demonstrate that $B$ appears roughly sinusoidal far from the
separatrix but becomes soliton-like (sharper peaks, flatter troughs) as the
separatrix is approached. Note that the $B$ shown in Fig. 2 is perfectly
quasisymmetric, as symmetry-breaking modes have been filtered out. The period
of $B$ increases as $L(\psi)\to\infty$ on the separatrix.

Figure 2: The value of $B$ in a quasiaxisymmetric device in the core ($s=1/8$)
and at the edge ($s=1$). The separatrix is outside of the device but close to
its edge. Note the increasing period as the separatrix is approached.

With the global description of the exact quasisymmetric $B$, (15) at our
disposal, we now investigate a key element of this theory: the physical
meaning of the roots $B_{i}$. In Fig. 3 b, we show the typical behavior of the
roots. Symmetry-breaking modes have been filtered out, and the roots are
obtained by a cubic fit of the numerical data. Points represent the actual
minimum and maximum values of $B$ on each flux surface. Surprisingly, the NAE
description of $B_{M}$ and $B_{m}$ continues to hold even far from the axis.
The third root $B_{X}$ depends quite sensitively on the $\iota$ profiles (Fig.
4). The relationship between the $\iota$ profile, shear, and $B_{X}$ will be
studied in more detail in a future publication. For now, we point out that the
constant value of $B_{X}$ predicted by the NAE (16) is valid only when the
magnetic shear is negligible (see Fig. 3(b)). We show in Fig. 4 that for small
but finite shear, $B_{X}$ tracks the $\iota$ profile quite closely but varies
more for positive shear.

Understanding the roots enables us to estimate the maximum volume that a
quasisymmetric field can occupy since the periodicity of $B$ is maintained as
long as $0\leq m<1$. As we saw earlier, the limit $(m\to 1,B_{m}\to B_{X})$
was the limit in which the connection length diverges. Using the simple NAE
estimates for the roots $B_{m}$ and $B_{X}$, we can estimate where the two
roots will intersect, thereby signaling the breakdown of periodicity. We have
demonstrated in Fig. 3 how this estimate can predict the maximum volume of
nested surfaces in precise QA extended using coils developed recently.
Wechsung  _et al._ (2022) The appearance of the island chain in Fig. 3 is due
to the failure of QS (which guarantees nested flux surfaces Rodríguez,
Helander, and Bhattacharjee (2020)) and not due to a resonance with the
helicity of the QA. Rodríguez and Bhattacharjee (2021)

Figure 3: Prediction of maximum toroidal volume that can be made
quasisymmetric in precise QA. The boundary of the extended precise QA and two
other flux surfaces are shown inside a Poincare plot of the islands in (a),
and the roots are plotted in (b). The boundary of the original precise QA is
shown in red. Note that the islands occur where the second and third roots
touch approximately.

Figure 4: The $B_{X}$ and $\iota$ profiles plotted for precise QA versions
with negative shear (a) and positive shear (b).

In summary, we have demonstrated that periodic soliton solutions of the KdV
equation lead to a broad class of exactly quasisymmetric $B$. Several results
follow from the connection between QS and KdV.

Firstly, the description of a class of exactly quasisymmetric $B$ can be
reduced to understanding the behavior of the roots $B_{i}$ of a cubic
polynomial. These roots correspond to extrema or saddle points of $B$ on a
surface. In particular, $B$, which satisfies the periodic KdV, is a solution
of the periodic Schrodinger equation where $B_{i}$’s are related to the
endpoints of the band gap.(Kamchatnov, 2000; Novikov  _et al._ , 1984) Since
the spectrum of KdV is time-independent, the roots do not depend on time. This
confirms the well-known statement that the extrema or saddle points of $B$
depend only on the flux label in QS.Helander (2014) Furthermore since various
aspects of the flux surface shaping can be deduced (Rodriguez, 2022) from the
Fourier coefficients of B, the hidden lower dimensionality is beneficial for
understanding shaping.

Secondly, the hierarchy of quasisymmetric invariants, as given by (4), can be
understood in terms of the hierarchy of the conserved quantities of KdV.
Evolution under KdV guarantees the field-line independence of these
quantities. Thirdly, we can estimate the maximum toroidal volume QS allows
from the properties of the roots—the overlap of the third root with the second
signals the breakdown of periodicity of $B$, as demonstrated numerically.
Finally, the connection length can be much larger than the major radius,
typically assumed in the literature. A longer connection length may have
important implications for linear and nonlinear stability in devices with QS.

Although we have not imposed any constraints from ideal MHD and therefore have
avoided the overdetermination problem, we have verified that several existing
numerically-obtained configurations are well described by our theory. Given
the well-known robust stability of soliton solutions,Tao (2009) further work
is needed to elucidate the possible implications for the stability of QS
equilibria.

We conclude by thanking E. Rodriguez, H. Weitzner, V. Duarte, F.P. Diaz, and
M. Landreman for helpful discussions and suggestions. This research was
supported by a grant from the Simons Foundation/SFARI (560651, AB) and DoE
Grant No. DE-AC02-09CH11466.

## References

  * Charalambous, Dubovsky, and Ivanov (2021) P. Charalambous, S. Dubovsky, and M. M. Ivanov, “Hidden symmetry of vanishing love numbers,” Physical Review Letters 127, 101101 (2021).
  * Lee  _et al._ (2021) D. Lee, S. Bogner, B. A. Brown, S. Elhatisari, E. Epelbaum, H. Hergert, M. Hjorth-Jensen, H. Krebs, N. Li, B.-N. Lu, _et al._ , “Hidden spin-isospin exchange symmetry,” Physical Review Letters 127, 062501 (2021).
  * Liu and Tegmark (2022) Z. Liu and M. Tegmark, “Machine learning hidden symmetries,” Physical Review Letters 128, 180201 (2022).
  * Landreman and Paul (2022) M. Landreman and E. Paul, “Magnetic fields with precise quasisymmetry for plasma confinement,” Physical Review Letters 128, 035001 (2022).
  * Moser (1979) J. Moser, “Hidden symmetries in dynamical systems,” American Scientist 67, 689–695 (1979).
  * Leach and Andriopoulos (2008) P. G. Leach and K. Andriopoulos, “The ermakov equation: a commentary,” Applicable Analysis and Discrete Mathematics , 146–157 (2008).
  * Ovsienko and Khesin (1987) V. Y. Ovsienko and B. A. Khesin, “Korteweg-de vries superequation as an euler equation,” Functional Analysis and Its Applications 21, 329–331 (1987).
  * Gardner  _et al._ (1967) C. S. Gardner, J. M. Greene, M. D. Kruskal, and R. M. Miura, “Method for solving the korteweg-devries equation,” Physical review letters 19, 1095 (1967).
  * Novikov  _et al._ (1984) S. Novikov, S. V. Manakov, L. P. Pitaevskii, and V. E. Zakharov,  _Theory of solitons: the inverse scattering method_ (Springer Science & Business Media, 1984).
  * Lewis Jr (1967) H. R. Lewis Jr, “Classical and quantum systems with time-dependent harmonic-oscillator-type hamiltonians,” Physical Review Letters 18, 510 (1967).
  * Gjaja and Bhattacharjee (1992) I. Gjaja and A. Bhattacharjee, “Asymptotics of reflectionless potentials,” Physical review letters 68, 2413 (1992).
  * Brezhnev (2008) Y. V. Brezhnev, “What does integrability of finite-gap or soliton potentials mean?” Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 366, 923–945 (2008).
  * Berry and Howls (1990) M. Berry and C. Howls, “Fake airy functions and the asymptotics of reflectionlessness,” Journal of Physics A: Mathematical and General 23, L243 (1990).
  * Boozer (1983) A. H. Boozer, “Transport and isomorphic equilibria,” The Physics of Fluids 26, 496–499 (1983).
  * Helander (2014) P. Helander, “Theory of plasma confinement in non-axisymmetric magnetic fields,” Reports on Progress in Physics 77, 087001 (2014).
  * Rodriguez (2022) E. Rodriguez, “Quasisymmetry,” Ph.D. thesis (2022).
  * Freidberg (2014) J. P. Freidberg,  _Ideal MHD_ (Cambridge University Press, 2014) p. 302.
  * Burby, Kallinikos, and MacKay (2020) J. W. Burby, N. Kallinikos, and R. S. MacKay, “Some mathematics for quasi-symmetry,” Journal of Mathematical Physics 61, 093503 (2020).
  * Plunk and Helander (2018) G. G. Plunk and P. Helander, “Quasi-axisymmetric magnetic fields: weakly non-axisymmetric case in a vacuum,” Journal of Plasma Physics 84, 905840205 (2018).
  * Garren and Boozer (1991a) D. A. Garren and A. H. Boozer, “Magnetic field strength of toroidal plasma equilibria,” Physics of Fluids B: Plasma Physics 3, 2805–2821 (1991a).
  * Garren and Boozer (1991b) D. A. Garren and A. H. Boozer, “Existence of quasihelically symmetric stellarators,” Physics of Fluids B: Plasma Physics 3, 2822–2834 (1991b).
  * Landreman and Sengupta (2019) M. Landreman and W. Sengupta, “Constructing stellarators with quasisymmetry to high order,” Journal of Plasma Physics 85, 815850601 (2019).
  * Constantin, Drivas, and Ginsberg (2021) P. Constantin, T. D. Drivas, and D. Ginsberg, “On quasisymmetric plasma equilibria sustained by small force,” Journal of Plasma Physics 87, 905870111 (2021).
  * Rodríguez, Helander, and Bhattacharjee (2020) E. Rodríguez, P. Helander, and A. Bhattacharjee, “Necessary and sufficient conditions for quasisymmetry,” Physics of Plasmas 27, 062501 (2020).
  * Rodríguez, Sengupta, and Bhattacharjee (2022) E. Rodríguez, W. Sengupta, and A. Bhattacharjee, “Weakly quasisymmetric near-axis solutions to all orders,” Physics of Plasmas 29, 012507 (2022).
  * Landreman and Sengupta (2018) M. Landreman and W. Sengupta, “Direct construction of optimized stellarator shapes. Part 1. Theory in cylindrical coordinates,” Journal of Plasma Physics 84, 905840616 (2018).
  * Schief (2003) W. Schief, “Nested toroidal flux surfaces in magnetohydrostatics. generation via soliton theory,” Journal of plasma physics 69, 465–484 (2003).
  * Kamchatnov (2000) A. M. Kamchatnov,  _Nonlinear periodic waves and their modulations: an introductory course_ (World Scientific, 2000).
  * Courant and Snyder (1958) E. D. Courant and H. S. Snyder, “Theory of the alternating-gradient synchrotron,” Annals of physics 3, 1–48 (1958).
  * Magnus and Winkler (2013) W. Magnus and S. Winkler,  _Hill’s equation_ (Courier Corporation, 2013).
  * Lax (1968) P. D. Lax, “Integrals of nonlinear equations of evolution and solitary waves,” Communications on pure and applied mathematics 21, 467–490 (1968).
  * Tao (2009) T. Tao, “Why are solitons stable?” Bulletin of the American Mathematical Society 46, 1–33 (2009).
  * Gesztesy and Weikard (1996) F. Gesztesy and R. Weikard, “Picard potentials and hill’s equation on a torus,” Acta Mathematica 176, 73–107 (1996).
  * Dubrovin, Krichever, and Novikov (1990) B. A. Dubrovin, I. M. Krichever, and S. P. Novikov, “Integrable systems. i,” in  _Dynamical systems IV_ (Springer, 1990) pp. 173–280.
  * Froman (1979) N. Froman, “Dispersion relation for energy bands and energy gaps derived by the use of a phase-integral method, with an application to the Mathieu equation,” Journal of Physics A: Mathematical and General 12, 2355 (1979).
  * Wechsung  _et al._ (2022) F. Wechsung, M. Landreman, A. Giuliani, A. Cerfon, and G. Stadler, “Precise stellarator quasi-symmetry can be achieved with electromagnetic coils,” Proceedings of the National Academy of Sciences 119, e2202084119 (2022).
  * Rodríguez and Bhattacharjee (2021) E. Rodríguez and A. Bhattacharjee, “Islands and current singularities in quasisymmetric toroidal plasmas,” Physics of Plasmas 28, 092506 (2021).

##  Appendix A Derivation of $\mathcal{D}(\psi)$ in equation (12)

We present here the derivation of the function $\mathcal{D}(\psi)$, which
appears in

$\displaystyle\left(\frac{\partial
B}{\partial\ell}\right)^{2}=\mathcal{D}(\psi)(B_{M}-B)(B-B_{m})(B-B_{X}),$
(12)

in terms of the roots $B_{M}(\psi),B_{m}(\psi),B_{X}(\psi)$, $\iota(\psi)$,
the rotational transform, and the quasihelicity $N$. Helander (2014);
Landreman and Sengupta (2018)

For the derivation, we shall utilize the following facts. Firstly, $B$ is
periodic with period $L(\psi)$. Thus, $B(\ell)=B(\ell+L(\psi))$. Secondly, the
average $\oint Bd\ell$ is a flux function. To show this, we use Boozer
coordinates $(\psi,\vartheta,\phi)$ Landreman and Sengupta (2018), where the
Jacobian is given by

$\displaystyle\bm{B}\cdot\bm{\nabla}\vartheta=\iota_{N}\frac{B^{2}}{G(\psi)+\iota(\psi)I(\psi)},\quad\iota_{N}=\iota(\psi)-N.$
(17)

Note that $G(\psi),I(\psi)$ are flux functions related to the poloidal and
toroidal currents. Helander (2014) Since
$\bm{B}\cdot\bm{\nabla}=B\partial_{\ell}$ in $(\psi,\alpha,\ell)$, (17) takes
the form

$\displaystyle\frac{\partial\theta}{\partial\ell}=\iota_{N}\frac{B}{G+\iota
I}.$ (18)

Integrating (18) with respect to $\ell$ from $0$ to $L$ and noting that the
net change in the Boozer angle $\theta$ is $2\pi$, we get

$\displaystyle\int_{0}^{L}Bd\ell=2\pi\frac{G+\iota I}{\iota_{N}}.$ (19)

Finally, we shall need some basic identities satisfied by elliptic functions
(see Appendix A of reference [28]). In particular, we shall use

$\displaystyle\text{dn}(z,m)=\text{dn}(z+2\text{K}(m),m)$
$\displaystyle\text{dn}^{2}(z,m)=1-m+m\text{cn}^{2}(z,m)$
$\displaystyle\int_{0}^{\text{K}(m)}\text{dn}^{2}(u,m)du=\text{E}(m),$ (20)

where dn$(z,m)$, cn$(z,m)$ are standard Jacobi elliptic functions, and K$(m)$,
E$(m)$ are the complete elliptic integrals of the first and second
kind.Kamchatnov (2000)

We are now in a position to derive an expression for $\mathcal{D}(\psi)$. To
simplify the derivation, we shall work in the frame of reference where
$\partial_{\alpha}B=0$ (see discussion under (3)). In this frame, the solution
to (12) is

$\displaystyle B$
$\displaystyle=B_{X}+(B_{M}-B_{X})\text{dn}^{2}\left(\sqrt{\frac{B_{M}-B_{X}}{2}}\sqrt{\frac{\mathcal{D}}{2}}\ell,m\right),$
$\displaystyle m$ $\displaystyle=\frac{B_{M}-B_{m}}{B_{M}-B_{X}},\quad 0\leq
m\leq 1,$ (21)

which follows from (13),(15) and (20).

From the periodicity of $B$ and the Jacobi elliptic function dn (20), it
follows that

$\displaystyle\sqrt{\mathcal{D}}L(\psi)=\frac{4\text{K}(m)}{\sqrt{B_{M}-B_{X}}}.$
(22)

Moreover, integrating (21) over $\ell$ we obtain

$\displaystyle\int_{0}^{L}Bd\ell=L\left(B_{X}+(B_{M}-B_{X})\frac{\text{E}(m)}{\text{K}(m)}\right)$
(23)

Combining together the two different expressions of the $\oint Bd\ell$ given
by (19) and (23), and using the relation between $\mathcal{D}$ and $L$ from
(22), we finally obtain

$\displaystyle\sqrt{\mathcal{D}}=\frac{\iota_{N}}{G+\iota
I}\frac{\text{K}(m)}{\pi/2}\frac{\left(B_{X}+(B_{M}-B_{X})\frac{\text{E}(m)}{\text{K}(m)}\right)}{\sqrt{B_{M}-B_{X}}}$
(24)

##  Appendix B Additional configurations that satisfy the integrated KdV
equation (12)

Here, we present figures similar to Figs. 1 and 3 in the main text, using data
from different stellarator devices.

The precise QA configuration with larger negative shear in Fig. 6 was obtained
by a series of optimizations starting from the precise QA.Landreman and Paul
(2022) Specifically, we start from the same objective function used to
optimize the precise QA, and the precise QA configuration itself. The mean
shear $\bar{\hat{s}}$ was extracted from the precise QA configuration, and a
new least-squares term $(\bar{\hat{s}}-\bar{\hat{s}}_{*})^{2}$ was added to
the objective function, where the target value for mean shear
$\bar{\hat{s}}_{*}$ is taken to be slightly different from the extracted
value. This new objective is then optimized locally, using the precise QA as
the initial state. This ensures that the additional least-squares term is
almost zero, so the precise QA is approximately an optimum to the modified
objective function. Hence, the new local optimum can be expected to be close
to the initial configuration. This process is then repeated starting from the
slightly modified optimum, each step generating a similar equilibrium with a
slightly different mean shear. Mean shear is here defined as $b/\bar{\iota}$,
where $b$ is the result of fitting a first-order polynomial $a+bs$ to the iota
profile, where $s$ is the normalized toroidal flux; $\bar{\iota}$ is the
rotational transform averaged over all radii. The mean shear in the original
precise QA is about $-0.018$, and we incremented the target negative shear in
steps of $0.005$.

Figure 5: The value of $(dB/d\ell)^{2}$ on the outermost flux surface in the
precise QH Landreman and Paul (2022) as a function of $B$, alongside a cubic
polynomial fit (a), and the errors of quadratic, cubic, and quartic fits on
each flux surface (b). Panel (c) shows the behavior of the roots in the
precise QH.

Figure 6: The value of $(dB/d\ell)^{2}$ on the outermost flux surface in a
precise QA version with average shear -0.08 as a function of $B$, alongside a
cubic polynomial fit (a), and the errors of quadratic, cubic, and quartic fits
on each flux surface (b). Panel (c) shows the behavior of the roots in this
version of the precise QA.

Figure 7: The value of $(dB/d\ell)^{2}$ on the outermost flux surface in a
precise QA version with average shear 0.04 as a function of $B$, alongside a
cubic polynomial fit (a), and the errors of quadratic, cubic, and quartic fits
on each flux surface (b). Panel (c) shows the behavior of the roots in this
version of the precise QA.

Figure 8: The value of $(dB/d\ell)^{2}$ on the outermost flux surface in the
QA equilibrium that was used in Figure 2 in the main text as a function of
$B$, alongside a cubic polynomial fit (a), and the errors of quadratic, cubic,
and quartic fits on each flux surface (b). Panel (c) shows the behavior of the
roots in this QA equilibrium.

Figure 9: The value of $(dB/d\ell)^{2}$ on the outermost flux surface in a QA
equilibrium similar to that in Fig. 8 as a function of $B$, alongside a cubic
polynomial fit (a), and the errors of quadratic, cubic, and quartic fits on
each flux surface (b). Panel (c) shows the behavior of the roots in this QA
equilibrium.

Figs. 8 and 9 show equilibria that were not derived from the precise QA/QH.
Instead, they were only optimized for quasisymmetry on the outermost flux
surface, with the quasisymmetry degrading as the axis is approached. For both
cases, the ratio of symmetry-breaking modes to quasisymmetric modes near the
axis is $\sim 10^{-3}$. While this may seem a small value, the resulting
ripple creates enough noise in the $(dB/d\ell)^{2}$ vs. $B$ plot to render it
meaningless, and the error for all fits is $\sim 1$ in that region. The
symmetry-breaking modes were filtered out when calculating the roots shown in
panels (c). Note that the equilibrium shown in Fig. 8 is the same one used in
Fig. 2 in the main text.


# AutoLay: Benchmarking amodal layout estimation for autonomous driving

Kaustubh Mani∗ ∗ denotes equal contributionCorresponding author:
<EMAIL_ADDRESS>Robotics Research Center, KCIS IIIT Hyderabad  N. Sai
Shankar∗ Robotics Research Center, KCIS Krishna Murthy Jatavallabhula  Mila -
Quebec AI Institute, Montreal  Université de Montréal  K. Madhava Krishna
Robotics Research Center, KCIS IIIT Hyderabad

###### Abstract

Given an image or a video captured from a monocular camera, amodal layout
estimation is the task of predicting semantics and occupancy in bird’s eye
view. The term _amodal_ implies we also reason about entities in the scene
that are occluded or truncated in image space. While several recent efforts
have tackled this problem, there is a lack of standardization in task
specification, datasets, and evaluation protocols. We address these gaps with
_AutoLay_ , a dataset and benchmark for amodal layout estimation from
monocular images. _AutoLay_ encompasses driving imagery from two popular
datasets: KITTI [1] and Argoverse [2]. In addition to fine-grained attributes
such as lanes, sidewalks, and vehicles, we also provide semantically annotated
3D point clouds. We implement several baselines and bleeding edge approaches,
and release our data and code.111Project page:
https://hbutsuak95.github.io/AutoLay/.

##  I Introduction

Commercial interest in autonomous driving has led to the emergence of several
interesting and challenging problems, particularly in terms of perception. In
this work, we focus on the problem of _amodal scene layout estimation_
(introduced in [3]). _Amodal_ perception, as studied by cognitive scientists,
refers to the phenomenon of “imagining" or “hallucinating" parts of the scene
that do not result in sensory stimulation. In the context of urban driving, we
formulate the _amodal layout estimation_ task as predicting a bird’s eye view
semantic map from monocular images that explicitly reasons about entities in
the scene that are not visible from the image(s).

Amodal scene layout estimation is an emerging area of study where prior
efforts [3, 4, 5] have only focused on amodal completion for _coarse_
categories such as roads and sidewalks. While recent efforts have incorporated
dynamic objects [3], they fall short of modeling higher-level behaviors in
urban driving scenarios such as lane-keeping, merging, etc. Further, each of
these efforts [3, 4, 5] lack consensus in problem specifications and
evaluation protocols.

We address both the above issues. We first formalize the problem of _amodal
scene layout estimation_ that unifies existing work and extends the task to
several fine-grained categories. To foster research in this area, we provide
_AutoLay_ , a dataset comprising of over $16000$ images over a distance of
$12$ kilometers, annotated with fine-grained _amodal_ completions in 3D as
well as in bird’s eye view. _AutoLay_ includes $44$ sequences from KITTI [1]
and $89$ sequences from the Argoverse [2] datasets. We implement several
baselines and prior art, and open-source our code, models, and data [3, 4, 6].
We additionally propose _VideoLayout_ , a simple approach that achieves top
performance on the _AutoLay_ benchmark by leveraging temporal information
across image sequences, advancing the state-of-the-art of amodal layout
estimation.

Figure 2: Dataset: (_Left to right_) Sample images from the _KITTI_ split of
_AutoLay_. Corresponding annotated lidar pointclouds. Amodal scene layout in
bird’s eye view. (_Last column_) Distribution of semantic classes (bar plot),
and scene types (pie chart).

##  II Related work

Interpreting 3D scenes from an RGB image has been a longstanding challenge for
computer vision systems. In this section, we enlist a few approaches to
_amodal_ perception. For a comprehensive survey of general 3D scene
understanding, we refer the interested reader to Ozyecsil _et al._ [7].

#### Amodal perception for indoor scene understanding

In the context of indoor 3D scene understanding, a few approaches to _amodal_
perception have been proposed [8, 9]. While these approaches use coarse voxel
grids to represent objects, we instead focus on an orthographic top-down view
that restricts semantic understanding to road regions. We note that most of
the semantic understanding necessary for driving-related tasks can be obtained
on the road regions, making this simplification attractive.

#### Amodal perception for autonomous driving

To the best of our knowledge, there are no previous works that provide a
unified approach to obtain an amodal scene layout estimation in bird’s eye
view for fine-grained static and dynamic classes like lanes, sidewalks,
vehicles, etc. Existing approaches [4, 5, 10, 11] reason about “coarse-
grained" static classes such as roads and sidewalks. Schulter _et al._ [5]
obtain occlusion-aware bird’s eye view road layouts by hallucinating behind
objects tagged as _foreground_ in an image. Wang _et al._ [10] develop a
parametric representation of road scenes that reason about the number of
lanes, width of each lane, presence of an intersection, etc. Lu _et al._ [4]
use a variational autoencoder to predict road, sidewalk, terrain, and non
free-space in bird’s eye view. Lu  _et al._ [11] also perform vehicle shape
completion, albeit independent of the static layout estimation task. Garnett
_et al._ [12] predict static road and lane layouts by estimating lane
boundaries in 3D.

For dynamic scene understanding, several approaches aim to detect object
layouts in 3D. While some of these [13, 14, 15] combine information from
images and lidar, others [16, 17, 18] operate by converting images to bird’s
eye view representations, followed by object detection. However, these
techniques are often developed independently of static layout estimation
methods.

In earlier work, we presented MonoLayout [3]\- perhaps the first approach to
amodally reason about the static and dynamic layout of an urban driving scene.
While the interest in this nascent area is rapidly increasing, the absence of
standardized task specification and evaluation have been slowing down
research. This forms the primary motivation behind the _AutoLay_ dataset and
benchmark.

##  III Amodal scene layout estimation

We begin by formally defining the task of amodal scene layout estimation.
Given an image (or a sequence of images) $\mathcal{I}$, usually in perspective
view, we wish to learn a _labeling function_ $\Phi$ of _all_ points within an
area $\mathcal{R}$ (usually rectangular) around the camera. The region of
interest $\mathcal{R}$ is often in a different view than that of the
perspective image. In this paper, we focus on bird’s eye view, which refers to
a top-down orthographic view of the ground plane. The labeling function $\Phi$
must produce a label distribution (over a set of $N$ classes) for world points
within the swath $\mathcal{R}$, _regardless of whether or not they are imaged
in_ $\mathcal{I}$.

In particular, we propose the estimation of the following semantic classes
tasks in the context of this problem: road, sidewalk, crosswalk, lanes (ego-
lane, other lanes), other road, vehicle. Depending on the exact setup used
(single-image vs image-sequences, semantic categories to be estimated, etc.),
this enables fair evaluation of all approaches. We defer a detailed
description of evaluation protocols and its challenges to Sec. VI-D.

##  IV AutoLay Dataset

Over the last decade, several public datasets [1, 19, 20, 21, 2] have enabled
massive strides in autonomous driving research. While nearly all the above
datasets provide out-of-the-box support (annotations, evaluation utilities,
metrics, benchmarking support) for popular tasks like object detection,
semantic segmentation, multi-object tracking, trajectory forecasting, none of
these currently benchmark amodal scene layout estimation.

Obtaining training data for the task of amodal layout estimation involves
multiple challenges. It involves perceiving scene segments that are not even
imaged by the camera. While most datasets have lidar scans that often span a
larger sensor swath, such lidar beams are not dense enough to perceive thin
scene structures. Approaches such as _MonoLayout_ [3] compensate for this lack
of precision training data by adversarial learning, using OpenStreetMap [22]
data. More recently, Wang _et al._ [10] release a dataset that provides a
_parametric_ bird’s eye view representation for images from the KITTI [1] and
nuScenes [20] datasets, but the top-view representations are synthesized views
of a simulated parametric surface model. In summary, there is currently no
dataset that provides off-the-shelf support for benchmarking amodal layout
estimation. We fill this void with _AutoLay_ , a dataset for amodal layout
estimation in bird’s eye view (Fig. LABEL:fig:teaser_new).

###  IV-A Dataset overview

We use $44$ video sequences from the KITTI Raw dataset [1] in _AutoLay_. We
provide per-frame annotations in perspective, orthographic (bird’s eye view),
as well as in 3D. Of the $44$ annotated sequences, $24$ sequences—containing
$10705$ images—are used for training. The other $20$ sequences—comprising
$5626$ images—form the test set. This makes for nearly $16$K annotated images,
across a distance of $12$ Km, and a variety of urban scenarios (_residential_
, _city_ , _road_) (_cf._ Fig 2). The semantic classes considered in this
dataset are _road_ , _sidewalk_ , _vehicle_ , _crosswalk_ , and _lane_. Each
_lane_ segment is provided a unique id, which we classify further222We perform
this subclassification for our specific task, by projecting lane segments to
bird’s eye view. The _lane_ class is further classified as _ego-lane_ and
_other lane_. We also have an _other road_ class for road areas that do not
fall under any of the above categories.

###  IV-B Data annotation

Since annotating amodal layout from a single image is a hard task even for
professional annotators, we leverage multiple sources of information to guide
this procedure. We first build a point cloud map of the entire scene, annotate
the map (in full 3D), and then generate the other views (bird’s eye view). To
build a map of the entire area, we use the lidar scans corresponding to each
frame and register them to a global view. We use the precise odometry
information provided in KITTI for registration, to be robust to outliers, as
many scenes contain dynamic objects (vehicles/pedestrians). To reduce manual
effort, we semi-automate the data annotation pipeline to the extent possible.
As a first step, we identify lane markers by choosing a range of remission
values of the lidar and thresholding the 3D map. Once a set of lane markers
are identified, we tag as _lanes_ the area between pairs of lane markers.
Manual verification is performed to ensure no discrepancies creep in. Each
identified _lane_ is labeled with a different _lane id_ , unique within a
given sequence. Annotators are then presented with this preprocessed map in an
orthographic view and asked to amodally complete annotations. Specifically,
chunks of missing points in a lane are _filled in_ as _lanes_ , and so on.
Other identified chunks are annotated as _sidewalk_ , _crosswalk_ , or _other
road_. We repurpose the annotation tool from Semantic KITTI [23] for this.

Note that the above annotation procedure can only be used for static parts of
the scene. For dynamic scene components, such as vehicles, a 3D bounding box
annotation is performed per-frame. This entire annotated sequence is then
projected into bird’s eye view and semantics are made available in every
camera frame in the sequence.

##  V Approach

Figure 3: Weak Supervision setup: We show our automated (noisy) label
generation scheme herein. Points from the lidar frame are projected down to
semantically segmented images (a), to obtain sparse static layouts (b). These
sparse layouts are stacked across an image sequence to generate dense static
layouts (c). In the next step (d), road and lane boundaries are extracted and
combined to obtain lane layouts (e).

Learning the mapping function described in Sec. III which maps the perspective
RGB images to the static and dynamic layouts in bird’s eye view is quite a
challenging problem. The problem demands an accurate understanding and
modeling of the 3D scene which necessitates the requirement for learning good
visual features that can encode the depth and the semantics of the 3D scene.
Most approaches [5, 3, 4] to monocular layout estimation operate on a single
RGB image input, which often leads to temporally inconsistent predictions.

In this section, we tackle the aforementioned problem by proposing a neural
network architecture that takes a sequence of images as input and generates
temporally consistent layouts as output.

###  V-A _VideoLayout_ : Temporally Consistent Layout Estimation

Given a sequence of RGB images
$\mathcal{I}_{1},\mathcal{I}_{2},...,\mathcal{I}_{t}$ as input, our model
predicts the posterior distribution
$P(\mathcal{S}_{t},\mathcal{D}_{t}|{\mathcal{I}_{1},\mathcal{I}_{2},...,\mathcal{I}_{t}})$,
where $\mathcal{S}_{t}$ and $\mathcal{D}_{t}$ denote the predicted static and
dynamic layouts corresponding to image $\mathcal{I}_{t}$.

Our network architecture consists of four subparts:

  * •

A feature encoder which takes as input a sequence of RGB images
$\mathcal{I}_{1},\mathcal{I}_{2},...,\mathcal{I}_{t}$ and generates rich
multi-scale features maps
$\mathcal{C}_{1},\mathcal{C}_{2},...,\mathcal{C}_{t}$ corresponding to each
rgb image.

  * •

A stacked Convolutional LSTM submodule which aggregates image features
$\mathcal{C}_{1},\mathcal{C}_{2},...,\mathcal{C}_{t}$ and encodes a temporal
representation useful for estimating consistent layouts.

  * •

Similar to [3], we maintain two set of decoder weights corresponding to the
static and dynamic decoders. Both decoders share a similar architecture except
for the use of sampling layer at the beginning of the static decoder in order
to generate smooth looking road and lane layouts. Instead of directly
processing the encoded output representation of convLSTM, we instead sample
from a gaussian distribution with the feature map as the mean values and a
fixed standard deviation.

  * •

A Refinement Network, which helps in improving the quality of the output
static and dynamic layouts by regularizing the predicted layouts in order to
resemble the true data distribution.

###  V-B Consistency Loss

The weights of the _VideoLayout_ architecture are updated via backpropagation
using the loss function defined in Eq. 4.

$\mathcal{L}_{sup}=\sum_{i=1}^{N}f(\bar{S}_{i},S_{i})+f(\bar{D}_{i},D_{i})$
(1)

$\mathcal{L}_{short}^{c}=\sum_{i=1}^{N}\sum_{j=1}^{seqlen-1}f(\bar{S}_{i}^{j},\bar{S}_{i}^{j+1})+f(\bar{D}_{i}^{j},\bar{D}_{i}^{j+1})$
(2)

$\mathcal{L}_{long}^{c}=\sum_{i=1}^{N}\sum_{j=1}^{seqlen-1}\sum_{k=j+2}^{seqlen}f(\bar{S}_{i}^{j},\bar{S}_{i}^{k})+f(\bar{D}_{i}^{j},\bar{D}_{i}^{k})$
(3)

$\mathcal{L}=\lambda_{sup}*\mathcal{L}_{sup}+\lambda_{short}^{c}*\mathcal{L}_{short}^{c}+\lambda_{long}^{c}*\mathcal{L}_{long}^{c}$
(4)

Here, $\bar{S}$ and $\bar{D}$ denote the predicted static and dynamic layouts
respectively, $S$ and $D$ denote the ground truth (weak/strong) layouts. $N$
is the mini batch size, $seqlen$ is the length of the input image sequence per
sample. $f(\cdot,\cdot)$ is the cross-entropy loss function.
$\mathcal{L}_{sup}$ is a supervised loss term that penalizes the deviation of
the predicted static and dynamic layouts. $\mathcal{L}_{short}^{c}$ is the
short-range consistency loss and $\mathcal{L}_{long}^{c}$ is the long-range
consistency loss. $\lambda_{sup}$, $\lambda_{short}^{c}$ and
$\lambda_{long}^{c}$ are the weights corresponding to the supervised, short-
range consistency and long-range consistency losses respectively. Finally,
$\mathcal{L}$ is the total weighted loss used for backpropagating gradients
through the network.
333$\lambda_{sup}>\lambda_{short}^{c}>>\lambda_{long}^{c}$. Consistency loss
acts as a soft constraint on the layout prediction objective
$\mathcal{L}_{sup}$, which enables the model to learn more temporally
consistent layout.

###  V-C Refinement Network

Similar to [3], we use adversarial regularization in order to restrict the
encoder-decoder network into predicting conceivable layouts. For adversarial
regularization, we use a Patch Discriminator which takes the output of the
encoder-decoder and a sample from the true data distribution. Unlike [3], we
make use of ground-truth layouts to do paired training with the discriminator
which further improves the performance of the model. Adding the sampling layer
in the static decoder converts the Encoder-Decoder network into a generator
and further helps in regularization for static layouts.

###  V-D Data Preparation: Weak Supervision

Since the amodal scene layout labels are not provided for almost every
autonomous driving dataset, we propose a generic layout generation method to
generate noisy or weak supervision layout labels for static classes like
roads, lanes, and sidewalks (_cf._ Fig. 3). Our weak supervision layout
generation method is similar to the sensor fusion approach adopted in [3].
Further, we provide a solution to obtain weak supervision layout labels for
lanes as well.

The process of aggregating / registering the lidar points for several image
frames and projecting them to an orthographic view by assuming a flat plane is
shown in Fig. 3. To obtain lanes, we perform the registering process for road
and lane marker semantic classes. We then obtain lane boundary curves by
isolating individual lane boundaries through Density Based Spatial Clustering
and fit third order polynomial curves on these individual lane boundaries.
From the registered road layout, we estimate the road boundaries and group the
road and lane boundaries together to obtain lane contours.

Additionally, we also obtain improved weak supervision layout for roads by
addressing cases where there are too many stationary obstacles in a sequence
occluding lidar from capturing the road layout. We register lidar points
belonging to those stationary obstacles that lie on the road.

##  VI Experiments

Figure 4: Static layout estimation: _VideoLayout_ predicts fine-grained
attributes of the road scene including lanes, side roads and ego-lane. Each
individual color represents a single lane. Ego-lane is shown in blue and side-
roads are shown in light orange. _VideoLayout_ produces decent static layout
estimates for both AutoLay and Argoverse datasets. Its able to hallucinate
occluded regions in the scene very reliably specially for Argoverse dataset.

So far, there has been a lack of consensus on datasets and evaluation protocol
for amodal layout estimation. We describe the _AutoLay_ evaluation protocol in
detail, and evaluate baselines and existing approaches on the benchmark.

###  VI-A Datasets

To ensure fair comparison with existing methods, we show our results on the
Argoverse [2] and KITTI RAW split used in [3] and [5] respectively. Argoverse
[2] provides detailed high-resolution semantic maps in bird’s eye view for
roads, vehicles and lanes, which makes it suitable for evaluation on all the
tasks proposed in Sec. III. We present a breakdown of the _AutoLay_ benchmark
below.

  1. 1.

_KITTI_ split: The _KITTI_ split of _AutoLay_ comprises $10705$ train images
and $5626$ test images. We report results for models trained using weak
(proposed data preparation) and strong (ground-truth) supervision.

  2. 2.

_KITTI RAW_ split: For fair evaluation with Schulter  _et al._ [5], we use the
KITTI RAW split comprising $16269$ train images and $2305$ test images).

  3. 3.

_Argoverse_ split: The Argoverse [2] split contains $6722$ train images and
$2418$ test images.

###  VI-B Implementation Details

####  VI-B1 PseudoLidar input

We evaluate the performance of a set of approaches that take PseudoLidar[18]
as input for amodal scene layout estimation. The PseudoLidar representation is
a processed point cloud obtained by projecting the RGB image pixels to the
camera coordinate system using the per-pixel depth values obtained from
unsupervised monocular depth estimation methods like [24]. We substitute the
remission value by the normalized mean of RGB channel intensities. We then
obtain a voxel representation of the point cloud by slicing it vertically
along the X-Z plane with a resolution of 0.15625 m and horizontally along the
Y-axis into 10 channels starting from 0.4 m above the camera to 2 m below the
camera. This PseudoLidar representation is provided as the input to ENet [25]
or Unet [26] for amodal scene layout estimation.

####  VI-B2 Lane layout convention

In order to convert lane estimation, which is essentially an instance
segmentation/detection task in bird’s eye view to an occupancy prediction
task, we adhere to the following convention while performing lane layout
estimation. While creating the per frame training labels using both weak and
strong supervision, we classify the individual lane layouts on the road with
the ego-vehicle, (_ego-road_), as either _ego lane_ or _other lanes_. The ego
lane and the other lanes are provided individual lane ids. The lane ids to the
other lanes are given on the basis of their position with respect to the ego-
lane.

###  VI-C Evaluation Metrics

####  VI-C1 Vehicle Layout

Similar to [3], We evaluate the vehicle layouts on both mean intersection-
over-union(mIoU) and mean average-precision(mAP). Since we are evaluating
different methods on the task of vehicle occupancy prediction in bird’s eye
view, we don’t go for the traditional evaluation metric($AP_{IoU>0.7}$ or
$AP_{IoU>0.5}$) used by 3D object detection methods or object detection
methods in bird’s eye view.

####  VI-C2 Road Layout

We adopt mean intersection-over-union(mIoU) and mean average-precision(mAP) as
our metric for evaluating road layouts on the datasets. To evaluate
hallucination capabilities of different methods, we also make use of the
occluded mean intersection-over-union (IoU) used in [5], [3]. This involves
calculating IoU for only the portions of the road which are occluded.

####  VI-C3 Lane Layout

As mentioned in Sec. III, Lane Layouts are evaluated on two separate tasks:
Ego-lane estimation and Overall lane detection.

  1. 1.

Ego-Lane Estimation: Ego-Lane estimation task is evaluated on the mean
intersection-over-union (mIoU) and mean average-precision (mAP) metric.

  2. 2.

Overall Lane Detection: For this task, we use $AP_{IoU>0.7}$, a popular metric
used for evaluating object detection and instance segmentation methods. A
predicted lane is counted as a detection if its intersection-over-union(IoU)
with one of the ground truth lanes is greater than 0.7(70%). $AP_{IoU>0.7}$
provides information about the average-precision of predicted detections. We
also report $Recall_{IoU>0.7}$, which gives us an idea about the prediction
capability of the method.

###  VI-D Evaluated Methods

We evaluate the performance of the following approaches.

  * •

_Schulter _et al.__ : The static scene layout estimation approach proposed in
[5].

  * •

_MonoOccupancy_ : The static scene layout estimation approach proposed in [4].

  * •

_MonoOccupancy-ext_ : We extend MonoOccupancy [4] to predict vehicle
occupancies.

  * •

_PseudoLidar-ENet_ : A ENet [25] architecture with PseudoLidar input for
amodal scene layout estimation.

  * •

_PseudoLidar-UNet_ : A UNet [26] architecture with PseudoLidar input for
amodal scene layout estimation.

  * •

_MonoLayout_ : Amodal scene layout estimation architecture proposed in [3]

  * •

_VideoLayout_ : The full _VideoLayout_ architecture trainned with temporal-
consistency loss and the adversarial refinement network.

Vehicle Layout Static Layout Estimation Road Lane Dataset Method Ego-Lane
Overall mIoU mAP mIoU mAP mIoU mAP $AP_{\small{iou>0.7}}$ $Recall_{iou>0.7}$
KITTI PseudoLidar-UNet $20.10$ $52.47$ $57.62$ $70.90$ $51.98$ $68.94$ $8.53$
$19.09$ PseudoLidar-ENet $15.26$ $40.05$ $55.70$ $67.62$ $54.42$ $68.15$
$13.01$ $24.50$ MonoOccupancy-ext $27.03$ $42.92$ $63.10$ $77.56$ $58.39$
$69.16$ $26.11$ $35.36$ MonoLayout[3] $30.88$ $53.63$ $63.83$ $77.53$ $58.27$
$72.08$ $16.09$ $34.91$ VideoLayout $\mathbf{34.60}$ $\mathbf{52.87}$
$\mathbf{68.73}$ $\mathbf{84.89}$ $\mathbf{62.68}$ $\mathbf{76.80}$
$\mathbf{38.56}$ $\mathbf{44.33}$ Argoverse MonoOccupancy-ext $16.22$ $38.66$
$72.41$ $79.62$ $75.26$ $85.51$ $39.76$ $51.65$ MonoLayout[3] $28.31$ $46.07$
$73.25$ $84.56$ $71.73$ $82.00$ $26.18$ $42.61$ VideoLayout $\mathbf{32.77}$
$\mathbf{50.48}$ $\mathbf{76.42}$ $\mathbf{88.01}$ $\mathbf{77.62}$
$\mathbf{87.77}$ $\mathbf{46.03}$ $\mathbf{54.86}$

Table I: Quantitative results: We benchmark _VideoLayout_ , _MonoLayout_ [3],
MonoOccupancy-ext and PseudoLidar based approaches on KITTI and Argoverse
datasets.

Road Road Sidewalk + Lane Method Sidewalk mIOU mIOU occ mIOU
$AP_{\small{iou>0.7}}$ MonoOccupancy[4] $56.16$ $18.18$ $28.24$ $26.64$
Schulter _et al._ [5] $68.89$ $30.35$ $61.06$ - MonoLayout[3] $73.86$ $32.86$
$67.42$ $16.20$ _VideoLayout_ $\mathbf{75.92}$ $\mathbf{37.74}$
$\mathbf{71.01}$ $\mathbf{36.64}$

Table II: State-of-the-art comparision for static layout(Road, Sidewalk and
Lane) with weak supervision(_cf._ Sec. V-D) on KITTI RAW split used in [5]
Figure 5: Dynamic Layout Estimation:. _VideoLayout_ provides crisp and
accurate vehicle occupancies. The grey boxes indicate vehicle occupancies.
Observe the ability of _VideoLayout_ to precisely localize vehicles that are
distant and partially occluded.

###  VI-E Road Layout Estimation

We evaluate PseudoLidar based (PseudoLidar-UNet, PseudoLidar-ENet) and RGB
image based architectures (MonoOccupancy [4], _MonoLayout_ and _VideoLayout_)
for the task of road layout estimation using the ground truth labels. Table I
provides a detailed benchmark of these methods on the _AutoLay_ and
Argoverse[2] datasets. We also perform state-of-the-art comparision(Table II)
with previously proposed methods in literature on KITTI RAW split proposed in
[5] on the task of road and lane layout estimation. For this comparision, we
use the weak data generation method mentioned in Sec V-D.

We observe that _MonoLayout_ and _VideoLayout_ perform better than
PseudoLidar-based methods for road layout estimation. This can be attributed
to the sparsity of the PseudoLidar points as the distance from camera
increases. Also, PseudoLidar based inputs don’t allow the use of deep feature
encoders like ResNet[27] needed to extract the rich visual features necessary
for hallucinating amodal scene layouts. From Table I it can be seen that
MonoOccupancy-ext performs better than the PseudoLidar counterparts. This
further exemplifies the narrative that with better supervision, an RGB image-
based architecture can reason for occluded regions better than pseudo-lidar
based approaches.

Within the RGB image based approaches, we observe that _VideoLayout_
outperforms _MonoLayout_ , MonoOccupancy [4] by considerable margins for all
the evaluation metrics. The same can be also be observed in the results on
KITTI RAW (_cf._ Table II).

###  VI-F Vehicle Layout Estimation

From Table I, we can see that PseudoLidar-based methods have inferior
performance in comparision to monocular methods[3, 4] on vehicle layout
estimation. This can again be attributed to the fact that monocular depths are
inprecise and sparse at large distances, which makes the PseudoLidar input
unreliable for this task.

Similar to road layout estimation, _VideoLayout_ performs better than
_MonoLayout_ due to the temporal consistency loss and the adversarial
regularization via the refinement network (_cf._ Fig. 5). Also, MonoOccupany
[4] performs poorly on vehicle layout estimation, presumably due to
_blurriness_ in the variational autoencoder used therein leading to low mAP
scores.

###  VI-G Lane Layout Estimation

We split the lane layout estimation into two tasks: _ego-lane estimation_ and
_overall lane detection_. This exclusive treatment of ego-lane estimation is
due to its prominence in semi and fully autonomous driving. Unlike objects,
which can be “detected" as bounding boxes, lanes are objects whose extents
span potentially the entire width/height of the bird’s eye view image, and
this calls for an appropriate evaluation protocol. We thus treat lane
estimation akin to a “segmentation" problem, and compare performance in terms
of average precision and recall at varying IoU thresholds ($0.5$, $0.7$). From
Table I, we see that that MonoOccupancy [4] performs significantly better than
PseudoLidar methods on ego-lane estimation.

_VideoLayout_ significantly outperforms other methods, owing to its ability to
encode temporal information and the use of a separate refinement network. The
refinement network corrects blobby lane estimates that do not match the true
data distribution, resulting in performance boosts at higher IoU thresholds. A
few qualitative results are presented in Fig. 4.

Ego-Lane OverAll Method mIOU mAP $AP_{\small{iou>0.7}}$ $R_{\small{iou>0.7}}$
VideoLayout (no samp., no reg.) $59.81$ $73.13$ $20.95$ $36.55$ VideoLayout
(no reg.) $61.15$ $75.72$ $35.89$ $41.92$ VideoLayout (full) $62.68$ $76.80$
$38.56$ $44.33$

Table III: Analyzing effect of various subparts of the _VideoLayout_
architecture on Lane Layout estimation performance. The sampling layer and
adversarial regularization improve performance.

Method $AP_{\small{iou>0.5}}$ $R_{\small{iou>0.5}}$ $AP_{\small{iou>0.7}}$
$R_{\small{iou>0.7}}$ PseudoLidar-UNet $19.32$ $43.24$ $8.53$ $19.09$
PseudoLidar-ENet $25.10$ $47.39$ $13.01$ $24.50$ MonoLayout[3] $25.38$ $55.06$
$16.09$ $34.91$ VideoLayout $57.02$ $65.49$ $38.56$ $44.33$

Table IV: Analyzing Lane layout scores at IoU thresholds of $0.5$ and $0.7$.
$AP$ denotes average precision. $R$ denotes Recall. _VideoLayout_ consistently
performs better across IoU thresholds.

###  VI-H Ablation Studies

We conduct ablation studies to analyze the performance of various components
in the pipeline. These lead to a number of interesting observations that we
enlist below.

####  VI-H1 Sensitivity of lane estimates to IoU thresholds

In Table IV, we analyze the sensitivity of lane estimates to IoU thresholds.
At lower thresholds ($AP_{IoU>0.5}$ and $R_{IoU>0.5}$), PseudoLidar-based
techniques have similar performance to that of MonoLayout [3]. But as the
threshold increases ($AP_{IoU>0.7}$ and $R_{IoU>0.7}$), PseudoLidar methods
exhibit a dramatic performance drop, particularly in terms of recall. We
attribute this to the fact that PseudoLidar relies on monocular depth
estimates which are sparse and inaccurate for vehicles farther from the
camera. On the other hand, _VideoLayout_ performs consistently across both IoU
thresholds. Particularly at $AP_{IoU>0.5}$, we observe a significant
performance increase compared to _MonoLayout_.

####  VI-H2 Effect of Sampling and Adversarial Refinement

Table III shows the performance of different variants of _VideoLayout_ for
lane layout estimation. _VideoLayout_ (no samp., no reg.) refers to a variant
without a sampling layer in the static decoder and without adversarial
regularization. _VideoLayout_ (no reg.) uses sampling layer in the static
decoder. _VideoLayout_ (full) uses both sampling layer and an adversarial
regularizer. It can be seen that each component meaningfully contributes to
performance boosts. The sampling layer enhances the capability of the model to
capture the distribution of _plausible_ layouts, improving precision
($AP_{IoU>0.7}$). The adversarial regularizer improves the sharpness of the
predicted samples, improving the recall $R_{IoU>0.7}$.

Figure 6: Failure Cases of _VideoLayout_. (_Top row_) Failure to predict ego-
lane. (_Middle row_) Failure in heavy traffic scenarios. (_Bottom row_)
Failure in predicting lane width.

###  VI-I Failure cases

Despite its impressive performance, _VideoLayout_ fails in multiple scenarios.
A few representative failure cases are shown in Fig. 6. Often, such failures
are due to high-dynamic range conditions, where shadows are mistaken for
lanes, or in heavy traffic scenarios.

###  VI-J Runtime

On an NVIDIA RTX 1080Ti, the inference rate of _VideoLayout_ is slightly over
$40$ frames per second, which is suitable for real-time performance.

##  VII Conclusion

Amodal layout estimation is an emerging perception task for autonomous
driving. Solving this task necessitates an understanding of not just entities
present in an image, but also a reasoning of occluded portions. Benchmarks
have driven progress in allied perception tasks such as object detection,
segmentation, tracking, depth estimation, and more. In that spirit, we
introduce a new benchmark (_AutoLay_) for amodal layout estimation. We
implement several baselines and prior art, and make all our code and data
available. We further propose _VideoLayout_ , a simple yet effective
technique, and demonstrate that reasoning about image sequences leads to
coherent and robust layout estimates. We hope the dataset and benchmark serve
as a breeding ground for a new class of approaches that will take us a step
closer to understanding the 3D world from a sequence of its projections.

## References

  * [1] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in _CVPR_ , 2012.
  * [2] M.-F. Chang, J. Lambert, _et al._ , “Argoverse: 3d tracking and forecasting with rich maps,” in _CVPR_ , 2019.
  * [3] K. Mani, S. Daga, _et al._ , “Monolayout: Amodal layout estimation from a single image,” _IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2020.
  * [4] C. Lu, M. J. G. van de Molengraft, and G. Dubbelman, “Monocular semantic occupancy grid mapping with convolutional variational encoder-decoder networks,” _IEEE Robotics and Automation Letters_ , 2019.
  * [5] S. Schulter, M. Zhai, N. Jacobs, and M. Chandraker, “Learning to look around objects for top-view representations of outdoor scenes,” in _ECCV_ , 2018\.
  * [6] Y. You, Y. Wang, _et al._ , “Pseudo-lidar++: Accurate depth for 3d object detection in autonomous driving,” _arXiv preprint_ , 2019.
  * [7] O. Özyeşil, V. Voroninski, R. Basri, and A. Singer, “A survey of structure from motion*.” _Acta Numerica_ , vol. 26, pp. 305–364, 2017.
  * [8] A. Kar, S. Tulsiani, J. Carreira, and J. Malik, “Amodal completion and size constancy in natural scenes,” in _International Conference on Computer Vision (ICCV)_ , 2015.
  * [9] S. Tulsiani, S. Gupta, _et al._ , “Factoring shape, pose, and layout from the 2d image of a 3d scene,” in _Computer Vision and Pattern Regognition (CVPR)_ , 2018.
  * [10] Z. Wang, B. Liu, S. Schulter, and M. Chandraker, “A parametric top-view representation of complex road scenes,” in _CVPR_ , 2019.
  * [11] C. Lu and G. Dubbelman, “Hallucinating beyond observation: Learning to complete with partial observation and unpaired prior knowledge,” _arXiv preprint_ , 2019.
  * [12] N. Garnett, R. Cohen, _et al._ , “3d-lanenet: end-to-end 3d multiple lane detection,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 2921–2930.
  * [13] J. Ku, M. Mozifian, _et al._ , “Joint 3d proposal generation and object detection from view aggregation,” in _IROS_ , 2018.
  * [14] X. Chen, H. Ma, _et al._ , “Multi-view 3d object detection network for autonomous driving,” in _CVPR_ , 2017.
  * [15] M. Liang, B. Yang, S. Wang, and R. Urtasun, “Deep continuous fusion for multi-sensor 3d object detection,” in _ECCV_ , 2018.
  * [16] B. Yang, W. Luo, and R. Urtasun, “Pixor: Real-time 3d object detection from point clouds,” in _CVPR_ , 2018.
  * [17] T. Roddick, A. Kendall, and R. Cipolla, “Orthographic feature transform for monocular 3d object detection,” _arXiv preprint_ , 2018.
  * [18] Y. Wang, W.-L. Chao, _et al._ , “Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving,” in _CVPR_ , 2019.
  * [19] W. Maddern, G. Pascoe, C. Linegar, and P. Newman, “1 Year, 1000km: The Oxford RobotCar Dataset,” _The International Journal of Robotics Research (IJRR)_ , vol. 36, no. 1, pp. 3–15, 2017.
  * [20] H. Caesar, V. Bankiti, _et al._ , “nuscenes: A multimodal dataset for autonomous driving,” _arXiv preprint arXiv:1903.11027_ , 2019.
  * [21] X. Huang, P. Wang, _et al._ , “The apolloscape open dataset for autonomous driving and its application,” _arXiv preprint arXiv:1803.06184_ , 2018.
  * [22] OpenStreetMap contributors, “Planet dump retrieved from https://planet.osm.org ,” https://www.openstreetmap.org, 2017.
  * [23] J. Behley, M. Garbade, _et al._ , “Semantickitti: A dataset for semantic scene understanding of lidar sequences,” in _ICCV_ , 2019.
  * [24] C. Godard, O. Mac Aodha, M. Firman, and G. Brostow, “Digging into self-supervised monocular depth estimation,” _arXiv preprint_ , 2018.
  * [25] A. Paszke, A. Chaurasia, S. Kim, and E. Culurciello, “Enet: A deep neural network architecture for real-time semantic segmentation,” _arXiv preprint arXiv:1606.02147_ , 2016.
  * [26] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_ , 2015.
  * [27] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.


the text how they were calculated and reference the corresponding figures or
tables in the text.

  36. 8.

Experiments Compute Resources

  37. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

  38. Answer: [Yes]

  39. Justification: See Appendix I

  40. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The paper should indicate the type of compute workers CPU or GPU, internal
cluster, or cloud provider, including relevant memory and storage.

     * •

The paper should provide the amount of compute required for each of the
individual experimental runs as well as estimate the total compute.

     * •

The paper should disclose whether the full research project required more
compute than the experiments reported in the paper (e.g., preliminary or
failed experiments that didn’t make it into the paper).

  41. 9.

Code Of Ethics

  42. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

  43. Answer: [Yes]

  44. Justification: The paper meets the NeurIPS Code of Ethics.

  45. Guidelines:

     * •

The answer NA means that the authors have not reviewed the NeurIPS Code of
Ethics.

     * •

If the authors answer No, they should explain the special circumstances that
require a deviation from the Code of Ethics.

     * •

The authors should make sure to preserve anonymity (e.g., if there is a
special consideration due to laws or regulations in their jurisdiction).

  46. 10.

Broader Impacts

  47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

  48. Answer: [Yes]

  49. Justification: See abstract, introduction and conlusions of our work.

  50. Guidelines:

     * •

The answer NA means that there is no societal impact of the work performed.

     * •

If the authors answer NA or No, they should explain why their work has no
societal impact or why the paper does not address societal impact.

     * •

Examples of negative societal impacts include potential malicious or
unintended uses (e.g., disinformation, generating fake profiles,
surveillance), fairness considerations (e.g., deployment of technologies that
could make decisions that unfairly impact specific groups), privacy
considerations, and security considerations.

     * •

The conference expects that many papers will be foundational research and not
tied to particular applications, let alone deployments. However, if there is a
direct path to any negative applications, the authors should point it out. For
example, it is legitimate to point out that an improvement in the quality of
generative models could be used to generate deepfakes for disinformation. On
the other hand, it is not needed to point out that a generic algorithm for
optimizing neural networks could enable people to train models that generate
Deepfakes faster.

     * •

The authors should consider possible harms that could arise when the
technology is being used as intended and functioning correctly, harms that
could arise when the technology is being used as intended but gives incorrect
results, and harms following from (intentional or unintentional) misuse of the
technology.

     * •

If there are negative societal impacts, the authors could also discuss
possible mitigation strategies (e.g., gated release of models, providing
defenses in addition to Attacks, mechanisms for monitoring misuse, mechanisms
to monitor how a system learns from feedback over time, improving the
efficiency and accessibility of ML).

  51. 11.

Safeguards

  52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

  53. Answer: [N/A]

  54. Justification: The paper poses no such risks.

  55. Guidelines:

     * •

The answer NA means that the paper poses no such risks.

     * •

Released models that have a high risk for misuse or dual-use should be
released with necessary safeguards to allow for controlled use of the model,
for example by requiring that users adhere to usage guidelines or restrictions
to access the model or implementing safety filters.

     * •

Datasets that have been scraped from the Internet could pose safety risks. The
authors should describe how they avoided releasing unsafe images.

     * •

We recognize that providing effective safeguards is challenging, and many
papers do not require this, but we encourage authors to take this into account
and make a best faith effort.

  56. 12.

Licenses for existing assets

  57. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

  58. Answer: [Yes]

  59. Justification: The creators or original owners of assets we use are get from public access and cited.

  60. Guidelines:

     * •

The answer NA means that the paper does not use existing assets.

     * •

The authors should cite the original paper that produced the code package or
dataset.

     * •

The authors should state which version of the asset is used and, if possible,
include a URL.

     * •

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

     * •

For scraped data from a particular source (e.g., website), the copyright and
terms of service of that source should be provided.

     * •

If assets are released, the license, copyright information, and terms of use
in the package should be provided. For popular datasets,
paperswithcode.com/datasets has curated licenses for some datasets. Their
licensing guide can help determine the license of a dataset.

     * •

For existing datasets that are re-packaged, both the original license and the
license of the derived asset (if it has changed) should be provided.

     * •

If this information is not available online, the authors are encouraged to
reach out to the asset’s creators.

  61. 13.

New Assets

  62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

  63. Answer: [Yes]

  64. Justification: It is available at the link we offer in the abstract.

  65. Guidelines:

     * •

The answer NA means that the paper does not release new assets.

     * •

Researchers should communicate the details of the dataset/code/model as part
of their submissions via structured templates. This includes details about
training, license, limitations, etc.

     * •

The paper should discuss whether and how consent was obtained from people
whose asset is used.

     * •

At submission time, remember to anonymize your assets (if applicable). You can
either create an anonymized URL or include an anonymized zip file.

  66. 14.

Crowdsourcing and Research with Human Subjects

  67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

  68. Answer: [N/A]

  69. Justification: The paper does not involve crowdsourcing nor research with human subjects.

  70. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Including this information in the supplemental material is fine, but if the
main contribution of the paper involves human subjects, then as much detail as
possible should be included in the main paper.

     * •

According to the NeurIPS Code of Ethics, workers involved in data collection,
curation, or other labor should be paid at least the minimum wage in the
country of the data collector.

  71. 15.

Institutional Review Board (IRB) Approvals or Equivalent for Research with
Human Subjects

  72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

  73. Answer: [N/A]

  74. Justification: The paper does not involve crowdsourcing nor research with human subjects.

  75. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Depending on the country in which research is conducted, IRB approval (or
equivalent) may be required for any human subjects research. If you obtained
IRB approval, you should clearly state this in the paper.

     * •

We recognize that the procedures for this may vary significantly between
institutions and locations, and we expect authors to adhere to the NeurIPS
Code of Ethics and the guidelines for their institution.

     * •

For initial submissions, do not include any information that would break
anonymity (if applicable), such as the institution conducting the review.