# On the supersymmetry of the Klein-Gordon oscillator
Dedicated to Akira Inomata on the occasion of his 90th birthday

Georg Junker<EMAIL_ADDRESS><EMAIL_ADDRESS>European Organization for
Astronomical Research in the Southern Hemisphere, Karl-Schwarzschild-Straße 2,
D-85748 Garching, Germany
and
Institut für Theoretische Physik I, Universität Erlangen-Nürnberg,
Staudtstraße 7, D-91058 Erlangen, Germany;

###### Abstract

The three-dimensional Klein-Gordon oscillator is shown to exhibit an algebraic
structure known from supersymmetric quantum mechanics. The supersymmetry is
found to be unbroken with a vanishing Witten index, and it is utilized to
derive the spectral properties of the Klein-Gordon oscillator, which is
closely related to that of the non-relativistic harmonic oscillator in three
dimensions. Supersymmetry also enables us to derive a closed-form expression
for the energy-dependent Green’s function.

Klein-Gordon oscillator, supersymmetric quantum mechanics, Green’s function

##  I Introduction

Starting with Galileo’s pendulum experiment [1] in 1602 and with Hook’s law of
elasticity [2] from 1678, harmonic oscillators played significant roles in
classical physics. More importantly, the harmonic oscillator was the first
system to which early quantum theory was successfully applied by Planck [3] in
1900 when developing his law of black body radiation. Nowadays the harmonic
oscillator is a standard part of any introductory text book on non-
relativistic quantum mechanics. In relativistic quantum mechanics the harmonic
oscillator was initially studied within Dirac’s theory of electrons in the
1960s [4, 5, 6] but attracted considerable attention only with the seminal
work by Moshinsky and Szczepaniak [7], see also Quesne and Moshinsky [8].
Inspired by this so-called Dirac oscillator, the Klein-Gordon oscillator (KGO)
has been studied by various authors [9, 10, 11].

The KGO Hamiltonian characterises a relativistic spin-zero particle with mass
$m$ minimally coupled to a complex linear vector potential. Since its
introduction, the KGO has and still is attracting interest. The spectral
properties of the one-dimensional system were discussed, for example, in [12,
13]. For a treatment in non-commutative space see the work [14, 15], for
recent results in a non-trivial topology see [16, 17, 18, 19] and references
therein.

At least since 1990 it is know that the Dirac oscillator exhibits a
supersymmetric (SUSY) structure, which in turn allows for explicit solutions
[20, 21, 23, 22]. More recently, SUSY also enabled us to formulate Feynman’s
path integral approach for Dirac systems [24]. Let us emphasize that SUSY in
the current context is not based on the original idea which transforms between
states with different internal spin-degree of freedom. SUSY here refers to
what nowadays is commonly known as supersymmetric quantum mechanics, see for
example [25] and reference therein.

The purpose of the present work is two-fold. First we show that the Klein-
Gordon oscillator possesses a hidden SUSY in the aforementioned sense.
Secondly we will derive an explicit expression for the Green function of the
KGO. In doing so we will closely follow the generic approach for SUSY in
relativistic Hamiltonians with fixed but arbitrary spin [26].

In the next section we will set up the stage with a brief discussion on the
KGO Hamiltonian in three space dimensions and show that this Hamiltonian
exhibits a SUSY structure by mapping it onto a SUSY quantum mechanical system.
This fact is then utilized to derive explicit results of the system. That is,
in section 3 we will derive the eigenvalues and associated eigenstates,
followed by section 4 where we derive the corresponding Green’s function in a
closed form. Finally, section 5 closes with a summary and some comments.

##  II Supersymmetry of the KGO

The Hamiltonian form of the Klein-Gordon equation with an arbitrary vector
potential was originally introduced by Feshbach and Villars [27] from which
the KGO Hamiltonian may be constructed via the minimal coupling
$\vec{p}\to\vec{\pi}:=\vec{p}-\textrm{i}m\omega\vec{r}$, where $m>0$ stands
for the mass of the spin-less Klein-Gordon particle and $\omega>0$ is a
coupling constant to be identified with the harmonic oscillator frequency.

This minimal coupling might be interpreted as a complex-valued vector
potential of the form $\vec{A}(\vec{r}):=\textrm{i}(mc\omega/q)\vec{r}$ with
$q$ being the charge of the particle and $c$ the speed of light. However, such
a vector potential is not linked to any kind of gauge invariance as
$\vec{A}(\vec{r})=\textrm{i}(mc\omega/2q)\vec{\nabla}r^{2}$ cannot be gauged
away by a pure phase-factor in the wave function due to the presence of the
imaginary unit.

First explicit expressions of the KGO Hamiltonian were presented by Debergh et
al [9] for an isotropic system. The KGO Hamiltonian for a more general
anisotropic oscillator system is due to Bruce and Minning [10]. For the sake
of simplicity we will consider the isotropic system characterised by the
Hamiltonian

${\cal
H}:=\frac{\vec{\pi}^{\dagger}\cdot\vec{\pi}}{2m}\otimes\left(\tau_{3}+\textrm{i}\tau_{2}\right)+mc^{2}\otimes\tau_{3}$
(1)

acting on the Hilbert space $L^{2}(\mathbb{R}^{3})\otimes\mathbb{C}^{2}$. In
the above the $\tau_{i}$’s stand for the Pauli matrices

$\tau_{1}:=\left(\begin{array}[]{cc}0&1\\\
1&0\end{array}\right)\,,\qquad\textrm{i}\tau_{2}:=\left(\begin{array}[]{cc}0&1\\\
-1&0\end{array}\right)\,,\qquad\tau_{3}:=\left(\begin{array}[]{cc}1&0\\\
0&-1\end{array}\right)\,.$ (2)

Note that these Pauli matrices do not represent a spin-degree of freedom. The
2-spinors on which above Hamiltonian acts are those originally introduced by
Feshbach and Villars [27].

The KGO Hamiltonian (1) is pseudo-Hermitian [28, 29], that is ${\cal
H}^{\dagger}=\tau_{3}\,{\cal H}\,\tau_{3}$, and reads in an explicit $2\times
2$ matrix notation

${\cal H}=\left(\begin{array}[]{cc}M&A\\\ -A&-M\end{array}\right)\,,$ (3)

where we have set $M:=H_{\rm NR}+mc^{2}$ and $A:=H_{\rm NR}$, both being in
essence represented by the non-relativistic harmonic oscillator Hamiltonian in
three dimensions

$H_{\rm
NR}:=\frac{\vec{\pi}^{\dagger}\cdot\vec{\pi}}{2m}=\frac{1}{2m}\vec{p}\,^{2}+\frac{m}{2}\omega^{2}\vec{r}\,^{2}-\frac{3}{2}\hbar\omega\,.$
(4)

Here and in the following we will use calligraphic symbols for operators
acting on the full Hilbert space $L^{2}(\mathbb{R}^{3})\otimes\mathbb{C}^{2}$
and operators represented in italic act on the subspace
$L^{2}(\mathbb{R}^{3})$.

Obviously, the diagonal and off-diagonal elements in (3) commute, i.e.,
$[M,A]=0$. Hence, following the general approach of [26] it is possible to
establish an $N=2$ SUSY structure as follows.

$\begin{array}[]{l}{\cal H}_{\rm
SUSY}:=\displaystyle\frac{1}{2mc^{2}}\left({\cal M}^{2}-{\cal
H}^{2}\right)=\frac{1}{2mc^{2}}H^{2}_{\rm NR}\otimes 1\,,\\\\[11.38109pt]
{\cal Q}:=\displaystyle\frac{1}{\sqrt{2mc^{2}}}\left(\begin{array}[]{cc}0&A\\\
0&0\end{array}\right)\,,\qquad{\cal W}:=\left(\begin{array}[]{cc}1&0\\\
0&-1\end{array}\right)\equiv\tau_{3}\,,\end{array}$ (5)

where we have set ${\cal M}:=M\otimes 1$. The above SUSY operators obey the
SUSY algebra

$\begin{array}[]{c}{\cal H}_{\rm SUSY}=\\{{\cal Q},{\cal
Q}^{\dagger}\\}\,,\quad{\cal Q}^{2}=0={{\cal
Q}^{\dagger}}^{2}\,,\\\\[5.69054pt] [{\cal W},{\cal H}_{\rm
SUSY}]=0\,,\quad\\{{\cal Q},{\cal W}\\}=0=\\{{\cal Q}^{\dagger},{\cal
W}\\}\,.\end{array}$ (6)

Let us note that in the current context the third Pauli matrix plays the role
of the Witten party operator ${\cal W}$. Therefore, the upper and lower
components of a general 2-spinor belong to the subspace with positive and
negative Witten parity, respectively. We further remark that $\dim\ker{\cal
Q}=\dim\ker{\cal Q}^{\dagger}=\dim\ker H_{\rm NR}=1$. That is, SUSY is
unbroken as ${\cal H}_{\rm SUSY}$ has zero-energy eigenstates [25], but the
Witten index $\Delta$ still vanishes as

$\Delta:={\rm ind\,}{\cal Q}=\dim\ker{\cal Q}-\dim\ker{\cal Q}^{\dagger}=0\,.$
(7)

To the best of our knowledge this is the first quantum mechanical system with
an unbroken $N=2$ SUSY but vanishing Witten index, implying that the spectrum
of ${\cal H}$ is fully symmetric with respect to the origin as we will see in
the following section.

##  III Spectral properties of the KGO

As has been shown recently [26], SUSY in a relativistic Hamiltonian implies
the existence of a Foldy-Wouthuysen transformation which brings that
Hamiltonian into a block-diagonal form. In the case of the KGO this
transformation operator ${\cal U}$, which is a pseudo-unitary operator in the
sense that ${\cal U}^{-1}=\tau_{3}\,{\cal U}^{\dagger}\tau_{3}$, reads

${\cal U}:=\frac{|{\cal H}|+\tau_{3}{\cal H}}{\sqrt{2({\cal H}^{2}+{\cal
M}|{\cal H}|)}}$ (8)

leading to the block-diagonal Foldy-Wouthuysen Hamiltonian

${\cal H}_{\rm FW}:={\cal U}\,{\cal H}\,{\cal U}^{-1}=H_{\rm
FW}\otimes\tau_{3}\,,\qquad H_{\rm FW}:=\sqrt{2mc^{2}H_{\rm
NR}+m^{2}c^{4}}\,.$ (9)

To be a bit more explicit, let us define $\tanh\Theta:=A/M=H_{\rm NR}/(H_{\rm
NR}+mc^{2})$, which then allows us to write the transformation (8) in the
matrix form [9]

${\cal
U}=\left(\begin{array}[]{cc}\cosh\frac{\Theta}{2}&\sinh\frac{\Theta}{2}\\\\[5.69054pt]
\sinh\frac{\Theta}{2}&\cosh\frac{\Theta}{2}\end{array}\right)=\frac{1}{\sqrt{2}}\left(\begin{array}[]{cc}\sqrt{\frac{M}{H_{\rm
FW}}+1}&\sqrt{\frac{M}{H_{\rm FW}}-1}\\\\[5.69054pt] \sqrt{\frac{M}{H_{\rm
FW}}-1}&\sqrt{\frac{M}{H_{\rm FW}}+1}\end{array}\right)\,.$ (10)

Above expressions are functions of operators all of which may be expressed in
terms of $H_{\rm NR}$. Hence, using the spectral theorem these are well-
defined. In fact, with the spectral properties of the non-relativistic
harmonic-oscillator Hamiltonian (4) one can directly obtain those of (1). Let
$\psi_{n\ell\mu}$ denote the well-known eigenfunctions of $H_{\rm NR}$
corresponding to the eigenvalue $\varepsilon_{n\ell}$, then we have

$\begin{array}[]{l}H_{\rm
NR}\,\psi_{n\ell\mu}=\varepsilon_{n\ell}\,\psi_{n\ell\mu}\,,\qquad\varepsilon_{n\ell}=\hbar\omega(2n+\ell)\,,\qquad
n,\ell\in\mathbb{N}_{0}\,,\\\\[5.69054pt]
\displaystyle\psi_{n\ell\mu}(\vec{r})=\left(\frac{m\omega}{\hbar}\right)^{\ell/2+3/4}\sqrt{\frac{2\,n!}{\Gamma(n+\ell+3/2)}}\,r^{\ell}\,\textrm{e}^{-m\omega
r^{2}/\hbar}\,{\rm
L}^{\ell+1/2}_{n}\left(\frac{m\omega}{\hbar}\,r^{2}\right){\rm
Y}_{\ell\mu}(\vec{e})\,,\\\\[11.38109pt]
\mu\in\\{-\ell,-\ell+1,\ldots,\ell-1,\ell\\}\,,\qquad
r:=|\vec{r}|\,,\qquad\vec{e}:=\vec{r}/r\,,\end{array}$ (11)

where ${\rm L}^{\ell+1/2}_{n}$ and ${\rm Y}_{\ell\mu}$ denote the associated
Laguerre polynomials and the spherical harmonics, respectively. See, for
example, ref. [30]. Then, the eigenvalues and eigenfunctions of (1) are
explicitly given by

$\begin{array}[]{l}\displaystyle{\cal
H}\Psi^{\pm}_{n\ell\mu}=E^{\pm}_{n\ell}\Psi^{\pm}_{n\ell\mu}\,,\qquad
E_{n\ell}^{\pm}=\pm
mc^{2}\sqrt{1+\frac{2\varepsilon_{n\ell}}{mc^{2}}}\,,\\\\[11.38109pt]
\Psi^{+}_{n\ell\mu}(\vec{r})=\psi_{n\ell\mu}(\vec{r})\left(\begin{array}[]{r}\cosh\frac{\vartheta_{n\ell}}{2}\\\\[2.84526pt]
-\sinh\frac{\vartheta_{n\ell}}{2}\end{array}\right)\,,\qquad\Psi^{-}_{n\ell\mu}(\vec{r})=\psi_{n\ell\mu}(\vec{r})\left(\begin{array}[]{r}-\sinh\frac{\vartheta_{n\ell}}{2}\\\\[2.84526pt]
\cosh\frac{\vartheta_{n\ell}}{2}\end{array}\right)\,,\end{array}$ (12)

where
$\tanh\vartheta_{n\ell}:=\varepsilon_{n\ell}/(\varepsilon_{n\ell}+mc^{2})$.
These states form an orthonormal basis in
$L^{2}(\mathbb{R}^{2})\otimes\mathbb{C}^{2}$ with respect to the scalar
product [27].

$\langle\Psi_{1}|\Psi_{2}\rangle:=\int_{\mathbb{R}^{3}}\textrm{d}^{3}\vec{r}\,\,\overline{\Psi}_{1}(\vec{r})\,\tau_{3}\,\Psi_{2}(\vec{r})\,,$
(13)

where the overbar stands for the transposed and complex conjugated $2$-spinor.
That is,

$\langle\Psi^{\pm}_{n\ell\mu}|\Psi^{\pm}_{n^{\prime}\ell^{\prime}\mu^{\prime}}\rangle=\pm\delta_{nn^{\prime}}\delta_{\ell\ell^{\prime}}\delta_{\mu\mu^{\prime}}\,,\qquad\langle\Psi^{\pm}_{n\ell\mu}|\Psi^{\mp}_{n^{\prime}\ell^{\prime}\mu^{\prime}}\rangle=0\,.$
(14)

Obviously, the scalar product (13), which was already introduced by Feshbach
and Villars [27], is not positive definite and might raise some questions on
its probabilistic interpretation. However, Mostafazadeh’s theory of pseudo-
Hermitian operators does provide a solution to this obstacle. The Klein-Gordon
case was explicitly discussed in [28, 29].

Let us conclude this section by noting that the SUSY ground states associated
with the non-relativistic eigenvalue $\varepsilon_{00}=0$ are given by

$\Psi^{+}_{000}(\vec{r})=\left(\frac{m\omega}{\hbar\pi}\right)^{3/4}\textrm{e}^{-m\omega
r^{2}/\hbar}\left(\begin{array}[]{c}1\\\
0\end{array}\right)\,,\qquad\Psi^{-}_{000}(\vec{r})=\left(\frac{m\omega}{\hbar\pi}\right)^{3/4}\textrm{e}^{-m\omega
r^{2}/\hbar}\left(\begin{array}[]{c}0\\\ 1\end{array}\right)$ (15)

with corresponding eigenvalues $E_{00}^{\pm}=\pm mc^{2}$. We also remark that
the Foldy-Wouthuysen Hamiltonian (9) can be written as

${\cal H}_{\rm FW}=mc^{2}\sqrt{1+\frac{2H_{\rm
NR}}{mc^{2}}}\otimes\tau_{3}\,,$ (16)

a form already observed for other relativistic Hamiltonians exhibiting a SUSY
[26].

##  IV Green’s function

The SUSY established for the KGO in the previous section also allows us to
study Green’s function associated with the KGO Hamiltonian (1). Following the
general approach of [26] Green’s function being defined by

${\cal G}(z):=\frac{1}{{\cal H}-z}\,,\qquad z\in\mathbb{C}\backslash{\rm
spec\,}{\cal H}\,,$ (17)

can be expressed in terms of the iterated Green’s function ${\cal G}_{\rm I}$,
that is,

${\cal G}(z)=({\cal H}+z){\cal G}_{\rm I}(z^{2})\,,\qquad{\cal G}_{\rm
I}(z^{2}):=\frac{1}{{\cal H}^{2}-z^{2}}\,.$ (18)

Noting that ${\cal H}^{2}=2mc^{2}(H_{\rm NR}+mc^{2}/2)\otimes 1$, the iterated
Green’s function can be written in terms of the non-relativistic Green’s
function $G_{\rm NR}(\varepsilon):=(H_{\rm NR}-\varepsilon)^{-1}$ associated
with $H_{\rm NR}$ as follows.

${\cal G}_{\rm I}(z^{2})=\frac{1}{2mc^{2}}\,G_{\rm NR}(\varepsilon)\otimes
1\,,\qquad\varepsilon:=\frac{z^{2}}{2mc^{2}}-\frac{mc^{2}}{2}=2mc^{2}\left(\left(\frac{z}{2mc^{2}}\right)^{2}-\frac{1}{4}\right)\,.$
(19)

Inserting this into above relation (18) results in

${\cal G}(z)=\frac{1}{2mc^{2}}\left(\begin{array}[]{cc}(H_{\rm
NR}+mc^{2}+z)\,G_{\rm NR}(\varepsilon)&H_{\rm NR}\,G_{\rm NR}(\varepsilon)\\\
-H_{\rm NR}\,G_{\rm NR}(\varepsilon)&-(H_{\rm NR}+mc^{2}-z)\,G_{\rm
NR}(\varepsilon)\end{array}\right)\,.$ (20)

Using the defining relation $H_{\rm NR}\,G_{\rm
NR}(\varepsilon)=\varepsilon\,G_{NR}(\varepsilon)$ together with the second
relation in (19) leads us to the closed-form expression

${\cal G}(z)=G_{\rm
NR}(\varepsilon)\left(\begin{array}[]{cc}\left(\frac{1}{2}+\frac{z}{2mc^{2}}\right)\left(\frac{1}{2}+\frac{z}{2mc^{2}}\right)&\left(\frac{1}{2}+\frac{z}{2mc^{2}}\right)\left(\frac{z}{2mc^{2}}-\frac{1}{2}\right)\\\\[5.69054pt]
\left(\frac{1}{2}+\frac{z}{2mc^{2}}\right)\left(\frac{1}{2}-\frac{z}{2mc^{2}}\right)&\left(\frac{1}{2}-\frac{z}{2mc^{2}}\right)\left(\frac{z}{2mc^{2}}-\frac{1}{2}\right)\end{array}\right)\,,$
(21)

with $\varepsilon$ as defined in (19). The reader is invited to verify that
${\cal H}\,{\cal G}(z)=z\,{\cal G}(z)$. With the definition
$\tanh\vartheta:=\varepsilon/(\varepsilon+mc^{2})$ the above result may be put
into the form

${\cal G}(z)=\frac{G_{\rm
NR}(\varepsilon)}{\cosh\frac{\vartheta}{2}-\sinh\frac{\vartheta}{2}}\left(\begin{array}[]{cc}\cosh^{2}\frac{\vartheta}{2}&\cosh\frac{\vartheta}{2}\sinh\frac{\vartheta}{2}\\\\[5.69054pt]
-\cosh\frac{\vartheta}{2}\sinh\frac{\vartheta}{2}&-\sinh^{2}\frac{\vartheta}{2}\end{array}\right)\,.$
(22)

To complete the discussion let us finally note that the coordinate
representation of ${G}_{\rm
NR}(\vec{r},\vec{r}\,^{\prime},\varepsilon):=\langle\vec{r}|{G}_{\rm
NR}(\varepsilon)|\vec{r}\,^{\prime}\rangle$ is known for long, see for example
[31], and explicitly reads

$\begin{array}[]{l}\displaystyle{G}_{\rm
NR}(\vec{r},\vec{r}\,^{\prime},\varepsilon)=\frac{1}{rr^{\prime}}\sum_{\ell=0}^{\infty}{G}_{\ell}(r,r^{\prime},\varepsilon)\sum_{\mu=-\ell}^{\ell}{\rm
Y}^{*}_{\ell\mu}(\vec{e}\,^{\prime}){\rm
Y}_{\ell\mu}(\vec{e})\,,\\\\[5.69054pt]
\displaystyle{G}_{\ell}(r,r^{\prime},\varepsilon)=-\frac{\Gamma\left(\frac{\ell}{2}-\frac{\varepsilon}{2\hbar\omega}\right)}{\sqrt{rr^{\prime}}\,\hbar\omega}{\rm
W}_{\lambda,\nu}\left(r^{2}_{>}m\omega/\hbar\right){\rm
M}_{\lambda,\nu}\left(r^{2}_{<}m\omega/\hbar\right)\,,\end{array}$ (23)

where ${\rm W}_{\lambda,\nu}$ and ${\rm M}_{\lambda,\nu}$ denote Whittaker’s
functions, and we have set
$\lambda:=\frac{\varepsilon}{2\hbar\omega}+\frac{3}{4}$,
$\nu:=\frac{\ell}{2}+\frac{1}{4}$, $r_{>}:=\max\\{r,r^{\prime}\\}$ and
$r_{>}:=\min\\{r,r^{\prime}\\}$.

##  V Summary and Outlook

In this work we have shown that the KGO exhibits a SUSY structure following
closely the general approach of ref. [26]. The SUSY of the KGO is found to be
unbroken but, remarkably, with a vanishing Witten index. Despite the fact that
the eigenvalues in (12) are known for a long time, see for example [9, 10],
the associated eigenstates in (12) have to our knowledge never been presented.
Note that in [10] only the eigenstates of ${\cal H}_{\rm FW}$ are given. In
addition, SUSY has enabled us to calculate the KGO Green function in a closed
form.

Obviously, the current discussion for an isotropic oscillator may be extended
to that for the anisotropic oscillator following Bruce and Manning [10]. Here,
in essence, one needs to reduce the problem to three one-dimensional harmonic
oscillators. An explicit expression for the Green’s function may also be
obtained as the one-dimensional harmonic oscillator Green’s function is known
in closed form, too. See, for example, Glasser and Nieto [32] and the
discussion of the associated Dirac problem [33]. One may also pursue the path
integral approach for the KGO along the lines of the corresponding approach
for the Dirac oscillator [24]. Another route for further investigation would
be to look at generalised nonharmonic oscillators characterized by a potential
function $U_{a}(r):=\lambda_{a}r^{a}$ using the minimal substitution
$\vec{\pi}:=\vec{p}-\textrm{i}(\vec{\nabla}U_{a})(r)$. Such power-law
potentials are known to obey a duality symmetry in classical and non-
relativistic quantum mechanics, see the recent work [34] and references
therein. In particular, the harmonic potential $a=2$, which corresponds to the
discussed KGO case, is dual to the Kepler potential where $a=-1$.

Another extension, which might be entertained, is to apply the current SUSY
construction to the relativistic $S=1$ oscillator. However, we must note that,
as has been argued by Debergh et al [9], the diagonal and off-diagonal matrix
elements of the associated Hamiltonian no longer commute. Hence, it may not be
possible to establish a SUSY structure.

## References

  * [1] Palmieri, P. A phenomenology of Galileo’s experiments with pendulums. Brit. J. Hist. Sci. 2009, 42, 479-513.
  * [2] Hook, R. De Potentia Restitutiva, or of Spring. Explaining the Power of Springing Bodies; The Royal Society: London, 1678.
  * [3] Planck, M. Zur Theorie des Gesetzes der Energieverteilung im Normalspectrum. Verhandl. Dtsch. phys. Ges. 1900, 2, 237-245.
  * [4] Itô, D.; Mori, K.; Carriere, E. An example of dynamical systems with linear trajectory. Nuovo Cim. A 1967, 51, 1119-1121.
  * [5] Swamy, N.V.V.J. Exact Solution of the Dirac Equation with an Equivalent Oscillator Potential. Phys. Rev. 1969, 180, 1225-1226.
  * [6] Cook, P.A. Relativistic harmonic oscillators with intrinsic spin structure. Lett. Nuovo Cim. 1971, 1, 419-426.
  * [7] Moshinsky, M.; Szczepaniak, A. The Dirac oscillator. J. Phys. A 1989, 22, L817-L819.
  * [8] Quesne C.; Moshinsky, M. Symmetry Lie algebra of the Dirac oscillator. J. Phys. A 1990, 23, 2263-2272.
  * [9] Debergh, N.; Ndimubandi J.; Strivay, D. On relativistic scalar and vector mesons with harmonic oscillatorlike interactions. Z. Phys. C 1992, 56, 421-425.
  * [10] Bruce S.; Minning, P. The Klein-Gordon Oscillator. Nuovo Cim. A 1993, 106, 711-713.
  * [11] Dvoeglazov, V.V.; del Sol Mesa, A. Notes on oscillator-like interactions of various spin relativistic particles. NASA Conference Pub.  1994, 3286, 333-340; arXiv:hep-th/9403165.
  * [12] Rao N.A.; Kagali, B.A. Energy profile of the one-dimensional Klein–Gordon oscillator. Phys. Scr. 2008, 77, 015003 (4pp).
  * [13] Boumali, A.; Hafdallah A.; Toumi, A. Comment on ’Energy profile of the one-dimensional Klein–Gordon oscillator’. Phys. Scr. 2011, 84, 037001 (2pp).
  * [14] Mirza, B.; Mohadesi, M.; Zare, S. The Klein-Gordon and the Dirac Oscillators in a Noncommutative Space. Commun. Theor. Phys. 2004, 42, 664-668.
  * [15] Mirza, B.; Mohadesi, M. Relativistic Oscillators in a Noncommutative Space and in a Magnetic Field. Commun. Theor. Phys. 2011, 55, 405-409.
  * [16] Santos, L.C.N.; Mota, C.E.; Barros Jr., C.C. Klein–Gordon Oscillator in a Topologically Nontrivial Space-Time. Adv. High Energy Phys.  2019, Article ID 2729352, (7pp).
  * [17] Ahmed, F. The generalized Klein–Gordon oscillator in the background of cosmic string space-time with a linear potential in the Kaluza–Klein theory. Eur. Phys. J. C 2020, 80, 211 (12pp).
  * [18] Bragança, E.A.F.; Vitória, R.L.L.; Belich, H.; Bezerra de Mello, E.R. Relativistic quantum oscillators in the global monopole spacetime. Eur. Phys. J.  2020, 80, 206 (11pp).
  * [19] Zhong, L.; Chen, H.; Hassanabadi, H.; Long, Z.-W.; Long, C.-Y. The study of the generalized Klein-Gordon oscillator in the context of the Som-Raychaudhuri space-time. 2021; arXiv:2101.06356.
  * [20] Benítez, J.; Martínez y Romero, R.P.; Núñez-Yépez H.N.; Salas-Brito, A.L. Solution and Hidden Super- symmetry of a Dirac Oscillator. Phys. Rev. Lett. 1990, 64, 1643-1645; ERRATA Phys. Rev. Lett. 1990, 65, 2085.
  * [21] Beckers J.; Debergh, N. Supersymmetry, Foldy-Wouthuysen trausformations, and relativistic oscillators. Phys. Rev. D 1990, 42, 1255-1259.
  * [22] Martínez y Romero, R.P.; Morena, M.; Zentella, A. Supersymmetric properties and stability of the Dirac sea. Phys. Rev. D 1991, 43, 2036-2040.
  * [23] Quesne, C. Supersymmetry and the Dirac oscillator. Int. J. Mod. Phys.  1991, 6, 1567-1589.
  * [24] Junker, G.; Inomata, A. Path Integral and Spectral Representations for Supersymmetric Dirac-Hamiltonians. J. Math. Phys.  2018, 59, 052301 (16pp).
  * [25] Junker, G. Supersymmetric Methods in Quantum, Statistical and Solid State Physics. Enlarged and revised edition; IOP Publishing: Bristol, 2019.
  * [26] Junker, G. Supersymmetry of Relativistic Hamiltonians for Arbitrary Spin. Symmetry 2021, 12, 1590 (14pp).
  * [27] Feshbach, H.; Villars, F. Elementary Relativistic Wave Mechanics of Spin 0 and Spin 1/2 Particles. Rev. Mod. Phys. 1958, 30, 24-45.
  * [28] Mostafazadeh, A. Is Pseudo-Hermitian Quantum Mechanics an Indefinite-Metric Quantum Theory?. Czech. J. Phys.  2003, 53, 1079-1084.
  * [29] Mostafazadeh, A. A Physical Realization of the Generalized PT-, C-, and CPT-Symmetries and the Position Operator for Klein-Gordon Fields. Int. J. Mod. Phys. A 2006, 53, 2553-2572.
  * [30] Galindo, A.; Pascual, P. Quantum Mechanics I; Springer-Verlag: Berlin, 1990.
  * [31] Blinder, S.M. Propagators from integral representations of Green’s functions for the $N$-dimensional free particle, harmonic oscillator and Coulomb problems. J. Math. Phys. 1984, 25, 905-909.
  * [32] Glasser, M.L.; Nieto, L.M.; The energy level structure of a variety of one-dimensional confining potentials and the effects of a local singular perturbation. Can. J. Phys.  2015, 93, 1588-1596.
  * [33] Junker, G. Supersymmetric Dirac Hamiltonians in $(1+1)$ dimensions revisited. Eur. Phys. J. Plus 2020, 135, 464 (13pp).
  * [34] Inomata, A., Junker, G. Power Law Duality in Classical and Quantum Mechanics. Symmetry 2021, 13, 409 (50pp).


ACP

    Attention-based Costmap Prediction
AIRL

    Approximate Inverse Reinforcement Learning
BNN

    Bayesian Neural Network
CNN

    Convolutional Neural Network
DAgger

    Data Aggregation
DDP

    Differential Dynamic Programming
DL

    Deep Learning
E2E

    end-to-end
IL

    Imitation Learning
iLQG/MPC-DDP

    iterative Linear Quadratic Gaussian/Model Predictive Control Differential Dynamic Programming
IRL

    Inverse Reinforcement Learning
LSTM

    Long Short Term Memory
Conv-LSTM

    Convolutional LSTM
MDP

    Markov Decision Process
MPC

    Model Predictive Control
MPPI

    Model Predictive Path Integral
NN

    Neural Network
RL

    Reinforcement Learning

#  Approximate Inverse Reinforcement Learning from Vision-based Imitation
Learning

Keuntaek Lee, Bogdan Vlahov, Jason Gibson, James M. Rehg, and Evangelos A.
Theodorou  Georgia Institute of Technology
Email<EMAIL_ADDRESS>
###### Abstract

In this work, we present a method for obtaining an implicit objective function
for vision-based navigation. The proposed methodology relies on Imitation
Learning, Model Predictive Control (MPC), and Deep Learning. We use Imitation
Learning as a means to do Inverse Reinforcement Learning in order to create an
approximate costmap generator for a visual navigation challenge. The resulting
costmap is used in conjunction with a Model Predictive Controller for real-
time control and outperforms other state-of-the-art costmap generators
combined with MPC in novel environments. The proposed process allows for
simple training and robustness to out-of-sample data. We apply our method to
the task of vision-based autonomous driving in multiple real and simulated
environments using the same weights for the costmap predictor in all
environments.
Supplementary video: https://youtu.be/WyJfT5lc0aQ

##  I Introduction

In robotics, vision-based control has become a popular topic as it allows
navigating in a variety of environments. A notable contribution is the ability
to work in areas where positional information such as GPS or VICON data is not
possible to obtain. While vision-based controls are harder to analytically
write equations for when compared to positional-based controls, it has been
shown to work by millions of humans using it every day.

Using neural networks (NNs) for vision-based control has become ubiquitous in
literature. Drews [1] uses an architecture that separates the vision-based
control problem into a costmap generation task and then uses an MPC controller
for generating the control. This method allows for the less principled area of
feature extraction and interpretation for autonomous driving to be done by the
NN, and solve the stochastic optimal control problem in a principled way. This
architecture provides better observability into the learning process as
compared to traditional end-to-end (E2E) control approaches [2]. Additionally,
this decouples the state estimation and controller, allowing us to leverage
standard state estimation techniques with a vision-based controller.

In general, most of the NN models suffer from the generalization problem; a
trained NN model does not work well on a new test dataset if the training and
testing dataset are very different from each other. To solve this
generalization problem in new environments, in this work, we focus on
generalizing vision-based control systems to new previously unseen
environments. This will focus on the ability of a single network to generate
reasonable costmaps even in a novel environment not seen during training. We
propose an automatic way to generate the costmap without requiring access to a
pre-defined costmap or any labels with which to perform segmentation,
classification, or recognition. The key idea is using a vision-based E2E
Imitation Learning (IL) framework [2]. In this E2E control approach, we only
need to query the expert’s action to learn a costmap of a specific task.
During training, the model implicitly learns a mapping from sensor input to
the control output. Specifically in vision-based autonomous driving, if we
train a deep NN by imitation learning and analyze an intermediate layer by
reading the weights of the trained network and the activated neurons of it, we
see the mapping converged to extracting important features that link the input
and the output (Fig. 2).

In a broad sense, the convolutional layer parts of the trained E2E network
become a function that extracts important features in the input scene. This
can be viewed as an implicit image segmentation done inside the deep
convolutional neural network where the extracted features will depend on the
task at hand. For example, if the task is learning to visually track an
object, the network will implicitly find the object as an important feature.
In another case, if the task is to perform autonomous lane-keeping, the
boundaries of the lane will become important.

Our work is obtaining a costmap based on an intermediate convolutional layer
activation, but the middle layer output is not directly trained to predict a
costmap; instead, it is generating an implicit objective function related to
relevant features. This allows our work to produce reasonable costmap on
unseen data where direct costmap prediction methods [1] would fail because the
data would be out of their prediction domain. As an analogy, our method is
similar to learning the addition operator $a+b=c$ whereas a prediction method
would be similar to a mapping between numbers $(a,b)\rightarrow c$.

###  I-A Related Work

In classic path planning of robotic systems, sensor readings and optimization
are all done in a world coordinate frame. Specifically, monocular vision-based
planning has shown a lot of success performing visual servoing in ground
vehicles [3, 4, 5, 6, 1], manipulators [7], and aerial vehicles [8].

Drews et al. [6] learns to generate a costmap from camera images with a
bottleneck network structure using a Long Short Term Memory (LSTM) layer. This
costmap is then given to a particle filter which uses it as a sensor
measurement to improve state estimation and is also used as a costmap for the
Model Predictive Control (MPC) controller. Drews [1] tries to generalize this
approach by using a Convolutional LSTM (Conv-LSTM) [9] and a softmax attention
mechanism and shows this method working on previously unseen tracks. However,
the training of this architecture requires having a predetermined costmap to
imitate and the track it was shown to generalize to had visually similar
components (dirt track and black track borders) to recognize.

Loquercio et al. [10] constructs a system for vision-based agile drone flight
that generalizes to new environments. They separate their system into a
perception and control pipeline. The perception pipeline was a Convolutional
Neural Network (CNN), taking in raw images and producing a desired direction
and velocity, trained in simulation on a large mixture of random backgrounds
and gates. By providing many random textures, the perception pipeline is
trained to be more generalized than training on just real data. They show this
system can perform similarly or better than a system trained on real-world
data alone from real drones. However, their method is still best applied to
drones where it is relatively easy to match a desired direction and velocity.

E2E learning has been shown to work in various lane-keeping applications [11,
12, 13]. However, these methods only control the steering angle and assume
constant velocity. While this is a valid assumption in most cases of driving,
there are times where the coupling of steering and throttle is necessary to
continue driving on the road.

Ollis et al. [14] proposed an image-space approach: planning a path for a
ground-based robot in the image-space of an onboard monocular camera. This
technique is most related to our approach since they applied a learned color-
to-cost mapping to transform a raw image into a costmap-like image, and
performed path planning directly in the image space.

###  I-B Contribution and Organization

The contributions of this work are threefold:

  * •

We introduce a novel inverse reinforcement learning method which approximates
a cost function from an intermediate layer of an end-to-end policy trained
with imitation learning.

  * •

We perform a sampling-based stochastic optimal control in image space, which
is perfectly suitable for our driver-view binary costmap.

  * •

Compared to state-of-the-art methods, our proposed method is shown to
generalize by generating usable costmaps in environments outside of its
training data.

The remaining of the paper is organized as follows: In Section II, we briefly
review some preliminaries used in our work with some literature reviews.
Section III introduces the Model Predictive Path Integral (MPPI) control
algorithm in image space and in Section IV-C, we introduce our Approximate
Inverse Reinforcement Learning algorithm. Section V details vision-based
autonomous driving experiments with analysis and comparisons of the proposed
methods. Finally, we conclude and discuss future directions in Section VI and
Section VII.

##  II Preliminaries

###  II-A Inverse Reinforcement Learning

Markov Decision Processes are used as a framework for modeling both
Reinforcement Learning (RL) and Inverse Reinforcement Learning (IRL) problems
[15]. An MDP is generally formulated as the following tuple $(X,U,T,R,\gamma)$
where $X$ is the space of possible states, $U$ is the space of possible
actions, $T(x^{\prime}|x,u)$ is the transition function from state $x$ to
$x^{\prime}$ using action $u$, $R(x,u,x^{\prime})$ is the reward for going to
state $x^{\prime}$ from $x$ using action $u$, and $\gamma$ is the reward
discount factor. In RL, the reward function $R$ is unknown to the learning
agent; it receives observations at time $t$ of the reward, $r_{t}$, by moving
through $X$ and $U$. The goal of RL is to learn a policy $\pi:X\rightarrow U$
that achieves the maximum expected reward $\sum_{t=1}\gamma^{t}r_{t}$.

In IRL, there is an unknown expert policy, $\pi_{e}$, from which we receive
observations in the form of $(x_{t},u_{t})$ at time $t$, acting according to
some optimal reward $R^{*}$. IRL is then learning a reward function $\hat{R}$
that describes the expert policy $\pi_{e}$ [16].

While this formulation can be easy to write, IRL can be considered a harder
problem to solve than RL. There is generally not a single reward function that
can describe an expert behavior [16]. Assumptions need to be made on the
formulation of the reward function and properly evaluating the learned
$\hat{R}$ involves creating a new policy, $\hat{\pi_{e}}$ and comparing it to
the observed $\pi_{e}$. Creating this $\hat{\pi_{e}}$ then requires either
solving or approximating a solution to a new MDP $(X,U,T,\hat{R},\gamma)$.

Despite these difficulties, IRL can be an extremely useful tool. If we can
approximate a reward function from observations, we can then train new agents
to maximize this reward. It can be considered similar to IL in that sense, as
we could train agents to perform according to an expert behavior. However, it
is important to note that, unlike in IL, the learning agents could then
potentially outperform the expert behavior. This is especially true when the
expert behavior is suboptimal or applied in a different environment. Russell
[17] and Arora and Doshi [16] also describes how a learned reward function is
more transferable than an expert policy because as a policy can be easily
affected by different transition functions $T$ whereas the reward function is
can be considered a description of the ideal policy.

Most of the IRL work in the literature requires one more pipeline of training
to figure out the mapping between the input trajectories and the reward
function. For example, M. Babes and Subramanian. [18] introduced maximum
likelihood approach to IRL while the maximum entropy approach was introduced
in [19] to find a generative model that yields trajectories that are similar
to the expert’s. They find a weighted distribution of reward basis functions
in an iterative way. This step still requires some hand-tuning; for example,
picking proper basis functions to form the distribution.

However, in our work, we do not need any new design of basis functions because
we learn a policy end-to-end and can get an approximate $\hat{R}$ for ‘free’,
i.e. without any hand-tuning. Since optimal controllers can be considered as a
form of model-based RL, this $\hat{R}$ can then be used as the cost function
that our MPC controller optimizes with respect to.

###  II-B Imitation Learning

RL is one way to train agents to maximize some notion of task-specific
rewards. One of the major problems in RL is the sample-inefficiency problem:
the agents have to explore the action-state space without any prior knowledge
of the environment or task. However, IL uses supervised learning to train a
control policy and bypass this sample-inefficiency problem. In IL, a policy is
trained to accomplish a specific task by mimicking an expert’s control policy,
which in most cases, is assumed to be optimal. Accordingly, IL provides a
safer training process. In this work, we will use sections of a network
trained with End-to-End Imitation Learning (E2EIL) using MPC as the expert
policy. Literally, E2EIL trains agents to directly output optimal control
actions given image data from cameras; End(sensor reading) to End(control).

While IL provides benefits in terms of sample efficiency, it does have
drawbacks. Here, we shortly talk about three major problems in IL.

#### Generalizability

The training data collected from an optimal expert does not usually include
demonstrations of failure cases in unsafe situations. Ross et al. [20]
introduced an online Data Aggregation (DAgger) method, which mixes the
expert’s policy and the learner’s policy to explore various situations like
$\epsilon$-greedy. However, even with the online scheme of collecting
datasets, it is impossible to experience all kinds of unexpected scenarios.
Accordingly, E2EIL is vulnerable to out-of-training-data. There are little to
no guarantees on what a Neural Network (NN) trained with IL will output when
given an input vastly different from its training set. [21, 22, 23]
demonstrated failure cases of deep end-to-end controllers; the controllers
failed to predict a correct label from a novel (out-of-training-data) input
and there was no way to tell the output prediction is trustworthy without
considering the Bayesian technique.

#### Upper-bounded

The best job a learner can do is capped by the ability of a teacher since the
objective of the IL setting is to mimic the expert’s behavior.

#### Interpretability

Since the end-to-end approach uses a totally blackbox model from sensor input
to control output, it loses interpretability; when it fails, it is hard to
tell if it comes from noise in the input, if the input is different from the
training data, or if the model has just chosen a wrong control output due to
ending training prematurely.

From these reasons, E2E IL controllers are not widely used in the real-world
applications, such as self-driving cars. Our approach provides solutions to
these problems by leveraging the idea of using Deep Learning (DL) only in some
blocks of autonomy, hence becomes more interpretable.

In the case of autonomous driving, given a cost function to optimize and a
vehicle dynamics model, we can compute an optimal solution via an optimal
model predictive controller. Therefore, the problem simplifies from computing
a good action to computing a good approximation of the cost function. In this
paper, we provide evidence of better performance than the expert teacher by
showing a higher success rate of task completion when a task requires
generalization to new environments.

###  II-C Model Predictive Path Integral Control

MPC-based optimal controllers provide planned control trajectories given an
initial state and a cost function by solving the optimal control problem. In
general, a discrete-time optimal control problem whose objective is to
minimize a task-specific cost function $J(\mathbf{x},\mathbf{u})$ can be
formulated as follows:

$\displaystyle J(\mathbf{x}_{t},\mathbf{u}_{t})$
$\displaystyle=\phi(\mathbf{x}_{\mathrm{T}})+\sum_{t=t_{0}}^{\mathrm{T}-1}l(\mathbf{x}_{t},\mathbf{u}_{t}).$
(1) $\displaystyle V(\mathbf{x}_{t_{0}})$
$\displaystyle=\min_{\mathbf{u}_{t}}\Big{[}J(\mathbf{x}_{t},\mathbf{u}_{t})\Big{]},$
(2)

subject to discrete time, continuous state-action dynamical system

$\displaystyle\mathbf{x}_{t+1}=F(\mathbf{x}_{t},\mathbf{u}_{t}),$ (3)

where $\mathbf{x}_{t}\in\mathbb{R}^{n}$ represents the system states at time
$t$, $\mathbf{u}_{t}\in\mathbb{R}^{m}$ represents the control at time $t$,
$\phi$ is the state cost at the final time $\mathrm{T}$, $l$ is the running
cost, and $V$ is the state-value function. $J$ in this optimal control
settings corresponds to the negative reward ($-R$) in RL and $F$ corresponds
to the state transition function $T$ in RL. $F$ is assumed to be time-
invariant and a finite time-horizon $t\in[0,1,2,...\mathrm{T}-1]$ has the unit
of time determined by the control frequency of the system. The optimal control
is solved in a receding horizon fashion within an MPC framework to give us a
real-time optimal control sequence $u_{0,1,2,...\mathrm{T}-1}$.

We choose to use a sampling-based stochastic optimal controller MPPI [24]
because it can operate on non-linear learned dynamics and can have non-convex
cost functions. This enables us to run the controller directly on the output
costmap of the network. Furthermore, it enables using arbitrary learned
dynamics for a given system. Therefore, for our approach to generalize to a
completely different dynamical environment, we simply need to change the
dynamics used by MPPI and the approach continues to work.

MPPI outputs the optimal control sequence given a cost function. Sequences of
control vectors are sampled around a nominal trajectory and are propagated
forward $\Delta t$ in time using the dynamics model to generate state-action
pairs that are input into the cost function. Then a cost weighted average is
computed over the sampled controls. This result is shifted down a time step
and used as the nominal trajectory to sample controls from in the next
optimization round. Each trajectory has a time horizon of $\mathrm{T}$. The
optimization can be repeated $K$ times. The exploration variance $\Sigma$
represents the variance of the zero-mean Gaussian that MPPI uses when sampling
random controls.

For the navigation task, our cost function will follow a similar format as in
[1] with the squared cost on the desired speed since we are operating at low
speeds. Also $\phi(\mathbf{x}_{\mathrm{T}})=0$ in this application. The
running cost function at time $t$, which does not penalize the control, is as
shown in Eq. 4:

$\displaystyle l(\textbf{x}_{t})=C_{s}(v_{x}^{d}-v_{x,t})^{2}+\gamma^{t}\cdot
C_{c}I(\textbf{x}_{t}),$ (4)

where $C_{s},C_{c}$ are coefficients that represent the penalty applied for
speed and crash, respectively. $I$ is an indicator function that returns $1$
if the vehicle position in the image space is on top of the high-cost region
or another reason would cause the vehicle to crash, and returns $0$ otherwise.
$v_{x}$ and $v_{x}^{d}$ are measured body velocity in the $x$ direction and
desired velocity respectively. $\gamma^{t}$ is a discount. Obstacle checking
is implemented as a costmap lookup, since this gives computational efficiency.
How we convert an image into a costmap that can be queried by MPPI is covered
in Section III.

For this navigation task, we followed the same definition of the system state
and control in the MPPI paper [24] and [1]:
$\mathbf{x}=[x,y,yaw,roll,v_{x},v_{y},\dot{yaw}]$ is the vehicle state in a
world coordinate frame and $\mathbf{u}$ is $[throttle,steering]$.

##  III MPPI in image space

We used MPPI [24] as an expert in IL and also an optimal controller for
testing our costmap like in [5, 6, 1]. It is important to note that near-
perfect state estimation and a GPS track map is provided when MPPI is used as
the expert, but as in [1], only body velocity, roll, and yaw from the state
estimate is used when it is operating using vision.

Image space from a mounted camera on a robot is a local and fixed frame; i.e.
the state represented in the image space is relative to the robot’s camera.

MPPI uses a data-driven neural network model as a vehicle dynamics model. The
data-driven neural network model takes in time, control and state (roll, body
frame velocity in x,y and yaw rate) information as an input, and outputs the
next state derivatives as described in [24].

Since we are planning an optimal path given a costmap image in first-person-
view, the vehicle’s future state trajectory described in the world coordinates
must be transformed into a 2D image in a moving frame of reference. This
coordinate transformation technique is widely used in 3D computer graphics
[25]. The coordinate transformation consists of 4 steps:

$\displaystyle\text{World}\rightarrow\text{Robot}\rightarrow\text{Camera}\rightarrow\text{Film}\rightarrow\text{Pixel}.$

In this work, we follow the convention in the computer graphics community and
set the Z (optic)-axis as the vehicle’s longitudinal (roll) axis, the Y-axis
as the axis normal to the road, the positive direction being upwards, and the
X-axis as the axis perpendicular on the vehicle’s longitudinal axis, the
positive direction pointing to the right side of vehicle.

Let us define roll, pitch, yaw angles as $\phi,\theta,\psi$ and the camera
(vehicle) position $U_{\text{cam}},V_{\text{cam}},W_{\text{cam}}$ in the world
coordinates. The camera focal length is defined as $f$. Then, we construct the
rotation matrices around the U, V, W-axis $R_{U},R_{V},R_{W}$, the translation
matrix $T_{\text{tl}}$, the robot-to-camera coordinate transformation matrix
$T_{r\rightarrow c}$ and the projection matrix $T_{c\rightarrow f\rightarrow
p}$ as:

$\displaystyle R_{U}=\begin{bmatrix}1&0&0\\\ 0&cos\phi&-sin\phi\\\
0&sin\phi&cos\phi\end{bmatrix},R_{V}=\begin{bmatrix}cos\theta&0&sin\theta\\\
0&1&0\\\ -sin\theta&0&cos\theta\end{bmatrix},$ $\displaystyle
R_{W}=\begin{bmatrix}cos\psi&-sin\psi&0\\\ sin\psi&cos\psi&0\\\
0&0&1\end{bmatrix},T_{\text{tl}}=\begin{bmatrix}1&0&0&-U_{\text{cam}}\\\
0&1&0&-V_{\text{cam}}\\\ 0&0&1&-W_{\text{cam}}\end{bmatrix},$ $\displaystyle
T_{r\rightarrow c}=\begin{bmatrix}0&1&0\\\ 0&0&1\\\
1&0&0\end{bmatrix},T_{c\rightarrow f\rightarrow
p}=\begin{bmatrix}f/Z&0&o_{X}/Z\\\ 0&f/Z&o_{Y}/Z\end{bmatrix},$ (5)

where the projection matrix $T_{c\rightarrow f\rightarrow p}$ projects the
point $(X,Y,Z)$ in the camera coordinates into the film coordinates using the
perspective projection equations from [25] and the offsets $o_{X}$ and $o_{Y}$
transform the film coordinates to the pixel coordinates by shifting the
origin.

The total rotation matrix is computed as

$\displaystyle R=R_{W}R_{V}R_{U}\in\mathbb{R}^{3\times 3}$ (6)

and the matrix $T_{w\rightarrow r}$, transforming the world coordinates to the
robot coordinates by translation and rotation, is calculated as

$\displaystyle T_{w\rightarrow r}=RT_{\text{tl}}\in\mathbb{R}^{3\times 4}.$
(7)

Then, after converting the X,Y,Z-axes to follow the convention in the computer
vision community through $T_{r\rightarrow c}$, the projection matrix
$T_{c\rightarrow f\rightarrow p}$ converts the camera coordinates to the pixel
coordinates. Finally, we get the $T$ matrix, which transforms the world
coordinates to the pixel coordinates:

$\displaystyle T=T_{c\rightarrow f\rightarrow p}T_{r\rightarrow
c}T_{w\rightarrow r}\in\mathbb{R}^{2\times 4}.$ (8)

To obtain the vehicle (camera) position in the pixel coordinates (u,v):

$\displaystyle\begin{bmatrix}u^{\prime}\\\
v^{\prime}\end{bmatrix}=T\begin{bmatrix}U_{cam}\\\ V_{cam}\\\ W_{cam}\\\
1\end{bmatrix}.$ (9)

However, this coordinate-transformed point $[u^{\prime},v^{\prime}]$ in the
pixel coordinates has the origin at the top left corner of the image. In our
work, as we deal with the state trajectory of the vehicle, we define the new
origin at the bottom center of the image $[\frac{w}{2},h]$, where $h$ and $w$
represents the height and width of the image, and rotate the axes by switching
$u^{\prime}$ and $v^{\prime}$. Finally, we subtract $[v^{\prime},u^{\prime}]$
from $[\frac{w}{2},h]$ and get the final $[u,v]$:

$\displaystyle\begin{bmatrix}u\\\ v\end{bmatrix}=\begin{bmatrix}w/2\\\
h\end{bmatrix}-\begin{bmatrix}v^{\prime}\\\ u^{\prime}\end{bmatrix}.$ (10)

We still use the same system dynamics in Eq. 3 for model prediction, where
$\mathbf{x}=[x,y,yaw,roll,v_{x},v_{y},\dot{yaw}]$ is the vehicle state in a
world coordinate frame and $\mathbf{u}$ is $[throttle,steering]$. In other
words, MPPI still plans in regular driving space, in the world coordinates.

Through the coordinate transform at every timestep, the MPPI-planned final
future state trajectory mapped in image space on our costmap looks like Fig.
1.

We use the total cost function Eq. 1 with a running cost Eq. 4 and zero
terminal cost for an autonomous driving task. In this formulation of MPPI, two
cost terms predominate: speed and track-related crash costs. The track cost
depends on the costmap and it is a binary grid map (0, 1) describes occupancy
of features we want to avoid driving through, e.g. track boundaries or lane
boundaries on the road. Going off the image plane does not have a cost
associated with it.

Figure 1: MPPI in image space. The green trajectory is the MPPI-planned future
state trajectory in image space. The black and white background represents the
costmap that MPPI is optimizing.

##  IV Methods

In this work, we introduce a method for an inverse reinforcement learning
problem and the task is vision-based autonomous driving. More specifically, we
focus on lane-keeping and collision checking like in [5, 6, 1, 2, 12]. The
following methods are evaluated in Section V.

###  IV-A End-to-End Imitation Learning for Autonomous Driving

Pan et al. [2] constructed a CNN that takes in RGB images and spits out
control actions of throttle and steering angles for an autonomous vehicle. It
was trained using DAgger on data provided by an expert MPC controller. While
it can achieve aggressive driving targets and was shown to handle various
lighting conditions on the same track, it in general does not generalize to
brand new tracks. This is most likely due to the images creating a feature
space not seen in training. While the last layers may not be able to choose
the proper action, intermediate layers still perform some feature extraction.
This feature extraction is further discussed in Section IV-C.

###  IV-B Attention-based Costmap Prediction

Drews [1] provides a template NN architecture and training procedure to try to
generalize costmap prediction to new environments in a method we call
Attention-based Costmap Prediction (ACP). The concise description of this work
is to create a NN that can take in camera images and output a costmap used by
a MPC controller. The major contributions over [5, 6] are using a Conv-LSTM
layer to maintain the spatial information of states close together in time as
well as a softmax attention mechanism applied to sparsify the Conv-LSTM layer.
They show that the attention image mimics areas where humans focus when
driving a vehicle, which provides evidence of a generalization technique
similar to humans. The costmap generated by this NN architecture is then
provided to an MPPI controller. By separating the perception and low-level
control into two robust components, this system can be more resilient to small
errors in either. Their final model is trained on a mixture of real datasets
of a simple racetrack as well as simulation datasets from a more complex
track. The full system is then able to drive around the real world version of
the complex track in an aggressive fashion without crashing. It is the most
generalized method for achieving autonomous driving in new environments that
the authors of this paper have found in the literature.

###  IV-C Approximate Inverse Reinforcement Learning (AIRL)

Our method can be considered a mixture of the two previously mentioned; we
will be using both E2EIL and an MPC controller. Although our work relies on
E2EIL and MPC, we tackle a totally different problem: IRL from E2EIL. Our main
contribution is learning an approximate, ‘generalizable’ costmap ‘from’ E2EIL
with a minimal extra cost of adding a binary filter. On top of this AIRL, we
perform MPC in image space (Section III) with a real-time-generated agent-view
costmap. To repeat our problem statement, it is an inverse reinforcement
learning problem of learning a cost function and the task is autonomous
driving.

We extract middle convolutional layers from the trained E2EIL network and use
them as a costmap for MPC. The averaged activation map (heat map) of each
pixel in the middle layer of E2EIL network is used to generate a costmap.
Pixel-wise heatmaps or activation maps have been widely used to interpret and
explain the NN’s predictions and the information flow, given an input image
[26, 27]. After training, each neuron’s activation from the middle layer tells
us the relevance of input, important features (Fig. 3a), and output. We
interpret this intermediate stage, the activated heatmap, as important
features, which relates the input and the output. Under the optimal control
settings, we view these relevant features as cost function-related features,
the intermediate step between the observation and the final optimal decision.

Our proposed approach requires one assumption: just like typical IL settings,
we assume the expert’s behavior is optimal. In this work, we used a model
predictive optimal controller, MPPI [24], as the expert for E2EIL. This is
similar to [1] in that we have separated the perception pipeline from the
controls.

The training process is the same as the E2EIL controller; AIRL only requires a
dataset of images, wheel speed sensor readings, and the expert’s optimal
solution to train a costmap model (see Fig. 2). The input image size is
$160\times 128\times 3$ and the output costmap from the middle layer is
$40\times 32$. This 2D costmap comes from taking the average of the activated
neurons with respect to all 128 kernels ($128\times 40\times 32\times
3\rightarrow 40\times 32\times 3$), and converting the 3D RGB channel into
greyscale ($40\times 32\times 3\rightarrow 40\times 32$). This is then resized
to $160\times 128$ for MPPI.

Unlike [5, 6, 1], our method does not require access to a predetermined
costmap function in order to train. Also, due to the fact that E2EIL can be
taught from human data only [12], our approach can learn a cost function even
without teaching specific task-related objectives to a model.

Figure 2: Blue box: The same structure used in [2] for IL training. For IRL
testing, only the red-dashed part, from the image input to the second max
pooling layer is used. This middle layer internally and implicitly learns the
mapping from an input image to important features, which can be used as a
costmap.

We would like to use all of the activated middle layer neurons to generate a
costmap, but the magnitude of the activation is different for each feature.
Since we consider all the activated features important for a costmap, we add a
binary ($0$ or $1$) filter. The binary filter outputs 1 if the activation is
greater than 0. In this way, we equally regard all the activated features as
important ones. Adding a binary filter may look like a simple step, but this
is the biggest reason why our costmap generation is stable while the E2E
controller fails. Details will be discussed in Section VI.

Furthermore, for safe navigation, there are two ways to make collision-free
navigation. One way is to assume the robot to be larger than its actual size;
this is equivalent to putting safety margins around the robot. The other way
is to assume the obstacles to be larger than their real size, making the
optimal path planning more conservative in the real world. Both approaches
will result in a similar behavior of collision-averse navigation, but since
our paper focuses on generating a costmap,  we introduce a $3\times 3$
Gaussian blur filter on the costmap so that the pixels around the objects are
also highlighted and a trajectory crossing them is penalized (Fig. 3c). This
Gaussian blur is tunable, so the costmap also becomes tunable to match a user-
defined risk-sensitivity. Increasing the size of the blur will generate a more
risk-averse costmap for an optimal controller. As a result, with a risk-
sensitive costmap, the optimal controller drives the vehicle in low-speed
while gaining more safety (less collisions). On the other hand, risk-neutral
costmap allows the vehicle to drive at a high speed, but results in more risky
behavior (e.g. driving too close to the road boundaries). We also tested
blurring the features in the input image space, so that the pixels close to
the important features are also relevant. As expected, it showed similar
results compared to putting a Gaussian blur filter. The approaches introduced
in this paragraph are the extensions of the vanilla AIRL method and can be
used in the risk-sensitive control case. In the next section, we show the
experimental results of the vanilla AIRL and leave some room for the risk-
sensitive version for future works.

(a)

(b)

(c)

Figure 3: The two-step filtering process of making a safety-considered
costmap. (a) is the original output from the deep CNN middle layer. The white
colored pixels are the activated neurons averaged among the convolution
filters. After applying a binary filter (b), the white pixels have a cost of 1
and the black ones have a cost of 0. Finally a Gaussian blur of $3\times 3$ is
applied in (c)

##  V Experiments

We compare the methods mentioned in Section IV on the following scenarios:

  1. 1.

Track A (real world complex track)

  2. 2.

Track C (real world pavement)

  3. 3.

Track B (real world oval track)

  4. 4.

Track D (gazebo, simulated Track B)

  5. 5.

Track E (gazebo, simulated)

  6. 6.

TORCS driving simulator

  7. 7.

KITTI dataset

(a) The test vehicle

(b) Track A

(c) Track B

(d) Track C

(e) Track D

(f) Track E

Figure 4: a) The $1/5$ scaled ground vehicle used for experiments and b) the
track used for training (Track A) and c-g) tracks used for testing (Track B,
C, D, and E) Track D and E are from the ROS Gazebo simulator. Note that Track
E is a simulated version of Track A, and Track D is a simulated version of
Track B.

For a fair comparison, we trained all models with the same dataset used in
[6]. The data set consists of a vehicle running around a 170m-long track shown
in Fig. 4 as Track A. It includes various lighting conditions, and views on
the track. Also shown in the supplementary
video111https://youtu.be/WyJfT5lc0aQ, the testing environment includes
different lighting/shadow conditions and all the ruts, rocks, leaves, and
grass on the dirt track provide various textures. With a learning rate of
0.001, Adam [28] was used as an optimizer in Tensorflow [29]. The learning
converged with a training loss of $4e^{-3}$ after $400$ epochs. The rest of
this section will explore how each method performed on each track. These
methods were compared over various real and simulated datasets including the
TORCS open source driving simulator [30] dataset and the KITTI dataset [31].
For the TORCS dataset, we used the baseline test set collected by [32]. All
hardware experiments were conducted using the 1/5 scale AutoRally autonomous
vehicle test platform [33].

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

Figure 5: From Top: KITTI, TORCS, Track A, Track B, and Track C. Left Column:
Resized and cropped input image. Middle Column: AIRL (ours) costmap generation
without a blur filter. Right: Direct costmap prediction from ACP. E2EIL does
not provide a costmap output so it is not pictured here. Note that ACP works
well on Track A data, which it was trained on.

###  V-A Costmap Prediction

We first ran our costmap models AIRL and ACP on various datasets to show
reasonable outputs in varied environments. The datasets used are KITTI, TORCs,
Track A, Track B, and Track C as shown in Fig. 3. AIRL produced costmaps that
are interpretable by humans. The predicted costmap of the ACP is interpreted
similarly to our method. The vehicle is located at the bottom middle of the
costmap and black represents the low-cost region, white represents the high-
cost. The difference is that ACP produces a top-down-view/bird-eye-view
costmap, whereas our method, AIRL, produces a driver-view costmap. ACP
produced clear cost maps models in Track A (which it was trained on) and Track
C, though Track C’s costmap was incorrect. These results show an inability for
ACP to generalize to varied different environments whereas our method produces
similar looking costmaps throughout.

###  V-B Autonomous Driving

We then took all three methods and drove them on Tracks B, C, D, and E.For
Tracks B, D, and E,we ran each algorithm in both clockwise and counter-
clockwise for 20 lap attempts and measured the average travel distance. We can
see in Fig. 7, that our approach was the only method that was able to finish
the whole lap of driving Track B and D.Compared to other methods, AIRL tended
to hug track boundaries closely, presumably because of the sparsity of our
costmaps. We note in Section V-C a failure state of our method and potential
reasons for it.

The parameters we used for AIRL’s MPPI in image space for all trials are as
follows: $\mathbf{K}=2,v_{x}^{d}=5.0m/s$ for off-road driving,
$v_{x}^{d}=2.5m/s$ for on-road driving,
$\Delta_{t}=0.02,\mathrm{T}=60,\Sigma_{steer}=0.3,\Sigma_{throttle}=0.35,C_{speed}=1.8$,
and $C_{crash}=0.9$. $\mathrm{T}$ was set to correlate to approximately $6m$
long trajectories, as this covers almost all the drivable area in the camera
view (see Fig. 1).

ACP failed to drive more than half of Track B because the predicted costmap
was not stable as seen in Fig. 5. We ensured that the poor performance of ACP
was not due to improper tuning of MPPI by training another model of [1] on
Track B data only. We then tuned MPPI with this model and drove it around
Track B successfully for 10 laps straight before being manually stopped. After
this verification of MPPI parameters, we applied the same parameters to ACP.
Unfortunately, we did not see the same track coverage with properly tuned
MPPI. When looking at the costmaps generated from ACP in Fig. 6, we can see
that the model trained on Track A is not generating a clear costmap.

Figure 6: Costmap prediciton differences of ACP due to different training data
on Track B. Left: Resized and cropped input image from Track B. Middle:
costmap prediction from ACP trained on Track B. Right: costmap prediction from
ACP trained on Track A.

Overall, ACP performed best on Track E, which is a simulated version of the
track it was trained upon, Track A. The other Tracks have a similar issue to
Track B, i.e. unclear costmaps.

Figure 7: Generalization results of autonomous driving on tracks outside of
the training dataset.

Surprisingly, E2EIL was able to drive up to half of a lap on Track B. In all
of the sim datasets (Tracks D and E), it did not move. This is most likely due
to images not matching the training distribution of images. The ability to
drive on Track B is most likely due to the images being somewhat similar to
Track A as can be seen in Fig. 5.

We also verified the generalization of each method at a totally new on-road
environment, Track C. We made a 30m-long zigzag lane on the tarmac to look
like a real road situation. Since the training data was collected at Track A
(Fig. 4b), an off-road dirt track, the tarmac surface is totally new; in
addition, the boundaries of the course changed from black plastic tubes to
taped white lanes (Fig. 4d). The width between the boundaries varied from 0.5
m to 1.5 m and was in general much tighter than the off-road tracks. Moreover,
we ran our algorithm in the late afternoon, which has very different lighting
conditions compared to the training data as seen in Fig. 5.

Even under this large change of environments, AIRL with MPPI was still able to
accomplish a lane-keeping task most of the time, whereas the other two
methods, E2EIL and ACP immediately failed the task.

###  V-C Failure case

From Track E, we report a failure case of our method where the vehicle could
not proceed to move forward. Fig. 8 shows the case where at a specific turn,
the optimal controller does not provide a globally optimal solution because
the costmap it tries to solve does not include any meaningful or useful
information to make a control decision.

Figure 8: Left: Input image to AIRL. Right: Predicted costmap, which is not
able to tell MPPI which direction to take.

##  VI Discussion

###  VI-A Why did other methods fail?

If we split the typical autonomy pipeline in two, we can split it into a) a
pipeline from sensor measurements to task-specific objective functions
generation, and b) a pipeline from objective functions to corresponding
optimal path and control. In the costmap learning approach ACP, Drews [1] uses
deep learning to replace the pipeline a), and uses an MPC controller to handle
b). However, if the first pipeline fails to produce a correct objective
function, the second part of path planning will calculate a wrong result and
fail the task, no matter how well the controller or path planner is tuned. In
our experiments, we saw this happens frequently (Fig. 5) and we analyze that
this false prediction is came from the input image. The input might be new to
the network, i.e. the training data did not include that specific image, or
the trained network did not correctly learn the mapping from those input data
to a corresponding output.

Our approach also replaces the first pipeline a) with deep learning, but it
always outputs a correct costmap through AIRL no matter whether the input is
corrupted or new. In E2EIL, although the middle layer outputs meaningful
features/heatmap, a small change of each middle layer’s activation coming from
a novel input results in a random or false NN output. For this reason, we
cannot use the whole (same) architecture and its weights used in the E2EIL
training phase. However, without throwing it away, we can still use the CNN
portion of the original network for feature extraction, which shows great
generalizability after applying the binary filter we introduced in the AIRL
costmap generation step. As a result, in AIRL, a small change of the activated
heatmap does not affect the final costmap.

###  VI-B Comparison to a simple road detection method

A binary road-detection network + MPPI might perform as well as our proposed
approach. However, a road-detection network requires manual labeling of what
is a road and what is not. Also, to train a network which predicts the road,
we would have to cover all possible kinds of roads to have good
generalizability, but which increases the amount of labeling required
dramatically. Our approach uses the imitation learning framework, which does
not require any extra labeling, and learns the task-related costmap which
generalizes to various kinds of roads.

###  VI-C Max Speed compared to [1]

In [1], the authors set the target velocity to almost twice as fast as our
settings. They were able to do this for many reasons. First, predicting
drivable area [1] rather than obstacles (our approach) lends itself to faster
autonomous racing. Second, the costmap generated in [1] has more gradient
information than our binary costmap. This allows MPPI to compute trajectories
that are better globally. Third, the myopic nature of our algorithm is the
main reason why our algorithm cannot go as fast as [1]. Our approach finds a
locally optimal solution given a drivable costmap in front of the agent.
Unlike our approach, [1] specifically trained a costmap predictor to predict a
costmap 10-15$m$ ahead with a pre-defined global costmap although the camera
could not see that far ahead.

###  VI-D Beyond Autonomous Driving

Any vision-based MDP problems, especially for camera-attached agents (e.g.
manipulator, drone), are possible applications of the proposed approach. For a
manipulator reaching task or a drone flying task with obstacle avoidance, and
after imitation learning of the tasks, our middle layer heatmap will output a
binary costmap composed of specific features of obstacles (high cost) and
other reachable/flyable regions (low cost). With these cost maps, an optimal
MPC controller can run in image space as in Section III.

###  VI-E Future Directions

Our idea of extracting middle layers of CNNs and using them as a costmap
generator can be used to boost the training procedure of end-to-end
controllers; if we use a known costmap to train an end-to-end controller,
using moment matching like in [18, 19], we can train a deep CNN controller
with two loss functions, one to fit a costmap in the middle layer and the
other with the final action at the output. In this way, the end-to-end model
will simultaneously learn the costmap internally and the optimal policy
explicitly and become more robust under new or different environments.

Similar to the max speed problem in Section VI-C, our proposed method has a
problem being too myopic. It works well in navigation along with a model
predictive controller, but the MPC only solves an optimization problem with a
local costmap. To solve this problem, we can incorporate a recurrent framework
so that we can predict further into the future and find a better global
solution.

The problem of driving too close to the road boundaries or obstacles can be
solved by introducing a risk-sensitive AIRL with a blur filter introduced in
Section IV-C, but we can also solve the problem by converting our binary
costmap to have smooth gradient information like in [1]. This will help MPPI
or any gradient-based optimal controller to find a better solution which
drives the vehicle to stay in the middle of the road (the lowest cost area).

##  VII Conclusion

We introduced an Approximate Inverse Reinforcement Learning framework using
deep Convolutional Neural Networks. Transferring middle layers of an IL-
trained network as a cost function holds the promises to automate the feature
extraction and cost function design for other vision-based tasks. The proposed
method allows to avoid manually designing a cost map that is generally
required in supervised learning. Our approach outperforms other state-of-the-
art vision and deep-learning-based controllers in generalizing to new
environments.

## Acknowledgments

This work was supported by Amazon Web Services (AWS) and NASA.

## References

  * Drews [2019] Paul Drews.  _Visual Attention for High Speed Driving_.  PhD thesis, Georgia Institute of Technology, 2019.  URL https://smartech.gatech.edu/bitstream/handle/1853/61183/DREWS-DISSERTATION-2019.pdf.
  * Pan et al. [2018] Yunpeng Pan, Ching-An Cheng, Kamil Saigol, Keuntaek Lee, Xinyan Yan, Evangelos A. Theodorou, and Byron Boots.  Agile Autonomous Driving using End-to-End Deep Imitation Learning.  _Robotics: Science and Systems_ , 2018.  URL http://www.roboticsproceedings.org/rss14/p56.pdf.
  * Thrun [2006] S. et al Thrun.  Stanley: The robot that won the DARPA Grand Challenge.  _Journal of field Robotics_ , 23(9):661–692, 2006\.  URL http://robots.stanford.edu/papers/thrun.stanley05.pdf.
  * Williams et al. [2016] Grady Williams, Paul Drews, Brian Goldfain, James M. Rehg, and Evangelos A. Theodorou.  Aggressive driving with model predictive path integral control.  _2016 IEEE International Conference on Robotics and Automation (ICRA)_ , 2016.  URL {https://ieeexplore.ieee.org/abstract/document/7487277}.
  * Drews et al. [2017] Paul Drews, Grady Williams, Brian Goldfain, Evangelos A. Theodorou, and James M. Rehg.  Aggressive Deep Driving: Combining Convolutional Neural Networks and Model Predictive Control.  In _1st Annual Conference on Robot Learning, CoRL 2017, Mountain View, California, USA, November 13-15, 2017, Proceedings_ , pages 133–142, 2017\.  URL https://dblp.org/rec/bib/conf/corl/DrewsWGTR17.
  * Drews et al. [2019] Paul Drews, Grady Williams, Brian Goldfain, Evangelos A. Theodorou, and James M. Rehg.  Vision-Based High-Speed Driving With a Deep Dynamic Observer.  _IEEE Robotics and Automation Letters_ , 4(2):1564–1571, April 2019.  ISSN 2377-3774.  doi: 10.1109/LRA.2019.2896449.  URL https://ieeexplore.ieee.org/document/8630018.
  * Levine et al. [2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel.  End-to-End Training of Deep Visuomotor Policies.  _Journal of Machine Learning Research_ , 17(39):1–40, 2016.  URL http://jmlr.org/papers/v17/15-522.html.
  * Giusti et al. [2016] Alessandro Giusti, Jerome Guzzi, Dan Ciresan, Fang-Lin He, Juan Pablo Rodriguez, Flavio Fontana, Matthias Faessler, Christian Forster, Jurgen Schmidhuber, Gianni Di Caro, Davide Scaramuzza, and Luca Gambardella.  A Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots.  _IEEE Robotics and Automation Letters_ , 2016.  URL https://ieeexplore.ieee.org/abstract/document/7358076.
  * Xingjian et al. [2015] SHI Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo.  Convolutional lstm network: A machine learning approach for precipitation nowcasting.  In _Advances in neural information processing systems_ , pages 802–810, 2015.
  * Loquercio et al. [2019] Antonio Loquercio, Elia Kaufmann, René Ranftl, Alexey Dosovitskiy, Vladlen Koltun, and Davide Scaramuzza.  Deep drone racing: From simulation to reality with domain randomization.  _IEEE Transactions on Robotics_ , 2019.  URL https://ieeexplore.ieee.org/document/8877728.
  * Li et al. [2019] Weizi Li, David Wolinski, and Ming C. Lin.  ADAPS: Autonomous driving via principled simulations.  In _Proceedings - IEEE International Conference on Robotics and Automation_ , pages 7625–7631. Institute of Electrical and Electronics Engineers Inc., 2019.  ISBN 9781538660263.  doi: 10.1109/ICRA.2019.8794239.  URL https://ieeexplore.ieee.org/document/8794239.
  * Bojarski et al. [2016] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba.  End to End Learning for Self-Driving Cars.  _arXiv_ , Apr 2016.  URL https://arxiv.org/abs/1604.07316.
  * Xu et al. [2017] Huazhe Xu, Yang Gao, Fisher Yu, and Trevor Darrell.  End-to-end learning of driving models from large-scale video datasets.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2174–2182, 2017.  URL https://ieeexplore.ieee.org/document/8099859.
  * Ollis et al. [2008] M. Ollis, W. H. Huang, M. Happold, and B. A. Stancil.  Image-based path planning for outdoor mobile robots.  In _2008 IEEE International Conference on Robotics and Automation_ , pages 2723–2728, May 2008.  doi: 10.1109/ROBOT.2008.4543623.  URL https://ieeexplore.ieee.org/document/4543623.
  * Ng and Russell [2000] Andrew Y. Ng and Stuart J. Russell.  Algorithms for inverse reinforcement learning.  In _Proceedings of the Seventeenth International Conference on Machine Learning_ , ICML ’00, page 663–670, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc.  ISBN 1558607072.  URL http://users.umiacs.umd.edu/~hal/courses/2016F_RL/Ng00.pdf.
  * Arora and Doshi [2018] Saurabh Arora and Prashant Doshi.  A survey of inverse reinforcement learning: Challenges, methods and progress.  _arXiv preprint arXiv:1806.06877_ , 2018.
  * Russell [1998] Stuart Russell.  Learning agents for uncertain environments.  In _Proceedings of the eleventh annual conference on Computational learning theory_ , pages 101–103, 1998.  doi: 10.1145/279943.279964.  URL https://dl.acm.org/doi/pdf/10.1145/279943.279964.
  * M. Babes and Subramanian. [2011] M. Littman M. Babes, V. Marivate and K. Subramanian.  Apprenticeship learning about multiple intentions.  In _Proceedings of the 28th International Conference on Machine Learning_ , 2011.  URL http://www.icml-2011.org/papers/478_icmlpaper.pdf.
  * Kuderer et al. [2015] M. Kuderer, S. Gulati, and W. Burgard.  Learning driving styles for autonomous vehicles from demonstration.  In _2015 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 2641–2646, May 2015.  doi: 10.1109/ICRA.2015.7139555.  URL https://ieeexplore.ieee.org/document/7139555.
  * Ross et al. [2011] Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell.  A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning.  In _Proceedings of the 14th International Conference on Artificial Intelligence and Statistics_ , volume 15 of _JMLR_ , Fort Lauderdale, FL, USA, 2011.  URL http://proceedings.mlr.press/v15/ross11a/ross11a.pdf.
  * Lee et al. [2019] Keuntaek Lee, Kamil Saigol, and Evangelos A. Theodorou.  Early failure detection of deep end-to-end control policy by reinforcement learning.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 8543–8549, May 2019.  doi: 10.1109/ICRA.2019.8794189.
  * Lee et al. [2019a] Keuntaek Lee, Gabriel Nakajima An, Viacheslav Zakharov, and Evangelos A Theodorou.  Perceptual attention-based predictive control.  _3rd Conference on Robot Learning (CoRL)_ , 2019a.
  * Lee et al. [2019b] Keuntaek Lee, Ziyi Wang, Bogdan I Vlahov, Harleen K Brar, and Evangelos A Theodorou.  Ensemble bayesian decision making with redundant deep perceptual control policies.  _18th IEEE International Conference on Machine Learning and Applications (ICMLA)_ , 2019b.
  * Williams et al. [2017] Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M. Rehg, Byron Boots, and Evangelos A. Theodorou.  Information theoretic MPC for model-based reinforcement learning.  In _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 1714–1721, May 2017.  doi: 10.1109/ICRA.2017.7989202.  URL https://ieeexplore.ieee.org/abstract/document/7989202.
  * Trucco and Verri [1998] Emanuele Trucco and Alessandro Verri.  _Introductory Techniques for 3-D Computer Vision_.  Prentice Hall PTR, Upper Saddle River, NJ, USA, 1998.  ISBN 0132611082.
  * Montavon et al. [2018] Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller.  Methods for interpreting and understanding deep neural networks.  _Digital Signal Processing_ , 73:1 – 15, 2018.  ISSN 1051-2004.  doi: https://doi.org/10.1016/j.dsp.2017.10.011.  URL http://www.sciencedirect.com/science/article/pii/S1051200417302385.
  * Samek et al. [2019] Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller.  _Explainable AI: Interpreting, Explaining and Visualizing Deep Learning_ , volume 11700.  Springer, Cham, 2019.  doi: 10.1007/978-3-030-28954-6.  URL http://dx.doi.org/10.1007/978-3-030-28954-6.
  * Kingma and Ba [2014] Diederik P. Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  _Proceedings of the 3rd International Conference on Learning Representations (ICLR)_ , abs/1412.6980, 2014.  URL http://arxiv.org/abs/1412.6980.
  * Abadi et al. [2015] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems, 2015.  URL http://tensorflow.org/.  Software available from tensorflow.org.
  * Wymann et al. [2015] Bernhard Wymann, Christos Dimitrakakisy, Andrew Sumnery, and Christophe Guionneauz.  Torcs: The open racing car simulator, 2015.  URL http://www.cse.chalmers.se/~chrdimi/papers/torcs.pdf.
  * Geiger et al. [2013] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  _International Journal of Robotics Research (IJRR)_ , 2013.  doi: 10.1177/0278364913491297.  URL https://journals.sagepub.com/doi/full/10.1177/0278364913491297.
  * Chen et al. [2015] Chenyi Chen, Ari Seff, Alain Kornhauser, and Jianxiong Xiao.  DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving.  _Proceedings of 15th IEEE International Conference on Computer Vision_ , 2015.  URL http://deepdriving.cs.princeton.edu/paper.pdf.
  * Goldfain et al. [2019] B. Goldfain, P. Drews, C. You, M. Barulic, O. Velev, P. Tsiotras, and J. M. Rehg.  Autorally: An open platform for aggressive autonomous driving.  _IEEE Control Systems Magazine_ , 39(1):26–55, Feb 2019.  URL https://ieeexplore.ieee.org/document/8616931.

  *[E2E]: end-to-end
  *[IL]: Imitation Learning
  *[LSTM]: Long Short Term Memory
  *[MPC]: Model Predictive Control
  *[Conv-LSTM]: Convolutional LSTM
  *[CNN]: Convolutional Neural Network
  *[MPPI]: Model Predictive Path Integral
  *[RL]: Reinforcement Learning
  *[IRL]: Inverse Reinforcement Learning
  *[MDP]: Markov Decision Process
  *[DAgger]: Data Aggregation
  *[NN]: Neural Network
  *[DL]: Deep Learning
  *[ACP]: Attention-based Costmap Prediction
  *[AIRL]: Approximate Inverse Reinforcement Learning