# Tricritical curve of massive chiral Gross-Neveu model with isospin

Michael<EMAIL_ADDRESS>Institut für Theoretische Physik,
Universität Erlangen-Nürnberg, D-91058, Erlangen, Germany

###### Abstract

We reconsider the two-flavor version of the massive, chiral Gross-Neveu model
in 1+1 dimensions. Its phase diagram as a function of baryon chemical
potential, isospin chemical potential and temperature has previously been
explored. We recapitulate the results, adding the missing tricritical curves.
They can be determined exactly by extending the standard stability analysis,
using fourth order almost degenerate perturbation theory. Results for three
different bare masses are presented and discussed.

##  I Introduction

The two best known Gross-Neveu (GN) models L1  are the original one with
discrete chiral symmetry ($\psi\to\gamma_{5}\psi$) and Lagrangian

${\cal L}_{\rm
GN}=\bar{\psi}i\partial\\!\\!\\!/\psi+\frac{g^{2}}{2}\left(\bar{\psi}\psi\right)^{2}$
(1)

and the chiral GN model with U(1) chiral symmetry ($\psi\to
e^{i\gamma_{5}\alpha}\psi$),

${\cal L}_{\rm
NJL}=\bar{\psi}i\partial\\!\\!\\!/\psi+\frac{g^{2}}{2}\left[(\bar{\psi}\psi)^{2}+(\bar{\psi}i\gamma_{5}\psi)^{2}\right].$
(2)

We will only be dealing with 1+1 dimensions and the large $N_{c}$ limit of
fermions with a U($N_{c}$) “color” symmetry. Color indices will be suppressed
as usual ($\bar{\psi}\psi=\sum_{k=1}^{N_{c}}\bar{\psi}_{k}\psi_{k}$ etc.).
Extending model (2) to SU(2) chiral symmetry by introducing isospin, we arrive
at the Lagrangian

${\cal L}_{\rm
isoNJL}=\bar{\psi}i\partial\\!\\!\\!/\psi+\frac{G^{2}}{2}\left[(\bar{\psi}\psi)^{2}+(\bar{\psi}i\gamma_{5}\vec{\tau}\psi)^{2}\right],$
(3)

familiar from the standard Nambu–Jona-Lasinio (NJL) model in 3+1 dimensions L2
. As already indicated in the subscripts of the Lagrangians, we shall refer to
(1) as GN model, (2) as NJL model and (3) as NJL model with isospin (isoNJL).
Actually, the focus of the present work will be on the massive versions of
these models obtained by adding a Dirac mass term (bare mass $m_{b}$)

$\delta{\cal L}=-m_{b}\bar{\psi}\psi.$ (4)

However, by way of introduction, we find it appropriate to briefly recall what
is known about the phase diagrams of models (1)–(3) in the chiral limit L3 .

The starting point for solving the isoNJL model (3) in the large $N_{c}$ limit
is the Dirac Hartree-Fock (HF) equation

$\left(-i\gamma_{5}\partial_{x}+\gamma^{0}S+i\gamma^{1}\vec{\tau}\cdot\vec{P}-\mu-\nu\tau_{3}\right)\psi=\omega\psi.$
(5)

We have introduced a baryon chemical potential $\mu$ and an isospin chemical
potential $\nu$. The scalar and pseudoscalar mean fields $S,\vec{P}$ satisfy
the following self-consistency conditions,

$\displaystyle S$ $\displaystyle=$
$\displaystyle-G^{2}\langle\bar{\psi}\psi\rangle,$ $\displaystyle\vec{P}$
$\displaystyle=$
$\displaystyle-G^{2}\langle\bar{\psi}i\gamma_{5}\vec{\tau}\psi\rangle,$ (6)

where the brackets denote either ground state or thermal averages. Let us now
assume that the charged pseudoscalar condensate vanishes, $\vec{P}_{\perp}=0$.
Then the HF Hamiltonian becomes diagonal in isospin space with

$\left(-i\gamma_{5}\partial_{x}+\gamma^{0}S\pm
i\gamma^{1}P_{3}-\mu\mp\nu\right)\psi=\omega\psi$ (7)

for isospin up and down, respectively. In each isospin channel, the Dirac HF
equation reduces to that of a single-flavor NJL model. For isospin up, the
mean field is $\Delta=S-iP_{3}$ and the chemical potential $\mu+\nu$. For
isospin down, the corresponding parameters are $\Delta=S+iP_{3}$ and
$\mu-\nu$. Although the HF Hamiltonian is diagonal in isospin, the two isospin
channels are still coupled through the self-consistency condition, as $P_{3}$
involves the difference between up and down contributions.

It is now easy to see that the phase diagram of the isoNJL model in
($\mu,\nu,T$) space can be constructed from the known phase diagrams of the GN
and NJL models in ($\mu,T$) space. To this end, consider first the special
cases where one of the chemical potentials vanishes. For $\nu=0$ (pure baryon
chemical potential), the NJL equations for isospin up and down have the same
chemical potential $\mu$ and complex conjugate mean fields $\Delta=S\mp
iP_{3}$. This is only possible if $\Delta$ is real ($P_{3}=0$). Thus the
isoNJL phase diagram in the $\nu=0$ plane is identical to the GN phase diagram
in the ($\mu,T$) plane, as indeed first noticed in the numerical study L4 .
The GN phase diagram in turn is known analytically L5  and features three
phases, a chirally restored one, a homogeneous massive one and a soliton
crystal. For $\mu=0$ on the other hand (pure isospin chemical potential
$\nu$), the NJL equations (5) have opposite chemical potentials $\pm\nu$ and
the mean fields are complex conjugates. This is exactly what it takes to solve
both equations with the standard NJL solution, so that the phase diagram of
the isoNJL model in the $\mu=0$ plane is that of the one-flavor NJL model with
chemical potential $\nu$. It is also well known analytically and consists of a
chirally restored phase and a soliton crystal of “chiral spiral” type L6 ; L7
. Thus, the phase diagram of the isoNJL model on the boundaries $\nu=0,\mu=0$
is completely determined by the single-flavor GN and NJL phase diagrams, see
Fig 1.

How can these boundary phase diagrams be continued into the bulk of
($\mu,\nu,T$) space? As noticed in Ref. L3 , the same trick used originally to
derive the chiral spiral, namely a chiral rotation with linearly $x$-dependent
phase, can be applied to the isospin case as well. This shows that the phase
boundaries of the isoNJL model are independent of $\nu$, just like the phase
boundary of the NJL model is independent of $\mu$. The full phase diagram can
thus be generated by simply translating the GN phase diagram rigidly into the
direction of the $\nu$ axis, see Fig. 2. The resulting mean field is the
product of the GN mean field and the NJL chiral spiral phase factor,

$S\mp iP_{3}=S_{\rm GN}(\mu,T,x)e^{\pm 2i\nu x}.$ (8)

The structure of a double helix emerges where up- and down contributions have
opposite handedness. This gives rise to three distinct phases of the massless
isoNJL model: A chirally restored one ($I$), a double chiral spiral with
constant radius ($II$) and a double chiral spiral with $x$-dependent radius,
modulated by the shape of the GN kink crystal ($III$). One finds that this
solution is self-consistent, in spite of the fact that $P_{1,2}=0$ has been
assumed from the outset. No better HF solution (with neutral and charged pion
condensates) has been found so far, including the earlier variational
calculations L8 ; L9 .

This is the status of the phase diagrams of models (1)–(3) in the chiral
limit. The purpose of the present paper is to continue investigating the phase
diagram of the massive isoNJL model. As already known from the one-flavor
models, we cannot expect an analytical solution any more, but have to engage
in numerical calculations as well. This complicates matters significantly, but
the phase diagram is also expected to be richer than in the chiral limit.

The paper is organized as follows. In Sect. II we review what is known about
the phase diagram of the massive isoNJL model to date. In Sect. III we outline
a method recently proposed to find the exact tricritical point in the massive
NJL model. In Sect. IV we adapt this method to the massive isoNJL model and
present detailed results for the full phase diagram, including for the first
time the tricritical curve in ($\mu,\nu,T$) space.

Figure 1: Boundary phase diagram of massless isoNJL model if one of the
chemical potentials vanishes. At $\nu=0$, same as massless GN model. At
$\mu=0$, same as massless NJL model with chemical potential $\nu$. Figure 2:
Phase diagram of massless isoNJL model in the bulk of ($\mu,\nu,T$) space. The
phase boundary sheets result from moving the GN phase diagram rigidly into the
direction of the $\nu$ axis. $I$) chirally restored phase, $II$) chiral spiral
with constant radius, $III$) chiral spiral with radius modulated by GN soliton
crystal.

##  II Constructing the phase diagram of the massive ${\bf isoNJL}$ model

We now turn to the massive versions of models (1)–(3) by adding the Dirac mass
term (4) to each Lagrangian. Here the phase diagrams of the GN and NJL model
are again known analytically (massive GN L10 ) or at least numerically
(massive NJL L11 ). Recall that during renormalization, the two bare
parameters $m_{b}$ and $G^{2}$ are traded for the physical fermion mass in
vacuum ($m$) and the so-called “confinement parameter” ($\gamma$) according to
the gap equation

$\frac{\pi}{2N_{c}G^{2}}=\gamma+\ln\frac{\Lambda}{m},\quad\gamma=\frac{\pi
m_{b}}{2N_{c}G^{2}m}=\frac{m_{b}}{m}\ln\frac{\Lambda}{m}.$ (9)

This version belongs to the two-flavor isoNJL model (3). In the case of the
one-flavor models (1), (2), replace $G^{2}$ by $g^{2}/2$. We use units such
that $m=1$ for any $\gamma$ in the following. The Dirac HF equation for the
massive isoNJL model is unchanged as compared to (5), but the self-consistency
conditions (6) now read

$\displaystyle S-m_{b}$ $\displaystyle=$
$\displaystyle-G^{2}\langle\bar{\psi}\psi\rangle,$ $\displaystyle\vec{P}$
$\displaystyle=$
$\displaystyle-G^{2}\langle\bar{\psi}i\gamma_{5}\vec{\tau}\psi\rangle.$ (10)

We assume once again that the charged components of $\vec{P}$ vanish. Then the
reasoning used in the chiral limit goes through literally, yielding a
Hamiltonian diagonal in isospin space and Eq. (5) for up- and down quarks. Let
us start again by looking at the special cases where one of the chemical
potentials vanishes. The problem then reduces to the single-flavor GN
($\nu=0$) and NJL ($\mu=0$) phase diagrams, now for the massive models. Here,
only two phases exist, a massive homogeneous one and an inhomogeneous one with
spatially periodic order parameter, see Fig. 3. The massless phase is
forbidden since chiral symmetry is explicitly broken by the bare mass. On the
NJL side, the “horizontal”, solid line is a perturbative, 2nd order phase
boundary determined via a stability analysis. The “vertical”, dotted curve is
a first order phase boundary inferred from a numerical HF calculation. The
homogeneous solution and a periodic inhomogeneous solution with finite
amplitude coexist along this line. These two curves meet at a tricritical
point indicated by a dot. Originally, this point had been found by pushing the
numerical HF calculation towards the endpoint of the first order line (“bottom
up” approach L12 ). More recently, a new method based on next-to-leading-order
(NLO) perturbation theory has been devised to find the exact position of the
tricritical point from the perturbative side (“top down” approach L13 ),
superseding the earlier numerical result. On the GN side, the horizontal part
also belongs to a 2nd order phase transition accessible via a stability
analysis. Upon crossing it, the system becomes unstable against the creation
of a periodic structure of infinitesimal amplitude. The vertical part is non-
perturbative. Here, the system is unstable against formation of a single
baryon. At the cusp, there is a tricritical point and the wave number of the
inhomogeneous phase vanishes. The massive GN results have been obtained in an
analytical way L10 . Let us also mention that the base points of the phase
boundaries at $T=0$ are located at the masses of the most strongly bound
baryons in both models. Their values are known from independent works,
analytically in the GN model L14  and numerically in the NJL model L15 .

Due to the different character of the phase diagrams on the boundaries $\nu=0$
and $\mu=0$, it is interesting to study how the massive isoNJL model will
manage to interpolate between the two graphs if both chemical potentials are
non-vanishing. Fig. 3 immediately shows that the interpolating phase boundary
sheet must depend non-trivially on all three coordinates, in contrast to the
chiral limit of Fig. 2. An inspection of Fig. 3 suggests to split the problem
into four distinct questions: How are the horizontal, perturbative phase
boundaries connected? What is the curve in the $T=0$ plane, connecting the two
base points and separating homogeneous from inhomogeneous phases at zero
temperature? What is the shape of the phase boundary sheet connecting the two
vertical, non-perturbative curves? And, finally, what is the shape of the
tricritical curve connecting the two tricritical points? The first three
questions have already been answered L3 ; L12 . We illustrate the solution in
subsections II.1–II.3, using as example the case $\gamma=0.1$. The last
question is the main topic of the present work and will be covered in Sects.
III and IV.

Figure 3: Boundary phase diagram of massive isoNJL model ($\gamma=0.1$) if one
of the chemical potentials vanishes. At $\nu=0$, same as massive GN model. At
$\mu=0$, same as massive NJL model with chemical potential $\nu$.

###  II.1 Perturbative sheet and stability analysis

If we set $P_{1}=P_{2}=0$, the grand canonical potential of the isoNJL model
can be written as a sum over two NJL model expressions,

$\displaystyle\Psi_{\rm isoNJL}(\mu,\nu,T,S,P_{3})$ $\displaystyle=$
$\displaystyle\Psi_{\rm NJL}(\mu+\nu,T,S-iP_{3})$ $\displaystyle+$
$\displaystyle\Psi_{\rm NJL}(\mu-\nu,T,S+iP_{3}).$

This equation holds provided we use $G^{2}=g^{2}/2$ and the same $\gamma$
parameter on both sides. Solving the isoNJL model is therefore closely related
to solving the NJL model. We start with the easiest part of the phase diagram,
the perturbative 2nd order phase boundaries separating the homogeneous from
the inhomogeneous phases. The stability analysis consists in the following
steps: perturb the massive Dirac Hamiltonian by a harmonic potential of the
form

$V=\gamma^{0}2S_{1}\cos(2Qx)-i\gamma^{1}2P_{1}\sin(2Qx).$ (12)

Evaluate analytically the shift of the single particle energies in 2nd order
perturbation theory. Expand the grand canonical potential to leading order in
the correction. This still has to be minimized with respect to
$m,Q,S_{1},P_{1}$. At the phase boundary, $m$ is the same as the fermion mass
in the homogeneous phase. Due to (II.1), many formulas can be taken over from
the NJL model. The minimizations amount to setting the Hessian determinant and
its derivative with respect to $Q$ equal to 0,

${\rm det}{\cal M}=0,\quad\partial_{Q}{\rm det}{\cal M}=0,$ (13)

with ${\cal M}$ the Hessian matrix

${\cal
M}=\left(\begin{array}[]{cc}\partial_{S_{1}}^{2}\Psi&\partial_{S_{1}}\partial_{P_{1}}\Psi\\\
\partial_{P_{1}}\partial_{S_{1}}\Psi&\partial_{P_{1}}^{2}\Psi\end{array}\right).$
(14)

In such a leading order (LO) stability analysis, not all of the parameters can
be determined. Aside from the critical temperature as a function of $\mu,\nu$,
one can extract the ratio $R=S_{1}/P_{1}$ and the wave number $Q$, but not the
overall strength of the perturbation. $R$ and $Q$ characterize the unstable
mode. They are not immediately relevant for the location of the phase
boundary, but will play a role for finding the tricritical point.

Although technically quite simple, such a stability analysis gives already a
fairly good impression of the full phase diagram. It is available for a number
of $\gamma$ values L3 . In the example at hand ($\gamma=0.1$), the result is
shown in Fig. (4), smoothly interpolating between GN and NJL perturbative
critical curves. As expected from the boundary phase diagrams at $\mu=0$ and
$\nu=0$, this calculation leaves open the details of the homogeneous “wound”
around $\mu=0,\nu=0$. This will be the subject of the upcoming subsections and
sections.

Figure 4: Adding the perturbative 2nd order sheet to the phase diagram of Fig.
3. The surface is the result of a LO stability analysis L3  and interpolates
between the horizontal, perturbative 2nd order phase boundaries of GN and NJL
model at $\nu=0$ and $\mu=0$, respectively.

###  II.2 Non-perturbative phase boundary in the $T=0$ plane and baryon masses

Figure 5: Adding the non-perturbative phase boundary in the $T=0$ plane to the
phase diagram of Fig. 3. As shown in Ref. L12 , it has the shape of the first
quadrant of an octogon and is fully determined by three baryon masses of the
isoNJL model, see main text.

The next step is to find the phase boundary at $T=0$, connecting the base
points of the GN and NJL phase diagrams. These latter are given by the masses
of the most strongly bound baryons of the two models and are known
analytically ($M_{\rm GN}$) or numerically ($M_{\rm NJL}$) in the massive
models, as a function of $\gamma$. Since these base points are now also part
of the isoNJL phase diagram at $T=0$, the two-flavor isoNJL model must possess
baryons with the same masses. As shown in Ref. L12 , $M_{\rm GN}$ is the mass
of the isoNJL baryon with maximal baryon number and zero isospin, consisting
of $N_{c}$ up quarks and $N_{c}$ down quarks. $M_{\rm NJL}$ is the mass of the
baryon with zero baryon number and maximal isospin made out of $N_{c}$ up
quarks and $N_{c}$ down antiquarks. Many other multifermion bound states with
different baryon number and isospin are expected to also play a role along the
phase boundary in the $T=0$ plane. This has been investigated in Ref. L12 .
Somewhat surprisingly, apart from $M_{\rm GN}$ and $M_{\rm NJL}$, only one
other baryon mass enters into the construction of the phase boundary: the mass
$M_{\rm up}$ of the baryon with half maximal baryon number and half maximal
isospin, consisting solely of $N_{c}$ up quarks. The result for the phase
boundary is the first quadrant of an octogon with vertices at the points
$(\mu,\nu)=(M_{\rm GN},2M_{\rm up}-M_{\rm GN})$ and $(2M_{\rm up}-M_{\rm
NJL},M_{\rm NJL})$, see Fig. 5. It can be made up by intersecting the three
lines

$\nu=M_{\rm NJL},\quad\mu=M_{\rm GN},\quad\mu+\nu=2M_{\rm up},$ (15)

where $M_{\rm NJL}=0.3853$, $M_{\rm GN}=0.7240$, $M_{\rm up}=0.4129$ at
$\gamma=0.1$. Actually, the octogon shape results from constructing the
envelope of a whole family of straight lines and seems to be universal for all
$\gamma$ values. In order to construct this family and the envelope, the
masses of all possible baryons had to be computed numerically in HF, even
though only three masses are needed eventually.

###  II.3 First order sheet at finite $T$ and summary

Figure 6: Adding the first order sheet at $T>0$ (crosses) to the phase diagram
of Fig. 3. It interpolates between the vertical, non-perturbative phase
boundaries of the GN ($\nu=0$) and NJL ($\mu=0$) models. From numerical HF
calculation L12 .

The most tedious part of the calculation is the non-perturbative sheet
interpolating between the vertical, non-perturbative phase boundaries of GN
and NJL models. It requires a full numerical HF calculation. One has to
evaluate the grand canonical potential for the mean fields $S(x),P_{3}(x)$.
These are assumed to be periodic with wave number $q$ and parameterized in
terms of their Fourier components $S_{\ell},P_{\ell}$. For a given point
($\mu,\nu,T$), one has to minimize the effective potential with respect to all
the $S_{\ell},P_{\ell}$ and $q$. One chooses a trajectory across the
anticipated phase boundary and compares the result with the homogeneous
solution. If one finds two curves intersecting at a point along the
trajectory, this point belongs to the first order phase boundary sheet. The
result for $\gamma=0.1$ from Ref. L12  is shown in Fig. 6 (crosses). It
interpolates between the non-perturbative curves on the boundary and also
matches nicely onto the octogon shape of the $T=0$ phase boundary. The
technique used here is identical to the one developed previously for the
massive NJL model, so that we do not go into further details. All points shown
were clearly identified as 1st order transitions.

Finally, we put all ingredients discussed so far together in one plot, Fig. 7.
This summarizes the state of the art of the massive isoNJL phase diagram at
present, for the example of $\gamma=0.1$. We find a consistent picture,
supporting our assumption that $\vec{P}_{\perp}=0$. All the different pieces
based on independent calculations and a variety of techniques fit together
very well, like the pieces of a a jigsaw puzzle. The most glaring deficit is
probably the fact that we do not know yet how to interpolate between the
tricritical points. Hence the line separating the first and second order
sheets remains poorly defined. As mentioned in Ref. L12 , numerical HF
calculations in this region were not precise enough, a difficulty first
encountered in the massive NJL model. Recently, a better method has been
proposed, tailored to the tricritical points and in principle exact L13 . This
will be reviewed and applied to the case at hand, the massive isoNJL model, in
the next two sections.

Figure 7: Present status of the massive isoNJL phase diagram at $\gamma=0.1$,
constructed by putting together the ingredients shown in Figs. 3–6. A
tricritical curve connecting the tricritical points of the GN and NJL models
on the boundaries is still missing. From L12 .

##  III Precise determination of the tricritical curve

Figure 8: Proof that the NLO stability analysis developed for the NJL model is
also capable of locating the tricritical point of the massive GN model. The
example $\gamma=0.3$ is shown. The 2nd derivative of the effective potential
could be followed as a function of $\mu$ down to $\mu=0.8$ (circles).
Extrapolation to 0 yields the value $\mu=0.799$, in excellent agreement with
the known tricritical point.

Recently, a novel method of locating the tricritical point has been devised
and tested successfully in the massive NJL model L13 . The basic idea is to
start from the stability analysis, but pushing perturbation theory to NLO (4th
order in $S_{1},P_{1}$). In the case of the single-flavor NJL model, the
procedure may be summarized as follows:

  1. 1.

Find a point on the perturbative sheet close to where the tricritical point is
expected via a LO stability analysis. Find the coordinates ($\mu,\nu,T$)
together with the ratio $R=S_{1}/P_{1}$ and the wave number $Q$ of the
unstable mode.

  2. 2.

Do a 4th order perturbative evaluation, first of the HF single particle
energies and then of the thermodynamic potential, keeping $R,Q$ fixed at the
LO values. The remaining parameters are $m,P_{1}$.

  3. 3.

Choose 3 masses $m_{0},m_{0}\pm\Delta m$ with $m_{0}$ the mass of the
homogeneous solution at point ($\mu,\nu,T$) on the perturbative sheet and
$\Delta m\ll m_{0}$. Minimize the grand canonical potential for each mass
value with respect to $P_{1}$. The resulting effective potential is a function
of $m$ only,

$\widetilde{\Psi}(m)=\min_{P_{1}}\Psi(m,P_{1}).$ (16)

  4. 4.

The criterion for the tricritical point is the vanishing of the 2nd derivative
of the effective potential,
$\partial_{m}^{2}\left.\widetilde{\Psi}\right|_{m_{0}}$ or, in discretized
form,

$\frac{\widetilde{\Psi}(m_{0}+\Delta
m)-2\widetilde{\Psi}(m_{0})+\widetilde{\Psi}(m_{0}-\Delta m)}{\Delta
m^{2}}=0.$ (17)

The main technical difficulty which had to be overcome is the fact that
perturbation theory breaks down near the spectral gaps of a periodic
potential. This becomes more serious in higher order PT where the usual LO
“almost degenerate perturbation theory” (ADPT) cannot be applied any more. The
way out is to go via an effective Hamiltonian in the space of strongly mixed
states and diagonalize it exactly, using a formalism from many-body
perturbation theory due to Lindgren L16 . The method has been shown to work
very well for the massive NJL model and is supposedly exact, just like the LO
stability analysis for the perturbative sheet.

When applying this method to the isoNJL model, all we have to do is use Eq.
(II.1) and reduce the technicalities to those of the NJL model. One question
which then arises is: Can we be sure that the criterion developed for the
massive NJL model also applies to the isoNJL model? The nature of the two
tricritical points on the boundaries $\nu=0$ (GN) and $\mu=0$ (NJL) seems to
be very different at first glance. In the NJL model, the two phase boundaries
adjacent to the tricritical point meet tangentially under $0^{\circ}$, whereas
the corresponding angle is $180^{\circ}$ at the cusp of the GN model. On the
other hand, since we have the representation (II.1) of the isoNJL model as a
sum of two NJL models, it is plausible that the methods for finding the
tricritical points in both models are closely related. As an additional check,
we have applied the method of Ref. L13  to the GN model. Since $P_{1}=0$,
there are minor changes in the stability analysis. The unstable mode is only
characterized by $Q$, there is no ratio $R$. In the 4th order perturbative
calculation, one can use the machinery developed in Ref. L13  by simply
setting $P_{1}=0$. We have evaluated the 2nd derivative (17) for several $\mu$
values near the tricritical point. The result is shown in Fig. 8 for
$\gamma=0.3$. We have not quite succeeded in reaching the point where the 2nd
derivative vanishes. However, as the figure shows, we can come very close to
it. A linear extrapolation from the two last points then yields the value
$\mu=0.799$, in excellent agreement with the known GN tricritical point at the
same $\gamma$ ($\mu=0.7986$). The same kind of agreement was reached at other
values of $\gamma$. Apparently, the tricritical point in the GN model lies
right at the edge of the perturbative sheet, but this is not expected for any
other point on the tricritical curve. Thus we are confident that the
perturbative method of locating the tricritical point can also be trusted in
the isoNJL model. The results will be presented in the following section.

##  IV Results and discussion

Figure 9: Final phase diagram for massive isoNJL model at $\gamma=0.1$. The
tricritical curve has been added to the previous phase diagram Fig. 7, now
delimiting more clearly second order and first order sheets of the phase
boundary between homogeneous and inhomogeneous phases. Figure 10: Projection
of tricritical curve onto the $T=0$ plane, for $\gamma=0.1$. The circles are
the points actually computed, the fat curve is just an interpolation which has
been used for Fig. 9. The thin polygon is the phase boundary at $T=0$ and
$\gamma=0.1$ discussed in Sect. II.2. Relevant baryon masses: $M_{\rm
GN}=0.7240,M_{\rm NJL}=0.3853,M_{\rm up}=0.4129$. Figure 11: Same as Fig. 9,
but for $\gamma=0.3$ Figure 12: Same as Fig. 10, but for $\gamma=0.3$.
Relevant baryon masses: $M_{\rm GN}=0.8041,M_{\rm NJL}=0.6142,M_{\rm
up}=0.4546$.

We have computed the tricritical curve still missing in the phase diagram Fig.
7 for the massive isoNJL at $\gamma=0.1$. The method has been sketched above
and explained in more detail in Ref. L13 , where it was applied to the massive
NJL model. The main complication of the two-flavor model is the fact that we
now have to determine a full curve of unknown shape and location, except for
the endpoints. This means that we have to repeat the NLO perturbative
calculation many times, each calculation being basically the same as for the
one-flavor NJL model. Our result for $\gamma=0.1$ is shown in Fig. 9. The
tricritical curve thus obtained is fully compatible with the other building
blocks of the phase diagram and enables us to delimit the inhomogeneous region
of the phase diagram in a quantitative way.

Figure 13: Same as Fig. 9, but for $\gamma=0.5$ Figure 14: Same as Fig. 12,
but for $\gamma=0.5$. Relevant baryon masses: $M_{\rm GN}=0.8501,M_{\rm
NJL}=0.7334,M_{\rm up}=0.4710$.

In order to exhibit the shape of the tricritical curve in greater detail, we
propose to look at its projection onto the $T=0$ plane. The result is shown in
Fig. 10 for $\gamma=0.1$. We have marked the points which have actually been
calculated by circles. The fat solid line is an interpolating curve and
corresponds to the tricritical curve shown in the 3d Fig. 9. Due to the
complicated shape of the tricritical curve, a fairly large number of points
was needed. We have also included the $T=0$ phase boundary into Fig. 10, the
thin polygon. The tricritical curve and the base curve are obviously
correlated. At $T=0$, we know that the instability along the three straight
line segments is determined by three different baryons. Correspondingly, we
shall refer to the three segments as GN (near $\nu=0$), up (tilted) and NJL
(near $\mu=0$) parts. A similar division can be made for the first order sheet
and the tricritical line. The GN and NJL parts are still strongly influenced
by the boundary phase diagrams. In these regions, the temperature drops to a
common level. The up region seems to be the one with the simplest features.
The projected tricritical curve follows the slope of the line $\mu+\nu={\rm
const.}$, the first order sheet is steep and essentially planar and the
temperature is slowly varying. This is perhaps the most interesting part,
since it has no correspondence in the one-flavor phase diagrams. As speculated
in Ref. L13 , it is quite likely that there are additional phase boundaries
inside the crystal phase, starting from the vertices of the $T=0$ octogon and
going inside the crystal region. If this is true, the question arises what
happens if we go up in temperature. If the internal phase boundary would
persist all the way through the inhomogeneous phase, there should be phase
boundaries in addition to the tricritical curve discussed so far. We have
searched for this with our method of finding tricritical points, but without
success. This indicates that if such internal phase boundaries really exist at
$T=0$, they disappear at some temperature below the perturbative sheet.

Figure 15: Projection of tricritical curves onto the $\nu=0$ plane. From top
to bottom: $\gamma=0.1,0.3,0.5$. Figure 16: Projection of tricritical curves
onto the $\mu=0$ plane. From top to bottom: $\gamma=0.1,0.3,0.5$.

In Ref. L13 , the non-perturbative sheet of the massive isoNJL phase diagram
has been evaluated at three values of $\gamma$, 0.1, 0.2, and 0.3. Therefore
we have also repeated the calculation of the tricritical curve for
$\gamma=0.2$ and $\gamma=0.3$ to see the evolution with increasing bare
fermion mass. The 3d plots are shown in Figs. 11 and 13, along with the
projections onto the $T=0$ plane and the $T=0$ phase boundaries in Figs. 12
and 14. Qualitatively, the three phase diagrams and tricritical curves at the
three different $\gamma$ values look very similar. As is particularly striking
in the projections, the up part becomes more dominant with increasing $\gamma$
and the simpler features of that part of the phase boundary show up more
clearly. Thus, at the highest $\gamma$, the $T=0$ line and the projection of
the tricritical curve are almost indistinguishable and account for a large
part of the phase boundary, see Fig. 14. The strong variation of the projected
tricritical curve near the largest $\nu$ values (the “nose”) is a reflection
of the part of the curve which rises steeply in temperature, as can be
inferred from the 3d plot, Fig. 13.

As mentioned above, the up parts of the tricritical curves have essentially
constant temperature, and at the same time the lowest one along the whole
tricritical curve. In order to exhibit this effect more clearly, we have also
plotted projections of the three tricritical curves onto the ($\mu,T$) and
($\nu,T$) planes, see Figs. 15 and 16. Common to all bare masses is the
observation that the temperature decreases in the GN section, stays constant
through the up section and increases more strongly in the NJL section. This
reinforces the impression that the physics in the up section should be most
easily accessible, perhaps by using some heavy quark approximation for large
$\gamma$.

## References

  * (1) D. J. Gross and A. Neveu, Phys. Rev. D 10, 3235 (1974).
  * (2) Y. Nambu and G. Jona-Lasinio, Phys. Rev. 124, 246 (1961).
  * (3) M. Thies, Phys. Rev. D 101, 014010 (2020).
  * (4) A. Heinz, F. Giacosa, M. Wagner, D. H. Rischke, Phys. Rev. D 93, 014007 (2016).
  * (5) O. Schnetz, M. Thies, and K. Urlichs, Ann. of Phys. 314, 425 (2004).
  * (6) V. Schön and M. Thies, At the Frontier of Particle Physics: Handbook of QCD, Boris Ioffe Festschrift, vol. 3, ed. M. Shifman (Singapore: World Scientific), ch. 33, p. 1945 (2001).
  * (7) G. Basar, G. V. Dunne, and M. Thies, Phys. Rev. D 79, 105012 (2009).
  * (8) T. G. Khunjua, K. G. Klimenko, R. N. Zhokhov, V. C. Zhukovsky, Phys. Rev. D 95, 105010 (2017).
  * (9) T. G. Khunjua, K. G. Klimenko, and R. N. Zhokhov, Phys. Rev. D 100, 034009 (2019).
  * (10) O. Schnetz, M. Thies, and K. Urlichs, Ann. of Phys. 321, 2604 (2006).
  * (11) C. Boehmer, U. Fritsch, S. Kraus, M. Thies, Phys. Rev. D 78, 065043 (2008).
  * (12) M. Thies, Phys. Rev. D 101, 074013 (2020).
  * (13) M. Thies, Phys. Rev. D 105, 116003 (2022).
  * (14) M. Thies and K. Urlichs, Phys. Rev. D 71, 105008 (2005).
  * (15) C. Boehmer, F. Karbstein, and M. Thies, Phys. Rev. D 77, 125031 (2008).
  * (16) I. Lindgren, J. Phys. B: Atom. Molec. Phys. 7, 2441 (1974).


# Model-based versus Model-free Deep Reinforcement Learning for Autonomous
Racing Cars

Axel Brunnbauer1∗, Luigi Berducci1∗, Andreas Brandstätter1∗, Mathias Lechner2,
Ramin Hasani3a, Daniela Rus3, Radu Grosu1 * Indicates authors with equal
contributions1 CPS, Technische Universität Wien (TU Wien), Austria2 Institute
of Science and Technology Austria (IST Austria)3 CSAIL, Massachusetts
Institute of Technology (MIT), MA, USAa Correspondence to Ramin Hasani:
<EMAIL_ADDRESS>
###### Abstract

Despite the rich theoretical foundation of model-based deep reinforcement
learning (RL) agents, their effectiveness in real-world robotics-applications
is less studied and understood. In this paper we therefore investigate how
such agents generalize to real-world autonomous-vehicle control-tasks, where
advanced model-free deep RL algorithms fail. In particular, we set up a series
of time-lap tasks for an F1TENTH racing robot, equipped with high-dimensional
LiDAR sensors, on a set of test tracks with a gradual increase in their
complexity. In this continuous-control setting, we show that model-based
agents capable of learning in imagination, substantially outperform model-free
agents with respect to performance, sample efficiency, successful task
completion, and generalization. Moreover, we show that the generalization
ability of model-based agents strongly depends on the observation-model
choice. Finally, we provide extensive empirical evidence for the effectiveness
of model-based agents provided with long enough memory horizons in sim2real
tasks.

##  I Introduction

Deploying deep reinforcement-learning (RL) agents in the real-world is
difficult. This is because they require running a significantly large amount
of episodes to obtain reasonable performance [1]. This performance is only
tractable in simulation environments. Subsequently, the agents should also
overcome the challenges of transferring learned dynamics from simulation to
the real world.

In RL settings, it was shown that if one learns a representation of the state-
space model from high-dimensional input observations and use it as a
predictive model to train policies in imagination, one gains performance,
sample efficiency, and robustness [2]. This world-model algorithm [3] was
called Dreamer, and it demonstrated performance in learning long-horizon
visual control tasks by imagination.

Providing a model of the world (state transition probability) to the agents
(even though in simulation) improves sample efficiency and, in some cases, the
performance of a deep RL agent [3, 4, 2]. However, there are still open
research questions to be investigated: For instance, how would world-model
compartments improve the sim2real performance of RL agents in real-world
applications? Where is the boundary between the generalization capability of
model-based algorithms such as Dreamer, compared to advanced model-free agents
such as soft actor-critic [5], distributional models [6], and policy gradient
[7, 8], in sim2real applications?

In this paper, we aim to provide answers to the above questions. We construct
a series of time-lap autonomous racing tasks with varying degrees of
complexity for an F1TENTH robot. The task is to learn to drive autonomously
directly from high-dimensional LiDAR inputs to finish a lap without collisions
successfully.

We observe that as the complexity of the map increases, model-free agents get
stuck in similar local minima and cannot learn to complete the task.
Contrarily, Dreamer can learn a proper state transition model, and given
enough imagination horizon, solves time-lap tasks of arbitrary complexity (See
Fig. 1). Moreover, we discover that model-based techniques demonstrate
desirable transferability: Dreamer agents trained on a single map can
generalize well to test tracks they have never seen before. In summary, our
main contributions are as follows:

  * •

We demonstrate the effectiveness of advanced model-based deep RL compared to
model-free agents in the real-world application of autonomous racing.

  * •

We show the transferability of advanced model-based deep RL agents to the
real-world applications where model-free agents fail.

  * •

We empirically show that the learning performance and generalization ability
of Dreamer in sim2real applications highly depends on the choice of the
observation model, its resulting state transition probability model, and the
imagination horizon.

Figure 1: Model-based deep-RL (Dreamer) solves autonomous racing in complex
tracks where all advanced model-free methods fail.

##  II Problem Definition

We set out to design deep RL models that are able to learn to autonomously
complete time-lap racing tasks. Considering the real-world conditions, we
formalize the problem as a Partially-Observable Markov Decision Process
(POMDP).

Notation and terminology. A POMDP is defined as a tuple
$(S,A,\Omega,\mathcal{O},\mathcal{T},\mathcal{R})$, where $S,A,\Omega$ are the
sets of states, actions and observations, respectively. $\mathcal{O}$ and
$\mathcal{T}$ denote stochastic observation and transition functions, and
$\mathcal{R}$ is a deterministic reward function. The transition function
$\mathcal{T}$ models the system dynamics and includes its uncertainty. It is
defined as a stochastic function $\mathcal{T}:S\times A\times
S\rightarrow[0,1]$ which returns the transition probability between two states
by applying actions. The observation function $\mathcal{O}$ models the
system’s perception, including its uncertainty. Its stochastic formulation
$\mathcal{O}:S\times\Omega\rightarrow[0,1]$ returns the probability of
perceiving an observation in a given state. The reward function is
deterministic $\mathcal{R}:S\times A\times S\rightarrow\mathbb{R}$, which
returns the credit assigned to a transition. We discuss our reward shaping
subsequently.

Racing-agent setup. In autonomous racing, the car drives along a race track by
controlling its actuators with continuous actions
$\boldsymbol{a}=(F,\alpha)^{T}$, where $F$ is the motor force applied to reach
a fixed target velocity, and $\alpha$ is the steering angle. In this scenario,
the observations are range measurements obtained by a LiDAR sensor with 1080
range measurements evenly distributed over a 270° field of view (see Fig. 2).

How do we aim to solve this racing objective? Traditional control techniques
model the state space as a set of continuous variables such as pose, velocity,
and acceleration of the car. These quantities are usually estimated from
observations by dedicated perception and filtering modules. State-space
planning commonly uses sophisticated dynamics models for cars. This demands
non-trivial system identification techniques. Contrarily, we aim to replace
the entirety of such modules by learning an abstract state representation and
transition model in a self-supervised manner by building upon recent advances
of model-based RL [2].

Figure 2: Racing-agent setup: observations, actions and reward.

##  III Related Work

We include works that are closely related to our contributions.

Model-free RL in robot control. Robot control has been a playground for RL
algorithm for decades [1, 9]. Model-free algorithms [7, 5, 6] achieved state-
of-the-art results in various continuous control tasks, but their inherent
sample inefficiency [10] makes it challenging to apply them in real-world
settings. Off-policy [11, 5] methods reduce the sample complexity by reusing
old experience to some degree. However, they might induce errors when deployed
in closed-loop with the real-world environment. Thus, model-free approaches
are often trained in simulation before being deployed on real robots. This
leads to difficulties when reproducing learned behaviors in the real setting
[12, 13]. To overcome simulation mismatches, some approaches rely on domain
randomization and subsequent fine-tuning. In order to learn more robust
policies [14], other works refined the physics simulator [15, 16] or directly
trained the policy on real-world robots [17, 18].

Model-based RL in robot control. Model-based approaches, on the other hand,
can leverage their learned dynamics model by either planning or generating new
training experience [19]. Problems arise when no sufficiently accurate
dynamics model can be learned from data. An early representative of world-
model algorithms is PILCO [20] which learned a Gaussian process from high-
dimensional states. Recent works on world-models [3, 4, 2, 21] proved the
feasibility of learning a dynamics model for POMDP’s by using noisy
observations instead of accurate states. They achieve better than state-of-
the-art results on various simulated control tasks while being significantly
more sample efficient. However, to the best of our knowledge, their
application and challenges in real-world testbeds have been less attended,
which we will explore in the present work.

Optimal control for autonomous racing. Autonomous racing has been an active
field of research in the control community. Traditional control does not solve
the problem in an end-to-end fashion but divides it into independent sub-
problems: perception, planning, and control. Recent success in the context of
Formula Student [22, 23, 24] has been achieved by Model Predictive Control
(MPC) and by engineering the perception pipeline using sensor fusion of LiDAR
and RGB cameras. Several approaches plan an optimal trajectory [25, 26, 27,
28]. However, these techniques usually require detailed global information
(e.g., map) and an accurate dynamics model, not available in a POMDP setting,
as in our case. Here, we use a fully automated process that learns an
approximated dynamics model in an unsupervised fashion by only having access
to LiDAR observations.

RL for autonomous racing. A large body of research made use of camera images
and adopt model-free methods [29, 30, 31]. In [32], the authors used SAC to
control a racing car in simulation by feeding the controller with state
information and ad-hoc features about the road curvature. To deal with the
complexity of continuous action spaces, a common technique is to discretize
the domain, but this approach is not scalable. Another recurrent trend
consists of combining MPC and deep RL [33, 34]. However, MPC requires to
efficiently perform sampling through the model to scale with respect to the
time constraints. Conversely, the adoption of a policy network is more
efficient and suitable for an online setting.

What about Imitation Learning? – Imitation learning consist of the process of
learning observation to action mapping from supervised data [35]. Imitation
learning allows for behavior cloning by data aggregation (Dagger) [36] via
supervised learning modes or by inverse RL [37]. Imitation learning has been
further adapted to imperfect [38] and incomplete demonstrations [39].

In the context of end-to-end control of autonomous vehicles, learning from
expert demonstrations is the dominant choice of algorithm [40, 41], because
other RL methods would require efficient simulation platforms to achieve
desirable performance [42]. For autonomous racing, as we already have a sample
efficient simulator available, we settle to use model-based RL algorithms over
imitation learning.

##  IV Formulating Dreamer for Autonomous Racing

In this section, we discuss the main considerations we had to address in order
to design a dreamer architecture [2] for autonomous racing. For this purpose,
we start by a brief recap of the Dreamer algorithm. We then present two
observation models that we utilized to learn the latent-dynamics model.
Finally, we formulate our specialized reward function.

Introduction. Dreamer is a model-based deep-RL algorithm that has recently
shown very impressive performance in simulated standard RL benchmarks [2].

Dreamer learns an accurate model of the dynamics from high-dimensional
observations in an unsupervised fashion and uses it to produce latent-state
sequences to train an agent using an actor-critic algorithm [2]. Dreamer
implements the dynamics model as a recurrent state-space model (RSSM) with
both stochastic and deterministic components [4].

The dynamics-model consists of the four distributions below, where $p$
highlights the interaction with the environment, $q$ the use in latent
imagination, too. All are implemented by deep neural networks [2]:

Representation model: $\displaystyle p_{\theta}(s_{t}|s_{t-1},a_{t-1},o_{t})$
(1) Observation model: $\displaystyle q_{\theta}(o_{t}|s_{t})$ (2) Reward
model: $\displaystyle q_{\theta}(r_{t}|s_{t})$ (3) Transition model:
$\displaystyle q_{\theta}(s_{t}|s_{t-1},a_{t-1})$ (4)

All components are jointly optimized to maximize the variational lower-bound
as follows, where $D_{\mathrm{KL}}\left(\left.\left.P\right|\right|Q\right)$
is the Kullback–Leibler divergence of distributions $P$ and $Q$:

$\displaystyle\mathcal{J}^{t}_{\mathcal{O}}=\ln q(o_{t}|s_{t})$
$\displaystyle\mathcal{J}^{t}_{\mathcal{R}}=\ln q(r_{t}|s_{t})$
$\displaystyle\mathcal{J}^{t}_{\mathcal{D}}=-\beta
D_{\mathrm{KL}}\left(\left.\left.p(s_{t}|s_{t-1},a_{t-1},o_{t})\right|\right|q(s_{t}|s_{t-1},a_{t-1})\right)$
$\displaystyle\mathcal{J_{REC}}=\mathbb{E}_{p}\left[\sum_{t}\mathcal{J}^{t}_{\mathcal{O}}+\mathcal{J}^{t}_{\mathcal{R}}+\mathcal{J}^{t}_{\mathcal{D}}\right]$

In contrast to [2], our representation model uses a multilayer perceptron
(MLP) to encode the observations, instead of convolutional layers. The reason
is that LiDAR observations have $1080$ dimensions, and further compression
does not show better performance in our empirical evaluation.

Observation model. This model determines how the observations are represented
by the RL algorithm. It is a Dreamer component that facilitates learning a
good latent representation. The model is used only in the training phase to
provide learning signals to the representation model. In this work, we first
propose two observation models: one based on reconstructing distance
measurements, and the other is based on reconstructing occupancy maps,
respectively. We then highlight their importance in the generalization
performance of a Dreamer agent.

_Distance reconstruction_ is formulated in terms of the distance measured by
each LiDAR ray at every time step. We call the resulting agent Dreamer-
Distance. In this case, the observation is both the input of the
representation-model and the output of the observation model. The observation
model is an MLP that computes the mean and standard deviation of a Gaussian
distribution for each ray.

_Occupancy reconstruction_ builds the surrounding occupancy map at each time
step. We call the resulting agent Dreamer-Occupancy. Specifically, the
reconstructed area is centered on the agent position and has a radius of 5
meters. The observation-model is a convolutional network that computes the
mean of a Bernoulli distribution. Each value is an indicator of the pixel
occupancy in the surrounding map. In Figure 3 we show a snapshot of the
simulation, the associated LiDAR observation, and the two reconstructions.

Note that the input of the representation model is slightly different from the
output of both observation-models.

Figure 3: Observation models and their reconstruction methods. Observations
are in agent coordinates. Row 1: bird-view of the racecar in simulation, row
2: LiDAR scan in racecar coordinates, row 3: reconstructed LiDAR scan, row 4:
reconstructed local occupancy map

Actor-critic on latent imagination. Having first learned a world model by
using complex observations, Dreamer then learns a policy purely on imagined
sequences [2]. This approach allows Dreamer to efficiently produce thousands
of training sequences, without requiring any direct interaction with the
environment. This results in safe and data-efficient learning. Starting from
these imagined sequences, Dreamer uses an actor-critic algorithm to train the
agent. The action model (policy) aims to synthesize the optimal action in each
latent state, and the value model (critic) estimates the value of each latent
state [2]:

Action model: $\displaystyle q_{\phi}(a_{t}|s_{t})$ (5) Value model:
$\displaystyle q_{\psi}(v_{t}|s_{t})$ (6)

Among the various techniques for approximating state values and balancing bias
and variance [1], Dreamer [2] uses an exponentially weighted sum over
different horizons. Moreover, the training process leverages the availability
of a learned dynamics model by back-propagating through it. This leads to more
informative gradients.

Reward shaping. The reward design (or shaping) is critical in determining the
quality of the obtained policy. The sparsity of rewards over long episodes
makes the learning problem challenging. In Figure 4 we show precomputed
gridmaps for each of our tracks. These gridmaps show the normalized distance,
which we interpret as progress, from the starting line in race direction.

Figure 4: Top-view of the race tracks and relative Progress Maps adopted for
reward design. Austria (length = $79.45m$), Columbia ($61.2m$), Treitlstrasse
($51.65m$) are used for training, Barcelona ($201m$) is used only for
evaluation.

The reward signal we propose is computed by

$c*|s_{t}-s_{t-1}|=c*\Delta s_{t}.$ (7)

In Equation 7, $s_{t}$ denotes the progress value at time $t$ and $c$ is a
constant scalar. In case of collision, the agent receives a penalty, and the
episode terminates (see also Figure 2).

##  V Experiments

Here, we describe our experimental setup and evaluate the performance of
model-free versus model-based methods.

###  V-A Experimental Setup

We train the models in simulation and transfer them to real cars. Here, we
provide an overview of the simulation and real-world setup as well as the
training process.

Simulation environment. To simulate a car model, we use the open-source
physics engine PyBullet [43]. The car model is a rigid body system based on
the URDF model in [44]. Our training environment can simulate a broad set of
sensory inputs, such as LiDAR sensors, RGB cameras, and odometry. The model is
actuated by applying force to the steering and acceleration joints. See
supplements for more details.

Training pipeline. During training in simulation, we place the agent randomly
on the track at the beginning of an episode. Each training episode has a
maximum length of $2\,000$ timesteps, resulting in 20 seconds of real-time
experience. Agents learn to directly maximize the progress covered in a small,
predefined time interval. Observations and actions are normalized. To account
for latency experienced during testing and to increase the effectiveness of
actions, we repeat each action multiple times. We regularly evaluate the agent
by placing it on a fixed starting position for each track and let it run for
at most $4\,000$ timesteps (i.e., 40 seconds) and average the maximum progress
reached over five consecutive trials. Dreamer is trained for 2 million
timesteps. The model-free baselines are trained for 8 million timesteps.

Hardware setup. The hardware platform for the race car is based on the F1TENTH
race series [45]. It consists of an off-the-shelf model race car chassis, with
a Traxxas Velineon 3351R brushless DC electric motor, which is driven by a
VESC 6 MkIV electronic speed controller (ESC). Computation and control tasks
are performed on an NVIDIA Jetson TX2. The environment is sensed by a Hokuyo
UST-10LX LiDAR sensor. It is running Ubuntu 18.04 as the base operating
system. Robot Operating System (ROS) [46] is used to provide an infrastructure
for sending and receiving messages and for the hardware abstraction. Our
dreamer agent is running inside a Docker container. A ROS interface node is
used to translate observation and action messages. The motor force commands
are processed by integration in order to get the desired speed values. For the
steering commands, we added an adaptive low pass filter in order to protect
the servo from high frequent steering operations.

Track difficulty. In order to compare the difficulty of the tracks, we
classified them according to several characteristics (as also discussed in
[47] and [48]). In particular, we measure the minimal track width, the track
length (shortest path), and its minimum curve radius. For curves, we measure
the radius of the largest circle that tangentially touches the outside of the
curve and also touches the inside of the curve. In Table I we summarize the
characteristics of the the tracks.

Figure 5: Learning curves of model-free methods (top row) over $8M$ steps and Dreamers (bottom row) over $2M$ steps. The dashed lines report the maximum performance obtained by the other algorithms as baselines. n=5 TABLE I: Track characteristics. Track | Min. Width | Length | Min. Radius
---|---|---|---
Austria | 1.86 m | 79.45 m | 2.78 m
Columbia | 3.53 m | 61.20 m | 7.68 m
Treitlstrasse | 0.89 m | 51.65 m | 3.55 m
Barcelona | 1.86 m | 201.00 m | 2.98 m

###  V-B Experimental Evaluation

In this section, we first introduce the baselines and then discuss in detail
the performance of the algorithms in various racing scenarios.

Model-free baselines. We compare the performance of the Dreamer agent against
the following advanced model-free baselines: D4PG, an enhanced version of DDPG
[6], MPO, a stable off-policy algorithm [8], SAC, an off-policy actor-critic
algorithm with less sensitivity to hyperparameters [17], PPO, an on-policy
algorithm [7], and PPO-LSTM, the recurrent version of PPO using long short-
term memory [49]. PPO-LSTM is chosen as a baseline since policies built by
recurrent neural network approximators demonstrate remarkable performance in
learning to control [50, 51, 52, 53].

We conduct three different experiments: I) Evaluation of the learning curves
of Dreamer and model-free algorithms in simulation, II) Evaluation of the
generalization ability by testing the trained models in simulation, and III)
Evaluation of the transferability on our real testing platform.

Learning performance and sample efficiency. In Figure 5 we give the learning
curves of the model-free and model-based algorithms in 3 different tracks.
Except for the simple track Columbia, all advanced model-free methods fail to
successfully perform one lap in more complex maps. On the other hand, the
Dreamer agents efficiently learn to complete the tasks regardless of the
degrees of complexity of the tracks.

As shown in Figure 5, the performance of the Dreamer agents equipped with
distance and occupancy reconstructions are comparable in Austria and Columbia.
However, the experiments on Treitlstrasse show better performance with
occupancy-map reconstruction, with which the agent can almost complete two
full laps over the evaluation-time window. This experiment suggests that the
occupancy-map reconstruction speeds up the training process. However, it
biases the learning process on the track on which it was trained. This affects
the policy’s generalizability, as illustrated in the next experiment.

What length of imagination horizon? Figure 6 shows that as we increase the
imagination horizon for a Dreamer agent, it learns to perform better. Our
experiments show that the horizon plays an important role in learning long-
range tasks.

Figure 6: Dreamer’s performance using different reconstructions for different
values of horizon. The batch length is fixed to $50$ and action repeat is $4$.
The figure reports the mean performance over 5 trials.

Performance on unseen tracks. In this experiment, we aim to evaluate the
cross-track generalizability of the learned policies and demonstrate the
domain-adaptation skills of Dreamer. We trained polices on complex tracks,
Austria (AUT) and Treitlstrasse (TRT) individually, and reported their test
performance on other unseen tracks: Columbia (COL) and Barcelona (BRC). We
compared the performance of the best performing model-free policy, MPO, to
Dreamer-Distance and Dreamer-Occupancy policies.

As shown in Figure 7 (top-left and top-right), while all algorithms can
complete the simple track, COL, only Dreamer can generalize the racing task
and transfer the learned skills to other complex tracks. Generally, policies
trained in AUT achieve better performance as they can complete at least one
lap in each other track.

Conversely, even if TRT contains very challenging turns, the trained agents
show bad performance on unseen complex tracks (AUT). The reason might be that
most turns in TRT have the same direction, and consequently, the trained
policy cannot generalize to altered scenarios.

Moreover, we observed that the Dreamer-Occupancy agent shows lower
generalization skills compared to Dreamer-Distance. The learned policy results
in a more aggressive strategy, presenting a lower lap-time as shown in Figure
7 (bottom-left and bottom-right). However, the car often gets dangerously
close to the track walls. This results in unsafe behavior that increases the
probability of collisions.

In conclusion, these observations suggest that reconstructing the occupancy
area would bias the agent towards learning the track on which it was trained.
This results in faster learning but induces a less robust behavior to changes
in the domain and worsens generalization.

Figure 7: Maximum progress and lap time of trained models over different
tracks in simulation. The bars show the result averaged over 10 episodes on
each track. The delimiters show the minimum and maximum achieved. For Lap-Time
results, we consider only the run completing one full lap.

Sim2Real transfer. In this experiment, we evaluate Dreamer policies with
respect to their sim2real transferability. We test trained dreamers deployed
on the car in a physical test-track. A video111The video can be viewed at:
https://youtu.be/IlN3vJxC30w
doi: 10.5281/zenodo.4577412 demonstrating the driving performance of the
Dreamer agent is supplied with the submission. It presents the Dreamer agent
completing a full lap in TRT in the forward direction. Then, to evaluate its
generalization, we tested the agent in reversed direction. We observe that
even if the agent was not trained in the reversed direction and not directly
on the TRT track, the agent can complete a successful lap. Finally, we placed
two obstacles after the most challenging turn and ran the agent in this
configuration. The Dreamer agents can complete the lap and securely avoid
obstacles.

##  VI Conclusions

We showed that Dreamer, a model-based deep RL algorithm, performs
significantly better in driving autonomous race cars in complex tracks
compared to advanced model-free agents. We empirically show that increasing an
agent’s imagination horizon accounts for this discrepancy. We showed how
observation models affect generalization and domain adaptation of learned
policies, and that Dreamer agents can enable robust autonomy in real-world
settings.222All code, data and appendix are available at:
https://github.com/CPS-TUWien/racing_dreamer 333All Experimental logs are
archived at: doi: 10.5281/zenodo.4588599

## Acknowledgments

L.B. was supported by the Doctoral College Resilient Embedded Systems. M.L.
was supported in part by the Austrian Science Fund (FWF) under grant Z211-N23
(Wittgenstein Award). R.H. and D.R. are supported by Boeing. R.G. was
partially supported by the Horizon-2020 ECSEL Project grant No. 783163
(iDev40) and A.B. by FFG Project ADEX.

## References

  * [1] R. S. Sutton and A. G. Barto, _RL: An introduction_. MIT press, 2018.
  * [2] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi, “Dream to control: Learning behaviors by latent imagination,” _arXiv:1912.01603_ , 2019.
  * [3] D. Ha and J. Schmidhuber, “Recurrent world models facilitate policy evolution,” in _Advances in Neural Information Processing Systems_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31. Curran Associates, Inc., 2018. [Online]. Available: https://proceedings.neurips.cc/paper/2018/file/2de5d16682c3c35007e4e92982f1a2ba-Paper.pdf
  * [4] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson, “Learning latent dynamics for planning from pixels,” in _Int. Conf. on Machine Learning_. PMLR, 2019, pp. 2555–2565.
  * [5] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in _Int. Conf. on Machine Learning_ , 2018, pp. 1861–1870.
  * [6] G. Barth-Maron, M. W. Hoffman, D. Budden, W. Dabney, D. Horgan, D. TB, A. Muldal, N. Heess, and T. Lillicrap, “Distributed Distributional Deterministic Policy Gradients,” in _International Conference on Learning Representations_ , 2018.
  * [7] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017.
  * [8] A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller, “Maximum a posteriori policy optimisation,” in _International Conference on Learning Representations_ , 2018. [Online]. Available: https://openreview.net/forum?id=S1ANxQW0b
  * [9] R. Hasani, M. Lechner, A. Amini, D. Rus, and R. Grosu, “A natural lottery ticket winner: Reinforcement learning with ordinary neural circuits,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 4082–4093.
  * [10] Y. Yu, “Towards sample efficient reinforcement learning,” in _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ , ser. IJCAI’18. AAAI Press, 2018, p. 5739–5743.
  * [11] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” _arXiv preprint arXiv:1312.5602_ , 2013.
  * [12] H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V. Kumar, and S. Levine, “The ingredients of real world robotic reinforcement learning,” in _International Conference on Learning Representations_ , 2020. [Online]. Available: https://openreview.net/forum?id=rJe2syrtvS
  * [13] W. Zhao, J. P. Queralta, and T. Westerlund, “Sim-to-Real Transfer in Deep RL for Robotics: a Survey,” in _IEEE Symposium Series on Computational Intelligence (SSCI)_ , 2020, pp. 737–744.
  * [14] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-real transfer of robotic control with dynamics randomization,” _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , May 2018. [Online]. Available: http://dx.doi.org/10.1109/ICRA.2018.8460528
  * [15] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke, “Sim-to-Real: Learning Agile Locomotion For Quadruped Robots,” in _Proceedings of Robotics: Science and Systems_ , Pittsburgh, Pennsylvania, June 2018.
  * [16] M. Kaspar, J. D. Muñoz Osorio, and J. Bock, “Sim2Real Transfer for Reinforcement Learning without Dynamics Randomization,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2020, pp. 4383–4388.
  * [17] T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine, “Learning to walk via deep reinforcement learning,” in _Proceedings of Robotics: Science and Systems_ , FreiburgimBreisgau, Germany, June 2019.
  * [18] A. Singh, L. Yang, C. Finn, and S. Levine, “End-To-End Robotic Reinforcement Learning without Reward Engineering,” in _Proceedings of Robotics: Science and Systems_ , 2019.
  * [19] R. S. Sutton, “Dyna, an integrated architecture for learning, planning, and reacting,” _SIGART Bull._ , vol. 2, no. 4, p. 160–163, Jul. 1991. [Online]. Available: https://doi.org/10.1145/122344.122377
  * [20] M. P. Deisenroth and C. E. Rasmussen, “PILCO: A Model-Based and Data-Efficient Approach to Policy Search,” in _Proceedings of the 28th International Conference on Machine Learning_ , ser. ICML’11. Madison, WI, USA: Omnipress, 2011, p. 465–472.
  * [21] W. Schwarting, T. Seyde, I. Gilitschenski, L. Liebenwein, R. Sander, S. Karaman, and D. Rus, “Deep latent competition: Learning to race using visual control policies in latent space,” _arXiv preprint arXiv:2102.09812_ , 2021.
  * [22] J. Kabzan, M. de la Iglesia Valls, V. Reijgwart, H. F. C. Hendrikx, C. Ehmke, M. Prajapat, A. Bühler, N. Gosala, M. Gupta, R. Sivanesan, A. Dhall, E. Chisari, N. Karnchanachari, S. Brits, M. Dangel, I. Sa, R. Dubé, A. Gawel, M. Pfeiffer, A. Liniger, J. Lygeros, and R. Siegwart, “Amz driverless: The full autonomous racing system,” 2019.
  * [23] J. Kabzan, L. Hewing, A. Liniger, and M. N. Zeilinger, “Learning-based model predictive control for autonomous racing,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 4, pp. 3363 – 3370, 2019-10.
  * [24] L. Andresen, A. Brandemuehl, A. Honger, B. Kuan, N. Vödisch, H. Blum, V. Reijgwart, L. Bernreiter, L. Schaupp, J. J. Chung, M. Burki, M. R. Oswald, R. Siegwart, and A. Gawel, “Accurate Mapping and Planning for Autonomous Racing,” in _2020 IEEE/RSJ Int. Conference on Intelligent Robots and Systems (IROS)_ , 2020, pp. 4743–4749.
  * [25] E. Velenis and P. Tsiotras, “Minimum Time vs Maximum Exit Velocity Path Optimization During Cornering,” in _Proceedings of the IEEE International Symposium on Industrial Electronics, 2005. ISIE 2005._ , vol. 1, 2005, pp. 355–360.
  * [26] A. Rucco, G. Notarstefano, and J. Hauser, “An Efficient Minimum-Time Trajectory Generation Strategy for Two-Track Car Vehicles,” _IEEE Trans on Control Systems Tech_ , vol. 23, no. 4, pp. 1505–1519, 2015\.
  * [27] J. P. Timings and D. J. Cole, “Minimum maneuver time calculation using convex optimization,” _Journal of Dynamic Systems, Measurement, and Control_ , vol. 135, no. 3, 2013.
  * [28] J. L. Vázquez, M. Brühlmeier, A. Liniger, A. Rupenyan, and J. Lygeros, “Optimization-based hierarchical motion planning for autonomous racing,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2020, pp. 2397–2403.
  * [29] M. Jaritz, R. de Charette, M. Toromanoff, E. Perot, and F. Nashashibi, “End-to-end race driving with deep reinforcement learning,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 2070–2075.
  * [30] M. Riedmiller, M. Montemerlo, and H. Dahlkamp, “Learning to drive a real car in 20 minutes,” in _2007 Frontiers in the Convergence of Bioscience and Information Technologies_ , 2007, pp. 645–650.
  * [31] A. Kendall, J. Hawke, D. Janz, P. Mazur, D. Reda, J. Allen, V. Lam, A. Bewley, and A. Shah, “Learning to drive in a day,” in _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 8248–8254.
  * [32] F. Fuchs, Y. Song, E. Kaufmann, D. Scaramuzza, and P. Duerr, “Super-Human Performance in Gran Turismo Sport Using Deep Reinforcement Learning,” _arXiv preprint arXiv:2008.07971_ , 2020. [Online]. Available: https://arxiv.org/abs/2008.07971
  * [33] G. Bellegarda and K. Byl, “An Online Training Method for Augmenting MPC with Deep Reinforcement Learning,” in _2020 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2020, pp. 5453–5459.
  * [34] G. Williams, N. Wagener, B. Goldfain, P. Drews, J. M. Rehg, B. Boots, and E. A. Theodorou, “Information theoretic mpc for model-based reinforcement learning,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , 2017, pp. 1714–1721.
  * [35] S. Schaal, “Is imitation learning the route to humanoid robots?” _Trends in cognitive sciences_ , vol. 3, no. 6, pp. 233–242, 1999.
  * [36] S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in _Proceedings of the fourteenth international conference on artificial intelligence and statistics_. JMLR Workshop and Conference Proceedings, 2011, pp. 627–635.
  * [37] A. Y. Ng, S. J. Russell _et al._ , “Algorithms for inverse reinforcement learning.” in _Icml_ , vol. 1, 2000, p. 2.
  * [38] Y.-H. Wu, N. Charoenphakdee, H. Bao, V. Tangkaratt, and M. Sugiyama, “Imitation learning from imperfect demonstration,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 6818–6827.
  * [39] M. Sun and X. Ma, “Adversarial imitation learning from incomplete demonstrations,” _arXiv preprint arXiv:1905.12310_ , 2019.
  * [40] M. Lechner, R. Hasani, A. Amini, T. A. Henzinger, D. Rus, and R. Grosu, “Neural circuit policies enabling auditable autonomy,” _Nature Machine Intelligence_ , vol. 2, no. 10, pp. 642–652, 2020.
  * [41] M. Lechner, R. Hasani, M. Zimmer, T. A. Henzinger, and R. Grosu, “Designing worm-inspired neural networks for interpretable robotic control,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 87–94.
  * [42] A. Amini, I. Gilitschenski, J. Phillips, J. Moseyko, R. Banerjee, S. Karaman, and D. Rus, “Learning robust control policies for end-to-end autonomous driving from data-driven simulation,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 1143–1150, 2020.
  * [43] E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” http://pybullet.org, 2016–2019.
  * [44] V. S. Babu and M. Behl, “f1tenth. dev-an open-source ros based f1/10 autonomous racing simulator,” in _16th Int. Conf. on Automation Science and Engineering (CASE)_. IEEE, 2020, pp. 1614–1620.
  * [45] M. O’Kelly, H. Zheng, A. Jain, J. Auckley, K. Luong, and R. Mangharam, “Tunercar: A superoptimization toolchain for autonomous racing,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 5356–5362.
  * [46] Stanford Artificial Intelligence Laboratory et al., “Robotic operating system.” [Online]. Available: https://www.ros.org
  * [47] F. Braghin, F. Cheli, S. Melzi, and E. Sabbioni, “Race driver model,” _Computers & Structures_, vol. 86, no. 13, pp. 1503–1516, 2008, structural Optimization. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0045794908000163
  * [48] F. Lamiraux and J. . Lammond, “Smooth motion planning for car-like vehicles,” _IEEE Transactions on Robotics and Automation_ , vol. 17, no. 4, pp. 498–501, 2001.
  * [49] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [50] R. Hasani, M. Lechner, A. Amini, D. Rus, and R. Grosu, “Liquid time-constant networks,” _arXiv preprint arXiv:2006.04439_ , 2020.
  * [51] M. Lechner, R. Hasani, D. Rus, and R. Grosu, “Gershgorin loss stabilizes the recurrent neural network compartment of an end-to-end robot learning scheme,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 5446–5452.
  * [52] R. Hasani, A. Amini, M. Lechner, F. Naser, R. Grosu, and D. Rus, “Response characterization for auditing cell dynamics in long short-term memory networks,” in _2019 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2019, pp. 1–8.
  * [53] M. Lechner and R. Hasani, “Learning long-term dependencies in irregularly-sampled time series,” _arXiv preprint arXiv:2006.04418_ , 2020\.
  * [54] M. Hoffman, B. Shahriari, J. Aslanides, G. Barth-Maron, F. Behbahani, T. Norman, A. Abdolmaleki, A. Cassirer, F. Yang, K. Baumli, S. Henderson, A. Novikov, S. G. Colmenarejo, S. Cabi, C. Gulcehre, T. L. Paine, A. Cowie, Z. Wang, B. Piot, and N. de Freitas, “Acme: A research framework for distributed reinforcement learning,” 2020.
  * [55] A. Hill, A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, and Y. Wu, “Stable baselines,” https://github.com/hill-a/stable-baselines, 2018.
  * [56] A. Raffin, A. Hill, M. Ernestus, A. Gleave, A. Kanervisto, and N. Dormann, “Stable baselines3,” https://github.com/DLR-RM/stable-baselines3, 2019\.
  * [57] J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, “Algorithms for hyper-parameter optimization,” in _Advances in Neural Information Processing Systems_ , J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger, Eds., vol. 24. Curran Associates, Inc., 2011. [Online]. Available: https://proceedings.neurips.cc/paper/2011/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdf
  * [58] J. Bergstra, D. Yamins, and D. Cox, “Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures,” in _Proceedings of the 30th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, S. Dasgupta and D. McAllester, Eds., vol. 28, no. 1. Atlanta, Georgia, USA: PMLR, 17–19 Jun 2013, pp. 115–123. [Online]. Available: http://proceedings.mlr.press/v28/bergstra13.html
  * [59] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A next-generation hyperparameter optimization framework,” 2019.
  * [60] V. Sezer and M. Gokasan, “A novel obstacle avoidance algorithm: “follow the gap method”,” _Robotics and Autonomous Systems_ , vol. 60, no. 9, pp. 1123–1134, 2012. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0921889012000838
  * [61] U. of North Carolina, “UNC CS team wins F1/10 autonomous racing challenge,” https://cs.unc.edu/news/unc-cs-team-wins-f1tenth-autonomous-racing-challenge/, 2019\.

Appendix

##  S1 Hyperparameters

###  S1-A Dreamer

For generating initial training data, we use a Follow-the-Gap approach (see
section S2 for details). This choice is motivated by the fact that random
actions will not lead to actual driving behavior and that data of longer
trajectories contain more useful information to train the dynamics model.

In our experiments, we focused on the sensitivity of Dreamer to variations of
the parameters _Imagination Horizon_ and _Action Repeat_. We carried out our
experiments by fixing one configuration and systematically evaluate variations
of the other parameters. The configuration details are described in Table S1,
where the column _Value_ reports the configuration that we used to produce the
learning curves presented in the main text.

Parameter | Value | Search Space
---|---|---
Initial Dataset Size | 5000 steps | -
Update Steps | 100 | -
Batch Size | 50 | -
Sequence Length | 50 | -
Action Repeat | 8 | 4, 8
Imagination Horizon | 15 | 2, 20, 50, 100
Learning Rate $\theta$ | 6e-4 | -
Learning Rate $\psi$ | 8e-5 | -
Learning Rate $\phi$ | 8e-5 | -
Discount | 0.99 | -
$\lambda$ | 0.95 | -
TABLE S1: Hyperparameters for Dreamer

###  S1-B Model Free Baselines

To compare the performance of Dreamer to other RL algorithms, we use stable
implementations of model-free algorithms provided by multiple open-source RL
frameworks. Specifically, we used MPO, D4PG from ACME[54] and SAC, PPO, PPO-
LSTM from Stable Baselines [55, 56].

For the model-free baseline algorithms MPO, D4PG, PPO and SAC we choose the
same neural network architecture as for the latent actor-critic model of
Dreamer. The recurrent version of PPO uses a shared recurrent observation
network (see Table S3). For MPO and D4PG we choose ELU activation functions,
for PPO(-LSTM) and SAC we use the default settings of the implementation [55,
56]. For all details, please refer to the code-repository:
https://github.com/CPS-TUWien/racing_dreamer.

Module | Layers
---|---
Actor Network | MLP with 4 Layers, 400 neurons each
Critic Network | MLP with 3 Layers, 400 neurons each
TABLE S2: MPO, D4PG, PPO and SAC Architectures Module | Layers
---|---
Shared Observation Network | MLP with 2 Layers, 200 neurons each
LSTM, 256 Cells
Actor Network | MLP with 3 Layers, 400 neurons each
Critic Network | MLP with 2 Layers, 400 neurons each
TABLE S3: PPO-LSTM Architecture

Tables S4, S5, S6 and S7 show the sets of parameters for the algorithms as
well as the search space of the hyperparameter study, which we conducted for
MPO, D4PG and PPO. It turned out that in most of the cases, the default
parameters worked best, as they allowed a more stable learning. The study
optimizes hyperparameters of the algorithms by doing 100 training runs for 1
million timesteps on the Austria track. The objective value is the average
progress achieved within at most 4000 timesteps over 5 consecutive runs. As an
optimization procedure we make use of the Tree-structured Parzen Estimator
[57, 58] and HyperBand for pruning unpromising examples. We use the Optuna
hyperparameter optimization library to conduct these experiments [59].

Parameter | Value | Search Space
---|---|---
Policy Learning Rate | $10^{-4}$ | $\interval{10^{-5}}{10^{-3}}$
Critic Learning Rate | $10^{-4}$ | $\interval{10^{-5}}{10^{-3}}$
Batch Size | 256 | -
Epsilon | 0.1 | -
Epsilon Mean | $10^{-3}$ | $\interval{5\times 10^{-4}}{10^{-3}}$
Epsilon Stddev. | $10^{-6}$ | $\interval{5\times 10^{-7}}{10^{-5}}$
Replay Buffer Size | $5\times 10^{5}$ | -
TABLE S4: Hyperparameters for MPO [8] Parameter | Value | Search Space
---|---|---
Policy Learning Rate | $10^{-4}$ | $\interval{10^{-5}}{10^{-3}}$
Critic Learning Rate | $10^{-4}$ | $\interval{10^{-5}}{10^{-3}}$
Batch Size | 256 | -
Sigma | 0.3 | -
Atoms | 51 | -
Discount | 0.99 | -
Replay Buffer Size | $5\times 10^{5}$ | -
TABLE S5: Hyperparameters for D4PG [6] Parameter | Value | Search Space
---|---|---
Learning Rate | $3\times 10^{-4}$ | $\interval{10^{-5}}{3\times 10^{-3}}$
Batch Size | 64 | -
Discount | 0.99 | -
GAE-$\lambda$ | 0.95 | -
Clip Range | 0.2 | $\interval{0.1}{0.3}$
Entropy Coef. | 0.0 | $\interval{0.0}{0.01}$
Value Function Coef. | 0.5 | $\interval{0.5}{1.0}$
TABLE S6: Hyperparameters for PPO [7] and PPO-LSTM Parameter | Value
---|---
Learning Rate | $3\times 10^{-4}$
Batch Size | 256
Tau | 0.005
Discount | 0.99
Replay Buffer Size | $5\times 10^{5}$
TABLE S7: Hyperparameters for SAC [5]

##  S2 Comparison with Follow the Gap

Follow the Gap. Achieving good driving performance without using a map is
difficult because of the lack of global information on the track. In 2012, a
novel method called Follow the Gap (FTG) was introduced [60].

The algorithm proposes a simple obstacle-avoidance strategy: given the sensed
distances by the LiDAR, FTG chooses the steering angle to avoid the closest
obstacle and trying to identify the largest gap in the sensed distances.
Concerning the velocity, FTG adaptively tunes the reference velocity to be
proportional to the maximum distance scanned in front of the car. The velocity
ranges within a desired interval to avoid the car goes out of control.

This strategy is safe by construction, extraordinarily effective, and easy to
implement on real cars. In fact, FTG was the base algorithm recently adopted
by UNC to win the F1TENTH Montreal 2019 Competition [61]. For these reasons,
we use FTG as a baseline for some experiments conducted in simulation.

Dreamer versus FTG. Figure S1 shows the performance of our Dreamer
implementations, MPO, and FTG concerning maximum progress and lap-time
averaged over 10 episodes.

As already explained in the main text, the training track and observation
model play an important role in the training process. For this reason, some
configurations cannot complete one full lap, resulting as missing bars in
Figure S1 (bottom).

However, considering the Dreamer agents trained in Austria, a few remarks are
worth mentioning. In fact, despite FTG is a programmed algorithm and its
parameter has been tuned to achieve good performance and lap time comparable
with our Dreamer agent, Figure S1 shows that FTG also struggles in completing
Austria, Treitlstrasse, and Barcelona.

The maximum-progress achieved by Dreamer-Distance is comparable to FTG. The
lap time of FTG is slightly better, and the overall performance shows less
variance. This is not surprising considering that we optimize this algorithm
for these tasks.

Figure S1: Comparison of average performance and lap-time on unseen track over
10 evaluation episodes. The bar reports the average metric achieved over
several race tracks and the minimum and maximum are described by the
delimiters.

Analysis of trajectories. One of the drawbacks of FTG is its zig-zagging
behavior resulting from its gap-oriented policy. To analyze the policy learned
by our Dreamer agent, we collect 10 trajectories by running the best agent of
each configuration on all the evaluation tracks. The visualization of the
trajectories is reported in Figure S2 and we can identify the problematic
turns that cause collisions in each tracks. Note that to avoid to overload the
visualization, we select 10 trajectories, while in the previous experiments in
Figure S1 we average the performance over multiple agents, obtaining much
larger statistics.

The zig-zagging behavior of FTG is evident in Austria, Treitlstrasse, and
Barcelona, where the car continually changes its orientation in the straight
part of the track. Conversely in wider tracks like Columbia, FTG shows a
smoother trajectory and drives close to the internal wall.

Dreamer’s trajectories result more stable and centered in the track in the
straight sections. Figure S2 also highlights that Dreamer reduces the velocity
in the turns’ proximity. The mitigation of these sharp changes in velocity
could benefit the racing performance and will be addressed in future work.

Figure S2: Comparison of trajectories obtained by the trained policies on
Austria (AUT) and Treitlstrasse (TRT) with the programmed method Follow The
Gap (FTG). For each track and agent, we show 10 trajectories to highlight the
section of the track, which results in difficulty for the agent. The color of
the trajectories denotes the speed in $m/s$.

##  S3 Race Car Hardware

The hardware platform for the race car is shown in Figure S3 (left). It is
based on the F1TENTH race series [45]. It consists of a off-the-shelf model
race car chassis _Traxxas Ford Fiesta ST_. Propulsion is provided by a
brushless DC electric motor _Traxxas Velineon 3351R_ which is driven by a
_VESC 6 MkIV_ electronic speed controller (ESC). Computation and control tasks
are performed on a _NVIDIA Jetson TX2_ embedded computing platform. Supported
by a _Tech Orbitty Carrier_ adapter board it interfaces the sensors and
actuators by Ethernet, USB and RS-232. The environment is sensed by a _Hokuyo
UST-10LX_ LiDAR sensor. The _SparkFun 9DoF Razor IMU M0_ inertial measurement
unit provides data about the acceleration and angular velocity of the race car
(these measurements were not used throughout this work). For human interaction
during the operation of the car there is a _Logitech F710_ Gamepad controller
attached. This input is used to stop the car in case of emergency.

##  S4 Race Car Software Setup

Figure S3 (right) shows the software setup of the actual racing car. It is
running Ubuntu 18.04 as the base operating system and uses Robot Operating
System (ROS) [46] to provide an infrastructure for sending and receiving
messages and for the hardware abstraction. Third-party device drivers
(urg_node) for the LiDAR sensor and VESC (vesc_node) are used as an interface
between the hardware components and ROS. Auxiliary nodes for specific tasks
(e.g., emergency stop, translating messages) are based on the general software
setup of [45]. In order to have a similar environment in simulation and on the
racing car on an operating system level, we are using a docker container to
run our dreamer agent. Inside this docker container A ROS interface node is
used to translate observation and action messages. The motor force commands
are processed by integration in order to get the desired speed values. For the
steering commands, we added an adaptive low pass filter in order to protect
the servo from high frequent steering operations.

Figure S3: Race car hardware platform and software architecture.