††thanks: Corresponding author: E-mail address:
<EMAIL_ADDRESS>E-mail address<EMAIL_ADDRESS>
# Fractality in tumor growth at the avascular stage from a generalization of
the logistic-Gompertz dynamics

Luiza M. S. Miranda  Andre M. C. Souza  Departamento de Fisica, Universidade
Federal de Sergipe, 49100-000, Sao Cristovao-SE, Brazil

###### Abstract

The dynamics between healthy and malignant cells at the avascular stage of
growth is described by a set of chemical reactions representing the
populations of both types of cells. We obtain a generalization of the
logistic-Gompertz dynamics and study the consequences in terms of the
properties of tumor growth associated with the Warburg effect, mitosis and
nutrient and space apoptosis. As a result of the stochastic nature of this
process, a fractal morphology is observed in this region. Our results agree
with experimental results for growth of tumor cell colonies.

logistic map; avascular tumor growth; fractal edge

##  I Introduction

The dominant physical processes involved in the growth of cancerous tumors
still pose a major challenge for science. Researches have observed the
presence of characteristics associated with deterministic chaos, internal and
external noise, dynamic behavior associated with nonlinear systems, relative
robustness to internal and external perturbations and plasticity ara ; sinh .
A mechanism for the dynamics of growth that encompasses all of these of
complexity has not been described by any theory. Several models of tumor
growth have been introduced based on the assumption of only some of the above
characteristics sinh ; ende ; roo ; ampa ; low ; izq3 ; cel0 . Of course, it
should be clear that the choice of some of the above characteristics must
still be able to represent the growth of the tumor and certain aspects of its
complexity, in order to make a solution possible.

Recently, a heuristic model based on simple chemical reactions was proposed by
Izquierdo-Kulich et al. to describe the dynamics of cancerous tumor growth at
the avascular stage izq1 . The processes of mitosis and apoptosis and the
action of the host are considered and the behavior is related to the logistic-
Gompertz dynamics ara .

In this work, we generalize the chemical reaction network proposed by
Izquierdo-Kulich et al. izq1 . We also add the Warburg effect to our model, to
take into consideration the fact that acidification of the medium of a normal
cell can create a tumor war1 ; war2 . In the step that describes apoptosis, we
separate the apoptosis based on nutrients and space processes and introduce a
parameter $q$ that is associated with the density of the tumor cells. The
above processes do not exhaust all the mechanisms of tumor growth, since the
complexity of the tumor surface, which is associated with the geometric
fractal dimension of the contour, clearly shows that we cannot ignore the
stochastic environment in which the tumor is found. We therefore also
introduce a noise term. A generalized logistic-Gompertz dynamics is presented
that includes these new components of the tumor growth.

From the results obtained here, we develop a relation between the fractal
structure of the cell and the parameter $q$ of our formalism.

The paper is organized as follows. In Sec. II, we introduce our chemical model
of avascular tumor growth, and present some results derived from our model. In
Sec. III, we develop a relation between the fractal structure of the cell and
the parameter $q$ proposed in our formalism. Finally, Sec. IV presents our
concluding remarks.

##  II The chemical model of avascular tumor growth

Izquierdo-Kulich et al. proposed a heuristic model (henceforward referred to
as the IRTN model) based on a network of simple chemical reactions izq1 . In
the avascular phase, the tumor grows until it reaches a dormant state. If $A$
represents the normal cell population, $N$ is the number of proliferating
tumor cells, $B$ is the host action, and $P$ represents a non-cancerous
process, the tumor dynamics can be written in the form of the following
chemical reactions:

(I) $N+A\rightarrow 2N$

(II) $2N\rightarrow P$

(III) $N+B\rightarrow P$

Step I is related to the process of mitosis and the reaction is associated
with the mitosis constant $K_{m}$. Steps II and III correspond to apoptosis
with the constant $K_{a}$ and host cell action with the constant $K_{b}$,
respectively. It is easy to see that the dynamic behavior of the system is
given by the following logistic-Gompertz equation:

$\frac{dN(t)}{dt}=\gamma^{\prime}N(t)-k_{a}N(t)^{2},$ (1)

where $\gamma^{\prime}=Ak_{m}-Bk_{b}$. It is important to carry out a
dimensional analysis of the model parameters. $N$, $A$ and $B$ are
dimensionless, $t$ has the dimension of $[time]$, and the constants $k_{a}$,
$k_{b}$, $k_{m}$ and $\gamma^{\prime}$ have the dimension of $[time]^{-1}$. By
implicitly setting $k_{b}=1$, we have assumed a timescale of $(k_{b})^{-1}$.
The solution to the temporal dynamics of the number of tumor cells is

$N(t)=\frac{\gamma^{\prime}N(0)}{k_{a}N(0)(1-e^{-\gamma^{\prime}t})+\gamma^{\prime}e^{-\gamma^{\prime}t}}.$
(2)

For $\gamma^{\prime}>0$ the stationary state has
$N(t\rightarrow\infty)=\gamma^{\prime}/k_{a}$ whereas for $\gamma^{\prime}<0$
we have $N(t\rightarrow\infty)=0$. A bifurcation occurs at
$\gamma^{\prime}=0$, which act as a second-order phase transition izq1 ; izq2
, consistent with the result of Prigogine and Lefever pri .

Tumor cells not only differ from normal cells in terms of their appearance,
but their entire energy metabolism is reassembled. The energy consumption of
metabolic activities in normal cells depends mainly on mitochondrial oxidative
phosphorylation, which is efficient and generates more adenosine triphosphate
(ATP) than glycolysis. However, one of the metabolic characteristics of tumor
cells is that they mainly convert glucose through aerobic glycolysis. This
pathway for energy production in tumor cells is known as the Warburg effect
war1 ; war2 . In the present work, we improve the IRTN model of this effect
through the addition of a new step for the chemical equations. Finally, in the
step that describes apoptosis, we separate the apoptosis based on the nutrient
and space processes, to allow the generalized model to be written in the form:

(I) $A+N\rightarrow 2N$

(II) $A\rightarrow N$

(III) $qN\rightarrow P$

(IV) $N+B\rightarrow P$.

The mitosis process is described in Step I, in which there are $N$ tumor cells
and $A$ normal cells. Step II describes the Warburg effect, in which a normal
cell can become a tumor following acidification of the medium. Step III
describes the process of apoptosis by nutrients, where for this to happen,
there must be a density of tumor cells $q$. Step IV describes apoptosis by
space, in which $B$ defense cells of the host tissue generate apoptosis. The
new dynamic behavior of the system is given by the following logistic-Gompertz
equation:

$\frac{dN(t)}{dt}=\gamma N(t)-k_{an}N(t)^{q}+Ak_{mw},$ (3)

where $\gamma=Ak_{m}-Bk_{ae}$. We have assumed that the timescale is
$(k_{ae})^{-1}$. We take into consideration two processes of mitosis. The
first term refers to the duplication of cancer cells at a constant rate
$k_{m}$. The possible transformation of normal cells into cancer cells linked
to the Warburg effect takes place at a constant rate $k_{mw}$. In this case,
the acidity of the medium, favored by the anaerobic respiration of cancer
cells, causes fermentation, and therefore serves as a catalyst for the
appearance of new cancer cells.

Apoptosis is linked to the mechanisms for the destruction of cancer cells by
the body’s defense systems. We can divide these mechanisms into two types:
apoptosis by nutrients at a constant rate $k_{an}$, and apoptosis by space,
with a constant rate $k_{ae}$. Nutrient apoptosis involves the mechanisms of
competition for nutrients between cells, in which the high density of cancer
cells in each region of the tissue favors a lack of nutrients and thus their
destruction. However, the lack of nutrients does not fully describe some basic
features of tumor growth. Cell proliferation is greatly inhibited inside the
colony or tumor bru1 , and the geometric and dynamic properties of cell colony
growth are affected by external pressure brup . In view of this behavior, an
expression for tumor growth cannot be extracted considering only apoptosis by
nutrients, and it is therefore of fundamental importance to add apoptosis by
space as a mechanism for inhibiting tumor growth. In the present model, this
mechanism is introduced with a rate reaction $k_{ae}$, and is proportional to
the density of cancer cells in the region, represented by the parameter $q$.
The IRTN model is obtained for $k_{mw}=0$, $k_{ae}=1$, $k_{an}=k_{n}$,
$\gamma=\gamma^{\prime}$ and $q=2$.

We emphasize that a similar parameter $q$ was introduced in a previous work
tsacel , as part of a model based on the ideas of nonextensive statistical
mechanics tsa1 ; tsa2 . From different values of the nonextensive parameter
$q$, different types of tumor evolution were obtainedtsacel . Based on the
ideas underlyng nonextensive entropy, the study in tsacel  showed that tumor
growth depends on the nonextensive parameter $q$: the smaller the value of
$q$, the more aggressive the tumor becomes. It is important to note that
although the parameter $q$ used in the present work has similarities with that
used in tsacel , it is not the same, since the equations for dynamics are not
the same in both works, and the Warburg effect and stochastic noise are not
considered in tsacel .

These processes are not likely to be an exhaustive description of all the
mechanisms of tumor growth, since the complexity of the tumor surface, which
isassociated with the geometric fractal dimension of the contour, clearly
shows that we cannot ignore the stochastic environment in which the tumor is
found bruprl . In addition to the processes of mitosis and apoptosis, we
consider a noise term $\eta_{i}$ taken out of a box uniform distribution
within $[-W/2,W/2]$, where $W$ is the disorder strength and $i$ the sample.

Given the above, the dynamics of tumor growth can be described by the
following expression

$\frac{dN(t)}{dt}=\gamma N(t)-k_{an}N(t)^{q}+Ak_{mw}+\eta_{i}.$ (4)

Fig. 1 shows the evolution of the number of cancer cells as a function of time
for $q=1.5$,$1.8$,$1.9$ and $2.0$. The Warburg effect and noise are not
considered, i.e., $k_{mw}=0$ and $\eta=0$. In this limit, Eq. 4 becomes a
Bernoulli differential equation, and it is therefore to show that

$N(t)=\frac{\gamma^{\frac{1}{q-1}}N(0)}{\left[k_{an}N(0)^{q-1}(1-e^{-(1-q)\gamma
t})+\gamma e^{-(1-q)\gamma t}\right]^{\frac{1}{q-1}}}.$ (5)

The tumor grows to a certain size and then goes into a dormant state for every
value of $q$. An arbitrary value of $q$ does not change the universality class
of the IRTN model ($q=2$). A bifurcation occurs at $\gamma=0$ and for
$\gamma>0$ the stationary state has
$N(t\rightarrow\infty)=(\gamma/k_{an})^{1/(q-1)}$, while for $\gamma<0$ we
have $N(t\rightarrow\infty)=0$. Considering that $q$ is directly linked to the
density of tumor cells so that there is apoptosis, we can see that the higher
the density of the tumor cells, the faster the tumor stabilizes. This is
because a lack of nutrients is associated with the mechanisms of competition
for nutrients between cells. Thus, the high density of tumor cells in each
region of the tissue favors a lack of nutrients and thus their destruction.
Tumor growth is restricted by the ability of the tissue to transport
nutrients. Note that in the initial stage of tumor growth the role of $q$ is
almost imperceptible. As noted previously in tsacel , the smaller values of
$q$, the more aggressive the tumor becomes. Since the number of infected cells
$N(t)$ in the steady state becomes larger as $q$ decreases.

Figure 1: Time evolution of tumor cells $N(t)$ in the absence of the Warburg
effect ($k_{mw}=0$) and a noise term($\eta=0$). We use the timescale as
$(k_{ae})^{-1}$. We consider $q=1.5$ (red curve), $1.8$ (green curve), $1.9$
(blue curve) and $2.0$ (black curve). Figure 2: Time evolution of tumor cells
$N(t)$ in the absence of a noise term($\eta=0$) and $q=2$, for Warburg rates
$k_{mw}=0$ (black curve), $10.0$ (blue curve) and $20.0$ (green curve). We use
a timescale $(k_{ae})^{-1}$.

The Warburg effect is shown in Fig. 2 for a fixed value of $q=2$. The
analytical solution for the dynamics of tumor growth is

$N(t)=\frac{N(0)\left[\Delta+\gamma+(\Delta-\gamma)e^{-\Delta
t}\right]+2k_{mw}(1-e^{-\Delta t})}{2k_{an}N(0)(1-e^{-\Delta
t})+(\Delta+\gamma)e^{-\Delta t}+\Delta-\gamma},$ (6)

where $\Delta=\sqrt{\gamma^{2}+4k_{an}k_{mw}}$. it can be seen from Fig. 2
that with an increase in the Warburg effect constant $k_{mw}$, tumor growth
tends to become faster. This is because cellular respiration by aerobic
glycolysis is faster despite being an inefficient means of ATP production. In
the Warburg effect, tumor cells conduct metabolism mainly by aerobic
glycolysis, producing lactate even in the presence of oxygen and fully
functioning mitochondria. This process increases the intracellular pH and
acidifies the extracellular environment, turning normal cells into tumor
cells. This metabolic characteristic makes the tumor more robust and
aggressive. Similar results are obtained for the other cases of $q$.

For the case where $\eta\neq 0$, the quantities can be determined by their
average $\langle...\rangle_{\eta}$ over $m$ independent realizations of the
noise term $\eta$. We find that the noise does not change the size of the
tumor. From this point, we have explored the model solutions numerically. We
obtain the result $\langle N(t)\rangle_{\eta}\approx\langle
N(t)\rangle_{\eta=0}=N(t)$. The inset to Fig 3 shows this result, in which the
time evolution of $\langle N(t)\rangle_{\eta}$ is similar for three different
values of $W$.

The noise effect is found by analyzing the standard deviation of the
variables. For $\langle N(t)\rangle_{\eta}$, the standard deviation is

$\langle\sigma(t)\rangle_{\eta}=\sqrt{\langle N(t)^{2}\rangle_{\eta}-\langle
N(t)\rangle_{\eta}^{2}}.$ (7)

Fig. 3 shows $\langle\sigma(t)\rangle_{\eta}$ for typical values of the model
parameters. We consider $m=100$ samples to noise term $\eta_{i}$ average. It
is possible to see that the standard deviation with stochastic noise increases
during tumor growth and then decreases when the tumor enters the dormant
state. Of course, increasing the disorder strength $W$ also increases the
deviation $\langle\sigma(t)\rangle_{\eta}$. For fixed values of the other
parameters of the model, we find that the standard deviation increases as $q$
decreases, whereas it increases as $k_{mw}$ increases.

Figure 3: Time evolution of the standard deviation of tumor cells
$\langle\sigma(t)\rangle_{\eta}$. The curves are: black color ($q=2$,
$k_{mw}=0$, $W=0.1$); red color ($q=2$, $k_{mw}=10.0$, $W=0.1$); blue color
($q=2$, $k_{mw}=0$, $W=1.0$); green color ($q=1.8$, $k_{mw}=0$, $W=1.0$). We
use a timescale $(k_{ae})^{-1}$ and $m=100$ samples. INSET: $\langle
N(t)\rangle_{\eta}$ versus $t$ for $q=2$, $k_{mw}=0.0$ and $W=0$ (circles),
$W=0.1$ (squares), $W=1.0$ (triangles). Differences in the curves are not
noticeable at this scale. Figure 4: Tumor interface versus radius $r(t)/R$ at
times $t=0.2$ (pink curve), $0.4$ (green curve), $0.6$ (red curve), and $0.8$
(brown curve) for $q=2$, $k_{am}=0.1$, $k_{mw}=0$, $\gamma=10.0$ and $W=1.0$.
Figure 5: Fractal dimension of the tumor interface calculated with the box-
counting method at time $t$, corresponding to $\langle
r(t)\rangle_{\eta}\approx 9R$ as a function of $q$. We use the parameters
$k_{am}=0.1$, $k_{mw}=0$, $\gamma=10.0$ and $W=1.0$.

##  III Fractal structure of a tumor cell

Experiments performed in mice have shown that the dynamics of tumor growth
create a host-tumor interface with a rough structure bruprl . There is even a
change in the growth dynamics of cell colonies that is induced by a change in
pressure brup . Considering the surface roughness structure of the tumor, it
is possible to determine the scale invariance of the fractal interface from
the calculation of its fractal dimension $d_{f}$ bru1 .

Different cell lines and different types of tumors were studied and fractal
dimensions were obtained for a range of $d_{f}\in[1.05,1.35]$. A direct
relationship between the fractal dimension and the type of tumor was not
conclusive. However, it became very clear that there was a need for further
studies to relate these features of the tumor and to some clinical strategies
bru1 . Within this context, we present an analysis that relate the fractal
dimension of the tumor surface and to the components of the formalism
introduced in this work. Our main hypothesis is the assumption that the
surface radius of the tumor $r(t)$ is proportional to the square root of the
number of tumor cells $N(t)$. We consider that the tissue that forms the
cellular substrate is two-dimensional, and hence $r(t)=R\sqrt{N(t)}$. The
different radius values are obtained from different realizations of the noise
term $\eta_{i}$.

In Fig. 4, we see the tumor contours as a function of the radius $r(t)/R$ for
different times. The times are $k_{ae}t=0.2$, $0.4$, $0.6$ and $0.8$, which
were chosen based on the growth dynamics shown in previous figures. We used
values of $q=2$, $k_{am}=0.1$, $k_{mw}=0$, $\gamma=10.0$ and $W=1.0$, and
$m=360$ samples (where each represented one of the $360$ angles in the
figure). Similar results were found for arbitrary values of $q$. We performed
the same analysis for different numbers of angles, and the results obtained
for the fractal dimension were the same, indicating that the number of samples
does not affect the roughness structure of the dynamics in the present model.

The box-counting method was used to calculate the fractal dimension $d_{f}$.
Fig. 5 shows the fractal dimension of the tumor at time $t$, corresponding to
$\langle r(t)/R\rangle_{\eta}\approx 9$ as a function of $q$. We use the
parameters $k_{am}=0.1$, $k_{mw}=0$, $\gamma=10.0$ and $W=1.0$. It is
straightforward to observe that the value of $d_{f}$ are within the interval
obtained from the experimental results under normal pressure conditions bru1 .
We find that increasing $q$ decreases $d_{f}$, since the value of $q$
indicates the density of the tumor cells in the process of apoptosis in
dispute for nutrients. We find that an increase in $d_{f}$ indicates a lower
density of cells in dispute for nutrients, this is because the higher the
density of malignant cells, the faster the tumor stabilizes, as the standard
deviation $\langle\sigma(t)\rangle_{\eta}$ is smaller. We found that for tumor
growth, $d_{f}$ initially increases until it reaches a maximum, within an
intermediate evolution time, and then decreases as the tumor enters the steady
state.

##  IV Conclusions

We propose a model for tumor growth at the avascular stage based on a new
chemical reaction network, in which we consider the process of mitosis by cell
division, the Warburg effect, and apoptosis by nutrients and space. In
addition, we include the stochastic noise, and fluctuations at the avascular
stage of the tumor are observed. From the logistic-Gompertz dynamics, we see
that at the outset, tumor growth is fast, whereas over time the cell
population saturates and reaches a dormant state. The time to reach a
steady/dormant state depends on the index $q$, which represents the density of
tumor cells. The Warburg effect clearly affects tumor growth, as tumor cells
conduct their metabolism mainly by aerobic glycolysis. This metabolic
characteristic makes cancer more robust and aggressive, due to acidification
of the extracellular environment. The relationship between the fractal
dimension of the tumor surface and the tumor cell density $q$ is implicit in
the dynamics of the model. It is important to mention that the results
obtained here agree with the experimental results recently presented for the
growth of tumor cell colonies bru1 ; brup . Our results suggest that the
fractal dimension of the tumor’s borders increases to a maximum value in a
certain intermediate time of its evolution, and then decreases as the tumor
moves towards its stationary state. Experiments to test this hypothesis are
welcome.

## ACKNOWLEDGMENTS

We acknowledge the support of the CNPq (Brazilian agency).

## References

  * (1) R. P. Araujo, and D. L. S. McElwain, A History of the Study of Solid Tumour Growth: The Contribution of Mathematical Modelling, Bulletin of Mathematical Biology 66 (2004) 1039-1091.
  * (2) S. Sinha, and P. Singh, Mathematical modeling and simulation of mechano-chemical effect on two-phase avascular tumor, International Journal of Modern Physics C 33 (2022) 2250063.
  * (3) H. Enderling, Cancer stem cells: small subpopulation or evolving fraction?, Integr. Biol. 7 (2015) 14-23.
  * (4) T. Roose, S. J. Chapman, and K. Ph. Maini, Mathematical models of avascular tumor growth, SIAM Review 49 (2007) 179-208.
  * (5) P. Ampatzoglou, G, Dassios, M. Hadjinicolaou, H. P. Kourea, and M. N. Vrahatis, A chemical energy approach of avascular tumor growth: multiscale modeling and qualitative results, SpringerPlus 4 (2015) 660.
  * (6) J. S. Lowengrub, H. B. Frieboes, F. Jin, Y. Chuang, X. Li, P. Macklin, S. Wise, and V. Cristini, Nonlinear modelling of cancer: bridging the gap between cells and tumours, Nonlinearity 23 (2010) R1-R9.
  * (7) E. Izquierdo-Kulich, and J. M. Nieto-Villar, Mesoscopic model for tumor growth, Mathematical Biosciences and Engineering 4, (2007) 687-698.
  * (8) E. Izquierdo-Kulich, J. M. Nieto-Villar, Morphogenesis of the tumor patterns, Math. Biosci. Eng. 5 (2008) 299-313.
  * (9) E. Izquierdo-Kulich, I. Rebelo, E. Tejera, and J. M. Nieto-Villar, Phase transition in tumor growth: I avascular development, Physica A 392 (2013) 6616-6623.
  * (10) O. Warburg, The metabolism of carcinoma cells, The Journal of Cancer Research 9 (1925) 148-163.
  * (11) M. V. Liberti, and J. W. Locasale, The Warburg Effect: How Does it Benefit Cancer Cells?, Trends Biochem Sci. 41 (2016) 211-218.
  * (12) J. A. Llanos-Pérez, A. Betancourt-Mar, M. P. De Miguel, E. Izquierdo-Kulich, M. Royuela-Garcia, E. Tejera, and J. M. Nieto-Villar, Phase transitions in tumor growth: II prostate cancer cell lines, Physica A 426 (2015) 88-92.
  * (13) I. Prigogine, and R. Lefever, Stability problems in cancer growth and nucleation, Comparative Biochemistry and Physiology 67B(1980) 389-393.
  * (14) A. Bru, S. Albertos, J. L. Subiza,J. L. Garcıa-Asenjo, and I. Bru, The universal dynamics of tumor growth, Biophys. J. 85 (2003) 2948-2961.
  * (15) A. Bru, and D. Casero, The effect of pressure on the growth of tumour cell colonies, J. Theor. Biology 243 (2006) 171-180.
  * (16) J. A. González, and I. Rondón, Cancer and nonextensive statistics, Physica A 369 (2006) 645-654.
  * (17) C. Tsallis, Possible Generalization of Boltzmann-Gibbs Statistics, J. Stat. Phys. 52 (1988) 479-487.
  * (18) C. Tsallis, S. V. F. Levy, A. M. C. de Souza, and R. Maynard, Statistical-Mechanical Foundation of the U biquity of Lévy Distributions in Nature, Phys. Rev. Lett. 75 (1995) 3589-3593.
  * (19) A. Bru, S. Albertos, J. A. L. Garcıa-Asenjo, and I. Bru, Pinning of Tumoral Growth by Enhancement of the Immune Response, Phys. Rev. Lett. 92 (2004) 238101.


# ObserveNet Control: A Vision-Dynamics Learning Approach to Predictive
Control in Autonomous Vehicles

Cosmin Ginerica1 Mihai Zaha1 Florin Gogianu2 Lucian Busoniu3 Bogdan Trasnea1
and Sorin Grigorescu1 Manuscript received: February 24, 2021; Revised: May 28,
2021; Accepted: July 6, 2021. This paper was recommended for publication by
Editor Jens Kober upon evaluation of the Associate Editor and Reviewers’
comments.1 Cosmin Ginerica, Mihai Zaha, Bogdan Trasnea and Sorin Grigorescu
are with RovisLab (Robotics, Vision and Control Laboratory, Romania)2 Florin
Gogianu is with BitDefender, Romania3 Lucian Busoniu is with Technical
University of Cluj-Napoca, Romania Digital Object Identifier (DOI): see top of
this page.

###### Abstract

A key component in autonomous driving is the ability of the self-driving car
to understand, track and predict the dynamics of the surrounding environment.
Although there is significant work in the area of object detection, tracking
and observations prediction, there is no prior work demonstrating that raw
observations prediction can be used for motion planning and control.

In this paper, we propose ObserveNet Control, which is a vision-dynamics
approach to the predictive control problem of autonomous vehicles. Our method
is composed of a i) deep neural network able to confidently predict future
sensory data on a time horizon of up to $10s$ and ii) a temporal planner
designed to compute a safe vehicle state trajectory based on the predicted
sensory data. Given the vehicle’s historical state and sensing data in the
form of Lidar point clouds, the method aims to learn the dynamics of the
observed driving environment in a self-supervised manner, without the need to
manually specify training labels. The experiments are performed both in
simulation and real-life, using CARLA and RovisLab’s AMTU mobile platform as a
1:4 scaled model of a car. We evaluate the capabilities of ObserveNet Control
in aggressive driving contexts, such as overtaking maneuvers or side cut-off
situations, while comparing the results with a baseline Dynamic Window
Approach (DWA) and two state-of-the-art imitation learning systems, that is,
Learning by Cheating (LBC) and World on Rails (WOR).

##  I Introduction

With the rise of Artificial Intelligence and Deep Neural Networks (DNN),
research in the area of autonomous vehicles has advanced significantly in the
last decade, being driven both by academia and industry alike. An autonomous
car is an intelligent agent which observes its environment, makes decisions
and performs actions based on these decisions. The state-of-the-art approaches
for self-driving car control are based on the traditional perception-planning-
act pipeline [1], where first the environment is reconstructed from sensory
data. Secondly, a safe vehicle state trajectory path is computed by a path
planner. Finally, the path is translated to motion commands which are executed
by a controller.

A key difficulty is that the control strategy is affected by unpredictable
external factors that can arise in the driving environment. It has been
observed that the control quality increases when the motion of the objects
around the car is tracked and predicted. However, such an approach is strictly
dependent on the accuracy of the object detection system. If the detection
system fails, than the motions of the objects cannot be predicted anymore,
hence degrading the performance of the controller.

Figure 1: ObserveNet Control. A deep neural network is trained in a self-
supervised fashion to predict raw sensory data over a finite prediction
horizon $[t+1,t+\tau_{o}]$. The predictions, as well as historic states and
observations, are iteratively used by a temporal state trajectory planner to
calculate and execute a safe vehicle motion.

In this paper, we propose the ObserveNet Control method depicted in Fig. 1,
which is an approach to derive control signals from predicted raw sensory
data. The algorithm is composed of a DNN which receives as input historical
system states and measurements from a Lidar range sensing device. The network
is composed of convolutional and recurrent layers trained in a self-supervised
fashion, without the need of manually labeling training data. The output of
the DNN represents the most likely future state trajectory of the vehicle,
where each state in the state trajectory is augmented with predicted sensory
data.

Our work was mainly inspired by [2], which served as a baseline for developing
the architecture of the DNN. Apart from the application area (autonomous
driving vs. navigating a simulated maze), we highlight the several key
differences in our work, as detailed in Section IV-A. For performance
evaluation, we have tested our algorithm on two distinct platforms, namely
within the CARLA simulator, as well as on our RovisLab Autonomous Mobile
Testing Platform (AMTU) from Fig 6.

The key contributions of the paper are:

  * •

the deep neural network architecture of ObserveNet, used to predict future
state trajectories and raw sensory measurements along a finite prediction
horizon;

  * •

the self-supervised training of ObserveNet with raw range sensing information
acquired from a Lidar device;

  * •

a temporal planner used to calculate a safe vehicle path based on the output
of ObserveNet.

The rest of the paper is organized as follows. Section II covers the related
work. The methodology of ObserveNet Control is given in Section III. The
experimental validation is given in Section IV. Finally, conclusions are
stated in Section V.

##  II Related Work

Similar to the perception-planning-act [1] paradigm, our approach divides the
algorithm into raw observations prediction and state trajectory planning. This
is opposed to End-To-End learning [3], where sensory data is directly mapped
to control signals via a deep neural network.

As a result of progress in AI, several model-free approaches have emerged in
the field of learning-based control for self-driving vehicles. One such
paradigm is End-To-End Learning, where input data is mapped directly to
control signals. Another approach is Deep Reinforcement Learning (DRL), where
an agent is controlled via action-reward systems [4]. In terms of training
data, self-supervised learning is a technique heavily used in modern learning-
based architectures, consisting of algorithms that learn independently,
without the need of human intervention in the form of manual data labelling.

From a data-driven control perspective, our work builds on goal-conditioned
imitation learning [5, 6, 7, 8]. As shown in [6], this class of methods
combines learning from an expert driver with the advantages of goal-directed
planning based on dynamic models and reward functions. However, in addition
learning the optimal driving path, we also employ the prediction of future raw
observations within the control task.

Additional to accurate object detection, the performance of autonomous
vehicles is directly influenced by their ability to predict future positions
of dynamic agents, such as other cars and pedestrians [9]. Instead of focusing
on label-intensive object detection and tracking, we proposes to learn to
predict future sensory data via self-supervised learning and plan future
trajectories based on the predicted sensory data. Raw observations predictions
include PredNet [10], which was trained to predict future frames in a video
sequence, while the work in [11, 12, 13, 14] is focused on forecasting an
intermediate occupancy map representation of the surrounding traffic
environment. An inverted pose forecasting pipeline is proposed in [15], where
the first phase of the algorithm predicts future 3D point clouds, followed by
object detection and pose estimation, instead of first detecting objects and
then predicting their future motion. Apart from sensory data, LaneGCN [16] and
VectorNet [17] are used to fuse object detection and tracking with HD map data
in order to better forecast the motion of dynamic objects within complex road
networks. However, using our ObserveNet sensory data predictor, we also
propose to derive control signals for the vehicle, thus showing that automatic
control based on predicted observations is viable.

The authors of [2] use a temporal neural network to construct an agent’s
belief. The belief is used in a RL setup to build a world model. The agent’s
observations are modeled as one-hot encoded obstacles in an occupancy grids.
The goal is to learn an information seeking behavior for an agent, endowing it
with a strong drive towards discovering its environment. In our work, we use a
different observation modelling mechanism for Lidar data. While the agent in
[2] moves based on four discrete control actions (left, right, up, down), in
our work the actions are represented by vehicle control signals. Another
difference is that the authors in [2] use a control method based on
Reinforcement Learning, as opposed to our ObserveNet Control pipeline.

A key contribution of our work is therefore to extend the neural network
architecture from [2], adapt it to our sensor setup, and illustrate that this
modified architecture can provide results in a closed-loop system, tested
under dynamic traffic conditions and with a real-world robotic system.

##  III Methodology

Throughout the paper, we use the following notation. The value of a variable
is defined either for a single discrete time step $t$, written as superscript
$<t>$, or as a discrete sequence defined in the $<t,t+k>$ time interval, where
$k$ denotes the length of the sequence. For example, the value of a state
variable $\mathbf{z}$ is defined either at discrete time $t$ as
$\mathbf{z}^{<t>}$, either within a sequence interval $\mathbf{z}^{<t,t+k>}$.
The difference between a measured and a predicted quantity is made using the
hat notation(e.g $\mathbf{z}$ and $\hat{\mathbf{z}}$ are measured and
predicted values, respectively). Vector and matrices are indicated by bold
symbols.

###  III-A Problem definition

The autonomous driving problem can be stated as follows. Given a global
reference trajectory $\mathbf{z}^{<t-\infty,t+\infty>}_{ref}$, we want to
calculate and execute a safe vehicle state trajectory
$\mathbf{z}^{<t+1,t+\tau_{o}>}$, given past and current sensory observations
$\mathbf{I}^{<t-\tau_{i},t>}$, as well as predicted raw observations
$\mathbf{\hat{I}}^{<t+1,t+\tau_{o}>}$. $t$ is the discrete time, while
$\tau_{i}$ and $\tau_{o}$ are the length of the sequence of past observations
and the control horizon, respectively.

A state trajectory $\mathbf{z}^{<t-i,t+i>}$ is defined as a sequence of
vehicle states:

$\mathbf{z}_{ref}=[x,y,v,\phi]$ (1)

where $x$ and $y$ are the Cartesian coordinates of the ego-vehicle, $v$ is the
velocity and $\phi$ the orientation.

###  III-B ObserveNet Predictive Control

The objective of the control loop is to compute a safe state trajectory
$\mathbf{z}^{<t+1,t+\tau_{o}>}_{d}$ which is to be executed by a motion
controller. Fig 2 illustrates our proposed ObserveNet predictive control
approach. Our main contributions are the Observations and Trajectory
Prediction module, along with the Temporal State Trajectory Planning algorithm
which consumes past and predicted observations and states, while computing the
input to a Constrained Non-linear Model Predictive (NMPC) controller. Within
its optimization loop, NMPC calculates optimal the control signals
$\mathbf{u}^{<t>}=(v^{<t>}_{cmd},\delta^{<t>}_{cmd})$, where $v^{<t>}_{cmd}$
and $\delta^{<t>}_{cmd}$ are the longitudinal velocity and steering angle,
respectively.

Figure 2: Block diagram of the ObserveNet Predictive Control algorithm. Based
on historical Localization & Perception data stored in the Augmented Memory
component, the method predicts raw sensory Observations and State Trajectories
which are used by a Temporal Planner to calculate a safe desired vehicle
trajectory executed via Constrained Non-linear Model Predictive control. The
dotted lines illustrate the flow of data used during self-supervised training.

The Localization & Perception component from Fig. 2 constructs a top-view 2D
occupancy grid out of 3D Lidar point clouds, while computing the state of the
vehicle using inertial measurements from an IMU, geolocation data from a GPS
receiver and wheels odometry. The Ackermann vehicle model is used both by the
state estimator and the controller:

$\hat{\mathbf{z}}^{<t+1>}=\mathbf{z}^{<t>}+v^{<t>}\cdot\begin{bmatrix}\cos(\phi^{<t>})\\\
\sin(\phi^{<t>})\\\ 1\\\ \frac{1}{L}\tan(\delta^{<t>})\end{bmatrix}\Delta t$
(2)

where $\delta$ is the steering angle, $L$ is the length of the vehicle and
$\Delta t$ is the sampling time.

The predictions are integrated by taking into consideration the vehicle’s
dynamics from Eq. 2, so that obstacle information is projected onto the
occupancy grid according to the ego-vehicle’s state for each predicted set of
observations. In the following, we detail ObserveNet’s deep neural network
architecture and the methodology within the temporal state trajectory planner.

###  III-C ObserveNet Architecture and Training

The goal of the ObserveNet neural network from Fig. 2 is to predict future
observations $\mathbf{\hat{I}}^{<t+1,t+\tau_{o}>}$ based on past sensory
measurements $\mathbf{I}^{<t-\tau_{i},t>}$, the global reference route and the
past vehicle’s state trajectory $\mathbf{z}^{<t-\tau_{i},t>}$. This past and
predicted data is used to calculate the reference of the constrained NMPC
controller which will execute the motion of the car.

Figure 3: ObserveNet deep neural network architecture. The DNN receives input
data in the form of historical observations, vehicle states and reference
route, with the goal of computing a safe desired state trajectory
$\mathbf{z}^{<t+1,t+\tau_{o}>}_{d}$ using predicted observations and temporal
path planning. ObserveNet computes a dynamic output, having a size dependent
on the time horizon for which the future observations are predicted.

In order to increase the computational speed, the input Lidar point clouds are
firstly converted into an OcTree representation. An octree is a data structure
used for representing 3D spatial features by subdividing them recursively into
eight child units. The obtained 3D grid map can be calculated at a finer or
more coarse resolution, while preserving a low-memory footprint, making the
representation suitable for embedded computing. Within the conversion loop, we
use several downsampling mechanisms, as follow.

First, we introduced a constraint for points which lay farther than a distance
threshold:

$p_{r}=p[i],\iff d(s,p[i])<T_{d},\forall i\in[1,N]$ (3)

where $p[i]$ and $p_{r}$ are input and output point clouds, respectively.
$d(s,p[i])$ is the Euclidean distance between the ego-vehicle’s origin and the
current point, $T_{d}$ is the distance threshold and $N$ is the original point
cloud size.

Secondly, points which are below a certain height or higher than our ego-
vehicle’s height are discarded:

$p_{r}=p[i],\iff V_{L}\leq p[i]_{Z}\leq V_{H},\forall i\in[1,N]$ (4)

where $p[i]_{Z}$ is height of point, $V_{L}=0.25m$ is the minimum height,
$V_{H}$ is the ego-vehicle height and $N$ is the point cloud size. The goal of
$V_{L}$ is to filter our the pointclouds representing the road.

Finally, points which are outside of an $160^{\circ}$deg field of view from
the front and rear of the vehicle are filtered out:

$p_{r}=p[i]\iff|\angle(p[i],V_{O})|\leq T_{v}\forall i\in[1,N]$ (5)

where $V_{O}$ is the vehicle origin, $T_{v}$ is the field of view threshold
and $N$ is the point cloud size. The problem of slopes, when the road would be
visualized as obstacles in the pre-processed pointcloud, can be easily
counteracted by updating the extrinsic parameters of the Lidar, which in turn
relate its orientation with respect to the vehicle.

The obtained OcTree is stored as historical data in the Augmented Memory
component from Fig. 2. In order to further increase the computational speed on
ObServeNet, we project the OcTree onto a top-view 2D occupancy grid which is
feed to the deep network. The network outputs observation predictions for
$\tau_{o}$ future frames.

ObserveNet’s architecture is shown in Fig. 3. The basic building blocks are
convolutional layers for spatial data encoding and Gated Recurrent Units (GRU)
for extracting temporal dependencies. The input sequences are normalized and
fed through the network’s convolutional layers. The proposed DNN has a dynamic
output, dependent on the length of the prediction horizon $\tau_{o}$. Namely,
each predicted observation $\hat{\mathbf{I}}^{<t+k>}$ is given by a separate
network output branch $k$, where $k\leq\tau_{o}$.

As illustrated in Fig. 3, apart from the differentiable layers in the network,
that is, its convolutional, recurrent and dense units, the architecture also
embodies non-differentiable layers used to map each predicted observation to
the most probable future state of the vehicle. The Plan & Simulate layer
recursively uses our temporal planner to simulate future states, while
replanning the vehicle desired trajectory $\mathbf{z}^{<t+1,t+\tau_{o}>}_{d}$.
The predicted states are smoothed using an extended Kalman filter. A couple of
observations predictions an their ground truth are shown in Fig. 4, as well as
in Fig. 5 in the context of the temporal planner.

Figure 4: Predicted and ground truth observations at three different future
timesteps. (top) Ground truth observations superimposed on ObserveNet
predictions, along with the reference path marked in red and the planned state
trajectory illustrated in green. For visualization purposes, we have
considered only observations within a $50m$ range. (bottom) Images from the
front of the vehicle at the considered timestamps.

ObserveNet has been trained for $10.000$ epochs in a self-supervised fashion
using the Adam optimizer, with learning rate of 0.0003, while minimizing the
mean squared error loss function between measured and predicted observations.

###  III-D Temporal State Trajectory Planning and Control

The temporal path planner is based on the Dynamic Window Approach (DWA)
method, modified to iteratively compute optimal vehicle paths based on
current, as well as predicted observations. The Dynamic Window Approach (DWA)
is an online collision avoidance strategy for mobile robots, which uses robot
dynamics and constraints imposed on the robot’s velocities and accelerations
to calculate a collision free path in the 2D plane. We have implemented DWA
based on the Robot Operating System (ROS) DWA local planner. DWA takes as
input the distances from the ego-vehicle to the obstacles present in the
scene, calculated as perception rays. In our case, we tune DWA to take into
consideration both past, as well as future predicted information.

Figure 5: Temporal state trajectory planning using RovisLab AMTU. While
tracking the red reference path, ObserveNet predicts the future observations
marked with blue, green and yellow, respectively, as well as the most probable
state trajectory of the vehicle. The blue lines represent candidate
trajectories at each future timestamp, while the green line represents the
output of the temporal planner. The white rectangles show predicted
observations of the person depicted on the left, considered in this case as a
moving obstacle.

A desired vehicle state is iteratively predicted at each time step in the
future, based on the temporal vehicle dynamics and observations. The input
data is both historical ($<t-\tau_{i},t>$), as well as predicted
($<t,t-\tau_{o}>$):

$\mathbf{z}^{<t+i>}=f(\hat{\mathbf{I}}^{<t-\tau_{i},t+i>},\mathbf{z}^{<t+i-1>})$
(6)

where $i$ is the prediction timestep and $f({\cdot})$ if the temporal path
planning function. $f({\cdot})$ is used to interpolate the outputs of the DWA
planner along the time horizon $[t-\tau_{i},t+\tau_{o}]$ and can be decomposed
as:

$f({\cdot})=\sum_{t=-\tau_{i}}^{0}DWA(\mathbf{I}^{<t>},\mathbf{z}^{<t>})+\sum_{t=1}^{\tau_{o}}DWA(\hat{\mathbf{I}}^{<t>},\hat{\mathbf{z}}^{<t>})$
(7)

where the first term represents the interpolated trajectories of the planner
given past observations, while the second term computes the interpolation for
the planned trajectories based on predicted observations. The states are
iteratively integrated into the desired state trajectory
$\mathbf{z}^{<t+1,t+\tau_{o}>}_{d}$ feed as input to the NMPC.

A snapshot of our temporal path planner can be seen in Fig. 5. The planner
generates candidate trajectories at each prediction step (shown with blue
lines in Fig. 5). The optimal trajectory is selected based on the vehicle
dynamics from Eq. 2 and predicted observations. The final trajectory computed
using Eq. 7, marked with green in Fig. 5.

##  IV Experiments

The performance of ObserveNet Control was benchmarked against the classical
Dynamic Window Approach (DWA) baseline method, as well as against two state-
of-the-art methods based on imitation learning, that is, Learning by Cheating
(LBC) [18] and World on Rails (WOR) [8]. Since ObserveNet is a derivation of
imitation learning, we have chosen to compare it to LBC and WOR, which are
methods that do not require object-bound boxes, tracks and HD maps, like [16]
or [17]. DWA and ObserveNet use the same Constrained NMPC controller for
executing the motion of the vehicle.

The evaluation of the four competing algorithms was performed on aggressive
driving scenarios simulated in the city environments Town 1 to 6 in CARLA, as
well as on CARLA Leaderboard [19]. Real-world experiments have been performed
using DWA and ObserveNet on indoor and outdoor navigation using the RovisLab
Autonomous Mobile Test Platform (AMTU) from Fig. 6.

Figure 6: RovisLab AMTU (Autonomous Mobile Test Unit). The platform is used in
the context of a 1:4 scaled car model.

Global reference trajectories were recorded while controlling both simulated
and real vehicles manually and used afterwards as reference trajectories for
the autonomous vehicle controller to track.

As performance measures, we have recorded the number of collisions $N_{c}$
between the ego-vehicle and other traffic participants, the cross-track error
$e_{ct}$, the vehicle’s orientation $e_{\phi}$, the rate of change of the
steering and acceleration commands $e_{\delta}$ and $e_{a}$, respectively, as
well as the ability of the vehicle to successfully complete the pre-recorded
track, quantified as $C_{100\%}$.

The cross-track error represents the difference between the reference
trajectory and the vehicle’s position along the $y$ coordinate:

$e_{ct}=g(x)-y$ (8)

where $g(x)$ represents the polynomial approximation of the trajectory,
evaluated at $x$.

The vehicle orientation error is defined as:

$e_{\phi}=\phi-\phi_{d}+\frac{v}{L}\delta$ (9)

where $\phi$ is the vehicle orientation, $\phi_{d}$ is the desired vehicle
orientation, $v$ is the ego-vehicle velocity, L is the vehicle wheelbase and
$\delta$ is the steering angle.

The rate of change of the steering and acceleration commands is computed as:

$\begin{cases}e_{\delta}&=\delta^{<t+1>}-\delta^{<t>}\\\
e_{a}&=a^{<t+1>}-a^{<t>}\end{cases}$ (10)

Figure 7: Mean (solid line) and standard deviation (shaded region) of the
performance metrics recorded during testing in the five considered scenarios.
Per sampling time cross-track error, orientation error, consecutive
acceleration difference and consecutive steering angle difference analysis.
From left to right: overtake, opposite direction, side cut-off left, side cut-
off right and multiple vehicle scenarios statistics.

The workflow of the experiments is as follows:

  * •

collect training data from driving recordings;

  * •

format training data as historical sequences of length $N$ and prediction
sequences of length $\tau_{o}$;

  * •

train the ObserveNet deep network from Fig. 3;

  * •

evaluate on simulated and real-world driving scenarios.

To train the network, we use data collected from CARLA and RovisLab AMTU, with
the ego-vehicle being manually driven in the test environment, while
encountering various traffic participants. In order to to mitigate
overfitting, all vehicles were driven in a different manner each time. During
training, we have used an 80/10/10 train - validation - test data split.

###  IV-A Simulation Experiments

CARLA [20] was used for testing ObserveNet Control in simulation. We have
simulated a $360^{\circ}$ Lidar sensor, spanning a maximum range of $50$m.
Additionally, camera images, inertial data and odometry information was
acquired.

Due to the high flexibility of the CARLA simulator, we were able to simulate
real-world situations like overtaking, opposite direction lane swerving, side
cut-off, etc. During the evaluation step, our algorithm was used to control
the ego-vehicle, while the other traffic participants were controlled using a
simple feedback loop directly in CARLA.

Towns 1-6 from CARLA are used for training and testing, amounting to $90$K
pairs of Lidar and vehicle state data. We have investigated city driving with
two or more simulated vehicles driving in the environment. We tackle
aggressive driving, mainly overtaking and cut-off situations, where a classic
controller would have difficulties due to the scene’s high dynamics. The ego-
vehicle navigated a pre-recorded trajectory, while several challenging
conditions were added. Other traffic participants were simulated as follows:
i) travelling at a dangerously low speed, leading to the need of aggressive
overtaking maneuvers, ii) vehicles incoming from the opposite direction at an
unsafe distance and iii) vehicles cutting off the ego-vehicle from the left
and right sides.

The goal of our experiments was to check if we can eliminate collisions, as
well as to smooth the state trajectory, thus generating a safer more pleasant
ride, with less jerking. We have implemented five challenging scenarios, with
quantitative results shown in Fig. 7.

The overtake scenario represents a challenging situation, particularly at
higher speed. The other vehicle accelerates at a slower pace than the ego-
vehicle, towards a lower top speed, generating an overtake situation. State
data is recorded for generating statistics.

In the opposite direction experiment, the ego-vehicle travels straight, while
the other vehicle is coming from the opposite direction, spawning a near-miss
situation. Using the network’s predictions, the ego-vehicle should sense the
other vehicle sooner than the setup without predictions, generating a safer
trajectory.

The side cut-off scenario consists of the ego-vehicle travelling straight and
approaching an intersection. When it comes close to the intersection, another
vehicle cuts it off, generating a collision. Using the predicted data, the
ego-vehicle needs to be able to avoid the collision. This scenario was tested
in two symmetrical situations: left side cut-off and right side cut off.

The rear pass-by scenario simulates a vehicle coming from behind the ego-
vehicle on the same side, at a greater velocity, passing it at an unsafe
distance. Using the predictions, the other vehicle’s position is predicted,
thus allowing the ego-vehicle’s path planner to plan accordingly, widening the
gap between the vehicles.

The multi-vehicle scenario has the most complexity, where the ego-vehicle is
traveling a larger map section, encountering vehicles, overtaking and avoiding
collisions. The scenario is designed to simulate real world complex scenes,
where the ego-vehicle has limited maneuver space.

For all five experimental scenarios, the corresponding quality measures are
summarized in Table I. The experiments show that ObserveNet achieves better
results than the regular DWA planner, providing less jerky movements. This is
due to the fact that the obstacles are available before being actually
observed.

The ego-vehicle successfully avoided obstacles in all five experimental
scenarios. The cross-track error remained relatively low, although in some
circumstances it was greater than the DWA baseline for all three imitation
learning methods (e.g. when running the side cut-off or multi-vehicle
scenarios). This behavior happens when the ego-vehicle is overtaking, thus
deviating from the reference trajectory. We believe that the lower performance
of ObserveNet, LBC and WOR against DWA in the side cut-off and multi-vehicle
scenarios is due to the randomness of these scenarios, as well as due to the
high dynamics of the involved participants. An additional amount of training
data should increase performance.

The overtaking maneuvers also cause spikes in orientation error levels. When
running the overtake scenario solely with DWA, the ego-vehicle collided
violently with the other traffic participant, thus causing large cross-track
and orientation errors. Consecutive acceleration and steering angle
differences were in some instances better without predictions, however the
error values were very close.

Scenario | Algorithm | $N_{C}$ | $\bar{e_{ct}}$ | $\bar{e_{\phi}}$ | $\bar{e_{a}}/\bar{e_{\delta}}$ | $C_{100\%}$
---|---|---|---|---|---|---
Overtake | DWA | 1 | 0.06 | 0.02 | 0.001 / 0.001 | 100
LBC | 0 | 0.076 | 0.027 | 0.0069 / 0.0023 | 100
WOR | 0 | 0.065 | 0.019 | 0.0061 / 0.0012 | 100
Ours | 0 | 0.6 | 0.00001 |  0.0006 / 0.0006 | 100
Opposite | DWA | 1 | 0.72 | 0.003 | 0.006 / 0.001 | 100
LBC | 0 | 0.2 | 0.0023 | 0.042 / 0.005 | 100
WOR | 0 | 0.17 | 0.04 | 0.031 / 0.01 | 100
Ours | 0 | 0.13 | 0.004 |  0.0008 / 0.0008 | 100
Side cut-off | DWA | 1 | 0.002 | 0.06 |  0.002 / 0.002 | 100
LBC | 1 | 0.10 | 0.23 | 0.064 / 0.020 | 100
WOR | 0 | 0.05 | 0.04 | 0.0096 / 0.010 | 100
Ours | 0 | 0.003 | 0.005 | 0.004 / 0.001 | 100
Rear pass-by | DWA | N/A | 1.57 | 0.007 | 0.01 / 0.003 | 100
LBC | N/A | 0.45 | 0.02 | 0.063 / 0.020 | 100
WOR | N/A | 0.62 | 0.005 | 0.0067 / 0.010 | 100
Ours | N/A | 0.39 | 0.02 |  0.004 / 0.004 | 100
Multi-vehicle | DWA | 4 | 0.005 | 0.04 | 0.0003 / 0.0003 | 24.56
LBC | 1 | 0.006 | 0.34 |  0.0002 / 0.0004 | 100
WOR | 1 | 0.006 | 0.13 | 0.00047 / 0.0011 | 100
Ours | 0 | 0.01 | 0.01 | 0.0005 / 0.0005 | 100

Table I: Quantitative results on simulated scenarios in CARLA.

###  IV-B CARLA Leaderboard Experiments

On top of our own CARLA simulation experiments, we have also assessed
ObserveNet within Carla Autonomous Driving Leaderboard [19], which is an open
platform designed to ease testing of autonomous driving systems. The platform
provides several urban environments, as well as varying weather and lighting
conditions and can be configured to simulate a variety of challenging traffic
scenarios, such as vehicle control loss, obstacle avoidance, lane changing,
crossing traffic running a red light and intersection handling. Three metrics
are used for performance evaluation: total driving score, route completion and
infraction penalty. In addition to these metrics, the framework provides
collision information, off-road driving data, route deviations and running red
lights and stop signs.

Since our focus is on eliminating collisions between the ego-vehicle and other
traffic participants, we aim to optimize the number of collisions, route
deviations and route completion. We have conducted a total of $40$ routes,
each repeated three times for each evaluated algorithm, comparing side by side
collision metrics and other infractions, as shown in Fig. 8. Although the
number of infractions like stop sign violations and red light violations are
more frequent in our approach, the average number of collisions remained
overall lower than for LBC and WOR.

From Fig. 8 it can be observed that LBC and WOR tend to perform better than
our approach for stop sign and red light violations. This is happening because
we use range sensing only, where stop signs and traffic lights are not
recognized.

Figure 8: Experimental results on CARLA Leaderboard [19]. Average number of
infractions for all tested routes. From left to right: layout collisions
($C_{Layout}$), pedestrian collisions ($C_{Ped}$), vehicle collisions
($C_{Veh}$), red light violations ($R_{Violation}$), stop sign violations
($S_{Violation}$) and blocked vehicle violations ($V_{Blocked}$).

###  IV-C Ablation Study

In order to asses the core components of the algorithm, we have performed the
following ablations to the ObserveNet structure from Fig. 3: i) ablation of
the Kalman filter, ii) ablation of the prediction horizon and iii) control
based on the prediction of the next vehicle’s state. In the last case, the
Kalman filter and the temporal planner were removed, together with the
$[t+2,t+\tau_{o}]$ branches. The kept $t+1$ branch was trained to predict only
the next state of the vehicle based on $[t-\tau_{i},t]$ observations. Fig.
9(a) shows abblations against the cross-track error $e_{ct}$, while Fig. 9(b)
takes into consideration pedestrian and vehicle collisions.

Figure 9: Ablation study results. The Kalman filter was switched on/off, while
the prediction horizon was set to $1s$, $2s$, $5s$, $10s$ and $20s$,
respectively.

The cross-track error $e_{ct}$ is proportional to the length of prediction
horizon $\tau_{o}$. The further in time we predict observations, the larger
the cross-track error is. Our intuition for this increase in $e_{ct}$ is that
ObserveNet is anticipating the visual dynamics of the obstacles encoded in the
raw measurements, thus optimizing the vehicle’s trajectory against collisions,
while diverging from the reference path. This phenomenon can be seen in Fig.
9(b), where the pedestians and other vehicles collisions are decreasing
proportional to the length of $\tau_{o}$.

Without the smoothing property of the Kalman filter, the values of the control
signals are higher, leading to a larger error. A decrease in performance is
observed when the vehicle is controlled solely by predicting its next state.

###  IV-D RovisLab AMTU Experiments

In addition to the CARLA simulated experiments, we have tested ObserveNet and
DWA on real-world navigation tasks using the RovisLab AMTU robot from Fig. 6.
Due to its better performance over LBC and WOR, we have chosen to conduct
real-world experiments only with ObserveNet as the learning-based controller
and DWA. RovisLab AMTU is an AgileX Scout 2.0 platform which acts as a 1:4
scaled car, equipped with a $360^{\circ}$ Hesai Pandar 40 Lidar, 4x e-130A
cameras providing a $360^{\circ}$ visual perception of the surroundings, a
VESC inertial measurement unit, GPS and an NVIDIA AGX Xavier board for data
processing and control. The robot navigated indoor and outdoor environments,
while avoiding dynamic obstacles.

The prediction horizon $\tau_{o}$ was set to $5s$ based on trial-and-error
experiments, while $10km$ of training data was acquired for the indoor and
outdoor experiments each. Both real-world experiments have been conducted at
Transilvania University’s research institute, amounting to $38.000$ pairs of
Lidar and vehicle state data.

We have replicated similar scenarios as the ones performed in simulation, with
the difference that the moving vehicles from CARLA have been replaced by
persons interacting with the robot in similar ways as in the five testing
schemes described above. Table II shows quantitative results on the real-world
experiments. The results show an increased improvement in tracking accuracy
and obstacle avoidance in the case of ObserveNet Control.

Scenario | Algorithm | $N_{C}$ | $\bar{e_{ct}}$ | $\bar{e_{\phi}}$ | $\bar{e_{a}}/\bar{e_{\delta}}$ | $C_{100\%}$
---|---|---|---|---|---|---
Overtake | DWA | 1 | 0.27 | 0.055 | 0.0022 / 0.0022 | 100
Ours | 0 | 0.55 | 0.001 |  0.0004 / 0.0004 | 100
Opposite | DWA | 1 | 1.32 | 0.006 | 0.005 / 0.002 | 100
Ours | 0 | 0.19 | 0.002 |  0.0007 / 0.0007 | 100
Side cut-off | DWA | 1 | 0.12 | 0.1 | 0.003 / 0.004 | 100
Ours | 0 | 0.09 | 0.007 |  0.001 / 0.001 | 100
Rear pass-by | DWA | N/A | 2.23 | 0.06 | 0.009 / 0.0025 | 100
Ours | N/A | 0.72 | 0.09 |  0.006 / 0.006 | 100
Multi-obstacles | DWA | 3 | 0.035 | 0.08 | 0.009 / 0.008 | 70.32
Ours | 0 | 0.067 | 0.01 |  0.005 / 0.005 | 100

Table II: Quantitative results on RovisLab AMTU.

##  V Conclusions

This paper introduces ObserveNet Control, which is an observation prediction
control method for autonomous vehicles composed of a DNN for observations
predictions and a temporal path planner. We have shown that our control
approach has an improved path tracking and obstacles avoidance accuracy when
augmented with predicted observations. Additionally, the training of the
system is self-supervised from driving recordings, which requiring manual
human annotations. Because we only predict raw observations, one drawback of
ObserveNet is that it lacks driving context, hence resulting in a large number
of traffic violations. This can be easily corrected by integrating our method
into autonomus driving systems that use visual data to detect additional cues,
such as lanes, or traffic signs. In such a case, ObserveNet would contribute
with its raw observations predictions to the reconstructed virtual environment
model used by the vehicle to plan its path. As future work, we plan to
increase the time prediction horizon, as well as its possible application in
mobile robotics and simultaneous localization and mapping.

## References

  * [1] S. Pendleton, H. Andersen, X. Du, X. Shen, M. Meghjani, Y. Eng, D. Rus, and M. Ang, “Perception, planning, control, and coordination for autonomous vehicles,” _Machines_ , vol. 5, no. 1, p. 6, 2017.
  * [2] M. G. Azar, B. Piot, B. A. Pires, J. Grill, F. Altché, and R. Munos, “World discovery models,” _CoRR_ , vol. abs/1902.07685, 2019.
  * [3] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, _et al._ , “End to end learning for self-driving cars,” _arXiv:1604.07316_ , 2016.
  * [4] M. Jaritz, R. Charette, M. Toromanoff, E. Perot, and F. Nashashibi, “End-to-end race driving with deep reinforcement learning,” in _Int. Conf. on Robotics and Automation ICRA_ , Brisbane, Australia, 2018.
  * [5] F. Codevilla, M. Müller, A. López, V. Koltun, and A. Dosovitskiy, “End-to-end driving via conditional imitation learning,” in _Int. Conf. on Robotics and Automation (ICRA)_ , 2018, pp. 4693–4700.
  * [6] N. Rhinehart, R. McAllister, and S. Levine, “Deep imitative models for flexible inference, planning, and control,” _CoRR_ , vol. abs/1810.06544, 2018.
  * [7] Y. Ding, C. Florensa, P. Abbeel, and M. Phielipp, “Goal-conditioned imitation learning,” in _Advances in Neural Information Processing Systems_ , vol. 32. Curran Associates, Inc., 2019\.
  * [8] D. Chen, V. Koltun, and P. Krähenbühl, “Learning to drive from a world on rails,” in _arXiv preprint_ , 2021.
  * [9] A. Rudenko, L. Palmieri, M. Herman, K. M. Kitani, D. M. Gavrila, and K. O. Arras, “Human motion trajectory prediction: A survey,” _Int. Journal of Robotics Research_ , vol. 39, no. 8, pp. 895–935, 2020.
  * [10] W. Lotter, G. Kreiman, and D. Cox, “Deep predicting coding networks for video prediction and unsupervised learning,” _ArXiv_ , vol. abs/1605.08104, 2016\.
  * [11] K. Luo, S. Casas, R. Liao, X. Yan, Y. Xiong, W. Zeng, and R. Urtasun, “Safety-oriented pedestrian motion and scene occupancy forecasting,” _ArXiv_ , vol. abs/2101.02385, 2021.
  * [12] G. Oh and J. Valois, “HCNAF: Hyper-Conditioned Neural Autoregressive Flow and its Application for Probabilistic Occupancy Map Forecasting,” _IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [13] N. Mohajerin and M. Rohani, “Multi-step prediction of occupancy grid maps with recurrent neural networks,” in _IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [14] M. Schreiber, S. Hoermann, and K. Dietmayer, “Long-term occupancy grid prediction using recurrent neural networks,” 05 2019.
  * [15] X. Weng, J. Wang, S. Levine, K. Kitani, and N. Rhinehart, “Inverting the pose forecasting pipeline with spf2: Sequential pointcloud forecasting for sequential pose forecasting,” _arXiv_ , 2020.
  * [16] M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun, “Learning lane graph representations for motion forecasting,” in _European Conference on Computer Vision (ECCV)_ , 2020.
  * [17] J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid, “Vectornet: Encoding hd maps and agent dynamics from vectorized representation,” in _IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 11 522–11 530.
  * [18] D. Chen, B. Zhou, V. Koltun, and P. Krähenbühl, “Learning by cheating,” in _Proceedings of the Conference on Robot Learning_ , ser. Proceedings of Machine Learning Research, nov 2020, pp. 66–75.
  * [19] “Carla autonomous driving leaderboard [online].”
  * [20] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “CARLA: An open urban driving simulator,” in _Proceedings of the 1st Annual Conference on Robot Learning_ , 2017, pp. 1–16.