# Path Following and Stabilisation of a Bicycle Model using a Reinforcement
Learning Approach

Sebastian Weyrera, Peter Manzla, A. L. Schwabb, Johannes Gerstmayra$\ast$ ∗
Corresponding author. E-mail<EMAIL_ADDRESS>aDepartment of
Mechatronics, University of Innsbruck, Innsbruck, Austria; bDepartment of
Biomechanical Engineering, Delft University of Technology, Delft, The
Netherlands

###### Abstract

Over the years, complex control approaches have been developed to control the
motion of a bicycle. Reinforcement Learning (RL), a branch of machine
learning, promises easy deployment of so-called agents. Deployed agents are
increasingly considered as an alternative to controllers for mechanical
systems. The present work introduces an RL approach to do path following with
a virtual bicycle model while simultaneously stabilising it laterally. The
bicycle, modelled as the Whipple benchmark model and using multibody system
dynamics, has no stabilisation aids. The agent succeeds in both path following
and stabilisation of the bicycle model exclusively by outputting steering
angles, which are converted into steering torques via a PD controller.
Curriculum learning is applied as a state-of-the-art training strategy.
Different settings for the implemented RL framework are investigated and
compared to each other. The performance of the deployed agents is evaluated
using different types of paths and measurements. The ability of the deployed
agents to do path following and stabilisation of the bicycle model travelling
between $2\text{\,}\mathrm{m}\mathrm{s}^{-1}$ and
$7\text{\,}\mathrm{m}\mathrm{s}^{-1}$ along complex paths including full
circles, slalom manoeuvres, and lane changes is demonstrated. Explanatory
methods for machine learning are used to analyse the functionality of a
deployed agent and link the introduced RL approach with research in the field
of bicycle dynamics.

###### keywords:

reinforcement learning, path following, stabilisation, bicycle model,
curriculum learning, machine learning

##  1 Introduction

Bicycle dynamics is a discipline within dynamics studying the motion of
bicycles under the influence of forces. Understanding bicycle dynamics is
essential for developing controllers for bicycle models.

Traditionally, controllers for mechanical systems like bicycles are designed
using principles from control engineering. Alongside the approaches from
control engineering, there is an emerging interest in alternative approaches
using machine learning methods to control mechanical systems. There is an
increasing trend in multibody systems to apply machine learning methods, in
the range between feed-forward networks for surrogate models [1] and large
language models to model dynamic systems [2]. In particular, Reinforcement
Learning (RL), a sub-area of machine learning, has been successfully used to
solve control tasks with various multibody systems [3]. The idea of RL is that
an agent learns to solve a task by observing the dynamic environment,
receiving a reward signal, and returning actions to the dynamic environment
without being supervised [4].

This raises the question of whether RL could serve as a viable approach for
controlling bicycle models, similar to the controllers designed using
traditional control engineering methods. A possible application of the RL
approach alongside conventional control engineering approaches for controlling
bicycles is not only of theoretical interest but could also be in practical
robotic systems in the future. Against the background of this question, the
state-of-the-art is considered from two perspectives. Firstly, research on the
control of the motion of a bicycle, focusing on control engineering methods,
is presented below. Secondly, existing work on using RL and related approaches
with bicycle models is considered.

The control of the motion of a bicycle has attracted many researchers over the
years [5]. Probably because of the speed-dependent lateral stability and the
non-minimal phase behaviour, that is, to stabilise a bicycle one has to steer
into the direction of the undesired fall [6]. Most of the attention has been
directed towards stabilising the lateral dynamics of the bicycle and not so
much on path following. A good overview on the dynamics and control of a
bicycle is presented in [7]. After 2013, more work emerged on the control of
the bicycle. The focus of the present work is on path tracking, which of
course must include lateral stability control. A passive velocity field
controller for path following has been introduced by Yin and Yamakita [8].
Their system is then passive with respect to external force perturbations.
Turnwald et al. [9] introduce a passivity-based path following control for a
bicycle. The steering and forward speed are controlled using a generalised
canonical transformation on their port-Hamiltonian system of the bicycle. A
classic multi-loop control structure is used by Shafiei and Emami [10] for
path following of an unmanned bicycle. A classic LQR based controller for
straight path following and cornering for a bicycle robot has been designed
and built by Seekhao et al. [11], where the lateral dynamics of the bicycle is
stabilised by an additional non-inverted pendulum. A model predictive
controller with constraints on the lean, steer, heading as well as position of
the bicycle is used by Persson et al. [12] for the design of a path following
controller for a riderless bicycle. He et al. [13] present a learning based
control framework for path following of a bicycle robot. For path following
both steer and forward speed actuation are used. The lateral dynamics of the
bicycle robot is stabilised by a controlled non-inverted pendulum.

The idea of using an RL approach to control bicycle models is not new. An
article from Randlov and Alstrøm [14] published in 1998 shows that RL can be
used to keep a bicycle model upright and steer it to a target point that is
$1000\text{\,}\mathrm{m}$ away from the initial position of the vehicle. The
paper suggests using shaping, where two separate neural networks are used. One
net keeps balance and the other steers the bicycle towards the target point.
The path the bicycle can take to the target point is not predefined. A good 20
years later, the articles from Le and Chung [15] and Le et al. [16] show that
shaping is not necessary when further evolved RL algorithms are used, as a
single agent learns to steer the bicycle model to the target point without it
falling over. The latter two works mainly adopt the bicycle model from the
work [14]. This bicycle model is characterised by not having a trail. The
trail is a geometric property of a bicycle, substantially contributing to the
speed-dependent lateral stability of the bicycle [5]. In an approach that does
not use RL per se, but shows that neural networks, in general, can be useful
for controlling bicycles, Cook [17] manually devised a neural network
controller with only two neurons as an example of a simple human bicycle
tracking controller. The first neuron is a proportional controller on the
heading with a threshold function. It outputs a desired roll angle, which is
an input for the second neuron which in turn outputs a steering torque based
on PD control. The desired heading is set using waypoints enabling the bicycle
to perform tracking tasks. He finds that the controller is relatively robust
as gain values weakly depend on speed and do not have to be perfectly adjusted
to the specific bicycle. The controller works at a range of velocities, but it
fails at low speed. A work from Zhu et al. [18] uses an RL approach to steer a
bicycle model along a predefined path, but does not consider lateral
stabilisation of it. The therein used bicycle model has an integrated inertia
wheel to enhance its stability. Taken together, the referenced works do not
show that the RL approach can be used both to stabilise the bicycle model and
to do path following, as they specialise in either one of these tasks.
Moreover, the aforementioned works on using RL to stabilise a bicycle model do
not clarify whether RL can induce a speed-dependent lateral stabilisation
policy, as required for riding the positive trail bicycle we ride every day.

The objective of the present work is to show for the first time that an RL
approach can be used to do path following with a bicycle model while
simultaneously stabilising it laterally. In the present work, the bicycle
model has no balance aids such as inertia wheels or non-inverted pendulums.
The agent must learn to do path following and stabilisation of the bicycle
model exclusively by turning the handlebar of the bicycle. The bicycle in the
present work is modelled using the so-called Whipple bicycle [19], which is
regarded as a benchmark model to study bicycle dynamics [20]. The Whipple
bicycle reflects the dynamic characteristics of the bicycle found on the road
today, having a positive trail and being unstable when moving at low forward
velocities and self-stable when moving at a specific speed range [5]. This
brings another novelty to the present work, namely the investigation of
whether an RL approach is able to induce a speed-dependent policy to stabilise
a bicycle model having a positive trail. A further feature of the present work
is that the agent is not allowed to change the forward velocity of the bicycle
model by itself but must be able to cope with different forward velocities,
underlining the practical significance of the present work by addressing a
system in which the velocity of the bicycle is set by the cyclist while the
agent does path following and stabilisation of the bicycle model by doing
steering movements. Another specification of the present work is that the
trained agent must be able to steer the bicycle model along arbitrary paths
containing full circles, polynomials, straight lines, and curves. By using
explanatory methods for machine learning, the present work aims to verify that
the agents learn the basic mechanisms of controlling a bicycle.

An additional aspect of the present work is that the environment is virtual,
as such that the bicycle is implemented as a multibody model. Using a virtual
environment is accompanied by simplifications resulting from the bicycle model
and its surrounding. It is assumed that the bicycle model drives on a
perfectly levelled surface.

The simplifications and mandatory preparatory work associated with the bicycle
model to apply an RL approach mark the beginning of this article. Afterwards,
the applied RL framework is described, followed by an outline of the learning
process with the application of curriculum learning as a training strategy.
The subsequent section evaluates how accurately the agents can do path
following and stabilisation of the bicycle model at different forward
velocities and along two different types of paths. Explanatory methods for
machine learning are used to analyse the functioning of the agents. Finally,
the presented results are discussed.

##  2 Bicycle Model and Associated Preparatory Work

The bicycle model used in the present work is the Whipple bicycle [19]. The
Whipple bicycle is often used to study bicycle dynamics [20] and the
linearised equations of motion for the Whipple bicycle are presented in a
benchmark paper [5]. The linearised equations of motion are confirmed by
comparing the eigenvalues of the system predicted by the linearised equations
of motion with experimentally measured eigenvalues of a real bicycle [21]. By
using the Whipple bicycle, the RL approach shown in the present work is
applied to a system of which the control task changes with the forward
velocity. In this section, the bicycle model itself and the preparatory work
that is required to use the bicycle model with the RL approach are described.

Figure 1: Whipple bicycle shown in the reference configuration where it stands
upright without the handlebar being turned. The position of the Center of Mass
(COM) of the rear wheel, the rear body, the handlebar, and the front wheel are
marked, as well as the two ground contact points $\mathrm{P}$ and
$\mathrm{Q}$, the two wheel radii $\mathrm{r_{R}}$ and $\mathrm{r_{F}}$, and
the trail $c$ of the bicycle. The global frame, denoted as $\mathrm{0}$-frame,
is shown.

###  2.1 Characteristics

The Whipple bicycle, shown in Figure 1, consists of four rigid bodies, namely
the Rear wheel (R), the Front wheel (F), the rear Body (B) being the bicycle
frame and the rider, and the Handlebar (H) that includes the fork. The bodies
are interconnected by three frictionless revolute joints, one at the bicycle
head tube and two at the wheel hubs. The Whipple bicycle has a tilted steering
axis and a fork offset. Due to the fork offset of the bicycle, the front wheel
hub is not located on the steering axis, but at a constant distance to it. The
Whipple bicycle has a positive trail $c$. The rider body is assumed to be
rigidly connected to the frame of the bicycle. The wheels are modelled using
geometrically ideal thin discs, each having one contact point with the flat
level ground. The contact points of the wheels with the ground are called
$\mathrm{P}$ and $\mathrm{Q}$. The wheels are slip-free, both in the direction
of travel and to the side. The geometrical and mechanical parameters of the
bicycle model are taken from [5]. The bicycle model is subjected to
gravitational acceleration. The bicycle model is self-stable when travelling
at a forward velocity between $4.3\text{\,}\mathrm{m}\mathrm{s}^{-1}$ and
$6\text{\,}\mathrm{m}\mathrm{s}^{-1}$ [5], as initial perturbations of the
bicycle model subside over time if moving in this range for the forward
velocity.

###  2.2 Minimal Coordinates

To use RL, a state must be defined which describes the environment in such a
way that the agent can select a suitable action [4]. In the present work, the
state contains the minimal coordinates of the bicycle model. For the degree of
freedom on positional base, $f_{\mathrm{L}}=6n-b_{\mathrm{L}}=24-17=7$
follows, where $n$ is the number of rigid bodies of the system and
$b_{\mathrm{L}}$ the number of geometric constraints in the system [5]. The
number of geometric constraints is the sum of the number of constraints
resulting from the revolute joints plus two, as the wheels cannot penetrate
the ground. In Figure 2 the minimal coordinates on positional base of the
system are illustrated. Since the bicycle has non-holonomic constraints, the
degree of freedom on velocity base is reduced to
$f=f_{\mathrm{L}}-b^{*}=7-4=3$, where $b^{*}$ is the number of non-holonomic
constraints [5]. Per wheel, two non-holonomic constraints are added since
neither side-slipping nor free spinning of the wheels is possible. In Table
2.2 the minimal coordinates on positional and velocity base are given. The
vector $\mathbf{q}\in\mathbb{R}^{10}$ of the minimal coordinates reads

$\mathbf{q}=\left[\begin{array}[]{cccccccccc}x_{\mathrm{P}}&y_{\mathrm{P}}&\Psi&\varphi&\delta&\theta_{\mathrm{R}}&\theta_{\mathrm{F}}&\dot{\varphi}&\dot{\delta}&\dot{\theta}_{\mathrm{F}}\end{array}\right]^{\mathrm{T}}\,.$
(1)

Figure 2: Whipple bicycle drawn with its minimal coordinates on positional
base and the pitch angle $\theta_{\mathrm{B}}$. The upright cylinder marked
with $\Psi$ represents the yaw angle, the mounting attached to the rear
dropout is used to illustrate the roll angle $\varphi$ that is independent of
the pitch angle $\theta_{\mathrm{B}}$ of the rear body. Note that the pitch
angle $\theta_{\mathrm{B}}$ is not a minimal coordinate of the bicycle, but is
needed later for the coordinates mappings.

Minimal coordinates of the Whipple bicycle that are part of the state
describing the environment. The $\mathrm{T_{1}}$-frame used is defined in
Appendix A.  notation description $x_{\mathrm{P}}$ x-coordinate of the global
position of the rear wheel contact point $y_{\mathrm{P}}$ y-coordinate of the
global position of the rear wheel contact point $\Psi$ yaw angle between the
x-axes of the $\mathrm{0}$-frame and the $\mathrm{T_{1}}$-frame around the
z-axis of the $\mathrm{0}$-frame $\varphi$ roll angle around the x-axis of the
$\mathrm{T_{1}}$-frame, where $\varphi=0$ means that the z-axes of the
$\mathrm{T_{1}}$-frame and the $\mathrm{0}$-frame are parallel $\delta$
steering angle $\theta_{\mathrm{R}}$ rear wheel rotation angle
$\theta_{\mathrm{F}}$ front wheel rotation angle $\dot{\varphi}$ roll angular
velocity $\dot{\delta}$ steering angular velocity $\dot{\theta}_{\mathrm{F}}$
front wheel rotation angular velocity

###  2.3 Coordinates Mappings

For the multibody model, a redundant formulation is used, representing the
configuration of each rigid body with a translation and rotation. The
redundant formulation is used to make the RL approach expandable in the
future, that is, to use flexible bodies instead of rigid bodies for the
bicycle model or to use an uneven pavement instead of a flat level ground on
which the bicycle model drives. By using a redundant formulation, a mapping of
the redundant coordinates to the minimal coordinates and vice versa must be
established before setting up the RL framework. In Table 2.3 the two mappings
are described, as well as why the two mappings are required.

Description of the coordinates (coord.) mappings that must be established for
using the multibody model of the bicycle in the RL framework.  redundant
coord. $\rightarrow$ minimal coord. minimal coord. $\rightarrow$ redundant
coord. compute the minimal coord. (on positional and velocity base) from the
current configuration of the bicycle since the minimal coord. are part of the
state vector describing the environment; without this transformation the state
of the environment could not be computed compute the redundant coord. (on
positional and velocity base) when the environment is reset since the
environment is reset by randomly initialising the minimal coord. of the
bicycle model; without this transformation the simulation could not restart
after the environment is reset

In Appendix A, the mapping of the redundant to the minimal coordinates with
sensors that are defined in the multibody simulation code and the mapping of
the minimal to the redundant coordinates are shown. The mappings include both
the positional and velocity base.

###  2.4 Model of the Steering Drive

The agent does path following and stabilisation of the bicycle model by
predicting the steering angle $\delta_{\mathrm{set}}$ of the bicycle model.
The prediction for the steering angle is converted into a steering torque
$\tau$ by a PD controller. The steering torque $\tau$ is imprinted between the
rear body and the fork of the bicycle model and acts around the steering axis,
modelling a drive sitting in the bicycle head. The steering torque $\tau$
reads

$\tau=P(\delta-\delta_{\mathrm{set}})+D(\dot{\delta}-\dot{\delta}_{\mathrm{set}})\,,$
(2)

where $\delta$ is the steering angle of the bicycle model and $\dot{\delta}$
is the steering angular velocity. For the set value of the steering angular
velocity $\dot{\delta}_{\mathrm{set}}=0$ applies, as the agent sets the
steering angle without specifying the steering angular velocity
$\dot{\delta}$. The two parameters of the PD controller are set
$P=$9\text{\,}\mathrm{N}\mathrm{m}$$ and
$D=$1.6\text{\,}\mathrm{N}\mathrm{m}\mathrm{s}$$. In Appendix B it is
described how the values for $P$ and $D$ are found.

##  3 Reinforcement Learning Framework

In an RL framework, the agent can choose an action based on the state of the
environment to solve a defined control task. The action chosen by the agent
changes the dynamic environment. The new state and a reward are passed back to
the agent, closing the agent-environment loop, see Figure 3. Based on the new
state, the agent again selects an action. The reward is a numerical signal.
The agent tries to maximise the cumulative reward and should thus learn to
solve the control task. Learning means that the data obtained when passing
through the agent-environment loop is used to adapt the behaviour of the
agent. The method used to adapt the behaviour is defined by the specific RL
algorithm used. In the present work, mainly the off-policy Soft Actor Critic
(SAC) algorithm is used, first introduced in the work [22]. The SAC algorithm
is off-policy, such that it uses a replay buffer, where the data obtained when
running through the RL framework is saved. After every single run through the
RL framework, a batch sampled from the replay buffer is used to adapt the
behaviour of the agent [22]. If the agent is implemented using neural
networks, the concept of RL is extended to Deep Reinforcement Learning (DRL).
Adapting the behaviour of the agent in the context of DRL means that the
weights and biases in the neural networks are adapted [4]. In the present
work, the terms RL and DRL are used synonymously, as all agents are
implemented using neural networks. If an agent is deployed, it can be used in
a similar way to a controller. To deploy an agent when using the SAC algorithm
means that at a selected point during the learning process a neural network of
the agent is saved. The saved neural network is the approximator for the state
value function, predicting an action to be executed depending on the state of
the environment [22]. When deployed, the agent no longer needs the reward
signal and, at best, solves the control task.

Figure 3: Scheme of the RL framework. An agent can choose an action based on
the state of a dynamic environment. A numerical reward signal is passed pack
to the agent.

The framework set up for the present work is Python (version 3.11.5) based.
The multibody simulation code Exudyn111https://github.com/jgerstmayr/EXUDYN
(used version: 1.8.0) is used for the multibody simulation of the bicycle
model. Exudyn is based on C++ and available as Python package [23]. Exudyn
provides an interface class that connects the multibody system to
OpenAIGym222https://github.com/Farama-Foundation/Gymnasium (used version:
0.21.0) [24]. OpenAIGym in turn provides a standardised port to already
existing implementations of RL algorithms. The RL algorithms used in the
present work are part of Stable-Baselines3333https://github.com/DLR-RM/stable-
baselines3 (used version: 1.7.0) [25], which provides implementations of seven
different RL algorithms.

In the following, four main parts of the RL framework are described, namely
the environment, the state, the reward, and the action. At certain points,
several possible settings for the RL framework are given. The different
settings for the RL framework are compared to each other later in the present
work.

###  3.1 Environment

The environment consists of the simulated bicycle model and the path the
bicycle should follow. As outlined in Section 2.3, the multibody model of the
bicycle is implemented using a redundant formalism. The differential algebraic
equations resulting from the redundant formalism and assembled by Exudyn are
solved using the generalised-$\alpha$ solver implemented in Exudyn. The step
size of the solver is set $0.005\text{\,}\mathrm{s}$. The RL algorithm
interacts every $h=$0.05\text{\,}\mathrm{s}$$ with the environment, with $1/h$
giving what is known as controller frequency in control engineering. Thus,
$10$ simulation steps are made until a new action chosen by the agent is
applied to the multibody system. The update time $h$ is set higher than the
step size of the solver so that the dynamics of the environment can be learnt
in fewer interactions between the RL algorithm and the environment. The
visualisation of the bicycle model is shown in Figure 4, left.

Figure 4: (Left) The visualisation of the virtual bicycle model is shown.
(Right) Examples of the three element types with which the paths in the
present work are constructed are drawn with their corresponding geometric
properties.

The paths the bicycle model should follow are built using three different
element types, namely linear elements, polynomial elements, and circular
elements, see Figure 4, right. The linear element is defined by its length
$L$. The circular element is defined using an angle $\Phi$ and a radius $R$.
The polynomial element is defined by its length $L_{\mathrm{P}}$ and width
$W_{\mathrm{P}}$. The polynomial elements are all of degree five. In the
present work, two families of paths are used, see Figure 5. For the learning
of the agents, paths are initialised randomly in a wide range using the reset
procedure, so that the agents not only learn to follow a specific path, but to
follow paths as generally as possible. The reset procedure is described later
in detail. For the performance evaluation of the deployed agents,
additionally, a benchmark path is introduced, including a full circle, a
slalom, curves, two lane changes, and straight elements.

Figure 5: (Left) Four examples of randomly generated paths which are used in
the learning process are shown. Short parts of the paths are drawn each to
prevent clustering in the figure. (Right) The benchmark path that is used to
evaluate the performance of the agents is shown. For the benchmark path,
$a=$5\text{\,}\mathrm{m}$$ applies.

###  3.2 State

The state in an RL framework includes important geometric and kinematic
quantities of the environment which are essential for the agent. The state in
the present work contains the minimal coordinates of the bicycle model and
preview information describing the path the bicycle should follow. Both parts
are described below.

In the preparatory work associated with the bicycle model, its minimal
coordinates $\mathbf{q}$ are defined, see Section 2.2 and Equation (1). One
part of the state vector $\mathbf{s}$ is the vector $\mathbf{q}$. An
alternative redundant formulation $\mathbf{q}^{\prime}\in\mathbb{R}^{11}$ of
the minimal coordinates is introduced, that can be used instead of the vector
$\mathbf{q}$. In the alternative formulation, the yaw angle $\Psi$ is
represented using a unity vector with the two components

$x_{\mathrm{\Psi}}=\cos{\Psi}\qquad\text{and}\qquad
y_{\mathrm{\Psi}}=\sin{\Psi}\,.$ (3)

Consequently, the alternative redundant formulation for $\mathbf{q}$ reads

$\mathbf{q}^{\prime}=\left[\begin{array}[]{ccccccccccc}x_{\mathrm{P}}&y_{\mathrm{P}}&x_{\mathrm{\Psi}}&y_{\mathrm{\Psi}}&\varphi&\delta&\theta_{\mathrm{R}}&\theta_{\mathrm{F}}&\dot{\varphi}&\dot{\delta}&\dot{\theta}_{\mathrm{F}}\end{array}\right]^{\mathrm{T}}\,.$
(4)

The preview information of the path is given to the agent using the vector
$\mathbf{t}_{\mathrm{P}}\in\mathbb{R}^{(n_{\mathrm{prev}}+1)}$, where
$n_{\mathrm{prev}}=4$ is the number of preview points used to compute the
vector $\mathbf{t}_{\mathrm{P}}$. The vector $\mathbf{t}_{\mathrm{P}}$
represents the course of the path ahead of the bicycle model given in the
$\mathrm{C}$-frame. The $\mathrm{C}$-frame is defined in Equation (A3). To
obtain the vector $\mathbf{t}_{\mathrm{P}}$, it is started with the point
$\mathrm{P_{P0}}$ illustrated in Figure 6. The vector pointing from the rear
wheel contact point to $\mathrm{P_{P0}}$ is projected on the y-axis of the
$\mathrm{C}$-frame. The length of the projection is the first entry of the
vector $\mathbf{t}_{\mathrm{P}}$, denoted as $t_{\mathrm{P0}}$. The point
$\mathrm{P_{P0}}$ is moved along the path by the preview distance $\Delta s$.
This gives the first preview point, called $\mathrm{P_{P1}}$. Again, the
vector that is pointing from $\mathrm{P}$ to $\mathrm{P_{P1}}$ is projected on
the y-axis of the $\mathrm{C}$-frame, which gives the second entry
$t_{\mathrm{P1}}$ of the vector $\mathbf{t}_{\mathrm{P}}$. Repeating this
procedure, the vector $\mathbf{t}_{\mathrm{P}}$ finally reads

$\mathbf{t}_{\mathrm{P}}=\left[\begin{array}[]{ccccc}t_{\mathrm{P0}}&t_{\mathrm{P1}}&t_{\mathrm{P2}}&t_{\mathrm{P3}}&t_{\mathrm{P4}}\end{array}\right]^{\mathrm{T}}\,.$
(5)

Taken together, the state vector $\mathbf{s}$ follows

$\mathbf{s}=\left[\begin{array}[]{c}{\mathbf{q}\vspace{0.1cm}}\\\
{\mathbf{t}_{\mathrm{P}}}\end{array}\right]\qquad\text{or}\qquad\mathbf{s}=\left[\begin{array}[]{c}{\mathbf{q}^{\prime}\vspace{0.1cm}}\\\
{\mathbf{t}_{\mathrm{P}}}\end{array}\right]\,.$ (6)

The two formulations for the state vector $\mathbf{s}$ are compared to each
other in the section on the results, as are two ways to compute the preview
distance $\Delta s$.

Figure 6: Scheme of the procedure to get preview information of the path. The
preview points required for the preview information are obtained by moving
along the path by the preview distance $\Delta s$, starting with the point
$\mathrm{P_{P0}}$. The point $\mathrm{P_{P0}}$ is the point on the path
nearest to the rear wheel contact point $\mathrm{P}$. The point
$\mathrm{P_{Q0}}$ is the point on the path nearest to the front wheel contact
point $\mathrm{Q}$.

###  3.3 Reward

The agent should solve the task of doing path following and stabilisation of
the bicycle model. To represent the two components of the task, the reward
$\rho\in[0,1]$ is the linear combination of the two parts $\rho_{\mathrm{y}}$
and $\rho_{\mathrm{\varphi}}$.

$\rho=\chi_{1}\cdot\rho_{\mathrm{y}}+\chi_{2}\cdot\rho_{\mathrm{\varphi}}$ (7)

For the two reward weights $\chi_{1}+\chi_{2}=1$ applies.

The first part $\rho_{\mathrm{y}}\in[0,1]$ rewards the agent when the bicycle
is close to the path. A distance referred to as $t_{\mathrm{0}}$ is used. The
distance $t_{\mathrm{0}}$ can either be the Euclidean distance
$d(\mathrm{P},\mathrm{P_{P0}})$ between the rear wheel contact point
$\mathrm{P}$ and $\mathrm{P_{P0}}$ or the Euclidean distance
$d(\mathrm{Q},\mathrm{P_{Q0}})$ between the front wheel contact point
$\mathrm{Q}$ and $\mathrm{P_{Q0}}$. The points $\mathrm{P_{P0}}$ and
$\mathrm{P_{Q0}}$ are shown in Figure 6. A relationship must be introduced to
compute the part $\rho_{\mathrm{y}}$ with the distance $t_{\mathrm{0}}$.
Either a linear relationship reading

$\rho_{\mathrm{y}}=\rho_{\mathrm{y}}^{\mathrm{lin}}=\frac{\eta_{\mathrm{y}}-t_{\mathrm{0}}}{\eta_{\mathrm{y}}}$
(8)

or a relationship forming the normalised Gaussian bell curve, reading

$\rho_{\mathrm{y}}=\rho_{\mathrm{y}}^{\mathrm{exp}}=\exp{\left(-0.5t_{\mathrm{0}}^{2}\right)}$
(9)

can be used. The distance threshold $\eta_{\mathrm{y}}$ is set to
$3.5\text{\,}\mathrm{m}$. As explained later, the environment is reset if the
bicycle model drives further away than $\eta_{\mathrm{y}}$ from the path.
Consequently, when the distance between the bicycle model and the path becomes
maximum, the part $\rho_{\mathrm{y}}$ of the reward equals $0$.

The part $\rho_{\mathrm{\varphi}}\in[0,1]$ is computed using the roll angle
$\varphi$.

$\rho_{\mathrm{\varphi}}=\frac{\eta_{\mathrm{\varphi}}-|\varphi|}{\eta_{\mathrm{\varphi}}}$
(10)

By computing the part $\rho_{\mathrm{\varphi}}$ this way, the agent is
penalised if the bicycle leans. The roll angle threshold
$\eta_{\mathrm{\varphi}}$ is set to $None$. If the magnitude of the roll angle
$\varphi$ exceeds $\eta_{\mathrm{\varphi}}$, the environment is reset.
Consequently, when the magnitude of the roll angle $\varphi$ becomes maximum,
the part $\rho_{\mathrm{\varphi}}$ of the reward equals $0$. Although this
reward part is conflicting with training the agent to follow paths including
curves, since with a curve the roll angle $\varphi$ must be non-zero, it will
be investigated later whether and to what extent this interferes with the
learning progress. Note that this reward part can be ignored by setting the
reward weights $\chi_{1}=1$ and $\chi_{2}=0$.

The two described options to obtain the distance $t_{\mathrm{0}}$ will be
compared to each other later, as well as using Equation (8) or Equation (9) to
compute the part $\rho_{\mathrm{y}}$. Furthermore, the influence of the reward
weights $\chi_{1}$ and $\chi_{2}$ is analysed, i.e. how considering the part
$\rho_{\mathrm{\varphi}}$ in the reward changes the learning process of the
agents.

###  3.4 Action

The agent does path following and stabilisation of the bicycle model
exclusively by taking steering actions. The agent predicts a steering angle
$\delta_{\mathrm{set}}\in[-$$,$$]$ that is applied to the bicycle model as
described in Section 2.4. The interval given for $\delta_{\mathrm{set}}$ is
substantiated in Appendix B.

##  4 Learning Process with Curriculum Learning

The learning process runs through the RL framework to optimise the weights of
the neural networks the agent consists of. The learning process thus is the
process that generates data with the multibody simulation environment and uses
this data to optimise the function approximators of the agent. The learning
processes in the present work consist of $n_{\mathrm{tot}}=4\cdot 10^{6}$
learning steps, since, as shown later, the validation results no longer change
significantly at this level of training. One learning step is one run-through
of the RL framework. This means that one learning process uses data generated
in $n_{\mathrm{tot}}\cdot h=55.5$ hours of simulated bicycle rides, with $h$
being the time between two interactions of the agent, see Section 3.1. The
learning process can be divided into episodes. An episode ends when at least
one of the conditions listed below is met:

  * •

The episode consists of $n_{\mathrm{max}}=1200$ learning steps so that one
episode represents a bicycle ride with a maximum duration of
$n_{\mathrm{max}}\cdot h=$60\text{\,}\mathrm{s}$$.

  * •

At least one of the two thresholds $\eta_{\mathrm{y}}$ and
$\eta_{\mathrm{\varphi}}$ is exceeded, i.e. the environment is reset, and the
episode consists of at least $n_{\mathrm{min}}=800$ learning steps. This
number of learning steps is determined by the ratio $n_{\mathrm{min}}\cdot
1.5=n_{\mathrm{max}}$ defined for the present work. The two thresholds
$\eta_{\mathrm{y}}$ and $\eta_{\mathrm{\varphi}}$ are defined in Section 3.3.

As long as the learning process does not contain $n_{\mathrm{tot}}$ learning
steps, a new episode starts when the previous episode ended.

In addition to running through the RL framework, the learning process must
reset the environment when certain conditions are met. Furthermore,
validations are performed during the learning process to monitor the progress
of learning. During the learning process, a training strategy called
Curriculum Learning (CL) is applied. The reset procedure, the validation
procedure, and the CL are explained now.

###  4.1 Resetting the Environment

Primarily, the environment of the RL framework must be reset if the agent
failed in solving the control task. The reset procedure is therefore called
when the roll angle $\varphi$ of the bicycle model exceeds the roll angle
threshold $\eta_{\mathrm{\varphi}}$ or when the distance between the rear
wheel and the point $\mathrm{P_{P0}}$ exceeds the distance threshold
$\eta_{\mathrm{y}}$. In other words, reset is called when the bicycle has
fallen over or driven far away from the path. If two consecutive episodes
contain the maximum number of learning steps $n_{\mathrm{max}}$ and no reset
was called between the episodes, the environment is also reset. Resetting the
environment at this condition ensures that the agent that already succeeds in
solving the control task without exceeding the thresholds $\eta_{\mathrm{y}}$
and $\eta_{\mathrm{\varphi}}$ still finds a variety of initial conditions in
the remaining learning process. The reset procedure initialises the
configuration of the bicycle model as well as the path the bicycle model
should follow.

The configuration of the bicycle model is initialised by randomly setting the
minimal coordinates of the bicycle model in the intervals given in the upper
part of Table 4.1. The roll angle $\varphi$, the steering angle $\delta$, and
their angular velocities are not set to zero when initialising the bicycle
model. By initially perturbing the bicycle model, the agent should learn how
to stabilise the bicycle by just observing the beginnings of every simulated
bicycle ride. The forward velocity of the bicycle model is set randomly
between $v_{\mathrm{min}}$ and $v_{\mathrm{max}}$. Note that the bicycle
cannot accelerate or decelerate in the simulated bicycle rides, as required in
the introductory Section 1. Since the forward velocity is not a minimal
coordinate of the bicycle model, it is converted into the front wheel rotation
angular velocity $\dot{\theta}_{\mathrm{F}}$ using

$\dot{\theta}_{\mathrm{F}}=\frac{v}{\mathrm{r_{F}}}\,,$ (11)

with $\mathrm{r_{F}}$ being the radius of the front wheel. See Appendix A to
see how the forward velocity is considered in the coordinates mappings. The
parameters not necessary for solving the control task, i.e. the contact point
$\mathrm{P}=(x_{\mathrm{P}},y_{\mathrm{P}})$, the yaw angle $\Psi$, and the
wheel rotation angles $\theta_{\mathrm{R}}$ and $\theta_{\mathrm{F}}$, are
initialised arbitrarily so that the agent learns their irrelevance. Remember
that the used multibody simulation of the bicycle model is based on a
redundant formulation, which is why the minimal coordinates must be mapped to
redundant coordinates after calling reset, as elaborated in Section 2.3.

The randomly initialised path the bicycle model should follow lies in the
plane spanned by the x-axis and the y-axis of the $\mathrm{0}$-frame. The
beginning of the path is placed where the contact point $\mathrm{P}$ of the
rear wheel is initialised. The beginning of the path is rotated such that the
angle of the tangent of the path at the point $\mathrm{P}$ is inside the
interval $[\Psi-$$,\Psi+$$]$, where $\Psi$ is the yaw angle the bicycle is
initialised with. The path is initialised by appending a random sequence of
the three element types the path can consist of. The three element types are
explained in Section 3.1 and illustrated in Figure 4. Not only the sequence of
element types, but also the geometric properties of each individual element of
the path are initialised randomly in a given interval. The intervals for the
geometric properties of the elements are given in the lower part of Table 4.1.
The curvature of the circular elements is selected at random so that with a
probability of $50\text{\,}\mathrm{\char 37\relax}$ the element is positively
or negatively curved. The same is done for the polynomial elements to specify
the direction of the transfer of the width $|W_{\mathrm{P}}|$. It is specified
that the initialised path does not start with a circular element so that after
resetting, where the bicycle model may be perturbed, the task does not become
unachievable. Additionally, it is defined that $40\text{\,}\%$ of the elements
in the initialised path should be circular elements, $40\text{\,}\%$ should be
polynomial elements, and $20\text{\,}\%$ should be straight elements. This is
specified so that the path does not contain long straight sections with which
the agent exploits a behaviour being able to follow the straight elements, but
not the other two types of elements. The family of paths assembled in the way
of the present work cannot be followed by the bicycle model exactly, because
the dynamic characteristics of the bicycle impose limitations in its
manoeuvrability [26]. In particular, step-like changes in the curvature of the
path result in a path error between the bicycle model and the path, alike a
bicycle travelling on real roads. To start the turn of a bicycle, the
handlebar must be turned in the opposite direction, known as counter-steering,
so that the bicycle leaves the desired path [5, 26]. This dynamic property of
the bicycle model must be learnt by the agents, trying to follow the path as
closely as possible.

Intervals in which the parameters of the environment are initialised when
reset is called, where for each parameter $x$ an interval $a\leq x\leq b$ is
given. The table indicates which initialisation intervals are used to apply
CL, see also Table 4.3.  parameter initialisation interval CL $a$ $b$ bicycle
model coordinates $x_{\mathrm{P}}$ and $y_{\mathrm{P}}$ of the contact point
$\mathrm{P}$ $-10\text{\,}\mathrm{m}$ $10\text{\,}\mathrm{m}$ no yaw angle
$\Psi$ $-\pi\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$
$\pi\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$ no roll angle $\varphi$
$-0.01\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$
$0.01\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$ no steering angle $\delta$
$-0.01\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$
$0.01\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$ no wheel rotation angles
$\theta_{\mathrm{R}}$ and $\theta_{\mathrm{F}}$
$-\pi\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$
$\pi\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$ no roll angular velocity
$\dot{\varphi}$ $-0.05\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}\mathrm{s}^{-1}$
$0.05\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}\mathrm{s}^{-1}$ no steering
angular velocity $\dot{\delta}$
$-0.01\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}\mathrm{s}^{-1}$
$0.01\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}\mathrm{s}^{-1}$ no front wheel
rotation angular velocity $\dot{\theta}_{\mathrm{F}}$ set with the forward
velocity $v$ - forward velocity $v$ $v_{\mathrm{min}}$ $v_{\mathrm{max}}$ yes
path length of linear elements $L$ $5\text{\,}\mathrm{m}$
$15\text{\,}\mathrm{m}$ no angle of circular elements $\Phi$
$\pi/4\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$
$\pi/2\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$ no radius of circular elements
$R$ $R_{\mathrm{min}}$ $R_{\mathrm{max}}$ yes length of polynomial elements
$L_{\mathrm{P}}$ $L_{\mathrm{P,min}}$ $L_{\mathrm{P,max}}$ yes width of
polynomial segments $W_{\mathrm{P}}$ $W_{\mathrm{P,min}}$ $W_{\mathrm{P,max}}$
yes

###  4.2 Validation during the Learning Process

The validation procedure is to monitor the success of the learning process. In
the learning process, a validation is always carried out after an episode with
$n_{\mathrm{max}}$ learning steps and if the last validation was $4000$
learning steps ago. The first condition is set since it is assumed that an
agent that manages to complete the maximum number of steps in an episode
already masters the task to some extent. The latter condition is set to save
computing time, as a well-trained agent is not validated after each episode. A
validation consists of $10$ simulated bicycle rides of a duration of
$30\text{\,}\mathrm{s}$. To do path following and stabilisation of the bicycle
model in these simulated bicycle rides, the agent in its current state of
training is used as a kind of controller. Before each of the simulated bicycle
rides in the validation, the environment is reset using the procedure and
initialisation intervals described in Section 4.1. Each simulated bicycle ride
is scored using a normalised error $e_{i}\in[0,1]$.

$e_{i}=\begin{cases}1&\text{if $\eta_{\mathrm{y}}$ or
$\eta_{\mathrm{\varphi}}$ is exceeded}\\\
\frac{\bar{t}_{\mathrm{0,$80\text{\,}\%$}}}{\eta_{\mathrm{y}}}&\text{else}\end{cases}$
(12)

If the bicycle model has fallen over or driven far away from the path, which
means that the distance threshold $\eta_{\mathrm{y}}$ or the roll angle
threshold $\eta_{\mathrm{\varphi}}$ is exceeded, $e_{i}=1$ is set. Otherwise,
the error $e_{i}$ is computed using the mean of $t_{\mathrm{0}}$ measured over
the last $80\text{\,}\%$ of the duration of the simulated bicycle ride,
denoted as $\bar{t}_{\mathrm{0,$80\text{\,}\%$}}$. The first $20\text{\,}\%$
of the bicycle ride are ignored for this computation, in order to make $e_{i}$
independent of the perturbations the bicycle model is initialised with. Remark
that the distance $t_{\mathrm{0}}$ is calculated the same as for the reward
$\rho$, which is described in Section 3.3. An error threshold
$\eta_{\mathrm{e}}$ is introduced, reading

$\eta_{\mathrm{e}}=\frac{$0.2\text{\,}\mathrm{m}$}{\eta_{\mathrm{y}}}=\frac{$0.2\text{\,}\mathrm{m}$}{$3.5\text{\,}\mathrm{m}$}\approx
0.057\,.$ (13)

In a validation process, a single simulated bicycle ride is called successful
if $e_{i}\leq\eta_{\mathrm{e}}$ holds. Since a validation consists of $10$
simulated bicycle rides, a validation results in $10$ errors $e_{i}$.

###  4.3 Curriculum Learning

Curriculum Learning (CL) is a training strategy for the field of machine
learning that was first formalised in the work [27]. When doing CL, the task
to be solved becomes increasingly complex as the learning proceeds in order to
improve the quality of the learnt policy and the speed of the learning process
as compared to learning the complex task directly. The CL is applied in the
present work by changing the initialisation interval of some parameters,
marked in Table 4.1, throughout learning.

Intervals in which the parameters that are changed throughout the learning
process are initialised in each part of the four-part curriculum, where for
each parameter $x$ an interval $a\leq x\leq b$ is given.  parameter
initialisation range at curriculum part 1 2 3 4 a b a b a b a b forward
velocity $v$ in $\mathrm{m}\mathrm{s}^{-1}$ $4$ $4$ $2$ $7$ $2$ $7$ $2$ $7$
radius of circular elements $R$ in $\mathrm{m}$ $14$ $14$ $12$ $14$ $10$ $14$
$8$ $14$ length of polynomial elements $L_{\mathrm{P}}$ in $\mathrm{m}$ $20$
$22$ $18$ $22$ $16$ $22$ $14$ $22$ width of polynomial elements
$W_{\mathrm{P}}$ in $\mathrm{m}$ $2$ $2$ $2$ $4$ $2$ $6$ $2$ $8$

The adaption of the initialisation intervals is done using discrete stages,
called parts of the curriculum. The functionality of the curriculum applied in
the present work is schematically illustrated in Figure 7. A linear
relationship over $n_{\mathrm{dec}}=10000$ learning steps is used in the
present work to extend the initialisation intervals. In Table 4.3 the
initialisation intervals that are changed with the parts of the curriculum are
given. It can be seen that the curriculum in the present work extends the
forward velocity at which the bicycle is initialised from a constant value of
$4\text{\,}\mathrm{m}\mathrm{s}^{-1}$ to a wide range of speeds. The speed
range goes from $2\text{\,}\mathrm{m}\mathrm{s}^{-1}$, a velocity at which the
bicycle is in a highly unstable mode, via the self-stable velocity range from
$4.3\text{\,}\mathrm{m}\mathrm{s}^{-1}$ to
$6\text{\,}\mathrm{m}\mathrm{s}^{-1}$, up to
$7\text{\,}\mathrm{m}\mathrm{s}^{-1}$, where the bicycle is again mildly
unstable [5]. This makes the task more complex in two ways, since the policy
must get speed-dependent, see introductory Section 1, and the slower bicycle
is more difficult to stabilise [5]. Furthermore, the curriculum makes the
paths to be followed more complex as the learning proceeds by adding sharper
turns and steeper transfers to the assembled paths.

Figure 7: Scheme of a four-part curriculum in which the initialisation range
of one parameter is adjusted. When a validation is fully successful, with all
errors $e_{i}\leq\eta_{\mathrm{e}}$, the next part of the curriculum is
initiated. During changing the initialisation range within $n_{\mathrm{dec}}$
learning steps, no validation is done. A fully successful validation in the
last part of the curriculum represents the end of the curriculum.

##  5 Results

The section on the results is divided into five parts. Firstly, different
settings for the RL framework are listed. Remark that each option for the
settings is explained in more detail in Section 3, where the implementation of
the RL framework is described. Secondly, the learning process when using the
different settings is analysed. Afterwards, the performance of the trained and
deployed agents when they are used to do path following and stabilisation of
the bicycle model is evaluated. Then, a simulated bicycle ride along the
benchmark path is analysed in more detail. Finally, explanatory methods are
applied to explain the predictions a trained agent makes. For statistical
significance, five learning processes, referred to as runs, initialized with a
different random seed, are done with the investigated settings. Therefore, in
this section, numerous selection procedures are performed to start with a
general analysis and then analyse a single agent as if it were a controller
designed using control engineering techniques.

###  5.1 Investigated Settings for the RL Framework

The settings for the RL framework concern the state vector $\mathbf{s}$, the
design of the reward $\rho$, and the used RL algorithm.

In the state vector $\mathbf{s}$, either the vector $\mathbf{q}$ for the
minimal coordinates of the bicycle model, see Equation (1), or a redundant
formulation $\mathbf{q}^{\prime}$ with the yaw of the bicycle given as a unity
vector, see Equation (4), is used. Furthermore, the preview distance $\Delta
s$ used to obtain the preview information is either set constant with $\Delta
s=$2\text{\,}\mathrm{m}$$ or scaled with the forward velocity $v$ of the
bicycle, reading

$\Delta s=v\;$0.4\text{\,}\mathrm{s}$\,.$ (14)

The distance $t_{\mathrm{0}}$ used to compute the part $\rho_{\mathrm{y}}$ of
the reward is either the distance $d(\mathrm{P},\mathrm{P_{P0}})$ using the
rear wheel contact point $\mathrm{P}$, or the distance
$d(\mathrm{Q},\mathrm{P_{Q0}})$ using the front wheel contact point
$\mathrm{Q}$. With this distance, either a linear relationship
$\rho_{\mathrm{y}}^{\mathrm{lin}}$, see Equation (8), or an exponential
relationship $\rho_{\mathrm{y}}^{\mathrm{exp}}$, see Equation (9), can be used
to compute the part $\rho_{\mathrm{y}}$ of the reward. The reward weights
$\chi_{1}$ and $\chi_{2}$ are adjustable, see Equation (7). In the present
work, either $\chi_{1}=1$ and $\chi_{2}=0$ or $\chi_{1}=\chi_{2}=0.5$ is used.
The latter setting is used to investigate how the inconsistency of the two
reward parts $\rho_{\mathrm{y}}$ and $\rho_{\mathrm{\varphi}}$, which is
described in Section 3.3, disrupts the learning progress.

While any RL algorithm can be used in the introduced RL framework, as long as
a continuous action space is supported by the algorithm, the SAC algorithm
[22] performed significantly better in initial tests than the A2C algorithm
introduced in the work [28] and the PPO algorithm introduced in the work [29].
The results shown were therefore achieved exclusively with the SAC algorithm.
The implementation of SAC in Stable-Baselines3 with the hyperparameters given
in [22] is used.

In Table 5.1, the investigated settings are listed. Proposed settings are
defined, to which each alternative combination of settings differs in one
single parameter to better understand their influence on learning and the
trained agent.

Variation of the RL framework. The single parameter that distinguishes the
alt(1-5) settings from the proposed (prop.) settings is highlighted.  settings
$\mathbf{q}\vee\mathbf{q}^{\prime}$ $\Delta s$ $t_{\mathrm{0}}$
$\rho_{\mathrm{y}}$ $\chi_{1}$ algorithm prop. $\mathbf{q}^{\prime}$
$v\;$0.4\text{\,}\mathrm{s}$$ $d(\mathrm{P},\mathrm{P_{P0}})$
$\rho_{\mathrm{y}}^{\mathrm{lin}}$ $1$ SAC alt(1) $\mathbf{q}$
$v\;$0.4\text{\,}\mathrm{s}$$ $d(\mathrm{P},\mathrm{P_{P0}})$
$\rho_{\mathrm{y}}^{\mathrm{lin}}$ $1$ SAC alt(2) $\mathbf{q}^{\prime}$
$2\text{\,}\mathrm{m}$ $d(\mathrm{P},\mathrm{P_{P0}})$
$\rho_{\mathrm{y}}^{\mathrm{lin}}$ $1$ SAC alt(3) $\mathbf{q}^{\prime}$
$v\;$0.4\text{\,}\mathrm{s}$$ $d(\mathrm{Q},\mathrm{P_{Q0}})$
$\rho_{\mathrm{y}}^{\mathrm{lin}}$ $1$ SAC alt(4) $\mathbf{q}^{\prime}$
$v\;$0.4\text{\,}\mathrm{s}$$ $d(\mathrm{P},\mathrm{P_{P0}})$
$\rho_{\mathrm{y}}^{\mathrm{exp}}$ $1$ SAC alt(5) $\mathbf{q}^{\prime}$
$v\;$0.4\text{\,}\mathrm{s}$$ $d(\mathrm{P},\mathrm{P_{P0}})$
$\rho_{\mathrm{y}}^{\mathrm{lin}}$ $0.5$ SAC

###  5.2 Learning Process

To analyse the learning process with the different settings for the RL
framework, the required learning steps, the progression of the validation
error over the learning steps, and the learning steps required in the parts of
the curriculum are considered.

The number of learning steps required when using specific settings for the RL
framework is the mean number of learning steps the runs with these settings
take to end the curriculum. As can be seen in Figure 8, the alt(2) settings,
where the preview distance $\Delta s$ is not scaled with the forward velocity
$v$, and the alt(5) settings, where the reward weights $\chi_{1}$ and
$\chi_{2}$ are set equally, require more than $2.5\cdot 10^{6}$ learning
steps. The proposed and alt(1, 3-4) settings require less than $2\cdot 10^{6}$
learning steps. With regard to the alt(1) settings, this implies that the yaw
representation does not influence the required learning steps. Note that in
initial tests the yaw angle $\Psi$ was not initialised randomly, but at
$\Psi=0$, with which the curriculum could not be finished regardless of the
used yaw representation.

Figure 8: Required learning steps shown with standard deviation when different
settings for the RL framework are used. ($\ast$ Not all of the runs done with
these settings have reached the end of the curriculum.)

As described in Section 4.2, each validation during the learning process
results in $10$ errors $e_{i}$. To obtain the learning progress, the
validation error $e$, being the mean value of the errors $e_{i}$, is computed
over the learning steps for each run. Thus, five errors $e$ can be computed
over the learning steps for each setting. In Figure 9 the validation error $e$
is plotted for the proposed and the alt(1-2, 5) settings over the learning
steps, as well as the parts of the curriculum in which the runs with these
settings are. No significant difference is observed between the proposed and
the alt(1) settings, where the yaw of the bicycle is given as an angle value.
In the first part of the curriculum, the learning process with the proposed
and the alt(2) settings with a non velocity-scaled preview distance $\Delta s$
does not differ significantly, since the velocity scaling of $\Delta s$ has no
influence in the first part of the curriculum, with the forward velocity of
the bicycle being constant, see Table 4.3. In the second part of the
curriculum, the interval for the forward velocity $v$ is extended and the
learning processes with the alt(2) settings with a constant preview distance
$\Delta s$ take an average of $264\text{\,}\%$ of the learning steps taken
when using the proposed settings. Furthermore, the mean validation error rises
above $0.4$ with the alt(2) settings, while the validation error does not
exceed $0.26$ with the proposed settings. The learning with the alt(2)
settings does not recover from the delay in the second part of the curriculum
until the end of the learning process, necessitating the higher number of
required learning steps already shown in Figure 8. Two learning processes
finish the curriculum in $n_{\mathrm{tot}}$ learning steps if the alt(5)
settings with equally weighted reward parts are used. The other runs stay in
the second and third part of the curriculum until $n_{\mathrm{tot}}$ learning
steps are reached. The validation error starts to increase after $2.5\cdot
10^{6}$ learning steps, implying that including the part
$\rho_{\mathrm{\varphi}}$ in the reward disrupts the learning. Learning
processes with the alt (3) settings, where the front wheel contact point
$\mathrm{Q}$ is used to obtain the distance $t_{\mathrm{0}}$, and the alt(4)
settings, where the exponential relationship
$\rho_{\mathrm{y}}^{\mathrm{exp}}$ is used, show no significant difference
from learning processes with the proposed or the alt(1) settings.

Figure 9: (Top) Validation error $e$ with the proposed and alt(1-2, 5)
settings plotted over the learning steps. The mean validation error of the
five runs done with each of the settings is drawn, as well as the area between
the minimal and maximal validation error. A Simple Moving Average (SMA) filter
with $k=20$ is applied to the data. (Bottom) Parts of the curriculum in which
the individual runs with the different settings are, plotted over the learning
steps.

###  5.3 Performance of the Deployed Agents

In this section, the performance of the deployed agents is shown, as well as
the influence the settings for the RL framework have on the performance of the
deployed agents. Remember that a deployed agent takes the place of the
controller, applying steering actions based on the current state of the
environment to do path following and stabilisation of the bicycle model.
Firstly, it is described at which point in a run the agent is deployed and how
the performance of the agents is measured. Afterwards, the agents are tested
with the family of paths used during learning and along the benchmark path
that is described in Section 3.1.

####  5.3.1 Agent Selection and Performance Measure

The parameters of the agent, i.e. the neural network that predicts the
actions, are saved periodically during training. For the evaluation of
performance, the last agent from each learning process is chosen which passed
all tests of a validation and has finished the curriculum. Consequently, five
agents are deployed and evaluated each with the proposed and alt(1-4) settings
and two agents are deployed and evaluated with the alt(5) settings. During the
simulated bicycle rides using the deployed agents, the distance
$t_{\mathrm{0}}=d(\mathrm{P},\mathrm{P_{P0}})$ is tracked, that is the
smallest distance between the rear wheel contact point $\mathrm{P}$ and the
path. The mean number of the measured distance $t_{\mathrm{0}}$ during the
bicycle ride when using the agent $k$ at a forward velocity of $v$ is denoted
as $\bar{t}_{\mathrm{0,k}}(v)$. The maximum of the measured distance
$t_{\mathrm{0}}$ is denoted as $\hat{t}_{\mathrm{0,k}}(v)$. Both values are
used to quantify the performance of the agent $k$. Assuming that the agents
$k\in\mathbb{N}\cap[1,N]$ are deployed from the $N$ runs with the proposed
settings, the distance $T(v)$ reading

$T(v)=\frac{1}{N}\sum_{k=1}^{N}{\bar{t}_{\mathrm{0,k}}(v)}$ (15)

gives the performance of the agents trained with the proposed settings at the
forward velocity of $v$. In the following, $T(v)$ is computed for all the
settings shown in Table 5.1 with $$2\text{\,}\mathrm{m}\mathrm{s}^{-1}$\leq
v\leq$7\text{\,}\mathrm{m}\mathrm{s}^{-1}$$ applying for the forward velocity
$v$. Note that during the simulated bicycle rides, the same settings apply as
during learning. If, for example, an agent is trained with a non velocity-
scaled preview distance, it is also evaluated with the preview distance being
non velocity-scaled.

####  5.3.2 Performance along Randomly Generated Paths

The randomly generated paths for evaluation always start in the origin of the
$\mathrm{0}$-frame. The bicycle model is initialized in the reference
configuration, see Figure 1, with the rear wheel contact point $\mathrm{P}$ on
the path to avoid inconsistencies in the performance due to different
initialisations. The duration of the simulated bicycle rides is set to
$30\text{\,}\mathrm{s}$. In the upper part of Figure 10, the distance $T(v)$
is plotted for different settings. If one or more of the five agents that can
be deployed from one setting fails at a specific forward velocity $v$, the
corresponding marker in the plot is shown translucent and connected by dotted
lines. From the outset, this is the case for all markers of the alt(5)
settings since only two agents finish the curriculum and are thus deployed. It
is seen that all of the agents trained using the proposed and the alt(1-4)
settings are able to do path following and stabilisation of the bicycle model
along the randomly generated paths over the entire speed range. Taken
together, for the evaluation of the proposed and alt(1-4) settings $525$
simulated bicycle rides are made in total ($21$ velocities are tested with the
agents deployed from the five runs for each of the five settings) and in all
$525$ bicycle rides the bicycle is stabilised and steered along the paths. In
other words, none of the agents trained with the proposed and alt(1-4)
settings fails in solving the control task for the type of paths the agents
are trained with. Note that for each simulated bicycle ride the path is
initialized randomly. The distance $T(v)$ for the agents trained with the
proposed settings is below $0.2\text{\,}\mathrm{m}$ over the investigated
speed range. The same applies for the alt(1, 3-4) settings.

Figure 10: (Top, Middle) Distance $T(v)$ plotted over the forward velocity $v$
for the investigated settings when two different types of paths are used
(schematic representation of the paths shown). (Bottom) Two individual agents
are analysed in more detail. The mean $\bar{t}_{\mathrm{0,1}}(v)$ and maximal
distance $\hat{t}_{\mathrm{0,1}}(v)$ are plotted, giving the performance of
the two agents over the investigated velocity range along the benchmark path.

####  5.3.3 Performance along the Benchmark Path

The bicycle model is again initialised in the reference configuration, see
Figure 1, with the rear wheel contact point $\mathrm{P}$ standing at the
starting point of the benchmark path. The simulated bicycle rides along the
benchmark path do not have a time limit, but the simulations stop when the
bicycle returns to the start point. Figure 10 shows that all of the agents
trained with the proposed settings manage to do path following and
stabilisation of the bicycle model along the benchmark path over the entire
velocity range. The same applies to the agents trained with the alt(3)
settings, where the reward is computed using an exponential relationship
$\rho_{\mathrm{y}}^{\mathrm{exp}}$. The distance $T(v)$ when using the
proposed settings is below $0.2\text{\,}\mathrm{m}$ over the entire speed
range. At $4.25\text{\,}\mathrm{m}\mathrm{s}^{-1}$ the distance $T$ equals
$0.08\text{\,}\mathrm{m}$, being the smallest distance $T(v)$ for the proposed
settings. It is found that at the lower end of the investigated velocity
range, $T$ reads $0.19\text{\,}\mathrm{m}$ and at the upper end $T$ equals
$0.15\text{\,}\mathrm{m}$. One agent each deployed using the alt(1-2) settings
does not succeed at $2\text{\,}\mathrm{m}\mathrm{s}^{-1}$. One agent using the
alt(4) does not succeed at $2\text{\,}\mathrm{m}\mathrm{s}^{-1}$ and at
$2.25\text{\,}\mathrm{m}\mathrm{s}^{-1}$. This fact will be part of the
closing discussion of the present work.

With the proposed and the alt(3) settings one deployed agent each is now
analysed in more detail. To select the best agent from the agents deployed
when using these two settings, the maximal distance
$\hat{t}_{\mathrm{0,k}}(v)$ is obtained for every agent $k$ deployed from the
settings. The value for $\hat{t}_{\mathrm{0,k}}(v)$ is maximal at a certain
forward velocity. The agent with the smallest maximal value for
$\hat{t}_{\mathrm{0,k}}(v)$ is selected. Doing so, for the proposed settings
and the alt(3) settings one agent, referred to as agent 1 ($k=1$) is chosen
each. In the bottom part of Figure 10 the distance values
$\hat{t}_{\mathrm{0,1}}(v)$ and $\bar{t}_{\mathrm{0,1}}(v)$ are plotted over
the velocity when these agents are used to do path following and stabilisation
of the bicycle model. Remember that the figure shows what can be seen as the
performance of the controller depending on the forward velocity of the
bicycle, with the mean distance to the benchmark path and the maximal distance
being given. Table 5.3.3 shows the maximal values of these two distances when
using the two selected agents.

Maximal values for $\hat{t}_{\mathrm{0,1}}(v)$ and $\bar{t}_{\mathrm{0,1}}(v)$
when the deployed agents are used to do path following and stabilisation of
the bicycle model along the benchmark path. The velocities at which the
maximal values occur are given.  name value in $\text{\,}\mathrm{m}$ velocity
$v$ in $\text{\,}\mathrm{m}\mathrm{s}^{-1}$ agent 1 trained with the proposed
settings maximal value for $\hat{t}_{\mathrm{0,1}}(v)$ $0.69$ $7$ maximal
value for $\bar{t}_{\mathrm{0,1}}(v)$ $0.09$ $7$ agent 1 trained with the
alt(3) settings maximal value for $\hat{t}_{\mathrm{0,1}}(v)$ $0.58$ $7$
maximal value for $\bar{t}_{\mathrm{0,1}}(v)$ $0.13$ $2.25$

Taken together, an agent is trained with the proposed settings for the RL
framework that can stabilise the bicycle model in the velocity range from
$2\text{\,}\mathrm{m}\mathrm{s}^{-1}7\text{\,}\mathrm{m}\mathrm{s}^{-1}$ while
the rear wheel contact point $\mathrm{P}$ is never more than
$0.69\text{\,}\mathrm{m}$ away from the benchmark path. The largest mean
distance to the benchmark path when using this agent as a kind of controller
for the bicycle model is found as $0.09\text{\,}\mathrm{m}$. This corresponds
to $9\text{\,}\mathrm{\char 37\relax}$ of the wheelbase of the used bicycle
model. The agent trained with the alt(3) shows similar performance.

###  5.4 Bicycle Ride along the Benchmark Path in Detail

In this section, a simulated bicycle ride along the benchmark path is analysed
in more detail, with the agent 1 trained with the proposed settings doing path
following and stabilisation of the bicycle model. The forward velocity is
initialised at $4\text{\,}\mathrm{m}\mathrm{s}^{-1}$.

In Figure 11, the roll angle $\varphi$, the steering angle $\delta$, and the
(negated) curvature $\kappa$ of the benchmark path are plotted over the first
$200\text{\,}\mathrm{m}$ of the benchmark path. It is seen that the roll angle
$\varphi$ of the bicycle model is proportional to the curvature $\kappa$ of
the path. In the zoomed-in area, it is seen that the steering angle $\delta$
is permanently adjusted by the agent and that $\delta$ gets negative prior to
the left curved circle. Such a counter-steering action is necessary to steer
the bicycle in the desired direction [26], refer to Section 4.1. Figure 12,
frame (a) also shows this counter-steering action done before the bicycle
follows the full circle. With the counter-steering, the roll angle $\varphi$
gets negative. Along the full circle, a negative roll angle $\varphi$, induced
by the prior counter-steering, is kept. The agent sets the steering angle
$\delta$ positive, keeping the bicycle model upright by steering into the
direction of the undesired fall [6], see also introductory Section 1, and
doing path following along the circle. Frame (b) refers to a point where the
curvature $\kappa$ changes the sign. Frame (c) shows the bicycle travelling
along the flat curve of the path. In frame (d) the bicycle enters the first
polynomial transfer. Frames (e) and (f) show the bicycle travelling along the
so-called hard lane change. It is seen that the hard lane change is
challenging to follow with the bicycle travelling at
$4\text{\,}\mathrm{m}\mathrm{s}^{-1}$.

Figure 11: Roll angle $\varphi$, steering angle $\delta$, and curvature
$\kappa$ of the path plotted over the first $200\text{\,}\mathrm{m}$ of the
benchmark path. Agent1 trained with the proposed settings is used to do path
following and stabilisation of the bicycle model. The bicycle model travels at
$4\text{\,}\mathrm{m}\mathrm{s}^{-1}$.

Figure 12: Rear wheel contact point $\mathrm{P}$ tracked when the agent 1
trained with the proposed settings is used to do path following and
stabilisation of the bicycle model along the benchmark path. Six different
frames (a)-(f) from the visualisation of the multibody simulation are shown at
specific points of the simulated bicycle ride. The circles shown on the street
illustrate the points $\mathrm{P_{P0}},\dots,\mathrm{P_{P4}}$ used to get the
preview information of the path, see Figure 6.

###  5.5 Explanation for the Predictions of the Agent

In Section 5.4, a simulated bicycle ride along the benchmark path using the
agent 1 trained with the proposed settings is described in detail. During the
slalom part of the benchmark path, the state vector $\mathbf{s}$ is saved
every time the agent interacts with the environment so that observations are
gained that equally represent the bicycle model doing path following along
left and right curves. Using these observations, the predictions of the agent
are explained using the SHapley Additive exPlanations (SHAP) values. SHAP
values are a measure for the feature importance, introduced in the work [30].
The features of a neural network are the elements in the input vector, being
the elements of the state vector $\mathbf{s}$. The permutation explainer,
implemented in the eponymous Python package
SHAP444https://github.com/shap/shap (used version: 0.45.0) is used to compute
the SHAP values. For each individual observation, being a saved state vector
$\mathbf{s}$, every feature can be assigned a SHAP value. A positive SHAP
value of a feature in an observation means that the feature positively
influences the prediction of the agent in that observation, while a negative
SHAP value means that the feature negatively influences the prediction.
Remember that the prediction of the agent is the set value
$\delta_{\mathrm{set}}$ for the steering angle.

Figure 13: Impact of the features on the prediction of a trained agent
measured by the SHAP values. Each dot represents a single observation. The
vertical alignment of the dot indicates whether the output of the agent is
positively or negatively influenced by the corresponding feature in the
observation. The colour of the dot indicates whether the value of this feature
is high or low in the observation.

In Figure 13, the importance of the features measured by the SHAP values are
shown. The features are ordered by their mean absolute SHAP value. The roll
angle $\varphi$ is the feature having the greatest impact on the predictions
of the agent. In the observations with a positive roll angle $\varphi$, the
roll angle $\varphi$ contributes negatively to the prediction of the agent.
Thus, a positive roll angle $\varphi$ pulls the set value
$\delta_{\mathrm{set}}$ for the steering angle down and vice versa. The entry
$t_{\mathrm{P3}}$ of the preview vector $\mathbf{t}$ is the second most
important feature based on the mean absolute SHAP values. Observations with a
positive value for $t_{\mathrm{P3}}$ negatively impact the prediction of the
agent. The same applies to $t_{\mathrm{P2}}$, but in the observations this
feature does not contribute that much to the output of the agent as
$t_{\mathrm{P3}}$. There are observations, in which the roll angular velocity
$\dot{\varphi}$ has a relatively high impact on the output of the agent.
However, in most of the predictions, the roll angular velocity $\dot{\varphi}$
is near to zero, not influencing the set value $\delta_{\mathrm{set}}$ for the
steering angle. The mean absolute SHAP value of $t_{\mathrm{P4}}$, which
represents the preview point that is most ahead of the bicycle, is smaller
than that of the roll angular velocity $\dot{\varphi}$. The other features are
taken together as others, as they do not influence the prediction of the agent
as much as the other features do, according to the SHAP values. The fact that
$x_{\mathrm{\Psi}}$ and $y_{\mathrm{\Psi}}$ describing the yaw of the bicycle
are part of others is in line with the finding that the yaw representation
does not influence the learning process, see Section 5.2, since the yaw is not
considered to a high degree by the agent.

##  6 Discussion

This section starts by discussing the different investigated settings for the
RL framework. Afterwards, the results shown for the performance of the
deployed agents are debated. Finally, the explanation for the predictions of
the agent given by the SHAP values are compared to already known mechanisms
for controlling a bicycle.

In Section 5.2, it is shown that using a constant preview distance $\Delta s$
or including the part $\rho_{\mathrm{\varphi}}$ in the reward lowers the
learning performance. On the former, we find that scaling $\Delta s$ with the
forward velocity reduces the learning volume of the agent. The learnt policy
must, because of using the Whipple bicycle, take the forward velocity into
account to stabilise the bicycle laterally. The scaling of the preview
distance eliminates another speed-dependence of the policy. What is striking
at first is that the speed-dependence of the policy is not represented by the
SHAP values, see Figure 13, since the front wheel rotation angular velocity
$\dot{\theta}_{\mathrm{F}}$ is classified into others. The reason for this is
not the lack of speed-dependence in the policy, but the way the SHAP values
are computed. In the simulated bicycle ride used to obtain the observations
for the explanation, the bicycle travels at a constant forward velocity of
$4\text{\,}\mathrm{m}\mathrm{s}^{-1}$, see Section 5.5. Thus, the permutation
explainer iterates over almost constant values for the front wheel rotation
angular velocity $\dot{\theta}_{\mathrm{F}}$ with the prediction of the agent
not changing within this iteration. Regarding the reward design, we discuss
whether including the roll angle $\varphi$ in a more sophisticated way than
using the deviation from zero, see Equation (10), would make the consideration
of the part $\rho_{\mathrm{\varphi}}$ beneficial for the learning progress,
i.e. using the deviation from an optimal roll angle that must be computed for
the current position of the bicycle. We also think of using the roll angular
velocity $\dot{\varphi}$ and a threshold $\eta_{\mathrm{\dot{\varphi}}}$
instead of the roll angle $\varphi$ and the roll angle threshold
$\eta_{\mathrm{\varphi}}$ in Equation (10). Initial tests with
$\chi_{1}=\chi_{2}=0.5$ show that with these changes the learning improves in
comparison to that with the alt(5) settings, but is still worse than when only
using $\rho_{\mathrm{y}}$.

In Section 5.3, it is shown that all agents trained with the proposed and
alt(1-4) settings succeed along the type of paths that is used during the
learning process. However, individual agents fail along the benchmark path at
the lower end of the investigated range for the forward velocity. To explain
this, we have a look at the bicycle rides along the benchmark path and find
that the individual agents fail at two specific segments of the benchmark
path, namely the full circle and the hard lane change. We suspect that these
two segments are not part of the learning process and therefore cause
difficulties, which we want to demonstrate in the following. Due to the
geometric properties of the hard lane change, see Figure 5, the hard lane
change does not occur in the paths during learning, see Table 4.3. To estimate
the probability $P$ of a full circle being part of a single path during
learning, it is assumed that for the bicycle travelling at
$2\text{\,}\mathrm{m}\mathrm{s}^{-1}$ the path is $120\text{\,}\mathrm{m}$
long, resulting from $n_{\mathrm{max}}$ that is specified in Section 4. In the
first step, $10^{4}$ paths are generated at random using the reset procedure
described in Section 4.1 and the initialisation intervals from the last part
of the curriculum. Using the expected value for the uniformly distributed
angle $\Phi$ of a circular element, a full circle is made out of approximately
five consecutive circular elements with the same curvature. In the randomly
generated paths, $20$ paths have five circular elements in a row with the same
curvature, resulting in $P=20/10^{4}=0.002$ for the probability of a full
circle. The two segments are therefore most likely neither part of the
learning process nor of the validation process, which, in combination with a
low forward velocity, leads to individual agents failing. The speed-dependence
is due to the increasing difficulty of stabilising the bicycle model as
velocity decreases and the real part of the Whipple bicycle’s eigenvalues
increases [5]. On the one hand, the results for the benchmark path show that
segments that are not part of paths used during the learning process can cause
difficulties for the deployed agents, in particular at low forward velocities.
On the other hand, the results show that most of the deployed agents have
induced a policy with which path following and stabilisation of the bicycle
model can be done at all investigated forward velocities along a path that
includes two segments that the agents statistically have not seen during
learning. Looking, for example, at the agent 1 trained with the proposed
settings, a maximal mean distance of $0.09\text{\,}\mathrm{m}$ is found along
the benchmark path at $7\text{\,}\mathrm{m}\mathrm{s}^{-1}$, being
$9\text{\,}\mathrm{\char 37\relax}$ of the wheelbase of the bicycle model. We
feel that this is very accurate, considering that the benchmark path cannot be
followed exactly due to the step-like curvature changes along the path and the
hard lane change that is challenging to follow especially at higher forward
velocities.

In Section 5.5, SHAP values are used to explain the predictions of an agent.
For a better understanding of this explanation, we discuss a scenario in which
the bicycle model is travelling on a straight and is to turn left. At first,
the elements of the preview vector take on positive values. A combination of
three of these elements, with $t_{\mathrm{P3}}$ being considered the most, see
Figure 13, negatively influences the steering angle predicted by the agent.
Thus, the agent does not steer the bicycle to the left, but to the right. The
agent does counter-steering, which is what we expect the agent to learn, as
explained in Section 4.1. With this counter-steering, the contact points
$\mathrm{P}$ and $\mathrm{Q}$ move to the right, the roll angle $\varphi$ gets
negative, and the bicycle starts to lean into the left turn. As the agent
learns to steer into the direction of the undesired fall, see Section 1, a
negative roll angle $\varphi$ pulls the prediction of the agent up, which
means that the agent turns the handlebar to the left, as the bicycle leans to
the left. The steering angle $\delta$ gets positive and the leaned bicycle
follows the left turn. We find that the agent also considers the roll angular
velocity $\dot{\varphi}$, which is suspected, as some bicycle controllers use
$\dot{\varphi}$ as controlled variable, see for example the intuitive approach
described in [31]. Taken together, in the trained agent we find two
fundamental mechanisms of controlling bicycles considering the SHAP values,
that is, to steer into the direction of the undesired fall and to do counter-
steering.

##  7 Conclusion

Before future research is addressed, the present work is concluded by five key
statements:

  * •

It is demonstrated that a Reinforcement Learning (RL) approach can be used to
do path following with the Whipple bicycle while simultaneously stabilising it
laterally. No stabilisation aids are needed. The agent learns to do path
following and stabilisation of the bicycle model exclusively by setting the
steering angle.

  * •

It is shown that the RL approach works for a wide range of forward velocities
and that a speed-dependent policy is induced.

  * •

Curriculum learning as a training strategy is used to make the RL approach
applicable for the wide range of forward velocities and arbitrary paths.

  * •

An agent is presented that can be used like a conventional controller, being
able to steer the bicycle model along a benchmark path with a maximum mean
distance of $0.09\text{\,}\mathrm{m}$ over the entire velocity range from
$2\text{\,}\mathrm{m}\mathrm{s}^{-1}$ to
$7\text{\,}\mathrm{m}\mathrm{s}^{-1}$. The benchmark path consists of a full
circle, a slalom, curves, two lane changes, and straight elements.

  * •

The explanation of the agent’s predictions shows that the agent has learnt
basic mechanisms for controlling bicycles. This builds a bridge to research in
the field of bicycle dynamics and to controllers designed using control
engineering.

Future research might address the practical realisation of a system where a
trained agent does path following and stabilisation of the bicycle while its
forward velocity is set by the cyclist. Here, above all, the modelling of the
bicycle must be refined in order to facilitate the Sim-to-Real transfer,
explained in the work [32] for RL in the field of robotics. The adaption of
the model concerns not only the bicycle model itself, but also the path on
which the bicycle travels, being, for example, uneven. In addition, future
work must consider a way of obtaining the preview information of the path.
Possible approaches might be camera systems or the usage of mapped cycling
routes.

## Acknowledgments

The computational results presented here have been achieved partly using the
LEO HPC infrastructure of the University of Innsbruck.

## References

  * [1] Choi HS, An J, Han S, Kim JG, Jung JY, Choi J, et al.  Data-driven simulation for general-purpose multibody dynamics using Deep Neural Networks.  Multibody System Dynamics. 2021;51:419-54.  doi:10.1007/s11044-020-09772-8.
  * [2] Gerstmayr J, Manzl P, Pieber M.  Multibody Models Generated from Natural Language.  Multibody System Dynamics. 2024.  Epub ahead of print.  doi:10.1007/s11044-023-09962-0.
  * [3] Hashemi A, Orzechowski G, Mikkola A, McPhee J.  Multibody dynamics and control using machine learning.  Multibody System Dynamics. 2023;58:397-431.  doi:10.1007/s11044-023-09884-x.
  * [4] Zai A, Brown B.  Deep Reinforcement Learning in Action.  Manning Publications; 2020.
  * [5] Meijaard JP, Papadopoulos JM, Ruina A, Schwab AL.  Linearized dynamics equations for the balance and steer of a bicycle: A benchmark and review.  Proc R Soc A. 2007 08;463:1955-82.  doi:10.1098/rspa.2007.1857.
  * [6] Kooijman JDG, Meijaard JP, Papadopoulos JM, Ruina A, Schwab AL.  A Bicycle Can Be Self-Stable Without Gyroscopic or Caster Effects.  Science. 2011;332(6027):339-42.  doi:10.1126/science.1201959.
  * [7] Schwab AL, Meijaard JP.  A review on bicycle dynamics and rider control.  Vehicle System Dynamics. 2013;51:1059-90.  doi:10.1080/00423114.2013.793365.
  * [8] Yin S, Yamakita M.  Passive velocity field control approach to bicycle robot path following.  In: 2016 55th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE); 2016. p. 1654-9.  doi:10.1109/SICE.2016.7749208.
  * [9] Turnwald A, Schäfer M, Liu S.  Passivity-based trajectory tracking control for an autonomous bicycle.  In: IECON 2018-44th Annual Conference of the IEEE Industrial Electronics Society; 2018. p. 2607-12.  doi:10.1109/IECON.2018.8591382.
  * [10] Shafiei MH, Emami M.  Design of a robust path tracking controller for an unmanned bicycle with guaranteed stability of roll dynamics.  Systems Science & Control Engineering. 2019;7(1):12-9.  doi:10.1080/21642583.2018.1555062.
  * [11] Seekhao P, Tungpimolrut K, Parnichkun M.  Development and control of a bicycle robot based on steering and pendulum balancing.  Mechatronics. 2020;69.  doi:10.1016/j.mechatronics.2020.102386.
  * [12] Persson N, Ekström MC, Ekström M, Papadopoulos AV.  Trajectory tracking and stabilisation of a riderless bicycle.  In: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC); 2021. p. 1859-66.  doi:10.1109/ITSC48978.2021.9564958.
  * [13] He K, Deng Y, Wang G, Sun X, Sun Y, Chen Z.  Learning-based trajectory tracking and balance control for bicycle robots with a pendulum: A Gaussian process approach.  IEEE/ASME Transactions on Mechatronics. 2022;27(2):634-44.  doi:10.1109/TMECH.2022.3140885.
  * [14] Randlov J, Alstrøm P.  Learning to Drive a Bicycle Using Reinforcement Learning and Shaping.  In: Proceedings of the Fifteenth International Conference on Machine Learning. ICML ’98. Association for Computing Machinery; 1998. p. 463-71.
  * [15] Le TP, Chung T.  Controlling bicycle using deep deterministic policy gradient algorithm.  In: 2017 14th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI); 2017. p. 413-7.  doi:10.1109/URAI.2017.7992765.
  * [16] Le TP, Choi S, Nguyen QD, Layek MA, Lee S, Chung T.  Toward Self-Driving Bicycles Using State-of-the-Art Deep Reinforcement Learning Algorithms.  Symmetry. 2019;11(2).  doi:10.3390/sym11020290.
  * [17] Cook M.  It takes two neurons to ride a bicycle.  In: Advances in Neural Information Processing Systems 17 (NIPS 2014) (demo); 2004. Available from: https://paradise.caltech.edu/cook/papers/TwoNeurons.pdf.
  * [18] Zhu X, Zheng X, Zhang Q, Chen Z, Liu Y, Liang B.  Natural Residual Reinforcement Learning for Bicycle Robot Control.  In: 2021 IEEE International Conference on Mechatronics and Automation (ICMA); 2021. p. 1201-6.  doi:10.1109/ICMA52036.2021.9512587.
  * [19] Whipple FJ.  The stability of the motion of a bicycle.  Quarterly Journal of Pure and Applied Mathematics. 1899;30(120):312-48.  Available from: http://ruina.tam.cornell.edu/research/topics/bicycle_mechanics/Whipple.pdf.
  * [20] Xiong J, Wang N, Liu C.  Stability analysis for the Whipple bicycle dynamics.  Multibody System Dynamics. 2020 03;48:311-35.  doi:10.1007/s11044-019-09707-y.
  * [21] Kooijman JDG, Schwab AL, Meijaard JP.  Experimental validation of a model of an uncontrolled bicycle.  Multibody System Dynamics. 2008;19:115-32.  doi:10.1007/s11044-007-9050-x.
  * [22] Haarnoja T, Zhou A, Abbeel P, Levine S.  Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor.  ArXiv preprint. 2018.  Preprint.  doi:10.48550/arXiv.1801.01290.
  * [23] Gerstmayr J.  Exudyn – A C++ based Python package for flexible multibody systems.  Multibody System Dynamics. 2023;60:533-61.  doi:10.1007/s11044-023-09937-1.
  * [24] Brockman G, Cheung V, Pettersson L, Schneider J, Schulman J, Tang J, et al.  OpenAI Gym.  ArXiv preprint. 2016.  Preprint.  doi:10.48550/arXiv.1606.01540.
  * [25] Raffin A, Hill A, Gleave A, Kanervisto A, Ernestus M, Dormann N.  Stable-Baselines3: Reliable Reinforcement Learning Implementations.  Journal of Machine Learning Research. 2021;22(268):1-8.  Available from: http://jmlr.org/papers/v22/20-1364.html.
  * [26] Astrom KJ, Klein RE, Lennartsson A.  Bicycle dynamics and control: adapted bicycles for education and research.  IEEE Control Systems Magazine. 2005;25(4):26-47.  doi:10.1109/MCS.2005.1499389.
  * [27] Bengio Y, Louradour J, Collobert R, Weston J.  Curriculum learning.  In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09. Association for Computing Machinery; 2009. p. 41-8.  doi:10.1145/1553374.1553380.
  * [28] Mnih V, Badia AP, Mirza M, Graves A, Lillicrap T, Harley T, et al.  Asynchronous methods for deep reinforcement learning.  In: Proceedings of The 33rd International Conference on Machine Learning; 2016. p. 1928-37.  doi:10.48550/arXiv.1602.01783.
  * [29] Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O.  Proximal Policy Optimization Algorithms.  ArXiv preprint. 2017.  Preprint.  doi:10.48550/arXiv.1707.06347.
  * [30] Lundberg SM, Lee SI.  A Unified Approach to Interpreting Model Predictions.  In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, et al., editors. Advances in Neural Information Processing Systems 30 (NIPS 2017). vol. 30; 2017. Available from: https://papers.nips.cc/paper/2017.
  * [31] Schwab AL, Kooijman JDG, Meijaard JP.  Some recent developments in bicycle dynamics and control.  In: Fourth European Conference on Structural Control (4ECSC); 2008. Available from: http://bicycle.tudelft.nl/schwab/Publications/SchwabKooijmanMeijaard2008.pdf.
  * [32] Zhao W, Queralta JP, Westerlund T.  Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey.  In: 2020 IEEE Symposium Series on Computational Intelligence (SSCI); 2020\. p. 737-44.  doi:10.1109/SSCI47803.2020.9308468.
  * [33] Henderson DM.  Euler angles, quaternions, and transformation matrices for space shuttle analysis.  NASA; 1977.  Available from: https://ntrs.nasa.gov/citations/19770019231.
  * [34] Psiaki ML.  Bicycle Stability: A Mathematical And Numerical Analysis [bathesis].  Princeton University; 1979.  Available from: http://ruina.tam.cornell.edu/research/topics/bicycle_mechanics/Psiaki_Princeton_thesis.pdf.
  * [35] Peterson D, Hubbard M.  Analysis of the Holonomic Constraint in the Whipple Bicycle Model.  In: Estivalet M, Brisson P, editors. The Engineering of Sport 7. vol. 2; 2008. doi:10.1007/978-2-287-99056-4_75.

##  Appendix A Coordinates Mappings for the Whipple Bicycle

In this section, it is shown how the output of a multibody simulation of the
Whipple bicycle can be mapped to the minimal coordinates of the bicycle model.
Afterwards, the mapping from the bicycle’s minimal coordinates to the
redundant coordinates is described. The relation between the minimal
coordinates and the redundant coordinates on both the positional and velocity
base of this non-holonomic system is shown in detail. In Table A the notation
used in this section is given. The reference frames and geometric properties
drawn in Figure 14 are used in the following.

Notation and spatial transformations used for the coordinates mappings.
notation description $\mathbf{0}$ zero matrix (or zero vector) $\mathbf{1}$
unity matrix
$\prescript{\mathrm{K}}{}{\mathbf{r}}_{\mathrm{P_{1}}\mathrm{P_{2}}}\,$ vector
pointing from the point $\mathrm{P_{1}}$ to the point $\mathrm{P_{2}}$ with
its coordinates formulated in the $\mathrm{K}$-frame
$\prescript{\mathrm{K}\mathrm{M}}{}{\mathbf{R}}\,$ rotation that rotates
coordinates of vectors formulated in the $\mathrm{M}$-frame into the
$\mathrm{K}$-frame:
$\prescript{\mathrm{K}}{}{\mathbf{r}}\,=\prescript{\mathrm{K}\mathrm{M}}{}{\mathbf{R}}\,\cdot\prescript{\mathrm{M}}{}{\mathbf{r}}\,$
$\prescript{\mathrm{K}\mathrm{M}}{}{\mathbf{T}}\,$ homogeneous transformation
that transforms coordinates of vectors formulated in the $\mathrm{M}$-frame
into the $\mathrm{K}$-frame $\widetilde{\mathbf{a}}$ skew symmetric matrix of
the vector $\mathbf{a}$; used to calculate the cross product
$\mathbf{c}=\mathbf{a}\times\mathbf{b}$ as a matrix vector multiplication:
$\mathbf{c}=\widetilde{\mathbf{a}}\cdot\mathbf{b}$
$\mathrm{Rot_{\mathrm{e}}\left(\alpha\right)}$ rotation by $\alpha$ around the
unity vector $\mathbf{e}$

Figure 14: Whipple bicycle shown in the reference configuration, see also
Figure 1, with characteristic geometric parameters and coordinate systems that
are needed for the coordinates mappings. Each coordinate system
$\mathcal{K}_{i}$ is described by its origin and unity vectors
$\mathbf{x}_{i}$, $\mathbf{y}_{i}$, and $\mathbf{z}_{i}$.

The global $\mathrm{0}$-frame, the $\mathrm{T_{1}}$-frame, and the
$\mathrm{T_{3}}$-frame lie in the ground plane. The origin of the latter two
frames each lies in the contact point of the wheel and its z-axis points
towards the respective wheel hub. The $\mathrm{R}$-, $\mathrm{B}$-,
$\mathrm{H}$-, and $\mathrm{F}$-frame are body-fixed frames of the eponymous
bodies with their origin each being located at the COM of the rigid body. The
$\mathrm{R}$\- and the $\mathrm{F}$-frame are defined in a way such that their
x-axes are the rotation axes of the wheels. The $\mathrm{B}$\- and the
$\mathrm{H}$-frame can each be translated to two further points on the
corresponding rigid body which is indicated with the subscript $\mathrm{1}$
and $\mathrm{2}$ for the rear body and with $\mathrm{2}$ and $\mathrm{3}$ for
the handlebar. Two reference frames are marked with a star. They are obtained
by rotating the $\mathrm{B}$-frame by the negated frame angle
$\mathrm{\alpha}$ around its y-axis and the $\mathrm{H}$-frame by the negated
steering axis tilt $\mathrm{\lambda}$ around its y-axis. In the following
mappings also the two wheel radii $\mathrm{r_{R}}$ and $\mathrm{r_{F}}$, the
frame length $\mathrm{d_{1}}$, the fork length $\mathrm{d_{2}}$, and the fork
offset $\mathrm{d_{3}}$ are used.

###  A.1 Mapping of Redundant to Minimal Coordinates using Sensors

To calculate the minimal coordinates, sensors are defined in the interface of
the multibody simulation code. One sensor measures the position of the contact
point $\mathrm{P}$ and thus provides the minimal coordinates $x_{\mathrm{P}}$
and $y_{\mathrm{P}}$. Another sensor gives the rotation matrix
$\prescript{\mathrm{0}\mathrm{B}}{}{\mathbf{R}}\,$ describing the spatial
orientation of the rear body in the global frame. Since the global frame can
be turned into the $\mathrm{B}$-frame by performing a sequence of turns

$\prescript{\mathrm{0}\mathrm{B}}{}{\mathbf{R}}\,=\mathrm{Rot_{z}\left(\Psi\right)}\cdot\mathrm{Rot_{x}\left(\varphi\right)}\cdot\mathrm{Rot_{y}\left(\theta_{\mathrm{B}}\right)}\,,$
(A1)

the yaw angle $\Psi$ and the roll angle $\varphi$ can be computed using the
equations given in [33]. The angle $\theta_{\mathrm{B}}$ is the pitch of the
frame and not further needed for the mapping of redundant to minimal
coordinates. The steering angle $\delta$, the rear wheel rotation angle
$\theta_{\mathrm{R}}$, and the front wheel rotation angle
$\theta_{\mathrm{F}}$ are output variables of sensors measuring the
corresponding revolute joint angles. The roll angular velocity $\dot{\varphi}$
is the component of the rear body angular velocity in the direction of the
x-axis of the $\mathrm{T_{1}}$-frame. Thus, it can be written as the inner
product

$\dot{\varphi}=\langle\prescript{\mathrm{0}}{}{\mathbf{x}}_{\mathrm{T_{1}}}\,,\prescript{\mathrm{0}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{B}}\,\rangle\,,$ (A2)

where
$\prescript{\mathrm{0}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{B}}\,$ is the output quantity of a sensor attached to the rear body.
The x-axis $\prescript{\mathrm{0}}{}{\mathbf{x}}_{\mathrm{T_{1}}}\,$ is known
via a homogeneous transformation matrix reading

$\prescript{\mathrm{0}\mathrm{T_{1}}}{}{\mathbf{T}}\,=\underbrace{\left[\begin{array}[]{cc}\mathrm{Rot_{z}\left(\Psi\right)}&\prescript{\mathrm{0}}{}{\mathbf{r}}_{\mathrm{0}\mathrm{P}}\vspace{0.1cm}\\\
\mathbf{0}&1\end{array}\right]}_{\prescript{\mathrm{0}\mathrm{C}}{}{\mathbf{T}}\,}\cdot\left[\begin{array}[]{cc}\mathrm{Rot_{x}\left(\varphi\right)}&\mathbf{0}\vspace{0.1cm}\\\
\mathbf{0}&1\end{array}\right]\,.$ (A3)

The remaining two minimal coordinates, namely the steering angular velocity
$\dot{\delta}$ and the front wheel rotation angular velocity
$\dot{\theta}_{\mathrm{F}}$ are measured by sensors attached to the
corresponding revolute joint.

###  A.2 Mapping of Minimal to Redundant Coordinates

In the following, it is shown how the redundant coordinates on positional and
velocity base of the four rigid bodies the Whipple bicycle consists of can be
computed from the minimal coordinates of the Whipple bicycle.

At first, four homogeneous transformation matrices are defined, representing
the position and orientation of the bodies. The pose of the rear body is given
by

$\prescript{\mathrm{0}\mathrm{B}}{}{\mathbf{T}}\,=\underbrace{\prescript{\mathrm{0}\mathrm{T_{1}}}{}{\mathbf{T}}\,\cdot\left[\begin{array}[]{cc}\mathrm{Rot_{y}\left(\theta_{\mathrm{B}}\right)}&\prescript{\mathrm{T_{1}}}{}{\mathbf{r}}_{\mathrm{P}\mathrm{P_{1}}}\vspace{0.1cm}\\\
\mathbf{0}&1\end{array}\right]}_{\prescript{\mathrm{0}\mathrm{B_{1}}}{}{\mathbf{T}}\,}\cdot\left[\begin{array}[]{cc}\mathbf{1}&\prescript{\mathrm{B}}{}{\mathbf{r}}_{\mathrm{P_{1}}\mathrm{B_{COM}}}\vspace{0.1cm}\\\
\mathbf{0}&1\end{array}\right]\,.$ (A4)

$\prescript{\mathrm{0}\mathrm{T_{1}}}{}{\mathbf{T}}\,$ follows from equation
(A3). The pitch angle $\theta_{\mathrm{B}}$ satisfies the fact that the rear
body can pitch, which means that it can rotate around its y-axis, see Figure
2. In [34] it is shown that an analytical solution for $\theta_{\mathrm{B}}$
given the bicycle minimal coordinates exists. In [35] a geometric way to
derive a fourth degree polynomial in $\theta_{\mathrm{B}}$ is introduced. In
Appendix C a convenient way using rotation matrices to get the polynomial in
$\theta_{\mathrm{B}}$ and thus $\theta_{\mathrm{B}}$ is derived. The rear
wheel is represented with

$\prescript{\mathrm{0}\mathrm{R}}{}{\mathbf{T}}\,=\prescript{\mathrm{0}\mathrm{B_{1}}}{}{\mathbf{T}}\,\cdot\left[\begin{array}[]{cc}\mathrm{Rot_{z}\left(\frac{\pi}{2}\right)}&\mathbf{0}\vspace{0.1cm}\\\
\mathbf{0}&1\end{array}\right]\cdot\left[\begin{array}[]{cc}\mathrm{Rot_{x}\left(\theta_{\mathrm{R}}\right)}&\mathbf{0}\vspace{0.1cm}\\\
\mathbf{0}&1\end{array}\right]\,.$ (A5)

The transformation $\prescript{\mathrm{0}\mathrm{B_{1}}}{}{\mathbf{T}}\,$ used
therein results from Equation (A4). Using Equation (A4), also the handle can
be represented as follows:

$\prescript{\mathrm{0}\mathrm{H}}{}{\mathbf{T}}\,=\prescript{\mathrm{0}\mathrm{B}}{}{\mathbf{T}}\,\cdot\left[\begin{array}[]{cc}\mathrm{Rot_{y}\left(-\mathrm{\lambda}\right)}&\prescript{\mathrm{B}}{}{\mathbf{r}}_{\mathrm{B_{COM}}\mathrm{H_{COM}}}\vspace{0.1cm}\\\
\mathbf{0}&1\end{array}\right]\cdot\left[\begin{array}[]{cc}\mathrm{Rot_{z}\left(\delta\right)}&\mathbf{0}\vspace{0.1cm}\\\
\mathbf{0}&1\end{array}\right]\cdot\left[\begin{array}[]{cc}\mathrm{Rot_{y}\left(\mathrm{\lambda}\right)}&\mathbf{0}\vspace{0.1cm}\\\
\mathbf{0}&1\end{array}\right]\,.$ (A6)

The pose of the front wheel reads

$\prescript{\mathrm{0}\mathrm{F}}{}{\mathbf{T}}\,=\prescript{\mathrm{0}\mathrm{H}}{}{\mathbf{T}}\,\cdot\left[\begin{array}[]{cc}\mathrm{Rot_{z}\left(\frac{\pi}{2}\right)}&\prescript{\mathrm{H}}{}{\mathbf{r}}_{\mathrm{H_{COM}}\mathrm{P_{3}}}\vspace{0.1cm}\\\
\mathbf{0}&1\end{array}\right]\cdot\left[\begin{array}[]{cc}\mathrm{Rot_{x}\left(\theta_{\mathrm{F}}\right)}&\mathbf{0}\vspace{0.1cm}\\\
\mathbf{0}&1\end{array}\right]\,,$ (A7)

where $\prescript{\mathrm{0}\mathrm{H}}{}{\mathbf{T}}\,$ is taken from
Equation (A6).

Furthermore, the translational and angular velocities of the rigid bodies must
be computed, using the bicycle’s minimal coordinates. The following sequence
of computations is followed in the present work:

  1. 1.

angular velocity of $\mathrm{B}$-frame

  2. 2.

angular velocity of $\mathrm{H}$-frame

  3. 3.

angular velocity of $\mathrm{F}$-frame

  4. 4.

translational velocity of $\mathrm{F}$-frame

  5. 5.

translational velocity of $\mathrm{H}$-frame

  6. 6.

translational velocity of $\mathrm{B}$-frame

  7. 7.

translational velocity of $\mathrm{R}$-frame

  8. 8.

angular velocity of $\mathrm{R}$-frame

It is started with the angular velocity of the rear body, that reads

$\prescript{\mathrm{0}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{B}}\,=\left[\begin{array}[]{c}{0\vspace{0.1cm}}\\\
{0\vspace{0.1cm}}\\\
{\dot{\Psi}}\end{array}\right]+\prescript{\mathrm{0}\mathrm{T_{1}}}{}{\mathbf{R}}\,\left[\begin{array}[]{c}{\dot{\varphi}\vspace{0.1cm}}\\\
{\dot{\theta}_{\mathrm{B}}\vspace{0.1cm}}\\\ {0}\end{array}\right]\,.$ (A8)

In Appendix C it is described how the pitch angular velocity
$\dot{\theta}_{\mathrm{B}}$ can be computed using rotation matrices. The yaw
angular velocity $\dot{\Psi}$ results from

$\dot{\Psi}=\frac{1}{w}\sin{(\mu_{1}-\Psi)}v\,,$ (A9)

where the forward velocity $v$ of the bicycle is computed using the front
wheel rotation angular velocity $\dot{\theta}_{\mathrm{F}}$ and the radius
$\mathrm{r_{F}}$ of the front wheel:
$v=\mathrm{r_{F}}\cdot\dot{\theta}_{\mathrm{F}}$. The wheelbase $w$ of the
bicycle is computed using Equation (C6). The angle $\mu_{1}$ describes the
direction of motion of the front wheel contact point $\mathrm{Q}$. Since the
rotation matrix $\prescript{\mathrm{0}\mathrm{F}}{}{\mathbf{R}}\,$ is already
known from Equation (A7), this angle can be calculated by taking

$\prescript{\mathrm{0}\mathrm{F}}{}{\mathbf{R}}\,=\mathrm{Rot_{z}\left(\mu_{1}\right)}\cdot\mathrm{Rot_{x}\left(\mu_{2}\right)}\cdot\mathrm{Rot_{y}\left(\mu_{3}\right)}$
(A10)

into account, using the formulas shown in [33]. The angles $\mu_{2}$ and
$\mu_{3}$ are not needed. The angular velocity of the handle is computed using
the relative angular velocity between the rear body and the handle which
results exclusively from the revolute joint connecting the rear body and the
handle. It follows

$\prescript{\mathrm{B}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{B}\mathrm{H}}\,=\mathrm{Rot_{y}\left(-\mathrm{\lambda}\right)}\left[\begin{array}[]{c}{0\vspace{0.1cm}}\\\
{0\vspace{0.1cm}}\\\ {\dot{\delta}}\end{array}\right]\,,$ (A11)

with $\mathrm{\lambda}$ being the steering axis tilt. Finally, for the handle
applies

$\prescript{\mathrm{0}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{H}}\,=\prescript{\mathrm{0}\mathrm{B}}{}{\mathbf{R}}\,\left(\prescript{\mathrm{B}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{B}}\,+\prescript{\mathrm{B}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{B}\mathrm{H}}\,\right)\,.$ (A12)

The same procedure is used to get the angular velocity of the front wheel

$\prescript{\mathrm{0}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{F}}\,=\prescript{\mathrm{0}\mathrm{H}}{}{\mathbf{R}}\,\left(\prescript{\mathrm{H}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{H}}\,+\prescript{\mathrm{H}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{H}\mathrm{F}}\,\right)\,,$ (A13)

where the relative angular velocity between the handle and the wheel results
from the front wheel hub and reads

$\prescript{\mathrm{H}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{H}\mathrm{F}}\,=\left[\begin{array}[]{ccc}0&\dot{\theta}_{\mathrm{F}}&0\end{array}\right]^{\mathrm{T}}\,.$
(A14)

The translational velocity of the front wheel follows with Euler’s first
theorem for kinematics to

$\prescript{\mathrm{0}}{}{\mathbf{v}}_{\mathrm{F}}\,=\prescript{\mathrm{0}}{}{\mathbf{v}}_{\mathrm{H_{3}}}\,=\prescript{\mathrm{0}}{}{\mathbf{v}}_{\mathrm{T_{3}}}\,+\prescript{\mathrm{0}}{}{\,\widetilde{\mbox{$\boldsymbol{\omega}$}\\!\\!}}_{\leavevmode\nobreak\
\mathrm{T_{3}}}\,\prescript{\mathrm{0}}{}{\mathbf{r}}_{\mathrm{Q}\mathrm{P_{3}}}\,\,.$
(A15)

The therein used velocity of the front wheel contact point $\mathrm{Q}$, that
is $\prescript{\mathrm{0}}{}{\mathbf{v}}_{\mathrm{T_{3}}}\,$, is computed
using the angle $\mu_{1}$ and the bicycle’s forward velocity $v$.

$\prescript{\mathrm{0}}{}{\mathbf{v}}_{\mathrm{T_{3}}}\,=v\left[\begin{array}[]{c}{\cos{\mu_{1}\vspace{0.1cm}}}\\\
{\sin{\mu_{1}\vspace{0.1cm}}}\\\ {0}\end{array}\right]$ (A16)

Remember that $v$ is computed using the front wheel rotation angular velocity
$\dot{\theta}_{\mathrm{F}}$. The angular velocity of the
$\mathrm{T_{3}}$-frame needed for Equation (A15) results from that of the
front wheel, but without its rotation angular velocity:

$\prescript{\mathrm{0}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{T_{3}}}\,=\prescript{\mathrm{0}\mathrm{T_{3}}}{}{\mathbf{R}}\,\left(\prescript{\mathrm{T_{3}}\mathrm{0}}{}{\mathbf{R}}\,\prescript{\mathrm{0}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{F}}\,+\left[\begin{array}[]{c}{0\vspace{0.1cm}}\\\
{-\dot{\theta}_{\mathrm{F}}\vspace{0.1cm}}\\\ {0}\end{array}\right]\right)\,.$
(A17)

Now, $\prescript{\mathrm{0}}{}{\mathbf{v}}_{\mathrm{F}}\,$ can be computed and
it is continued with the velocity of the handle, reading

$\prescript{\mathrm{0}}{}{\mathbf{v}}_{\mathrm{H}}\,=\prescript{\mathrm{0}}{}{\mathbf{v}}_{\mathrm{H_{3}}}\,+\prescript{\mathrm{0}}{}{\,\widetilde{\mbox{$\boldsymbol{\omega}$}\\!\\!}}_{\leavevmode\nobreak\
\mathrm{H}}\,\prescript{\mathrm{0}}{}{\mathbf{r}}_{\mathrm{P_{3}}\mathrm{H_{COM}}}\,\,.$
(A18)

With the translational velocity of the bicycle’s head

$\prescript{\mathrm{0}}{}{\mathbf{v}}_{\mathrm{B_{2}}}\,=\prescript{\mathrm{0}}{}{\mathbf{v}}_{\mathrm{H_{2}}}\,=\prescript{\mathrm{0}}{}{\mathbf{v}}_{\mathrm{H}}\,+\prescript{\mathrm{0}}{}{\,\widetilde{\mbox{$\boldsymbol{\omega}$}\\!\\!}}_{\leavevmode\nobreak\
\mathrm{H}}\,\prescript{\mathrm{0}}{}{\mathbf{r}}_{\mathrm{H_{COM}}\mathrm{P_{2}}}\,\,,$
(A19)

the one of the rear body follows to

$\prescript{\mathrm{0}}{}{\mathbf{v}}_{\mathrm{B}}\,=\prescript{\mathrm{0}}{}{\mathbf{v}}_{\mathrm{B_{2}}}\,+\prescript{\mathrm{0}}{}{\,\widetilde{\mbox{$\boldsymbol{\omega}$}\\!\\!}}_{\leavevmode\nobreak\
\mathrm{B}}\,\prescript{\mathrm{0}}{}{\mathbf{r}}_{\mathrm{P_{2}}\mathrm{B_{COM}}}\,\,.$
(A20)

The translational velocity of the rear wheel reads

$\prescript{\mathrm{0}}{}{\mathbf{v}}_{\mathrm{R}}\,=\prescript{\mathrm{0}}{}{\mathbf{v}}_{\mathrm{B_{1}}}\,=\prescript{\mathrm{0}}{}{\mathbf{v}}_{\mathrm{B}}\,+\prescript{\mathrm{0}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{B}}\,\prescript{\mathrm{0}}{}{\mathbf{r}}_{\mathrm{B_{COM}}\mathrm{P_{1}}}\,\,.$
(A21)

Finally, the angular velocity of the rear wheel can be expressed by

$\prescript{\mathrm{0}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{R}}\,=\prescript{\mathrm{0}\mathrm{B}}{}{\mathbf{R}}\,\left(\prescript{\mathrm{B}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{B}}\,+\prescript{\mathrm{B}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{B}\mathrm{R}}\,\right)\,.$ (A22)

The relative angular velocity between the rear body and the rear wheel results
from the rear wheel hub. It is denoted as

$\prescript{\mathrm{B}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{B}\mathrm{R}}\,=\left[\begin{array}[]{ccc}0&\dot{\theta}_{R}&0\end{array}\right]^{\mathrm{T}}\,.$
(A23)

Since the therein used rear wheel rotation angular velocity is not an element
of the state vector, see Equation (1), it must be computed. Using the angular
velocity of the $\mathrm{T_{1}}$-frame that reads

$\prescript{\mathrm{T_{1}}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{T_{1}}}\,=\prescript{\mathrm{T_{1}}\mathrm{0}}{}{\mathbf{R}}\,\prescript{\mathrm{0}}{}{\mbox{$\boldsymbol{\omega}$}\\!\\!}_{\leavevmode\nobreak\
\mathrm{B}}\,+\left[\begin{array}[]{c}{0\vspace{0.1cm}}\\\
{-\dot{\theta}_{\mathrm{B}}\vspace{0.1cm}}\\\ {0}\end{array}\right]\,,$ (A24)

the translational velocity of it in its own frame can be computed as shown
below.

$\prescript{\mathrm{T_{1}}}{}{\mathbf{v}}_{\mathrm{T_{1}}}\,=\prescript{\mathrm{T_{1}}}{}{\mathbf{v}}_{\mathrm{R}}\,+\prescript{\mathrm{T_{1}}}{}{\,\widetilde{\mbox{$\boldsymbol{\omega}$}\\!\\!}}_{\leavevmode\nobreak\
\mathrm{T_{1}}}\,\prescript{\mathrm{T_{1}}}{}{\mathbf{r}}_{\mathrm{P_{1}}\mathrm{P}}\,$
(A25)

Only the x-component of this vector is different from zero, which results from
the fact that the wheels are slip free to the side. Consequently, the absolute
velocity of the rear contact point $\mathrm{P}$ reads
$v_{\mathrm{T_{1}}}=\left[\begin{array}[]{ccc}1&0&0\end{array}\right]\cdot\prescript{\mathrm{T_{1}}}{}{\mathbf{v}}_{\mathrm{T_{1}}}\,$.
The velocity $v_{\mathrm{T_{1}}}$ only results from the rear wheel rotational
angular velocity $\dot{\theta}_{\mathrm{R}}$. Thus, the rear wheel rotation
angular velocity can be computed with

$\dot{\theta}_{\mathrm{R}}=\frac{v_{\mathrm{T_{1}}}}{\mathrm{r_{R}}}\,,$ (A26)

where the radius $\mathrm{r_{F}}$ of the rear wheel is used. This completes
the mapping from the minimal to the redundant coordinates.

##  Appendix B Threshold Value for the Steering Angle of the Whipple Bicycle
and Step Response of the Steering Angle

In this section, a threshold value for the steering angle $\delta$ of the
Whipple bicycle is defined. The threshold value is used to define a range in
which the set value $\delta_{\mathrm{set}}$ for the steering angle the agent
can output must be. With this threshold value, the step response of the
steering angle $\delta$ is shown, which may be used to design controllers for
the bicycle model, such as a PD controller mapping a set value
$\delta_{\mathrm{set}}$ for the steering angle to a steering torque.

As a result of the tilted steering axis and the fork offset of the Whipple
bicycle, the direction of motion of the front wheel contact point $\mathrm{Q}$
relative to the $\mathrm{T_{1}}$-frame is not equal to the steering angle
$\delta$ of the Whipple bicycle, see Table B. In the table it can be seen that
the direction of motion of the front wheel contact point, given by the angle
$\mu_{1}$, deviates significantly from the steering angle $\delta$, especially
in configurations of the Whipple bicycle where $|\varphi|>>0$ applies. The
angle $\mu_{1}$ is computed using Equation (A10). Since the magnitude of the
roll angle of the bicycle model in the present work should not exceed the
threshold $\eta_{\mathrm{\varphi}}=$$$, see Section 3.3, and the direction of
motion of the front wheel contact point $\mathrm{Q}$ must not exceed $None$, a
limit for the steering angle $\delta$ can be defined. It is assumed that
limiting the steering angle $\delta$ to an value of $None$ is sufficient
(although $|\mu_{1}|$ slightly exceeds $None$ when the bicycle is rolled by
$|\varphi|=$$$). Consequently, the set value $\delta_{\mathrm{set}}$ the agent
can output for the steering angle must be in the interval of $None$ to $None$.

Tabular representation of the dependency between the roll angle $\varphi$, the
steering angle $\delta$, and the direction of motion $\mu_{1}$ of the front
wheel contact point $\mathrm{Q}$ of the Whipple bicycle. The minimal
coordinates except of $\varphi$ and $\delta$ of the Whipple bicycle are
zeroed.  roll angle $\varphi$ steering angle $\delta$ direction of motion of
$\mathrm{Q}$ given by $\mu_{1}$ $None$ $None$ $None$ $None$ $None$ $None$
$None$ $None$ $None$ $None$ $None$ $None$ $None$ $None$ $None$ $None$ $None$
$None$ $None$ $None$ $None$ $None$ $None$ $None$ $None$ $None$ $None$

The step response of the steering angle $\delta$ is used to determine the two
parameters $P$ and $D$ of a PD controller that gives the torque $\tau$ that is
imprinted between the rear body and the fork of the bicycle model. To get the
step response, the stationary bicycle model is simulated when the set value
$\delta_{\mathrm{set}}$ for the steering angle follows

$\delta_{\mathrm{set}}(t)=\begin{cases}$$&t<0\\\ $$&t\geq 0\,.\end{cases}$
(B1)

For this particular simulation, the gravitational acceleration is zeroed so
that the bicycle does not fall over. In Figure 15 the step response of the
steering angle $\delta$ is shown. It is defined that the two parameters $P$
and $D$ should be set in a way so that the maximal value of the step response
does not overshoot the set value by more than $10\text{\,}\%$. By setting
$P=$9\text{\,}\mathrm{N}\mathrm{m}$$ and
$D=$1.6\text{\,}\mathrm{N}\mathrm{m}\mathrm{s}$$ the specified criterion is
fulfilled.

Figure 15: Step response of the steering angle $\delta$ with the bicycle being
stationary when the set value $\delta_{\mathrm{set}}$ for the steering angle
follows the function shown. To obtain the step response, the gravitational
acceleration is zeroed. The defined maximal permissible value
$\delta_{\mathrm{max}}$ for the step response is also shown.

##  Appendix C Pitch Angle and Pitch Angular Velocity of the Whipple Bicycle
with Rotation Matrices

In the following, a way to compute the pitch angle $\theta_{\mathrm{B}}$ of
the Whipple bicycle’s rear body using rotation matrices is shown, when the
minimal coordinates of the bicycle model are given. The notation and reference
frames that are used in Appendix A are also used in the following.

It is started by getting the z-axis
$\prescript{\mathrm{0}}{}{\mathbf{z}}_{\mathrm{T_{1}}}\,$ from the
$\mathrm{T_{1}}$-frame formulated in the global $\mathrm{0}$-frame. This
vector is the last column of the rotation matrix

$\prescript{\mathrm{0}\mathrm{T_{1}}}{}{\mathbf{R}}\,=\mathrm{Rot_{z}\left(\Psi\right)}\cdot\mathrm{Rot_{x}\left(\varphi\right)}\,.$
(C1)

To rotate the $\mathrm{0}$-frame into the $\mathrm{B^{*}}$-frame, the rotation
matrix

$\prescript{\mathrm{0}\mathrm{B^{*}}}{}{\mathbf{R}}\,=\underbrace{\prescript{\mathrm{0}\mathrm{T_{1}}}{}{\mathbf{R}}\,\cdot\mathrm{Rot_{y}\left(\theta_{\mathrm{B}}\right)}}_{\prescript{\mathrm{0}\mathrm{B}}{}{\mathbf{R}}\,}\cdot\mathrm{Rot_{y}\left(-\mathrm{\alpha}\right)}$
(C2)

is used, with the pitch angle $\theta_{\mathrm{B}}$ being unknown. The x-axis
$\prescript{\mathrm{0}}{}{\mathbf{x}}_{\mathrm{B^{*}}}\,$ is the first column
of this matrix. The rotation matrix for the $\mathrm{H^{*}}$-frame follows

$\prescript{\mathrm{0}\mathrm{H^{*}}}{}{\mathbf{R}}\,=\prescript{\mathrm{0}\mathrm{B}}{}{\mathbf{R}}\,\cdot\underbrace{\mathrm{Rot_{y}\left(-\mathrm{\lambda}\right)}\cdot\mathrm{Rot_{z}\left(\delta\right)}}_{\prescript{\mathrm{B}\mathrm{H^{*}}}{}{\mathbf{R}}\,}\,.$
(C3)

The axes $\prescript{\mathrm{0}}{}{\mathbf{x}}_{\mathrm{H^{*}}}\,$,
$\prescript{\mathrm{0}}{}{\mathbf{y}}_{\mathrm{H^{*}}}\,$, and
$\prescript{\mathrm{0}}{}{\mathbf{z}}_{\mathrm{H^{*}}}\,$ are the columns of
this matrix. As shown in [35], it is possible to calculate a unity vector
$\prescript{\mathrm{0}}{}{\mathbf{h}}\,$ that points from the front wheel hub
in the direction of $\mathrm{Q}$ by performing two steps:

$\prescript{\mathrm{0}}{}{\mathbf{f}}\,=-\left[\begin{array}[]{c}{0\vspace{0.1cm}}\\\
{0\vspace{0.1cm}}\\\
{1}\end{array}\right]+\left(\prescript{\mathrm{0}}{}{\mathbf{y}}_{\mathrm{H^{*}}}\,\cdot\left[\begin{array}[]{c}{0\vspace{0.1cm}}\\\
{0\vspace{0.1cm}}\\\
{1}\end{array}\right]\right)\,,\qquad\prescript{\mathrm{0}}{}{\mathbf{h}}\,=\frac{\prescript{\mathrm{0}}{}{\mathbf{f}}\,}{|\mathbf{f}|}\,.$
(C4)

A vector pointing from $\mathrm{P}$ to $\mathrm{Q}$ can be computed by

$\prescript{\mathrm{0}}{}{\mathbf{r}}_{\mathrm{P}\mathrm{Q}}\,=\mathrm{r_{R}}\prescript{\mathrm{0}}{}{\mathbf{z}}_{\mathrm{T_{1}}}\,+\mathrm{d_{1}}\prescript{\mathrm{0}}{}{\mathbf{x}}_{\mathrm{B^{*}}}\,-\mathrm{d_{2}}\prescript{\mathrm{0}}{}{\mathbf{z}}_{\mathrm{H^{*}}}\,+\mathrm{d_{3}}\prescript{\mathrm{0}}{}{\mathbf{x}}_{\mathrm{H^{*}}}\,+\mathrm{r_{F}}\prescript{\mathrm{0}}{}{\mathbf{h}}\,\,.$
(C5)

Here, $\mathrm{r_{R}}$, $\mathrm{r_{F}}$, $\mathrm{d_{1}}$, $\mathrm{d_{2}}$,
and $\mathrm{d_{3}}$ are bicycle geometries independent of the bicycle’s
configuration. The wheelbase $w$ of the Whipple bicycle in its current
configuration reads

$w=|\mathbf{r}_{\mathrm{P}\mathrm{Q}}|\,.$ (C6)

Setting the constraint that both wheels each touch the ground at one point
leads to a scalar equation reading

$\prescript{\mathrm{0}}{}{\mathbf{r}}_{\mathrm{P}\mathrm{Q}}\,\left[\begin{array}[]{ccc}0&0&1\end{array}\right]^{\mathrm{T}}=0\,.$
(C7)

If the bicycle’s minimal coordinates are given, the only unknown in this
equation is the pitch angle $\theta_{\mathrm{B}}$. Consequently, a calculation
of it with Newton’s method with the initial guess $\theta_{\mathrm{B}}=0$ [35]
is possible.

A function for the pitch angular velocity $\dot{\theta}_{\mathrm{B}}$ of the
shape
$\dot{\theta}_{\mathrm{B}}(\theta_{\mathrm{B}},\varphi,\delta,\dot{\varphi},\dot{\delta})$
can be gained by deriving Equation (C7) with respect to time. Thus, also the
pitch angular velocity $\dot{\theta}_{\mathrm{B}}$ is known.