# Autonomous Control of a Novel Closed Chain Five Bar Active Suspension via
Deep Reinforcement Learning

Nishesh Singh  Department of Mechanical Engineering, Manipal Institute of
Technology  Sidharth Ramesh  Department of Computer Science Engineering,
Manipal Institute of Technology  Jyotishka Duttagupta  Department of
Mechanical Engineering, Manipal Institute of Technology  Abhishek Sankar
Department of Mechanical Engineering, Manipal Institute of Technology  Leander
Stephen D’Souza  Department of Mechatronics, Manipal Institute of Technology
Sanjay Singh  Department of Information and Communication Technology, Manipal
Institute of Technology

(June 2021)

##  1 Abstract

Planetary exploration requires traversal in environments with rugged terrains.
In addition, Mars rovers and other planetary exploration robots often carry
sensitive scientific experiments and components onboard, which must be
protected from mechanical harm. This paper deals with an active suspension
system focused on chassis stabilisation and an efficient traversal method
while encountering unavoidable obstacles. Soft Actor-Critic (SAC) was applied
along with Proportional Integral Derivative (PID) control to stabilise the
chassis and traverse large obstacles at low speeds. The model uses the rover’s
distance from surrounding obstacles, the height of the obstacle, and the
chassis’ orientation to actuate the control links of the suspension
accurately. Simulations carried out in the Gazebo environment are used to
validate the proposed active system.

Keywords: Deep Reinforcement Learning, Soft Actor-Critic, Active suspension,
Mars Exploration Rover (MER), Five bar mechanism, Space robotics, OpenAI Gym,
gym-gazebo

##  2 Introduction

Locomotion over rough terrain remains a formidable challenge for mobile
robots. Never much more so than in the sphere of planetary exploration. In
just the past few years, mobile robots are being employed at an ever-
increasing rate to explore the surfaces of numerous extraterrestrial bodies
[29][30]. The robot provides mobility to onboard scientific instruments, which
can be carried to specific objective sites, enabling scientists to gather
valuable information and conduct research. Hence, increased traversal
capabilities are essential for the robot to navigate the environment and
accomplish these tasks.

NASA’s rocker-bogie system is the most prominent of the several mobile systems
that have been proposed [12]. It was first used on the Mars rover Sojourner
and over time became the suspension of choice for Mars exploration vehicles,
given its superior stability and obstacle climbing ability. The rocker-bogie
suspension, which features a lateral differential mechanism, enables a six-
wheeled rover to passively keep all six of its wheels in contact with the
ground while traversing across severely uneven and rugged terrain [20]. This
feature has two crucial advantages. First, it equilibrates the pressure of the
wheels on the ground, and second, it helps to nominally keep the wheels in
contact with the ground while climbing over uneven obstacles [12].

Figure 1: The bogie overturn problem [2] describes a greater than 90
°revolution of the bogie about the bogie-rocker revolute joint, which leads to
a collapse of the mechanism and renders the rover immobile.

The bogie link is a centrally pivoted link with a wheel mounted on its two
ends. A problem commonly associated with the rocker-bogie suspension is bogie
overturn [2]. It is a term used to describe a greater than 90 °revolution of
the bogie about the bogie-rocker revolute joint, which essentially flips the
bogie link and renders the rover immobile. Such a situation cannot be
corrected remotely and requires manual intervention to bring the rover out of
its stranded state. The subjection of the suspension to extreme traversal
situations such as descending steep drops, colliding head-on with an obstacle
at high speed, and the middle wheel getting lodged on an obstacle, can result
in bogie overturn. By manipulating the design of the suspension, one can avoid
the occurrence of bogie overturn. Solutions include manipulating the geometry
of the linkage mechanism, usage of mechanical stops, and generally operating
at slower speeds. All these solutions significantly reduce the mobility of the
suspension and the speed of the rover. This work attempts to develop a
solution by collecting, identifying and presenting a set of desirable
characteristics for a mobile wheeled rover suspension. The objective for
determining these characteristics is to evaluate the performance of a system
and provide reviewers with a tool to compare the performance of a new system
impartially concerning the present state of the art.
Characteristics for a suspension system to be used for navigating and
exploring uneven terrain, especially in operations that are beyond the scope
of manual intervention, are identified as follows:

  * •

The suspension should be able to preserve the chassis’s lateral and
longitudinal stability at all times, even when negotiating large obstacles and
varied terrain [2]. The longitudinal stability is a direct measure of the
pitch (see Figure 3) of the chassis and, lateral stability is a direct measure
of the roll.

  * •

The suspension must possess increased traversability, which refers to a rovers
ability to traverse uneven and irregular terrain while ensuring ground contact
on as many wheels as possible [1]. This characteristic is of paramount
importance in the scope of the present work.

  * •

The suspension should be capable of changing its heading on command and
navigate through cluttered environments. This property is referred to as the
manoeuvrability of a mobile robot [1].

  * •

The suspension must have greater trafficability, which refers to the ability
of a rover to generate traction and overcome resistances [1].

  * •

The suspension must be able to scale obstacles several times its tire radius.
Usual consumer Automobiles can only climb obstacles with height less than or
equal to the radius of the tire [2].

  * •

The suspension must absorb and offload any impact loads the rover might
experience while navigating the terrain.

  * •

The suspension mechanism must not have positions in its motion that might
render the rover immobile, such as mechanical singularities or cases such as
bogie-overturn in the rocker-bogie mechanism.

This work outlines a proposal for a novel suspension system, considering the
challenges highlighted with the current state of the art and the performance
evaluation indices outlined above.

Figure 2: Kinematic diagram of the proposed planar five bar mechanism Figure
3: Pictorial representation of the chassis pitch. Pitch is defined as the
angle subtended between the longitudinal axis of the chassis and the ground
plane.

The new suspension system is meant to be used for a wheeled robot. It is based
on a modified five-bar mechanism [4] that has outer wheels mounted to
extrapolated sections of links 1 and 2, and the middle wheel mounted at the
revolute joint between links 1 and 2 as shown in Fig. 2. The wheel secured on
the extrapolated section of link 1 is henceforth referred to as the front
wheel. Each side of the suspension consists of two bogies, two vertical links
3 and 4 (henceforth referred to as control links), and three wheels. A
laterally mirrored setup on the other side of the chassis completes the six-
wheeled suspension setup.The suspension has all six wheels powered, making it
a six-wheel drive. The control links are mounted to the chassis, which forms
the fifth Link. The closed chain planar five-bar linkage mechanism forms the
basis of the suspension.
The proposed suspension system is not structurally stable in its present
configuration as the 5 bar mechanism cannot retain its structural integrity
under loading. Thus the use of a mechanism to constrain the five-bar mechanism
to the chassis becomes necessary. One can achieve the same using either
passive or active methods.

In passive systems, the mechanical configuration of the linkages itself is
responsible for the interaction of the wheels with the terrain. No complex
control systems dictating the position of the links or the reaction of the
wheels with the ground is required. A passive suspension system typically
comprises of a combination of shock absorbers and springs. It is characterised
by an inability to actively alter suspension parameters (such as suspension
stiffness, damping coefficient, suspension travel, geometry and more.) in
response to different road/terrain conditions [27]. A practical method to
constrain the five-bar mechanism to the chassis is to restrict the motion of
the turning pairs between (see Fig. 2) links 3 and 5 and links 4 and 5 using
torsion springs. Torsion springs’ rate is defined in N-mm per deg, i.e. it
provides increasing torque in response to increasing angular displacement.
This setup provides a simple, cost-effective, and implementable method of
constraining the 5 bar mechanism. In addition to this, it confers shock
absorption properties onto the suspension.

|
---|---

(a)

(b)

Figure 4: Prototype rover comprising of a torsion spring constrained passive
five bar suspension system.

Compared to passive systems, active systems present significantly higher
levels of complexity concerning the adaption of the wheels with the ground.
However, unlike purely passive suspension systems, an active system provides a
high degree of control over the reconfiguration capabilities of the suspension
geometry. Thus, it overall confers onto the rover, more remarkable
traversability. Furthermore, an active system makes use of actuators to change
and modify the kinematics of the suspension. Not only can it achieve wheel
adaption to the ground but also actively reconfigure the geometry to aid in
traversability by, for example, increasing traction on the ground . An active
control solution for the same entails controlling the rotation of the turning
pairs between (refer to Figure 2) links 3 & 5, and links 4 & 5 using rotary
actuators.

Considering the different types of obstacles that appear in various unknown
terrains, the control method for the suspension must be adaptable and able to
make decisions without having seen its exact surroundings at any last moment.
It cannot be assumed that the rover has access to all the required details of
its surroundings, making it essential that the control method is proactive in
deciding to traverse an obstacle rather than simply reactive to its current
state. Keeping in mind the proactivity of the state, we chose deep
reinforcement learning as the method of controlling the suspension. In the
field of robotics, Deep Reinforcement Learning is currently seen as one of the
most promising avenues of research for an end to end automation of robots.
[21].

Reinforcement Learning is a form of machine learning different from supervised
and unsupervised learning, where an intelligent agent learns to take a series
of actions by maximising a cumulative reward in order to extrapolate or
generalise to situations not present in the training set [28]. While there are
many approaches to solving reinforcement learning problems, this study mainly
focuses on using policy gradient methods that perform better in continuous
action spaces. In particular, an off-policy algorithm Soft Actor-Critic (SAC),
introduced in [10] and improved in [11] is used to predict the actions the
rovers control links must take in order to traverse obstacles blocking their
path by predicting the required angle. The control links are then actuated via
PID control to climb the obstacle. The trained model is then used to generate
data that is used as a benchmark to compare the efficiency of the proposed
suspension type.

Summary of the main contributions of the paper are as follows:

  * •

We developed and deployed a novel 5 bar suspension mechanism for our rover
suspension that eliminates bogie overturn typically associated with the
traditional rocker-bogie mechanism and its variants. Furthermore, the
developed model outperforms the rocker-bogie model and provides greater
stability during traversal.

  * •

We compared the active and passive variants of the proposed suspension model.
We demonstrated that the active model outperforms the passive model on the
chosen metrics that evaluate the suspension’s effectiveness.

  * •

Instead of deploying traditional supervised methods, we demonstrated the use
of deep Reinforcement Learning to train our agent. The latter addresses the
inherent problems of the former, like abstracting huge amounts of labelled
data before training and its inability to generalise to continuous action
spaces.

##  3 Related Works

Brian H Wilcox et al. in [31] detail the design of a Lunar Utility Vehicle
called ATHLETE developed by NASA, Stanford University and Boeing Company.
Several other works [13], [6] and [5] showcase different variants of the
Sherpa rover suspension system. These designs (ATHLETE and Sherpa) utilise a
highly actuated hybrid legged-wheeled suspension system that possesses higher
mobility associated with active legged robots and wheeled robots’ simplicity
and energy efficiency. The suspension consists of drive wheels mounted at the
extremities of an articulating leg with actuated joints. The system provides
much greater modularity and redundancies than passive wheeled counterparts,
including actively elevating wheels from the ground and actively altering the
vehicle footprint and support polygon, which provides greater stability on
slopes and enables the robot to fit into compact stow volumes. These proposed
designs aim to address drawbacks typically associated with currently deployed
wheeled rovers (e.g. NASA-JPL MER missions) and make a strong case for the use
of actuated suspension systems. Mohamed Krid and Faiz Benamar [18] explore the
development of a model predictive controller with a fast anti-roll suspension
design. The suspension is suitable for rovers meant to travel at high speeds.
However, the controller focuses only on minimising load transfer and energy
consumption and has no provisions for easily climbing steep obstacles.

Recently, machine learning has been widely investigated in control problems
and applied to vehicle suspension. For example, Ikbal Eski and Sahin Yildirim
in [7] compare two different control structures to control vehicle vibrations,
displacements instigated by an active suspension model. They have developed a
robust neural network and have compared its results to a fine-tuned PID
controller. Simulations show the effectiveness of the proposed neural network
model over the standard PID control system. The neural network control method
designed in [26], [7]makes full use of the fact that neural networks are
suitable for the nonlinear system, control and achieve good results. However,
as a supervised learning method, it requires the system to provide many
samples with labels. Moreover, in generating the control strategy, only the
current state is considered, but not the future state, which severely limits
the method’s utility. The study by Mubin Khan [15] focuses on devising a
Reinforcement Learning algorithm applied to an active suspension control
system. The algorithm aims to vary the model’s damper coefficient to control
the articulation of the axle shaft. Q learning is chosen to be the most
appropriate algorithm for this study, thus making the model inefficient in
extending to continuous action spaces. Furthermore, the environment is
formulated as a fully observable Markov Decision Process that ignores the
state uncertainty under the pretext of sensor readings being deemed 100 %
accurate and does not account for the stochastic nature of the agent’s
environment.

Ahmad Fares and Ahmad Bani Younes [8] have used online Reinforcement Learning
with the Temporal Difference (TD) advantage actor-critic to train an active
suspension system controller. This model is compared to a controller proposed
in [17] where a neural network was trained by the optimal PID and surpassed it
under parameter uncertainties. The results showed that the trained
Reinforcement Learning model obtains more optimal results under parameter
uncertainty as compared to the supervised learning method proposed in [17].
Furthermore, the results encourage further studies by testing state-of-the-art
continuous action Reinforcement Learning algorithms like the Deep
Deterministic Policy Gradient (DDPG) [19] and Asynchronous Actor-Critic (A3C)
[23]. Considering the continuity, stochasticity and complexity of the state
and action space, Liu Ming et al. [22] base their semi-active suspension
control system on the Deep Deterministic Policy Gradient (DDPG) algorithm
[19]. The algorithm aims towards adjusting the active control force of a
linear motor actuator, which further affects model parameters like body
acceleration, displacement and the dynamic deflection of the suspension. The
experimentation carried out in the time domain shows that the performance of
the proposed model is much better than that of its passive counterpart when
tested upon a randomly generated road profile. Mark N Howell et al. [14]
introduced a new Reinforcement Learning algorithm (CARLA) which focuses on
generalising to continuous action spaces. The control objective of the
algorithm is to minimise the mean squared acceleration of the vehicle body.
The results indicate the proposed model to fair better than conventional
passive models.

Passive suspension models have an inherent disadvantage of a delayed response
since motion is induced due to forces that act on the suspension during
traversal. These responses are unpredictable in nature and magnitude due to
the stochasticity of the environment, which introduces increased vibrations
and forces to the chassis and affects the stability by altering its
configuration, namely the pitch. Integrating an active suspension model
provides the robot with the ability to predict the response well in advance
whilst interacting with the environment. The agent aims to perceive the
environment variables and generate action space values, thus enabling the
robot to emulate the response well in advance. Most supervised learning
methods fail to capture the inherent stochasticity in the environment due to a
lack of labelled data. Our work adopts a model-free Reinforcement Learning
algorithm, Soft Actor-Critic (SAC), which models the environment accurately
and generalises to perturbations in the environment without the necessity of
collecting labelled data points. It also outperforms and converges faster than
other model-free algorithms due to the use of a maximum entropy objective to
generalise training to noises and perturbations in the environment thoroughly.

##  4 Simulation and Setup

The environment for training the active suspension of the rover is simulated
in Gazebo [16]. In addition, this environment has been incorporated with ROS
to ensure seamless communication and integration with all the robot’s onboard
sensors and controllers.

###  4.1 Unified Robot Description Format (URDF) Creation

The rover’s Solidworks robot model is exported into a universal robot
description format file, or URDF, using the SW2URDF (Solidworks assembly to
URDF exporter) plugin, which is available to the Solidworks users. A tree
structure comprising parent link and child link connections is defined to
simulate the five bar closed-chain mechanism appropriately. In addition,
separate definitions are provided for joint types, joint positions, rotation
axes, and joint limitations. The exporter’s link properties configuration
window is then used to configure the links’ inertial properties to replicate
real-world scenarios.

###  4.2 Simulation Description Format (SDF) Creation

The main downside of using a URDF file format is that it prevents the user
from creating a closed-chain mechanism. So, the links forming the closed chain
are detached in the URDF and reassembled in the SDF file. We create the SDF by
launching the robot’s URDF in Gazebo, then editing and saving using the
inbuilt model editor. The generated model is then saved and stitched by hand,
which closed the chain.

|
---|---

(a) Before completing the closed chain (URDF)

(b) After completing the closed chain (SDF)

Figure 5: The rover simulated in Gazebo

###  4.3 Controllers and Plugins Added

The multi-wheel hector differential drive plugin
(http://wiki.ros.org/hector_gazebo_plugins) is used to teleoperate the robot.
In addition, we have incorporated the Inertial Measurement Unit (IMU) plugin
to record the Euler angle variations emerging from the robot. A PID controller
that has been manually tuned is attached to all the four motors of the control
links.

###  4.4 Training Environment

gym-gazebo [32] a wrapper for the OpenAI Gym [3] for Gazebo environments is
used to build our training environment in the gym format. Our training
environment consists of a randomly generated obstacle with a minimum height of
25 cm to a maximum height of 32 cm present in the robot’s traversal path. The
obstacle faces the rover with an edge perpendicular to the ground, as this
scenario is the worst-case scenario for the rover to traverse. The obstacle is
sufficient to be considered unavoidable, meaning the rover is forced to
traverse it to reach its goal.

###  4.5 Problem Description

The environment is formulated as a Markov Decision Process (MDP) to apply deep
reinforcement learning. The notation used is presented as is from the Soft-
Actor Critic paper [10]. The MDP can be represented in tuple form as
$\mathcal{(S,A,P,R)}$, where the state space $\mathcal{S}$ and action space
$\mathcal{A}$ are continuous. The unknown state transition probability
$\mathcal{P}:\mathcal{S}\times\mathcal{S}\times\mathcal{A}$ $\rightarrow$
$[0,\infty)$ represents the probability density of next state
$s_{t+1}\in\mathcal{S}$ given the current state $s_{t}\in\mathcal{S}$ and
action $a_{t}\in\mathcal{A}$. The policy will be represented as
$\pi_{\phi}(a_{t}|s_{t})$ where the policy represents the probability of
taking action $a_{t}$ at state $s_{t}$. The environment emits a bounded reward
$\mathcal{R:S\times A}$ $\rightarrow$ $[r_{min},r_{max}]$ at every transition.

The environment’s observation space $\mathcal{O}$ is a vector
$\mathcal{O}\in\mathbb{R}^{4}$. The observations available to the rover are as
an array of form $[pitch,roll,distance,height]$ with an upper bound of 50 and
a lower bound of -50, which is obtained empirically. $Pitch$ is defined as the
angle subtended between the longitudinal axis of the chassis with the ground,
while $roll$ is defined as the angle subtended between the lateral axis of the
chassis with the ground. The rover can learn a decent representation of its
environment given these observations, which are part of the state space
$\mathcal{S}$. The $pitch$ and $roll$ of the rover at state $s_{t}$ are
obtained from an Inertial Measurement Unit. The $distance$ is the distance of
the rover from the face of the obstacle. $height$ is the obstacle height as
sourced from the environment. The action space $\mathcal{A}$ of the rover is
another 4 dimensional vector $\mathcal{A}\in\mathbb{R}^{4}$ in an array of
form $\mathit{[a_{0},a_{1},a_{2},a_{3}]}$ with an upper bound of 1 and lower
bound of -1. The angles $\mathit{a_{0},a_{1}}$ actuate the control links for
the movement of the middle and rear pair. The angles $\mathit{a_{2},a_{3}}$
are used to actuate the control links of the two wheels approaching the
obstacle. The angles are converted to the actual values in degrees by
multiplying them by 37 (the maximum angle allowed to actuate the control link)
before passing them to the ROS publishers for the joint position controllers.

Within the environment, when the rover reaches a certain threshold distance
from the obstacle, the agent is made to start taking its steps. At every
instance, the angles produced by the algorithm as part of its action space are
enforced by the use of a PID controller provided in ROS control. The PID
controller is tuned by hand to ensure no error in the angle achieved by the
motor and the angle predicted by the agent.

|  |
---|---|---
|  |

(a)

(b)

(c)

(d)

(e)

(f)

Figure 6: Trained model - SAC in action. The figure demonstrates how the
trained model enables the rover to climb over a randomly generated obstacle of
considerable size.

###  4.6 Soft Actor Critic (SAC)

We have used the Soft-Actor Critic (SAC) [11] algorithm for reinforcement
learning. The choice of this algorithm is made based on the findings of Tuomas
Haarnoja et al.[11] showing that SAC generalises well to real-world robotics
applications. The SAC algorithm aims to maximise a trade-off between two
goals, the maximisation of reward and entropy, which is a measurement of the
randomness of the policy. The use of the maximum entropy objective ensures the
policy explores more widely while dropping unpromising avenues. Tuomas
Haarnoja et al.[11] and John Schulman et al. [24] show that the use of this
objective makes the model converge faster compared to the conventional method.
Thus the optimal policy aims to maximize the reward as well as the entropy at
each visited state. This is given in Equation 1

$\pi^{*}=\arg\max_{\pi}\sum_{t}\operatorname{\mathbb{E}}_{({\mathbf{s}_{t}},{\mathbf{a}_{t}})\sim\rho_{\pi}}\left[\mathcal{R}({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\alpha\mathcal{H}(\pi({\,\cdot\,}|{\mathbf{s}_{t}}))\right]$
(1)

where $\alpha$ is the temperature term for entropy maximization that
determines the importance of entropy maximization relative to the reward.
While for a complete derivation of the algorithm, we would encourage the
readers to refer to the paper, in order to make the work self-contained, we
will provide a look at the basic equations involved.

Algorithm 1 Soft Actor-Critic

Input: $\theta_{1}$, $\theta_{2}$, ${\phi}$ $\triangleright$ Initial
parameters
Output: $\theta_{1}$, $\theta_{2}$, ${\phi}$ $\triangleright$ Optimized
parameters

$\bar{\theta}_{1}\leftarrow\theta_{1}$, $\bar{\theta}_{2}\leftarrow\theta_{2}$
$\triangleright$ Initialize target network weights

$\mathcal{D}\leftarrow\emptyset$ $\triangleright$ Initialize an empty replay
pool

for each iteration do

for each environment step do

${\mathbf{a}_{t}}\sim\pi_{\phi}({\mathbf{a}_{t}}|{\mathbf{s}_{t}})$
$\triangleright$ Sample action from the policy

${\mathbf{s}_{t+1}}\sim
p({\mathbf{s}_{t+1}}|{\mathbf{s}_{t}},{\mathbf{a}_{t}})$ $\triangleright$
Sample transition from the environment

$\mathcal{D}\leftarrow\mathcal{D}\cup\left\\{({\mathbf{s}_{t}},{\mathbf{a}_{t}},\mathcal{R}({\mathbf{s}_{t}},{\mathbf{a}_{t}}),{\mathbf{s}_{t+1}})\right\\}$
$\triangleright$ Store the transition in the replay pool

end for

for each gradient step do

$\theta_{i}\leftarrow\theta_{i}-\lambda_{Q}\hat{\nabla}_{\theta_{i}}J_{Q}(\theta_{i})$
for $i\in\\{1,2\\}$ $\triangleright$ Update the Q-function parameters

${\phi}\leftarrow{\phi}-\lambda_{\pi}\hat{\nabla}_{\phi}J_{\pi}({\phi})$$\triangleright$
Update policy weights

$\alpha\leftarrow\alpha-\lambda\hat{\nabla}_{\alpha}J(\alpha)$
$\triangleright$ Adjust temperature

$\bar{\theta}_{i}\leftarrow\tau\theta_{i}+(1-\tau)\bar{\theta}_{i}$ for
$i\in\\{1,2\\}$$\triangleright$ Update target network weights

end for

end for

SAC makes use of two soft Q functions. The soft Q function is calculated
iteratively by applying the modified Bellman operator $\mathcal{T}^{\pi}$ as

$\mathcal{T}^{\pi}\mathcal{Q}(s_{t},a_{t})\triangleq\mathcal{R}(s_{t},a_{t})+\gamma\mathbb{E}_{s_{t+1}\backsim
p}[V(s_{t+1})]$ (2)

where $V(s_{t})$ is the soft value function defined by

$V({\mathbf{s}_{t}})=\operatorname{\mathbb{E}}_{{\mathbf{a}_{t}}\sim\pi}\left[Q({\mathbf{s}_{t}},{\mathbf{a}_{t}})-\alpha\log\pi({\mathbf{a}_{t}}|{\mathbf{s}_{t}})\right]$
(3)

The policy has to be updated towards the exponential of the new soft Q
function. A tractable policy is chosen which can correspond to a parameterized
family of probability distributions. The information projection is defined in
terms of the Kullback-Leibler Divergence. The policy obtained by the policy
improvement step is updated by

$\pi_{\mathrm{new}}=\arg\underset{\pi^{\prime}\in\Pi}{\min}\mathrm{D_{KL}}\left(\pi^{\prime}(\centerdot|s_{t})\;\middle\|\;\frac{\exp\left(\frac{1}{\alpha}Q^{\pi_{\mathrm{old}}}(s_{t},\centerdot)\right)}{Z^{\pi_{\mathrm{old}}}(s_{t})}\right)$
(4)

where $\alpha$ is the temperature parameter for entropy maximization and $Z$
is a partition function which normalizes the distribuion but is ignored as it
doesn’t contribute to gradient calculations. Several Objective functions are
trained to optimise the parameters. The soft Q functions are parameters are
trained by minimising the soft Bellman residual

$J_{Q}(\theta)=\operatorname{\mathbb{E}}_{({\mathbf{s}_{t}},{\mathbf{a}_{t}})\sim\mathcal{D}}\left[\frac{1}{2}\left(Q_{\theta}({\mathbf{s}_{t}},{\mathbf{a}_{t}})-\left(\mathcal{R}({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\gamma\operatorname{\mathbb{E}}_{{\mathbf{s}_{t+1}}\sim
p}\left[V_{\bar{\theta}}({\mathbf{s}_{t+1}})\right]\right)\right)^{2}\right]$
(5)

This is optimized by the use of the stochastic gradient given in 6

$\hat{\nabla}_{\theta}J_{Q}(\theta)=\nabla_{\theta}Q_{\theta}({\mathbf{a}_{t}},{\mathbf{s}_{t}})\left(Q_{\theta}({\mathbf{s}_{t}},{\mathbf{a}_{t}})-\left(r({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\gamma\left(Q_{\bar{\theta}}({\mathbf{s}_{t+1}},{\mathbf{a}_{t+1}})-\alpha\log\left(\pi_{\phi}({\mathbf{a}_{t+1}}|{\mathbf{s}_{t+1}}\right)\right)\right)\right)$
(6)

The update uses the target Q function parameters $\bar{\theta}$ calculated as
the exponential moving average of the soft Q function weights.

The policy parameter $\phi$ can be learnt by minimizing equation 4. In order
to lower the variance estimate from the Q function, which is represented by a
neural network and is differentiable, the reparameterization trick can be
applied for the policy. Equation 7 shows the reparameterization,

${\mathbf{a}_{t}}=f_{\phi}(\epsilon_{t};{\mathbf{s}_{t}}),$ (7)

where $\epsilon_{t}$ is a noise vector sampled from a fixed distribution such
as a Gaussian. The final objective function for the parameter $\phi$ is given
by

$J_{\pi}({\phi})=\operatorname{\mathbb{E}}_{{\mathbf{s}_{t}}\sim\mathcal{D},\epsilon_{t}\sim\mathcal{N}}\left[\alpha\log\pi_{\phi}(f_{\phi}(\epsilon_{t};{\mathbf{s}_{t}})|{\mathbf{s}_{t}})-Q_{\theta}({\mathbf{s}_{t}},f_{\phi}(\epsilon_{t};{\mathbf{s}_{t}}))\right],$
(8)

where $\pi_{\phi}$ is reparameterized in terms of $f_{\phi}$. The stochastic
gradient can be approximated with 9

$\hat{\nabla}_{\phi}J_{\pi}({\phi})=\nabla_{\phi}\alpha\log\left(\pi_{\phi}({\mathbf{a}_{t}}|{\mathbf{s}_{t}})\right)+(\nabla_{\mathbf{a}_{t}}\alpha\log\left(\pi_{\phi}({\mathbf{a}_{t}}|{\mathbf{s}_{t}})\right)-\nabla_{\mathbf{a}_{t}}Q({\mathbf{s}_{t}},{\mathbf{a}_{t}}))\nabla_{\phi}f_{\phi}(\epsilon_{t};{\mathbf{s}_{t}})$
(9)

The final parameter to be tuned is the temperature for the maximization of
entropy. To get the optimal temperature $\alpha$ the dual equation in equation
10 must be solved.

$\alpha_{t}^{*}=\arg\min_{\alpha_{t}}\operatorname{\mathbb{E}}_{{\mathbf{a}_{t}}\sim\pi_{t}^{*}}\left[-\alpha_{t}\log\pi_{t}^{*}({\mathbf{a}_{t}}|{\mathbf{s}_{t}};\alpha_{t})-\alpha_{t}\bar{\mathcal{H}}\right]$
(10)

Solving the recursive equation for this dual problem would give us the optimal
temperature, but we use function approximators like neural networks and
gradient descent in practice. Dual Gradient Descent is used by [11] to solve
the objective in 11 to get the optimal temperature.

$J(\alpha)=\operatorname{\mathbb{E}}_{{\mathbf{a}_{t}}\sim\pi_{t}}\left[-\alpha\log\pi_{t}({\mathbf{a}_{t}}|{\mathbf{s}_{t}})-\alpha\bar{\mathcal{H}}\right].$
(11)

These objectives form the core of the SAC algorithm, and the parameters are
approximated via stochastic gradient descent. The minimum of the two soft Q
functions is used to calculate the stochastic gradient in 6 and the policy
gradients in 9 in order to minimise the positive biases in the policy
improvement step [11]. This is also known to speed up training on harder tasks
significantly. The final algorithm for SAC is given in algorithm 1.

For our Gazebo simulation, we have compared SAC to similar off-policy
algorithms such as Deep Delayed Deterministic Policy Gradients (DDPG) [19] and
Twin Delayed Deep Deterministic Policy Gradients (TD3) [9]. It has also been
compared to a popular on-policy learning algorithm, Proximal Policy
Optimization (PPO) [25].

###  4.7 Reward Function

The rewards obtained by the agent are incredibly crucial to the learning
process. Rewards indicate the quality of actions chosen by the agent in
reaching the next state. Positive rewards encourage the agent to accumulate as
much reward as possible, while negative rewards encourage the agent to reach
the desired state quickly. The reward for the rover at every step $s_{t}$ is
defined by algorithm 2.

Algorithm 2 The Reward function for every step

Input: $s_{t+1}$
Output: $r$

Episode End = $False$;

Reward $r$=0;

if $pitch$ > 20° then

$r$=-100;

Episode End = $True$;

end if

if $yaw$> 10° then

$r$=-100;

Episode End = $True$;

end if

if Obstacle is crossed then

$r$=100;

Episode End = $True$;

end if

if Obstacle is not crossed and time steps>430 then

$r$=-50;

Episode End = $True$;

end if

This simple reward formulation penalises the rover for veering off course
while attempting to climb or failing to cross the obstacle fast enough. If the
rover successfully crosses the obstacle, it gets a single positive reward, and
the next episode begins. In line with the rover’s objective to maintain the
chassis’ stability, a negative reward is provided if the rover is tilted
beyond a threshold and the episode ends. No intermediate rewards were required
in the training process to guiding the policy, and a single positive reward on
reaching the goal helps the model converge fastest.

###  4.8 Software and Hardware

The environment has been created from scratch by the authors. All the
algorithms make use of the Stable Baselines framework for their
implementation. PyTorch is used as the Reinforcement Learning framework, and
all the programs are implemented in Python. The programs were developed on
Ubuntu 20.04. Multiple training runs were conducted with each algorithm, all
resulting in convergence. All training runs were conducted on a Lenovo
ThinkPad L-470 with a quadcore i5-7200 CPU and 7.9 GB RAM. The codebase for
the environment and agent can be found in https://github.com/Mars-Rover-
Manipal/Active-Suspension.

$0$$0.5$$1$$\cdot 10^{5}$$-60$$-40$$-20$Timesteps Actor loss  (a) Actor loss

$0$$0.5$$1$$\cdot 10^{5}$$0$$0.2$$0.4$$0.6$$0.8$Timesteps Entropy coefficient
(b) Entropy coefficient

Figure 7: Training Plots of SAC obtained after being trained on the
GazeboMarsLsd-v0 environment

##  5 Results and Discussion

Figure 7 shows the convergence of the model as the actor loss and the entropy
coefficient $\alpha$ both decrease as the model converges to its optimal
policy. Furthermore, $\alpha$ starts to approach zero as the model finds the
optimal policy indicating a lesser need for exploration and more focus on
maximising the reward.

$0$$0.5$$1$$\cdot 10^{5}$$60$$80$$100$Timesteps Episode reward mean  (a) DDPG

$0$$0.5$$1$$\cdot 10^{6}$$20$$40$$60$$80$$100$Timesteps Episode reward mean
(b) TD3

$0$$0.5$$1$$\cdot 10^{6}$$20$$40$$60$$80$$100$Timesteps Episode reward mean
(c) PPO

$0$$0.5$$1$$\cdot 10^{5}$$0$$50$$100$Timesteps Episode reward mean  (d) SAC

Figure 8: Mean episode rewards plotted as a function of timesteps

The total average return of evaluation rollouts during training, i.e. the mean
reward obtained for progressive episodes and the mean episode length is
plotted progressively for 1M timesteps for PPO and 100k timesteps for DDPG,
SAC and TD3. SAC learns considerably faster than PPO due to the large batch
sizes PPO needs to learn stably on more high-dimensional and complex tasks
[11]. The plots in Figure 8 depicts that SAC converges to an optimal
stochastic policy in much fewer timesteps as compared to other tested on and
off policy baselines.

$0$$0.5$$1$$\cdot 10^{5}$$800$$850$$900$Timesteps Mean episode length  (a)
DDPG

$0$$0.5$$1$$\cdot 10^{5}$$900$$950$$1{,}000$Timesteps Mean episode length  (b)
TD3

$0$$0.5$$1$$\cdot 10^{6}$$500$$600$$700$$800$Timesteps Mean episode length
(c) PPO

$0$$0.5$$1$$\cdot 10^{5}$$100$$120$$140$Timesteps Mean episode length  (d) SAC

Figure 9: Mean episode lengths plotted as a function of timesteps. SAC shows
the least mean episode length amongst the other aforementioned algorithms.

Figure 9 depicts that SAC also exhibits the least mean episode length amongst
all trained algorithms and its successive convergence through learning as
opposed to TD3 and DDPG. Overall, this implies that SAC outperforms the other
baselines models for our provided task with a large margin. The results above
suggest that algorithms based on the maximum entropy principle can outperform
conventional Reinforcement Learning methods on challenging tasks [11].

As aforementioned, chassis pitch is the angle subtended between the
longitudinal axis of the chassis and the plane of the ground. The pitch of the
chassis must is minimised in order to maintain vehicle stability. Toppling
occurs when the projection of the centre of gravity of the vehicle on the
ground moves outside the vehicle’s support polygon [2]. High chassis pitch
increases the risks of toppling, especially on inclined surfaces. Frequent
fluctuation in pitch also causes scientific components and instruments to
experience inertial forces, which is highly undesirable. Transient responses
of both passive and active suspension system are determined in the time
domain. Time domain analysis is carried out for a step profile of a height of
32 cm. The suspension and the terrain primarily determine the nature of
variation of the pitch. From Fig 10, the peak amplitude of the pitch is
21.31°for the passive system, whereas, for the active system, it is 10.73°.
The peak amplitude is reduced by a magnitude of 10.58°, demonstrating the
effectiveness of the proposed active suspension in maintaining chassis
stability, which is the singular most important factor in suspension design.

$0$$1$$2$$3$$0$$10$$20$Time $\left[\mathrm{$\sec$}\right]$Pitch
$\left[\mathrm{$\mathrm{\SIUnitSymbolDegree}$}\right]$PassiveActive Figure 10:
The plot depicts the chassis pitch variation with respect to time for active
and passive suspension model whilst traversing over an obstacle.

The travel velocity is the velocity of the rover during its course of travel
over an obstacle. Figure 11 is obtained by plotting the travel velocity for
both the active and passive models for a 32 cm step obstacle. In a passive
suspension system, the motion of the suspension mechanism is induced purely in
response to terrain variations. Thus, while traversing the obstacle as each
wheel engages, the mechanism idles as reaction forces need to reach a
threshold value to cause the suspension mechanism to articulate. Hence the
passive system demonstrates lower velocity values while traversing the
obstacle than the active counterpart proposed in this paper, which exhibits no
time lag as it can preemptively actuate the mechanism in response to perceived
terrain undulations. Furthermore, the velocity plot (see Fig 11) corresponding
to the passive model indicates successive rise and fall in the velocity as
each wheel engages with the vertical face of the step. On the other hand, the
active model succeeds in maintaining an almost constant velocity profile while
encountering the step due to its ability to predict the motor angles well
before encountering it. The plot shows that the proposed active model fares
better than its passive counterpart in maintaining its constant travel
velocity of 0.7 m/s (chosen for the simulation empirically). The negative
portion of the graph in the velocity profile of the passive model corresponds
to the rebound the rover experiences when the wheels hit the obstacle. This
rebound being absent in the active model makes traversal smoother and faster.

$0$$100$$200$$300$$0$$0.2$$0.4$$0.6$$0.8$Timesteps Velocity
$\left[\mathrm{m/sec}\right]$PassiveActive Figure 11: Travel velocity of the
model plotted as a function of time.

##  6 Conclusion

The work proposes a novel five bar suspension architecture that eliminates the
bogie overturn problem, an inherent disadvantage present in the conventional
rocker-bogie suspension model. Furthermore, the adaptation of an active model
has proven advantageous compared to its passive counterpart as it minimises
the chassis pitch fluctuations greatly and succeeds in maintaining a nearly
constant velocity during its course of travel. We have presented an end to end
learning system for the automation of active suspensions for Mars rovers that
use a partially observed environment to control the actuation of the
suspension control links. Our experiments converged on a policy for traversing
unavoidable obstacles within 4 hours of training time without using a powerful
GPU. The algorithm, Soft Actor-Critic (SAC), based on an entropy constrained
Reinforcement Learning objective outperforms other model-free Reinforcement
Learning methods in our environment. However, we obtained the current results
through simulation studies. Whether the practical application has similar
effects still needs our further validation. Future works also include
integrating the current learning architecture with an efficient path planning
algorithm and multi-task learning of obstacle avoidance and traversal.

##  7 Acknowledgements

We would like to thank Mars Rover Manipal for providing us with the necessary
resources to complete the research.

## References

  * [1] Dimitrios S Apostolopoulos.  Analytical configuration of wheeled robotic locomotion.  The Robotics Institute of Carnegie Mellon University Technical Report CMU-RI-TR-01-08, 2001.
  * [2] Fırat Barlas.  Design of a mars rover suspension mechanism.  Master’s thesis, Izmir Institute of Technology, 2004.
  * [3] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym, 2016.
  * [4] Lucas Campos, Francis Bourbonnais, Ilian A Bonev, and Pascal Bigras.  Development of a five-bar parallel robot with large workspace.  In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, volume 44106, pages 917–922, 2010.
  * [5] Florian Cordes, Frank Kirchner, and Ajish Babu.  Design and field testing of a rover with an actively articulated suspension system in a mars analog terrain.  Journal of Field Robotics, 35(7):1149–1181, 2018.
  * [6] Florian Cordes, Christian Oekermann, Ajish Babu, Daniel Kuehn, Tobias Stark, Frank Kirchner, and DRIC Bremen.  An active suspension system for a planetary rover.  In Proceedings of the International Symposium on Artificial Intelligence, Robotics and Automation in Space (i-SAIRAS), pages 17–19, 2014\.
  * [7] Ikbal Eski and Şahin Yıldırım.  Vibration control of vehicle active suspension system using a new robust neural network control system.  Simulation Modelling Practice and Theory, 17(5):778–793, 2009.
  * [8] Ahmad Fares and Ahmad Bani Younes.  Online reinforcement learning-based control of an active suspension system using the actor critic approach.  Applied Sciences, 10(22):8060, 2020.
  * [9] Scott Fujimoto, Herke van Hoof, and David Meger.  Addressing function approximation error in actor-critic methods.  In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1587–1596. PMLR, 10–15 Jul 2018\.
  * [10] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861–1870. PMLR, 10–15 Jul 2018\.
  * [11] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine.  Soft actor-critic algorithms and applications, 2019.
  * [12] Brian D. Harrington and Chris Voorhees.  The challenges of designing the rocker-bogie suspension for the mars exploration rover.  In In 37th Aerospace Mechanisms Symposium, 2004.
  * [13] Javier Hidalgo and Florian Cordes.  Kinematics modeling of a hybrid wheeled-leg planetary rover.  In International Symposium on Artificial Intelligence, Robotics and Automation in Space, 2012.
  * [14] Mark N Howell, Geoff P Frost, Timothy J Gordon, and Qing H Wu.  Continuous action reinforcement learning applied to vehicle suspension control.  Mechatronics, 7(3):263–276, 1997.
  * [15] Mubeen Khan.  Reinforcement learning based suspension dampening control system for automotive applications.  Master’s thesis, Stanford University, 450 Serra Mall, Stanford, CA 94305, United States, 12 2018.  This study explores the application of Reinforcement Learning (RL) methods to a suspension dampening control system used to regulate the articulation of the control arm connecting the wheel assembly to the chassis of an ordinary road going automobile.
  * [16] N. Koenig and A. Howard.  Design and use paradigms for gazebo, an open-source multi-robot simulator.  In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566), volume 3, pages 2149–2154 vol.3, 2004.
  * [17] Aleksey Konoiko, Allan Kadhem, Islam Saiful, Navid Ghorbanian, Yahya Zweiri, and M Necip Sahinkaya.  Deep learning framework for controlling an active suspension system.  Journal of Vibration and Control, 25(17):2316–2329, 2019.
  * [18] Mohamed Krid and Faiz Benamar.  Design and control of an active anti-roll system for a fast rover.  In 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 274–279, 2011.
  * [19] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning, 2019.
  * [20] R.A. Lindemann and C.J. Voorhees.  Mars exploration rover mobility assembly design, test and performance.  In 2005 IEEE International Conference on Systems, Man and Cybernetics, volume 1, pages 450–455 Vol. 1, 2005.
  * [21] A. Rupam Mahmood, Dmytro Korenkevych, Gautham Vasan, William Ma, and James Bergstra.  Benchmarking reinforcement learning algorithms on real-world robots.  In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto, editors, Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pages 561–591. PMLR, 29–31 Oct 2018\.
  * [22] Liu Ming, Li Yibin, Rong Xuewen, Zhang Shuaishuai, and Yin Yanfang.  Semi-active suspension control based on deep reinforcement learning.  IEEE Access, 8:9978–9986, 2020.
  * [23] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In International conference on machine learning, pages 1928–1937. PMLR, 2016.
  * [24] John Schulman, Xi Chen, and Pieter Abbeel.  Equivalence between policy gradients and soft q-learning, 2018.
  * [25] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms, 2017.
  * [26] Zhu Si-hong, Lu Bao-zhan, Wang Hui, Zhang Ying, and He Liang.  Neural network control method of automotive semi-active air suspension [j].  Journal of Traffic and Transportation Engineering, 4, 2006.
  * [27] Weichao Sun, Huijun Gao, and Peng Shi.  Advanced Control for Vehicle Active Suspension Systems.  Springer International Publishing, 2020.
  * [28] Richard S. Sutton and Andrew G. Barto.  Reinforcement Learning: An Introduction.  A Bradford Book, Cambridge, MA, USA, 2018.
  * [29] Wikipedia contributors.  List of rovers on extraterrestrial bodies — Wikipedia, the free encyclopedia, 2021.  [Online; accessed 25-June-2021].
  * [30] Wikipedia contributors.  Mars rover — Wikipedia, the free encyclopedia, 2021.  [Online; accessed 25-June-2021].
  * [31] Brian H Wilcox, Todd Litwin, Jeff Biesiadecki, Jaret Matthews, Matt Heverly, Jack Morrison, Julie Townsend, Norman Ahmad, Allen Sirota, and Brian Cooper.  Athlete: A cargo handling and manipulation robot for the moon.  Journal of Field Robotics, 24(5):421–434, 2007.
  * [32] Iker Zamora, Nestor Gonzalez Lopez, Victor Mayoral Vilches, and Alejandro Hernandez Cordero.  Extending the openai gym for robotics: a toolkit for reinforcement learning using ros and gazebo.  arXiv preprint arXiv:1608.05742, 2016.