# Neural Lyapunov and Optimal Control

Daniel Layeghi, Steve Tonneau, Michael Mistry
The University of Edinburgh
<EMAIL_ADDRESS>

###### Abstract

Optimal control (OC) is an effective approach to controlling complex dynamical
systems. However, traditional approaches to parameterising and learning
controllers in optimal control have been ad-hoc, collecting data and fitting
it to neural networks. However, this can lead to learnt controllers ignoring
constraints like optimality and time variability. We introduce a unified
framework that simultaneously solves control problems while learning
corresponding Lyapunov or value functions. Our method formulates OC-like
mathematical programs based on the Hamilton-Jacobi-Bellman (HJB) equation. We
leverage the HJB optimality constraint and its relaxation to learn time-
varying value and Lyapunov functions, implicitly ensuring the inclusion of
constraints. We show the effectiveness of our approach on linear and nonlinear
control-affine problems. Additionally, we demonstrate significant reductions
in planning horizons (up to a factor of 25) when incorporating the learnt
functions into Model Predictive Controllers.

##  1 Introduction

Optimal control has been widley used to optimise the behaviour of complex
systems such as robots, aircrafts, powergrids and manufacturing plants.
Parametersing these policies or value functions allows us to learn a rich set
of control behaviours that can be adaptable to many scenarios. Traditional
approaches to learning within optimal control parameterise these ingredients
in an ad-hoc and offline manner. On the other hand deep reinforcement learning
(RL), which shares many of the same theoretical groundings, is able to learn
within a single framework. However, RL approaches typically do not leverage
problem structure such as dynamic models and their respective derivative
information[1]. Consider a scenario where we have solved an trajectory
optimisation task using off the shelf solvers. However we would like to learn
a neural network based policy or value function in order to generalise for
different initial conditions. However, in order to synthesise these
controllers efficiently it is necessary to process and optimise data in a
massively parallel form. This presents an opportunity for a new approach to
efficiently and effectively solve parameterised OC problems while leveraging
the strengths of the machinery of RL. As a result we need to leverage suitable
optimisation machinery. Deep learning has had an important impact on many
fields, and there is potential to leverage its massive data and parallel
optimisation frameworks within the context of OC. This could simplify and
unify the learning process and potentially lead to further improvements down
the line.

Previous approaches to parameterised optimal control firstly formulate an
optimisation problem over a time horizon. The problem is then solved using
theoretical grounds that define optimality with respect state and time. The
data generated is then used separately to fit to a neural network. This
process is time-consuming and as it is a supervised approach, ignores
important implicit constraints such as optimality conditions and the time
varying nature of the generated data when fitting neural networks. One of the
main reasons a unified approach to this problem had not been considered before
is that the concept of optimisation through time within neural networks, which
is fundamental to formulating optimal control problems, had not been studied
enough to find efficient methods. Additionally, methods such as
backpropagation through time are expensive both in terms of memory and
complexity and do not provide a close grounding to OC. Neural Ordinary
Differential Equations (ODEs) provide a new outlook to this problem. They
treat the hidden layers of a network as an ODE with its depth defined as an
integration horizon. In order to optimise this ODE it leverages a core concept
to OC, the maximum principle. As a result they provide similar theoretical
grounding.

We propose a unified approach that solves finite time parameterised optimal
control problems, leveraging neural ODEs and the deep learning machinery to
learn value and Lyapunov functions. This unification allows to implicitly
consider optimality constraints through problem formulation and temporal
constraints by including time as a parameter. We show that our method is able
to learn value and Lyapunov functions for linear and nonlinear control-affine
problems. In addition we show that the learnt functions when used in an model
predictive control context, enable us to solve finite-time optimal control
problems with using significantly shorter planning horizons and in our best
result reducing it by a factor of 25.

##  2 Related Work

Approaches to learning optimal control derived policy and value functions have
been explored in many different settings. Many seminal works exist within the
context of offline settings, where off-the-shelf solvers are used to solve OC
problems, and the generated data is then used for training. For example [2],
applied this formulation to character control. The authors used second order
nonlinear optimal control to generate state trajectories. A neural network was
then used to train a policy that can generate these trajectories. The results
show the network learns complex dynamic behaviour. However, the consequential
constraints that are implicit within the trajectory optimisation problem such
as inverse dynamics are not considered within the regression. Additionally,
this process is extremely computationally expensive and requires compute
clusters for solving multiple OC problems in parallel. There are other works
within the offline setting that focus on the value function space. [3]
formulate a supervised approach that uses neural networks to learn to fit
approximate cost-to-gos collected from trajectory optimisation. The authors
tackle the complex task of humanoid foothold selection. The aim of this
approach is inspired by [4] based the notion that informed terminal
constraints and value functions can reduce planning horizons. However, due to
the task complexity and the difficulty of the data generation process, the
authors only consider two initial conditions. Additionally as they solve a
supervised problem, implicit constraints such as Bellman back up are not
typically considered in the regression. Over all offline approaches require
extensive compute to collect relevant data in order to solve many optimal
control problems. Additionally, the training process is mainly supervised
which ignores the implicit dynamics of the generated data that is being
learnt.

Other online and more unified approaches have formulated methods that learn
Lyapunov and control barrier functions. These methods train neural networks to
inherently satisfy Lyapunov stability by minimising the penalty form of
Lyapunov constraints. [5] jointly optimises over Lyapunov and a policy
network. [6] modifies this problem and additionally optimises over a barrier
function as well. [7] uses a similar approach. Additionally it introduces a
verification procedure for certifying the learnt functions. However, these
methods are not applicable to finite time horizon problems such as trajectory
optimisation, as they solve for time invariant Lyapunov functions. As result
cannot account for time-varying Lyapunov functions which are necessary for
bounded time formulations such as trajectory optimisation. [8] extends this to
a finite horizon formulation however, similar to before it requires offline
data generation by nominal controllers.

Perhaps the closest line of work is done by [9] where the authors exploit the
structure of neural ODEs with application to policy learning given known
dynamics. In our work we show that neural ODEs can be naturally extended to
Lyapunov and value function synthesis while considering the time-varying
nature of the problem. Similar to [9], [10] explores the constrained version
of the policy based formulation. [11] considers the time parameterised version
of the policy. Additionally, it introduces further regularisation using
offline data. The above methods are formulated in policy space and cannot be
applied to value and Lyapunov functions synthesis. [12] work also considers
similar domain where authors learn the costate vector using maximum principle
formulation. This vector only considers the value function in its time
differential space making the learnt function limited to the trajectories on
which it was trained.

In this work we introduce a unified approach that solves finite time
parameterised optimal control problems while leveraging neural ODEs to learn
the corresponding time-varying Lyapunov and value functions. We test our
method on linear and nonlinear control-affine systems. Additionally, we show
that by utilising the learnt value and Lyapunov functions to warmstart
traditional MPC solvers we are able to significantly reduce the planning
horizon.

##  3 Preliminaries

Optimal Control:  optimal control is grounded within the Hamilton-Jacobi-
Bellman (HJB) equation.

$-v_{t}(\mathbf{x},t)=\ell(\mathbf{x},\mathbf{u},t)+\mathbf{f}(\mathbf{x},\mathbf{u})^{\top}v_{\mathbf{x}}(\mathbf{x},t)$
(1)

where $\dot{\mathbf{x}}=\mathbf{f}(\mathbf{x},\mathbf{u})$ represents the
deterministic dynamics given state $\mathbf{x}\in{\mathbb{R}}^{n}$ and control
$\mathbf{u}\in{\mathbb{R}}^{m}$, $\ell(\mathbf{x},\mathbf{u},t)$ represents
the state-control cost, $v_{t}(\mathbf{x},t)$ and $v_{x}(\mathbf{x},t)$
represent the Legendre notation for partial derivatives of the value function
with respect to time and state respectively. This is a partial differential
equation (PDE) defining the time evolution of the value function, in other
words the compact description of the relationship between optimal costs for
different states and times. In the case of the existence of a $C^{1}$ value
function, cost and dynamics, one can analytically compute a closed loop
feedback law for every initial condition $\mathbf{x}(0)$. The following shows
this optimal control for a control-affine dynamical system where
$\mathbf{\dot{x}}=\mathbf{f}(\mathbf{x})+\mathbf{g}(\mathbf{x})\mathbf{u}$
given a quadratic regularisation of control $\|\mathbf{u}\|_{\mathbf{R}}$.
Where $\mathbf{R}$ is a positive definite regularisation matrix.

$\displaystyle\operatorname*{min}_{\mathbf{u}\in\mathbf{U}}$
$\displaystyle\left[v_{t}(\mathbf{x},t)+\ell(\mathbf{x},\mathbf{u},t)+\mathbf{f}(\mathbf{x},\mathbf{u})^{\top}v_{\mathbf{x}}(\mathbf{x},t)\right]$
(2)
$\displaystyle\pi^{*}(\mathbf{x},t)=-\frac{1}{2}\mathbf{R}^{-1}\mathbf{g}(\mathbf{x})^{\top}v_{\mathbf{x}}(\mathbf{x},t)^{\top}$

Above policy gives us a time-varying optimal control $\mathbf{u^{*}}(t)$ for
any initial condition. This is an important property of the HJB equation which
however relies on the strong assumption of existence of a known value
function. On the other hand Portraying Maximum Principle (PMP) alleviates this
problem by operating in the tangential space. It converts this PDE into and
ODE by interpreting $v_{x}$ with a vector $\mathbf{p}$ known as a costate
vector. Referring to the right-hand side of equation 1 as Hamiltonian $H$, the
PMP aims to minimise $H$ over a horizon $T$ using an optimal control
trajectory $\mathbf{u^{*}}$. This trajectory is determined given an initial
condition $\mathbf{x}(0)$ and a terminal cost
$\ell_{\mathbf{x}}(\mathbf{x}_{T})=\mathbf{p}(T)$, subject to following
conditions:

$\displaystyle\mathbf{\dot{x}}$ $\displaystyle=H_{\mathbf{p}}|_{*},\hskip
20.00003pt\mathbf{\dot{p}^{*}}=-H_{\mathbf{x}}|_{*}$ (3)
$\displaystyle\mathbf{u^{*}}(t)$
$\displaystyle=\operatorname*{arg\,max}_{\mathbf{u}\in\mathbf{U}}H(\mathbf{x^{*}}(t),\mathbf{u},\mathbf{p^{*}}(t))$

Due to its tractability, this framework underpins the majority of trajectory
optimisation algorithms such as iterative LQR [13] or differential dynamic
programming (DDP) [14]. However, it only generates an open loop trajectory
valid for a single initial condition. This can be a significant limitation as
new initial conditions require reoptimisation. For further discussion around
these topics we refer the reader to [15].

Neural ODEs:  neural ODEs are both practically and theoretically fundamental
to our work. In this section we briefly describe neural ODEs from an OC
prospective. The general objective for neural ODEs is defined as:

$L\left(\mathbf{x}(T)\right)=L\left(\mathbf{x}(0)+\int_{t_{0}}^{T}\mathbf{f}\left(\mathbf{x}(t),\mathbf{u},t\right)dt\right)$
(4)

Neural ODEs differ from traditional optimal control methods in two ways: the
control parameter $\mathbf{u}$ is a time-invariant variable interpreted as
network weights, and the loss is evaluated at the end of the trajectory due to
the analogous nature of integration time step to hidden layers in a standard
neural network. As a result, the canonical equations of neural ODEs are a
special case of PMP, with the costate vector
$\mathbf{\dot{p}^{*}}=-H_{\mathbf{x}}|_{*}=-\mathbf{f}_{\mathbf{x}}^{\top}(\mathbf{x}(t),\mathbf{\theta},t)\mathbf{p}(t)$
ignoring the effects of running loss $L_{\mathbf{x}}(\mathbf{x}(t))$.
Therefore, although neural ODEs create a familiar grounding to OC they cannot
be immediately applied to problems where change of state along time matters.
[9] provides a reformulation of neural ODE’s gradient estimator for policy
learning. In the next segment we show how this can also be extended to the
value and Lyapunov space.

##  4 Learning Lyapunov and value functions

Our approach provides a straightforward and effective framework for solving
learning value and Lyapunov parameterised optimal control problems while
respecting implicit OC constraints.

Value functions: let us focus on the HJB equation 1 under the optimal policy
$\pi^{*}(\mathbf{x},t)$. By moving the inner product between $v_{\mathbf{x}}$
and the dynamics $\mathbf{f}(\mathbf{x},\mathbf{u})$ or
$\frac{d\mathbf{x}}{dt}$ to the left hand side we can rewrite HJB as a
definition for the total rate of change of the value function

$\displaystyle\frac{\partial v(\mathbf{x},t)}{\partial t}+\frac{\partial
v(\mathbf{x},t)}{\partial\mathbf{x}}\frac{d\mathbf{x}}{dt}$
$\displaystyle=-\ell(\mathbf{x},\pi^{*}(\mathbf{x},t),t)$ (5)
$\displaystyle\frac{dv(\mathbf{x},t)}{dt}$
$\displaystyle=-\ell(\mathbf{x},\pi^{*}(\mathbf{x},t),t)$

The above can be interpreted as a constraint on the rate of change of the
value function under the optimal policy. Integrating over a horizon T allows
us to evaluate the consistency of optimal policies with respect to the above
constraint. This leads to:

$\displaystyle\int_{0}^{T}\frac{dv(\mathbf{x},t)}{dt}$
$\displaystyle=-\int_{0}^{T}\ell(\mathbf{x},\pi^{*}(\mathbf{x},t),t)$ (6)
$\displaystyle v(\mathbf{x}(T),T)-v(\mathbf{x}(0),0)$
$\displaystyle=-\int_{0}^{T}\ell(\mathbf{x},\pi^{*}(\mathbf{x},t),t)$

Equation 5, provides us with an equality that can be converted into a soft
penalty to define a loss function in order to learn parameterised value
functions. We can formulate this as a mathematical program where we
approximate the value function with a neural network,
$\tilde{v}(\mathbf{x},t;\theta)$. In our case we evaluate this problem on
control-affine dynamics and consider the same closed loop feedback policy in
equation 2 under the approximate value function. In order to learn a complete
value function we consider multiple initial conditions $\mathbf{x}_{0}\in
X_{0}$ where $|X_{0}|=N$.

$\displaystyle\min_{\theta}\Bigg{[}\frac{1}{N}\sum_{k=0}^{N}\bigg{(}\tilde{v}(\mathbf{x}_{k}(T),T;\theta)-\tilde{v}(\mathbf{x}_{k}(0),0;\theta)+\int_{t=0}^{T}\|\mathbf{u}^{*}_{k}(t))\|_{\mathbf{R}}+\ell(\mathbf{x}_{k}(t))dt\bigg{)}^{2}\Bigg{]}$
(7) s.t:
$\displaystyle\qquad\mathbf{\dot{x}}(t)=\mathbf{f}(\mathbf{x}(t))+\mathbf{g}(\mathbf{x}(t))\mathbf{u}^{*}(t)$
$\displaystyle\qquad\mathbf{u}^{*}(\mathbf{x},t)=-\frac{1}{2}\mathbf{R}^{-1}\mathbf{g}(\mathbf{x})^{\top}v_{\mathbf{x}}(\mathbf{x},t;\theta)^{\top}$

The above mathematical program defines a finite time parameterised optimal
control problem that aims to learn a value function that approximately
minimises the Bellman backup condition in equation 5 and consequently solves
the task.

Lyapunov functions:  finite time Lyapunov analysis states that if we have a
continuously differentiable positive definite function $v(\mathbf{x},t)$,
where $v(\mathbf{x},t)>0$ for $\mathbf{x}\neq 0$ and $v(0,t)=0$ then
$\mathbf{f}(\mathbf{x},t)$ is stable if $\dot{v}(\mathbf{x},t)=\frac{\partial
v}{\partial\mathbf{x}}\mathbf{f}(\mathbf{x})+\frac{\partial v}{\partial t}<0,\
\dot{v}(0,t)=0$. This condition must hold for all $\mathbf{x}$ and all $t$.
However, finding such Lypunov functions is not trivial and is unknown apriori.
Equation 5 and the conditions above impose constraints on the rate of change
of function $v$ in both the contexts of value and Lyapunov functions. However,
the Lyapunov condition is a relaxed version of the HJB equality constraint.
Meeting this lower bound results in transforming the Lyapunov function into a
value function. We aim to learn an approximate Lyapunov function that allows
for generating controls that can sub optimally complete a task. To formulate
this we relax the HJB condition and define the Lyapunov constraint with
respect to a task loss $\ell(\mathbf{x})$.

$\displaystyle\frac{\partial v(\mathbf{x},t)}{\partial t}+\frac{\partial
v(\mathbf{x},t)}{\partial\mathbf{x}}\mathbf{f}(\mathbf{x},\mathbf{u})$
$\displaystyle\leq-\ell(\mathbf{x},t)$ (8)
$\displaystyle\frac{dv(\mathbf{x},t)}{dt}$
$\displaystyle\leq-\ell(\mathbf{x},t)$

Performing the same integration over T as in equation 6 we can define a soft
inequality as a loss function in order to learn parameterised Lyapunov
functions. We formulate this using the following mathematical program:

$\displaystyle\min_{\theta}\Bigg{[}\frac{1}{N}\sum_{k=0}^{N}\max\bigg{(}\tilde{v}(\mathbf{x}_{k}(T),T;\theta)-\tilde{v}(\mathbf{x}_{k}(0),0;\theta)+\int_{t=0}^{T}\ell(\mathbf{x}_{k}(t))d{t},0\bigg{)}\Bigg{]}$
(9) s.t:
$\displaystyle\qquad\mathbf{\dot{x}}(t)=\mathbf{f}(\mathbf{x}(t))+\mathbf{g}(\mathbf{x}(t))\mathbf{u}^{*}(t)$
$\displaystyle\qquad\mathbf{u}^{*}(\mathbf{x},t)=-\frac{1}{2}\mathbf{R}^{-1}\mathbf{g}(\mathbf{x})^{\top}v_{\mathbf{x}}(\mathbf{x},t;\theta)^{\top}$

We make use of the same HJB feedback policy used in value learning. This is
because, as we show in the Appendix the policy is a valid control Lyapunov
function (CLF) for a given Lyapunov function. We show this in the Theorem 1 of
the supplementary material. However, other choices for a CLF such as Sontag’s
universal controller [16] are also valid. To parameterise the Lyapunov
function while respecting Lyapunov constrains we make use of input convex
neural networks (ICNNs) [17] with an additional positive constant term to
approximate the positive definite Lyapunov function:
$v(\mathbf{x},t;\theta)=v_{\text{ICNN}}(\mathbf{x},t;\theta)+\epsilon\|\mathbf{x}\|_{2}^{2}$.

Neural ODEs:  the mathematical programs LABEL:eq:_value_prog and
LABEL:eq:_lyap_prog have a bounded time horizon. While neural ODEs are
suitable for finite horizon optimisation, they cannot be directly applied to
these programs as the task loss needs to be evaluated over the entire horizon
T. To alleviate this we follow the same modifications shown in [9]. These
modification aim convert the canonical equations of neural ODEs to the PMP
case.

$\displaystyle\dot{\mathbf{a}}$
$\displaystyle=-\mathbf{a}(t)^{\top}\mathbf{f}_{\mathbf{x}}(\mathbf{x},\mathbf{u}(\mathbf{x},t,\theta))+\ell_{\mathbf{x}}(\mathbf{x},\mathbf{u}(\mathbf{x},t;\theta))$
(10) $\displaystyle\dot{\ell}_{\theta}$
$\displaystyle=-\mathbf{a}(t)^{\top}\mathbf{f}_{\mathbf{u}}(\mathbf{x},\mathbf{u}(\mathbf{x},t,\theta))\mathbf{u}_{\theta}(\mathbf{x},t,\theta)+\ell_{\mathbf{u}}(\mathbf{x}(t),\mathbf{u}(\mathbf{x},t;\theta))\mathbf{u}_{\theta}(\mathbf{x},t,\theta)$

These canonical equations differ from vanilla neural ODEs mentioned in [18].
The resulting gradient estimator is less efficient in complexity compared to
its vanilla form. However, it is a necessary addition and our results show
that computation time is still feasable in table 1.

We have shown how parameterised optimal control problems can be formulated
using the mathematical programs LABEL:eq:_lyap_prog and LABEL:eq:_value_prog.
In the next segment we will demonstrate how these problems can be solved using
the gradient estimator 10. We obtain these results on systems with control-
affine dynamics.

##  5 Empirical results

Our approach successfully learns approximate time-varying Lyapunov and value
functions to classical control problems, including the double integrator,
cartpole stabilisation, cartpole swing up, and the planar two link
environment. We use the Mujoco simulator for rendering[19]. We demonstrate the
effectiveness of our method by learning time-varying Lyapunov functions for
linear or locally linear dynamics, as seen in the double integrator and
cartpole balancing tasks. Furthermore, we showcase our ability to learn time-
varying value functions for more complex nonlinear problems, such as the
cartpole swing up and planar two link environments. Finally, by incorporating
these learnt Lyapunov and value functions to warmstart Model Predictive
Control (MPC) solver MPPI [20], we show that significant reduction in planning
horizon can be obtained when compared to vanilla MPPI.

Double Integrator (DI): The objective of DI is to drive the state
$\mathbf{x}\in{\mathbb{R}}^{2}$ to the origin in finite time T, using a
unconstrained control input $\mathbf{u}\in{\mathbb{R}}^{1}$.

Cartpole (CP) Stabilisation: The goal of CP stabilisation is to bring a four-
dimensional state $\mathbf{x}\in{\mathbb{R}}^{4}$ to the unstable equilibrium
point $\mathbf{x}=[0,0,0,0]$ in finite time T. The control input
$\mathbf{u}\in{\mathbb{R}}^{2}$ is a one-dimensional force applied to the
cart. To avoid wrap-around effects the revolute joint is encoded using
$\hat{q}_{\text{pole}}=\cos(q_{\text{pole}})-1$

Cartpole Swing Up: The objective is to swing the CP from downward position to
a vertical state $\mathbf{x}=[0,0,0,0]$ within a finite time T. The state
$\mathbf{x}\in{\mathbb{R}}^{4}$ and control input
$\mathbf{u}\in{\mathbb{R}}^{1}$. We have used the same model mentioned in [21]

Planar Two link (TL): In this control problem the state
$\mathbf{x}\in{\mathbb{R}}^{4}$ representing the position and velocity of a
two link (TL) arm. The aim is to move the arm to reach the target state
$\mathbf{x}=[0,0,0,0]$ within a finite time T. The control input
$\mathbf{u}\in{\mathbb{R}}^{2}$ consists of torques applied to each joint. We
obtained the model from [22].

For all tasks we used a our implementation of neural ODEs gradient estimator
defined in equation 10. The results were obtained on a core i9 Intel processor
with Nvidia GeForce RTX 4070 GPU.

Table 1: Training Statistics Task | Runtime (m) | Epochs | Normalised Cost | Horizon (ms)
---|---|---|---|---
Double Integrator | 12.51 | 65 | 0.29 | 700
Cartpole Balance | 1.48 | 30 | 0.90 | 100
Two link arm | 2.35 | 50 | 0.54 | 300
Cartpole Swingup | 8.49 | 150 | 0.82 | 300

###  5.1 Lyapunov Function Results

Figure 1: a) Normalised average trajectory cost per epoch for CP. b) Levels
sets of the DI Lyapunov function at T=0. c) Normalised average trajectory cost
per epoch for DI.

First we look at the ability of our method to learn approximate Lyapunov
functions for linear tasks. In this case, we use the double integrator and the
mathematical program shown in LABEL:eq:_lyap_prog. The cost is defined by
$\mathbf{Q}=\text{diag}(1,0)$ and $\mathbf{R}=0.1$. For the double integrator
the Lyapunov function is parameterised by
$\tilde{V}_{di}:{\mathbb{R}}^{2+1}\rightarrow{\mathbb{R}}^{1}$: (3-4-4-1
ICNN). In figure 1 c we show the convergence of this Lyapunov function across
5 different seeds with 20 initial conditions and over a horizon of 7 seconds.
All networks converge within 60 epochs. Figure 1b shows the level-sets of the
converged Lyapunov function along with controlled trajectories.

We also test this framework on locally linear tasks, in this case cartpole
balancing. The tasks cost defined by $\mathbf{Q}=\text{diag}(0.5,1,0.01,0.01)$
and $\mathbf{R}=0.1$. The Lyapunov function
$\tilde{V}_{cp}:{\mathbb{R}}^{4+1}\rightarrow{\mathbb{R}}^{1}$: (5-200-500-1
ICNN) is much larger than the linear task. 1a shows the average trajectory
cost per epoch across 5 seeds and 100 initial condition with $T=1s$. Learning
for all networks converges at 15 iterations.

For Lyapunov learning we use Euler integration scheme with time steps 0.01s
and 0.008s respectively. We use a smaller integration time step for the
balancing problem due to the unstable nature of the system. The networks are
optimised using the Adam optimiser [23].

###  5.2 Value Function Results

We also apply our method to nonlinear tasks. In this setting we utilise the
mathematical program LABEL:eq:_value_prog to learn time varying value
functions for cartpole swing and two link arm control. The cost function for
the swing up tasks is defined by $\mathbf{Q}=\text{diag}(1,1,0.1,0.1)$ and
$\mathbf{R}=0.1$. The value function
$\tilde{V}_{cp}:{\mathbb{R}}^{4+1}\rightarrow{\mathbb{R}}^{1}$: (5-128-128-1
FCN). Figure 2a shows the convergence of average task cost across 100 initial
conditions, 5 seeds and $T=3s$. Notably the network learns swing up value over
140 epochs.

We also apply the same framework to a two link arm problem where
$\mathbf{Q}=\text{diag}(1,1,0.1,0.1)$ and $\mathbf{R}=\text{diag}(0.15,0.15)$
and $\tilde{V}_{tl}:{\mathbb{R}}^{4+1}\rightarrow{\mathbb{R}}^{1}$:
(5-128-128-1 FCN). From figure 2b we can see that the task cost is minimised
at approximately 10 epochs. This result are consistent across 5 seeds and 100
initial conditions over $T=3s$

Figure 2: a) Normalised average trajectory cost per epoch for CP swing up. b)
Normalised average trajectory cost per epoch for TL.

Both models are optimised using the Adam optimiser. For swing up we use 0.08s
time step and 0.01 for two link arm.

###  5.3 MPC with value and Lyapunov functions

Finally we compare the effect of incorporating value functions into a model
predictive controller, in this case MPPI. To do this we warmstart every MPC
iteration with the one step Bellman policy shown in equation 2. We compare the
results on the swing up, balancing and the two link arm tasks. Our results
show that we are able to reduce the planning horizon in the maximum case
(swing up and two link) by a factor of 25. Additionally our results show that
even with shorter time horizons the informed policy consistently performs
better than vanilla MPC. The results below are obtained across 20 initial
conditions using the same cost functions described above.

Table 2: Comparison of MPPI and MPPI+value function in different environments Environment | MPPI | MPPI+value function
---|---|---
Avg. Iter. | Horizon (ms) | Avg. Cost | Avg. Iter. | Horizon (ms) | Avg. Cost
CartPole Swing Up | 254 | 200 | 3218.35 | 223 | 8 | 2051.40
CartPole Balance | 46 | 100 | 2743.20 | 71 | 8 | 2181.15
Two-Link Arm | 336 | 200 | 856.23 | 321 | 8 | 786.3

##  6 Conclusion

In this paper we proposed a method that simultaneously learns Lyapunov and
value functions and utilises them to solve optimal control problems. This
method greatly simplifies the process of synthesising these functions. Our
formulation inherently considers important constraints such as optimality
criterion and time varying conditions. We test our method on both linear and
nonlinear control-affine systems. Our results show that our method is able to
successfully learn these value and Lyapunov functions. Additionally, we use
the learnt functions to warmstart an MPC solver, leading to significant
reduction in planning horizons. We envision that this method can allow optimal
control practitioners to tackle value and Lyapunov function learning in a more
unified framework.

## References

  * [1] B. Recht, “A tour of reinforcement learning: The view from continuous control,” Annual Review of Control, Robotics, and Autonomous Systems, vol. 2, pp. 253–279, 2019.
  * [2] I. Mordatch, K. Lowrey, G. Andrew, Z. Popovic, and E. V. Todorov, “Interactive control of diverse complex characters with neural networks,” Advances in neural information processing systems, vol. 28, 2015.
  * [3] J. Wang, T. S. Lembono, S. Kim, S. Calinon, S. Vijayakumar, and S. Tonneau, “Learning to guide online multi-contact receding horizon planning,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 12942–12949, 2022.
  * [4] H. Li, R. J. Frei, and P. M. Wensing, “Model hierarchy predictive control of robotic systems,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 3373–3380, 2021.
  * [5] C. Dawson, Z. Qin, S. Gao, and C. Fan, “Safe nonlinear control using robust neural lyapunov-barrier functions,” in Conference on Robot Learning, pp. 1724–1735, PMLR, 2022.
  * [6] W. Xiao, T.-H. Wang, R. Hasani, M. Chahine, A. Amini, X. Li, and D. Rus, “Barriernet: Differentiable control barrier functions for learning of safe robot control,” IEEE Transactions on Robotics, pp. 1–19, 2023.
  * [7] Y.-C. Chang, N. Roohi, and S. Gao, “Neural lyapunov control,” Advances in neural information processing systems, vol. 32, 2019.
  * [8] W. Xiao, R. Hasani, X. Li, and D. Rus, “BarrierNet: A Safety-Guaranteed Layer for Neural Networks,” arXiv e-prints, p. arXiv:2111.11277, Nov. 2021.
  * [9] S. Ainsworth, K. Lowrey, J. Thickstun, Z. Harchaoui, and S. Srinivasa, “Faster policy learning with continuous-time gradients,” in Learning for Dynamics and Control, pp. 1054–1067, PMLR, 2021.
  * [10] I. O. Sandoval, P. Petsagkourakis, and E. A. del Rio-Chanona, “Neural odes as feedback policies for nonlinear optimal control,” arXiv preprint arXiv:2210.11245, 2022.
  * [11] X. Zhang, J. Long, W. Hu, W. E, and J. Han, “Initial value problem enhanced sampling for closed-loop optimal control design with deep neural networks,” 2023\.
  * [12] Y. D. Zhong, B. Dey, and A. Chakraborty, “Symplectic ode-net: Learning hamiltonian dynamics with control,” in International Conference on Learning Representations, 2020.
  * [13] Y. Tassa, T. Erez, and E. Todorov, “Synthesis and stabilization of complex behaviors through online trajectory optimization,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4906–4913, IEEE, 2012.
  * [14] D. H. Jacobson, “New second-order and first-order algorithms for determining optimal control: A differential dynamic programming approach,” Journal of Optimization Theory and Applications, vol. 2, pp. 411–440, 1968.
  * [15] D. Liberzon, Calculus of variations and optimal control theory: a concise introduction.  Princeton university press, 2011.
  * [16] Y. Lin and E. D. Sontag, “A universal formula for stabilization with bounded controls,” Systems & control letters, vol. 16, no. 6, pp. 393–397, 1991\.
  * [17] B. Amos, L. Xu, and J. Z. Kolter, “Input convex neural networks,” in Proceedings of the 34th International Conference on Machine Learning, vol. 70 of Proceedings of Machine Learning Research, pp. 146–155, PMLR, 2017.
  * [18] R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud, “Neural ordinary differential equations,” Advances in neural information processing systems, vol. 31, 2018.
  * [19] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026–5033, IEEE, 2012.
  * [20] G. Williams, P. Drews, B. Goldfain, J. M. Rehg, and E. A. Theodorou, “Information-theoretic model predictive control: Theory and applications to autonomous driving,” IEEE Transactions on Robotics, vol. 34, no. 6, pp. 1603–1622, 2018.
  * [21] R. Tedrake, Underactuated Robotics.  2023\.
  * [22] E. Todorov and W. Li, “Optimal control methods suitable for biomechanical systems,” in Proceedings of the 25th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE Cat. No.03CH37439), vol. 2, pp. 1758–1761 Vol.2, 2003.
  * [23] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.