# A Gradient Descent-Ascent Method for Continuous-Time Risk-Averse Optimal
Control

Gabriel Velho  Jean Auriol  Riccardo Bonalli

###### Abstract

In this paper, we consider continuous-time stochastic optimal control problems
where the cost is evaluated through a coherent risk measure. We provide an
explicit gradient descent-ascent algorithm which applies to problems subject
to non-linear stochastic differential equations. More specifically, we
leverage duality properties of coherent risk measures to relax the problem via
a smooth min-max reformulation which induces artificial strong concavity in
the max subproblem. We then formulate necessary conditions of optimality for
this relaxed problem which we leverage to prove convergence of the gradient
descent-ascent algorithm to candidate solutions of the original problem.
Finally, we showcase the efficiency of our algorithm through numerical
simulations involving trajectory tracking problems and highlight the benefit
of favoring risk measures over classical expectation.

###### keywords:

Gradient descent-ascent , Coherent risk measures , Stochastic optimal control
, Non-linear control , Risk-averse control

††journal: Systems and Control Letters

[Université Paris-Saclay]organization=Université Paris-Saclay,
CentraleSupélec, CNRS, Laboratoire des signaux et systèmes,city=Gif-sur-
Yvette, country=France

##  1 Introduction

Risk-averse stochastic optimal control is a powerful tool for designing
control laws that feature robustness against uncertainties. Relevant
applications of this theory range from safe financial investment to safe
control of autonomous systems, as evidenced by the monographs [1, 2] and
references therein. For example, when controlling non-linear stochastic
systems in robotics or space applications, robustness against uncertainties is
often leveraged to ensure safety during motion. The problem also arises in
finance, where, to mitigate unrecoverable losses, an agent may want to invest
safe assets uniquely, even in exchange for a smaller average gain. This is in
contrast with “classical” stochastic optimal control, which rather aims at
minimizing the expectation of a stochastic cost. Although high reward on
average might be attained, these risk-neutral methods result in optimal
controls that may generate trajectories that may cause catastrophic losses
with small, though non-zero, probability. As such, these solutions are
considered unacceptable in terms of safety in many applications [3].

To safely control systems whose dynamics are subject to uncertainties, various
methods have been developed. A first possible approach is robust control,
where optimal control problems are directly solved in the worst-case scenarios
[4]. Although robust control theoretically guarantees resilience to
unfavorable situations, it can be over-conservative and does not apply to
dynamics with unbounded uncertainties, e.g., stochastic differential equations
[3]. A more flexible approach is risk-averse control, which aims at minimizing
a suitable risk measure of the stochastic cost, offering a good trade-off
between minimizing the expected cost and mitigating catastrophic scenarios.
Many risk-averse control methods involve discretizing the dynamics before
proceeding with optimization [2]. For discrete problems, risk-averse Model
Predictive Control (MPC) is a popular choice [5, 6]. It operates similarly to
standard MPC but optimizes a risk measure instead of the expectation of the
cost. Continuous-time risk-averse control methods are less common. Although
risk-averse analogs of the Hamilton-Jacobi-Bellman (HJB) equation [7] and the
Pontryagin Maximum Principle (PMP) [8] have been established, these methods do
not directly provide tractable numerical schemes.

Designing a risk-averse method that scales with many different risk measures
is a challenging task. Therefore, the focus has mostly been on developing
methods that minimize specific risk measures. A significant hindrance of risk
measures is their lack of time consistency, meaning that the optimal solution
at the current time may not solely depend on the current state’s values [9]. A
class of time-consistent risk measures exists for which dynamic programming
can be employed to solve the associated Markov Decision Process (MDP) [10,
11]. Nevertheless, such methods do not apply to the widely adopted Conditional
Value at Risk (CV@R), a risk measure that is not time consistent [1]. To
address this issue, a state-space lifting approach has been proposed in [12].
Although effective, these approaches are limited to discrete-time and finite
discrete space MDP. A continuous-time and continuous space approach for the
CV@R has been proposed in [13], where a representation of the CV@R is used to
re-write the problem into a double minimization that is solved with a gradient
descent approach. Another important class of risk measures is the family of
risk measures that can be expressed as a linear function of the state’s
probability distribution [1, Chapter 6]. To minimize these risk measures, a
mean field game approach can be applied. In [14] and [15], the Frank-Wolf
algorithm is used to solve the problem, with proof of convergence properties
to a necessary optimality condition. Yet, these approaches work with very
specific risk measures.

In this paper, we propose a gradient-based method to solve continuous-time,
non-convex risk-averse control problems where the corresponding risk measure
is only coherent, a fairly general requirement encompassing all the
aforementioned risk measures. We will only consider the case of a diffusion
independent of the control to keep the computation of gradients tractable.
This is a reasonable assumption for applications such as aerospace and
robotics, where the diffusion process often models external perturbations
(e.g., gusts of wind on a robot), which are then uncorrelated from the
control. The motivation to pursue this research direction came from the fact
that gradient descent methods have already seen a lot of success in the more
classical setting of average-oriented minimization [16], non-stochastic
control [17], as well as some risk-averse methods [13, 15, 18]. Minimizing
directly the risk measure is challenging. We, therefore, leverage the duality
properties of coherent risk measures to transform the minimization problem
into a min-max problem of a smooth, non-convex/linear function. Min-max
optimization has been widely studied in the convex-concave setting [19].
Still, recent applications to machine learning (particularly Generative
Adversarial Networks) have pushed studies toward the non-convex / strongly
concave setting [20]. The gradient descent-ascent method has proven to be a
successful technique for addressing these problems in finite-dimensional
spaces. In [21], the proposed algorithm reaches an $\epsilon$ distance from a
critical point within a finite time frame. In [22], the algorithm’s
convergence to the critical point is guaranteed, given a specific geometric
property of the cost function.

Our contribution consists of applying gradient descent ascent to solve the
min-max problem, which stems from dual representations of coherent risk
measures, by extending the work in [22]. This is not an easy task in that our
problem is not smooth nor strongly concave. To address such an issue, we
propose to apply gradient descent-ascent to a modified version of the problem
which is smooth, non-convex / strongly concave, and arbitrarily “close” to the
original formulation. Via appropriate proofs of convergence, we then show that
the gradient of the cost converges to 0. By requiring more regularity in the
dynamics of the system, we can also prove that the sequence of controls
generated by our algorithm converges (in an appropriate sense) to the set of
critical points of the original problem. We finally implement the algorithm on
a realistic robotic application to showcase the efficiency of our easy-to-use
method.

The paper is organized as follows. In Section 2, we begin by providing a brief
summary of some basic concepts of stochastic calculus and risk measures.
Section 3 outlines the specific settings of dynamics and control strategies
under consideration. Next, we present in Section 4 our algorithm, providing
details for the computation of the gradient. In Section 5, we prove various
convergence properties under different assumptions on the dynamics. Finally,
we analyze in Section 6 numerical convergence, and we showcase the
effectiveness of our approach through a series of simulations.

##  2 Notations and Preliminary Results

In this section, we briefly present some basic concepts and results of
stochastic calculus we need in our work. More details can be found in [23] and
[24, Chapter 1.6]. From now on, we fix $n,m,d\in\mathbb{N}$, a finite time
horizon $T>0$, and $p\in[1,+\infty)$. Let $E$ be a vector space. We denote
$E^{*}$ its dual. As a result of Riesz theorem, if $E$ is also a Hilbert
space, then it is isomorphic to its dual space $E^{*}$. Consequently,
throughout the paper, elements in $(L^{2})^{*}$ are implicitly assumed as
elements in $L^{2}$. We denote $\|\cdot\|$ as the Euclidian norm. Other norms
will be properly introduced as they are needed. We consider random variables
in a probability space $(\Omega,\mathcal{G},\mathbb{P})$. For any sub-sigma
algebra $\mathcal{S}\subset\mathcal{G}$, we denote by
$L^{p}_{\mathcal{S}}(\Omega,\mathbb{R}^{n})$ the Banach space of random
variables $z:\Omega\rightarrow\mathbb{R}^{n}$ that are
$\mathcal{S}$-measurable and satisfy:

$\|z\|_{L^{p}}\triangleq\mathbb{E}\left[\|z\|^{p}\right]^{1/p}<\infty.$

It is a standard consequence of Riesz theorem that
$L^{p}_{\mathcal{S}}(\Omega,\mathbb{R}^{n})^{*}\cong
L^{q}_{\mathcal{S}}(\Omega,\mathbb{R}^{n})$, with $q$ the conjugate exponent
of $p$. Let $(W_{t})_{t\in[0,T]}$ be a $d$-dimensional Wiener process,
generating a filtration

$\mathcal{F}\triangleq(\mathcal{F}_{t})_{t\in[0,T]}=(\sigma(W_{s},0\leq s\leq
t))_{t\in[0,T]}.$

We denote by $L^{p}_{\mathcal{F}}([0,T]\times\Omega,\mathbb{R}^{n})$ the
Banach space of processes $x:[0,T]\times\Omega\rightarrow\mathbb{R}^{n}$ which
are progressive with respect to $\mathcal{F}$ and such that:

$\|x\|_{L^{p}_{\mathcal{F}}}\triangleq\mathbb{E}\left[\int_{0}^{T}\|x\|^{p}(t)dt\right]^{1/p}<\infty.$

In addition, we denote by
$C^{p}_{\mathcal{F}}([0,T]\times\Omega,\mathbb{R}^{n})$ the Banach space of
$\mathcal{F}$-adapted processes $x$ which have continuous sample paths and
finite sup norm

$\|x\|_{C^{p}_{\mathcal{F}}}\triangleq\mathbb{E}\left[\underset{0\leq t\leq
T}{\sup}\|x\|^{p}(t)\right]^{1/p}<\infty.$

In particular, $C^{p}_{\mathcal{F}}([0,T]\times\Omega,\mathbb{R}^{n})\subset
L^{p}_{\mathcal{F}}([0,T]\times\Omega,\mathbb{R}^{n})$. In what follows, we
often use the notation $x(t)$, $t\in[0,T]$ to denote progressive processes. A
$\mathcal{F}$-adapted process $x:[0,T]\times\Omega\rightarrow\mathbb{R}^{n}$
such that $x(t)\in L^{1}_{\mathcal{F}_{T}}$ for every $t\in[0,T]$ is called a
martingale provided that

$\mathbb{E}[x(t)|\mathcal{F}_{s}]=x(s),$

for all $0\leq s\leq t\leq T$. We then say that a martingale
$x:[0,T]\times\Omega\rightarrow\mathbb{R}^{n}$ is uniformly bounded in
$L^{p}_{\mathcal{F}}([0,T]\times\Omega,\mathbb{R}^{n})$ if there exists a
constant $C>0$ such that

$\underset{t\in[0,T]}{\textrm{sup}}\|x(t)\|_{L^{p}}\leq C.$

For every $x=(x^{1}|\dots|x^{d})\in
L^{p}_{\mathcal{F}}([0,T]\times\Omega,\mathbb{R}^{n\times d})$, we write

$y(t)=\int_{0}^{T}x(t)dW_{t}\triangleq\sum_{i=1}^{d}\int_{0}^{T}x^{i}(t)dW^{i}_{t},$

for the Itô integral of $x$ with respect to $W$, and recall that $y$ is then a
martingale in $C^{p}_{\mathcal{F}}([0,T]\times\Omega,\mathbb{R}^{n})$. We also
recall the Burkholder-Davis-Gundy inequality for stochastic integrals:

$\mathbb{E}\left[\underset{0\leq t\leq T}{\textrm{sup}}\|y(t)\|^{p}\right]\leq
C_{p}\mathbb{E}\left[\left(\int_{0}^{T}\|x(t)\|^{2}dt\right)^{p/2}\right],$
(1)

where $C_{p}$ is a constant which depends on $p$ uniquely.

As outlined in the introduction, our work is centered on controlling the
extreme values or tail of the distribution of a stochastic cost rather than
controlling its expectation uniquely. To estimate these extreme values, we
leverage risk measures. Unlike expectation, risk measures are not always
linear or Fréchet-differentiable. However, they possess several interesting
properties we discuss below. We refer to [1, Chapter 6] for all the results
presented below on risk measures.

######  Definition 2.1.

A finite risk measure $\rho$ is a mapping from a space
$\mathcal{Z}=L^{p}(\Omega,\mathbb{R})$ with $p\in[1,+\infty)$ to $\mathbb{R}$.
A finite risk measure is said to be coherent if it verifies the following
properties:

Convexity:

$\forall(Z,Z^{\prime})\in\mathcal{Z}\quad\forall\lambda\in[0,1],\quad\rho(\lambda
Z+(1-\lambda)Z^{\prime})\leq\lambda\rho(Z)+(1-\lambda)\rho(Z^{\prime}).$

Monotonicity111for $(Z,Z^{\prime})\in\mathcal{Z}$, it is said that $Z\leq
Z^{\prime}$ if $\mathbb{P}(\\{\omega:Z(\omega)\leq Z^{\prime}(\omega)\\})=1$.:

$\forall(Z,Z^{\prime})\in\mathcal{Z},\quad Z\leq
Z^{\prime}\Rightarrow\rho(Z)\leq\rho(Z^{\prime}).$

Translation equivariance:

$\forall Z\in\mathcal{Z},\quad\forall
a\in\mathbb{R},\quad\rho(Z+a)=\rho(Z)+a.$

Positive homogeneity:

$\forall Z\in\mathcal{Z},\quad\forall
t\in\mathbb{R}^{+},\quad\rho(tZ)=t\rho(Z).$

Definition 2.1 yields some interesting properties that are used throughout our
work.

######  Theorem 2.2 ([1]).

A finite coherent risk measure $\rho:\mathcal{Z}\rightarrow\mathbb{R}$
verifies the following properties:

  1. 1.

The convex subdifferential of $\rho$ at 0, denoted by $\partial\rho(0)$, is a
nonempty closed and bounded subset of $\mathcal{Z}^{*}$. It is, in particular,
weakly-* compact.

  2. 2.

For every $Z\in\mathcal{Z}$, the risk measure can be represented as

$\rho(Z)=\underset{\zeta\in\partial\rho(0)}{\max}\ \mathbb{E}[\zeta Z].$

  3. 3.

For every $Z\in\mathcal{Z}$, $\partial\rho(Z)$ is a nonempty closed and
bounded subset of $\mathcal{Z}^{*}$, and it can be expressed as

$\partial\rho(Z)=\underset{\zeta\in\partial\rho(0)}{\arg\max}\
\mathbb{E}[\zeta Z].$

  4. 4.

For every $Z,H\in\mathcal{Z}$, the mapping $\rho$ is Hadamard directionally
(sub-) differentiable at $Z$ along $H$, and the differential can be expressed
as

$\rho^{\prime}(Z,H)=\underset{\zeta\in\partial\rho(Z)}{\max}\ \mathbb{E}[\zeta
H].$

The most commonly adopted risk measure is the Conditional-Value-at-Risk,
denoted by CV@R, which we will regularly consider in our examples.
Intuitively, CV@R represents the average of the highest values that a random
variable can attain. Consequently, it is a highly valuable metric for
assessing potential losses in a worst-case scenario.

######  Definition 2.3.

For $Z$ in $L^{2}(\Omega,\mathbb{R})$ and $\alpha\in(0,1]$, the Conditional-
Value-at-Risk of parameter $\alpha$ of $Z$ is given by:

$\textrm{CV@R}_{\alpha}[Z]=\inf\
\left\\{t+\frac{\mathbb{E}[\max(0,Z-t)]}{\alpha}:t\in\mathbb{R}\right\\}.$

The parameter $\alpha$ sets up the number of “worst-case scenarios”. For
instance, setting $\alpha=0.05$ means that $\textrm{CV@R}_{\alpha}[Z]$
corresponds to the average of the top $5\%$ values that $Z$ can attain. Note
that the subdifferential at 0 of the Conditional-Value-at-Risk can be
explicitly calculated, which is extremely helpful for numerical simulations.

######  Proposition 2.4.

The subdifferential of $\textrm{CV@R}_{\alpha}$ at 0 is given by:

$\partial(\textrm{CV@R}_{\alpha})(0)=\\{\zeta\in
L^{2}(\Omega,\mathbb{R}):\mathbb{E}[\zeta]=1,\ \zeta\in[0,\alpha^{-1}]\
\textrm{a.s.}\\}.$ (2)

##  3 Problem Formulation

In what follows, we detail the settings in which we study the controlled
stochastic dynamics. We then devise the optimization problem and present some
necessary conditions for optimality an optimal solution must satisfy. As
already mentioned, we focus on two different stochastic dynamics:

  1. 1.

Setting A. A general non-linear stochastic system with no control term in the
diffusion, which is coherent with our framework of modeling exterior
perturbations affecting a mechanical system. The proposed algorithm can be run
when considering such dynamics, although only partial theoretical guarantees
of convergence may be derived. Nevertheless, through numerical simulations, we
show that reliable solutions are often generated.

  2. 2.

Setting B. A control affine stochastic system with no control term in the
diffusion. This system is a particular case of the general dynamics which are
often encountered in many applications, such as aerospace and robotics [25,
26, 27, 28]. Importantly, under such dynamics, it is possible to theoretically
prove the convergence of our algorithm.

From now on, we assume state variables take values in $\mathbb{R}^{n}$,
whereas control variables take values in $\mathbb{R}^{m}$ . For the sake of
clarity in the exposition and without loss of generality, from now on, we
assume the dynamics are perturbed through a weighted one-dimensional Wiener
process $W$, as below.

###  3.1 Setting A: The general non-linear control system

In this general setting, we consider non-linear stochastic dynamics in the
form

$\left\\{\begin{array}[]{ll}dX_{u}(t)&=\hskip
10.00002ptb(t,X_{u}(t),u(t))dt+\sigma(t,X_{u}(t))dW_{t},\\\ X_{u}(0)&=\hskip
10.00002ptX_{0},\end{array}\right.$ (3)

where $X_{0}\in\mathbb{R}^{n}$ is a deterministic initial condition (note that
extending our results where the initial condition is a random variable is
straightforward). In particular, we consider diffusion terms $\sigma$
independent from the control input. This is generally the case in many
applications such as aerospace and robotics [29]. We also consider the
following classical assumptions on the mappings
$b:[0,T]\times\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}^{n}$ and
$\sigma:[0,T]\times\mathbb{R}^{n}\to\mathbb{R}^{n}$ to hold true:

  1. 1.

The mappings

$b(\cdot,x,u):[0,T]\rightarrow\mathbb{R}^{n}\quad\sigma(\cdot,x):[0,T]\rightarrow\mathbb{R}^{n}$

are continuous for every $(x,u)\in\mathbb{R}^{n}\times\mathbb{R}^{m}$.

  2. 2.

For all $t$ in [0,T], the mappings

$b(t,\cdot,\cdot):\mathbb{R}^{n}\times\mathbb{R}^{m}\rightarrow\mathbb{R}^{n}\quad\sigma(t,\cdot):\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$

are differentiable with bounded $L$-Lipschitz gradient. That is, there exists
a constant $L>0$ such that

$\begin{split}&\left\|\frac{\partial b}{\partial
x}(t,x,u)\right\|+\left\|\frac{\partial b}{\partial
u}(t,x,u)\right\|+\left\|\frac{\partial\sigma}{\partial x}(t,x)\right\|\leq
L,\\\ &\left\|\frac{\partial b}{\partial x}(t,x,u)-\frac{\partial b}{\partial
x}(t,y,v)\right\|\leq L(|x-y|+|u-v|),\\\ &\left\|\frac{\partial b}{\partial
u}(t,x,u)-\frac{\partial b}{\partial u}(t,y,v)\right\|\leq L(|x-y|+|u-v|),\\\
&\left\|\frac{\partial\sigma}{\partial x}(t,x)-\frac{\partial\sigma}{\partial
x}(t,y)\right\|\leq L|x-y|.\end{split}$

In particular, given a progressive control input $u\in
L^{2}_{\mathcal{F}}([0,T]\times\Omega,\mathbb{R}^{m})$, the assumptions above
guarantee the existence and uniqueness of a solution $X_{u}\in
C^{2}_{\mathcal{F}}([0,T]\times\Omega,\mathbb{R}^{n})$ to (3), see, e.g., [24,
Chapter 1.6]. The additional assumption on Lipschitz gradients, in particular,
will be required to compute the Fréchet derivative of $X_{u}$ with respect to
$u$ (see Section 4.2).

We consider cost functions of the form

$J(u)\triangleq\int_{0}^{T}f(t,X_{u}(t),u(t))dt+g(T,X_{u}(T)),\quad\textrm{with
$X_{u}$ solution of \eqref{eq:base_system_nonlinear},}$

where $f$ and $g$ are positive continuous functions, such that
$f(t,\cdot,\cdot)$ and $g(t,.)$ are $C^{1}$ with Lipschitz gradient in the
state and control variables for all $t$ in $[0,T]$. Note that these functions
cover a variety of costs that are commonly used in many applications. For
instance, in trajectory tracking $f$ consists of the quadratic difference
between a reference trajectory and the output trajectory [25].

###  3.2 Setting B: control affine non-linear systems

In this setting, we consider the particular case of dynamics of the form

$\left\\{\begin{array}[]{ll}dX_{u}(t)&=\hskip
10.00002pt\left[b_{0}(t,X_{u}(t))+\sum_{i=0}^{m}u_{i}(t)b_{i}(t,X_{u}(t))\right]dt+\sigma(t,X_{u}(t))dW_{t},\\\
X_{u}(0)&=\hskip 10.00002ptX_{0},\end{array}\right.$ (4)

where the mappings $b_{i}:[0,T]\times\mathbb{R}^{n}\to\mathbb{R}^{n}$ and
$\sigma:[0,T]\times\mathbb{R}^{n}\to\mathbb{R}^{n}$ are $C^{1}$ with Lipschitz
gradient in the state variables for all $t$ in $[0,T]$. Again, these
assumptions ensure existence and uniqueness of $X_{u}\in
C^{2}_{\mathcal{F}}([0,T]\times\Omega,\mathbb{R}^{n})$ solution to (4). We
additionally require the functions $b_{i}$ to be uniformly bounded, which
still allows for a wide variety of dynamics. Finally, for this setting, we
consider slightly less generic stochastic cost functions of the form

$\displaystyle J(u)\triangleq
C_{u}\|u\|^{2}_{\mathcal{U}}+\int_{0}^{T}f(t,X_{u}(t))dt+g(T,X_{u}(T)),$
$\displaystyle\hskip 215.2771pt\textrm{with $X_{u}$ solution of
\eqref{eq:affine_system_nonlinear}},$

where $f$ and $g$ follow the same assumptions as above. Additionally, we
require that $f$ and $g$ satisfy

$\left\|\frac{\partial(f,g)}{\partial x}(t,x)-\frac{\partial(f,g)}{\partial
x}(t,y)\right\|\leq L\|x-y\|,\quad\left\|\frac{\partial(f,g)}{\partial
x}(t,0)\right\|\leq L,$

for some constant $L>0$. These assumptions, for instance, enable the use of
quadratic functionals. Note that such cost functions are still relevant for
many applications, including trajectory tracking-type problems [25].

###  3.3 Optimization problem

We seek optimal controls in the space
$\mathcal{U}=L^{2}([0,T],\mathbb{R}^{m})\times\mathbb{R}^{q}$ for some fixed
$q\in\mathbb{N}$, which parametrizes feedback controls in
$L^{2}_{\mathcal{F}}([0,T]\times\Omega,\mathbb{R}^{m})$. More specifically, to
each control parameter $u=(v,\lambda)\in\mathcal{U}$ we associate a feedback
control in the following manner:

$u_{v,\lambda}(t,x)=v(t)+K_{\lambda}(x),$ (5)

where $K_{\lambda}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$ is a vector-valued
function that is smoothly parametrized by the vector $\lambda$. We assume that

$\begin{split}\forall
x,y\in\mathbb{R}^{n},\forall\lambda\in\mathbb{R}^{q},\quad&\|K_{\lambda}(x)-K_{\lambda}(y)\|\leq
C(1+\|\lambda\|)\|x-y\|\\\ \forall
x\in\mathbb{R}^{n},\forall\lambda,\mu\in\mathbb{R}^{q},\quad&\|K_{\lambda}(x)-K_{\mu}(x)\|\leq
C(1+\|x\|)\|\lambda-\mu\|.\end{split}$ (6)

We also assume the function $(x,\lambda)\mapsto K_{\lambda}(x)$ has a
lipschitz gradient, meaning

$\|\nabla_{\lambda,x}K_{\lambda}(x)-\nabla_{\lambda,x}K_{\mu}(y)\|\leq
L(\|x-y\|+\|\lambda-\mu\|)$ (7)

for some constant $L>0$. In our implementations, we use a feedback law that is
linear in $\lambda$ and $x$, which therefore automatically verifies both
assumptions, but more generic choices are clearly possible. We denote by
$X_{v,\lambda}\in C^{2}_{\mathcal{F}}([0,T]\times\Omega,\mathbb{R}^{n})$ the
unique solution to (3) (or (4)) which is generated by the control
$u_{v,\lambda}$. These control laws will be composed of a deterministic term
to steer the trajectory and a stochastic term to compensate for the diffusion.
This choice of controls is motivated by the fact that feedback-type controls
are easy to implement and are widely used in practice for risk-averse control
[3]222Note that in this work, we assume that the state variables are
completely observable, leaving the investigation of a more general setting as
a future research direction.. One easily verifies that under (6), the
controlled systems under these controls still satisfy the assumptions of
Setting A or Setting B. In what follows, we endow the space $\mathcal{U}$ with
the scalar product

$\langle(v_{1},\lambda_{1}),(v_{2},\lambda_{2})\rangle_{\mathcal{U}}\
\triangleq\int_{0}^{T}v_{1}(t)\cdot v_{2}(t)dt+\lambda_{1}\cdot\lambda_{2}.$

From now on, since clear from the context we equivalently denote
$u=(v,\lambda)=u_{v,\lambda}$ and $X_{u}=X_{v,\lambda}=X_{u_{v,\lambda}}$ for
simplicity.

Let us now state and reformulate our optimal control problem as an infinite-
dimensional min-max optimization problem using the coherent measure properties
described in Theorem 2.2. In particular, let $\rho$ be a finite coherent risk
measure defined on $L^{1}_{\mathcal{F}_{T}}(\Omega,\mathbb{R})$. The main
objective of this paper is to find a local optimum of the problem

$\underset{u\in\mathcal{U}}{\min}\ \rho(J(u)).$ (8)

This problem is not easily solvable as $\rho$ is not always Fréchet
differentiable. Given the properties of coherent risk measures we listed
previously, we can rewrite (8) via the following min-max formulation:

$\underset{u\in\mathcal{U}}{\min}\underset{\zeta\in\partial\rho(0)}{\max}\
\mathbb{E}[\zeta J(u)].$ (9)

Let us remark that $\partial\rho(0)$ is a subset of
$L^{\infty}_{\mathcal{F}_{T}}(\Omega,\mathbb{R})\subset
L^{2}_{\mathcal{F}_{T}}(\Omega,\mathbb{R})$. Using this formulation, we can
leverage saddle point algorithms to solve problem (8). Saddle point algorithms
have been extensively investigated for min-max optimization, motivating our
approach [19, 20, 21, 22].

Before introducing our method to solve (9) (and therefore (8)), we
characterize its optimal solutions through appropriate necessary conditions
for optimality.

###  3.4 Necessary conditions for optimality

In what follows, we assume our problem is in the most general Setting A. As of
common use in min-max optimization settings, we propose to solve Problem (9)
by finding extremal points [30]. For this, let us first introduce appropriate
first-order necessary conditions for optimality which are satisfied by any
optimal solution $u^{*}$ to (8). These conditions involve the differential of
the cost function $J$ that we first need to establish clearly.

######  Lemma 3.1.

The cost function $J:\mathcal{U}\rightarrow L^{2}_{\mathcal{F}_{T}}$ is
Fréchet differentiable. Moreover, let $\omega\in\Omega$ and $u\in\mathcal{U}$.
Thanks to the Riesz representation theorem in the Hilbert space $\mathcal{U}$,
we can represent the differential $D_{u}J(u,\omega)$ by an element of
$\mathcal{U}$ denoted $\nabla J(u,\omega)$ such that

$\forall h\in\mathcal{U},\quad D_{u}J(u,\omega)\cdot h=\langle\nabla
J(u,\omega),h\rangle_{\mathcal{U}}.$

The proof can be found in A.1, along with an explicit formula of $\nabla J$.

Additional notations: In what follows, we call gradient of $J$ the function
$\nabla J(\cdot,\cdot)$, that we see as a mapping $\nabla
J:\mathcal{U}\rightarrow L^{2}_{\mathcal{F}_{T}}(\Omega,\mathcal{U})$. We may
need to differentiate the two components for clarity, in which case we will
write $\nabla J=(\nabla_{v}J,\nabla_{\lambda}J)$, with $\nabla_{v}J(u)\in
L^{2}_{\mathcal{F}_{T}}(\Omega,L^{2}([0,T],\mathbb{R}^{m}))$ and
$\nabla_{\lambda}J(u)\in L^{2}_{\mathcal{F}_{T}}(\Omega,\mathbb{R}^{q})$.
Additionally, for an element $z$ in
$L^{2}_{\mathcal{F}_{T}}(\Omega,\mathcal{U})$ and $u$ in $\mathcal{U}$, we
will denote by $\langle z,u\rangle_{\mathcal{U}}$ the random variable in
$L^{2}_{\mathcal{F}_{T}}(\Omega,\mathbb{R})$ such that

$\forall\omega\in\Omega,\quad\langle z,u\rangle_{\mathcal{U}}(\omega)=\langle
z(\omega),u\rangle_{\mathcal{U}}.$

In particular, we can “invert” the expectation and the scalar product as
follows:

$\mathbb{E}[\langle
z,u\rangle_{\mathcal{U}}]=\langle\mathbb{E}[z],u\rangle_{\mathcal{U}}.$ (10)

With these properties in mind, we can establish a necessary condition any
optimal solution to (9) must verify as follows:

######  Proposition 3.2.

Let $\rho$ be a finite coherent risk measure and let $u^{*}$ be an optimal
solution to (8). There exists $\zeta^{*}\in\partial\rho(0)$ such that

$\langle\mathbb{E}[\zeta^{*}\nabla J(u^{*})],h\rangle_{\mathcal{U}}\
=0,\quad\textnormal{for every}\quad h\in\mathcal{U}.$ (11)

The proof of Proposition 3.2 is quite similar to the proof given in [31] or
[8]. Nevertheless, we report the full proof of Proposition 3.2 in A.2 for the
sake of completeness.

###  3.5 The modified problem

Seeking extremals of min-max problems is generally a hard task, and not that
many algorithms which enjoy convergence guarantees exist in the literature.
Notably, for finite-dimensional problems with strongly concave costs, some
convergence results are provided in [22]. Inspired by this latter work, we
propose an algorithm that works on a modified version of problem (9) whose
cost is strongly concave and whose solutions are arbitrarily close to the
solutions of the original problem. More specifically, the modified problem
takes the following form:

$\underset{u\in\mathcal{U}}{\min}\ \underset{\zeta\in\partial\rho(0)}{\max}\
\mathbb{E}[(\zeta-\gamma\zeta^{2})J(u)],$ (12)

where $\gamma>0$ is a “concavifying” parameter, which controls how close (12)
is to the original problem. Note that the term $\mathbb{E}[\zeta^{2}J(u)]$ is
well defined as $\zeta\in\partial\rho(0)\subset
L^{\infty}_{\mathcal{F}_{T}}(\Omega,\mathbb{R})$. In Section 5, we discuss in
what sense the solutions to (12) converge to solutions to (9) for $\gamma\to
0$, endowing our methodology with guarantees of success. Also, from a
numerical point of view, we show later that $\gamma>0$ may be chosen in the
order of the machine precision.

##  4 Ascent-Descent Algorithm

To solve the min-max problem (9), we first introduce in this section a
gradient descent-ascent-based approach to solve the modified problem (12). A
gradient descent-ascent is an iterative optimization algorithm that can be
used to find saddle points. Descent-ascent algorithms represent arguably sound
paradigms to seek extremals for general min-max problems. Importantly, under
appropriate assumptions, convergence to extremals may be theoretically proven,
as we will show later. Until further notice, we start by considering the
setting of general non-linear dynamics (Setting A).

###  4.1 Update rule

The gradient descent-ascent algorithm is an iterative optimization algorithm
for min-max problems. At each iteration, it computes the Riesz representation
of the gradient of the function being maximized and the gradient of the
function being minimized. It then updates the estimates by moving in the
direction of their respective gradients. With obvious notations, our gradient
descent-ascent algorithm updates at each iteration $n$ our estimates
$\zeta_{n}$ and $(v_{n},K_{\lambda_{n}})$ with the following rule:

$\begin{split}&v_{n+1}=v_{n}-\eta\mathbb{E}[\zeta_{n}\nabla_{v}J(v_{n},K_{\lambda_{n}})],\\\
&\lambda_{n+1}=\lambda_{n}-\eta\mathbb{E}\left[\zeta_{n}\nabla_{\lambda}J(v_{n},K_{\lambda_{n}})\right],\\\
&\zeta_{n+1}=\mathcal{P}_{\partial\rho(0)}(\zeta_{n}+\beta
J(v_{n},K_{\lambda_{n}})),\end{split}$ (13)

where $\mathcal{P}_{\partial\rho(0)}$ is the projection on the closed convex
set $\partial\rho(0)$, and $\eta$ and $\beta$ are two positive constants that
represent the step size of the gradient. To increase convergence speed, one
could make these steps vary from iteration to iteration based on the change of
direction of the gradient [32]. We will keep them constant for the sake of
simplicity. All convergence results should be valid for a range of step sizes
that are both upper and lower bounded, although proof of this improvement is
left for future work.We recall that $\partial\rho(0)$ can be viewed as a
subset of $L^{\infty}_{\mathcal{F}_{T}}(\Omega,\mathbb{R})$, which justifies
the above mathematical operations between $\zeta_{n}$ and the other terms.
Note that it is important to project $\zeta_{n}+\beta J(u_{n})$ onto
$\partial\rho(0)$, as there is no a priori guarantee that $\zeta_{n}+\beta
J(u_{n})$ stays inside $\partial\rho(0)$.

Unfortunately, through the scheme (13), no clear convergence result is
available. To develop theoretical guarantees of convergence, we propose to
rather leverage the following alternative rule, which essentially represents
an update rule for Problem (12):

$\begin{split}&v_{n+1}=v_{n}-\eta\mathbb{E}[\zeta_{n}(1-\gamma\zeta_{n})\nabla_{v}J(v_{n},K_{\lambda_{n}})],\\\
&\lambda_{n+1}=\lambda_{n}-\eta\mathbb{E}\left[\zeta_{n}(1-\gamma\zeta_{n})\nabla_{\lambda}J(v_{n},K_{\lambda_{n}})\right],\\\
&\zeta_{n+1}=\mathcal{P}_{\partial\rho(0)}(\zeta_{n}+\beta(1-2\gamma\zeta_{n})J(v_{n},K_{\lambda_{n}})).\end{split}$
(14)

The algorithms described by (13) and (14) offer the advantage of being easily
implementable, provided explicit expressions of the gradient $\nabla J$ are
available. We provide such expressions in the following section.

###  4.2 Computation of gradients

The practical implementation of (13) and (14) requires computing the gradient
of the cost, and in particular of the trajectory, with respect to both the
variables $\zeta$ and $u$. Below, we summarize the expressions of this
gradient, referring the reader to A.1 for their proof. Let us clarify that we
denote $K_{\lambda}^{\prime}(x)$ the derivative of $K_{\lambda}(x)$ with
respect to $x$.

######  Proposition 4.1.

Fix a control $u=(v,\lambda)\in\mathcal{U}$. Under the assumptions of Setting
A, the Riesz representation of the gradient of $J$ at $u=(v,\lambda)$ is

$\displaystyle\nabla_{v}J(v,\lambda)(t)=I_{u}(t)\phi_{u}^{-1}(t)\partial_{u}b(t,X_{u}(t),u(t))$
$\displaystyle\nabla_{\lambda}J(v,\lambda)=\int_{0}^{T}I_{u}(s)\phi_{u}^{-1}(s)\partial_{u}b(s,X_{u}(s),u(s))\nabla_{\lambda}K_{\lambda}(X_{u}(s))ds.$

where

$\begin{split}I_{u}(s)&\triangleq\partial_{x}g(T,X_{u}(T))\phi_{u}(T)\\\
&+\int_{s}^{T}[\partial_{x}f(t,X_{u}(t),u(t))+\partial_{u}f(t,X_{u}(t),u(t))K_{\lambda}^{\prime}(X_{u}(t))]\phi_{u}(t)dt.\end{split}$

Here, $\phi_{u}(.)$ is the fundamental matrix solution to the linearized
problem, i.e., the unique solution to

$\left\\{\begin{array}[]{ll}&d\phi(t)=[\partial_{x}b(t,X_{u}(t),u(t))+\partial_{u}b(t,X_{u}(t),u(t))K_{\lambda}^{\prime}(X_{u}(t))]\phi(t)dt\\\
&\hskip 25.00003pt+\ \partial_{x}\sigma(t,X_{u}(t))\phi(t)dW_{t},\\\
&\phi(0)=Id_{n}.\end{array}\right.$ (15)

Thus, the primary challenge of implementing this method lies in simulating the
trajectories of $X_{u}$ and $\phi_{u}$ and computing expectations of random
variables that involve terms from those trajectories. To deal with these
technical details, we used a classical approach presented in [33, Chapter 10]
and refer to it for more extensive elaboration. The contents of this section
furnish the necessary components to implement the algorithm and evaluate its
performance for the general Setting A.

In the next section, we establish some convergence results that may be of
particular significance for Setting B, as they provide insight into the
convergence of the algorithm towards a critical point that satisfies the
necessary optimality condition (11).

##  5 Convergence Results

We leverage the method introduced in [22] to examine the convergence of the
smooth non-convex/strong concave gradient descent-ascent algorithm in the most
general Setting A. In what follows, we adapt the approach to an infinite-
dimensional space. With some minor adjustments, we show that the gradient of
the cost converges to 0 and that the cost function converges to a finite
value. While the study in [22] also establishes that in finite-dimensional
spaces, every converging subsequence of the control estimates tends to a
critical point, we were unable to prove this result in our general setting.
Nevertheless, in the slightly more restrictive Setting B, any weakly
converging subsequence of our control estimates converges to a point that is a
“$\gamma$-critical point.” That is, a control $\overline{u}_{\gamma}$ that
verifies

$\exists\overline{\zeta},\ \langle\mathbb{E}[\overline{\zeta}\ \nabla
J(\overline{u})],h\rangle_{\mathcal{U}}=\|h\|_{\mathcal{U}}O(\gamma)\quad\forall
h\in\mathcal{U}.$ (16)

This important result shows that by progressively selecting smaller values of
$\gamma$, we can obtain a control that verifies the necessary optimality
condition (11). In addition, taking $\gamma\to 0$ yields critical points for
the original problem, as we state at the end of this section.

###  5.1 Convergence of the gradient in Setting A

Let $(u_{n},\zeta_{n})$ be the estimates given by algorithm (14), in what
follows, we prove the convergence of $\|u_{n+1}-u_{n}\|_{\mathcal{U}}$ to 0 in
Setting A, as well as the convergence of the $\gamma$-approximated cost,
denoted $\Phi_{\gamma}:\mathcal{U}\to\mathbb{R}$ and defined as

$\Phi_{\gamma}(u)\triangleq\underset{\zeta\in\partial\rho(0)}{\max\
}\mathbb{E}[\zeta(1-\gamma\zeta)J(u_{n})].$

The proof of the convergence closely follows the one presented in [22], and
we, therefore, do not explicit all of the technical details. We indicate where
changes were needed and develop the modifications in the Appendix for
completeness. For the sake of clarity, we denote

$\zeta_{u}\triangleq\underset{\zeta\in\partial\rho(0)}{\arg\max\
}\mathbb{E}[\zeta(1-\gamma\zeta)J(u)]$

which is well defined as $\zeta\mapsto\mathbb{E}[\zeta(1-\gamma\zeta)J(u)]$ is
a weakly continuous and strongly concave function over a weakly compact set.

######  Proposition 5.1.

Let $L>0$ be such that $J$ has L-Lipschitz gradient, and $\mu>0$ such that
$\zeta\mapsto\mathbb{E}[\zeta(1-\gamma\zeta)\nabla J(u_{n})]$ is
$\mu$-strongly concave. We define $K=\frac{L}{\mu}$. If the gradient rates
$\eta$ and $\beta$ verify $\eta<\frac{1}{K^{3}(L+3)^{2}}$ and
$\beta<\frac{1}{L}$, then the sequences $(u_{n},\zeta_{n})$ given by the
update rule (14) verify:

$\|u_{n+1}-u_{n}\|_{\mathcal{U}}\rightarrow
0,\quad\|\zeta_{n+1}-\zeta_{n}\|_{L^{2}_{\mathcal{F}_{T}}}^{2}\rightarrow
0,\quad\|\zeta_{n}-\zeta_{u_{n}}\|_{L^{2}_{\mathcal{F}_{T}}}^{2}\rightarrow
0,$

###### Proof.

As mentioned above, we generalise the proof from [22, Proposition 2] with
$f(u,\zeta)=\mathbb{E}[\zeta(1-\gamma\zeta)J(u)]$, $g=0$ and
$h=\delta_{\partial\rho(0)}$, the convex characteristic function of
$\partial\rho(0)$. The key argument is to consider the function

$H(u,\zeta)=\Phi_{\gamma}(u)+\left(1-\frac{1}{4K^{2}}\right)\|\zeta-\zeta_{u}\|_{L^{2}_{\mathcal{F}_{T}}}^{2}$
(17)

and show that $H(u_{n},\zeta_{n})$ is decreasing over the iterations of the
algorithm. To prove it, we must show the Lipschitzianity of the functions
$u\mapsto\zeta_{u}$ and $u\mapsto\nabla\Phi_{\gamma}(u)$. The first one is
straightforward, as shown in [22, Proposition 1]. The second one however needs
some adjustments to our infinite-dimensional setting. Using the version of the
Danskin theorem found in [34] with the weak topology, we can first prove the
following lemma.

######  Lemma 5.2.

$\Phi_{\gamma}$ is Fréchet differentiable and its gradient is given by

$\nabla\Phi_{\gamma}=\mathbb{E}[\zeta_{u}(1-\gamma\zeta_{u})\nabla J(u)].$

The Lipschitzianity of $\nabla\Phi_{\gamma}(u)$ can be deduced as follows:

$\begin{split}\|\Phi(u_{2})-\Phi(u_{1})\|&=\|\nabla_{u}\mathcal{J}_{\gamma}(u_{2},\zeta_{u_{2}})-\nabla_{u}\mathcal{J}_{\gamma}(u_{1},\zeta_{u_{1}})\|\\\
&\leq L(\|u_{2}-u_{1}\|+\|\zeta_{u_{2}}-\zeta_{u_{1}}\|)\\\ &\leq
L(1+K)\|u_{2}-u_{1}\|\end{split}$

With these two Lipschitzianity results, we can execute the same computations
as in [22, Proposition 2] and prove the inequality

$H(u_{n+1},\zeta_{n+1})\leq
H(u_{n},\zeta_{n})-2\|u_{n+1}-u_{n}\|_{\mathcal{U}}^{2}-\frac{1}{4K^{2}}(\|\zeta_{n+1}-\zeta_{u_{n+1}}\|_{L^{2}_{\mathcal{F}_{T}}}^{2}+\|\zeta_{n}-\zeta_{u_{n}}\|_{L^{2}_{\mathcal{F}_{T}}}^{2}).$

From this inequality we can deduce the sequence $H(u_{n},\zeta_{n})$ is
decreasing and therefore converges to $H^{*}$. By summing the inequality over
the iterations, we obtain a telescoping series that yields

$\begin{split}\sum_{n=0}^{N}&\left[2\|u_{n+1}-u_{n}\|_{\mathcal{U}}^{2}-\frac{1}{4K^{2}}(\|\zeta_{n+1}-\zeta_{u_{n+1}}\|_{L^{2}_{\mathcal{F}_{T}}}^{2}+\|\zeta_{n}-\zeta_{u_{n}}\|_{L^{2}_{\mathcal{F}_{T}}}^{2})\right]\\\
&\leq\sum_{n=0}^{N}H(u_{n},\zeta_{n})-H(u_{n+1},\zeta_{n+1})\leq
H(u_{0},\zeta_{0})-H^{*}.\end{split}$

This shows that the series on the left is convergent, meaning that its terms
tend to zero, which proves our proposition.∎

This result additionally shows some other interesting properties of the
algorithm, which are of straightforward proof.

######  Corollary 5.3.

  1. 1.

Since $\|\zeta_{n}-\zeta_{u_{n}}\|_{L^{2}_{\mathcal{F}_{T}}}^{2}$ tends to
zero and H decreases, the sequence $\Phi_{\gamma}(u_{n})$ converges to a
finite limit.

  2. 2.

If the cost $J$ is coercive, the sequence $u_{n}$ is bounded in $\mathcal{U}$
and is thus in a weakly compact set.

To go further, one would wish to find a generalization of [22, Theorem 1] by
showing that all weak limit points of our sequence $u_{n}$ are ”gamma-critical
points” and verify equation (16). Unfortunately, we could not prove this in
the general case, as a weakly converging sequence of controls $u_{n}$ does not
necessarily yield a converging sequence of states $x_{u_{n}}$. Nonetheless, we
can demonstrate the result in the slightly more restrictive setting of linear
affine controls, as shown in the next subsection.

###  5.2 Convergence of the gradient in Setting B

As mentioned previously, restricting to control affine systems allows us to
use important theoretical properties and improve convergence guarantees for
our gradient ascent-descent algorithm. The most important property we take
advantage of is the compacity of the input-output mapping $u\mapsto X_{u}$, as
shown in the following lemma:

######  Lemma 5.4.

Let $u_{n}$ be a sequence controls in $\mathcal{U}$, and $p\geq 2$. Under the
hypothesis of Setting B, we have that

$u_{n}\overset{\mathcal{U}}{\rightharpoonup}\overline{u}\hskip
5.0pt\Rightarrow\hskip
5.0ptX_{u_{n}}\overset{C^{p}_{\mathcal{F}}}{\rightarrow}X_{\overline{u}}.$

Consequently, if we decompose the cost $J$ of Setting B as
$J(u)=C_{u}\|u\|^{2}_{\mathcal{U}}+J_{X}(X_{u})$, we have

$u_{n}\overset{\mathcal{U}}{\rightharpoonup}\overline{u}\hskip
5.0pt\Rightarrow\hskip
5.0pt\nabla_{u}(J_{X}(X_{u_{n}}))\overset{C^{2}_{\mathcal{F}}}{\rightarrow}\nabla_{u}(J_{X}(X_{\overline{u}})).$

The proof of this lemma requires some long and technical computations that we
detailed in A.3. This compacity result, in turn, allows us to prove the
following:

######  Proposition 5.5.

Let $\overline{u}$ be a weak limit of a subsequence of $u_{n}$, then for any
$\overline{\zeta}\in\partial\rho(0)$ that is a weak limit of a subsequence of
$\zeta_{n}$ we have that

$\langle\mathbb{E}[\overline{\zeta}\ \nabla
J(\overline{u})],h\rangle_{\mathcal{U}}=\|h\|_{\mathcal{U}}\
O(\gamma)\quad\forall v\in\mathcal{U}.$ (18)

###### Proof.

Let us first use our result on the convergence of the general algorithm. We
can see that

$\begin{split}v_{n+1}-v_{n}&=-\eta\mathbb{E}[(\zeta_{n}-\gamma\zeta_{n}^{2})\nabla_{v}J(u_{n})],\\\
&=-\eta\mathbb{E}[\zeta_{n}\nabla_{v}J(u_{n})]+\eta\gamma\mathbb{E}[\zeta_{n}^{2}\nabla_{v}J(u_{n})].\end{split}$

Since $\zeta_{n}$ is bounded in $L^{\infty}(\Omega,\mathbf{R)}$,
$\nabla_{v}J(u_{n})$ is bounded in $L^{2}(\Omega,\mathbf{R)}$, and
$(v_{n+1},\lambda_{n+1})-(v_{n},\lambda_{n})$ tends to 0, there exists a
constant $C>0$ such that

$\|\mathbb{E}[\zeta_{n}\nabla_{v}J(u_{n})]\|_{L^{2}(0,T)}\leq
C\gamma=O(\gamma).$ (19)

Below, we will implicitly overload the constant $C$. Similarly, we have

$|\mathbb{E}[\zeta_{n}\nabla_{\lambda}J(u_{n})]|\leq C\gamma=O(\gamma).$ (20)

We now go back to the equality we want to prove. For this, let $h$ be a
control in $\mathcal{U}$, and let $\overline{u}$ and $\overline{\zeta}$ be
weak limits of respectively $u_{n}$ and $\zeta_{n}$. We may compute

$\begin{split}\langle\mathbb{E}[\overline{\zeta}\nabla
J(\overline{u})],h\rangle_{\mathcal{U}}&=\langle\mathbb{E}[(\overline{\zeta}-\zeta_{n})\nabla
J(\overline{u})],h\rangle_{\mathcal{U}}\\\
&+\langle\mathbb{E}[\zeta_{n}(\nabla J(\overline{u})-\nabla
J(u_{n}))],h\rangle_{\mathcal{U}}\\\ &+\langle\mathbb{E}[\zeta_{n}\nabla
J(u_{n})],h\rangle_{\mathcal{U}}.\end{split}$

Using equations (19) and (20), we have that

$|\langle\mathbb{E}[\zeta_{n}\nabla
J(u_{n})],h\rangle_{\mathcal{U}}|=\|h\|_{\mathcal{U}}\ O(\gamma)\quad\forall
v\in\mathcal{U}.$

and since $\zeta_{n}$ weakly converges to $\zeta$, with an abuse of notation
we may write

$\langle\mathbb{E}[\overline{\zeta}\nabla
J(\overline{u})],h\rangle_{\mathcal{U}}=\langle\mathbb{E}[\zeta_{n}(\nabla
J(\overline{u})-\nabla J(u_{n}))],h\rangle_{\mathcal{U}}+\|h\|_{\mathcal{U}}\
O(\gamma).$

By further decomposing $J(u)=C_{u}\|u\|^{2}_{\mathcal{U}}+J_{X}(X_{u})$, we
additionally compute

$\begin{split}\langle\mathbb{E}[\overline{\zeta}\nabla
J(\overline{u})],h\rangle_{\mathcal{U}}&=2C_{u}\langle\mathbb{E}[\zeta_{n}(\overline{u}-u_{n})],h\rangle_{\mathcal{U}}\\\
&+\langle\mathbb{E}[\zeta_{n}(\nabla_{u}J_{X}(X_{\overline{u}})-\nabla_{u}J_{X}(X_{u_{n}}))],h\rangle_{\mathcal{U}}+\|h\|_{\mathcal{U}}\
O(\gamma).\end{split}$

Using Lemma 5.4 and the fact that $u=(v,\lambda)$ is deterministic, again with
an abuse of notation, we can write

$\langle\mathbb{E}[\overline{\zeta}\nabla
J(\overline{u})],h\rangle_{\mathcal{U}}=2C_{u}\mathbb{E}[\zeta_{n}]\langle\overline{u}-u_{n},h\rangle_{\mathcal{U}}+\|h\|_{\mathcal{U}}\
O(\gamma),$

and thanks to the weak convergence of $u_{n}$ to $u^{*}$ we finally obtain
that

$\langle\mathbb{E}[\overline{\zeta}\nabla
J(\overline{u})],h\rangle_{\mathcal{U}}=\|h\|_{\mathcal{U}}\ O(\gamma).$

∎

This result implies that, as we make $\gamma$ tend to 0, the solutions
obtained will get closer to our original problem, as summarized in the
following corollary.

######  Corollary 5.6.

Let $\overline{u}^{\gamma}$ be a weak limit of a sequence of controls
$u_{n}^{\gamma}$ obtained with the update rule (14) with parameter $\gamma$.
For any decreasing sequence $(\gamma_{k})_{k\in\mathbf{N}}$ tending to 0, we
can define a sequence $(\overline{u}^{\gamma_{k}})_{k\in\mathbf{N}}$ of
controls whose weak limits all verify the original optimality condition (11).

This corollary indicates that if we find with our algorithm a
$\gamma$-critical point for smaller and smaller values of $\gamma$, we can
find a control law that verifies the necessary optimality condition.
Theoretically, lowering $\gamma$ should imply lowering the gradient step
$\eta$ to satisfy the assumptions in Proposition 5.1, implying a slower
convergence in practice. However, the numerical results we present in the next
section show that the two convergences, i.e., with $\gamma=0$ or $\gamma\sim
0$, are essentially equivalent, thus there is no downside in taking $\gamma>0$
very small, e.g., in the order of the epsilon machine.

##  6 Numerical Simulations

In this section, we present some simulation results. Our algorithm was
implemented and tested on two systems, one in Setting A, and the other in
Setting B. We respectively denote them as System A and System B. By testing
our algorithm on these two different systems, we can evaluate its
effectiveness and robustness in solving optimal control problems in both
settings. System A is an academic non-linear system that simulates a steering
problem, where a vehicle is controlled through its angular velocity while
moving at a constant speed. It is subject to perturbations in the state which,
for instance, may model measurement errors. On the other hand, System B is a
control affine system that models a real-world robot called AstroBee [35],
designed to operate in a zero-gravity environment, such as a space station.
The diffusion term may, for instance, model sudden gusts of air randomly
generated by onboard air purification systems, which destabilizes the robot.
For simplicity, we designed and leveraged a very simple stochastic model to
model the disturbances; note that other models exist in the literature, e.g.,
[36, 37].

###  6.1 System presentation

For both systems, the objective is to move from one point to another in a
corridor while following a reference trajectory and avoiding obstacles. The
reference trajectory is given by a reference control that provides a
satisfying solution to the deterministic problem. However, when we add
perturbations to the system, the trajectories resulting from the reference
control often collide with obstacles. To overcome this challenge, our
algorithm searches for a control strategy that minimizes a risk measure of the
stochastic cost function. The cost function is composed of two terms: a
trajectory-tracking term that tracks the reference trajectory and a term
$J_{obstacles}(X)$ that heavily penalizes trajectories that collide with
obstacles. This penalization method as presented in [38] is a classic optimal
control approach to get rid of the constraints and is widely used in the
literature [39, 40, 41].

$J(u)=\|u\|_{\mathcal{U}}^{2}\ +\ \int_{0}^{T}\|X_{u}(t)-X_{ref}(t)\|^{2}dt\
+\ J_{obstacles}(X_{u}).$

The term $J_{obstacles}(X)$ acts as a potential that is extremely high in the
obstacles and thus forces the trajectories away. The risk measure
significantly weighs the worst-case scenarios, which means that the cost of a
control strategy will increase substantially if some random trajectories
produced by the control collide with an obstacle. A control minimizing the
risk measure should therefore avoid collisions with high probability.

###  6.2 Results

As shown in Figure 1, our algorithm converges arbitrarily close to solutions
for both System A and System B, as long as we use a sufficiently small step
size. Additionally, implementing the algorithm with an infinitesimal value of
$\gamma$ results in the same level of convergence as when using a non-
negligible value of $\gamma$. In particular, although theoretical proofs
typically require $\gamma$ to be strictly greater than 0, our numerical
results suggest that it may not always be necessary to achieve convergence.

(a) Plot of the gradient norm over the iterations, System A, $\alpha=0.05$

(b) Plot of the gradient norm over the iterations, System B, $\alpha=0.05$

Figure 1: Convergence rates obtained for $\alpha=0.05$ and different values of
$\gamma$, for both the steering system (left) and the AstroBee system (right).

(a) System A: 100 trajectories, $\alpha=0.05$, and $\gamma=0$.

(b) System B: 100 trajectories, $\alpha=0.05$, and $\gamma=0$.

Figure 2: 100 trajectories given by a trained control for both systems.

In Table 1, we evaluate the effectiveness of our algorithm under three
different values of $\alpha$: $\alpha=0.05$, $\alpha=0.2$, and $\alpha=1$ (the
latter value corresponds to the minimization of the mean). By testing our
algorithm under these different values of $\alpha$, we can assess the impact
of different risk levels on the control strategy. Our main metric is the
percentage of trajectories produced by the control strategy that resulted in
collisions with obstacles. We compare these percentages between strategies
obtained with different values of $\alpha$ to assess its impact on the level
of risk in the control strategy.

Systems | $\alpha=1$ | $\alpha=0.2$ | $\alpha=0.05$
---|---|---|---
System A | 0.353$\%$ | 0.112$\%$ | 0.011$\%$
System B | 0.049% | 0.007% | 0.004%
Table 1: Percentage of collisions with obstacles for different $\alpha$ and
$\gamma=0$. Control strategies trained with higher values of $\alpha$ yield
significantly more trajectories colliding with obstacles.

To evaluate the robustness of the control strategy, we can increase the size
of the obstacles to see how much the probability of collision increases. We
rapidly re-trained the controls in this modified problem, with obstacles of a
bigger size. This allows us to assess the ability of the control strategy to
handle more challenging and uncertain conditions. From Table 4, for both
System A and System B, we observe the control solutions that minimized the
risk measure with a low value of $\alpha$ are the most conservative. This
results in trajectories that keep a larger distance from obstacles but also
deviate more from the reference trajectory.

|  Obstacles bonus size | $\alpha=1$ | $\alpha=0.2$ | $\alpha=0.05$
---|---|---|---
+0$\%$ | 0.353$\%$ | 0.112$\%$ | 0.011$\%$
+5$\%$ | 0.367$\%$ | 0.111$\%$ | 0.011$\%$
+10$\%$ | 0.470$\%$ | 0.145$\%$ | 0.013$\%$
+15$\%$ | 0.600$\%$ | 0.177$\%$ | 0.015$\%$
+20$\%$ | 0.735$\%$ | 0.225$\%$ | 0.020$\%$
Table 2: Percentage of collisions with obstacles in the steering problem.

|  Obstacles bonus size | $\alpha=1$ | $\alpha=0.2$ | $\alpha=0.05$
---|---|---|---
+0$\%$ | 0.049% | 0.007% | 0.004%
+5$\%$ | 0.044% | 0.006% | 0.004%
+10$\%$ | 0.048% | 0.006% | 0.006%
+15$\%$ | 0.054% | 0.006% | 0.006%
+20$\%$ | 0.066% | 0.008% | 0.006%
Table 3: Percentage of collisions with obstacles in the AstroBee system.

Table 4: Percentage of collisions with obstacles for different $\alpha$ in the
two systems.

To further assess the impact of the parameter $\alpha$ on the trade-off
between mean scores and worst-case scenarios, we computed the probability
densities of the costs of the obtained control strategies. By comparing the
probability densities of the costs under different values of $\alpha$, we can
identify the optimal level of risk for each system and obtain a control
strategy that balances the trade-off between mean scores and worst-case
scenarios. As expected, density distributions have higher means but smaller
tails for lower values of $\alpha$. This suggests that a more conservative
risk estimate leads to a more robust control strategy with lower worst-case
scenarios but also a lower mean score.

(a) Density of the cost for System A.

(b) Density of the cost for System B.

Figure 3: Densities of the cost for different $\alpha$ and $\gamma=0$.

##  7 Conclusion and future directions

In this paper, we develop an algorithm to solve risk-averse stochastic optimal
control problems subject to non-linear stochastic differential equations,
where a risk measure replaces expectation in the cost. By leveraging duality
results for coherent risk measures, we recast our minimization problem into a
min-max problem that we solve with a gradient descent-ascent approach. We
prove convergence properties for this algorithm, which we showcase through
appropriate numerical implementation on non-trivial control systems. Our
results show our algorithm genuinely converges under very general assumptions
and settings. Importantly, our approach of minimizing the cost under a risk
measure instead of the expectation yields more robust control strategies,
which are capable of considerably better mitigating the uncertainties
generated by the diffusion term in the stochastic differential equation.

Several exciting avenues for further exploration and testing are listed
hereafter. One direction would consist in extending our algorithm to systems
that are modeled through more complex stochastic processes, such as processes
with control variables in the diffusion or processes with jumps. Control-
dependent diffusion would allow us to model systems where the uncertainties
are caused by inputs as well. While putting a control term in the diffusion is
already theoretically feasible, we found it to be computationally very
expensive as it requires computing conditional expectations, calling for
deeper investigation. In particular, recent techniques found in the literature
[42] may allow us to efficiently compute these conditional expectations.
Another direction consists of testing our approach for other risk measures
than just the CV@R. Additionally, we are interested in extending our approach
to more sophisticated agents, e.g., soft robots. Soft robots have recently
received a particular surge of interest as they appear in an increasing number
of applications. An emerging model for the dynamics of soft robots hinges upon
infinite-dimensional dynamics ruled by Partial Differential Equations (PDEs)
[43]. It would be beneficial to extend our approach to handle stochastic PDEs:
we look forward to future developments in this area.

##  Appendix A Technical proofs

###  A.1 Proof of Lemma 3.1

In this appendix, we give the proof of Lemma 3.1. We want to show the
differentiability of the cost function. We assume to be dealing with the cost
and dynamics in Setting A. We prove first that the function
$J:\mathcal{U}\rightarrow L^{2}_{\mathcal{F}_{T}}$ is Fréchet differentiable.
We also retrieve a numerically tractable formula for the Riesz representation
of the differential, that we denote $\nabla
J(u)=(\nabla_{v}J(u),\nabla_{\lambda}J(u))$. For the sake of clarity, we
recall that if $u=(v,\lambda)$ is a control in $\mathcal{U}$, we denote $u(t)$
the value $u(t)\triangleq v(t)+K_{\lambda}(X_{u}(t))$, where $X_{u}$ uniquely
solves (3) with control (5). We also denote $K_{\lambda}^{\prime}(x)$ the
differential of $K_{\lambda}(x)$ with respect to $x$. We first start giving
two preliminary lemmas.

######  Lemma A.1.

Under assumptions of Setting A, let $u=(v_{u},\lambda_{u})$ and
$h=(v_{h},\lambda_{h})$ be two controls in $\mathcal{U}$. There exists
$C_{u,h}$ a positive continuous function of $\|u\|_{\mathcal{U}}$ and
$\|h\|_{\mathcal{U}}$ such that

$\|X_{u}-X_{h}\|_{C^{2}_{\mathcal{F}}}^{2}\leq
C_{u,h}\|u-h\|_{\mathcal{U}}^{2}.$ (21)

###### Proof.

Let $u=(v_{u},\lambda_{u})$ and $h=(v_{h},\lambda_{h})$ be two controls in
$\mathcal{U}$. The difference in the two trajectories $X_{u}$ and $X_{h}$ is
given by

$\displaystyle X_{u}(s)-X_{h}(s)$
$\displaystyle=\int_{0}^{s}\left[b(r,X_{u}(r),u(r))-b(r,X_{h}(r),h(r))\right]dr$
$\displaystyle+\int_{0}^{s}\left[\sigma(r,X_{u}(r))-\sigma(r,X_{h}(r))\right]dW_{r}.$

where we denote the control value $u(r)=v_{u}(r)+K_{\lambda_{u}}(X_{u}(r))$.
If we note $C>0$ a positive constant (that will be overloaded in the rest of
the proof), we can bound the $C^{2}_{\mathcal{F}}$ norm of the difference
above by

$\begin{split}&\mathbb{E}\left[\underset{0\leq s\leq
t}{\sup}\|X_{u}(s)-X_{h}(s)\|^{2}\right]\leq C\mathbb{E}\left[\underset{0\leq
s\leq
t}{\sup}\left\|\int_{0}^{s}\left[\sigma(r,X_{u}(r))-\sigma(r,X_{h}(r))\right]dW_{r}\right\|^{2}\right]\\\
&+C\mathbb{E}\left[\underset{0\leq s\leq
t}{\sup}\left\|\int_{0}^{s}\left[b(r,X_{u}(r),u(r))-b(r,X_{h}(r),h(r))\right]dr\right\|^{2}\right].\end{split}$

Using Cauchy-Schwartz in the deterministic integral and the Burkholder-Davis-
Gundy inequality in the stochastic integral, we get:

$\begin{split}&\mathbb{E}\left[\underset{0\leq s\leq
t}{\sup}\|X_{u}(s)-X_{h}(s)\|^{2}\right]\leq
C\mathbb{E}\left[\int_{0}^{t}\|\sigma(r,X_{u}(r))-\sigma(r,X_{h}(r))\|^{2}dr\right]\\\
&+C\mathbb{E}\left[\int_{0}^{t}\|b(r,X_{u}(r),u(r))-b(r,X_{h}(r),h(r))\|^{2}dr\right].\end{split}$

By using the lipschitzianity of the dynamics and taking the sup in the
integral, we get

$\begin{split}\mathbb{E}\left[\underset{0\leq s\leq
t}{\sup}\|X_{u}(s)-X_{h}(s)\|^{2}\right]&\leq
C\mathbb{E}\left[\int_{0}^{t}\underset{0\leq r\leq
s}{\sup}\|X_{u}(r)-X_{h}(r)\|^{2}ds\right]\\\
&+C\mathbb{E}\left[\int_{0}^{t}\|u(s)-h(s)\|^{2}ds\right].\end{split}$

We can write

$\begin{split}u(s)-h(s)&=v_{u}(s)-v_{h}(s)+K_{\lambda_{u}}(X_{u}(s))-K_{\lambda_{h}}(X_{u}(s))\\\
&+K_{\lambda_{h}}(X_{u}(s))-K_{\lambda_{h}}(X_{h}(s)),\end{split}$

and therefore have

$\begin{split}&\mathbb{E}\left[\underset{0\leq s\leq
t}{\sup}\|X_{u}(s)-X_{h}(s)\|^{2}\right]\leq
C\mathbb{E}\left[\int_{0}^{t}\underset{0\leq r\leq
s}{\sup}\|X_{u}(r)-X_{h}(r)\|^{2}ds\right]\\\
&+C\mathbb{E}\left[\int_{0}^{t}\|v_{u}(s)-v_{h}(s)\|^{2}ds\right]+C\mathbb{E}\left[\int_{0}^{t}\|K_{\lambda_{u}}(X_{u}(s))-K_{\lambda_{h}}(X_{u}(s))\|^{2}ds\right]\\\
&+C\mathbb{E}\left[\int_{0}^{t}\|K_{\lambda_{h}}(X_{u}(s))-K_{\lambda_{h}}(X_{h}(s))\|^{2}ds\right].\end{split}$

Using the assumptions on $K_{\lambda}(X)$ in (6), we obtain that

$\begin{split}&\mathbb{E}\left[\underset{0\leq s\leq
t}{\sup}\|X_{u}(s)-X_{h}(s)\|^{2}\right]\leq
C\mathbb{E}\left[\int_{0}^{t}\underset{0\leq r\leq
s}{\sup}\|X_{u}(r)-X_{h}(r)\|^{2}ds\right]\\\
&+C\int_{0}^{t}\|v_{u}(s)-v_{h}(s)\|^{2}ds+C\left(1+\mathbb{E}\left[\int_{0}^{t}\|X_{u}(s)\|^{2}ds\right]\right)\|\lambda_{u}-\lambda_{h}\|^{2}\\\
&+C(1+\|\lambda_{h}\|^{2})\mathbb{E}\left[\int_{0}^{t}\|X_{u}(s)-X_{h}(s)\|^{2}ds\right].\end{split}$

The dynamics of the SDE (3) are continuous in $u=(v,\lambda)$ and the solution
$X_{u}$ changes continuously with the dynamics [24], implying that
$\mathbb{E}\left[\int_{0}^{t}\|X_{u}(s)\|^{2}ds\right]$ is a continuous
function of $u$. We may therefore introduce $C_{u,h}$, an overloaded constant
that depends continuously on $u$ and $h$. From this remark, by taking the sup
in the integral above, we may infer that

$\begin{split}\mathbb{E}\left[\underset{0\leq s\leq
t}{\sup}\|X_{u}(s)-X_{h}(s)\|^{2}\right]\leq&C_{u,h}\mathbb{E}\left[\int_{0}^{t}\underset{0\leq
r\leq s}{\sup}\|X_{u}(r)-X_{h}(r)\|^{2}ds\right]\\\
&+C_{u,h}\left[\int_{0}^{t}\|v_{u}(s)-v_{h}(s)\|^{2}ds+\|\lambda_{u}-\lambda_{h}\|^{2}\right].\end{split}$

Finally, the Gronwall lemma applied to $t\mapsto\underset{0\leq s\leq
t}{\sup}\|X_{u}(s)-X_{h}(s)\|^{2}$ yields

$\mathbb{E}\left[\underset{0\leq s\leq
t}{\sup}\|X_{u}(s)-X_{h}(s)\|^{2}\right]\leq
C_{u,h}\|u(s)-h(s)\|_{\mathcal{U}}^{2}.$

∎

In what follows, we denote $y_{u,h}$ the unique element of
$L^{2}_{\mathcal{F}}(\Omega\times[0,T],\mathbf{R}^{n})$ that verifies the
linearized SDE

######  Lemma A.2.

Let $u=(v_{u},\lambda_{u})$ and $h=(v_{h},\lambda_{h})$ be two controls in
$\mathcal{U}$, let us consider the following linear SDE with stochastic
dynamics:

$\left\\{\begin{array}[]{ll}dY(t)&=[\partial_{x}b(t,X_{u}(t),u(t))+\partial_{u}b(t,X_{u}(t),u(t))K_{\lambda_{u}}^{\prime}(X_{u}(t))]Y(t)\\\
&+\partial_{u}b(t,X_{u}(t),u(t))[v_{h}(t)+\nabla_{\lambda}K_{\lambda_{u}}(X_{u}(t))\lambda_{h}]dt\\\
&+\partial_{x}\sigma(t,X_{u}(t))Y(t)dW_{t},\\\ Y(0)&=0.\end{array}\right.$

This equation has a unique solution denoted $y_{u,h}\in
L^{2}_{\mathcal{F}}(\Omega\times[0,T],\mathbf{R}^{n})$. This solution can be
written explicitly as

$\displaystyle
y_{u,h}(t)=\phi_{u}(t)\int_{0}^{t}\phi_{u}^{-1}(s)\partial_{u}b(s,X_{u}(s),u(s))[v_{h}(s)+\nabla_{\lambda}K_{\lambda}(X_{u}(s))\lambda_{h}]ds$

where $\phi_{u}$, defined in (15), is the resolvant of the linear SDE (15).
Consider $\epsilon$ a positive constant and denote $\delta
X_{u,\epsilon,h}\triangleq X_{u+\epsilon h}-X_{u}$. Then,

$\|\delta X_{u,\epsilon,h}-\epsilon y_{u,h}\|_{C^{2}_{\mathcal{F}}}\leq
C(\epsilon)\epsilon\|h\|_{L^{2}_{\mathcal{F}}}=o(\epsilon)$

where $C(\epsilon)>0$ is a real positive function of $\epsilon$ that tends to
zero as $\epsilon\to 0$.

###### Proof.

The process $Y$ is the solution of a linear equation of the form

$dY(t)=(A(t)Y(t)+r(t))dt+C(t)Y(t)dW_{t},$

where $A(t),C(t)$ and $r(t)$ are $\mathcal{F}_{t}$-adapted processes. If
$A(t)$ and $C(t)$ are $L^{\infty}_{\mathcal{F}}(\Omega\times[0,T])$, and
$r(t)$ is $L^{2}_{\mathcal{F}}(\Omega\times[0,T])$, then [44] ensures us that
there exists a unique strong solution that can be written explicitly. The
assumptions made on $K_{\lambda}(x)$ ensure us that

$\displaystyle\|K_{\lambda}^{\prime}(x)\|\leq C(1+\|\lambda\|),$
$\displaystyle\|\nabla_{\lambda}K_{\lambda}(x)\|\leq C(1+\|x\|),$

which in turn implies

$\displaystyle\|A\|_{\infty}$
$\displaystyle\leq\|\partial_{x}b\|_{\infty}+C\|\partial_{u}b\|_{\infty}(1+\|\lambda_{u}\|)$
$\displaystyle\|C\|_{\infty}$
$\displaystyle\leq\|\partial_{x}\sigma\|_{\infty}$
$\displaystyle\|r\|_{L^{2}}$
$\displaystyle\leq\|\partial_{u}b\|_{\infty}(\|v_{h}\|_{L^{2}}+C(1+\|X_{u}\|_{C^{2}_{\mathcal{F}}})\|\lambda_{h}\|)$

The solution therefore exists and can be written explicitly.

As for the approximation, the functions $\delta X_{u,\epsilon,h}$ and
$y_{u,h}$ follow the same SDE up to a $o(\epsilon)$ term. Therefore, the proof
follows the exact same steps as the proof of Lemma A.1, which we then avoid
detailing here. ∎

We can now move to the proof of Lemma 3.1. To prove differentiability, we
start by estimating the difference

$\displaystyle\delta J_{u,\epsilon,h}$ $\displaystyle=J(u+\epsilon h)-J(u)$
$\displaystyle=\int_{0}^{T}[f(t,X_{u+\epsilon h}(t),v_{u}(t)+\epsilon
v_{h}(t)+K_{\lambda_{u}+\epsilon\lambda_{h}}(X_{u+\epsilon h}(t)))$
$\displaystyle-f(t,X_{u}(t),v_{u}(t)+K_{\lambda_{u}}(X_{u}(t)))]dt+g(T,X_{u+\epsilon
h}(T))-g(T,X_{u}(T))$
$\displaystyle=\int_{0}^{T}\partial_{x}f(t,X_{u}(t),u(t))\delta
X_{u,\epsilon,h}(t)dt$
$\displaystyle+\int_{0}^{T}\partial_{u}f(t,X_{u}(t),u(t))[\epsilon
v_{h}(t)+K_{\lambda_{u}+\epsilon\lambda_{h}}(X_{u+\epsilon
h}(t))-K_{\lambda_{u}}(X_{u}(t))]dt$
$\displaystyle+\partial_{x}g(T,X_{u}(T))\delta
X_{u,\epsilon,h}(T)+o(\epsilon)+o(\delta X_{u,\epsilon,h}).$

where $\delta X_{u,\epsilon,h}\triangleq X_{u+\epsilon h}-X_{u}$. Due to Lemma
A.1, we have

$\|X_{u+\epsilon h}-X_{u}\|^{2}_{C^{2}_{\mathcal{F}}}\leq
C\epsilon^{2}\|h\|^{2}_{\mathcal{U}},$

where $C$ is a positive constant, meaning that $o(\delta
X_{u,\epsilon,h})=o(\epsilon)$. Using this latter fact, and the assumption
that $(\lambda,x)\mapsto K_{\lambda}(x)$ has lipschitz gradient, meaning it
verifies (7), we may use the Taylor expansion of $K_{\lambda}(x)$ and write

$K_{\lambda_{u}+\epsilon\lambda_{h}}(X_{u+\epsilon
h}(t))-K_{\lambda_{u}}(X_{u}(t))=K_{\lambda_{u}}^{\prime}(X_{u}(t))\delta
X_{u,\epsilon,h}(t)+\nabla_{\lambda}K_{\lambda_{u}}(X_{u}(t))\epsilon\lambda_{h}+o(\epsilon),$

and then finally compute

$\displaystyle\delta J_{u,\epsilon,h}$
$\displaystyle=\int_{0}^{T}[\partial_{x}f(t,X_{u}(t),u(t))+\partial_{u}f(t,X_{u}(t),u(t))K_{\lambda}^{\prime}(X_{u}(t))]\delta
X_{u,\epsilon,h}(t)dt$
$\displaystyle+\int_{0}^{T}\partial_{u}f(t,X_{u}(t),u(t))\epsilon[v_{h}(t)+\nabla_{\lambda}K_{\lambda}(X_{u}(t))\lambda_{h}]dt$
$\displaystyle+\partial_{x}g(T,X_{u}(T))\delta
X_{u,\epsilon,h}(T)+o(\epsilon).$

We now need an estimate for $\delta X_{u,\epsilon,h}$ that is linear in
$\epsilon h$. In order to obtain this latter, we first notice that $\delta
X_{u,\epsilon,h}$ solves the following SDE

$\begin{split}d\delta
X_{u,\epsilon,h}(t)&=[\partial_{x}b(t,X_{u}(t),u(t))+\partial_{u}b(t,X_{u}(t),u(t))K_{\lambda}^{\prime}(X_{u}(t))]\delta
X_{u,\epsilon,h}(t)dt\\\
&+\partial_{u}b(t,X_{u}(t),u(t))\epsilon[v_{h}(t)+\nabla_{\lambda}K_{\lambda}(X_{u}(t))\lambda_{h}]dt+o(\epsilon)dt\\\
&+\left[\partial_{x}\sigma(t,X_{u}(t))\delta
X_{u,\epsilon,h}(t)+o(\epsilon)\right]dW_{t}.\end{split}$

Lemma A.2 yields that the solution of the linearized SDE is indeed a good
approximation. The gradients of $f$ and $g$ are bounded, we can therefore
replace $\delta X_{u,\epsilon,h}$ by the new estimate $\epsilon y_{u,h}$ and
obtain

$\displaystyle\delta J_{u,\epsilon,h}$
$\displaystyle=\int_{0}^{T}[\partial_{x}f(t,X_{u}(t),u(t))+\partial_{u}f(t,X_{u}(t),u(t))K_{\lambda}^{\prime}(X_{u}(t))]\epsilon
y_{u,h}(t)dt$
$\displaystyle+\int_{0}^{T}\partial_{u}f(t,X_{u}(t),u(t))\epsilon[v_{h}(t)+\nabla_{\lambda}K_{\lambda}(X_{u}(t))\lambda_{h}]dt$
$\displaystyle+\partial_{x}g(T,X_{u}(T))\epsilon y_{u,h}(T)+o(\epsilon).$

By injecting the explicit formula of $y_{u,h}$ and inverting the integral
signs, we finally obtain

$\begin{split}\delta
J_{u,\epsilon,h}&=\epsilon\int_{0}^{T}I_{u}(s)\phi_{u}^{-1}(s)\partial_{u}b(s,X_{u}(s),u(s))[v_{h}(s)+\nabla_{\lambda}K_{\lambda}(X_{u}(s))\lambda_{h}]ds\\\
&+o(\epsilon),\end{split}$ (22)

where $I_{u}$ is the term

$\begin{split}I_{u}(s)&\triangleq\partial_{x}g(T,X_{u}(T))\phi_{u}(T)\\\
&+\int_{s}^{T}[\partial_{x}f(t,X_{u}(t),u(t))+\partial_{u}f(t,X_{u}(t),u(t))K_{\lambda}^{\prime}(X_{u}(t))]\phi_{u}(t)dt.\end{split}$

The function $u=(v,\lambda)\mapsto J$ is therefore Fréchet differentiable and
we can compute the Riesz representation of its gradient, denoted $\nabla
J(u)$. We recall that $\nabla J(u)=(\nabla_{v}J(u),\nabla_{\lambda}J(u))$ is
the random variable such that

$\underset{\epsilon\rightarrow 0}{\lim}\frac{\delta
J_{u,\epsilon,h}(\omega)}{\epsilon}=\langle\nabla
J(u)(\omega),h\rangle_{\mathcal{U}}\quad\forall h\in\mathcal{U}.$

From the expression of $\delta J_{u,\epsilon,h}$ in (22) we can directly
identify the gradient as

$\displaystyle\nabla_{v}J(v,\lambda)(t)=I_{u}(t)\phi_{u}^{-1}(t)\partial_{u}b(t,X_{u}(t),u(t))$
$\displaystyle\nabla_{\lambda}J(v,\lambda)=\int_{0}^{T}I_{u}(s)\phi_{u}^{-1}(s)\partial_{u}b(s,X_{u}(s),u(s))\nabla_{\lambda}K_{\lambda}(X_{u}(s))ds.$

###  A.2 Proof of Proposition 3.2

Let $\rho$ be a finite coherent risk measure, $\mathcal{U}$ and
$\partial\rho(0)$ are both convex subsets of a Hilbert space. As shown in 3.1,
$u\mapsto J(u)$ is also Frechet differentiable. Additionally, the risk
function is Hadamard differentiable (H-differentiable) as stated in Theorem
2.2, and thus $u\mapsto\rho(J(u))$ is H-differentiable. From [45, Chapter
3.1], we obtain the following optimality condition for H-differentiable
functions:

######  Proposition A.3.

Let $X$ be a convex subset of a Banach space and $f:X\mapsto\mathbf{R}$ an
H-differentiable function. If $f$ has a minimum on $X$ reached at $x^{*}$, it
is necessary for $x^{*}$ to verify:

$f^{\prime}(x^{*})\cdot(x-x^{*})\geq 0,\quad\forall x\in X.$

Let $u^{*}$ be a solution to Problem (8). Therefore, the previous proposition
yields:

$-\rho^{\prime}(J(u^{*}))\cdot D_{u}J(u^{*})\cdot(u-u^{*})\leq 0,\quad\forall
u\in\mathcal{U}.$

In particular, from the formula of the differential of $\rho$ given in Theorem
2.2 and from the chain rule for directional differentials, we obtain that:

$\underset{\zeta\in\partial\rho(J(u^{*}))}{\inf}\ \mathbb{E}\left[-\zeta\
D_{u}J(u^{*})\cdot(u-u^{*})\right]\leq 0,\quad\forall u\in\mathcal{U}.$

By using the Riesz representation of $D_{u}J(u^{*})$ and using equation (10)
we get

$\underset{\zeta\in\partial\rho(J(u^{*}))}{\inf}\
\langle\mathbb{E}\left[-\zeta\ \nabla
J(u^{*})\right],u-u^{*}\rangle_{\mathcal{U}}\leq 0,\quad\forall
u\in\mathcal{U}.$ (23)

By taking the supremum of (23) over $u\in\mathcal{U}$, we infer that

$\underset{u\in\mathcal{U}}{\sup}\underset{\zeta\in\partial\rho(J(u^{*}))}{\inf}\langle\mathbb{E}\left[-\zeta\
\nabla J(u^{*})\right],u-u^{*}\rangle_{\mathcal{U}}\leq 0.$ (24)

Now, we need to invert the sup with the inf in the above expression, and for
this, we use the following version of Sion min-max theorem [46]:

######  Theorem A.4.

Let $X$ and $Y$ be two convex subsets of Haussdorff topological spaces with X
being compact, and consider a continuous map $f$ such that

  * 1.

$x\mapsto f(x,y)$ is convex in $X$ for all $y\in Y$,

  * 2.

$y\mapsto f(x,y)$ is concave in $Y$ for all $x\in X$.

Then, it holds that

$\underset{y\in Y}{\sup}\ \underset{x\in X}{\inf}\ f(x,y)=\underset{x\in
X}{\inf}\ \underset{y\in Y}{\sup}\ f(x,y).$

In our setting, we select $Y=\mathcal{U}$, which is convex, and
$X=\partial\rho(J(u^{*}))$, which is convex, weakly-* compact and therefore
weakly compact since it is a subset of a Hilbert space. Also, by definition,
the mapping

$(\zeta,u)\mapsto\langle\mathbb{E}\left[-\zeta\ \nabla
J(u^{*})\right],u-u^{*}\rangle_{\mathcal{U}}$ (25)

is weakly continuous and concave in $u$, as well as weakly continuous and
convex in $\zeta$. Therefore, Theorem A.4 can be applied to equation (24),
yielding

$\underset{\zeta\in\partial\rho(J(u^{*}))}{\inf}\underset{u\in\mathcal{U}}{\sup}\
\langle\mathbb{E}\left[-\zeta\ \nabla
J(u^{*})\right],u-u^{*}\rangle_{\mathcal{U}}\leq 0.$ (26)

Since the supremum of a family of lower semi-continuous functions is also
lower semi-continuous, the mapping

$\zeta\mapsto\underset{u\in\mathcal{U}}{\sup}\ \langle\mathbb{E}\left[-\zeta\
\nabla J(u^{*})\right],u-u^{*}\rangle_{\mathcal{U}}$

is lower semi-continuous, and therefore it attains a minimum which we denote
by $\zeta^{*}\in\partial\rho(J(u^{*}))$. Thanks to this latter remark, from
(26) we infer that

$\langle\mathbb{E}\left[\zeta^{*}\ \nabla
J(u^{*})\right],u-u^{*}\rangle_{\mathcal{U}}\geq 0,\quad\forall
u\in\mathcal{U},$

and the conclusion follows from the fact that $\mathcal{U}$ is a vector space.

###  A.3 Proof of Lemma 5.4 (compacity of the output of a control affine
system)

We recall we are under the hypothesis of Setting B.
Proof of the compacity of $u\mapsto X_{u}$: Let $u_{n}$ be a sequence of
controls in $\mathcal{U}$ weakly converging to $\overline{u}$. Let $p\geq 2$,
we prove below the more general convergence

$\left\|X_{u_{n}}-X_{\overline{u}}\right\|^{p}_{C^{p}_{\mathcal{F}}}\rightarrow
0$

The theorem of existence of strong solutions of SDEs in [24] ensures us that,
since our initial condition is deterministic, $X_{u}\in C^{p}_{\mathcal{F}}$.
Below, we denote by $C>0$ a constant that will be implicitly overloaded
throughout the proof. By definition, for every $n\in\mathbb{N}$ we may compute

$\begin{split}&\left\|X_{u_{n}}-X_{\overline{u}}\right\|^{p}_{C^{p}_{\mathcal{F}}}\leq
C\mathbb{E}\left[\underset{0\leq r\leq
t}{\sup}\left\|\int_{0}^{r}(b_{0}(s,X_{u_{n}}(s))-b_{0}(s,X_{\overline{u}}(s)))ds\right\|^{p}\right]\\\
&+C\mathbb{E}\left[\underset{0\leq r\leq
t}{\sup}\left\|\int_{0}^{r}(\sigma(s,X_{u_{n}}(s))-\sigma(s,X_{\overline{u}}(s)))dW_{s}\right\|^{p}\right]\\\
&+C\mathbb{E}\left[\underset{0\leq r\leq
t}{\sup}\left\|\int_{0}^{r}\sum_{i=1}^{m}\left[(v_{n})^{(i)}(s)-(\overline{v})^{(i)}(s)\right]b_{i}(s,X_{\overline{u}}(s))ds\right\|^{p}\right]\\\
&+C\mathbb{E}\left[\underset{0\leq r\leq
t}{\sup}\left\|\int_{0}^{r}\sum_{i=1}^{m}\left[K_{\lambda_{n}}(X_{u_{n}}(s))^{(i)}-K_{\overline{\lambda}}(X_{\overline{u}}(s))^{(i)}\right]b_{i}(s,X_{\overline{u}}(s))ds\right\|^{p}\right]\\\
&+C\mathbb{E}\left[\underset{0\leq r\leq
t}{\sup}\left\|\int_{0}^{r}\sum_{i=1}^{m}u_{n}^{(i)}(s)(b_{i}(s,X_{u_{n}}(s))-b_{i}(s,X_{\overline{u}}(s)))ds\right\|^{p}\right].\end{split}$

Let us denote

$h_{n}(t,\omega)\triangleq\int_{0}^{t}\sum_{i=1}^{m}\left[(v_{n})^{(i)}(s)-(\overline{v})^{(i)}(s)\right]b_{i}(s,X_{\overline{u}}(s,\omega))ds.$

We now appropriately bound the different integral terms. Hölder’s inequality
yields

$\begin{split}\underset{0\leq r\leq
t}{\sup}&\left\|\int_{0}^{r}(b_{0}(s,X_{u_{n}}(s))-b_{0}(s,X_{\overline{u}}(s)))ds\right\|^{p}\leq\\\
&\leq
C\int_{0}^{t}\left\|b_{0}(s,X_{u_{n}}(s))-b_{0}(s,X_{\overline{u}}(s))\right\|^{p}ds\\\
&\leq C\int_{0}^{t}\left\|X_{u_{n}}(s)-X_{\overline{u}}(s)\right\|^{p}ds\leq
C\int_{0}^{t}\underset{0\leq s^{\prime}\leq
s}{\sup}\left\|X_{u_{n}}(s^{\prime})-X_{\overline{u}}(s^{\prime})\right\|^{p}ds.\end{split}$

Using that the functions $b_{i}$ are bounded and the assumptions on the
feedback (6) paired with Hölder’s inequality, we have that

$\displaystyle\mathbb{E}\left[\underset{0\leq r\leq
t}{\sup}\left\|\int_{0}^{r}\sum_{i=1}^{m}\left[K_{\lambda_{n}}(X_{u_{n}}(s))^{(i)}-K_{\overline{\lambda}}(X_{\overline{u}}(s))^{(i)}\right]b_{i}(s,X_{\overline{u}}(s))ds\right\|^{p}\right]$
$\displaystyle\leq
C\mathbb{E}\left[\int_{0}^{t}\sum_{i=1}^{m}\|K_{\lambda_{n}}(X_{u_{n}}(s))^{(i)}-K_{\lambda_{n}}(X_{\overline{u}}(s))^{(i)}\|^{p}\|b_{i}(s,X_{\overline{u}}(s))\|^{p}ds\right]$
$\displaystyle+C\mathbb{E}\left[\int_{0}^{t}\sum_{i=1}^{m}\|K_{\lambda_{n}}(X_{\overline{u}}(s))^{(i)}-K_{\overline{\lambda}}(X_{\overline{u}}(s))^{(i)}\|^{p}\|b_{i}(s,X_{\overline{u}}(s))\|^{p}ds\right]$
$\displaystyle\leq
C\mathbb{E}\left[\int_{0}^{t}(1+\|\lambda_{n}\|^{p})\|X_{u_{n}}(s)-X_{\overline{u}}(s)\|^{p}ds\right]$
$\displaystyle+C\mathbb{E}\left[\int_{0}^{t}(1+\|X_{\overline{u}}(s)\|^{p})\|\lambda_{n}-\overline{\lambda}\|^{p}ds\right].$

Since $\lambda_{n}$ converges, it is bounded. Using the fact that
$X_{\overline{u}}$ is in $C^{p}_{\mathcal{F}}$, by taking the sup we finally
have that

$\displaystyle\mathbb{E}\left[\underset{0\leq r\leq
t}{\sup}\left\|\int_{0}^{r}\sum_{i=1}^{m}\left[K_{\lambda_{n}}(X_{u_{n}}(s))^{(i)}-K_{\overline{\lambda}}(X_{\overline{u}}(s))^{(i)}\right]b_{i}(s,X_{\overline{u}}(s))ds\right\|^{p}\right]$
$\displaystyle\leq C\mathbb{E}\left[\int_{0}^{t}\underset{0\leq s^{\prime}\leq
s}{\sup}\|X_{u_{n}}(s^{\prime})-X_{\overline{u}}(s^{\prime})\|^{p}ds\right]+C\|\lambda_{n}-\overline{\lambda}\|^{p}.$

For the stochastic integral, thanks to Burkholder-Davis-Gundy inequality (1)
combined with Hölder we may compute

$\begin{split}&\mathbb{E}\left[\underset{0\leq r\leq
t}{\sup}\left\|\int_{0}^{r}(\sigma(s,X_{u_{n}}(s))-\sigma(s,X_{\overline{u}}(s)))dW_{s}\right\|^{p}\right]\leq\\\
&\leq
C\mathbb{E}\left[\int_{0}^{t}\left\|\sigma(s,X_{u_{n}}(s))-\sigma(s,X_{\overline{u}}(s))\right\|^{2}ds\right]^{\frac{p}{2}}\\\
&\leq
C\mathbb{E}\left[\int_{0}^{t}\left\|X_{u_{n}}(s)-X_{\overline{u}}(s)\right\|^{p}ds\right]\leq
C\int_{0}^{t}\mathbb{E}\left[\underset{0\leq s^{\prime}\leq
s}{\sup}\left\|X_{u_{n}}(s^{\prime})-X_{\overline{u}}(s^{\prime})\right\|^{p}\right]ds.\end{split}$

Gathering all the previous bounds yields

$\begin{split}\mathbb{E}\left[\underset{0\leq s\leq
t}{\sup}\left\|X_{u_{n}}(s)-X_{\overline{u}}(s)\right\|^{p}\right]&\leq
C\int_{0}^{t}\mathbb{E}\left[\underset{0\leq s^{\prime}\leq
s}{\sup}\left\|X_{u_{n}}(s^{\prime})-X_{\overline{u}}(s^{\prime})\right\|^{p}\right]ds\\\
&+C\left\|\lambda_{n}-\overline{\lambda}\right\|^{p}+C\mathbb{E}\left[\underset{0\leq
s\leq t}{\sup}\|h_{n}(s,\omega)\|^{p}\right],\end{split}$

and an application of Grönwall inequality allows us to infer that

$\mathbb{E}\left[\underset{0\leq s\leq
t}{\textrm{sup}}\left\|X_{u_{n}}(s)-X_{\overline{u}}(s)\right\|^{p}\right]\leq
C\left[\mathbb{E}\left[\underset{0\leq s\leq
t}{\sup}\|h_{n}(s,\omega)\|^{p}\right]+\left\|\lambda_{n}-\overline{\lambda}\right\|^{p}\right].$

To conclude, we now need to prove that $\mathbb{E}\left[\underset{0\leq s\leq
t}{\sup}\|h_{n}(s,\omega)\|^{p}\right]$ tends to 0. The following lemma gives
us the convergence of $\underset{0\leq s\leq
t}{\textrm{sup}}\|h_{n}(s,\omega)\|^{p}$ for $\omega$ fixed in $\Omega$ (see
[25] for proof).

######  Lemma A.5.

Let $a,b\in\mathbf{R}$ and let $E$ be a normed vector space. For all
$n\in\mathbf{N}$, let $f_{n}:[a,b]\to E$ be uniformly $\alpha$-Hölder, that is

$\exists\alpha,K>0,\quad\forall n\in\mathbf{N},\quad\forall
x,y\in[a,b]\quad\|f_{n}(x)-f_{n}(y)\|\leq K\|x-y\|^{\alpha}$ (27)

If the sequence $f_{n}$ converges simply to an application $f$, then it
converges uniformly.

For $\omega\in\Omega$, $h_{n}(s,\omega)$ converges to 0 by weak convergence of
$u_{n}$. Let us recall that the functions $b_{i}$ are bounded by assumption.
We can now show that the sequence $h_{n}(\cdot,\omega)$ is uniformly Hölder
thanks to Cauchy Schwartz and to the fact that $v_{n}$ is uniformly bounded in
$L^{2}(0,T)$ :

$\begin{split}|h_{n}(t_{2},\omega)-&h_{n}(t_{1},\omega)|=\left|\int_{t_{1}}^{t_{2}}\sum_{i=1}^{m}\left[(v_{n})^{(i)}(s)-(\overline{v})^{(i)}(s)\right]b_{i}(s,X_{\overline{u}}(s,\omega))ds\right|\\\
&\leq
C\sum_{i=1}^{m}\left[\left(\int_{t_{1}}^{t_{2}}(|(v_{n})^{(i)}(s)|^{2}+|(\overline{v})^{(i)}(s)|^{2})ds\right)^{\frac{1}{2}}\left(\int_{t_{1}}^{t_{2}}\|b_{i}\|_{\infty}ds\right)^{\frac{1}{2}}\right]\hskip
30.00005pt\\\ &\leq C|t_{2}-t_{1}|^{\frac{1}{2}}\hskip 30.00005pt\end{split}$

Using Lemma A.5, we have that sequence of random variables
$g_{n}(\omega):=\underset{0\leq s\leq t}{\sup}\|h_{n}(s,\omega)\|^{p}$
converges pointwise to 0. The random variable $g_{n}$ can be uniformly bounded
by the deterministic value

$\begin{split}|g_{n}(\omega)|&\leq\left(Tm\int_{0}^{T}\sum_{i=1}^{m}(M_{u}+\|(\overline{v})^{(i)}(s)\|^{2})\|b_{i}\|^{2}_{\infty}\right)^{\frac{p}{2}}\end{split}$

with $M_{u}$ a uniform bound of the sequence of controls. We therefore have by
dominated convergence that $\mathbb{E}[g_{n}]$ tends to 0 , which in turn
gives

$\begin{split}\|X_{u_{n}}-X_{\overline{u}}\|_{C_{\mathcal{F}}^{p}}^{p}\rightarrow
0.\end{split}$

Proof of the compacity of $u\mapsto\nabla_{u}J_{X}(X_{u})$:

For more clarity, let us consider a cost where $g=0$. Let $u$ be a control in
$\mathcal{U}$, we can write the gradient as

$\nabla_{u}J_{X}(X_{u})^{(i)}(t)=\left[\int_{t}^{T}\partial_{x}f(s,X_{u}(s))\phi_{u}(s)ds\right]\phi_{u}^{-1}(t)\partial_{x}b_{i}(t,X_{u}(t))$

with $\phi_{u}$ satisfying for $t\in[0,T]$ the linear system

$\begin{split}\left\\{\begin{array}[]{ll}d\phi_{u}(t)&=\left[\partial_{x}b_{0}(t,X_{u}(t))+\sum_{i=1}^{m}u_{i}(t)\partial_{x}b_{i}(t,X_{u}(t))\right]\phi_{u}(t)dt\\\
&\quad+\left[\sum_{i=1}^{m}b_{i}(t,X_{u}(t))\right]K_{\lambda_{u}}^{\prime}(X_{u}(t))\phi_{u}(t)dt\\\
&\quad+\partial_{x}\sigma(t,X_{u}(t))\phi_{u}(t)dW_{t},\\\
\phi(0)&=Id_{N}.\end{array}\right.\end{split}$ (28)

Since functions $b_{i},\partial_{x}b_{i},\partial_{x}\sigma_{i}$ and
$K_{\lambda}^{\prime}$ are all uniformly bounded, we can prove in a similar
way as previously the compacity of $u\mapsto\phi_{u}$ in
$C_{\mathcal{F}}^{p}$. We can also prove the compacity of
$u\mapsto\phi^{-1}_{u}$ as it satisfies a similar SDE.

We then conclude using the Cauchy-Schwartz inequality twice on the expectation
of the four terms and obtaining

$\begin{split}\mathbb{E}&\left[\underset{0\leq t\leq
T}{\sup}\|\nabla_{u}J_{X}(X_{u})^{(i)}(t)-\nabla_{u}J_{X}(X_{u_{n}})^{(i)}(t)\|^{2}\right]\leq\\\
&\
C\left(\int_{0}^{T}\mathbb{E}\left[\|\partial_{x}f(s,X_{\overline{u}}(s))-\partial_{x}f(s,X_{u_{n}}(s))\|^{8}\right]ds\right)\|\phi_{\overline{u}}\|_{C_{\mathcal{F}}^{8}}\|\phi_{\overline{u}}^{-1}\|_{C_{\mathcal{F}}^{8}}\|\partial_{x}b_{i}\|^{8}_{\infty}\\\
&+C\|\partial_{x}f(\cdot,X_{u_{n}}(\cdot))\|_{C_{\mathcal{F}}^{8}}\|\phi_{\overline{u}}-\phi_{u_{n}}\|_{C_{\mathcal{F}}^{8}}\|\phi_{\overline{u}}^{-1}\|_{C_{\mathcal{F}}^{8}}\|\partial_{x}b_{i}\|^{8}_{\infty}\\\
&+C\|\partial_{x}f(\cdot,X_{u_{n}}(\cdot))\|_{C_{\mathcal{F}}^{8}}\|\phi_{u_{n}}\|_{C_{\mathcal{F}}^{8}}\|\phi_{\overline{u}}^{-1}-\phi_{u_{n}}^{-1}\|_{C_{\mathcal{F}}^{8}}\|\partial_{x}b_{i}\|^{8}_{\infty}\\\
&+C\|\partial_{x}f(\cdot,X_{u_{n}}(\cdot))\|_{C_{\mathcal{F}}^{8}}\|\phi_{u_{n}}\|_{C_{\mathcal{F}}^{8}}\|\phi_{u_{n}}^{-1}\|_{C_{\mathcal{F}}^{8}}\|\partial_{x}b_{i}(\cdot,X_{\overline{u}}(\cdot))-\partial_{x}b_{i}(\cdot,X_{u_{n}}(\cdot))\|_{C_{\mathcal{F}}^{8}}\end{split}$

The lipschitzianity of $\partial_{x}f$ ensures the convergence of the first
term to 0 and implies that

$\|\partial_{x}f(s,x)\|\leq C(1+\|x\|)$

which in turn yields

$\|\partial_{x}f(\cdot,X_{u_{n}}(\cdot))\|_{C_{\mathcal{F}}^{8}}\leq
C(1+\|X_{u_{n}}\|_{C_{\mathcal{F}}^{8}}),$

which is bounded thanks to the convergence of the sequence $X_{u_{n}}$.
Finally, we conclude with the lipschitzianity of $\partial_{x}b_{i}$ which
implies that the last term tends to 0. We therefore have

$\|\nabla_{u}(J_{X}(X_{u_{n}}))-\nabla_{u}(J_{X}(X_{\overline{u}}))\|^{2}_{C^{2}_{\mathcal{F}}}\rightarrow
0.$

## References

  * [1] A. Shapiro, D. Dentcheva, and A. Ruszczyński, Lectures on Stochastic Programming: Modeling and Theory.  Society for Industrial and Applied Mathematics, Jan. 2009.
  * [2] M. P. Chapman, R. Bonalli, K. M. Smith, I. Yang, M. Pavone, and C. J. Tomlin, “Risk-Sensitive Safety Analysis Using Conditional Value-at-Risk,” IEEE Transactions on Automatic Control, vol. 67, pp. 6521–6536, Dec. 2022.  Conference Name: IEEE Transactions on Automatic Control.
  * [3] Y. Wang and M. P. Chapman, “Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal control,” Artificial Intelligence, vol. 311, p. 103743, Oct. 2022.
  * [4] M. V. Kothare, V. Balakrishnan, and M. Morari, “Robust constrained model predictive control using linear matrix inequalities,” Automatica, vol. 32, pp. 1361–1379, Oct. 1996.
  * [5] Y.-L. Chow and M. Pavone, “A framework for time-consistent, risk-averse model predictive control: Theory and algorithms,” in 2014 American Control Conference, (Portland, OR, USA), pp. 4204–4211, IEEE, June 2014\.
  * [6] P. Sopasakis, M. Schuurmans, and P. Patrinos, “Risk-averse risk-constrained optimal control,” in 2019 18th European Control Conference (ECC), (Naples, Italy), pp. 375–380, IEEE, June 2019.
  * [7] A. Ruszczyński and J. Yao, “A Risk-Averse Analog of the Hamilton–Jacobi–Bellman Equation,” in 2015 Proceedings of the Conference on Control and its Applications (C. Bonnet, B. Pasik-Duncan, H. Ozbay, and Q. Zhang, eds.), pp. 462–468, Philadelphia, PA: Society for Industrial and Applied Mathematics, July 2015.
  * [8] R. Bonalli and B. Bonnet, “First-Order Pontryagin Maximum Principle for Risk-Averse Stochastic Optimal Control Problems,” May 2023.  arXiv:2204.03036 [math], accepted in SICON, to appear shortly.
  * [9] A. Shapiro, “On a time consistency concept in risk averse multistage stochastic programming,” Operations Research Letters, vol. 37, pp. 143–147, May 2009.
  * [10] A. Ruszczyński, “Risk-averse dynamic programming for Markov decision processes,” Mathematical Programming, vol. 125, pp. 235–261, Oct. 2010\.
  * [11] O. Çavuş and A. Ruszczyński, “Risk-Averse Control of Undiscounted Transient Markov Models,” SIAM Journal on Control and Optimization, vol. 52, pp. 3935–3966, Jan. 2014.  Publisher: Society for Industrial and Applied Mathematics.
  * [12] N. Bäuerle and J. Ott, “Markov Decision Processes with Average-Value-at-Risk criteria,” Mathematical Methods of Operations Research, vol. 74, pp. 361–379, Dec. 2011.
  * [13] C. W. Miller and I. Yang, “Optimal Control of Conditional Value-at-Risk in Continuous Time,” SIAM Journal on Control and Optimization, vol. 55, pp. 856–884, Jan. 2017.
  * [14] L. Pfeiffer, “Risk-averse Merton’s Portfolio Problem,” IFAC-PapersOnLine, vol. 49, no. 8, pp. 266–271, 2016.
  * [15] P. Laurent, “Optimality conditions in variational form for non-linear constrained stochastic control problems,” Mathematical Control and Related Fields, vol. 10, pp. 493–526, Aug. 2020.  Publisher: Mathematical Control and Related Fields.
  * [16] R. Archibald, “A Stochastic Gradient Descent Approach for Stochastic Optimal Control,” East Asian Journal on Applied Mathematics, vol. 10, pp. 635–658, June 2020.
  * [17] K. Konolige, “A gradient method for realtime robot control,” in Proceedings. 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2000) (Cat. No.00CH37113), vol. 1, (Takamatsu, Japan), pp. 639–646, IEEE, 2000.
  * [18] S. Garreis, T. M. Surowiec, and M. Ulbrich, “An Interior-Point Approach for Solving Risk-Averse PDE-Constrained Optimization Problems with Coherent Risk Measures,” SIAM Journal on Optimization, vol. 31, pp. 1–29, Jan. 2021.  Publisher: Society for Industrial and Applied Mathematics.
  * [19] A. Nedić and A. Ozdaglar, “Subgradient Methods for Saddle-Point Problems,” Journal of Optimization Theory and Applications, vol. 142, pp. 205–228, July 2009.
  * [20] A. Mokhtari, A. Ozdaglar, and S. Pattathil, “A Unified Analysis of Extra-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approach,” in Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, pp. 1497–1507, PMLR, June 2020.  ISSN: 2640-3498.
  * [21] T. Lin, C. Jin, and M. I. Jordan, “On gradient descent ascent for nonconvex-concave minimax problems,” in Proceedings of the 37th International Conference on Machine Learning, vol. 119 of ICML’20, pp. 6083–6093, JMLR.org, July 2020.
  * [22] Z. Chen, Y. Zhou, T. Xu, and Y. Liang, “Proximal Gradient Descent-Ascent: Variable Convergence under KŁ Geometry,” Jan. 2021\.
  * [23] J.-F. Le Gall, Brownian Motion, Martingales, and Stochastic Calculus, vol. 274 of Graduate Texts in Mathematics.  Cham: Springer International Publishing, 2016.
  * [24] J. Yong and X. Y. Zhou, Stochastic controls: Hamiltonian systems and HJB equations.  No. 43 in Applications of mathematics, New York: Springer, 1999.
  * [25] E. Trélat, “Contrôle optimal : théorie & applications,” HAL, vol. 2005, no. 0, 2005.
  * [26] R. Bonalli, B. Hérissé, and E. Trélat, “Solving optimal control problems for delayed control-affine systems with quadratic cost by numerical continuation,” in 2017 American Control Conference (ACC), pp. 649–654, May 2017.  ISSN: 2378-5861.
  * [27] R. Bonalli, B. Herisse, and E. Trelat, “Optimal Control of Endoatmospheric Launch Vehicle Systems: Geometric and Computational Issues,” IEEE Transactions on Automatic Control, vol. 65, pp. 2418–2433, June 2020\.
  * [28] R. Bonalli, Optimal Control of Aerospace Systems with Control-State Constraints and Delays.  phdthesis, Sorbonne Université, UPMC University of Paris 6, Laboratoire Jacques-Louis Lions ; ONERA – The French Aerospace Lab, Département TIS, Unité NGPA ; Inria Paris, Equipe CAGE, July 2018.
  * [29] R. Bonalli, T. Lew, and M. Pavone, “Sequential convex programming for non-linear stochastic optimal control,” ESAIM: Control, Optimisation and Calculus of Variations, vol. 28, p. 64, 2022.
  * [30] M. Benzi, G. H. Golub, and J. Liesen, “Numerical solution of saddle point problems,” Acta Numerica, vol. 14, pp. 1–137, May 2005.
  * [31] D. P. Kouri and T. M. Surowiec, “Existence and Optimality Conditions for Risk-Averse PDE-Constrained Optimization,” SIAM/ASA Journal on Uncertainty Quantification, vol. 6, pp. 787–815, Jan. 2018.  Publisher: Society for Industrial and Applied Mathematics.
  * [32] D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” Jan. 2017.  arXiv:1412.6980 [cs].
  * [33] P. E. Kloeden and E. Platen, Numerical Solution of Stochastic Differential Equations.  Berlin, Heidelberg: Springer, 1992.
  * [34] P. Bernhard and A. Rapaport, “On a theorem of Danskin with an application to a theorem of Von Neumann-Sion,” Nonlinear Analysis: Theory, Methods & Applications, vol. 24, pp. 1163–1181, Apr. 1995.
  * [35] T. Smith, J. Barlow, M. Bualat, T. Fong, C. Provencher, H. Sanchez, and E. Smith, “Astrobee: A New Platform for Free-Flying Robotics on the International Space Station,” (Beijing), June 2016.  NTRS Author Affiliations: NASA Ames Research Center, SGT, Inc., Universities Space Research Association NTRS Report/Patent Number: ARC-E-DAA-TN31584 NTRS Document ID: 20160007769 NTRS Research Center: Ames Research Center (ARC).
  * [36] L. Siurna, G. Bragg, and G. Reusing, “Transient solutions to a stochastic model of ventilation,” Building and Environment, vol. 24, pp. 265–277, Jan. 1989.
  * [37] A. Thavlov and H. Madsen, “A non-linear stochastic model for an office building with air infiltration,” International Journal of Sustainable Energy Planning and Management, pp. 55–66 Pages, Nov. 2015.  Artwork Size: 55-66 Pages Publisher: International Journal of Sustainable Energy Planning and Management.
  * [38] J.-F. Bonnans, J. C. Gilbert, C. Lemarechal, and C. A. Sagastizábal, Numerical Optimization: Theoretical and Practical Aspects.  Springer Science & Business Media, Oct. 2006.  Google-Books-ID: 88X5vN3ybvgC.
  * [39] E. Gilbert and D. Johnson, “Distance functions and their application to robot path planning in the presence of obstacles,” IEEE Journal on Robotics and Automation, vol. 1, pp. 21–30, Mar. 1985.  Conference Name: IEEE Journal on Robotics and Automation.
  * [40] Y. Rasekhipour, A. Khajepour, S.-K. Chen, and B. Litkouhi, “A Potential Field-Based Model Predictive Path-Planning Controller for Autonomous Road Vehicles,” IEEE Transactions on Intelligent Transportation Systems, vol. 18, pp. 1255–1267, May 2017.  Conference Name: IEEE Transactions on Intelligent Transportation Systems.
  * [41] R. Bonalli, A. Cauligi, A. Bylard, and M. Pavone, “GuSTO: Guaranteed Sequential Trajectory optimization via Sequential Convex Programming,” in 2019 International Conference on Robotics and Automation (ICRA), pp. 6741–6747, May 2019.  ISSN: 2577-087X.
  * [42] J. Chessari, R. Kawai, Y. Shinozaki, and T. Yamada, “Numerical methods for backward stochastic differential equations: A survey,” Probability Surveys, vol. 20, pp. 486–567, Jan. 2023.  Publisher: Institute of Mathematical Statistics and Bernoulli Society.
  * [43] C. Della Santina, C. Duriez, and D. Rus, “Model-Based Control of Soft Robots: A Survey of the State of the Art and Open Challenges,” IEEE Control Systems Magazine, vol. 43, pp. 30–65, June 2023.  Conference Name: IEEE Control Systems Magazine.
  * [44] J. Yong, “Linear forward-backward stochastic differential equations with random coefficients,” Probability Theory and Related Fields, vol. 135, pp. 53–83, May 2006.
  * [45] J. F. Bonnans and A. Shapiro, Perturbation Analysis of Optimization Problems.  New York, NY: Springer New York, 2000.
  * [46] S. Simons, “Minimax Theorems and Their Proofs,” in Minimax and Applications (P. Pardalos, R. Horst, D.-Z. Du, and P. M. Pardalos, eds.), vol. 4, pp. 1–23, Boston, MA: Springer US, 1995.  Series Title: Nonconvex Optimization and Its Applications.