# Optimal feedback control of dynamical systems via value-function
approximation

Karl Kunisch$*$ and  Daniel Walter$\dagger$

###### Abstract.

A self-learning approach for optimal feedback gains for finite-horizon
nonlinear continuous time control systems is proposed and analysed. It relies
on parameter dependent approximations to the optimal value function obtained
from a family of universal approximators. The cost functional for the training
of an approximate optimal feedback law incorporates two main features. First,
it contains the average over the objective functional values of the
parametrized feedback control for an ensemble of initial values. Second, it is
adapted to exploit the relationship between the maximum principle and dynamic
programming. Based on universal approximation properties, existence,
convergence and first order optimality conditions for optimal neural network
feedback controllers are proved.

$*$University of Graz, Institute of Mathematics and Scientific Computing,
Heinrichstr. 36, A-8010 Graz, Austria and Johann Radon Institute for
Computational and Applied Mathematics (RICAM), Austrian Academy of Sciences,
Altenberger Straße 69, 4040 Linz, Austria, (karl.kunisch@uni-graz.at).

$\dagger$Institut für Mathematik, Humboldt-Universität zu Berlin, Rudower
Chaussee 25, 10117 Berlin, Germany,(daniel.walter@hu-berlin.de).

Keywords: optimal feedback control, neural networks, Hamilton-Jacobi-Bellman
equation, self-learning, reinforcement learning.

AMS classification: 49J15, 49N35, 68Q32, 93B52, 93D15.

##  1\. Introduction

In this paper we focus on optimal feedback control for problems of the form

$\left\\{\begin{aligned}
\quad&\inf_{y,u}J(y,u)\coloneqq\frac{1}{2}\int_{0}^{T}\left(|Q_{1}(y(t)-y_{d}(t))|^{2}+\beta|u(t)|^{2}\right)~{}\mathrm{d}t+\frac{1}{2}|Q_{2}(y(T)-y^{T}_{d})|^{2}\\\
&s.t.\quad\dot{y}={f}(y)+g(y)u,\quad y(0)=y_{0},\text{ and }u\in
L^{2}(0,T;\mathbb{R}^{m}),\end{aligned}\right.$ ($P$)

with nonlinear dynamics described by
$f:[0,T]\times\mathbb{R}^{n}\to\mathbb{R}^{n}$. The system can be influenced
by choosing a control input $u$ which enters through a control operator
$g:\mathbb{R}^{n}\to\mathbb{R}^{n\times m}$. We assess the perfomance of a
given control by its objective functional value which comprises the (weighted)
distance between the associated state trajectory $y$ and a given desired state
$y_{d}$ as well as the norm of the control for some cost parameter $\beta>0$.
The weighting matrices $Q_{i}$, for $i=1,2$, are assumed to be symmetric
positive semi-definite. Searching for an optimal control $u^{*}$ in feedback
form requires to find a function
$F^{*}:[0,T]\times\mathbb{R}^{n}\to\mathbb{R}^{m}$ such that

$u^{*}(t)=F^{*}(t,y^{*}(t)),\text{ for }t\in(0,T).$

Here $(u^{*},y^{*})$ denotes an optimal control-trajectory pair associated to
($P$). Under appropriate conditions, see e.g. [12], the feedback mapping can
be expressed as

$F^{*}(t,y)=-\frac{1}{\beta}g^{\top}(t,y)\partial_{y}V^{*}(t,y),$ (1.1)

where $V^{*}$ stands for the value function associate to ($P$), i.e. for
$(T_{0},y_{0})\in[0,T]\times\mathbb{R}^{n}$:

$V^{*}(T_{0},y_{0})=\min_{y,u}J_{T_{0}}(y,u),\text{ subject to
}\dot{y}={f}(y)+g(y)u,\quad y(T_{0})=y_{0},$

and

$J_{T_{0}}(y,u)=\frac{1}{2}\int_{T_{0}}^{T}\left(|Q_{1}(y(t)-y_{d}(t))|^{2}+\beta|u(t)|^{2}\right)~{}\mathrm{d}t+\frac{1}{2}|Q_{2}(y(T)-y^{T}_{d})|^{2}.$

The value function $V^{*}$ satisfies a Hamilton-Jacobi-Bellman (HJB) equation
which is a time-dependent first order hyperbolic equation of spatial dimension
$n$. Numerical realisations, therefore, are plagued by the curse of
dimensionality. Indeed a direct solution of the HJB equation already becomes
computationally prohibitive for moderate dimensions $n$.

Therefore, for practical realization, the interest in alternative techniques
arises. In many situations of practical relevance researches have relied on
linear approximations to the nonlinear dynamical system and have treated the
resulting linear-quadratic problem by Riccati techniques. Much research has
concentrated on validating this approach locally around a reference
trajectory. Globally such a strategy may fail, see for instance [8, 16].

In this paper we follow an approach, possibly first proposed in [16],
circumventing the construction of the value function on the basis of solving
the HJB equation. Rather the feedback mapping is constructed by an
unsupervised self-learning technique. In practice, this requires the
approximation of $V^{*}$ by a family of functions $V_{\theta}$ which are
parametrized by a finite dimensional vector $\theta$ and satisfy a uniform
approximation property. Possible families of universal approximators include,
e.g., neural networks or piecewise polynomial approximations. Subsequently, in
view of (1.1), we introduce the corresponding feedback law

$F_{\theta}(t,y)=-\frac{1}{\beta}g^{\top}(y)\partial_{y}V_{\theta}(t,y),\text{
for }(t,y)\in[0,\infty)\times\mathbb{R}^{n},$ (1.2)

as approximation to $F^{*}$. An "optimal" parametrized feedback law is then
determined by a variant of the following self-learning, structure preserving,
variational problem:

$\begin{array}[]{l}\min_{\theta}\;J(y,\mathcal{F}_{\theta}(y))\\\\[6.45831pt]
\qquad\;+\frac{1}{2}\int^{T}_{0}{\gamma_{1}}|V_{\theta}(t,y(t))-J_{t}(y,F_{\theta}(\cdot,y))|^{2}+{\gamma_{2}}|\partial_{y}V_{\theta}(t,y(t))-p(t)|^{2}~{}\mathrm{d}t+\frac{\gamma_{\varepsilon}}{2}{|\theta|^{2}}\\\\[7.74998pt]
\text{s.t. }\quad\dot{y}={f}(y)+g(y)F_{\theta}(y),\quad y(0)=y_{0},\quad
p(T)=Q^{\top}_{2}Q_{2}(y(T)-y^{T}_{d})\\\\[6.45831pt]
\qquad\;-\dot{p}={f}(y)^{\top}p+[D{g}(y)^{\top}F_{\theta}(y)]p+Q_{1}^{\top}Q_{1}(y-y_{d}).\end{array}$
(1.3)

In this problem, minimization with respect to $u$ is replaced by minimizing
with respect to the parameters $\theta$ which characterize $V_{\theta}$ and
$F_{\theta}$. The cost functional of problem (1.3) consists of four parts: The
first term represents the objective functional of ($P$) where the control $u$
is replaced by the closed loop expression $F_{\theta}(y)$. The next two terms
realize the fact that $V_{\theta}$ is constructed as approximation to the
value function associated to $\eqref{def:refproblem}$ and exploit the well-
known property that, under certain conditions, the gradient of the value
function coincides with the solution of a suitable adjoint equation, see e.g.
[12, page 21]. The final term penalizes the norm of the structural parameters.
We point out that $V_{\theta}$ and $F_{\theta}$ are learned along the orbit
$\mathcal{O}=\\{y(t;y_{0}):t\in(0,\infty)\\}$ within the state space
$\mathbb{R}^{n}$. To accommodate the case that one trajectory does not provide
enough information, we propose to involve an ensemble of orbits departing from
a set $Y_{0}$ of initial conditions, and to reformulate problem (1.3)
accordingly. This will be done in Section 4 below.

In our earlier work on learning a feedback function, [16], we considered
infinite horizon optimal control problems. In that case, the time-dependent
HJB equation results in a stationary one. There we had not yet incorporated
the structure preserving terms involving $V_{\theta}$ and
$\partial_{y}V_{\theta}$ into the cost. Moreover we directly constructed an
approximation $F_{\theta}$ to the vector valued function $F^{*}$, rather than
approximating the scalar valued function $V^{*}$ and subsequently using (1.2).
In the present paper we provide the theoretical foundations for the learning
based technique that we propose to construct an approximation to the optimal
feedback function for ($P$). Recently in [20] a variant of the approach as in
[16] was used for interesting numerical investigations to construct optimal
feedback functions for finite horizon multi-agent optimal control problems.

Let us very briefly mention some of the vast literature on solving the HJB
equations. Semi-Lagrangian schemes and finite difference methods have been
deeply investigated to directly solve HJB equations directly, see e.g. [5, 11,
15]. Significant progress was made in solving high dimensional HJB equations
by the of use policy iterations combined with tensor calculus techniques, [8,
14, 9]. The use of Hopf formulas was proposed in e.g. [18, 6]. Interpolation
techniques, utilizing ensembles of open loop solutions have been analyzed in
the works of [2, 19], for example. Finally we mention that optimal feedback
control is intimately related to reinforcement learning, see e.g. the
monograph [4], and also the survey articles [17, 22, 23].

The manuscript is structured as follows. Some pertinent notation is gathered
in Section 2. In Section 3 concepts of optimal feedback control, semi-global
with respect to the initial condition $y_{0}$, are gathered. Section 4 is
devoted to describing the learning technique that we propose to approximate
the optimal feedback function. In Section 5 the required assumptions on
approximating subspaces are checked for a class of neural networks and a class
of piecewise polynomials. Existence of solutions to the approximating learning
problems is proved in Section 6. Their convergence is analyzed in Section 7.
The case of learning from finitely many orbits is the focus of Section 8.
Section 9 provides an example illustrating the numerical feasibility of the
proposed method. We do not aim for sophistication in this respect. The
appendix details the proofs of several necessary technical results.

##  2\. Notation

For $I:=(0,T)$, with $T>0$, we define $W_{T}=\\{\,y\in
L^{2}(I;\mathbb{R}^{n})\;|\;\dot{y}\in L^{2}(I;\mathbb{R}^{n})\,\\},$ where
the temporal derivative is understood in the distributional sense. We equip
$W_{T}$ with the norm induced by the inner product

$\displaystyle(y_{1},y_{2})_{W_{T}}=(\dot{y}_{1},\dot{y}_{2})_{L^{2}(I;\mathbb{R}^{n})}+(y_{1},y_{2})_{L^{2}(I;\mathbb{R}^{n})}\quad\text{for
}y_{1},y_{2}\in W_{T},$

making it a Hilbert space. We recall that $W_{T}$ embeds continuously into
$C(\bar{I};\mathbb{R}^{n})$. For a compact metric space $X$ we denote the
space of continuous functions between $X$ and $Y$ by $\mathcal{C}(X;Y)$ which
we endow with $\|\varphi\|_{\mathcal{C}(X;Y)}=\max_{x\in X}\|\varphi(x)\|_{Y}$
as norm. By $Y_{0}$ we denote a compact set of initial conditions in
$\mathbb{R}^{n}$. When arising as index, the space $\mathcal{C}(Y_{0};W_{T})$
will frequently be abbreviated by $\mathcal{C}$. The space
$\mathcal{C}^{1}(X;Y)$ of continuously differentiable functions is defined
analogously. Open balls of radius $\varepsilon$ in a Banach space $X$ with
center $x$ will be denoted by $B_{\varepsilon}(x)$. The space of bounded
linear operators between Banach spaces $X$ and $Y$, endowed with the canonical
norm, is denoted by $\mathcal{B}(X,Y)$. We further abbreviate
$\mathcal{B}(X):=\mathcal{B}(X,X)$.

##  3\. Semi-global optimal feedback control

Consider the controlled nonlinear dynamical system of the form

$\displaystyle\dot{y}=\mathbf{f}(y)+\mathbf{g}(y)u\quad\text{in}~{}L^{2}(I;\mathbb{R}^{n}),\quad
y(0)=y_{0},$ (3.1)

described by Nemitsky operators

$\begin{array}[]{l}\mathbf{f}\colon W_{T}\to
L^{2}(I;\mathbb{R}^{n}),\quad\mathbf{f}(y)(t)=f(t,y(t))\\\\[5.59721pt]
\mathbf{g}\colon
W_{T}\to\mathcal{L}(L^{2}(I;\mathbb{R}^{m});L^{2}(I;\mathbb{R}^{n})),\quad\mathbf{g}(y)(t)=g(t,y(t))\end{array}$
(3.2)

for $a.e.~{}t\in I$, $f\colon I\times\mathbb{R}^{n}\to\mathbb{R}^{m}$ and
$g\colon I\times\mathbb{R}^{n}\to\mathbb{R}^{n\times m}$. The smoothness
requirements on $f$ and $g$ will be detailed in Assumption 1 below. Our aim is
to choose a control input $u^{*}\in L^{2}(I;\mathbb{R}^{m})$ which keeps the
associated solution $y^{*}\in W_{T}$ close to a known reference trajectory
$y_{d}$, while keeping the control effort small. This is formulated as the
constrained minimization problem

$\left\\{\begin{aligned} \quad&\inf_{y\in W_{T},\,u\in
L^{2}(I;\mathbb{R}^{m})}J(y,u)\\\
&s.t.\quad\dot{y}=\mathbf{f}(y)+\mathbf{g}(y)u,\quad
y(0)=y_{0},\end{aligned}\right.$ ($P_{y_{0}}$)

where

$\displaystyle
J(y,u)=\frac{1}{2}\int_{I}\left(|Q_{1}(y(t)-y_{d}(t))|^{2}+\beta|u(t)|^{2}\right)~{}\mathrm{d}t+\frac{1}{2}|Q_{2}(y(T)-y^{T}_{d})|^{2},$

which incorporates the weighted misfit between the trajectory $y$ within the
time horizon $I=(0,T)$ and at the terminal time to desired states $y_{d}\in
L^{2}(I;\mathbb{R}^{n})$ and $y_{d}^{T}\in\mathbb{R}^{n}$, as well as the norm
of the control $u$. While this open loop optimal control problem captures well
the objective formulated above, it comes with several disadvantages. First,
its solution is a function of time only, and does not include the current
state $y(t)$. This makes the open loop approach susceptible to possible
perturbations in the dynamical system. Second, determining the control action
for a new initial condition requires to solve ($P_{y_{0}}$) from the start.

The aforementioned limitations of open loop optimal controls motivate the
study of semi-global optimal feedback control approaches to ($P_{y_{0}}$).
More precisely, given a compact set $Y_{0}\subset\mathbb{R}^{n}$, we look for
a feedback function $F^{*}\colon I\times\mathbb{R}^{n}\to\mathbb{R}^{m}$ which
induces a Nemitsky operator

$\displaystyle\mathcal{F}^{*}\colon
W_{T}\to{L^{2}}(I;\mathbb{R}^{m}),\quad\mathcal{F}^{*}(y)(t)=F^{*}(t,y(t))\quad\text{for
a.e.}~{}t\in I,$

such that for every $y_{0}\in Y_{0}$ the closed loop system

$\displaystyle\dot{y}=\mathbf{f}(y)+\mathbf{g}(y)\mathcal{F}^{*}(y),\quad
y(0)=y_{0},$ (3.3)

admits a unique solution $y^{*}(y_{0})\in W_{T}$ and
$(y^{*}(y_{0}),\mathcal{F}^{*}(y^{*}(y_{0})))$ is a minimizing pair of
($P_{y_{0}}$).

The determination of an optimal feedback function usually rests on the
computation of the value function to($P_{y_{0}}$) which is defined as

$\displaystyle V^{*}(T_{0},y_{0}):=\min_{\begin{subarray}{c}y\in
H^{1}(T_{0},T;\mathbb{R}^{n}),\\\ u\in
L^{2}(T_{0},T;\mathbb{R}^{m})\end{subarray}}J_{T_{0}}(y,u)\quad
s.t.\quad\dot{y}=\mathbf{f}(y)+\mathbf{g}(y)u,~{}\quad y(t_{0})=y_{0},$ (3.4)

where $(T_{0},y_{0})\in I\times\mathbb{R}^{n}$, and $J_{T_{0}}(y,u)$ is
defined as

$\displaystyle
J_{T_{0}}(y,u)=\frac{1}{2}\int^{T}_{T_{0}}\left(|Q_{1}(y(t)-y_{d}(t))|^{2}+\beta|u(t)|^{2}\right)~{}\mathrm{d}t+\frac{1}{2}|Q_{2}(y(T)-y_{d}(T))|^{2}.$

By construction $V^{*}$ satisfies the final time boundary condition

$\displaystyle
V^{*}(T,y_{0})=\frac{1}{2}|Q_{2}(y_{0}-y_{d}(T))|^{2}\quad\forall
y_{0}\in\mathbb{R}^{n}.$

If $V^{*}$ is continuously differentiable in a neighborhood of some
$(t,y_{0})\in I\times\mathbb{R}^{n}$ then it solves the instationary Hamilton-
Jacobi-Bellman (HJB) equation

$\displaystyle\partial_{t}V^{*}(t,y_{0})+(f(y_{0}),\partial_{y}V^{*}(t,y_{0}))_{\mathbb{R}^{n}}-\frac{1}{2\beta}|g(t,y_{0})^{\top}\partial_{y}V^{*}(t,y_{0})|^{2}+\frac{1}{2}|Q_{1}(y_{0}-y_{d}(t))|^{2}=0$
(3.5)

in the classical sense there, see e.g. [10, 12]. Here $\partial_{t}V^{*}$
denotes the partial derivative of the value function with respect to $t$ and
$\partial_{y}V^{*}$ is the gradient of $V^{*}$ with respect to the
$y$-variable. An optimal control for ($P_{y_{0}}$) in feedback form is then
given by
$u^{*}=-\frac{1}{\beta}\mathbf{g}(y^{*})^{\top}\partial_{y}\mathcal{V}^{*}(y^{*})$
where $\partial_{y}\mathcal{V}^{*}(y^{*})(t)=\partial_{y}V^{*}(t,y^{*}(t))$
for every $t\in I$, and $y^{*}=y^{*}(y_{0})\in W_{T}$ solves the closed loop
system

$\displaystyle\dot{y}=\mathbf{f}(y)-\frac{1}{\beta}\mathbf{g}(y)\mathbf{g}(y)^{\top}\partial_{y}\mathcal{V}^{*}(y),\quad
y(0)=y_{0}.$

Thus

$\displaystyle\left(y^{*}(y_{0}),-\frac{1}{\beta}\mathbf{g}(y^{*}(y_{0}))^{\top}\partial_{y}\mathcal{V}^{*}(y^{*}(y_{0}))\right)\in\operatorname*{arg\,min}\eqref{def:openloopproblem}$

and the function

$\displaystyle
F^{*}(\cdot,\cdot)=-\frac{1}{\beta}g(\cdot,\cdot)^{\top}\partial_{y}V^{*}(\cdot,\cdot)$

is an optimal feedback law.

Realizing the optimal feedback in this way requires a solution to (3.5) which
is a partial differential equation on $\mathbb{R}^{n}$. This can be extremely
challenging or even impossible depending on the dimension $n$ and the
computational facilities at hand. Similarly to our previous manuscript [16] we
take a different approach by formulating minimization problem over a suitable
set of feedback functions involving the closed loop system as a constraint.
This relates to a learning problem, within which the feedback functions are
trained to achieve optimal stabilization. This makes the problem
computationally amenable.

The procedure just described will be formalized in the following section. Here
we first summarize the assumptions on the nonlinear dynamical system that we
refer to throughout the paper.

######  Assumption 1.

  * A.1

The functions $f\colon I\times\mathbb{R}^{n}\to\mathbb{R}^{n}$ and $g\colon
I\times\mathbb{R}^{n}\to\mathbb{R}^{n\times m}$ are twice continuously
differentiable. Their Jacobians and Hessians with respect to the second
variable, denoted by $D_{y}f,D_{yy}f$, and $D_{y}g,D_{yy}g$, respectively, are
Lipschitz continuous on compact sets, uniformly for $t\in I$.

  * A.2

There exists a constant $M_{Y_{0}}>0$ such that the value function
$V(\cdot,\cdot)\colon I\times\mathbb{R}^{n}\to\mathbb{R}$ for ($P_{y_{0}}$) is
twice continuously differentiable on $I\times\bar{B}_{2\widehat{M}}(0)$ with
Lipschitz continuous gradient and Hessian (w.r.t. $y$ uniformly in $t\in I$)
where

$\displaystyle\widehat{M}=M_{Y_{0}}\,\|\imath\|_{\mathcal{B}(W_{T},\,\mathcal{C}(I;\mathbb{R}^{n}))},$
(3.6)

and $\imath$ denotes the embedding of $W_{T}$ into
$\mathcal{C}(I;\mathbb{R}^{n}))$.

As a consequence of (A.1), the Nemitsky operators $\mathbf{f},~{}\mathbf{g}$
are at least two times continuously differentiable with domains and ranges as
defined in (3.2). Their derivatives, denoted by $D\mathbf{f}$ and
$D\mathbf{g}$, are the Nemitsky operators induced by $D_{y}f$ and $D_{y}g$. We
point out
$D\mathbf{g}(y)\in\mathcal{B}(W_{T};L^{2}(I;\mathbb{R}^{m});L^{2}(I;\mathbb{R}^{n}))$.
Moreover $\mathbf{f},D\mathbf{f},\mathbf{g},D\mathbf{g}$ are Lipschitz
continuous and bounded, on bounded subsets of $L^{\infty}(I;\mathbb{R}^{n})$,
and thus in particular on $\mathcal{Y}_{ad}\subset W_{T}$, where

$\displaystyle\mathcal{Y}_{ad}:=\left\\{\,y\in W_{T}\;|\;\|y\|_{W_{T}}\leq
2M_{Y_{0}}\,\right\\}.$ (3.7)

Finally $D\mathbf{f}^{\top}\in{\mathcal{B}}(W_{T},L^{2}(I;\mathbb{R}^{n}))$
denotes the Nemitsky operator associated to $D_{y}f^{\top}$.

Analogously, due to (A.2), $V^{*}$ induces a twice Lipschitz continuously
Fréchet differentiable Nemitsky operator
$\mathcal{V}^{*}:\mathcal{Y}_{ad}\subset W_{T}\to
L^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}2}}(I)$.
Moreover $\mathcal{V}^{*}$ and its first derivative $D\mathcal{V}^{*}$ are
weak-to-strong continuous. Define the Nemitsky operator

$\displaystyle\mathcal{F}^{*}\colon\mathcal{Y}_{ad}\to
L^{2}(I;\mathbb{R}^{n}),{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\quad}\mathcal{F}^{*}(y)=-\frac{1}{\beta}\mathbf{g}(y)^{\top}\partial_{y}\mathcal{V}^{*}(y),$
(3.8)

where $\partial_{y}\mathcal{V}^{*}$ is the Nemitsky operator induced by the
gradient $\partial_{y}V^{*}=D_{y}V(\cdot,\cdot)^{\top}$. Note also that
$\mathcal{F}^{*}\in
C^{1}(W_{T};(L^{2}(I;\mathbb{R}^{m});L^{2}(I;\mathbb{R}^{n})))$. We further
assume the following:

  * A.3

For every $y_{0}\in Y_{0}$ there exists a unique function
$y=\mathbf{y}^{*}(y_{0})\in W_{T}$ satisfying

$\displaystyle\dot{y}=\mathbf{f}(y)+\mathbf{g}(y)\mathcal{F}^{*}(y),\quad
y(0)=y_{0},\quad\|y\|_{W_{T}}\leq M_{Y_{0}}.$

Moreover we have

$\displaystyle(y^{*}(y_{0}),\mathcal{F}^{*}(y^{*}(y_{0})))\in\operatorname*{arg\,min}\eqref{def:openloopproblem}\quad\forall
y_{0}\in Y_{0}.$

When referring to Assumption 1 we mean (A.1)-(A.3). We emphasize that the
constant $M$ appearing in (A2) and (A3) is assumed to be same. Note further
that as a consequence of (A3) problem ($P_{y_{0}}$) admits a solution for each
$y_{0}\in Y_{0}$, with the optimal control given by
$u^{*}=\mathcal{F}^{*}(y^{*}(y_{0})))$.

######  Remark 1.

Using $(\mathbf{A.1})$, $(\mathbf{A.3})$ as well as the implicit function
theorem it can be readily be verified that the mapping $\mathbf{y}^{*}\colon
Y_{0}\to W_{T}$ from ($\mathbf{A.3}$) is continuously differentiable. Given
$\delta y_{0}\in\mathbb{R}^{n}$ the directional derivative $\delta
y\coloneqq\partial\mathbf{y}^{*}(y_{0})(\delta y_{0})$ of $\mathbf{y}^{*}$ at
$y_{0}\in Y_{0}$ in direction $\delta y_{0}$ satisfies the linearized ODE
system

$\displaystyle\dot{\delta y}=D\mathbf{f}(\mathbf{y}^{*}(y_{0}))\delta
y+[D\mathbf{g}(\mathbf{y}^{*}(y_{0}))\delta
y]\mathcal{F}^{*}(\mathbf{y}^{*}(y_{0}))+\mathbf{g}(\mathbf{y}^{*}(y_{0}))D\mathcal{F}^{*}(\mathbf{y}^{*}(y_{0}))\delta
y,\,\delta y(0)=\delta y_{0}.$

Here $D\mathbf{g}$ is induced by $D_{y}g$ which is given by

$\displaystyle\left[D_{y}g(t,y)\delta
y\right]_{ij}=\left(\sum^{n}_{k=1}\partial_{k}g_{ij}(t,y)\delta
y_{k}\right)\quad\forall\delta y\in\mathbb{R}^{n},$

where $g(y)=(g_{ij})$ and $"\partial_{k}"$ denotes the partial derivative
w.r.t to the $k$-th component of $y$. The transposed $D\mathbf{g}(y)^{\top}$,
which will arise in the adjoint equation below, is induced by the tensor
$D_{y}g(t,\cdot)^{\top}=(D_{y}g(t,\cdot)_{kji})\in\mathbb{R}^{n\times n\times
m}$, with $t\in I$. In particular, we readily verify that
$D\mathbf{g}(\cdot)^{\top}\in\mathcal{B}(L^{2}(I;\mathbb{R}^{m});\mathcal{B}(W_{T};L^{2}(I;\mathbb{R}^{n})))$.

To end this section we collect structural information on the relation between
the adjoined state, denoted by ${p}$ below, the optima value function $V^{*}$,
and the induced optimal feedback law $\mathcal{F}^{*}$.

######  Proposition 1.

Let Assumption 1 hold. Then there exists a unique continuous mapping
$\mathbf{p}^{*}\colon Y_{0}\to W_{T}$ such that for each $y_{0}\in Y_{0}$ the
tuple $(y,p)=(\mathbf{y}^{*}(y_{0}),\mathbf{p}^{*}(y_{0}))$ satisfies

$\displaystyle\frac{d}{dt}y$
$\displaystyle=\mathbf{f}(y)+\mathbf{g}(y)\mathcal{F}^{*}(y),~{}y(0)=y_{0},$
(3.9) $\displaystyle-\frac{d}{dt}p$
$\displaystyle=D\mathbf{f}(y)^{\top}p+[D\mathbf{g}(y)^{\top}\mathcal{F}^{*}(y)]p+Q_{1}^{\top}Q_{1}(y-y_{d}),~{}p(T)=Q^{\top}_{2}Q_{2}(y(T)-y^{T}_{d}),$
(3.10) $\displaystyle\mathcal{F}^{*}(y)$
$\displaystyle=-\frac{1}{\beta}\mathbf{g}(y)^{\top}p.$ (3.11)

Moreover we have

$\displaystyle
V^{*}(t,y(t))=J_{t}(y(t),F^{*}(t,y(t))),~{}p(t)=\partial_{y}V(t,y(t))\quad\forall
t\in[0,T].$ (3.12)

###### Proof of Proposition 1.

By (A.3) problem ($P_{y_{0}}$) admits a solution for each $y_{0}\in Y_{0}$.
Then (A.1)-(A.2) guarantee that (3.10), with $y=y(y_{0})\in W_{T}$ the state
component of a solution to ($P_{y_{0}}$), admits a unique solution $p$ in
$W_{T}$ which continuously depends on $y\in W_{T}$. Moreover (3.9) - (3.11)
represent the first order necessary optimality condition for ($P_{y_{0}}$)
with the optimal control $u(t)=\mathcal{F}^{*}(y(t))$. Since
$\mathbf{y}^{*}\colon Y_{0}\to W_{T}$ is continuous as mentioned in Remark 1
and the solution to (3.10) depends continuously on $y\in W_{T}$, the claimed
continuity $\mathbf{p}^{*}\colon Y_{0}\to W_{T}$ follows. Equation (3.12) is a
direct consequence of the dynamic programming principle, and (A.3). ∎

##  4\. Optimal feedback control by value function approximation

This section is devoted to introducing a family of computationally tractable
minimization problems from which we will "learn" approximations of optimal
feedback laws. Our approach rests on two main pillars. First, given
$\varepsilon>0$, we consider a family of functions
$V^{\varepsilon}_{\theta}\in\mathcal{C}(I\times\mathbb{R}^{n})$ which are
finitely parametrized by
$\theta\in\mathcal{R}_{\varepsilon}\simeq\mathbb{R}^{N_{\varepsilon}}$,
$N_{\varepsilon}\in\mathbb{N}$. These serve as "discrete" approximations of
the optimal value function $V^{*}$. The following a priori estimate is
assumed, for some fixed $\varepsilon_{0}>0$:

######  Assumption 2.

For every $0<\varepsilon\leq\varepsilon_{0}$ there holds
$V_{\cdot}^{\varepsilon}\in\mathcal{C}^{4}(\mathcal{R}_{\varepsilon}\times\mathbb{R}\times\mathbb{R}^{n})$
and $V^{\varepsilon}_{\theta}(T,y_{0})=\frac{1}{2}|Q_{2}(y_{0}-y_{d}(T))|^{2}$
for every $y_{0}\in\mathbb{R}^{n}$ and $\theta\in\mathcal{R}_{\varepsilon}$.
Moreover there exists $\theta_{\varepsilon}\in\mathcal{R}_{\varepsilon}$ with

$\displaystyle\max_{\begin{subarray}{c}t\in I,\\\ |y|\leq
2\widehat{M}\end{subarray}}|V^{\varepsilon}_{\theta_{\varepsilon}}(t,y)-V^{*}(t,y)|+|\partial_{y}(V^{\varepsilon}_{\theta_{\varepsilon}}(t,y)-V^{*}(t,y))|+\|\partial_{yy}(V^{\varepsilon}_{\theta_{\varepsilon}}(t,y)-V^{*}(t,y))\|\leq
c\varepsilon$ (4.1)

for some $c>0$ independent of $\varepsilon\in(0,\varepsilon_{0}]$.

Now recall from (3.8) that the optimal feedback law $\mathcal{F}^{*}$ is the
superposition operator induced by
$F^{*}(t,y)=-(1/\beta)g(t,y)^{\top}\partial_{y}V^{*}(t,y)$. With the aim of
preserving the dependence of the feedback law on the value function in our
approximation, we define a set of parametrized feedback laws
$\mathcal{F}^{\varepsilon}_{\theta}$ associated to $V^{\varepsilon}_{\theta}$,
$\theta\in\mathcal{R}_{\varepsilon}$, by

$\displaystyle\mathcal{F}^{\varepsilon}_{\theta}(y)(t)=F^{\varepsilon}_{\theta}(t,y(t))=-\frac{1}{\beta}g(t,y(t))\partial_{y}V^{\varepsilon}_{\theta}(t,y(t))$

for all $y\in W_{T}$, $t\in\bar{I}$ and $\theta\in\mathcal{R}_{\varepsilon}$.
A first approach to obtain an optimal feedback law in the form
$\mathcal{F}^{\varepsilon}_{\theta}$ can then be found by replacing the open
loop control $u$ in ($P_{y_{0}}$) by the closed loop expression
$\mathcal{F}^{\varepsilon}_{\theta}(y)$ and minimizing for
$\theta\in\mathcal{R}_{\varepsilon}$:

$\displaystyle\min_{y\in
W_{T},\theta\in\mathcal{R}_{\varepsilon}}J(y,F^{\varepsilon}_{\theta}(y))+\frac{\gamma_{\varepsilon}}{2}\|\theta\|^{2}_{\mathcal{R}_{\varepsilon}}\quad
s.t.\quad\dot{y}=\mathbf{f}(y)+\mathbf{g}(y)\mathcal{F}^{\varepsilon}_{\theta}(y),~{}y(0)=y_{0},$
(4.2)

where $\|\cdot\|_{\mathcal{R}_{\varepsilon}}$ denotes a Hilbert space norm on
$\mathcal{R}_{\varepsilon}$, $\gamma_{\varepsilon}>0$ and $y_{0}\in Y_{0}$ is
fixed. This represents the goal of finding a feedback law
$\mathcal{F}^{\varepsilon}_{\theta}$ together with a trajectory $y\in W_{T}$
which satisfy
$(y,\mathcal{F}^{\varepsilon}_{\theta}(y))\in\operatorname*{arg\,min}\eqref{def:openloopproblem}$.
However, this approach falls short in several aspects. First, we cannot hope
to recover a solution of the semiglobal optimal feedback control problem for
all $y_{0}\in Y_{0}$, since the minimization in (4.2) is associate to a single
initial condition only. Secondly it misses to impose properties that would
guide ${\mathcal{F}}^{\varepsilon}_{\theta}(y)$ to be close to
$\mathcal{V}^{*}$, and it does not exploit the relation between the adjoint
state $p$, see (3.10), and the gradient of the value function
$\partial_{y}\mathcal{V}^{*}$. Incorporating this information into the problem
can, potentially, lead to improved learning results and improved parameterized
feedback laws which behave similarly to $\mathcal{F}^{*}$. These
considerations lead to the second pillar of our approach, namely a succinct
choice of the cost for the learning problem. For this purpose we use all of
$Y_{0}$ as "learning set" for initial conditions. It is endowed with the
normalized Lebesgue measure $\mathcal{L}$. Moreover we define the augmented
objective

$\begin{array}[]{l}J_{\varepsilon}(y,p,\theta)=J(y,\mathcal{F}^{\varepsilon}_{\theta}(y))\\\\[6.45831pt]
+\int^{T}_{0}\frac{\gamma_{1}}{2}|V^{\varepsilon}_{\theta}(t,y(t))-J_{t}(y,\mathcal{F}^{\varepsilon}_{\theta}(y))|^{2}+\frac{\gamma_{2}}{2}|\partial_{y}V^{\varepsilon}_{\theta}(t,y(t))-p(t)|^{2}~{}\mathrm{d}t\end{array}$
(4.3)

for penalty parameters $\gamma_{1},\gamma_{2}\geq 0$. The arguments in $J_{t}$
are the restriction of the solution $y$ to the equation in (4.2) and the
feedback $\mathcal{F}^{\varepsilon}_{\theta}(y)$ to $[t,T]$. The additional
terms in this new objective functional penalize the violation of the cost and
its gradient by means of the approximation based on
$V_{\theta}^{\varepsilon}$, i.e. they penalize the differences between
$J_{t}(y,\mathcal{F}^{\varepsilon}_{\theta}(y))$ and
$V^{\varepsilon}_{\theta}(t,y(t)$, as well as $p(t)$ and
$\partial_{y}V^{\varepsilon}_{\theta}(t,y(t)$.

Given a strictly positive weight function $\omega\in
L^{\infty}(Y_{0});~{}0<c\leq\omega$ a.e., we thus propose to find a feedback
law $\mathcal{F}^{\varepsilon}_{\theta}$ by solving the ensemble control
problem

$\displaystyle\min_{\begin{subarray}{c}\mathbf{y}\in\mathbf{Y}_{ad},\\\
\mathcal{F}^{\varepsilon}_{\theta}(\mathbf{y})\in\mathbf{U}_{ad},\\\
\mathbf{p}\in\mathcal{C}(Y_{0};W_{T})\\\
\theta\in\mathcal{R}_{\varepsilon}\end{subarray}}\mathcal{J}_{\varepsilon}(\mathbf{y},\mathbf{p},\theta)\coloneqq\int_{Y_{0}}\omega(y_{0})\,J_{\varepsilon}(\mathbf{y}(y_{0}),\mathbf{p}(y_{0}),\theta)~{}\mathrm{d}\mathcal{L}(y_{0})+\frac{\gamma_{\varepsilon}}{2}\|\theta\|^{2}_{\mathcal{R}_{\varepsilon}}$
($\mathcal{P}_{\varepsilon}$)

subject to the system of closed loop state and adjoint equations

$\displaystyle\dot{\mathbf{y}}(y_{0})=\mathbf{f}(\mathbf{y}(y_{0}))+\mathbf{g}(\mathbf{y}(y_{0}))\mathcal{F}^{\varepsilon}_{\theta}(\mathbf{y}(y_{0}))$
(4.4)
$\displaystyle-\dot{\mathbf{p}}(y_{0})=D\mathbf{f}(\mathbf{y}(y_{0}))^{\top}\mathbf{p}(y_{0})+[D\mathbf{g}(\mathbf{y}(y_{0}))^{\top}\mathcal{F}^{\varepsilon}_{\theta}(\mathbf{y}(y_{0}))]\mathbf{p}(y_{0})+\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}(\mathbf{y}(y_{0})-y_{d})$
(4.5)
$\displaystyle\mathbf{y}(y_{0})(0)=y_{0},~{}\mathbf{p}(y_{0})(T)=Q^{\top}_{2}Q_{2}(\mathbf{y}(y_{0})(T)-y_{d}^{T}),~{}\mathbf{y}(y_{0})\in\mathcal{Y}_{ad}$
(4.6)

for $\mathcal{L}$-a.e. $y_{0}\in Y_{0}$. Above
$\mathbf{Y}_{ad}\subset\mathcal{C}(Y_{0};W_{T})$ and $\mathbf{U}_{ad}\subset
L^{2}(Y_{0};L^{2}(I;\mathbb{R}^{m}))$ denote the admissible sets of ensemble
state trajectories and admissible controls. They will be specified in section
6.

##  5\. Examples

In this section we discuss two particular examples for the parameterized
mappings $V^{\varepsilon}$: deep residual networks and piecewise polynomial
functions of sufficiently high degree.

###  5.1. Residual networks

To explain the approximation of the value function by residual neural
networks, we first fix some notation. Let $L_{\varepsilon}\in\mathbb{N}$,
$L_{\varepsilon}\geq 2$, as well as $N^{\varepsilon}_{i}\in\mathbb{N}$,
$i=1,\dots,L_{\varepsilon}-1$ be given. We set $N^{\varepsilon}_{0}=n+1$ and
$N^{\varepsilon}_{L}=1$. Furthermore define

$\displaystyle\mathcal{R}_{\varepsilon}=\bigtimes^{L_{\varepsilon}-1}_{i=1}\left(\mathbb{R}^{N^{\varepsilon}_{i}\times
N^{\varepsilon}_{i-1}}\times\mathbb{R}^{N^{\varepsilon}_{i}\times
N^{\varepsilon}_{i-1}}\times\mathbb{R}^{N^{\varepsilon}_{i}}\right)\times\mathbb{R}^{N^{\varepsilon}_{L}\times
N^{\varepsilon}_{L-1}}.$

The space $\mathcal{R}_{\varepsilon}$ is uniquely determined by its
architecture

$\displaystyle\text{arch}(\mathcal{R}_{\varepsilon})=\left(N^{\varepsilon}_{0},N^{\varepsilon}_{1},\dots,N^{\varepsilon}_{L}\right)\in\mathbb{N}^{L_{\varepsilon}+1}.$

A set of parameters $\theta\in\mathcal{R}_{\varepsilon}$ given by

$\displaystyle\theta=\left(W_{11},W_{12},b_{1},\dots,W_{{L_{\varepsilon}}}\right)$

is called a neural network with $L_{\varepsilon}$ layers. Moreover let
$\sigma\in\mathcal{C}^{4}(\mathbb{R})$ be given and assume that $\sigma$ is
not a polynomial. The function

$\begin{array}[]{l}V^{\varepsilon}_{\theta}(t,y)=\frac{1}{2}|Q_{2}(y-y_{d}(T))|^{2}\\\\[6.45831pt]
\qquad\qquad+f^{\sigma}_{L_{\varepsilon},\theta}\circ
f^{\sigma}_{L_{\varepsilon}-1,\theta}\circ\cdots\circ
f^{\sigma}_{1,\theta}((t,y))-f^{\sigma}_{L_{\varepsilon},\theta}\circ
f^{\sigma}_{L_{\varepsilon}-1,\theta}\circ\cdots\circ
f^{\sigma}_{1,\theta}((T,y))\end{array}$ (5.1)

for $(t,y)\in\mathbb{R}\times\mathbb{R}^{n}$ where

$\displaystyle
f^{\sigma}_{L_{\varepsilon},\theta}(x)=W_{L_{\varepsilon}}x\quad\forall
x\in\mathbb{R}^{N^{\varepsilon}_{L-1}}$

as well as

$\displaystyle
f^{\sigma}_{i,\theta}(x)=\sigma(W_{i1}x+b_{i})+W_{i2}x\quad\forall
x\in\mathbb{R}^{N^{\varepsilon}_{i-1}},~{}i=1,\dots,L_{\varepsilon}-1$

is called the realization of $\theta$ with activation function $\sigma$. Here
the application of $\sigma$ is defined to act componentwise i.e. given an
index $i\in\\{1,\dots,L_{\varepsilon}-1\\}$ and
$x\in\mathbb{R}^{N^{\varepsilon}_{i}}$ we set

$\displaystyle\sigma(x)=(\sigma(x_{1}),\dots,\sigma(x_{N^{\varepsilon}_{i}}))^{\top}.$

By construction, $V^{\varepsilon}_{\theta}$ satisfies the terminal condition

$\displaystyle
V^{\varepsilon}_{\theta}(T,y)=\frac{1}{2}|Q_{2}(y-y_{d}(T))|^{2}\quad\forall
y\in\mathbb{R}^{n}.$

Moreover Assumption 2 is fulfilled as confirmed by the following result.

######  Theorem 2.

For every $\varepsilon>0$ there exists architectures
$\operatorname{\mathcal{R}}_{\varepsilon}$ and
$\theta_{\varepsilon}\in\mathcal{R}_{\varepsilon}$ such that
$V^{\varepsilon}\in\mathcal{C}^{4}(\mathcal{R}_{\varepsilon}\times\mathbb{R}\times\mathbb{R}^{n})$
and $V^{\varepsilon}_{\theta_{\varepsilon}}$ satisfies (4.1).

###### Proof.

Let us set $h(t,y)=V^{*}(t,y)$ for $(t,y)\in
I\times\bar{B}_{2\widehat{M}}(0)$. Then $h$ is twice continuously
differentiable on $I\times\bar{B}_{2\widehat{M}}(0)$ and
$h(T,y)=\frac{1}{2}|Q_{2}(y-y^{T}_{d})|^{2}$. A consequence of the universal
approximation theorem implies that for all $\varepsilon>0$ there exists
$\tilde{h}_{\varepsilon}\in\mathcal{M}_{\text{net}}$ such that

$\|h-\tilde{h}_{\varepsilon}\|_{C^{2}(I\times\bar{B}_{2\widehat{M}}(0))}\leq\frac{\varepsilon}{2},$
(5.2)

where $\mathcal{M}_{\text{net}}=\text{span}\\{\sigma(\tilde{w}\cdot
x+\tilde{b}):\tilde{w}\in\mathbb{R}^{n+1},\,\tilde{b}\in\mathbb{R}\\}$, see eg
[21, Theorem 4.1], [13]. Let us observe that $\tilde{h}_{\varepsilon}$ can be
expressed as a residual network. Indeed, since

$\tilde{h}_{\varepsilon}=\sum_{i=1}^{M}\tilde{c}_{i}\sigma(\tilde{w}_{i}\cdot
x+\tilde{b}_{i})$

for some $M\in\mathbb{N}$,
$\tilde{w}_{i}\in\mathbb{R}^{n+1},\tilde{b}_{i},\tilde{c}_{i}\in\mathbb{R}$,
choosing $L_{\epsilon}=2,W_{11}\in\mathbb{R}^{M\times(n+1)}$ with rows
$\\{\tilde{w}_{i}\\}_{i=1}^{M}$,

$b_{1}=\text{col}(\tilde{b}_{1},\dots,\tilde{b}_{M}),\,W_{2}=(\tilde{c}_{1},\dots,\tilde{c}_{M}),\,W_{12}=0,$

we have $\tilde{h}_{\varepsilon}=f^{\sigma}_{2,\theta}\circ
f^{\sigma}_{1,\theta}$. Moreover, $\tilde{h}_{\varepsilon}\in
C^{4}(I\times\bar{B}_{2\widehat{M}})$. Following (5.1) we define

$\displaystyle
V^{\varepsilon}_{\theta_{\varepsilon}}(t,y)=\frac{1}{2}|Q_{2}(y-y^{T}_{d})|^{2}+\tilde{h}_{\varepsilon}(t,y)-\tilde{h}_{\varepsilon}(T,y)\in
C^{4}(I\times\bar{B}_{2\widehat{M}}).$

and estimate

$\displaystyle\|V^{\varepsilon}_{\theta_{\epsilon}}(t,y)-V^{*}(t,y)\|_{C^{2}}$
$\displaystyle=\|\tilde{h}_{\varepsilon}(t,y)-\tilde{h}_{\varepsilon}(T,y)+V^{*}(T,y)-V^{*}(t,y)\|_{C^{2}}$
$\displaystyle\leq 2\|h-\tilde{h}_{\varepsilon}\|_{C^{2}}\leq\varepsilon,$

where all norms are taken over $I\times\bar{B}_{2\widehat{M}}(0)$. This ends
the proof. ∎

###  5.2. Piecewise polynomials

Fix $\varepsilon_{0}>0$, and let $\varepsilon\in(0,\varepsilon_{0}]$ be
arbitrarily fixed. Throughout this subsection we assume $(\mathbf{A.2})$ and
in particular we shall make use of the global Lipschitz continuity of
$D^{2}V^{*}$ on $\bar{K}=\bar{I}\times\bar{B}_{2\widehat{M}}(0)$. Since
$\bar{K}$ is compact and hence totally bounded, there exist
$n_{\varepsilon}\in\mathbb{N}$ and
$\\{(\bar{t}_{i},\bar{y}^{i}_{0})\\}_{i=1}^{n_{\varepsilon}}\in\mathbb{R}^{n+1}$
such that

$\displaystyle\bar{K}\subset\bigcup^{n_{\varepsilon}}_{i=1}K_{i}\quad\text{where}\quad
K_{i}=B_{\varepsilon}((t_{i},\bar{y}^{i}_{0})).$

Note that we do not highlight the dependence of $(t_{i},\bar{y}^{i}_{0})$ and
$K_{i}$ on $\varepsilon$. For each $i$ define the parametrized polynomial

$\displaystyle
V^{\varepsilon}_{i}(A,b,c,t,y)=(t-\bar{t}_{i},y-\bar{y}^{i}_{0})^{\top}A(t-\bar{t}_{i},y-\bar{y}^{i}_{0})+b^{\top}(t-\bar{t}_{i},y-\bar{y}^{i}_{0})+c$

with

$\displaystyle(A,b,c,t,y)\in\operatorname{Sym}(n+1)\times\mathbb{R}^{n+1}\times\mathbb{R}\times\mathbb{R}^{n+1},$

where ${Sym}(n)$ denotes the space of real symmetric $n\times n$ matrices.
Note that $V^{\varepsilon}_{i}$ is infinitely many times differentiable in all
of its arguments.

For each $\varepsilon\in(0,\varepsilon_{0}]$ we define a special partition of
unity $\\{\varphi_{i}\\}_{i=1}^{n_{\varepsilon}}$ subordinate to $K_{i}$ with
$\varphi_{i}\colon\mathbb{R}\times\mathbb{R}^{n}\to[0,1]$, satisfying
$\mathcal{C}^{4}$ and

$\left\\{\begin{array}[]{c}\operatorname{supp}\varphi_{i}=\bar{K}_{i},\qquad~{}\sum^{n_{\varepsilon}}_{i=1}\varphi_{i}(t,y)=1,\,\forall(t,y)\in\bar{K},\\\\[7.3194pt]
\|D^{j}\varphi_{i}\|_{C(\bar{K}_{i}\cap\bar{K})}=\bar{\mu}\varepsilon^{-j},\forall
i=1,\dots,n_{\varepsilon},\;\text{ and }j\in\\{1,2\\},\\\\[7.3194pt]
\text{card }\\{i:\varphi_{i}(t,y)\neq
0\\}\leq\mathfrak{m}\quad\forall(t,y)\in\bar{K},\varepsilon\in(0,\varepsilon_{0}],\end{array}\right.$
(5.3)

with $\bar{\mu}$ and $\mathfrak{m}$ positive constants independent of
$i,(t,y)\in\bar{K},\varepsilon\in(0,\varepsilon_{0}]$. Finally we define

$\displaystyle\mathcal{R}_{\varepsilon}=\bigtimes^{n_{\varepsilon}}_{i=1}(\operatorname{Sym}(n+1)\times\mathbb{R}^{n+1}\times\mathbb{R}),$

and introduce the family of parameterized functions on $\mathbb{R}^{n+1}$ by

$\displaystyle
V^{\varepsilon}_{\theta}(t,y)=\frac{1}{2}|Q_{2}(y-y_{d}(T))|^{2}+\sum^{n_{\varepsilon}}_{i=1}\varphi_{i}(t,y)\left(V^{\varepsilon}_{i}(A_{i},b_{i},c_{i},t,y)-V^{\varepsilon}_{i}(A_{i},b_{i},c_{i},T,y)\right)$
(5.4)

for
$\theta=(A_{1},b_{1},c_{1},\dots,A_{n_{\varepsilon}},b_{n_{\varepsilon}},c_{n_{\varepsilon}})\in\mathcal{R}\varepsilon$.
Obviously we have
$V^{\varepsilon}_{\cdot}(\cdot)\in\mathcal{C}^{4}(\mathcal{R}_{\varepsilon}\times\mathbb{R}\times\mathbb{R}^{n})$
and

$\displaystyle
V^{\varepsilon}_{\theta}(T,y)=\frac{1}{2}|Q_{2}(y-y^{T}_{d})|^{2}\quad\forall
y\in\mathbb{R}^{n}.$

Thus the final time condition in the HJB equation is fulfilled. Next we show
that $V^{\varepsilon}_{\theta}$ satisfies the approximation property in
Assumption 2 for the particular choice of

$\begin{array}[]{ll}\theta_{\varepsilon}=&\big{(}\partial_{yy}V^{*}(\bar{t}_{1},\bar{y}^{1}_{0}),\partial_{y}V^{*}(\bar{t}_{1},\bar{y}^{1}_{0}),V^{*}(\bar{t}_{1},\bar{y}^{1}_{0}),\\\\[6.02773pt]
&\dots,\partial_{yy}V^{*}(\bar{t}_{n_{\varepsilon}},\bar{y}^{n_{\varepsilon}}_{0}),\partial_{y}V^{*}(\bar{t}_{n_{\varepsilon}},\bar{y}^{n_{\varepsilon}}_{0}),V^{*}(\bar{t}_{n_{\varepsilon}},\bar{y}^{n_{\varepsilon}}_{0})\big{)},\end{array}$
(5.5)

i.e. $V^{\varepsilon}_{i}$ in (5.4) are chosen with

$(\bar{A}_{i},\bar{b}_{i},\bar{c}_{i})=(\partial_{yy}V^{*}(\bar{t}_{i},\bar{y}^{i}_{0}),\partial_{y}V^{*}(\bar{t}_{i},\bar{y}^{i}_{0}),V^{*}(\bar{t}_{i},\bar{y}^{i}_{0}))\quad
i=1,\dots,n_{\varepsilon}.$ (5.6)

######  Theorem 3.

Let $V^{\varepsilon}$ and $\theta_{\varepsilon}$ be chosen according to (5.4)
and (5.5), respectively, and suppose that $(\mathbf{A.2})$ and (5.3) are
satisfied. Then Assumption 2 holds.

###### Proof.

We already argued that $V_{\theta}^{\varepsilon}$ has the desired regularity.
It remains to prove the required approximation capabilities. For abbreviation
set
$V^{\varepsilon}_{i}(t,y)=V^{\varepsilon}_{i}({\bar{A}_{i}},{\bar{b}_{i}},{\bar{c}_{i}},t,y)$,
with $({\bar{A}_{i}},{\bar{b}_{i}},{\bar{c}_{i}})$ as in (5.6).

Since $V^{\varepsilon}_{i}$ is the second order Taylor expansion of $V$ at
$(\bar{t}_{i},\bar{y}^{i}_{0})$ we conclude that

$\|V^{*}-V^{\varepsilon}_{i}\|_{C^{2-j}(\bar{K}_{i}\cap\bar{K})}\leq\bar{c}\varepsilon^{j+1},\quad\text{for
}j\in{0,1,2,}$ (5.7)

for some $\bar{c}>0$ depending on the global Lipschitz constant of $V$ on
$\bar{K}$, and independent of $\varepsilon\in(0,\varepsilon_{0}]$ and $i$.
Still recall that the sets $K_{i}$ depend on $\varepsilon$. To estimate
$V^{*}(t,y)-V^{\varepsilon}_{\theta}(t,y)$ we recall that
$V^{*}(T,y)=\frac{1}{2}|Q_{2}(y-y^{T}_{d})|^{2}$, and express $V^{*}(t,y)$ as
$V^{*}(t,y)=V^{*}(T,y)+V^{*}(t,y)-V^{*}(T,y)$. This leads to

$\begin{array}[]{l}V^{*}(t,y)-V^{\varepsilon}_{\theta}(t,y)\\\\[6.45831pt]
\;=\sum_{i\in\\{1,\dots,n_{\varepsilon}\\}}\varphi_{i}(t,y)(V^{*}(t,y)-V^{\varepsilon}_{i}(t,y))+\sum_{i\in\\{1,\dots,n_{\varepsilon}\\}}\varphi_{i}(T,y)\
(V^{*}(T,y)-V^{\varepsilon}_{i}(T,y)),\end{array}$

for $(t,y)\in\bar{K}$. From (5.3) and (5.3) we deduce that
$\|V(t,y)-V^{\varepsilon}_{\theta}(t,y)\|_{C(\bar{K})}\leq
2\bar{c}\varepsilon^{3}$.

For the gradient with respect to $y$ we proceed similarly. Fixing
$(t,y)\in\bar{K}$ we estimate

$\displaystyle|\partial_{y}V^{*}(t,y)$
$\displaystyle-\partial_{y}V^{\varepsilon}_{\theta_{\varepsilon}}(t,y)|\leq
D_{1}+D_{2}$

where

$\displaystyle D_{1}$
$\displaystyle=\sum_{i\in\\{1,\dots,n_{\varepsilon}\\}}\left[\varphi_{i}(t,y)|\partial_{y}V^{*}(t,y)-\partial_{y}V^{\varepsilon}_{i}(t,y)|+|V^{\varepsilon}_{i}(t,y)-V^{*}(t,y)||\partial_{y}\varphi_{i}(t,y)|\right]$
$\displaystyle D_{2}$
$\displaystyle=\sum_{i\in\\{1,\dots,n_{\varepsilon}\\}}\left[\varphi_{i}(T,y)|\partial_{y}V^{*}(T,y)-\partial_{y}V^{\varepsilon}_{i}(T,y)|+|V^{\varepsilon}_{i}(T,y)-V^{*}(T,y)||\partial_{y}\varphi_{i}(t,y)|\right].$

By (5.7) with $j=1$ the first terms in $D_{1}$ and $D_{2}$ can be estimated by
$\bar{c}\varepsilon^{2}$. Using (5.3) and (5.7) the second terms in $D_{1}$
and $D_{2}$ can be bounded by $\mathfrak{m}\bar{\mu}\varepsilon^{2}$.
Combining these estimate we arrive at

$\|\partial_{y}V^{*}(t,y)-\partial_{y}V^{\varepsilon}_{\theta}(t,y)\|_{C(\bar{K})}\leq
2\varepsilon^{2}(\bar{c}+\mathfrak{m}\,\bar{\mu}).$

In an analogous manner one can obtain a bound of the order $O(\varepsilon)$ on
the difference of the Hessians of $V$ and $V^{\varepsilon}_{\theta}$. This
finishes the proof. ∎

In Appendix A it is shown how standard mollifiers can be used so that (5.3) is
satisfied. This requires some extra attention due to the required bounds on
the derivatives of $\varphi_{i}$.

##  6\. Existence of minimizers to ($\mathcal{P}_{\varepsilon}$)

This section is devoted to proving the existence of minimizing triples to
($\mathcal{P}_{\varepsilon}$). Throughout this section $c$ will denote a
generic constant independent of $\varepsilon>0$ and $y_{0}\in Y_{0}$.

###  6.1. Existence of admissible points

Recall from Assumption 1 and Remark 1 that the optimal ensemble state
$\mathbf{y}^{*}\in\mathcal{C}(Y_{0};W_{T})$ satisfies
$\|{\mathbf{y}^{*}}\|_{{\mathcal{C}}}\leq M_{Y_{0}}$. Accordingly we define
the set of admissible states and admissible controls as

$\displaystyle\mathbf{Y}_{ad}=\left\\{\,\mathbf{y}\in\mathcal{C}(Y_{0};W_{T})\;|\;\|\mathbf{y}\|_{\mathcal{C}}\leq
2M_{Y_{0}}\,\right\\},\quad\mathbf{U}_{ad}\coloneqq
L^{2}(Y_{0};L^{2}(I;\mathbb{R}^{m})).$

We also recall the definition $\mathcal{Y}_{ad}$ in (3.7).

To prove the existence of minimizers to ($\mathcal{P}_{\varepsilon}$) we first
argue that the admissible set

$\displaystyle\mathcal{N}^{\varepsilon}_{ad}=\left\\{(\mathbf{y},\mathbf{p},\theta)\\!\in\\!\mathbf{Y}_{ad}\\!\times\\!\mathcal{C}(Y_{0};W_{T})\\!\times\\!\mathcal{R}_{\varepsilon}|(\mathbf{y},\mathbf{p},\theta)~{}\text{satisfies}\\!~{}\eqref{eq:statepropapprox}\\!-\\!\eqref{eq:constraintprop},\mathcal{F}^{\varepsilon}_{\theta}(\mathbf{y})\\!\in\\!\mathbf{U}_{ad}\right\\}$
(6.1)

is nonempty for $\varepsilon$ small enough. For this purpose consider the
family $\theta_{\varepsilon}\in\mathcal{R}_{\varepsilon}$,
$0<\varepsilon\leq\varepsilon_{0}$, from Assumption 2 as well as the
associated closed loop system of state and adjoint equations

$\displaystyle\dot{y}_{\varepsilon}$
$\displaystyle=\mathbf{f}(y_{\epsilon})+\mathbf{g}(y_{\varepsilon})\mathcal{F}^{\varepsilon}_{\theta_{\epsilon}}(y_{\epsilon}),$
(6.2) $\displaystyle-\dot{p}_{\varepsilon}$
$\displaystyle=D\mathbf{f}(y_{\varepsilon})^{\top}p_{\varepsilon}+[D\mathbf{g}(y_{\varepsilon})^{\top}\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(y_{\varepsilon})]p_{\varepsilon}+\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}(y_{\varepsilon}-y_{d}),$
(6.3)

subject to the following initial and terminal conditions

$\displaystyle
y_{\epsilon}(0)=y_{0},~{}p_{\varepsilon}(T)=Q^{\top}_{2}Q_{2}(y_{\varepsilon}(T)-y_{d}^{T}),$

for every $y_{0}\in Y_{0}$. We first prove the following approximation result.

######  Theorem 4.

Let Assumptions 1 and 2 hold. There exists a constant $c$ such that for all
$\varepsilon>0$ small enough and for all $y_{0}\in Y_{0}$ the system (6.2) and
(6.3) admits unique solutions
$y_{\varepsilon}=\mathbf{y}_{\varepsilon}(y_{0})\in\mathcal{Y}_{ad}$ and
$p_{\varepsilon}=\mathbf{p}_{\varepsilon}(y_{0})\in W_{T}$. Furthermore
$\mathbf{y}_{\varepsilon}\in\mathcal{C}^{1}(Y_{0};W_{T})$,
$\mathbf{p}_{\varepsilon}\in\mathcal{C}(Y_{0};W_{T})$, and
$\mathcal{F}^{*}(\mathbf{y}^{*})\in\mathcal{C}(Y_{0};L^{2}(I;\mathbb{R}^{m}))$
hold and

$\displaystyle\|\mathbf{y}_{\varepsilon}-\mathbf{y}^{*}\|_{\mathcal{C}^{1}(Y_{0};W_{T})}+\|\mathbf{p}_{\varepsilon}-\mathbf{p}^{*}\|_{\mathcal{C}(Y_{0};W_{T})}+\|\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(\mathbf{y}_{\varepsilon})-\mathcal{F}^{*}(\mathbf{y}^{*})\|_{\mathcal{C}(Y_{0};L^{2}(I;\mathbb{R}^{m}))}\leq
c\varepsilon.$

In particular,
$(\mathbf{y}_{\varepsilon},\mathbf{p}_{\varepsilon},\theta_{\varepsilon})\in\mathcal{N}^{\varepsilon}_{ad}$
for all $\varepsilon>0$ small enough.

In order to prove this we require several auxiliary results.

######  Lemma 5.

There exists a constant $c$ such that for all $\varepsilon$ small enough there
holds

$\displaystyle\|(\mathcal{F}^{*}(y_{1})-\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(y_{1}))-(\mathcal{F}^{*}(y_{2})-\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(y_{2}))\|_{L^{2}(I;\mathbb{R}^{m})}\leq
c\varepsilon\|y_{1}-y_{2}\|_{W_{T}},\quad\forall
y_{1},y_{2}\in\mathcal{Y}_{ad}.$

###### Proof.

According to the definition of $\mathcal{F}^{*}$ and
$\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}$ we split

$\displaystyle\|(\mathcal{F}^{*}(y_{1})-\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(y_{1}))-(\mathcal{F}^{*}(y_{2})-\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(y_{2}))\|_{L^{2}(I;\mathbb{R}^{m})}\leq
D_{1}+D_{2}$

with

$\displaystyle D_{1}$
$\displaystyle=1/\beta\,\|\mathbf{g}(y_{1})^{\top}\|_{\mathcal{B}(L^{2}(I;\mathbb{R}^{n}),L^{2}(I;\mathbb{R}^{m}))}\|\partial_{y}((\mathcal{V}^{*}(y_{1})\\!-\\!\mathcal{V}^{\varepsilon}_{\theta_{\varepsilon}}(y_{1}))-(\mathcal{V}^{*}(y_{2})\\!-\\!\mathcal{V}^{\varepsilon}_{\theta_{\varepsilon}}(y_{2})))\|_{L^{2}(I;\mathbb{R}^{n})}$
$\displaystyle D_{2}$
$\displaystyle=1/\beta\,\|\mathbf{g}(y_{1})^{\top}-\mathbf{g}(y_{2})^{\top}\|_{\mathcal{B}(L^{2}(I;\mathbb{R}^{n}),L^{2}(I;\mathbb{R}^{m}))}\|\partial_{y}(\mathcal{V}^{*}(y_{2})-\mathcal{V}^{\varepsilon}_{\theta_{\varepsilon}}(y_{2}))\|_{L^{2}(I;\mathbb{R}^{n})}.$

Applying the integral mean value theorem yields

$\displaystyle\|\partial_{y}((\mathcal{V}^{*}(y_{1})-\mathcal{V}^{\varepsilon}_{\theta_{\varepsilon}}(y_{1}))$
$\displaystyle-(\mathcal{V}^{*}(y_{2})-\mathcal{V}^{\varepsilon}_{\theta_{\varepsilon}}(y_{2})))\|_{L^{2}(I;\mathbb{R}^{n})}$
$\displaystyle\leq\sup_{s\in[0,1]}\|\partial_{yy}(\mathcal{V}^{*}(y_{1}+sh)-\mathcal{V}^{\varepsilon}_{\theta_{\varepsilon}}(y_{1}+sh))\|_{\mathcal{B}(W_{T},L^{2}(I;\mathbb{R}^{n}))}\|h\|_{W_{T}}$

with $h=y_{2}-y_{1}\in W_{T}$. Note that $y_{1}+sh\in\mathcal{Y}_{ad}$ for all
$s\in[0,1]$. Thus we can use Assumption 2 for every $s\in[0,1]$ and $\delta
y\in W_{\infty}$ and estimate

$\displaystyle\|\partial_{yy}(\mathcal{V}^{*}(y_{1}+sh)$
$\displaystyle-\mathcal{V}^{\varepsilon}_{\theta_{\varepsilon}}(y_{1}+sh))\delta
y\|_{L^{2}(I;\mathbb{R}^{n})}$
$\displaystyle\leq\sqrt{\int_{0}^{T}~{}|\partial_{yy}(V^{*}(t,y_{1}(t)+sh(t))-V^{\varepsilon}_{\theta_{\varepsilon}}(t,y_{1}(t)+sh(t)))|^{2}_{\mathbb{R}^{n\times
n}}|\delta y(t)|^{2}\mathrm{d}t}$ $\displaystyle\leq c\varepsilon\|\delta
y\|_{L^{2}(I;\mathbb{R}^{n})}\leq\varepsilon c\|\delta y\|_{W_{T}}.$

Similarly we obtain

$\displaystyle\|\partial_{y}(\mathcal{V}^{*}(y_{2})-\mathcal{V}^{\varepsilon}_{\theta_{\varepsilon}}(y_{2}))\|_{L^{2}(I;\mathbb{R}^{n})}=\sqrt{\int^{T}_{0}|\partial_{y}({V}^{*}(t,y_{2}(t))-{V}^{\varepsilon}_{\theta_{\varepsilon}}(t,y_{2}(t)))|^{2}~{}\mathrm{d}t}\leq\sqrt{T}c\varepsilon.$

Last recall that $\mathbf{g}$ is Lipschitz continuous and uniformly bounded on
$\mathcal{Y}_{ad}$. Combining these facts yields the desired statement. ∎

With the same arguments the following a priori estimate can be obtained. For
the sake of brevity its proof is omitted.

######  Corollary 6.

There exists a constant $c$ such that for all $\varepsilon$ small enough there
holds

$\displaystyle\|\mathcal{F}^{*}(y)-\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(y)\|_{L^{2}(I;\mathbb{R}^{m})}\leq
c\varepsilon\|y\|_{W_{T}},\quad\forall y\in\mathcal{Y}_{ad}.$

Next we establish existence of a unique solution to (6.2) as well as a first
approximation result.

######  Proposition 7.

Let Assumptions 1 and 2 hold. Then for all $\varepsilon>0$ small enough there
is a unique $\mathbf{y}_{\varepsilon}\in\mathcal{C}^{1}(Y_{0};W_{T})$ such
that
$y_{\varepsilon}\coloneqq\mathbf{y}_{\varepsilon}(y_{0})\in\mathcal{Y}_{ad}$
satisfies (6.2) for all $y_{0}\in Y_{0}$. Moreover there exists a constant $c$
independent of $\varepsilon$ such that

$\displaystyle\|\mathbf{y}^{*}-\mathbf{y}_{\varepsilon}\|_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{C}(Y_{0};W_{T})}}+\|\mathcal{F}^{*}(\mathbf{y}^{*})-\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(\mathbf{y}_{\varepsilon})\|_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{C}(Y_{0};L^{2}(I;\mathbb{R}^{m}))}}\leq
c\varepsilon.$

In particular we have
$\|\mathbf{y}_{\varepsilon}\|_{\mathcal{C}(Y_{0};W_{T})}\leq 2M_{Y_{0}}$ for
all sufficiently small $\varepsilon$.

###### Proof.

The proof is based on a fixed-point argument. Let $y_{0}\in Y_{0}$ be
arbitrary but fixed. Define the set

$\displaystyle\mathcal{M}=\left\\{\,y\in
W_{T}\;|\;\|y\|_{W_{T}}\leq\frac{3}{2}M_{Y_{0}}\,\right\\}\subset\mathcal{Y}_{ad}.$

On $\mathcal{M}$ we consider the mapping $\mathcal{Z}\colon\mathcal{M}\to
W_{T}$, where $z=\mathcal{Z}(y)\in\mathcal{Y}_{ad}$ is the unique solution of

$\displaystyle\dot{z}=\mathbf{f}(z)+\mathbf{g}(z)\mathcal{F}^{*}(z)+\mathbf{g}(y)\mathcal{F}^{\varepsilon}_{\theta_{\epsilon}}(y)-\mathbf{g}(y)\mathcal{F}^{*}(y),\quad
z(0)=y_{0}.$ (6.4)

It is well-defined since the perturbation function
$v=\mathbf{g}(y)\mathcal{F}^{\varepsilon}_{\theta_{\epsilon}}(y)-\mathbf{g}(y)\mathcal{F}^{*}(y)\in
L^{2}(I;\mathbb{R}^{n})$ satisfies

$\displaystyle\|v\|_{L^{2}}\leq\|\mathbf{g}(y)\|_{\mathcal{B}(L^{2}(I;\mathbb{R}^{m}),L^{2}(I;\mathbb{R}^{n}))}\|\mathcal{F}^{*}(y)-\mathcal{F}^{\varepsilon}_{\theta_{\epsilon}}(y)\|_{L^{2}}\leq\frac{3}{2}c\varepsilon
M_{Y_{0}}\|\mathbf{g}(y)\|_{\mathcal{B}(L^{2}(I;\mathbb{R}^{m}),L^{2}(I;\mathbb{R}^{n}))}$

where we use Corollary 6 and the definition of $\mathcal{M}$. Hence
$\|v\|_{L^{2}}\leq c\varepsilon$. Here and below $c$ denotes a generic
constant which is independent of $y_{0}\in Y_{0}$ and all $\epsilon>0$
sufficiently small. We may invoke Proposition 25 and Corollary 26 from the
Appendix, to assert the existence of a unique solution $z\in\mathcal{Y}_{ad}$
to (6.4) with

$\displaystyle\|z\|_{W_{T}}$ $\displaystyle\leq
M_{Y_{0}}+c\|v\|_{L^{2}}\leq\frac{3}{2}M_{Y_{0}},\quad\forall y_{0}\in Y_{0},$

if $\varepsilon>0$ is chosen small enough. From this we particularly conclude
$\mathcal{Z}(\mathcal{M})\subset\mathcal{M}$ for all $y_{0}\in Y_{0}$ and
$\varepsilon>0$ small. It remains to prove that $\mathcal{Z}$ is a
contraction. To this end let $y_{1},~{}y_{2}\in\mathcal{M}$ be given. Applying
Corollary 26 yields the first inequality in

$\displaystyle\|\mathcal{Z}(y_{1})-\mathcal{Z}(y_{2})\|_{W_{T}}$
$\displaystyle\leq
c\|\mathcal{F}^{*}(y_{1})-\mathcal{F}^{\varepsilon}_{\theta_{\epsilon}}(y_{1})-\mathcal{F}^{*}(y_{2})+\mathcal{F}^{\varepsilon}_{\theta_{\epsilon}}(y_{2})\|_{L^{2}}\leq
c\varepsilon\|y_{1}-y_{2}\|_{W_{T}}$

with a constant $c>0$ independent of $y_{1},y_{2}\in\mathcal{M}$ as well as of
$y_{0}\in Y_{0}$, and $\epsilon$ sufficiently small. The last inequality
follows from Lemma 5. Choosing $\varepsilon>0$ small enough we conclude that
$\mathcal{Z}$ admits a unique fixed point
$y_{\varepsilon}=\mathcal{Z}(y_{\varepsilon})\in W_{T}$ on $\mathcal{M}$.
Clearly, the function $\mathbf{y}_{\varepsilon}(y_{0}):=y_{\varepsilon}$
satisfies (6.2), $y_{\varepsilon}\in{\mathcal{M}}\subset\mathcal{Y}_{ad}$ as
well as

$\displaystyle\|\mathbf{y}_{\varepsilon}(y_{0})-\mathbf{y}^{*}(y_{0})\|_{W_{T}}$
$\displaystyle=\|\mathcal{Z}(\mathbf{y}_{\varepsilon}(y_{0}))-\mathcal{Z}(0)\|_{W_{T}}\leq
c\varepsilon\|y_{\varepsilon}\|_{W_{T}}\leq c\varepsilon\frac{3}{2}M_{Y_{0}},$

and by Corollary 6

$\displaystyle\|\mathcal{F}^{*}(\mathbf{y}^{*}(y_{0}))$
$\displaystyle-\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(\mathbf{y}_{\varepsilon}(y_{0}))\|_{L^{2}}$
$\displaystyle\leq\|\mathcal{F}^{*}(\mathbf{y}^{*}(y_{0}))-\mathcal{F}^{*}(\mathbf{y}_{\varepsilon}(y_{0}))\|_{L^{2}}+\|\mathcal{F}^{*}(\mathbf{y}_{\varepsilon}(y_{0}))-\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(\mathbf{y}_{\varepsilon}(y_{0}))\|_{L^{2}}$
$\displaystyle\leq
c\|\mathbf{y}^{*}(y_{0})-\mathbf{y}_{\varepsilon}(y_{0})\|_{W_{T}}+c\varepsilon\|\mathbf{y}_{\varepsilon}(y_{0})\|_{W_{T}}\leq
c\varepsilon.$

Finally according to Proposition 25 the solution
$\mathbf{y}_{\varepsilon}(y_{0})$ is unique and the mapping
$\mathbf{y}_{\varepsilon}$ is at least of class $\mathcal{C}^{1}$ . ∎

Next we estimate the $W^{1,2}$ difference between $\mathbf{y}_{\varepsilon}$
and $\mathbf{y}^{*}$.

######  Proposition 8.

The mapping $\mathbf{y}_{\varepsilon}\in\mathcal{C}^{1}(Y_{0};W_{T})$ from
Theorem 7 satisfies

$\displaystyle\|\mathbf{y}_{\varepsilon}-\mathbf{y}^{*}\|_{\mathcal{C}^{1}(Y_{0};W_{T})}\leq
c\varepsilon$

for $c>0$ independently of $\varepsilon$ small enough.

###### Proof.

By the previous proposition the estimate is already known for
${\mathcal{C}^{1}(Y_{0};W_{T})}$ replaced by ${\mathcal{C}(Y_{0};W_{T})}$. Now
fix $y_{0}\in Y_{0}$ and $i\in\\{1,\dots,n\\}$. By the inverse mapping theorem
the partial derivatives of $\mathbf{y}^{*}$ and $\mathbf{y}_{\varepsilon}$ at
$y_{0}$ are given by
$\partial_{i}\mathbf{y}^{*}(y_{0})=T_{*}(y_{0})^{-1}(0,e_{i})$,
$\partial_{i}\mathbf{y}_{\varepsilon}(y_{0})=T_{\varepsilon}(y_{0})^{-1}(0,e_{i})$.
Here, $e_{i}$ denotes the i-th canonical basis vector in $\mathbb{R}^{n}$ and

$\displaystyle T_{*}(y_{0})^{-1},T_{\varepsilon}(y_{0})^{-1}\colon
L^{2}(I;\mathbb{R}^{n})\times\mathbb{R}^{n}\to W_{T}$

denote the linear continuous inverses of

$\displaystyle T_{*}(y_{0})\delta
y\\!=\\!\left(\begin{array}[]{c}\\!\\!\\!\dot{\delta
y}-D\mathbf{f}(\mathbf{y}^{*}(y_{0}))\delta
y-[D\mathbf{g}(\mathbf{y}^{*}(y_{0}))\delta
y]\mathcal{F}^{*}(\mathbf{y}^{*}(y_{0}))-\mathbf{g}(\mathbf{y}^{*}(y_{0}))D\mathcal{F}^{*}(\mathbf{y}^{*}(y_{0}))\delta
y\\\ \delta y(0)\\\ \end{array}\\!\\!\\!\right)$

and

$\displaystyle T_{\varepsilon}(y_{0})\delta
y\\!=\\!\left(\begin{array}[]{c}\\!\\!\\!\dot{\delta
y}-D\mathbf{f}(\mathbf{y}_{\varepsilon}(y_{0}))\delta
y-[D\mathbf{g}(\mathbf{y}_{\varepsilon}(y_{0}))\delta
y]\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(\mathbf{y}_{\varepsilon}(y_{0}))-\mathbf{g}(\mathbf{y}_{\varepsilon}(y_{0}))D\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(\mathbf{y}_{\varepsilon}(y_{0}))\delta
y\\\ \delta y(0)\\\ \end{array}\\!\\!\\!\right).$

Using Gronwall’s inequality, we readily verify that

$\displaystyle\max\left\\{\|T_{\varepsilon}(y_{0})^{-1}(\delta v,\delta
y_{0})\|_{W_{T}},\|T_{*}(y_{0})^{-1}(\delta v,\delta
y_{0})\|_{W_{T}}\right\\}\leq C(\|\delta v\|_{L^{2}(I;\mathbb{R}^{n})}+|\delta
y_{0}|_{\mathbb{R}^{n}})$ (6.5)

for all $\delta v\in L^{2}(I,\mathbb{R}^{n}),~{}\delta
y_{0}\in\mathbb{R}^{n}$, $y_{0}\in Y_{0}$ and some $C>0$ independent of
$y_{0},\delta v,\delta y_{0}$. Now we recall that
$\mathbf{y}_{\varepsilon}(y_{0}),\mathbf{y}^{*}(y_{0})\in\mathcal{Y}_{ad}$ and
that $D\mathbf{f},D\mathbf{g},\mathbf{g}$ are Lipschitz continuous, and thus
in particular bounded, on $\mathcal{Y}_{ad}$, see Assumption 1 $\mathbf{A.1}$.
Together with boundedness of
$\\{\|\mathcal{F}^{*}(\mathbf{y}(y_{0}))\|_{L^{2}}:y_{0}\in Y_{0}\\}$,
Corollary 6 and Theorem 7 we conclude

$\displaystyle\|(T_{*}(y_{0})-T_{\varepsilon}(y_{0}))\delta
y\|_{L^{2}(I;\mathbb{R}^{n})\times\mathbb{R}^{n}}\leq c\varepsilon\|\delta
y\|_{W_{T}}\quad\forall\delta y\in W_{T}$ (6.6)

for some $c>0$ again independent of $y_{0}\in Y_{0}$. Recalling that
$B^{-1}-A^{-1}=A^{-1}(A-B)B^{-1}$ for invertible bounded linear operators $A$
and $B$, we obtain

$\displaystyle\|\partial_{i}\mathbf{y}_{\varepsilon}(y_{0})-\partial_{i}\mathbf{y}^{*}(y_{0})\|_{W_{T}}$
$\displaystyle=\|T_{\varepsilon}(y_{0})^{-1}(0,e_{i})-T_{*}(y_{0})^{-1}(0,e_{i})\|_{W_{T}}$
$\displaystyle\leq C^{2}\sup_{\|\delta y\|_{W_{T}}\leq
1}\|(T_{*}(y_{0})-T_{\varepsilon}(y_{0}))\delta
y\|_{L^{2}(I;\mathbb{R}^{n})\times\mathbb{R}^{n}}\leq c\varepsilon,$

where $C>0$ is the constant from (6.5). Since all involved constants are
independent of $y_{0}\in Y_{0}$ we obtain the desired estimate
$\|\partial_{i}\mathbf{y}_{\varepsilon}-\partial_{i}\mathbf{y}^{*}\|_{\mathcal{C}}\leq
c\varepsilon.$ ∎

Next we address the solvability of the adjoint equation (4.5).

######  Proposition 9.

There exists a constant $c$ such that for all $\varepsilon$ small enough there
exists $\mathbf{p}_{\varepsilon}\in\mathcal{C}(Y_{0};W_{T})$ such that
$p_{\varepsilon}\coloneqq\mathbf{p}_{\varepsilon}(y_{0})\in W_{T}$ satisfies
(6.3) for all $y_{0}\in Y_{0}$ and

$\displaystyle\|\mathbf{p}_{\varepsilon}-\mathbf{p}^{*}\|_{\mathcal{C}}\leq
c\varepsilon.$

###### Proof.

Given $y\in\mathcal{Y}_{ad}$ consider the linear ordinary differential
equation

$\displaystyle-\dot{p}=D\mathbf{f}(y)p+[D\mathbf{g}(y)^{\top}\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(y)]p+\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}(y-y_{d}),~{}p(T)=Q^{\top}_{2}Q_{2}(y(T)-y^{T}_{d}).$

It admits a unique solution $p=P(y)\in W_{T}$ which is bounded independently
of $y\in\mathcal{Y}_{ad}$. Moreover the mapping $P\colon W_{T}\to W_{T}$ is
continuous on $\mathcal{Y}_{ad}$ in virtue of the Gronwall lemma and
Assumption 1. The existence of a mapping $\mathbf{p}_{\varepsilon}$ which
satisfies (6.3) then follows by setting
$\mathbf{p}_{\varepsilon}=P\circ\mathbf{y}_{\varepsilon}$.

It remains to prove the estimate for the difference between
$\mathbf{p}_{\varepsilon}$ satisfying (6.3) and $\mathbf{p}^{*}$ satisfying
(3.10). For this purpose we can use the same technique as in the proof of
Proposition 8 and therefore we only give the main estimates. Recall that
$D\mathbf{f}(\cdot)^{\top},D\mathbf{g}(\cdot)^{\top}$ are Lipschitz continuous
on $\mathcal{Y}_{ad}$. The the most involved term in the estimate analogous to
(6.6) is

$\displaystyle\|[D\mathbf{g}(\mathbf{y}_{\varepsilon}(y_{0}))^{\top}$
$\displaystyle\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(\mathbf{y}_{\varepsilon}(y_{0}))-D\mathbf{g}(\mathbf{y}^{*}(y_{0}))^{\top}\mathcal{F}^{*}(\mathbf{y}^{*}(y_{0}))]\delta
p\|_{L^{2}}$ $\displaystyle\leq
c(\|\mathbf{y}_{\varepsilon}(y_{0})-\mathbf{y}^{*}(y_{0})\|_{W_{T}}+\|\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(\mathbf{y}_{\varepsilon}(y_{0}))-\mathcal{F}^{*}(\mathbf{y}^{*}(y_{0}))\|_{L^{2}})\|\delta
p\|_{W_{T}}$

with $c>0$ independent of $\varepsilon>0$ and $\delta p\in W_{T}$. Now a
perturbation argument as in the proof of Propostion 8 provides us with

$\displaystyle\|$
$\displaystyle\mathbf{p}_{\varepsilon}(y_{0})-\mathbf{p}^{*}(y_{0})\|_{W_{T}}$
$\displaystyle\leq
c(\|\mathbf{y}_{\varepsilon}(y_{0})-\mathbf{y}^{*}(y_{0})\|_{W_{T}}+\|\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(\mathbf{y}_{\varepsilon}(y_{0}))-\mathcal{F}^{*}(\mathbf{y}^{*}(y_{0}))\|_{L^{2}}+|\mathbf{y}_{\varepsilon}(y_{0})(T)-\mathbf{y}^{*}(y_{0})(T)|)$
$\displaystyle\leq
c(\|\mathbf{y}_{\varepsilon}(y_{0})-\mathbf{y}^{*}(y_{0})\|_{W_{T}}+\|\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(\mathbf{y}_{\varepsilon}(y_{0}))-\mathcal{F}^{*}(\mathbf{y}^{*}(y_{0}))\|_{L^{2}})\leq
c\varepsilon$

where $W_{T}\hookrightarrow\mathcal{C}(\bar{I};\mathbb{R}^{n})$ is used in the
second inequality, and Proposition 8 and Corollary 6 are utilized in the final
one. Since all involved constants are again independent of $y_{0}\in Y_{0}$,
this finishes the proof. ∎

Summarizing all previous observations we arrive at the proof of Theorem 4.

###### Proof of Theorem 4.

This follows directly by combining Proposition 7, Proposition 8, and
Proposition 9. ∎

###  6.2. Closedness of $\mathcal{N}^{\varepsilon}_{ad}$

As a last prerequisite for proving existence to ($\mathcal{P}_{\varepsilon}$)
we argue that the admissible set $\mathcal{N}^{\varepsilon}_{ad}$ is closed.
The existence of at least one minimizing triple to
($\mathcal{P}_{\varepsilon}$) then follows by variational arguments. From here
on we always assume that $\mathcal{N}^{\varepsilon}_{ad}$ from (6.1) is
nonempty, i.e. that $\varepsilon$ is sufficiently small.

######  Proposition 10.

Let
$(\mathbf{y}_{k},\mathbf{p}_{k},\theta_{k})_{k\in\mathbb{N}}\subset\mathcal{N}^{\varepsilon}_{ad}$
be a sequence with weak limit $(\mathbf{y},\mathbf{p},\theta)$ in
$L^{2}(Y_{0};W_{T})^{2}\times\mathcal{R}_{\varepsilon}$. Then
$(\mathbf{y},\mathbf{p},\theta)\in\mathcal{N}^{\varepsilon}_{ad}$ and we have

$\displaystyle(\mathbf{y},\mathbf{p})\in\mathcal{C}(Y_{0};W_{T})^{2},~{}\lim_{k\rightarrow\infty}\mathbf{y}_{k}(y_{0})=\mathbf{y}(y_{0})\text{
and}~{}\lim_{k\rightarrow\infty}\mathbf{p}_{k}(y_{0})=\mathbf{p}(y_{0})\text{
in }W_{T},\quad\forall~{}y_{0}\in Y_{0}.$

The proof builds upon the following two lemmas.

######  Lemma 11.

Let the sequence
$(\mathbf{y}_{k},\mathbf{p}_{k},\theta_{k})_{k\in\mathbb{N}}\subset\mathcal{N}^{\varepsilon}_{ad}$
satisfy the prerequisites of Proposition 10. Then
$\mathbf{y}\in\mathbf{Y}_{ad}$,
$\mathbf{y}_{k}(y_{0})\rightarrow\mathbf{y}(y_{0})\text{ in}~{}W_{T},$
$\mathcal{F}^{\varepsilon}_{\theta_{k}}(\mathbf{y}_{k}(y_{0}))\rightarrow\mathcal{F}^{\varepsilon}_{\theta}({\mathbf{y}}(y_{0}))~{}\text{in}~{}L^{\infty}(I;\mathbb{R}^{m})$,
and

$\displaystyle\dot{\mathbf{y}}(y_{0})=\mathbf{f}({\mathbf{y}}(y_{0}))+\mathbf{g}({\mathbf{y}}(y_{0}))\mathcal{F}^{\varepsilon}_{\theta}({\mathbf{y}}(y_{0})),~{}{\mathbf{y}}(y_{0})(0)=y_{0},$
(6.7)

for all $y_{0}\in Y_{0}$.

###### Proof.

By assumption we have $\mathbf{y}_{k}\in\mathbf{Y}_{ad}$, and hence
$\|\mathbf{y}_{k}(y_{0})\|_{W_{T}}\leq 2M_{Y_{0}}$ for all $k\in\mathbb{N}$
and $y_{0}\in Y_{0}$, and $\mathbf{y}_{k}\in\mathcal{C}^{1}(Y_{0};W_{T})$ for
all $k\in\mathbb{N}$, see Proposition 7. Let us fix an arbitrary $y_{0}\in
Y_{0}$. and set $y_{k}\coloneqq\mathbf{y}_{k}(y_{0})$ for abbreviation. Then
there exists a subsequence, denoted by the same index, and $\tilde{y}\in
W_{T}$ such that $y_{k}\rightharpoonup\tilde{y}$ in $W_{T}$. Since
$W_{T}\hookrightarrow_{c}\mathcal{C}(\bar{I};\mathbb{R}^{n})\hookrightarrow
L^{p}(I;\mathbb{R}^{n})$, $1\leq p\leq+\infty$, we immediately get

$\displaystyle
y_{k}(0)\rightarrow\tilde{y}(0)~{}\text{in}~{}\mathbb{R}^{n},~{}\mathbf{f}(y_{k})\rightarrow\mathbf{f}(\tilde{y})~{}\text{in}~{}L^{2}(I;R^{n}),~{}\mathbf{g}(y_{k})\rightarrow\mathbf{g}(\tilde{y})~{}\text{in}~{}\mathcal{B}(L^{2}(I;\mathbb{R}^{m}),L^{2}(I;\mathbb{R}^{n}))$

as well as
$\mathcal{F}^{\varepsilon}_{\theta}(y_{k})\rightarrow\mathcal{F}^{\varepsilon}_{\theta}(\tilde{y})~{}\text{in}~{}L^{\infty}(I;\mathbb{R}^{m}).$
Moreover by Assumption 2 for every $\delta>0$ there exits
$K_{\delta}\in\mathbb{N}$ such that

$\displaystyle|\partial_{y}V^{\varepsilon}_{\theta_{k}}(t,y)-\partial_{y}V^{\varepsilon}_{\theta}(t,y)|\leq\delta\quad\forall(t,y)\in\bar{I}\times\bar{B}_{2\widehat{M}}(0)$
(6.8)

for all $k\geq K_{\delta}$. Here $\widehat{M}$ denotes the constant from
Assumption 1 $\mathbf{A.2}$. For all such $k$ we get utilizing (6.8) for a
constant $c$ independent of $k$

$\displaystyle\|\mathcal{F}^{\varepsilon}_{\theta_{k}}(y_{k})-\mathcal{F}^{\varepsilon}_{\theta}(\tilde{y})\|_{L^{\infty}}$
$\displaystyle\leq
c\|\mathcal{F}^{\varepsilon}_{\theta_{k}}(y_{k})-\mathcal{F}^{\varepsilon}_{\theta}(y_{k})\|_{L^{\infty}}+\|\mathcal{F}^{\varepsilon}_{\theta}(y_{k})-\mathcal{F}^{\varepsilon}_{\theta}(\tilde{y})\|_{L^{\infty}}$
$\displaystyle\leq
c\delta+\|\mathcal{F}^{\varepsilon}_{\theta}(y_{k})-\mathcal{F}^{\varepsilon}_{\theta}(\tilde{y})\|_{L^{\infty}}.$

This implies that
$\lim_{k\to\infty}\mathcal{F}^{\varepsilon}_{\theta_{k}}(y_{k})=\mathcal{F}^{\varepsilon}_{\theta}(\tilde{y})$
in $L^{\infty}(I;\mathbb{R}^{m})$. These observations imply

$\displaystyle\dot{y}_{k}=\mathbf{f}(y_{k})+\mathbf{g}(y_{k})\mathcal{F}^{\varepsilon}_{\theta_{k}}(y_{k})\rightarrow\mathbf{f}(\tilde{y})+\mathbf{g}(\tilde{y})\mathcal{F}^{\varepsilon}_{\theta}(\tilde{y}).$

Together with $y_{k}\rightharpoonup\tilde{y}$ in $W_{T}$ this implies that
$\mathbf{y}_{k}(y_{0})=y_{k}\rightarrow\tilde{y}$ in $W_{T}$ and

$\displaystyle\dot{\tilde{y}}=\mathbf{f}(\tilde{y})+\mathbf{g}(\tilde{y})\mathcal{F}^{\varepsilon}_{\theta}(\tilde{y}),~{}\tilde{y}(0)=y_{0}.$
(6.9)

Since the solution to this equation is unique, every weak accumulation point
of $y_{k}$ satisfies (6.9) and we have
$\mathbf{y}_{k}(y_{0})\rightarrow\tilde{y}~{}\text{in}~{}W_{T}$ for the whole
sequence. We repeat this construction for all $y_{0}\in Y_{0}$. This defines a
function $\tilde{\bf{y}}:Y_{0}\to W_{T}$ such that
$\mathbf{y}_{k}(y_{0})\rightarrow\tilde{\bf{y}}(y_{0})~{}\text{in}~{}W_{T}$
and such that (6.9) is satisfied with $\tilde{y}=\tilde{\bf{y}}(y_{0})$ for
each $y_{0}\in Y_{0}$. By Proposition 7 it is the unique solution to (6.7).

Lebesgue’s dominated convergence theorem for Bochner integrals [7, pg 45]
implies that $\mathbf{y}_{k}\rightarrow\tilde{\bf{y}}$ in
$L^{1}(Y_{0};W_{T})$, and by boundedness of
$\\{\|\mathbf{y}_{k}\|_{\mathcal{C}}\\}_{k=1}^{\infty}$ also in
$L^{2}(Y_{0};W_{T})$. By assumption ${\bf{y}}_{k}$ converges weakly in
$L^{2}(Y_{0};W_{T})$ to $\bf{y}$. Thus we have $\bf{y}=\tilde{\bf{y}}$.
Moreover $\|\mathbf{y}\|_{\mathcal{C}}\leq 2M_{Y_{0}}$ and hence
$\mathbf{y}\in\mathbf{Y}_{ad}$. ∎

Next we consider the behavior of the adjoint states $\mathbf{p}_{k}$.

######  Lemma 12.

Let
$(\mathbf{y}_{k},\mathbf{p}_{k},\theta_{k})_{k\in\mathbb{N}}\subset\mathcal{N}^{\varepsilon}_{ad}$
be a sequence with weak limit $(\mathbf{y},\mathbf{p},\theta)$ satisfying the
prerequisites of Proposition 10. Then $\|\mathbf{p}_{k}\|_{\mathcal{C}}\leq C$
for some $C>0$ and all $k\in\mathbb{N}$ large enough, and
$\mathbf{p}\in\mathcal{C}(Y_{0};W_{T})$. Moreover
$\mathbf{p}_{k}(y_{0})\rightarrow\mathbf{p}(y_{0})~{}\text{in}~{}W_{T}$, and

$\begin{array}[]{ll}-\dot{\mathbf{p}}(y_{0})=D\mathbf{f}(\mathbf{y}(y_{0}))^{\top}\mathbf{p}(y_{0})+[D\mathbf{g}(\mathbf{y}(y_{0}))^{\top}\mathcal{F}^{\varepsilon}_{\theta}(\mathbf{y}(y_{0}))]{\mathbf{p}}(y_{0})+\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}(\mathbf{y}(y_{0})-y_{d}),\\\\[6.88889pt]
\;\mathbf{p}(y_{0}))(T)=Q^{\top}_{2}Q_{2}(\mathbf{y}(T)(y_{0})-y^{T}_{d}),\end{array}$
(6.10)

for all $y_{0}\in Y_{0}$.

###### Proof.

From Lemma 11 recall that for the sequences
$~{}y_{k}\coloneqq\mathbf{y}_{k}(y_{0})\in\mathcal{Y}_{ad}$ and
$y\coloneqq\mathbf{y}(y_{0})$ we have for each $y_{0}\in Y_{0}$

$\displaystyle y_{k}\rightarrow
y~{}\text{in}~{}W_{T},~{}\mathcal{F}^{\varepsilon}_{\theta_{k}}(y_{k})\rightarrow\mathcal{F}^{\varepsilon}_{\theta}(y)~{}\text{in}~{}L^{\infty}(I;\mathbb{R}^{m}).$

Further for each $k\in\mathbb{N}$ and $y_{0}\in Y_{0}$, the element
$p_{k}\coloneqq\mathbf{p}_{k}(y_{0})\in W_{T}$ satisfies

$\displaystyle-\dot{p}_{k}=D\mathbf{f}(y_{k})^{\top}p_{k}+[D\mathbf{g}(y_{k})^{\top}\mathcal{F}^{\varepsilon}_{\theta_{k}}(y_{k})]p_{k}+\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}(y_{k}-y_{d}),~{}p_{k}(T)=Q^{\top}_{2}Q_{2}(y_{k}(T)-y^{T}_{d}).$
(6.11)

Recall from Assumption 2 that $\partial_{y}V_{\cdot}^{\varepsilon}$ is
uniformly continuous on compact sets. Thus for every $\delta>0$ there is
$K_{\delta}\in\mathbb{N}$ such that

$\displaystyle|F^{\varepsilon}_{\theta_{k}}(t,x)|\leq|F^{\varepsilon}_{\theta_{k}}(t,x)-F^{\varepsilon}_{\theta}(t,x)|+|F^{\varepsilon}_{\theta}(t,x)|\leq\delta+\max_{(t,x)\in
I\times\bar{B}_{2\widehat{M}}(0)}|F^{\varepsilon}_{\theta}(t,x)|<\infty$

for all $(t,x)\in I\times\bar{B}_{2\widehat{M}}(0)$ and $k\geq K_{\delta}$.
Consequently we obtain

$\displaystyle\sup_{k\geq K_{\delta}}\max_{(t,x)\in
I\times\bar{B}_{2\widehat{M}}(0)}\|A_{k}(t,x)\|_{\mathbb{R}^{n\times
n}}<\infty,\text{ where
}A_{k}(t,x)=Df(t,x)^{\top}+Dg(t,x)^{\top}F^{\varepsilon}_{\theta_{k}}(t,x).$

Applying Proposition 24 to the time-reversed equation (6.11) implies that

$\displaystyle\|p_{k}\|_{W_{T}}\leq
c\left(\|\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}(y_{k}-{y}_{d})\|_{L^{2}}+|y_{k}(T)-y^{T}_{d}|\right)$

for some $c>0$ independent of $y_{0}\in Y_{0}$ and all sufficiently large $k$.
Since $\|\mathbf{y}_{k}\|_{\mathcal{C}}\leq 2M_{Y_{0}}$ we finally conclude
$\|\mathbf{p}_{k}\|_{\mathcal{C}}\leq C$ for some $C>0$ independent of $k$
sufficiently large. We are now prepared to pass to the limit in (6.11). For
this purpose we proceed as in the proof of Lemma 11 and use

$\displaystyle
D\mathbf{f}(y_{k})+D\mathbf{g}(y_{k})^{\top}\mathcal{F}^{\varepsilon}_{\theta_{k}}(y_{k})\rightarrow
D\mathbf{f}(y)+D\mathbf{g}(y)^{\top}\mathcal{F}^{\varepsilon}_{\theta}(y)~{}\text{in}~{}\mathcal{B}(L^{2}(Y;\mathbb{R}^{n})),$

as well as

$\displaystyle\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}(y_{k}-{y}_{d})\rightarrow\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}(y-{y}_{d})~{}\text{in}~{}L^{2}(I;\mathbb{R}^{n}),$

and

$\displaystyle Q^{\top}_{2}Q_{2}(y_{k}(T)-y^{T}_{d})\rightarrow
Q^{\top}_{2}Q_{2}(y(T)-y^{T}_{d})~{}\text{in}~{}\mathbb{R}^{n}$

to show that every weak accumulation point $\tilde{p}\in W_{T}$ of $p_{k}$ is
in fact a strong accumulation point and satisfies the differential equation in
(6.10). Since the solution to this equation is unique we get
$p_{k}\rightarrow\tilde{p}$ in $W_{T}$ for the whole sequence. Finally
utilizing $\|\mathbf{p}_{k}\|_{\mathcal{C}}\leq C$ and Lebesgue’s dominated
convergence theorem we conclude $\tilde{p}=\mathbf{p}(y_{0})$ for all
$y_{0}\in Y_{0}$. ∎

###### Proof of Proposition 10.

This is a direct consequence of Lemma 11 and Lemma 12. ∎

###  6.3. Existence of minimizers

Finally we prove the existence of at least one minimizing triplet to
($\mathcal{P}_{\varepsilon}$).

######  Theorem 13.

Let Assumption 1 and 2 hold. Then for all $\varepsilon>0$ small enough,
Problem ($\mathcal{P}_{\varepsilon}$) admits at least one minimizing triplet
$(\mathbf{y}^{*}_{\varepsilon},\mathbf{p}^{*}_{\varepsilon},\theta^{*}_{\varepsilon})\in\mathcal{C}(Y_{0};W_{T})^{2}\times\mathcal{R}_{\varepsilon}$.

###### Proof.

According to Theorem 4, the admissible set $\mathcal{N}^{\varepsilon}_{ad}$ is
nonempty for $\varepsilon>0$ small enough. Fix such a $\varepsilon>0$ and let
$(\mathbf{y}_{k},\mathbf{p}_{k},\theta_{k})\in\mathcal{N}^{\varepsilon}_{ad}$
denote a minimizing sequence for $\mathcal{J}_{\varepsilon}$ i.e.

$\displaystyle\mathcal{J}_{\varepsilon}(\mathbf{y}_{k},\mathbf{p}_{k},\theta_{k})\rightarrow\inf_{(\mathbf{y},\mathbf{p},\theta)\in\mathcal{N}^{\varepsilon}_{ad}}\mathcal{J}_{\varepsilon}(\mathbf{y},\mathbf{p},\theta).$

Since $\mathbf{y}_{k}\in\mathbf{Y}_{ad}$ and
$\frac{\gamma_{\varepsilon}}{2}\|\theta_{k}\|^{2}_{\mathcal{R}_{\varepsilon}}\leq\mathcal{J}_{\varepsilon}(\mathbf{y}_{k},\mathbf{p}_{k},\theta_{k}),$
for all $k\in\mathbb{N}$, the sequence $\\{(\mathbf{y}_{k},\theta_{k})\\}\in
L^{2}(Y_{0};W_{T})\times\mathcal{R}_{\varepsilon}$ is bounded. Thus it admits
at least one subsequence, denoted by the same index, with

$\displaystyle(\mathbf{y}_{k},\theta_{k})\rightharpoonup(\mathbf{y}^{*}_{\varepsilon},\theta^{*}_{\varepsilon})~{}\text{in}~{}L^{2}(Y_{0};W_{T})\times\mathcal{R}_{\varepsilon}$

for some $(\mathbf{y}^{*}_{\varepsilon},\theta^{*}_{\varepsilon})$. As in the
proof of Lemma 12 we verify that $\|\mathbf{y}_{k}\|_{\mathcal{C}}\leq C$ and
$\|\mathbf{p}_{k}\|_{\mathcal{C}}\leq C$ for some $C>0$ independent of
$k\in\mathbb{N}$. Consequently, by possibly taking another subsequence we
arrive at

$\displaystyle(\mathbf{y}_{k},\mathbf{p}_{k},\theta_{k})\rightharpoonup(\mathbf{y}^{*}_{\varepsilon},\mathbf{p}^{*}_{\varepsilon},\theta^{*}_{\varepsilon})~{}\text{in}~{}L^{2}(Y_{0};W_{T})^{2}\times\mathcal{R}_{\varepsilon}$

for some
$(\mathbf{y}^{*}_{\varepsilon},\mathbf{p}^{*}_{\varepsilon},\theta^{*}_{\varepsilon})\in\mathcal{N}^{\varepsilon}_{ad}$.
For the following estimates it will be convenient to recall the augmented
functional $J_{\varepsilon}$, see (4.3), which arises in the running cost of
($\mathcal{P}_{\varepsilon}$) in compact form:

$\displaystyle
J_{\varepsilon}(y,p,\theta)=J(y,\mathcal{F}^{\varepsilon}_{\theta}(y))+\frac{\gamma_{1}}{2}\|\mathcal{V}(y)-J_{\bullet}(y,\mathcal{F}^{\varepsilon}_{\theta^{*}_{\varepsilon}}(y))\|^{2}_{L^{2}(I;\mathbb{R})}+\frac{\gamma_{2}}{2}\|p-\partial_{y}\mathcal{V}(y)\|^{2}_{L^{2}(I;\mathbb{R}^{n})},$
(6.12)

where $J_{t}$ was defined below (3.4). Now fix an arbitrary $y_{0}\in Y_{0}$
and set

$\displaystyle
y_{k}\coloneqq\mathbf{y}_{k}(y_{0}),~{}p_{k}\coloneqq\mathbf{p}_{k}(y_{0}),~{}{y^{*}}\coloneqq\mathbf{y}^{*}_{\varepsilon}(y_{0}),~{}p\coloneqq\mathbf{p}^{*}_{\varepsilon}(y_{0}).$

From Lemma 11 and Lemma 12 we get

$\displaystyle y_{k}\rightarrow\tilde{y},~{}p_{k}\rightarrow
p~{}\text{in}~{}W_{T},~{}\mathcal{F}^{\varepsilon}_{\theta_{k}}(y_{k})\rightarrow\mathcal{F}^{\varepsilon}_{\theta^{*}_{\varepsilon}}(\tilde{y})~{}\text{in}~{}L^{2}(I;\mathbb{R}^{n})$

and, again using the uniform continuity of $V_{\bullet}^{\varepsilon}$ and
$\partial_{y}V_{\bullet}^{\varepsilon}$, we conclude

$\displaystyle\mathcal{V}^{\varepsilon}_{\theta_{k}}(y_{k})\rightarrow\mathcal{V}^{\varepsilon}_{\theta^{*}_{\varepsilon}}(\tilde{y})~{}\text{in}~{}L^{2}(I),~{}\partial_{y}\mathcal{V}^{\varepsilon}_{\theta_{k}}(y_{k})\rightarrow\partial_{y}\mathcal{V}^{\varepsilon}_{\theta^{*}_{\varepsilon}}(\tilde{y})~{}\text{in}~{}L^{2}(I;\mathbb{R}^{n}),$

as well as the uniform boundedness of
$\mathcal{V}^{\varepsilon}_{\theta_{k}}(\mathbf{y}_{k})$ and
$\partial_{y}\mathcal{V}^{\varepsilon}_{\theta_{k}}(\mathbf{y}_{k})$ in
$\mathcal{C}(Y_{0};L^{2}(I))$ and
$\mathcal{C}(Y_{0};L^{2}(I;\mathbb{R}^{n}))$, respectively. Moreover we
readily verify that

$\displaystyle|J_{t}(y_{k},\mathcal{F}^{\varepsilon}_{\theta_{k}}(y_{k}))\\!-\\!J_{t}(\tilde{y},\mathcal{F}^{\varepsilon}_{\theta^{*}_{\varepsilon}}(\tilde{y}))|\\!\leq\\!c\\!\left(\|y_{k}-\tilde{y}\|_{L^{2}}+\|\mathcal{F}^{\varepsilon}_{\theta_{k}}(y_{k})\\!-\\!\mathcal{F}^{\varepsilon}_{\theta^{*}_{\varepsilon}}(\tilde{y})\|_{L^{2}}+|y_{k}(T)-\tilde{y}(T)|\right),$

for some $c>0$ independent of $y_{0}\in Y_{0}$, $t\in(0,T)$, and
$k\in\mathbb{N}$. Thus we arrive at

$\displaystyle
J_{{\bullet}}(y_{k},\mathcal{F}^{\varepsilon}_{\theta_{k}}(y_{k}))\rightarrow
J_{\bullet}(y,\mathcal{F}^{\varepsilon}_{\theta^{*}_{\varepsilon}}(y))~{}\text{in}~{}L^{\infty}(I).$

Summarizing the previous findings there holds

$\displaystyle\|\mathcal{V}(y_{k})\\!-\\!J_{\bullet}(y_{k},\mathcal{F}^{\varepsilon}_{\theta_{k}}(y_{k}))\|^{2}_{L^{2}}\\!+\\!\|p_{k}\\!-\\!\partial_{y}\mathcal{V}(y_{k})\|^{2}_{L^{2}}$
$\displaystyle\to\|\mathcal{V}(y)\\!-\\!J_{\boldsymbol{\cdot}}(y,\mathcal{F}^{\varepsilon}_{\theta^{*}_{\varepsilon}}(y))\|^{2}_{L^{2}}+\|p\\!-\\!\partial_{y}\mathcal{V}(y)\|^{2}_{L^{2}}$
$\displaystyle J(y_{k},\mathcal{F}^{\varepsilon}_{\theta_{k}}(y_{k}))$
$\displaystyle\rightarrow
J(y,\mathcal{F}^{\varepsilon}_{\theta^{*}_{\varepsilon}}(y)).$

Using these expressions in $J_{\varepsilon}$ as given in (6.12), and the
boundedness of $\|y_{k}\|_{L^{2}},|y_{k}(0)|,$
$\|p_{k}\|_{L^{2}}~{}\|\mathcal{F}^{\varepsilon}_{\theta_{k}}(y_{k})\|_{L^{2}},~{}\|\mathcal{V}^{\varepsilon}_{\theta_{k}}(y_{k})\|_{L^{2}}$
independent of $k\in\mathbb{N}$ and $y_{0}\in Y_{0}$ we finally get by using
Lebesgue’s dominated convergence theorem

$\displaystyle\mathcal{J}_{\varepsilon}(\mathbf{y}_{k},\mathbf{p}_{k},\theta_{k})\rightarrow\mathcal{J}_{\varepsilon}(\mathbf{y}^{*}_{\varepsilon},\mathbf{p}^{*}_{\varepsilon},\theta^{*}_{\varepsilon})=\inf_{(\mathbf{y},\mathbf{p},\theta)\in\mathcal{N}^{\varepsilon}_{ad}}\mathcal{J}_{\varepsilon}(\mathbf{y},\mathbf{p},\theta).$

∎

##  7\. Convergence towards optimal controls

In Proposition 7 and 9 it was established that the ensemble triple
$(\mathbf{y}^{*},\mathcal{F}(\mathbf{y^{*}}),\mathbf{p}^{*})$ can be
approximated by ensemble triples
$(\mathbf{y}_{\varepsilon},\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(\mathbf{y}_{\varepsilon}),\mathbf{p}_{\varepsilon})$
in the order $O(\varepsilon)$. In this section, the convergence of solutions
to ($\mathcal{P}_{\varepsilon}$) as $\varepsilon\rightarrow 0$ is addressed.
We first consider the terms in the definition $\mathcal{J}_{\varepsilon}$, see
(4.3). To obtain the desired asymptotic behavior a smallness condition on the
regularisation parameter $\gamma_{\epsilon}$, in relation to the norm of the
parameters $\theta_{\varepsilon}$ describing the approximation quality, is
required.

######  Theorem 14.

Let Assumptions 1 and 2 hold the latter with
$\theta_{\varepsilon}\in\mathcal{R}_{\varepsilon}$, and let
$(\mathbf{y}^{*}_{\varepsilon},\mathbf{p}^{*}_{\varepsilon},\theta^{*}_{\varepsilon})$,
denote an optimal triple to ($\mathcal{P}_{\varepsilon}$) for all
$\varepsilon>0$ small enough. If additionally
$\gamma_{\varepsilon}\|\theta_{\varepsilon}\|^{2}_{\mathcal{R}_{\varepsilon}}=O(\varepsilon)$,
then

$\displaystyle
0\leq\int_{Y_{0}}\omega(y_{0})\left[J(\mathbf{y}^{*}_{\varepsilon}(y_{0}),\mathcal{F}^{\varepsilon}_{\theta^{*}_{\varepsilon}}(\mathbf{y}^{*}_{\varepsilon}(y_{0}))-V^{*}(0,y_{0})\right]~{}\mathrm{d}\mathcal{L}(y_{0})\leq
c\,\varepsilon$

holds and, if $\gamma_{1},\gamma_{2}>0$, we also have

$\displaystyle\int_{Y_{0}}\\!\\!\omega(y_{0})(\|V^{\varepsilon}_{\theta^{*}_{\varepsilon}}(t,\mathbf{y}^{*}_{\varepsilon}(y_{0}))\\!-\\!J_{\bullet}(\mathbf{y}^{*}_{\varepsilon}(y_{0}),\mathcal{F}^{\varepsilon}(\mathbf{y}^{*}_{\varepsilon}(y_{0})))\|^{2}_{L^{2}}$
$\displaystyle+\|\partial_{y}V^{\varepsilon}_{\theta^{*}_{\varepsilon}}(t,\mathbf{y}^{*}_{\varepsilon}(y_{0}))\\!-\\!\mathbf{p}^{*}_{\varepsilon}(y_{0})\|^{2}_{L^{2}})\mathrm{d}\mathcal{L}(y_{0})$
$\displaystyle\leq c\,\varepsilon$

for some $c>0$ independent of $\varepsilon$.

###### Proof.

Let $\mathbf{y}_{\varepsilon},\mathbf{p}_{\varepsilon}$ denote the ensembles
of state and adjoint trajectories associated to $\theta_{\varepsilon}$, see
Theorem 4, for $\varepsilon>0$ small enough. Then we have

$\displaystyle\big{|}$ $\displaystyle
J(\mathbf{y}_{\varepsilon}(y_{0}),\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(\mathbf{y}_{\varepsilon}(y_{0}))-V^{*}(0,y_{0})\big{|}$
$\displaystyle\leq
C\big{(}\|\mathbf{y}_{\varepsilon}(y_{0})-\mathbf{y}^{*}(y_{0})\|_{W_{T}}+\|\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(\mathbf{y}_{\varepsilon}(y_{0}))-\mathcal{F}^{*}(\mathbf{y}^{*}(y_{0}))\|_{L^{2}(I;\mathbb{R}^{m})}\big{)}\leq
c\varepsilon$

for some $C>0$ independent of $\varepsilon$. Here we have used
$V^{*}(0,y_{0})=J(\mathbf{y}^{*}(y_{0}),\mathcal{F}^{*}(\mathbf{y}^{*}(y_{0})))$
for all $y_{0}\in Y_{0}$, the embedding
$W_{T}\hookrightarrow\mathcal{C}(\bar{I};\mathbb{R}^{n})$ as well as the a
priori estimates of Proposition 7. Next we utilize
$\mathbf{p}^{*}(y_{0})=\partial\mathcal{V}^{*}(\mathbf{y}^{*}(y_{0}))$,
$y_{0}\in Y_{0}$, to estimate

$\displaystyle\|\partial_{y}\mathcal{V}^{\varepsilon}_{\theta_{\varepsilon}}$
$\displaystyle(\mathbf{y}_{\varepsilon}(y_{0}))-\mathbf{p}_{\varepsilon}(y_{0})\|^{2}_{L^{2}(I;\mathbb{R}^{n})}$
$\displaystyle\leq
2\left(\|\partial_{y}\mathcal{V}^{\varepsilon}_{\theta_{\varepsilon}}(\mathbf{y}_{\varepsilon}(y_{0}))-\partial_{y}\mathcal{V}^{*}(\mathbf{y}^{*}(y_{0}))\|^{2}_{L^{2}(I;\mathbb{R}^{n})}+\|\mathbf{p}_{\varepsilon}(y_{0})-\mathbf{p}^{*}(y_{0})\|^{2}_{L^{2}(I;\mathbb{R}^{n})}\right)$
$\displaystyle\leq c\varepsilon^{2},$

where the last inequality is deduced from Proposition 7 and Proposition 9.
Proceeding analogously and using
$V^{*}(t,\mathbf{y}^{*}(y_{0})(t))=J_{t}(\mathbf{y}^{*}(y_{0}),\mathcal{F}^{*}(\mathbf{y}^{*}(y_{0})))$
for all $y_{0}\in Y_{0}$, $t\in I$, we obtain

$\displaystyle\int_{Y_{0}}\omega(y_{0})\int^{T}_{0}|V^{\varepsilon}_{\theta_{\varepsilon}}(t,\mathbf{y}_{\varepsilon}(y_{0})(t))-J_{t}(\mathbf{y}_{\varepsilon}(y_{0}),\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(\mathbf{y}_{\varepsilon}(y_{0}))|^{2}~{}\mathrm{d}t\,\mathrm{d}\mathcal{L}(y_{0})\leq
D_{1}+D_{2},$

where, using Assumption 2 and again Proposition 7

$\displaystyle D_{1}$
$\displaystyle\coloneqq\int_{Y_{0}}\omega(y_{0})\|\mathcal{V}^{\varepsilon}_{\theta_{\varepsilon}}(\mathbf{y}_{\varepsilon}(y_{0}))-\mathcal{V}^{*}(\mathbf{y}^{*}(y_{0}))\|^{2}_{L^{2}(I)}~{}\mathrm{d}\mathcal{L}(y_{0})\leq
c\varepsilon^{2},$ $\displaystyle D_{2}$
$\displaystyle\coloneqq\int_{Y_{0}}\omega(y_{0})\int^{T}_{0}|J_{t}(\mathbf{y}^{*}(y_{0}),\mathcal{F}^{*}(\mathbf{y}^{*}(y_{0}))-J_{t}(\mathbf{y}_{\varepsilon}(y_{0}),\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(\mathbf{y}_{\varepsilon}(y_{0}))|^{2}~{}\mathrm{d}t\mathrm{d}\mathcal{L}(y_{0})\leq
c\varepsilon^{2}.$

Combining the previous estimates with the optimality of
$(\mathbf{y}^{*},\mathbf{p}^{*},\theta^{*}_{\varepsilon})$, and the assumption
on the asymptotic behavior of $\gamma_{\epsilon}$ we deduce that

$\displaystyle 0$
$\displaystyle\leq\int_{Y_{0}}\omega(y_{0})\left[J_{\varepsilon}(\mathbf{y}^{*}_{\varepsilon}(y_{0}),\mathbf{p}^{*}_{\varepsilon}(y_{0}),\theta^{*}_{\varepsilon})-V^{*}(0,y_{0})\right]~{}\mathrm{d}\mathcal{L}(y_{0})+\frac{\gamma_{\varepsilon}}{2}\|\theta^{*}_{\varepsilon}\|^{2}_{\mathcal{R}_{\varepsilon}}$
$\displaystyle\leq\int_{Y_{0}}\omega(y_{0})\left[J_{\varepsilon}(\mathbf{y}_{\varepsilon}(y_{0}),\mathbf{p}_{\varepsilon}(y_{0}),\theta_{\varepsilon})-V^{*}(0,y_{0})\right]~{}\mathrm{d}\mathcal{L}(y_{0})+\frac{\gamma_{\varepsilon}}{2}\|\theta_{\varepsilon}\|^{2}_{\mathcal{R}_{\varepsilon}}\leq
c\,\varepsilon.$

Recalling the definition of $J_{\varepsilon}$, this yields all claimed
estimates and finishes the proof. ∎

Next the convergence of the ensemble trajectories
$(\mathbf{y}^{*}_{\varepsilon},\mathbf{p}^{*}_{\varepsilon})$, the feedback
controls
$\mathcal{F}^{\varepsilon}_{\theta^{*}_{\varepsilon}}(\mathbf{y}^{*}_{\varepsilon})$
as well as the approximate value function
$\mathcal{V}^{\varepsilon}_{\theta^{*}_{\varepsilon}}$ are analyzed. For this
purpose we make use of the additional regularity of ensemble solutions to the
closed loop system, see Proposition 8, and introduce further constraints to
($\mathcal{P}_{\varepsilon}$). Without changing the notation we henceforth set

$\displaystyle\mathbf{\hat{Y}}_{ad}=\left\\{\,\mathbf{y}\in\mathcal{C}^{1}(Y_{0};W_{T})\;|\;\|\mathbf{y}\|_{\mathcal{C}}\leq
2M_{Y_{0}},~{}\|\mathbf{y}\|_{W^{1,2}}\leq 2M_{W^{1,2}}\,\right\\},$ (7.1)

where $M_{W^{1,2}}>0$ is a constant with $\|\mathbf{y}^{*}\|_{W^{1,2}}\leq
M_{W^{1,2}}$, the function $y^{*}$ was introduced in A.3, and
$W^{1,2}=\\{\mathbf{y}\in L^{2}(I;W_{T}):\partial_{i}\mathbf{y}\in
L^{2}(Y_{0};W_{T}),i\in\\{1,\dots,n\\}\\}$ endowed with the natural norm. Next
we note that

$\displaystyle\frac{\beta}{2}\|\mathcal{F}^{*}(\mathbf{y}^{*}(y_{0}))\|^{2}_{L^{2}}\leq
J(\mathbf{y}^{*}(y_{0}),\mathcal{F}^{*}(\mathbf{y}^{*}(y_{0})))=V^{*}(0,y_{0})$

for all $y_{0}\in Y_{0}$. Thus, due to the continuity of the value function
$V^{*}$, see Assumption 1 $\mathbf{A.2}$, there is $M_{U}>0$ with
$\|\mathcal{F}^{*}(\mathbf{y}^{*})\|_{L^{\infty}}\leq M_{U}$. Correspondingly
we set

$\displaystyle\mathbf{\hat{U}}_{ad}=\left\\{\,\mathbf{u}\in
L^{\infty}(Y_{0};L^{2}(I;\mathbb{R}^{m}))\;|\;\|\mathbf{u}\|_{L^{\infty}}\leq
2M_{U}\,\right\\}.$ (7.2)

We point out that Theorem 13 remains valid despite the additional restriction
of the set of admissible states and controls. Problem
($\mathcal{P}_{\varepsilon}$) with $\mathbf{Y}_{ad},\mathbf{U}_{ad}$ replaced
by $\mathbf{\hat{Y}}_{ad},\mathbf{\hat{U}}_{ad}$ will be denoted by
$(\mathcal{\hat{P}}_{\epsilon})$.

######  Proposition 15.

Let Assumption 1 and 2 hold. Then for all $\varepsilon>0$ small enough,
Problem $(\mathcal{\hat{P}}_{\epsilon})$ admits at least one minimizing
triple.

###### Proof.

Let $(\mathbf{y}_{\varepsilon},\mathbf{p}_{\varepsilon},\theta_{\varepsilon})$
be defined as in Theorem 4. Then we have
$\mathbf{y}_{\varepsilon}\in\mathbf{\hat{Y}}_{ad}$, see Proposition 7 and
Proposition 8, as well as
$\mathcal{F}^{\varepsilon}_{\theta_{\varepsilon}}(\mathbf{y}_{\varepsilon})\in\mathbf{\hat{U}}_{ad}$,
according to Proposition 7, for all $\varepsilon>0$ small enough. Hence the
admissible set of $(\mathcal{\hat{P}}_{\epsilon})$ is not empty. The existence
of a minimizing triple then follows by repeating the arguments of the proof of
Theorem 13 noting that the admissible set

$\displaystyle\left\\{(\mathbf{y},\mathbf{p},\theta)\\!\in\\!\mathbf{\hat{Y}}_{ad}\\!\times\\!\mathcal{C}(Y_{0};W_{T})\\!\times\\!\mathcal{R}_{\varepsilon}|(\mathbf{y},\mathbf{p},\theta)~{}\text{satisfies}\\!~{}\eqref{eq:statepropapprox}\\!-\\!\eqref{eq:constraintprop},\mathcal{F}^{\varepsilon}_{\theta}(\mathbf{y})\\!\in\\!\mathbf{\hat{U}}_{ad}\right\\}$

is closed w.r.t to the weak topology on
$L^{2}(Y_{0};W_{T})^{2}\times\mathcal{R}_{\varepsilon}$. ∎

Let us next address the convergence of the optimal ensemble states
$\mathbf{y}^{*}_{\varepsilon}$, adjoint states $\mathbf{p}_{\varepsilon}$ and
the associated feedback controls
$\mathcal{F}^{\varepsilon}_{\theta^{*}_{\varepsilon}}(\mathbf{y}^{*}_{\varepsilon})$
as $\varepsilon$ tends to 0.

######  Theorem 16.

Let the prerequisites of Theorem 14 hold, and let $\varepsilon_{k}>0$ be a
strictly decreasing null sequence such that
$(\mathcal{\hat{P}}_{\varepsilon_{k}})$ admits a minimizing triple
$(\mathbf{y}^{*}_{k},\mathbf{p}^{*}_{k},\theta^{*}_{k})$. Then
$(\mathbf{y}^{*}_{k},\mathbf{p}^{*}_{k},\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{\varepsilon_{k}}}(\mathbf{y}^{*}_{\varepsilon_{k}}))$
contains at least one accumulation point
$(\bar{\mathbf{y}},\bar{\mathbf{p}},\bar{\mathbf{u}})\in
L^{\infty}(Y_{0};W_{T})^{2}\times L^{\infty}(Y_{0};L^{2}(I;\mathbb{R}^{m}))$
w.r.t the strong topology on $L^{2}(Y_{0};W_{T})^{2}\times
L^{2}(Y_{0};L^{2}(I;\mathbb{R}^{m}))$. For each accumulation point and
$\mathcal{L}$-a.e. $y_{0}\in Y_{0}$ we have that
$(\bar{y},\bar{p},\bar{u})\coloneqq(\bar{\mathbf{y}}(y_{0}),\bar{\mathbf{p}}(y_{0}),\bar{\mathbf{u}}(y_{0}))$
satisfies

$\displaystyle(\bar{y},\bar{u})\in\min\eqref{def:openloopproblem}$

as well as

$\displaystyle\dot{\bar{y}}$
$\displaystyle=\mathbf{f}(\bar{y})+\mathbf{g}(\bar{y})\bar{u},~{}\bar{y}(0)=y_{0},$
$\displaystyle-\dot{\bar{p}}$
$\displaystyle=D\mathbf{f}(\bar{y})^{\top}\bar{p}+[D\mathbf{g}(\bar{y})^{\top}\bar{u})]\bar{p}+\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}(\bar{y}-y_{d}),~{}\bar{p}(T)=Q^{\top}_{2}Q_{2}(\bar{y}(T)-y^{T}_{d}).$

###### Proof.

By choice of the admissible sets $\mathbf{\hat{Y}}_{ad}$ and
$\mathbf{\hat{U}}_{ad}$ we have that
$\\{(\mathbf{y}^{*}_{k},\mathcal{F}^{\varepsilon}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k}))\\}_{k=1}^{\infty}$
is bounded in $(W^{1,2}(Y_{0};W_{T})\cap L^{\infty}(Y_{0};W_{T}))\times
L^{\infty}(Y_{0};L^{2}(I;\mathbb{R}^{m}))$. By Gronwall’s inequality we can
argue that $\\{\mathbf{p}^{*}_{k}\\}_{k=1}^{\infty}$ is also bounded in
$L^{\infty}(Y_{0};W_{T})$. Thus, due to the Banach-Alaoglu theorem, there is a
subsequence, denoted by the same index, and
$(\bar{\mathbf{y}},\bar{\mathbf{p}},\bar{\mathbf{u}})\in
L^{\infty}(Y_{0};W_{T})^{2}\times L^{\infty}(Y_{0};L^{2}(I;\mathbb{R}^{m}))$
such that

$\displaystyle(\mathbf{y}^{*}_{k},\mathbf{p}^{*}_{k},\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k}))\rightharpoonup^{*}(\bar{\mathbf{y}},\bar{\mathbf{p}},\bar{\mathbf{u}})~{}\text{in}~{}L^{\infty}(Y_{0};W_{T})^{2}\times
L^{\infty}(Y_{0};L^{2}(I;\mathbb{R}^{m})),$

and $\dot{\mathbf{y}}^{*}_{k}\rightharpoonup\dot{\bar{\mathbf{y}}}$ in
$L^{2}(Y_{0};L^{2}(I;\mathbb{R}^{n}))$. By the compact embedding of
$W^{1,2}(Y_{0};W_{T})$ into $L^{2}(Y_{0};\mathcal{C}(I;\mathbb{R}^{n}))$, see
[1, Theorem 5.3] the subsequence can be chosen such that
$\mathbf{y}^{*}_{k}\to\bar{\mathbf{y}}$ strongly in
$L^{2}(Y_{0};\mathcal{C}(I;\mathbb{R}^{n}))$. These properties imply that
$(\bar{y},\bar{u})\coloneqq(\bar{\mathbf{y}}(y_{0}),\bar{\mathbf{u}}(y_{0}))$
satisfies

$\dot{\bar{y}}=\mathbf{f}(\bar{y})+\mathbf{g}(\bar{y})\bar{u},~{}\bar{y}(0)=y_{0},$
(7.3)

for $\mathcal{L}$-a.e. $y_{0}\in Y_{0}$. This also implies $V^{*}(0,y_{0})\leq
J(\bar{y},\bar{u})$ and thus, together with

$\displaystyle
J(\mathbf{y}^{*}_{k},\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k}))\rightarrow
V^{*}(0,\cdot)~{}\text{in}~{}L^{1}(Y_{0};W_{T}),$

see Theorem 14, we have
$(\bar{y},\bar{u})\in\operatorname*{arg\,min}\eqref{def:openloopproblem}$ for
$\mathcal{L}$-a.e. $y_{0}\in Y_{0}$. Moreover, again using the strong
convergence of $\mathbf{y}^{*}_{k}$ in
$L^{2}(Y_{0};\mathcal{C}(I;\mathbb{R}^{n}))$ and recalling the definition of
$J(\cdot,\cdot)$ as

$\displaystyle
J(y,u)=(1/2)\|\mathbf{Q}_{1}(y-y_{d})\|^{2}_{L^{2}(I;\mathbb{R}^{n})}+(\beta/2)\|u\|^{2}_{L^{2}(I;\mathbb{R}^{n})}+(1/2)|{Q}_{2}(y(T)-y^{T}_{d})|^{2},$

for all $y\in W_{T},~{}u\in L^{2}(I;\mathbb{R}^{m})$, we also conclude the
convergence of the $L^{2}(Y_{0};L^{2}(I;\mathbb{R}^{m}))$ norm of
$\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k})$ towards
the norm of $\bar{\mathbf{u}}$. Thus
$\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k})\rightarrow\bar{\mathbf{u}}$
strongly in $L^{2}(Y_{0};L^{2}(I;\mathbb{R}^{m}))$, and
$\mathbf{y}^{*}_{k}\to\bar{\mathbf{y}}$ strongly in $L^{2}(Y_{0};W_{T})$, by
Lebegue’s bounded convergence theorem.

It remains to address the strong convergence of $\mathbf{p}_{k}$. For this
purpose we show that the functions
$[D\mathbf{g}(\mathbf{y}^{*}_{k})^{\top}\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k}(\cdot))]\mathbf{p}^{*}_{k}(\cdot)$
converge weakly to
$[D\mathbf{g}(\bar{\mathbf{y}}(\cdot))^{\top}\bar{\mathbf{u}}(\cdot)]\bar{\mathbf{p}}(\cdot)$
in $L^{2}(Y_{0};L^{2}(I;\mathbb{R}^{n}))$. Fixing a test function $\varphi\in
L^{2}(Y_{0};L^{2}(I;\mathbb{R}^{n}))$ we first note that

$\displaystyle\lim_{k\rightarrow\infty}\big{(}\varphi$
$\displaystyle,[D\mathbf{g}(\bar{\mathbf{y}}(\cdot))^{\top}\bar{\mathbf{u}}(\cdot))](\mathbf{p}^{*}_{k}(\cdot)-\bar{\mathbf{p}})\big{)}_{L^{2}(Y_{0};L^{2}(I;\mathbb{R}^{n}))}=0.$

Second, for $\mathcal{L}$-a.e. $y_{0}\in Y_{0}$ we estimate

$\displaystyle\big{(}\varphi(y_{0}),[D\mathbf{g}(\mathbf{y}^{*}_{k}(y_{0}))^{\top}\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k}(y_{0}))-D\mathbf{g}(\bar{\mathbf{y}}(y_{0}))^{\top}\bar{\mathbf{u}}(y_{0})]\mathbf{p}^{*}_{k}(y_{0})\big{)}_{L^{2}(I;\mathbb{R}^{n})}$
$\displaystyle\leq\\!C\|\varphi(y_{0})\|_{L^{2}}\|\mathbf{p}^{*}_{k}(y_{0})\|_{W_{T}}\\!\left(\|\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k}(y_{0}))\|_{L^{2}}\|\mathbf{y}^{*}_{k}(y_{0})\\!-\\!\bar{\mathbf{y}}(y_{0})\|_{W_{T}}\\!+\\!\|\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k}(y_{0}))\\!-\\!\bar{\mathbf{u}}(y_{0})\|_{L^{2}}\\!\right)$
$\displaystyle\leq\\!C\|\varphi(y_{0})\|_{L^{2}}\left(\|\mathbf{y}^{*}_{k}(y_{0})-\bar{\mathbf{y}}(y_{0})\|_{W_{T}}+\|\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k}(y_{0}))-\bar{\mathbf{u}}(y_{0})\|_{L^{2}}\right)$

for some $C>0$ independent of $k\in\mathbb{N}$ and $y_{0}$. Here we made use
of the boundedness of $\\{\mathbf{y}^{*}_{k}\\}_{k=1}^{\infty}$ and
$\\{\mathbf{p}^{*}_{k}\\}_{k=1}^{\infty}$ in $L^{\infty}(Y_{0};W_{T})$, and of
$\\{\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k})\\}_{k=1}^{\infty}$
in $L^{\infty}(Y_{0};L^{2}(I;\mathbb{R}^{m}))$. Integrating both sides of the
inequality w.r.t to $\mathcal{L}$ and utilizing the strong convergence of
$\mathbf{y}^{*}_{k}$ and
$\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k})$ we
finally arrive at

$\displaystyle\lim_{k\rightarrow\infty}\big{(}\varphi,[D\mathbf{g}(\mathbf{y}^{*}_{k}(\cdot))^{\top}\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k}(\cdot))-D\mathbf{g}(\bar{\mathbf{y}}(\cdot))^{\top}\bar{\mathbf{u}}(\cdot)]\mathbf{p}^{*}_{k}(\cdot)\big{)}_{L^{2}(Y_{0},L^{2}(I;\mathbb{R}^{n}))}=0.$

By repeating this argument for the different terms appearing in the adjoint
equation we get that
$(\bar{y},\bar{p},\bar{u})\coloneqq(\bar{\mathbf{y}}(y_{0}),\bar{\mathbf{p}}(y_{0}),\bar{\mathbf{u}}(y_{0}))$
satisfies

$\displaystyle-\dot{\bar{p}}$
$\displaystyle=D\mathbf{f}(\bar{y})^{\top}\bar{p}+[D\mathbf{g}(\bar{y})^{\top}\bar{u}]\bar{p}+\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}(\bar{y}-y_{d}),~{}\bar{p}(T)=Q^{\top}_{2}Q_{2}(\bar{y}(T)-y^{T}_{d})$

for $\mathcal{L}$-a.e. $y_{0}\in Y_{0}$. Applying Gronwall’s inequality we
deduce

$\displaystyle\|\mathbf{p}_{k}(y_{0})-\bar{\mathbf{p}}(y_{0})\|_{W_{T}}\leq
C\left(\|\mathbf{y}^{*}_{k}(y_{0})-\bar{\mathbf{y}}(y_{0})\|_{W_{T}}+\|\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k}(y_{0}))-\bar{\mathbf{u}}(y_{0})\|_{L^{2}}\right)$

for $\mathcal{L}$-a.e. $y_{0}\in Y_{0}$ and $C>0$ independent of $y_{0}$ and
$k$. This yields $\mathbf{p}_{k}\rightarrow\bar{\mathbf{p}}$ strongly in
$L^{2}(Y_{0};W_{T})$. Since the weakly convergent subsequence was chosen
arbitrarily in the beginning, this finishes the proof. ∎

######  Remark 2.

If $\mathbf{g}(y(t))=B\in\mathbb{R}^{m\times n}$ then the statement of the
previous theorem also holds without constraints on the control (i.e. for
$\mathbf{U}_{ad}=L^{2}(Y_{0};L^{2}(I;\mathbb{R}^{m}))$). In this particular
case, the uniform boundedness of
$\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k})$ in
$L^{2}(Y_{0};L^{2}(I;\mathbb{R}^{m}))$ follows from

$\displaystyle\frac{\beta}{2}\|\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k})\|^{2}_{L^{2}}\leq
c\int_{Y_{0}}\omega(y_{0})J(\mathbf{y}^{*}_{k},\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k}(y_{0})))~{}\mathrm{d}\mathcal{L}(y_{0})\leq
C,$

see Theorem 14. Moreover the adjoint equation does no longer depend on the
control. Repeating the arguments of the last proof yields the subsequential
convergence of
$(\mathbf{y}^{*}_{k},\mathbf{p}^{*}_{k},\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k}))$
towards an element $(\bar{\mathbf{y}},\bar{\mathbf{p}},\bar{\mathbf{u}})\in
L^{\infty}(Y_{0};W_{T})^{2}\times L^{2}(Y_{0};L^{2}(I;\mathbb{R}^{m}))$ such
that
$(\bar{y},\bar{p},\bar{u})\coloneqq(\bar{\mathbf{y}}(y_{0}),\bar{\mathbf{p}}(y_{0}),\bar{\mathbf{u}}(y_{0}))$
satisfy the system of state and adjoint equations as well as
$(\bar{y},\bar{u})\in\operatorname*{arg\,min}\eqref{def:openloopproblem}$ for
$\mathcal{L}$-a.e. $y_{0}\in Y_{0}$. Then it only remains to argue the
additional regularity $\mathbf{u}\in L^{\infty}(Y_{0};L^{2}(Y_{0};W_{T}))$.
This is, however, a direct consequence of the first order necessary optimality
condition $\bar{\mathbf{u}}=(-1/\beta)B^{\top}\bar{p}$ for ($P_{y_{0}}$), see
Proposition 1.

We point out that the statement of Theorem 16 holds independently of the
values of the penalty parameters $\gamma_{1},\gamma_{2}$. If
$\gamma_{1},\gamma_{2}>0$ then we additionally obtain the following
convergence results for the approximate value function
$\mathcal{V}^{\varepsilon}_{\theta^{*}_{k}}$ and its derivative
$\partial_{y}\mathcal{V}^{\varepsilon}_{\theta^{*}_{k}}$ along optimal state
trajectories.

######  Proposition 17.

Let the prerequisites of Theorem 14 hold and let
$(\mathbf{y}^{*}_{k},\mathbf{p}^{*}_{k},\theta^{*}_{k})$ denote a sequence of
minimizing triplets as described in Theorem 16. Assume that
$(\mathbf{y}^{*}_{k},\mathbf{p}^{*}_{k},\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k}))$
converges to $(\bar{\mathbf{y}},\bar{\mathbf{p}},\bar{\mathbf{u}})$ in
$L^{2}(Y_{0};W_{T})^{2}\times L^{2}(Y_{0};L^{2}(I;\mathbb{R}^{m}))$ and
$\gamma_{1},\gamma_{2}>0$. Then we also have

$\displaystyle\mathcal{V}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k})\rightarrow\mathcal{V}^{*}(\bar{\mathbf{y}})~{}\text{in}~{}L^{2}(Y_{0};L^{2}(I)),~{}\partial_{y}\mathcal{V}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k})\rightarrow\bar{\mathbf{p}}~{}\text{in}~{}L^{2}(Y_{0};L^{2}(I;\mathbb{R}^{n})).$

###### Proof.

Due to the convergence of $\mathbf{y}^{*}_{k}\to\bar{\mathbf{y}}$ in
$L^{2}(Y_{0};L^{2}(I;\mathbb{R}^{n}))$ and
$\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k})\to\bar{\mathbf{u}}$
in $L^{2}(Y_{0};L^{2}(I;\mathbb{R}^{m}))$, we conclude that

$\displaystyle
J_{\bullet}(\mathbf{y}^{*}_{k},\mathcal{F}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k}))\rightarrow
J_{\bullet}(\bar{\mathbf{y}},\bar{\mathbf{u}})=\mathcal{V}^{*}(\bar{\mathbf{y}})~{}\text{in}~{}L^{2}(Y_{0};L^{2}(I)).$

Together with

$\displaystyle\lim_{k\rightarrow\infty}\int_{Y_{0}}\omega(y_{0})\|V^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k}(y_{0}))-J_{\bullet}(\mathbf{y}^{*}_{k}(y_{0}),\mathcal{F}^{\varepsilon_{k}}(\mathbf{y}^{*}_{k}(y_{0})))\|^{2}_{L^{2}}~{}\mathrm{d}\mathcal{L}(y_{0})=0,$

see Theorem 14, we arrive at
$\mathcal{V}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k})\rightarrow\mathcal{V}^{*}(\bar{\mathbf{y}})$
in $L^{2}(Y_{0};L^{2}(I))$. The statement on the convergence of
$\partial_{y}\mathcal{V}^{\varepsilon_{k}}_{\theta^{*}_{k}}(\mathbf{y}^{*}_{k})$
follows similarly from the strong convergence of $\mathbf{p}_{k}$. ∎

##  8\. Learning from a finite training set

We turn to analysing a discrete version of ($\mathcal{P}_{\varepsilon}$). In
this case we can proceed without the state-space constraint
$\mathbf{y}\in\textbf{Y}_{ad}$ provided certain growth bounds on $\mathbf{f}$
and $\mathbf{g}$ are satisfied. The numerical realization of
($\mathcal{P}_{\varepsilon}$) will always rely on such a discrete
approximation. Henceforth we fix a finite ensemble of initial conditions
$\\{y_{0}^{i}:i=1,\dots,N\\}\subset Y_{0}$. For positive weights $\omega_{i}$,
$i=1,\dots,N$, and $\varepsilon>0$ we consider

$\displaystyle\inf_{y_{i},p_{i}\in
W_{T},\theta\in\mathcal{R}_{\varepsilon}}\left[\sum^{N}_{i=1}\omega_{i}J_{\varepsilon}(y_{i},p_{i},\theta)+\frac{\gamma_{\varepsilon}}{2}\|\theta\|^{2}_{\mathcal{R}_{\varepsilon}}\right]$
($\mathcal{P}^{N}_{\varepsilon}$)

subject to

$\displaystyle\dot{y}_{i}$
$\displaystyle=\mathbf{f}(y_{i})+\mathbf{g}(y_{i})\mathcal{F}^{\varepsilon}_{\theta}(y_{i}),~{}y_{i}(0)=y^{i}_{0}$
$\displaystyle-\dot{p}_{i}$
$\displaystyle=D\mathbf{f}(y_{i})^{\top}p_{i}+[D\mathbf{g}(y_{i})^{\top}\mathcal{F}^{\varepsilon}_{\theta}(y_{i})]p_{i}+\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}(y_{i}-y_{d}),~{}p_{i}(T)=Q^{\top}_{2}Q_{2}(y_{i}(T)-y_{d}^{T}).$

Throughout this section, Assumptions 1 and 2 are supposed to hold. Further
$\varepsilon$ is supposed to be sufficiently small so that the set of
admissible solutions for ($\mathcal{P}^{N}_{\varepsilon}$) is nonempty,
compare Theorem 4. It will be convenient to introduce
$\mathbf{y}=\text{col}(y_{1},\dots,y_{N})$, and
$\mathbf{p}=\text{col}(p_{1},\dots,p_{N})$, which replace the ensemble states
and costates from the previous sections.

######  Proposition 18.

Let $\varepsilon>0$ be sufficiently small and let
$(\mathbf{y}^{k},\mathbf{p}^{k},\theta_{k})\in
W_{T}^{2N}\times\mathcal{R}_{\varepsilon}$ denote an infimizing sequence for
($\mathcal{P}^{N}_{\varepsilon}$). If
$\max_{i}\|y^{k}_{i}\|_{L^{\infty}(I;\mathbb{R}^{n})}\leq M_{\infty}$ for some
${M_{\infty}}>0$ independent of $k\in\mathbb{N}$, then Problem
($\mathcal{P}^{N}_{\varepsilon}$) admits at least one minimizer
$(\mathbf{y}^{*},\mathbf{p}^{*},\theta^{*})$.

###### Proof.

Since by assumption $(\mathbf{y}^{k},\mathbf{p}^{k},\theta_{k})$ is an
infimizing sequence for ($\mathcal{P}^{N}_{\varepsilon}$) and since $\beta>0$
we have

$\displaystyle\max_{i}\|{\bf{Q_{1}}}y^{k}_{i}\|^{2}_{L^{2}}+\max_{i}\|\mathcal{F}^{\varepsilon}_{\theta_{k}}(y^{k}_{i})\|^{2}_{L^{2}}\leq
C_{N}$ (8.1)

for some $C_{N}>0$ depending on $N$. Moreover there holds

$\displaystyle\|\dot{y}^{k}_{i}\|_{L^{2}}\leq\|\mathbf{f}(y^{k}_{i})\|_{L^{2}}+\|\mathbf{g}(y_{i})\mathcal{F}^{\varepsilon}_{\theta_{k}}(y^{k}_{i})\|_{L^{2}}\leq
C(\mathbf{f},\mathbf{g})M_{\infty}(1+C_{N})$

using the uniform $L^{\infty}$ and $L^{2}$ boundedness of $y^{k}_{i}$ and
$\mathcal{F}^{\varepsilon}_{\theta}(y^{k}_{i})$, respectively. Thus we also
have $\|y^{i}_{k}\|_{W_{T}}\leq\widehat{C}_{N}$ for all $k\in\mathbb{N}$, for
some $\widehat{C}_{N}>0$ which depends on $N$ but not on $k$ and $i$. The
proof can now be completed by the same steps as Theorem 13. ∎

######  Remark 3.

The $L^{\infty}$-boundedness of the minimizing sequence $y^{k}_{i}$ in
Proposition 18 can be be ensured by additional assumptions on the dynamics of
the problem. These include:

  * •

Add an additional state constraint $\|y_{i}\|_{L^{\infty}}\leq\widehat{M}$ to
($\mathcal{P}^{N}_{\varepsilon}$).

  * •

Assume that there are $a_{1},a_{2},a_{3}>0$ such that

$\displaystyle|f(x)|\leq a_{1}+a_{2}|x|+a_{3}|x|^{2},~{}\|g(x)\|\leq
a_{1}+a_{2}|x|\quad\forall x\in\mathbb{R}^{n},$

and that $Q_{1}$ is positive definite. Then by (8.1) the family
$\\{y^{k}_{i}\\}$ is uniformly w.r.t. $i\in\\{1,\dots,n\\}$ and $k=1,\dots$
bounded in $L^{2}(I;\mathbb{R}^{n})$ . Further we can readily verify that

$\displaystyle\|\dot{y}^{k}_{i}\|_{L^{1}}$
$\displaystyle\leq\|\mathbf{f}(y^{k}_{i})\|_{L^{1}}+\|\mathbf{g}(y^{k}_{i})\|_{L^{1}}$
$\displaystyle\leq
2a_{1}T+a_{2}\|y^{k}_{i}\|_{L^{1}}+a_{3}\|y^{k}_{i}\|^{2}_{L^{2}}+a_{2}\|y^{k}_{i}\|_{L^{2}}\|\mathcal{F}^{\varepsilon}_{\theta_{k}}(y^{k}_{i})\|^{2}_{L^{2}}\leq
M_{N}$

for an $N$-dependent bound $M_{N}>0$. Here we made use of the
$L^{2}$-boundedness of $y^{k}_{i}$ and
$\mathcal{F}^{\varepsilon}_{\theta_{k}}(y^{k}_{i})$ which follows from (8.1)
in the proof of Proposition 18, and the assumption that $Q_{1}>0$.
Consequently $y^{k}_{i}$ is uniformly bounded in $W^{1,1}(I;\mathbb{R}^{n})$
and thus also in $L^{\infty}(I;\mathbb{R}^{n})$.

  * •

Assume that $f(x)=Ax-h(x)$ where $A\in\mathbb{R}^{n\times n}$ and $h$ is
monotone i.e. $(x,h(x))_{\mathbb{R}^{n}}\geq 0$ for all $x\in\mathbb{R}^{n}$.
Moreover assume that $Q_{1}$ is positive definite and that

$\displaystyle\|g(x)\|\leq a_{1}+a_{2}|x|\quad\forall x\in\mathbb{R}^{n}.$

In this case, testing the equation satisfied by $y_{i}$ with $y_{i}$, and a
Gronwall argument yields

$\displaystyle|y^{k}_{i}(t)|^{2}\leq
C_{N}\left(|y^{i}_{0}|^{2}+\|y^{k}_{i}\|^{2}_{L^{2}}+\|\mathcal{F}^{\varepsilon}_{\theta_{k}}(y^{k}_{i})\|^{2}_{L^{2}}\right)$

for some $N$-dependent $C_{N}>0$ and all $t\in I$. Thus, the uniform
boundedness of $y^{k}_{i}$ in $L^{\infty}(I;\mathbb{R}^{n})$ follows again
from the $L^{2}$-estimates on $y^{k}_{i}$ and
$\mathcal{F}^{\varepsilon}_{\theta_{k}}(y^{k}_{i})$ in (8.1).

The convergence result as $\varepsilon\to 0^{+}$ of Theorem 16 can be
transferred to the finite training set setting as well.

######  Proposition 19.

Let the regularisation parameters satisfy
$\gamma_{\varepsilon}\|\theta_{\varepsilon}\|^{2}_{\mathcal{R}_{\varepsilon}}=O(\varepsilon)$.
Further let $\varepsilon_{k}>0$ be a positive null sequence such that for each
$k\in\mathbb{N}$ there exists a solution
$(\mathbf{y}^{k},\mathbf{p}^{k},\theta_{k})\in
W_{T}^{2N}\times\mathcal{R}_{\varepsilon}$ to
$(\mathcal{P}^{N}_{\varepsilon_{k}})$. If there is ${M_{\infty}}>0$ with
$\max_{i}\|y^{k}_{i}\|_{L^{\infty}}\leq M_{\infty}$ for all $k\in\mathbb{N}$,
then
$(\mathbf{y}^{k},\mathbf{p}^{k},\mathbf{\mathcal{F}}^{\varepsilon_{k}}_{\theta_{k}}(\mathbf{y}^{k}))$
admits at least one strong accumulation point
$(\mathbf{\bar{y}},\mathbf{\bar{p}},\mathbf{\bar{u}})$ in $W^{2N}_{T}\times
L^{2}(I;\mathbb{R}^{m})^{N}$. Each such point satisfies

$\displaystyle(\bar{y}_{i},\bar{u}_{i})\in\operatorname*{arg\,min}(P^{y^{i}_{0}}_{\beta}),\quad
i=1,\dots,N,$

as well as

$\displaystyle\dot{\bar{y}}_{i}$
$\displaystyle=\mathbf{f}(\bar{y}_{i})+\mathbf{g}(\bar{y}_{i})\bar{u}_{i},~{}\bar{y}_{i}(0)=y^{i}_{0}$
$\displaystyle-\dot{\bar{p}}_{i}$
$\displaystyle=D\mathbf{f}(\bar{y}_{i})^{\top}\bar{p}_{i}+[D\mathbf{g}(\bar{y}_{i})^{\top}\bar{u}_{i}]\bar{p}_{i}+\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}(\bar{y}_{i}-y_{d}),~{}\bar{p}_{i}(T)=Q^{\top}_{2}Q_{2}(\bar{y}_{i}(T)-y_{d}^{T}).$

###### Proof.

For every $\varepsilon_{k}$, with $k$ sufficiently large, denote by
$\theta_{\varepsilon_{k}}\in\mathcal{R}_{\varepsilon_{k}}$ the corresponding
parameters from Assumption 2, by $\mathbf{y}_{\varepsilon_{k}}$ the associated
ensemble solution, see Theorem 4, and by $\mathbf{p}_{\varepsilon_{k}}$ the
adjoint states. For abbreviation we set
$y^{\varepsilon_{k}}_{i}\coloneqq\mathbf{y}_{\varepsilon_{k}}(y^{i}_{0})$ and
$p^{\varepsilon_{k}}_{i}\coloneqq\mathbf{p}_{\varepsilon_{k}}(y^{i}_{0})$.
Then, by optimality, we have

$\displaystyle\sum^{N}_{i=1}\omega_{i}J(y^{k}_{i},\mathcal{F}^{\varepsilon_{k}}_{\theta_{k}}(y^{k}_{i}))\leq\sum^{N}_{i=1}\omega_{i}J_{\varepsilon}(y^{\varepsilon_{k}}_{i},p^{\varepsilon_{k}}_{i},\theta_{\varepsilon_{k}})+\frac{\gamma_{\varepsilon_{k}}}{2}\|\theta_{\varepsilon_{k}}\|^{2}_{\mathcal{R}_{\varepsilon}}.$
(8.2)

As in the proof of Theorem 14 we see that the righthandside of this inequality
converges to $\sum^{N}_{i=1}\omega_{i}V^{*}(0,y^{i}_{0})$ as
$k\rightarrow+\infty$. Thus it is bounded independently of $k\in\mathbb{N}$.
Similarly to Proposition 18 we then conclude the existence of $C_{N}>0$
depending on $N$, but not on $k$, such that

$\displaystyle\max_{i}\|{\mathbf{Q}_{1}}y^{k}_{i}\|^{2}_{L^{2}}+\max_{i}\|\mathcal{F}^{\varepsilon}_{\theta_{k}}(y^{k}_{i})\|^{2}_{L^{2}}\leq
C_{N}.$

Utilizing the state equation this can be improved to a $k$-independent bound
on the $W_{T}$-norm of $y^{k}_{i}$. By a Gronwall-type argument the same can
be shown for the adjoint states $p^{k}_{i}$. Now fix an arbitrary index
$i\in\\{1,\dots,N\\}$. Summarizing the previous observations we get the
uniform boundedness of
$(y^{k}_{i},p^{k}_{i},\mathcal{F}^{\varepsilon}_{\theta_{k}}(y^{k}_{i}))$ in
$W_{T}^{2}\times L^{2}(I;\mathbb{R}^{m})$ w.r.t. $k$, for each $i=1,\dots,N$.
Each of its weak accumulation points $(\bar{y}_{i},\bar{p}_{i},\bar{u}_{i})\in
W_{T}^{2}\times L^{2}(I;\mathbb{R}^{m})$ satisfies

$\displaystyle\dot{\bar{y}}$
$\displaystyle=\mathbf{f}(\bar{y})+\mathbf{g}(\bar{y})\bar{u},~{}\bar{y}(0)=y_{0}.$

From this we conclude that

$\displaystyle
0\leq\sum^{N}_{i=1}\omega_{i}V^{*}(0,y^{i}_{0})\leq\sum^{N}_{i=1}\omega_{i}J(\bar{y}_{i},\bar{u}_{i})\leq\lim_{k\to\infty}\sum^{N}_{i=1}\omega_{i}J(y^{k}_{i},\mathcal{F}^{\varepsilon_{k}}_{\theta_{k}}(y^{k}_{i}))\leq\sum^{N}_{i=1}\omega_{i}V^{*}(0,y^{i}_{0}),$

Since the second and third of the above inequalities also hold for each
summand we conclude that
$\lim_{k\to\infty}J(y^{k}_{i},\mathcal{F}^{\varepsilon_{k}}_{\theta_{k}}(y^{k}_{i}))\rightarrow
J(\bar{y}_{i},\bar{u}_{i})$ as well as
$J(\bar{y}_{i},\bar{u}_{i})=V^{*}(0,y^{i}_{0})$. Hence

$\displaystyle(\bar{y}_{i},\bar{u}_{i})\in\operatorname*{arg\,min}(P^{y^{i}_{0}}_{\beta}).$

The proof can now be concluded with minor adaptations to the proof of Theorem
16. ∎

A result analogous to that of Proposition 17 can also be obtained for Problem
($\mathcal{P}^{N}_{\varepsilon}$). For the sake of brevity we do not present
the details.

###  8.1. The reduced objective functional

In order to compute a solution to ($\mathcal{P}^{N}_{\varepsilon}$) we will
rely on gradient-based optimization methods. For this purpose we introduce a
reduced objective functional by eliminating the state and adjoint equations in
($\mathcal{P}^{N}_{\varepsilon}$). Subsequently, we characterize the
derivative of the reduced functional by means of adjoint techniques. To
simplify the presentation we fix an arbitrary index $i\in\\{1,\dots,N\\}$ in
the following. Moreover, for abbreviation, we define the mapping

$\displaystyle\mathbf{A}\colon
W_{T}\times\mathcal{R}_{\varepsilon}\to\mathcal{B}(W_{T};L^{2}(I;\mathbb{R}^{n})),~{}A(y,\theta)=D\mathbf{f}(y)^{\top}+[D\mathbf{g}(y)^{\top}\mathcal{F}^{\varepsilon}_{\theta}(y)].$

Using this notation, the adjoint equation in ($\mathcal{P}^{N}_{\varepsilon}$)
can be expressed compactly as

$\displaystyle-\dot{p}_{i}=\mathbf{A}(y_{i},\theta)p_{i}+\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}(y_{i}-y_{d}),~{}p_{i}(T)=Q^{\top}_{2}Q_{2}(y_{i}(T)-y_{d}^{T}).$

First, we argue the existence of parameter-to-state operators for the adjoint
and the state equation.

######  Lemma 20.

Define $G_{i}\colon W_{T}\times W_{T}\times\mathcal{R}_{\varepsilon}\to
L^{2}(I;\mathbb{R}^{n})\times
L^{2}(I;\mathbb{R}^{n})\times\mathbb{R}^{n}\times\mathbb{R}^{n}$ by

$\displaystyle
G_{i}(y,p,\theta)=\left(\begin{array}[]{c}\dot{y}-\mathbf{f}(y)-\mathbf{g}(y)\mathcal{F}^{\varepsilon}_{\theta}(y)\\\
-\dot{p}-A(y,\theta)p-\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}(y-y_{d})\\\
y(0)-y^{i}_{0}\\\ p(T)-Q^{\top}_{2}Q_{2}(y(T)-y_{d}^{T})\end{array}\right).$

Let $(\tilde{y},\tilde{p},\tilde{\theta})\in W_{T}\times
W_{T}\times\mathcal{R}_{\varepsilon}$ satisfy
$G(\tilde{y},\tilde{p},\tilde{\theta})=0$. Then there exists a neighbourhood
$\mathcal{N}_{i}(\tilde{y})\times\mathcal{N}_{i}(\tilde{p})\times\mathcal{N}_{i}(\tilde{\theta})$
as well as $\mathcal{C}^{1}$-mappings
$Y_{i}\colon\mathcal{N}_{i}(\tilde{\theta})\to\mathcal{N}_{i}(\tilde{y})\subset
W_{T}$,
$P_{i}\colon\mathcal{N}_{i}(\tilde{\theta})\to\mathcal{N}_{i}(\tilde{p})\subset
W_{T}$ such that

$\displaystyle
G_{i}(Y_{i}(\theta),P_{i}(\theta),\theta)=0\quad\forall\theta\in\mathcal{N}(\tilde{\theta}).$

Given $y_{i}:=Y_{i}(\theta)$ and $~{}p_{i}:=P_{i}(\theta)$, the Fréchet
derivatives of $Y_{i}$ and $P_{i}$ at
$\theta\in\mathcal{N}_{i}(\tilde{\theta})$, in direction
$\delta\theta\in\mathcal{R}_{\varepsilon}$, denoted by $\delta Y_{i}\coloneqq
Y^{\prime}_{i}(\theta)(\delta\theta)$, $\delta P_{i}\coloneqq
P^{\prime}_{i}(\theta)(\delta\theta)$ satisfy

$\displaystyle\dot{\delta Y_{i}}-\mathbf{A}(y_{i},\theta)^{\top}\delta
Y_{i}-\mathbf{g}(y_{i})D_{y}\mathcal{F}^{\varepsilon}_{\theta}(y_{i})\delta
Y=\mathbf{g}(y_{i})D_{\theta}\mathcal{F}^{\varepsilon}_{\theta}(y_{i})\delta\theta,$
$\displaystyle-\dot{\delta P_{i}}-\mathbf{A}(y_{i},\theta)\delta
P_{i}=[D_{y}\mathbf{A}(y_{i},\theta)\delta
Y_{i}]p_{i}+\mathbf{Q}_{1}\mathbf{Q}_{1}\delta
Y_{i}+[\partial_{\theta}\mathbf{A}(y_{i},\theta)\delta\theta]p_{i},$
$\displaystyle~{}\delta Y_{i}(0)=0,~{}\delta P_{i}(T)=Q_{2}^{\top}Q_{2}\delta
Y_{i}(T).$

###### Proof.

This is a direct consequence of the implicit function theorem applied to $G$
noting that the directional derivatives satisfy

$\displaystyle\begin{pmatrix}\partial_{y}G_{i}(y,p,\theta)&\partial_{p}G_{i}(y,p,\theta)\end{pmatrix}\begin{pmatrix}\delta
Y\\\ \delta P\end{pmatrix}=-\partial_{\theta}G_{i}(y,p,\theta)\delta\theta.$

∎

Now consider an admissible point
$(\tilde{\mathbf{y}},\tilde{\mathbf{p}},\theta)\in
W^{2N}_{T}\times\mathcal{R}_{\varepsilon}$ for
($\mathcal{P}^{N}_{\varepsilon}$). For every $i=1,\dots,N$, let
$\mathcal{N}_{i}(\tilde{\theta})$ and $Y_{i},P_{i}$ denote the corresponding
neighbourhoods and operators from Lemma 20. Setting
$\mathcal{N}(\tilde{\theta})=\bigcap^{N}_{i=1}\mathcal{N}_{i}(\tilde{\theta})$
define the reduced objective functional

$\displaystyle\mathcal{J}_{N}\colon\mathcal{N}(\tilde{\theta})\to[0,+\infty),~{}\mathcal{J}_{N}(\theta)=\sum^{N}_{i=i}\omega_{i}J_{\varepsilon}(Y_{i}(\theta),P_{i}(\theta),\theta)+\frac{\gamma_{\varepsilon}}{2}\|\theta\|^{2}_{\mathcal{R}_{\varepsilon}},$
(8.3)

and set

$\displaystyle\Phi_{i}(t)=\int^{t}_{0}(V^{\varepsilon}_{\theta}(s,y_{i}(s))-J_{s}(y_{i},u))~{}\mathrm{d}s.$

.

######  Proposition 21.

The functional $\mathcal{J}_{N}$ from (8.3) is at least of class
$\mathcal{C}^{1}$ on $\mathcal{N}(\tilde{\theta})$. Given
$\theta\in\mathcal{N}(\tilde{\theta})$, set $y_{i}\coloneqq Y_{i}(\theta)$,
$p_{i}\coloneqq P_{i}(\theta)$ as well as $\delta Y_{i}\coloneqq
Y^{\prime}_{i}(\theta)(\delta\theta)$, $\delta P_{i}\coloneqq
P^{\prime}_{i}(\theta)(\delta\theta)$. The directional derivative of
$\mathcal{J}_{N}$ at $\theta$ in the direction of
$\delta\theta\in\mathcal{R}_{\varepsilon}$ is given by

$\displaystyle\mathcal{J}^{\prime}_{N}(\theta)(\delta\theta)=\sum^{N}_{i=1}\omega_{i}\left((\widehat{y}_{i},\delta
Y_{i})_{L^{2}}+(\widehat{y}^{T}_{i},\delta
Y_{i}(T))_{\mathbb{R}^{n}}+(\widehat{p}_{i},\delta
P_{i})_{L^{2}}+(\widehat{\theta}_{i},\delta\theta)_{\mathcal{R}_{\varepsilon}}\right)+\gamma_{\varepsilon}(\theta,\delta\theta)_{\mathcal{R}_{\varepsilon}}$

with

$\displaystyle\widehat{y}_{i}=(1-\gamma_{1}\Phi_{i})$
$\displaystyle\mathbf{Q}_{1}\mathbf{Q}_{1}(y_{i}-y_{d})+\beta(1-\gamma_{1}\Phi_{i})D_{y}\mathcal{F}^{\varepsilon}_{\theta}(y_{i})^{\top}\mathcal{F}^{\varepsilon}_{\theta}(y_{i})$
$\displaystyle+\gamma_{1}(\mathcal{V}^{\varepsilon}_{\theta}(t,y_{i})-J_{\bullet}(y_{i},\mathcal{F}^{\varepsilon}_{\theta}(y_{i})))\partial_{y}\mathcal{V}^{\varepsilon}_{\theta}(y_{i})+\gamma_{2}D_{yy}\mathcal{V}^{\varepsilon}_{\theta}(y_{i})(\partial_{y}\mathcal{V}^{\varepsilon}_{\theta}(y_{i})-p_{i}),$

and

$\displaystyle\widehat{y}^{T}_{i}=\alpha(1-\gamma_{1}\Phi_{i}(0))Q_{2}Q_{2}(y_{i}(T)-y^{T}_{d}),$

as well as

$\displaystyle\widehat{p}_{i}=\gamma_{2}(p_{i}-\partial_{y}\mathcal{V}^{\varepsilon}_{\theta}(y_{i})),$

and

$\displaystyle\widehat{\theta}_{i}$
$\displaystyle=\gamma_{1}\int^{T}_{0}D_{\theta}V^{\varepsilon}_{\theta}(t,y_{i}(t))^{\top}(V^{\varepsilon}_{\theta}(t,y_{i}(t))-J_{t}(y_{i},\mathcal{F}^{\varepsilon}_{\theta}(y_{i})))~{}\mathrm{d}t$
$\displaystyle+\int^{T}_{0}\\![\beta(1\\!-\\!\gamma_{1}\Phi_{i}(t))D_{\theta}F^{\varepsilon}_{\theta}(t,y_{i}(t))^{\top}F^{\varepsilon}_{\theta}(t,y_{i}(t))\\!+\\!\gamma_{2}D_{y\theta}V^{\varepsilon}_{\theta}(t,y_{i}(t))^{\top}(\partial_{y}{V}^{\varepsilon}_{\theta}(t,y_{i}(t))\\!-\\!p_{i}(t))]~{}\mathrm{d}t.$

###### Proof.

The regularity of $\mathcal{J}_{N}$ follows immediately from Lemma 20 and the
chain rule. In order to compute the directional derivative we abbreviate

$\begin{array}[]{l}F_{1}(y,u,\theta)=\frac{\gamma_{1}}{2}\int^{T}_{0}|V^{\varepsilon}_{\theta}(t,y(t))-J_{t}(y,u)|^{2}~{}\mathrm{d}t,\\\\[6.02773pt]
F_{2}(y,p,\theta)=\frac{\gamma_{2}}{2}\int^{T}_{0}|\partial_{y}V^{\varepsilon}_{\theta}(t,y(t))-p(t)|^{2}~{}\mathrm{d}t\end{array}$

in the following. Thus we have

$\displaystyle J_{\varepsilon}(Y_{i}(\theta),P_{i}(\theta),\theta)$
$\displaystyle=J(Y_{i}(\theta),\mathcal{F}^{\varepsilon}_{\theta}(Y_{i}(\theta)))+F_{1}(Y_{i}(\theta),\mathcal{F}^{\varepsilon}_{\theta}(Y_{i}(\theta)),\theta)+F_{2}(Y_{i}(\theta),P_{i}(\theta),\theta)$
$\displaystyle=G_{1}(\theta)+G_{2}(\theta)+G_{3}(\theta).$

We readily verify

$\displaystyle G^{\prime}_{1}(\theta)(\delta\theta)=$
$\displaystyle(\mathbf{Q}_{1}\mathbf{Q}_{1}(y_{i}-y_{d}),\delta
Y_{i})_{L^{2}}+\beta(D_{y}\mathcal{F}^{\varepsilon}_{\theta}(y_{i})^{\top}\mathcal{F}^{\varepsilon}_{\theta}(y_{i}),\delta
Y_{i})_{L^{2}}$
$\displaystyle+\beta(D_{\theta}\mathcal{F}^{\varepsilon}_{\theta}(y_{i})^{\top}\mathcal{F}^{\varepsilon}_{\theta}(y_{i}),\delta\theta)_{\mathcal{R}_{\varepsilon}}+(Q_{2}Q_{2}(y_{i}(T)-y^{T}_{d}),\delta
Y_{i}(T))_{\mathbb{R}^{n}}.$

Recalling the definition of $\Phi_{i}$ we get

$\displaystyle G^{\prime}_{2}(\theta)$
$\displaystyle(\delta\theta)=\gamma_{1}(E_{1}+E_{2}+E_{3}+E_{4}),$

where

$\displaystyle E_{1}$
$\displaystyle=((\mathcal{V}^{\varepsilon}_{\theta}(y_{i})\\!-\\!J_{\bullet}(y_{i},\mathcal{F}^{\varepsilon}_{\theta}(y_{i})))\partial_{y}\mathcal{V}^{\varepsilon}_{\theta}(y_{i}),\delta
Y_{i})_{L^{2}}+(D_{\theta}\mathcal{V}^{\varepsilon}_{\theta}(y_{i})^{\top}(\mathcal{V}^{\varepsilon}_{\theta}(y_{i})\\!-\\!J_{\bullet}(y_{i},\mathcal{F}^{\varepsilon}_{\theta}(y_{i}))),\delta\theta)_{\mathcal{R}_{\varepsilon}}$
$\displaystyle=((\mathcal{V}^{\varepsilon}_{\theta}(y_{i})\\!-\\!J_{\bullet}(y_{i},\mathcal{F}^{\varepsilon}_{\theta}(y_{i})))\partial_{y}\mathcal{V}^{\varepsilon}_{\theta}(t,y_{i}),\delta
Y_{i})_{L^{2}}$
$\displaystyle\qquad+\left(\int^{T}_{0}D_{\theta}V^{\varepsilon}_{\theta}(t,y_{i}(t))^{\top}(V^{\varepsilon}_{\theta}(t,y_{i}(t))-J_{t}(y_{i},\mathcal{F}^{\varepsilon}_{\theta}(y_{i})))~{}\mathrm{d}t,\delta\theta\right)_{\mathcal{R}_{\varepsilon}},$
$\displaystyle E_{2}=-\int^{T}_{0}$
$\displaystyle(V^{\varepsilon}_{\theta}(t,y(t))-J_{t}(y,u))$
$\displaystyle\left(\int^{T}_{t}(Q_{1}Q_{1}(y(s)-y_{d}(s)),\delta
y(s))~{}\mathrm{d}s+(Q_{2}Q_{2}(y(T)-y^{T}_{d}),\delta
y(T))_{\mathbb{R}^{n}}\right)~{}\mathrm{d}t$
$\displaystyle=-(\Phi_{i}Q_{1}Q_{1}(y-y_{d}),\delta
y)_{L^{2}}-\Phi_{i}(0)(Q_{2}Q_{2}(y(T)-y^{T}_{d}),\delta
y(T))_{\mathbb{R}^{n}},$

as well as

$\displaystyle E_{3}$
$\displaystyle=-\int^{T}_{0}(V^{\varepsilon}_{\theta}(t,y(t))-J_{t}(y,u))\left(\beta\int^{T}_{t}(D_{y}F^{\varepsilon}_{\theta}(s,y_{i}(s))^{\top}F^{\varepsilon}_{\theta}(s,y_{i}(s)),\delta
Y_{i}(s))_{\mathbb{R}^{n}}~{}\mathrm{d}s\right)~{}\mathrm{d}t$
$\displaystyle=-\beta(\Phi_{i}D_{y}\mathcal{F}^{\varepsilon}_{\theta}(y_{i})^{\top}\mathcal{F}^{\varepsilon}_{\theta}(y_{i}),\delta
Y_{i})_{L^{2}},$

and

$\displaystyle E_{4}$
$\displaystyle=-\int^{T}_{0}(V^{\varepsilon}_{\theta}(t,y(t))-J_{t}(y,u))\left(\beta\int^{T}_{t}(D_{\theta}F^{\varepsilon}_{\theta}(s,y_{i}(s))^{\top}F^{\varepsilon}_{\theta}(s,y_{i}(s)),\delta\theta)_{\mathcal{R}_{\varepsilon}}~{}\mathrm{d}s\right)~{}\mathrm{d}t$
$\displaystyle=-\beta\left(\int^{T}_{0}\Phi_{i}(t)D_{\theta}F^{\varepsilon}_{\theta}(t,y_{i}(t))^{\top}F^{\varepsilon}_{\theta}(t,y_{i}(t))~{}\mathrm{d}t,\delta\theta\right)_{\mathcal{R}_{\varepsilon}},$

by means of partial integration. Finally we calculate

$\displaystyle G^{\prime}_{3}(\theta)(\delta\theta)=\gamma_{2}(D_{yy}$
$\displaystyle\mathcal{V}^{\varepsilon}_{\theta}(y_{i})(\partial_{y}\mathcal{V}^{\varepsilon}_{\theta}(y_{i})-p_{i}),\delta
Y_{i})_{L^{2}}-\gamma_{2}(\partial_{y}\mathcal{V}^{\varepsilon}_{\theta}(y_{i})-p_{i},\delta
P_{i})_{L^{2}}$
$\displaystyle+\gamma_{2}\left(\int^{T}_{0}D_{y\theta}V^{\varepsilon}_{\theta}(t,y_{i}(t))^{\top}(\partial_{y}{V}^{\varepsilon}_{\theta}(t,y_{i}(t))-p_{i}(t))~{}\mathrm{d}t,\delta\theta\right)_{\mathcal{R}_{\varepsilon}}.$

Summarizing the previous observations, we arrive at the claimed
characterization. ∎

Applying a gradient method to ($\mathcal{P}^{N}_{\varepsilon}$) requires the
computation of the gradient
$\nabla\mathcal{J}_{N}(\theta)\in\mathcal{R}_{\varepsilon}$ which satisfies

$\displaystyle\mathcal{J}^{\prime}_{N}(\theta)(\delta\theta)=(\nabla\mathcal{J}_{N}(\theta),\delta\theta)_{\mathcal{R}_{\varepsilon}}\quad\forall\delta\theta\in\mathcal{R}_{\varepsilon}.$

This can be done by computing $\mathcal{J}^{\prime}_{N}(\theta)(e_{j})$ for
the canonical basis
$\\{e_{j}\\}^{N_{\varepsilon}}_{j=1}\subset\mathcal{R}_{\varepsilon}$.
However, such reasoning leads to the necessity to solve
$2\operatorname{dim}(\mathcal{R}_{\varepsilon})N$ additional ODEs in order to
compute the sensitivities $Y^{\prime}_{i}(\theta)(e_{j})$ and
$P^{\prime}_{i}(\theta)(e_{j})$, respectively. Introducing suitable costate
equations, this can be reduced to $2N$ additional equation solves.

######  Lemma 22.

Let $\widehat{y}_{i},\widehat{y}^{T}_{i},\widehat{p}_{i}$ as well as $\delta
Y_{i},\delta P_{i}$ be defined as in Proposition 21. Then there holds

$\displaystyle(\widehat{y}_{i},\delta
Y_{i})_{L^{2}}+(\widehat{y}^{T}_{i},\delta
Y_{i}(T))_{\mathbb{R}^{n}}+(\widehat{p}_{i},\delta
P)_{L^{2}}=(D_{\theta}\mathcal{F}^{\varepsilon}_{\theta}(y_{i})^{\top}(\mathbf{g}(y_{i})^{\top}\zeta_{i}+[D\mathbf{g}(y_{i})\kappa_{i}]^{\top}p_{i}),\delta\theta)_{\mathcal{R}_{\varepsilon}}$

where $\zeta_{i},\kappa_{i}\in W_{T}$ satisfy

$\displaystyle-\dot{\zeta}_{i}$
$\displaystyle=\mathbf{A}(y_{i},\theta)\zeta+D_{y}\mathcal{F}^{\varepsilon}_{\theta}(y_{i})^{\top}\mathbf{g}(y_{i})^{\top}\zeta_{i}+[D_{y}\mathbf{A}(y_{i},\theta)^{\top}p]\kappa_{i}+\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}\kappa_{i}+\widehat{y}_{i}$
$\displaystyle\dot{\kappa}_{i}$
$\displaystyle=\mathbf{A}(y_{i},\theta)^{\top}\kappa_{i}+\widehat{p}_{i},$
$\displaystyle\zeta_{i}(T)$
$\displaystyle=Q_{2}^{\top}Q_{2}\kappa(T)+\widehat{y}^{T}_{i},~{}\kappa_{i}(0)=0.$

###### Proof.

For the sake of readability, we drop the subscript $i$ in the following. By
partial integration and Lemma 20 we obtain

$\displaystyle(\widehat{p},\delta P)_{L^{2}}$
$\displaystyle=(\dot{\kappa}-\mathbf{A}(y,\theta)^{\top}\kappa,\delta
P)=(-\dot{\delta P}-\mathbf{A}(y,\theta)\delta
P,\kappa)+(Q_{2}^{\top}Q_{2}\delta Y(T),\kappa(T))_{\mathbb{R}^{n}}$
$\displaystyle=([D_{y}\mathbf{A}(y,\theta)\delta
Y]p+\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}\delta
Y+[\partial_{\theta}\mathbf{A}(y,\theta)\delta\theta]p,\kappa)+(\delta
Y(T),\zeta(T)-\hat{y}_{i}^{T})_{\mathbb{R}^{n}}$

and

$\displaystyle(\hat{y}_{i},\delta Y)_{L^{2}}+(\hat{y}_{i}^{T},\delta
Y(T))_{\mathbb{R}^{n}}$
$\displaystyle=(-\dot{\zeta}\\!-\\!\mathbf{A}(y,\theta)\zeta\\!-\\!D_{y}\mathcal{F}^{\varepsilon}_{\theta}(y)^{\top}\mathbf{g}(y)^{\top}\zeta-[D_{y}\mathbf{A}(y,\theta)^{\top}p]\kappa-\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}\kappa,\delta
Y)_{L^{2}}+(\hat{y}^{T}_{i},\delta Y(T))_{\mathbb{R}^{n}}$
$\displaystyle=(\dot{\delta Y}-\mathbf{A}(y,\theta)^{\top}\delta
Y-\mathbf{g}(y)D_{y}\mathcal{F}^{\varepsilon}_{\theta}(y)\delta
Y,\zeta)_{L^{2}}$
$\displaystyle\quad-([D_{y}\mathbf{A}(y,\theta)^{\top}p]\kappa+\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}\kappa,\delta
Y)_{L^{2}}-(\delta Y(T),\zeta(T))_{\mathbb{R}^{n}}$
$\displaystyle=(\mathbf{g}(y)D_{\theta}\mathcal{F}^{\varepsilon}_{\theta}(y)\delta\theta,\zeta)_{L^{2}}-([D_{y}\mathbf{A}(y,\theta)^{\top}p]\kappa+\mathbf{Q}_{1}^{\top}\mathbf{Q}_{1}\kappa,\delta
Y)_{L^{2}}-(\delta Y(T),\zeta(T)-\hat{y}_{i}^{T})_{\mathbb{R}^{n}}.$

Adding both equations finally yields

$\displaystyle(\hat{y}_{i},\delta Y)_{L^{2}}+(\hat{y}_{i}^{T},\delta
Y(T))_{\mathbb{R}^{n}}+(p_{1},\delta P)_{L^{2}}$
$\displaystyle=(\mathbf{g}(y)D_{\theta}\mathcal{F}^{\varepsilon}_{\theta}(y)\delta\theta,\zeta)_{L^{2}}+([\partial_{\theta}\mathbf{A}(y,\theta)\delta\theta]p,\kappa)_{L^{2}}$
$\displaystyle=(D_{\theta}\mathcal{F}^{\varepsilon}_{\theta}(y)^{\top}(\mathbf{g}(y)^{\top}\zeta+[D\mathbf{g}(y)\kappa]^{\top}p),\delta\theta)_{\mathcal{R}_{\varepsilon}}$

which ends the proof. ∎

We arrive at the following characterization of the gradient
$\nabla\mathcal{J}_{N}(\theta)$.

######  Theorem 23.

Let $y_{i},p_{i},\zeta_{i},\kappa_{i}\in
W_{T},~{}\widehat{\theta}_{i}\in\mathcal{R}_{\varepsilon}$ be defined as in
Proposition 21 and Lemma 22. The gradient of $\mathcal{J}_{N}$ at $\theta$ is
given by

$\displaystyle\nabla\mathcal{J}_{N}(\theta)=\sum^{N}_{i=1}\omega_{i}\left(D_{\theta}\mathcal{F}^{\varepsilon}_{\theta}(y_{i})^{\top}(\mathbf{g}(y_{i})^{\top}\zeta_{i}+[D\mathbf{g}(y_{i})\kappa_{i}]^{\top}p_{i})+\widehat{\theta}_{i}\right)+\gamma_{\varepsilon}\theta.$

##  9\. Numerical example

We finish this paper by applying the proposed learning approach to one
particular instance of Problem ($P_{y_{0}}$). Setting $I=(0,T)$ and
$\Omega=(0,2\pi)$, we consider the parabolic bilinear optimal control problem

$\displaystyle\min_{\mathcal{Y}\in L^{2}(I\times\Omega),u\in
L^{2}(I;\mathbb{R}^{3})}\left[\frac{1}{2}\int_{I}\|\mathcal{Y}(t)-\mathcal{Y}_{d}(t)\|^{2}_{L^{2}(\Omega)}+\frac{\beta}{2}|u(t)|^{2}_{\mathbb{R}^{3}}~{}\mathrm{d}t\right]+\frac{\alpha}{2}\|\mathcal{Y}(T)-\mathcal{Y}_{d}(T)\|^{2}_{L^{2}(\Omega)}$

subject to

$\displaystyle\partial_{t}\mathcal{Y}-\bigtriangleup\mathcal{Y}+\left(u_{1}\chi_{1}+u_{2}\chi_{2}+u_{3}\chi_{3}\right)\mathcal{Y}=0,$
(9.1)

as well as

$\displaystyle\mathcal{Y}(t,x)=0\quad\text{on}~{}I\times\partial\Omega,~{}\mathcal{Y}(0,x)=\mathcal{Y}_{0}(x)\quad\text{on}~{}\Omega.$

Here $\alpha>0,\beta>0$, and $\mathcal{Y}_{d}$ denotes a given desired state.
The dynamics of this infinite-dimensional system can be influenced by choosing
a time-dependent three-dimensional control input $u\in
L^{2}(I;\mathbb{R}^{3})$ which acts on the subdomains $\Omega_{1}=(0.5,1)$,
$\Omega_{2}=(2,2.5)$ and $\Omega_{3}=(4,4.5)$, respectively. The associated
characteristic functions are denoted by $\chi_{i}$, $i=1,\dots,3$.

In order to fit this problem into the setting of the current manuscript, let
$\\{\lambda_{i},\varphi_{i}\\}\in\mathbb{R}_{+}\times L^{2}(\Omega)$ denote
the first $n\in\mathbb{N}$ normalized eigenpairs of the Dirichlet Laplacian on
$\Omega$. Approximating the state dynamics $\mathcal{Y}$ as well as the
desired state by

$\displaystyle\mathcal{Y}(t,x)\approx\sum^{n}_{i=1}Y_{i}(t)\varphi_{i}(t),~{}\mathcal{Y}_{d}(t,x)\approx\sum^{n}_{i=1}Y^{i}_{d}(t)\varphi_{i}(t),~{}$

we end up with

$\displaystyle\min_{{Y}\in L^{2}(I;\mathbb{R}^{1}0),u\in
L^{2}(I;\mathbb{R}^{3})}\left[\frac{1}{2}\int_{I}|Y(t)-Y_{d}(t)|^{2}+\frac{\beta}{2}|u(t)|^{2}_{\mathbb{R}^{3}}~{}\mathrm{d}t+\frac{\alpha}{2}|Y(T)-Y_{d}(T)|^{2}_{\mathbb{R}^{1}0}\right]$
(9.2)

subject to

$\displaystyle\dot{Y}(t)+AY(t)+\sum^{3}_{i=1}u_{i}M_{i}Y(t)=0,~{}Y(0)=Y_{0}.$

where $(Y_{0})_{i}=(\mathcal{Y}_{0},\varphi_{i})_{L^{2}}$, $i=1,\dots,n$, and
the symmetric matrices $A,M_{i}\in\mathbb{R}^{n\times n}$ are given by

$\displaystyle A_{jk}=\begin{cases}0&j\neq k\\\
\lambda_{j}&\text{else}\end{cases},~{}(M_{i})_{jk}=\int_{\Omega}\phi_{j}\phi_{k}\chi_{i}(x)~{}\mathrm{d}x,\quad
i=1,2,3,~{}j,k=1\dots,n.$

###  9.1. Learning & validation setup

In the following, we determine an approximate optimal feedback law for (9.2)
by applying the learning approach detailed in Section 4. The parametrized
model $V^{\varepsilon}_{\theta}$ for the value function is given by
realizations of residual networks, as described in Section 5.1, with
$L_{\varepsilon}=2$ layers, $\operatorname{arch}(\theta)=(11,60,1)$ and
activation function $\sigma$ given by

$\displaystyle\sigma(x)=\sin(x)+\cos(x).$

This yields a total of $1440$ trainable parameters. We emphasize that the
architecture as well as the activation function were chosen based on numerical
testing. In particular, the present tests should not be mistaken as a
_quantitative_ survey but as a _proof of concept_ which highlights the
potential of learned feedbacks for optimal control and puts a focus on the
role played by the penalty parameters $\gamma_{1}$ and $\gamma_{2}$.

Given a fixed reference vector $\bar{Y}_{0}$, we randomly generate a set
$\mathbf{Y}_{0}$ of $130$ initial conditions by sampling uniformly from the
closure of $B_{1}(\bar{Y}_{0})$, Subsequently, these are split into a training
set $\mathbf{Y}^{t}_{0}$ of $N=30$ initial conditions, which is used in the
learning problem ($\mathcal{P}^{N}_{\varepsilon}$) together with uniform
weights $w_{j}=1/N$, and a validation set
$\mathbf{Y}^{v}_{0}=\mathbf{Y}_{0}\setminus\mathbf{Y}^{T}_{0}$ which we later
utilize to assess the performance of the obtained feedback.

In order to obtain a candidate for the optimal network parameters
$\theta^{*}_{\varepsilon}$, a Barzilai-Borwein method [3], is applied to the
learning problems ($\mathcal{P}^{N}_{\varepsilon}$), based on the reduced
objective functional introduced in (8.3) as well as the characterization of
its gradient in Theorem 23. For every $Y_{0}\in\mathbf{Y}^{t}_{0}$, this
approach entails the computation of the state $Y\coloneqq Y_{\theta}(Y_{0})$
and the adjoint state $P\coloneqq P_{\theta}(Y_{0})$ which satisfy

$\displaystyle\dot{Y}(t)+\left(A+\sum^{3}_{i=1}{F}^{\varepsilon}_{\theta}(t,Y(t))_{i}M_{i}\right)Y(t)$
$\displaystyle=0,~{}Y(0)=Y_{0}$