# Smooth approximation of feedback laws for infinite horizon control problems
with non-smooth value functions

Karl Kunisch  Donato Vásquez-Varas

###### Abstract

In this work the synthesis of approximate optimal and smooth feedback laws for
infinite horizon optimal control problems is addressed. In this regards,
$L^{p}$ type error bounds of the approximating smooth feedback laws are
derived, depending on either the $C^{1}$ norm of the value function or its
semi-concavity. These error bounds combined with the existence of a Lyapunov
type function are used to prove the existence of an approximate optimal
sequence of smooth feedback laws. Moreover, we extend this result to the
Hölder continuous case by a diagonalization argument combined with the Moreau
envelope. It is foreseen that these error bounds could be applied to study the
convergence of synthesis of feedback laws via data driven machine learning
methods. Additionally, we provide an example of an infinite horizon optimal
control problem for which the value functions is non-differentiable but
Lipschitz continuous. We point out that in this example no restrictions on
either the controls or the trajectories are assumed.

###### keywords:

optimal feedback control, Hamilton-Jacobi-Bellman equation, learning approach,
non-differentiability.

††journal: Computers and Mathematics with Applications

[RICAM]organization=Radon Institute for Computational and Applied Mathematics,
Austrian Academy of Sciences, addressline=Altenbergerstraße 69, city=Linz,
postcode=4040, state=Upper Austria, country=Austria
[Graz]organization=Institute of Mathematics and Scientific Computing,
University of Graz, addressline=Heinrichstraße 36, city=Graz, postcode=8010,
state=Styria, country=Austria

AMS classification: 49L12, 49J15, 49N35, 68Q32, 93B52,

##  1 Introduction

The construction of feedback laws for infinite horizon control problems is a
hard task. The classical approach depends on solving the Hamilton Jacobi
Bellman equation (HJB). Among the wellknown methods for solving the HJB
equation we mention finite difference schemes [1], semi-Lagrangian schemes
[2], and policy iteration [3, 4, 5, 6]. However, it is known that in general
these approaches suffer from the _curse of dimensionality_. In the last years
significant efforts have been dedicated to overcome this difficulty by using
machine learning and deep learning techniques. For instance we can mention the
following contributions: representation formulas [7, 8, 9, 10], approximating
the HJB equation by neural networks [11, 12, 13, 14, 15, 16, 17, 18, 19], data
driven approaches [20, 21, 22, 23, 24, 25], max-plus methods [26, 27, 28],
polynomial approximation [29, 30], tensor decomposition methods [31, 32, 33,
34, 35, 36], POD methods [37, 38], tree structure algorithms [39], and sparse
grids techniques[40, 41, 42, 43], see also the proceedings volume [44]. In
general, there is no proof concerning the fact that feedback laws constructed
via data driven approaches or the resolution of HJB by neural networks provide
optimal controls, unless the value function some smoothness of the value
function is required as is the case in [25], where the value functions is
supposed to be an element of a reproducing kernel Hilbert space. There is a
group of techniques, based on training feedback laws involving the value
function, by minimizing the average of the objective function of the control
problem over a set of initial conditions [45, 16, 46, 47, 43]. For these
approaches, convergence results have been provided under the assumption that
the value function of the control problem is $C^{1,1}$ and the optimal states
are bounded. However, these assumptions can be weakened by assuming the
existence of a sequence of smooth feedback’s converging in such a manner that
the states generated by these feedback laws remain bounded. In this work, we
provide conditions which ensure the existence of such sequences of feedback
laws. We distinguish four cases depending on whether the value function is
$C^{1}$, semi-convex, semi-concave, or $\alpha$-Hölder continuous with
$\alpha\in\left(\frac{1}{2},1\right]$. For each of them we also assume the
existence of an appropriately chosen Lyapunov function. For this purpose we
derived $L_{p}$ type error bounds with respect to the optimality of smooth
feedback laws which could also be used to study the convergence of feedback
laws provided by data driven approaches.

The structure of this work is as follows. In Section 2 we present the control
problem and some important concept of the dynamical programming approach for
synthesizing feedback laws. In Section 3 we introduce preliminary results on
semi-concave functions, viscosity solutions, and their regularization by means
of the Moreau envelope. In Section 4 we introduce necessary definitions and we
present the main assumption about the stabilizability by feedback control. In
Section 5 we state our main results which are the construction of a sequence
of feedback laws such that the state given by this approximation is bounded
and the evaluation of the objective function in the obtained controls
converges to the value function. For the proofs we refer to later sections,
mainly to section 8. Section 6 is devoted to obtaining estimates on the error
between the value function and the evaluation of the objective function of the
control problem, assuming that the state is bounded. In Section 7, a result
for the stability of the trajectories of approximations of feedback laws
constructed by using an approximation of the value function is provided. The
final section is devoted to the detailed description of an example for an
optimal control problem which admits at least two globally optimal solutions
for initial conditions in an appropriately chosen subset of the state space.
As a consequence it will follow that the value function is not $C^{1}$ on this
set of initial conditions. However, all the assumption for the applicability
of the presented results can still be verified.

##  2 Control theory and dynamic programming

We consider the following control problem

$(P)\quad\min_{u}J(y_{0},u):=\int_{0}^{\infty}\left(\ell(y)+\frac{\beta}{2}|u|^{2}\right)dt$
(2.1) $s.t.\quad\dot{y}(t)=f(y(t))+B(y(t))u(t),\quad y(0)=y_{0},$ (2.2)

where $T>0$, $\ell\in Lip_{loc}(\mathbb{R}^{d})$ is bounded from below by 0,
with $\ell(0)=0$, $y_{0}\in\mathbb{R}^{d}$, $\beta>0$, $u\in
L^{2}((0,\infty);\mathbb{R}^{m})$, $f\in
Lip_{loc}(\mathbb{R}^{d};\mathbb{R}^{d})$ with $f(0)=0$, and $B\in
Lip_{loc}(\mathbb{R}^{d};\mathbb{R}^{d\times m})$. We define the associated
value function by

$V(y_{0})=\min_{u\in L^{2}((0,\infty);\mathbb{R}^{m})}J(y_{0},u).$ (2.3)

It is known that $V$ satisfies the dynamic programming principle, i.e., for
all $T>0$:

$V(y_{0})=\min_{u\in
L^{2}((0,T);\mathbb{R}^{m})}\left\\{\int_{0}^{T}\left(\ell(y(\cdot;y_{0}))+\frac{\beta}{2}|u|^{2}\right)dt+V(y(T;y_{0}))\right\\},$
(2.4)

where $y(\cdot;y_{0})$ is the solution of (2.2). Further, if problem (2.1) has
a solution for all $y_{0}\in\Omega$ with $\Omega\subset\mathbb{R}^{d}$ open,
then $V$ satisfies the Hamilton-Jacobi-Bellman equation $\mathcal{H}(y,\nabla
v(y))=0$ in the viscosity sense, where

$\mathcal{H}(y,p)=\max_{u\in\mathbb{R}^{M}}H(y,p,u),$

and

$H(y,p,u)=\left\\{-p\cdot(f(y)+B(y)u)-\ell(y)-\frac{\beta}{2}|u|^{2}\right\\}.$
(2.5)

Thus, in the case that the cost is quadratic in $u$ and the control enters
into (2.2) in an affine manner, the HJB equation becomes

$\mathcal{H}(y,\nabla v(y))=-\ell(y)+\frac{1}{2\beta}|B^{\top}(y)\nabla
v(y)|^{2}-\nabla v(y)^{\top}f(y)=0\mbox{ in }\Omega.$ (2.6)

The following properties of $H$ and $\mathcal{H}$ will be important in the
upcoming sections:

$\begin{array}[]{l}\displaystyle\left|H(y,p_{1},u_{1})-H(y,p_{2},u_{2})\right|\leqslant\frac{\beta}{2}|u_{1}+u_{2}|\cdot|u_{1}-u_{2}|+|p_{1}|\cdot|B(y)|\cdot|u_{1}-u_{2}|+|f(y)|\cdot|p_{1}-p_{2}|\\\
\vskip 6.0pt plus 2.0pt minus
2.0pt\cr\displaystyle+|u_{2}|\cdot|B(y)|\cdot|p_{1}-p_{2}|,\end{array}$ (2.7)
$\begin{array}[]{l}\displaystyle\left|\mathcal{H}(x,p_{1})-\mathcal{H}(y,p_{1})\right|\leqslant\left(\left\lVert\ell\right\rVert_{Lip(\Omega)}+\frac{1}{\beta}\left\lVert
B\right\rVert_{Lip(\Omega;\mathbb{R}^{m\times d})}^{2}|p_{1}|^{2}+\left\lVert
f\right\rVert_{Lip(\Omega;\mathbb{R}^{d})}|p_{1}|\right)|x-y|,\end{array}$
(2.8)

for all $u_{1},u_{2}\in\mathbb{R}^{M}$, $p_{1},p_{2}\in\mathbb{R}^{d}$, and
$x,y\in\Omega$, where we use the following definition for $\left\lVert
B\right\rVert_{Lip(\Omega;\mathbb{R}^{m\times d})}$

$\left\lVert B\right\rVert_{Lip(\Omega;\mathbb{R}^{m\times
d})}=\sup_{\begin{array}[]{c}x,y\in\Omega,\\\ i\in 1,\ldots m,\\\ j\in
1,\ldots,d\end{array}}\frac{|B_{i,j}(x)-B_{i,j}(y)|}{|x-y|}+|B_{i,j}(x)-B_{i,j}(y)|$
(2.9)

Additionally, if $V$ is smooth, then $V$ is a classical solution of the HJB
equation and this allows us to construct a feedback law. Namely, if for an
initial condition $y_{0}\in\Omega$ the optimal solution of (2.1) is such that
the respective state $y^{*}$satisfies $y^{*}(t)\in\Omega$ for all
$t\in(0,\infty)$, then we can define a feedback law by

$u^{*}(y)=\operatorname*{argmax}_{u\in R^{M}}H(y,\nabla V(y),u)$ (2.10)

along $y=y^{*}(t)$, where $y^{*}$ solves the closed loop problem

$\frac{d}{dt}y^{*}(t)=f(y^{*}(t))+B(y^{*}(t))u^{*}(y^{*}(t)),\quad
y^{*}(t_{0})=y_{0}.$ (2.11)

This implies that once we have solved the HJB equation, we can obtain an
optimal feedback law by using (2.10). However, in general the value function
is not differentiable. But we will see in the following sections that it is
possible to construct a feedback such that $J$ evaluated along the controls
provided by this feedback is close to the value function. We will explain this
in more detail in Section 4. First we need some preliminary results which are
given in the next section. Throughout $V$ denotes the value function of (2.1).
We assume that it is a continuous function in $\mathbb{R}^{d}$.

##  3 Preliminary results

In this section we provide results that we will need throughout the article.
We start by recalling the definition of a viscosity solution and some results
concerning its regularity. Then, we present a definition of semi-concave
functions together with some of their useful properties. All the results and
definitions in this section can be found in [48, Chapters 1 and 2], [49,
Chapters 1, 2 and 3] and [50, Chapter 2], except for those concerning the
Moreau envelope of a Hölder function. Throughout $\Omega\subset\mathbb{R}^{d}$
denotes an open.

######  Definition 3.1 (Viscosity solution, see [50, Chapter 2, Definition
1.1]).

Let $F\in C(\Omega\times\mathbb{R}^{d})$. We say that $v\in C(\Omega)$ is a
sub-solution of

$F(y,\nabla v(y))=0\mbox{ in }\Omega$ (3.1)

in the viscosity sense if for all $\bar{y}\in\Omega$ and $\phi\in
C_{loc}^{1}(\Omega)$ such that $v-\phi$ attains a local maximum at $\bar{y}$
it holds that

$F(\bar{y},\nabla\phi(\bar{y}))\leqslant 0.$ (3.2)

Analogously, we say that $v\in C(\Omega)$ is super-solution of (3.1) in the
viscosity sense if for all $\bar{y}\in\Omega$ and $\phi\in
C_{loc}^{1}(\Omega)$ such that $v-\phi$ attains a local minimum at $\bar{y}$
it holds that

$F(\bar{y},\nabla\phi(\bar{y}))\geqslant 0.$ (3.3)

We say that $v$ is a viscosity of (3.1) if it is a sub and a super solution of
(3.1).

######  Proposition 3.1 (see [50, Chapter 2, Proposition 1.9]).

Let $F\in C(\Omega\times\mathbb{R}^{d})$. If $v$ is a viscosity solution of
(3.1) which is differentiable at $\bar{y}$, then $F(\bar{y},\nabla
v(\bar{y}))=0$. Further, if $v$ is locally Lipschitz in $\Omega$, then

$F(y,\nabla v(y))=0\mbox{ a.e. in }\Omega.$

We now give a definition of a semi-concave functions which is suitable for
what we are intending. However, it is possible to find more general
definitions in the literature.

######  Definition 3.2 (Semi-concave function, see [50, Chapter 2, Section
4.2]).

Let $\Omega$ be convex. A function $v\in C(\Omega)$ is semi-concave in
$\Omega$, if there exists a constant $C>0$ such that $x\in\Omega\mapsto
v(x)-\frac{C}{2}|x|^{2}$ is concave. If $-v$ is semi-concave we say that $v$
is semi-convex.

There are several interesting properties of semi-concave functions, however,
we only need the following:

######  Proposition 3.2.

Let $\Omega$ be convex and $v$ be semi-concave in $\Omega$. Then,

  1. 1.

(see [49, Chapter 2, Theorem 2.3.1]) $v$ is locally Lipschitz continuous in
$\Omega$ and $\nabla v\in BV_{loc}(\overline{\Omega})^{d}$.

  2. 2.

(see [49, Chapter 3, Theorem 3.3.7]) If additionally $v$ is semi-convex, then
$v\in C^{1,1}(\Omega)$.

It will be necessary to regularize semi-concave function. For this purpose, it
is useful to recall the definition of a smooth mollifier and the mollification
of a function.

######  Definition 3.3.

A compactly supported function $\rho:\mathbb{R}^{d}\to\mathbb{R}_{+}$ is a
smooth mollifier if it is infinite differentiable and it satisfies

$\int_{\mathbb{R}^{d}}\rho(x)dx=1,\quad\lim_{\varepsilon\to
0^{+}}\int_{\mathbb{R}^{d}}\phi(x)\rho(x/\varepsilon)\varepsilon^{-d}dx=\phi(0)\mbox{
for all }\phi\in C(\mathbb{R}^{d}).$

For $\varepsilon>0$ and $\phi\in L_{loc}^{1}(\mathbb{R}^{d})$ we say that
$\phi_{\varepsilon}=\phi\ast\rho_{\varepsilon}$ is a mollification of $\phi$,
where $\rho_{\varepsilon}(x)=\frac{1}{\varepsilon}\rho(x/\varepsilon)$ and
$\rho$ a smooth mollifier.

The following property of a mollification of a semi-concave function is a
consequence of Proposition 1.3.3 in [49]:

######  Proposition 3.3.

Let $\Omega$ be convex and bounded, and $v\in C(\Omega)$ be a semi-concave
function with constant $C>0$. Let $v_{\varepsilon}$ be a mollification of $v$
and set
$\Omega_{\varepsilon}=\\{x\in\Omega:dist(x,\partial\Omega)>\varepsilon\\}$.
Then $v_{\varepsilon}$ is semi-concave with constant $C>0$ in
$\Omega_{\varepsilon}$ and it satisfies

$p^{\top}\nabla^{2}v_{\varepsilon}(x)p\leqslant C|p|^{2}\mbox{ for all
}x\in\Omega_{\varepsilon}\mbox{ and for all }p\in\mathbb{R}^{d}.$

We will derive error bounds depending on regularity and structural properties
of the value function. We start with the case that $V$ is $C^{1}$ regular. In
the case that the value function is not $C^{1}$ but semi-convex, we shall
employ a Moreau approximation which is $C^{1,1}$ regular, so that the the
$C^{1}$ error bound can be used. In the semi-concave case we shall make use a
mollification, which in view of Proposition 3.3 preserves the semi-concavity.
For the non semi-concave case we shall use again mollification combined with
the Moreau envelope. In this way we shall be able to control the blow-up of
the semi-concavity constant.

Let us start by recalling the definition of the Moreau envelope.

######  Definition 3.4.

For $\phi\in C(\Omega)\cap L^{\infty}(\Omega)$ and $\lambda>0$ the Moreau
envelope of $\phi$ is defined by

$\mathcal{M}_{\lambda}\phi(y)=\inf_{x\in\Omega}\phi(x)+\frac{1}{2\lambda}|x-y|^{2}.$

The next proposition summarizes some important features of
$\mathcal{M}_{\lambda}$ for $\phi$ bounded and continuous, including that
$\mathcal{M}_{\lambda}\phi$ is semi-concave. The results can be found in [50]
except for (3.7) which is contained in [51], and (3.5), which is verified
below.

######  Proposition 3.4.

Let $\phi\in C(\Omega)\cap L^{\infty}(\Omega)$. For $\lambda>0$ we denote

$\Omega^{\lambda}_{\phi}=\left\\{x\in\Omega:dist(x,\partial\Omega)\geqslant
2\lambda^{\frac{1}{2}}\left\lVert\phi\right\rVert_{L^{\infty}(\Omega)}^{\frac{1}{2}}\right\\}$

and for $x\in\Omega$

$\mathcal{A}_{\lambda}\phi(x)=\operatorname*{argmin}_{y\in\Omega}\phi(y)+\frac{1}{2\lambda}|x-y|^{2}.$

Then the following statements hold:

  1. 1.

(see [50, Chapter 2, Lemma 4.11 and Lemma 4.12]) For all
$x\in\Omega^{\lambda}_{\phi}$ we have
$\mathcal{A}_{\lambda}\phi(x)\neq\emptyset$ and for all
$y\in\mathcal{A}_{\lambda}\phi(x)$ we have

$|x-y|\leqslant
2\lambda^{\frac{1}{2}}\left\lVert\phi\right\rVert_{L^{\infty}(\Omega)}$ (3.4)

In addition, if $\phi$ is $\alpha$-Hölder continuous in $\overline{\Omega}$,
i.e., there exist $\alpha\in(0,1]$ and $C>0$ such that

$|\phi(x_{1})-\phi(x_{2})|\leqslant C|x_{1}-x_{2}|^{\alpha}\mbox{ for all
}x_{1},x_{2}\in\overline{\Omega},$

then

$|x-y|\leqslant(2C\lambda)^{\frac{1}{2-\alpha}}.$ (3.5)

  2. 2.

(see [50, Chapter 2, Lemma 4.12]) The function $\mathcal{M}_{\lambda}\phi$
converges to $\phi$ in $C_{loc}(\Omega)$ as $\lambda\to 0^{+}$ and

$\lim_{\lambda\to
0^{+}}\sup_{y\in\mathcal{A}_{\lambda}(x)}\frac{|x-y|^{2}}{2\lambda}=0\mbox{ in
}C_{loc}(\Omega).$ (3.6)

Moreover, if $\phi$ is $\alpha$-Hölder continuous we have

$|\mathcal{M}_{\lambda}\phi(x)-\phi(x)|\leqslant
C^{\frac{2}{2-\alpha}}2^{\frac{\alpha}{2-\alpha}}\lambda^{\frac{\alpha}{2-\alpha}}\mbox{
for all }x\in\Omega^{\lambda}\phi$ (3.7)

  3. 3.

(see [50, Chapter 2, Lemma 4.11 and Lemma 4.12]) For every $\lambda>0$ the
mapping $\mathcal{M}_{\lambda}\phi$ is semi-concave with constant
$\frac{1}{\lambda}$ in $\Omega^{\lambda}_{\phi}$. In particular, for almost
every $x\in\Omega^{\lambda}_{\phi}$ we have that
$\mathcal{A}_{\lambda}\phi(x)$ is a singleton and

$\nabla\mathcal{M}_{\lambda}\phi(x)=\frac{1}{\lambda}(x-y)\mbox{ where
}\mathcal{A}_{\lambda}\phi(x)=\\{y\\}.$

  4. 4.

(see [50, Chapter 2, Proposition 4.13]) Consider $F\in
C(\Omega\times\mathbb{R}^{d})$ such that for every $R>0$ there exists $h\in
C([0,\infty);[0,\infty))$ satisfying $h(0)=0$ and

$|F(y_{1},p)-F(y_{2},p)|\leqslant h(|y_{1}-y_{2}|\cdot(1+|p|))\mbox{ for all
}y_{1},y_{2}\in\Omega\cap B(0,R),\ p\in\mathbb{R}^{d}.$

If $\phi$ is a viscosity super-solution of $F(y,\nabla\phi(y))=0$, then

$F(y,\nabla\mathcal{M}_{\lambda}\phi(y))+g_{\lambda}(y)\geqslant 0\mbox{
almost everywhere in }\Omega^{\lambda}_{\phi},$

where

$g_{\lambda}(x)=\sup_{y\in\mathcal{A}_{\lambda}\phi(x)}h\left(|x-y|\left(1+\frac{|x-y|}{\lambda}\right)\right).$

Further, it holds that

$\lim_{\lambda\to 0^{+}}g_{\lambda}(x)=0\mbox{ in }C_{loc}(\Omega).$

  5. 5.

(see [49, Chapter 3, Theorem 3.5.3]) If $\phi$ is semi-convex with constant
$C>0$, then $\mathcal{M}_{\lambda}\phi$ is
C${}^{1,1}(\Omega^{\lambda}_{\phi})$ and convex for all
$\lambda\in\left(0,\frac{1}{C}\right)$.

As announced above, to verify (3.5) let $x\in\Omega$ and
$y\in\mathcal{A}_{\lambda}\phi(x)$, where we make use of the fact that
$\mathcal{A}_{\lambda}\phi(x)$ is nonempty. Then

$\frac{1}{2\lambda}|x-y|^{2}=\mathcal{M}_{\lambda}\phi(x)-\phi(y)\leqslant\phi(x)-\phi(y)\leqslant
C|x-y|^{\alpha},$

from which the desired estimate follows. The result in 3.4 (4) is slightly
restrictive. For instance, this result can not be applied to a super-solution
of (2.6), unless we assume Lipschitz continuity of the value function. In the
following result we extend this to non-Lipschitz functions by assuming Hölder
continuity with an exponent large enough. It will used in the context
approximating the HJB equation as well as for the use of a Lyapunov function
to approximate the escape time from the domain of the approximation of the HJB
equation.

######  Proposition 3.5.

Let us assume that $\Omega$ is convex and let $F\in
C(\Omega\times\mathbb{R}^{d})$ and $q\in[1,\infty)$ be such that for every
$R>0$, there exists $h_{i}\in C([0,\infty);[0,\infty))$, satisfying
$h_{i}(0)=0$, for $i\in\\{1,2,3\\}$, and

$|F(y_{1},p)-F(y_{2},p)|\leqslant
h_{1}(|y_{1}-y_{2}|)|p|^{q}+h_{2}(|y_{1}-y_{2}|)|p|+h_{3}(|y_{1}-y_{2}|))\mbox{
for all }y_{1},y_{2}\in\Omega\cap B(0,R),\ p\in\mathbb{R}^{d}.$ (3.8)

If $\phi$ is a viscosity super-solution of $F(y,\nabla\phi(y))=0$, then

$F(y,\nabla\mathcal{M}_{\lambda}\phi(y))+g_{\lambda}(y)\geqslant 0\mbox{ a.e.
in }\Omega^{\lambda}_{\phi},$ (3.9)

where

$g_{\lambda}(x)=\sup_{y\in\mathcal{A}_{\lambda}\phi(x)}h_{1}(|x-y|)\frac{|x-y|^{q}}{\lambda^{q}}+h_{2}(|x-y|)\frac{|x-y|}{\lambda}+h_{3}(|x-y|).$
(3.10)

If additionally, $\phi$ is $\alpha-$Hölder continuous with
$\alpha\in\left(1-\frac{1}{q},1\right]$ and there exists $C>0$ such that

$\ h_{i}(s)\leqslant Cs,\mbox{ for }s\geqslant 0\mbox{ and }i\in\\{1,2,3\\},$
(3.11)

then

$\lim_{\lambda\to 0^{+}}g_{\lambda}(x)=0\mbox{ in }C_{loc}(\Omega).$ (3.12)

###### Proof.

Let $\lambda>0$ and $x\in\Omega^{\lambda}_{\phi}$. Then by (1) we have
$\mathcal{A}_{\lambda}\phi(x)\neq\emptyset$. Since $\phi$ is a super solution
of $F(y,\nabla\phi(y))=0$ and $y\mapsto\phi(x)+\frac{1}{2\lambda}|x-y|^{2}$
attains its minimum at $y\in\mathcal{A}_{\lambda}\phi(x)$, we find

$F\left(y,\frac{1}{\lambda}(x-y)\right)\geqslant 0\mbox{ for all
}y\in\mathcal{A}_{\lambda}\phi(x).$

By using (3.8) we have for all $y\in\mathcal{A}_{\lambda}\phi(x)$ the
following

$\left|F\left(x,\frac{1}{\lambda}(x-y)\right)-F\left(y,\frac{1}{\lambda}(x-y)\right)\right|\leqslant
h_{1}(|x-y|)\frac{|x-y|^{q}}{\lambda^{q}}+h_{2}(|x-y|)\frac{|x-y|}{\lambda}+h_{3}(|x-y|).$

These last two inequalities imply that

$F\left(x,\frac{1}{\lambda}(x-y)\right)+g_{\lambda}(x)\geqslant 0\mbox{ for
all }y\in\mathcal{A}_{\lambda}\phi(x).$ (3.13)

If $\mathcal{M}_{\lambda}\phi$ is differentiable at $x$, then by 3.4(3) we
have that $\mathcal{A}_{\lambda}\phi(x)=\\{y\\}$ is a singleton and
$\nabla\mathcal{M}_{\lambda}(x)=\frac{1}{\lambda}(x-y)$. Combining this, the
almost everywhere differentiability in $\Omega^{\lambda}_{\phi}$ of
$\mathcal{M}_{\lambda}\phi$, and (3.13) we get (3.9).

For (3.12), we notice that if $\phi$ is $\alpha$-Hölder continuous, then by
(3.5) there exists a constant $C>0$ such that

$|x-y|\left(1+\frac{|x-y|^{q}}{\lambda^{q}}+\frac{|x-y|}{\lambda}\right)\leqslant
C\left(\lambda^{\frac{1}{2-\alpha}}+\lambda^{\frac{q+1}{2-\alpha}-q}+\lambda^{\frac{\alpha}{2-\alpha}}\right)\mbox{
for all }y\in\mathcal{A}_{\lambda}\phi(x),$

where the constant does not depend on $x$. Consider now $\omega\subset\Omega$
compact. There exists $\lambda_{0}>0$ such that for all
$\lambda\in(0,\lambda_{0})$ we have $\omega\subset\Omega^{\lambda}_{\phi}$.
Then, the previous inequality and the fact that $\frac{q+1}{2-\alpha}-q>0$ for
$\alpha>1-\frac{1}{q}$, imply that

$\lim_{\lambda\to
0^{+}}\sup_{x\in\omega,y\in\mathcal{A}_{\lambda}\phi(x)}|x-y|\left(1+\frac{|x-y|^{q}}{\lambda^{q}}+\frac{|x-y|}{\lambda}\right)=0.$

This together with condition (3.11) concludes the proof of (3.12). ∎

######  Remark 3.5.

We apply Proposition 3.5 with $F$ given by the Hamiltonian in
$\mathcal{H}(y,\nabla v(y))=0\mbox{ in }\Omega$, where $v$ will be the value
function $V\in L^{\infty}(\Omega)\cap C(\Omega)$ associated to (2.1) which is
a viscosity solution to this equation. By (2.8) assumption (3.8) is satisfied
with $q=2$, $h_{1}(s)=\frac{1}{\beta}\left\lVert
B\right\rVert_{Lip(\Omega;\mathbb{R}^{d\times m})}s$, $h_{2}(s)=\left\lVert
f\right\rVert_{Lip(\Omega;\mathbb{R}^{d})}s$ and
$h_{3}(s)=\left\lVert\ell\right\rVert_{Lip(\Omega)}s$ independently of $R$.
Therefore Proposition 3.5 implies that $\mathcal{M}_{\lambda}V$ is a super-
solution of

$\mathcal{H}(y,\nabla v(y))+h_{\lambda}(y)=0\mbox{ in }\Omega_{V}^{\lambda},$
(3.14)

with

$h_{\lambda}(x)=\sup_{y\in\mathcal{A}_{\lambda}V(x)}\left(\left\lVert\ell\right\rVert_{Lip(\Omega)}+\frac{1}{\beta}\left\lVert
B\right\rVert_{Lip(\Omega;\mathbb{R}^{m\times
d})}^{2}\frac{|x-y|^{2}}{\lambda^{2}}+\left\lVert
f\right\rVert_{Lip(\Omega;\mathbb{R}^{d})}\frac{|x-y|}{\lambda}\right)|x-y|.$
(3.15)

Moreover, assuming that $V$ is $\alpha$-Hölder continuous with
$\alpha\in\left(\frac{1}{2},1\right]$ in $\Omega$, by (3.5) we can bound
$h_{\lambda}$ by

$h_{\lambda}(x)\leqslant
C\left(\left\lVert\ell\right\rVert_{Lip(\Omega)}\lambda^{\frac{1}{2-\alpha}}+\frac{1}{\beta}\left\lVert
B\right\rVert_{Lip(\Omega;\mathbb{R}^{m\times
d})}^{2}\lambda^{\frac{2\alpha-1}{2-\alpha}}+\left\lVert
f\right\rVert_{Lip(\Omega;\mathbb{R}^{d})}\lambda^{\frac{\alpha}{2-\alpha}}\right)\mbox{
for all }x\in\Omega^{\lambda}_{V},$

where $C$ only depends on $\alpha$ and the Hölder constant of $V$ on $\Omega$.
Additionally, if $\lambda\in(0,1]$, the last inequality implies

$h_{\lambda}(x)\leqslant
C\left(\left\lVert\ell\right\rVert_{Lip(\Omega)}+\frac{1}{\beta}\left\lVert
B\right\rVert_{Lip(\Omega;\mathbb{R}^{m\times d})}^{2}+\left\lVert
f\right\rVert_{Lip(\Omega;\mathbb{R}^{d})}\right)\lambda^{\frac{2\alpha-1}{2-\alpha}}.$

In particular $h_{\lambda}$ tends to $0$ on $\Omega$ with the specified rate.

##  4 Statement of the problem and structural hypotheses

The main purpose of this work is the construction of a sequence of
approximating optimal feedback laws and to analyze their convergence. Here we
provide the exact problem formulation and state the main structural hypotheses
that will be needed. In the following $\Subset$ denotes strict inclusion.

######  Hypothesis 4.1.

Consider a locally Lipschitz continuous function
$u:\Omega\mapsto\mathbb{R}^{m}$, $\omega\Subset\Omega$ open and $T>0$. For all
$y_{0}\in\omega$, the triplet $(u,\omega,T)$ satisfies that
$y(t;y_{0},u)\in\Omega$ for all $t\in[0,T]$, where $y(\cdot;y_{0},u)$ is the
solution of

$y^{\prime}(t)=f(y(t))+B(y(t))u(y(t)),\quad y(0)=y_{0}.$ (4.1)

######  Definition 4.1.

For $(u,\omega,T)$ satisfying 4.1, we define $\mathcal{V}_{u,T}$ by

$y_{0}\in\omega\mapsto\mathcal{V}_{u,T}(y_{0})=\int_{0}^{T}\left(\ell(y(t;y_{0},u))+\frac{\beta}{2}|u(y(t;y_{0},u))|^{2}\right)dt.$

With this definition, our objective is to prove the existence of a sequence of
controls $u_{n}\in C^{1}(\Omega;\mathbb{R}^{m})$ in feedback form, and times
$T_{n}$ tending to infinity such that

$\lim_{n\to\infty}\left\lVert\mathcal{V}_{u_{n},T_{n}}-V\right\rVert_{L^{p}(\omega)}=0$
(4.2)

for some $p\in[1,\infty]$ depending on the regularity of the value function
$V$. One of the motivations to include the time horizon $T$ into the
approximation framework, rather than setting the time horizon to $(0,\infty)$
is motivated, besides intrinsic interest, by the fact that in the context of
numerical approximation techniques for the value function $V$, many strategies
rely on data for $V$ at sample points $x_{i}\in\Omega$. These values for
$V(x_{i})$ would require to solve infinite horizon optimal control problems,
which is unfeasible, whereas approximations by finite horizon problems can be
obtained.

The following definition characterizes the choice of the feedback functions
which will be made.

######  Definition 4.2.

For a function $v$ in $C^{1}(\Omega)$, we define $u_{v}\in
C(\Omega;\mathbb{R}^{m})$ by

$u_{v}(y_{0})=-\frac{1}{\beta}B(y_{0})^{\top}\nabla v(y_{0})\mbox{, for
}y_{0}\in\Omega.$ (4.3)

######  Remark 4.3.

For $v\in C^{1}(\Omega)$, we notice that (4.3) is equivalent to

$u_{v}(y_{0})\in\operatorname*{argmax}_{u\in\mathbb{R}^{d}}H(y_{0},\nabla
v(y_{0}),u)\mbox{, for }y_{0}\in\Omega.$ (4.4)

For such a control equation (2.2) admits a local solution if $v\in
C^{1}(\Omega)$ which is unique if moreover $v\in C^{1,1}(\Omega)$.

To achieve (4.2), we will need a stability hypothesis on the optimal
trajectories of (2.1)-(2.2). We express this stability condition in the
viscosity sense and make use of it in sections Section 7 and Section 8.

######  Hypothesis 4.2.

Let $\omega\Subset\Omega$ be open and $\phi\in C(\Omega)$. There exist
$\delta>0$, $w\in C^{1}(\Omega)$, and $g\in C(\Omega)$, bounded from below by
0, such that $\omega$ is strictly contained

$\omega_{\delta}:=\\{y\in\Omega:w(y)<\sup_{y_{0}\in\omega}w(y_{0})+\delta\\}$

and $\omega_{\delta}$ has a $C^{1}$ boundary $\partial\omega_{\delta}$, and
$\phi$ is a viscosity super solution of

$-\nabla w(y)^{\top}(f(y)+B(y)u_{\phi}(y))+g(y)=0\mbox{ in }\omega_{\delta},$

i.e. for every $\bar{y}\in\omega_{\delta}$ and every $h\in
C_{loc}^{1}(\Omega)$ such that $\phi-h$ attains a local minima at $\bar{y}$
the inequality

$\nabla w(\bar{y})^{\top}(f(\bar{y})+B(\bar{y})u_{h}(\bar{y}))\leqslant
g(\bar{y})$

holds.

The existence of a Lyapunov-like function $w$ as demanded by Hypothesis 4.2 is
discussed in [52]. The condition will be used with $\phi=V$ or an
approximation thereof. The set $\omega_{\delta}$ is the region where we
approximate the value function. The regularity assumption for the boundary of
$\omega_{\delta}$ holds if $\nabla w\neq 0$ on $\partial\omega_{\delta}$.

In the analysis of approximating the escape time from $\omega_{\delta}$, the
following quantities will be required. For $\varepsilon>0$ we set

$\sigma^{1}_{\varepsilon}=\sup_{x\in\omega_{\delta},y\in
B(x,\varepsilon)}\left|g(y)-\nabla w(y)^{\top}f(y)+\nabla
w(x)^{\top}f(x)\right|$ (4.5)

and

$\sigma^{2}_{\varepsilon}=\sup_{x\in\omega_{\delta},y\in
B(x,\varepsilon)}\left|B(x)^{\top}\nabla w(x)-B(y)^{\top}\nabla w(y)\right|.$
(4.6)

For $g=0$ these quantities tend to zero with $\varepsilon$. We also need

$\sigma^{1}_{\varepsilon,\lambda}=\sup_{x\in\omega_{\delta},y\in
B(x,\varepsilon)}\left|g_{\lambda}(y)-\nabla w(y)^{\top}f(y)+\nabla
w(x)^{\top}f(x)\right|$ (4.7)

where

$\begin{array}[]{l}\displaystyle
g_{\lambda}(x)=g(x)+\sup_{y\in\mathcal{A}_{\lambda}\phi(x)}\Big{\\{}h_{w}(|x-y|)\left(\left\lVert
f\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d})}+\frac{1}{\beta}\frac{|x-y|}{\lambda}\left\lVert
B\right\rVert^{2}_{L^{\infty}(\Omega;\mathbb{R}^{d\times m})}\right)\\\ \vskip
6.0pt plus 2.0pt minus 2.0pt\cr\displaystyle+\left\lVert
f\right\rVert_{Lip(\Omega;\mathbb{R}^{d})}\left\lVert\nabla
w\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d})}|x-y|+\frac{2}{\beta}\frac{|x-y|^{2}}{\lambda}\left\lVert
B\right\rVert_{Lip(\Omega);\mathbb{R}^{d\times M}}^{2}\left\lVert\nabla
w\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d})}\Big{\\}},\end{array}$ (4.8)

and $h_{w}$ is the modulus of continuity of $\nabla w$ in $\omega_{\delta}$,
i.e.

$|\nabla w(x_{1})-\nabla w(x_{2})|\leqslant h_{w}(|x_{1}-x_{2}|)\mbox{ for all
}x_{1},x_{2}\in\omega_{\delta}.$

In Remark 7.1 we shall provide a sufficient condition guaranteeing that
$\lim_{\lambda\to 0^{+}}g_{\lambda}=g$.

##  5 Main results: approximation of viscosity solutions

Here we state the approximation results which are part of the main
contributions of this paper, under various assumptions on the regularity of
the value function $V$. The proofs will be given in the following sections. We
start with $V\in C^{1}(\Omega)$.

######  Theorem 5.1.

Let $w\in C^{1}(\overline{\Omega})$ satisfy 4.2 with $\phi=V\in C^{1}(\Omega)$
the value function of (2.1) and $g=0$. Further let $u_{\varepsilon}\in
Lip(\Omega)$ be a family of funtions such that

$\lim_{\varepsilon\to 0^{+}}\left\lVert
B(u_{\varepsilon}-u_{V})\right\rVert_{C(\omega_{\delta};\mathbb{R}^{M})}=0.$
(5.1)

Choose $\kappa:(0,\infty)\to(0,\infty)$ and $s_{0}>0$ such that

$\lim_{s\to 0^{+}}\kappa(s)=\infty\mbox{ and
}\kappa(s)s\leqslant\frac{\delta}{\left\lVert\nabla
w\right\rVert_{L^{\infty}(\omega_{\delta};\mathbb{R}^{d})}}\mbox{ for all
}s\in(0,s_{0}),$ (5.2)

and set $\tau_{\varepsilon}=\kappa(\left\lVert
B(u_{\varepsilon}-u_{V})\right\rVert_{C(\omega_{\delta};\mathbb{R}^{M})})$.
Then, there exists $\varepsilon_{0}$ such that for all
$\varepsilon\in(0,\varepsilon_{0})$ the triplet
$(u_{\varepsilon},\omega_{\delta},\tau_{\varepsilon})$ satisfies 4.1,

$\left\lVert\mathcal{V}_{u_{\varepsilon},\tau_{\varepsilon}}+V\circ
y(\tau_{\varepsilon};\cdot,u_{\varepsilon})-V\right\rVert_{L^{\infty}(\omega_{\delta})}\leqslant\beta\kappa(\left\lVert
B(u_{\varepsilon}-u_{V})\right\rVert_{C(\omega_{\delta};\mathbb{R}^{M})})\left\lVert
u_{\varepsilon}-u_{V}\right\rVert_{C(\omega_{\delta};\mathbb{R}^{M})}^{2},$
(5.3)

and consequently

$\lim_{\lambda\to 0^{+}}\tau_{\varepsilon}=\infty,\mbox{ and
}\lim_{\varepsilon\to
0^{+}}\left\lVert\mathcal{V}_{u_{\varepsilon},\tau_{\varepsilon}}+V\circ
y(\tau_{\varepsilon};\cdot,u_{\varepsilon})-V\right\rVert_{L^{\infty}(\omega_{\delta})}=0.$
(5.4)

The following result holds under a semi-convexity assumption on $V$.

######  Theorem 5.2.

Let $\Omega\subset\mathbb{R}^{d}$ be convex and bounded, $\omega\Subset\Omega$
open, $\delta>0$, and $w\in C^{1}(\overline{\Omega})$, such that they satisfy
4.2 with $\phi=V$ the value function of (2.1) and $g=0$. Assume that $V$ is
semi-convex with constant $C>0$. For $\lambda>0$ and $\varepsilon>0$, set
$V_{\varepsilon,\lambda}=\rho_{\varepsilon}\ast\mathcal{M}_{\lambda}V$ with
$\rho$ a smooth mollifier. Then there exists $\lambda_{0}>$ such that for all
$\lambda\in(0,\lambda_{0})$ there exists $\varepsilon(\lambda)$ such that

$\left\lVert\nabla
V_{\varepsilon(\lambda),\lambda}-\nabla\mathcal{M}_{\lambda}V\right\rVert^{2}_{L^{\infty}(\omega_{\delta;\mathbb{R}^{d}})}\leqslant\lambda,\mbox{
and }\omega_{\delta}+B(0,\varepsilon(\lambda))\subset\Omega^{\lambda}_{V}.$
(5.5)

Set $u_{\lambda}=u_{V_{\varepsilon(\lambda),\lambda}}$, choose
$\kappa:(0,\infty)\to(0,\infty)$ such that

$\lim_{s\to 0^{+}}\kappa(s)=\infty\mbox{ and }\lim_{s\to 0^{+}}\kappa(s)s=0,$
(5.6)

and set

$\tau_{\lambda}=\min\left\\{\delta\left(\sigma^{1}_{\varepsilon(\lambda),\lambda}+\sigma_{\varepsilon(\lambda)}^{2}\frac{\left\lVert
B^{\top}\nabla\mathcal{M}_{\lambda}V\right\rVert_{L^{\infty}(\omega_{\delta};\mathbb{R}^{m})}}{\beta}\right)^{-1},\,\kappa(\lambda)\right\\},$
(5.7)

where $\sigma^{1}_{\varepsilon(\lambda),\lambda}$ and
$\sigma_{\varepsilon(\lambda)}^{2}$ were defined in (4.7) and (4.6),
respectively. Then, for all $\lambda\in(0,\lambda_{0})$ the triplet
$(u_{\lambda},\omega,\tau_{\lambda}$ satisfies Hypothesis 4.1, and

$\lim_{\lambda\to 0^{+}}\tau_{\lambda}=\infty,\mbox{ and }\lim_{\lambda\to
0^{+}}\left\lVert\mathcal{V}_{u_{\lambda},\tau_{\lambda}}+V\circ
y(\tau_{\lambda};\cdot,u_{\lambda})-V\right\rVert_{L^{\infty}(\omega_{\delta})}=0.$
(5.8)

In the following theorem we replace the $C^{1}$ regularity of $V$ employed in
Theorem 5.1 by the semi-concavity of $V$. This implies that $V$ is locally
Lipschitz continuous, which in turns tell us by the Rademacher Theorem that it
is differentiable almost everywhere. Hence $V$ satisfies (2.6) almost
everywhere.

######  Theorem 5.3.

Let $\Omega\subset\mathbb{R}^{d}$ be convex and bounded, $\omega\Subset\Omega$
open, $\delta>0$, and $w\in C^{1}(\overline{\Omega})$, such that they satisfy
4.2 with $g=0$ and $\phi=V\in C(\Omega)$, which is assumed to be semi-concave
in $\Omega$ with constant $C>0$. Set $u_{\varepsilon}=u_{V_{\varepsilon}}$
with $V_{\varepsilon}=\rho_{\varepsilon}\ast V$ a smooth mollification of $V$,
and $p\in[1,\infty)$. Let $\kappa:(0,1)\to(0,\infty)$ be such that

$\lim_{s\to 0^{+}}\kappa(s)=\infty\mbox{ and }\lim_{s\to
0^{+}}\kappa(s)^{\frac{p-1}{p}}(e^{K\kappa(s)}-1)^{\frac{1}{p}}s^{2}=0,$ (5.9)

where

$K=\left(\frac{1}{\beta}\left(mdC\left\lVert
B\right\rVert_{L^{\infty}(\omega_{\delta};\mathbb{R}^{m\times
d})}^{2}+md^{2}\left\lVert
B\right\rVert^{2}_{Lip(\omega_{\delta};\mathbb{R}^{d\times
m})}\left\lVert\nabla
V\right\rVert_{Lip(\omega_{\delta};\mathbb{R}^{d})}\right)+d\left\lVert
f\right\rVert_{Lip(\overline{\Omega};\mathbb{R}^{d})}\right),$

and set

$\tau_{\varepsilon}=\min\left\\{\delta\left(\sigma^{1}_{\varepsilon}+\frac{\sigma^{2}_{\varepsilon}}{\beta}\left\lVert\nabla
V\right\rVert_{L^{\infty}(\omega_{\delta};R^{d})}\right)^{-1},\kappa\left(\left\lVert
u_{\varepsilon}-u_{V}\right\rVert_{L^{p}(\omega_{\delta};\mathbb{R}^{m})}\right)\right\\},$
(5.10)

where $\sigma^{1}_{\varepsilon}$ and $\sigma_{\varepsilon}^{2}$ were defined
in (4.5) and (4.6), respectively. Then there exists $\varepsilon_{0}$ such
that for all $\varepsilon\in(0,\varepsilon_{0})$ the triplet
$(u_{\varepsilon},\omega_{\varepsilon},\tau_{\varepsilon})$ satisfies
Hypothesis 4.1 and consequently

$\lim_{\varepsilon\to 0^{+}}\tau_{\varepsilon}=\infty\mbox{ and
}\lim_{\varepsilon\to
0^{+}}\left\lVert\mathcal{V}_{u_{\varepsilon},\tau_{\varepsilon}}+V\circ
y(\tau_{\varepsilon};\cdot,u_{\varepsilon})-V\right\rVert_{L^{p}(\omega_{\delta})}=0.$
(5.11)

######  Remark 5.1.

It might seem that $\kappa$ is difficult to find, but for instance,
$s\mapsto-\frac{1}{\alpha}\log(s)$ for $s\in(0,1)$ and $\alpha>\frac{K}{p+1}$,
satisfies (5.9). It is also important to observe that this provides an upper
bound on the convergence of $\mathcal{V}_{u_{\varepsilon},T_{\varepsilon}}$ to
$V$ in $L^{p}(\omega_{\delta})$.

We next relax the semi-concavity assumption of Theorem 5.3 by using a
mollification of the Moreau envelope of the value function combined with Lemma
7.1(c) concerning the escape time from $\omega_{\delta}$.

######  Theorem 5.4.

Let $\Omega\subset\mathbb{R}^{d}$ be convex and bounded, $\omega\Subset\Omega$
open, $\delta>0$, and $w\in C^{1}(\overline{\Omega})$, such that they satisfy
4.2 with $g=0$ and $\phi=V$, which is assumed to be $\alpha-$Hölder continuous
in $\Omega$ with constant $C>0$ and $\alpha\in(\frac{1}{2},1]$. Further,
assume $\nabla w$ is $\sigma$-Hölder continuous with $\sigma\in(1-\alpha,1]$.
For $\lambda>0$ and $\varepsilon>0$, set
$V_{\varepsilon,\lambda}=\rho_{\varepsilon}\ast\mathcal{M}_{\lambda}V$ a
mollification of $\mathcal{M}_{\lambda}V$. Let $p\in[1,\infty)$ be fixed and
$\eta:(0,1)\to(0,\infty)$ be such that $\lim_{s\to 0^{+}}\eta(s)=0$. Then
there exists $\lambda_{0}$ such that for all $\lambda\in(0,\lambda_{0})$ there
exist $\varepsilon(\lambda)>0$ such that

$\left\lVert\nabla
V_{\varepsilon(\lambda),\lambda}-\nabla\mathcal{M}_{\lambda}V\right\rVert_{L^{2p}(\omega_{\delta})}^{2}\leqslant\eta(\lambda),\
\varepsilon(\lambda)\leqslant\lambda^{\frac{1}{2-\alpha}},\mbox{ and
}\omega_{\delta}+B(0,\varepsilon(\lambda))\subset\Omega^{\lambda}_{V}.$ (5.12)

Set $u_{\lambda}=u_{V_{\varepsilon(\lambda),\lambda}}$ and let
$\kappa:(0,1)\to(0,\infty)$ be such that

$\lim_{s\to 0^{+}}\kappa(s)=\infty\mbox{ and }\lim_{s\to
0^{+}}\kappa(s)^{\frac{p-1}{p}}(e^{K(s)\kappa(s)}-1)^{\frac{1}{p}}\eta(s)+\kappa(s)s^{\frac{2\alpha-1}{2-\alpha}}=0,$
(5.13)

where

$K(s)=\left(\frac{md(d+1)}{\beta s}\left\lVert
B\right\rVert_{Lip(\omega_{\delta};\mathbb{R}^{d\times M})}^{2}+d\left\lVert
f\right\rVert_{Lip(\omega_{\delta};\mathbb{R}^{d})}\right).$ (5.14)

Further set

$\tau_{\lambda}=\min\left\\{\delta\left(\sigma^{1}_{\varepsilon(\lambda),\lambda}+\frac{\sigma^{2}_{\varepsilon(\lambda)}}{\beta}\left\lVert\nabla\mathcal{M}_{\lambda}V\right\rVert_{L^{\infty}(\omega_{\delta};R^{d})}\right)^{-1},\kappa\left(\lambda\right)\right\\}.$
(5.15)

where $\sigma^{1}_{\varepsilon(\lambda),\lambda}$ and
$\sigma_{\varepsilon(\lambda)}^{2}$ were defined in (4.7) and (4.6),
respectively. Then for all $\lambda\in(0,\lambda_{0})$ the triplet
$(u_{\lambda},\omega_{\delta},\tau_{\lambda})$ satisfies Hypothesis 4.1 and

$\lim_{\lambda\to 0^{+}}\tau_{\lambda}=\infty\mbox{ and }\lim_{\lambda\to
0^{+}}\left\lVert\mathcal{V}_{u_{\lambda},\tau_{\lambda}}+V\circ
y(\tau_{\lambda};\cdot,u_{\lambda})-V\right\rVert_{L^{p}(\omega_{\delta})}=0.$
(5.16)

######  Remark 5.2.

In the previous theorem, an admissible option is choosing
$\kappa(s)=\frac{1}{s^{q}}$ with
$q\in\left(0,\frac{2\alpha-1}{2-\alpha}\right)$ and
$\eta(s)=\exp(-\frac{1}{p}K(s)\kappa(s)-\frac{1}{s^{2}})$. We notice that in
5.4 more smoothness on the approximation of the value function (i.e. smaller
$\varepsilon(\lambda)$) implies a decrement on the bound of the error between
$\mathcal{V}_{u_{\lambda},T_{\lambda}}$ and $\mathcal{M}_{\lambda}V$.

We close this section with a result concerning the convergence of the
trajectories of the closed loop system for a sequence of feedback laws.

######  Theorem 5.5.

Let $\Omega\subset\mathbb{R}^{d}$ be a bounded subset of $\mathbb{R}^{d}$ and
choose $y_{0}\in\Omega$. Consider a sequence of feedback laws
$u_{\varepsilon}\in C^{1}(\overline{\Omega};\mathbb{R}^{m})$ and times
$T_{\varepsilon}>0$ with $\lim_{\varepsilon\to 0^{+}}T_{\varepsilon}=\infty$
such that $y([0,T_{\varepsilon}];y_{0},u_{\varepsilon})\subset\Omega$ and

$\lim_{\varepsilon\to
0^{+}}\mathcal{V}_{T_{\varepsilon,u_{\varepsilon}}}(y_{0})+V(y(T_{\varepsilon};y_{0},u_{\varepsilon}))=V(y_{0}),$
(5.17)

and set $y_{\varepsilon}=y(\cdot;y_{0},u_{\varepsilon})$. Then, the following
statements hold:

  1. 1.

There exists at least one limit point of $(u_{\varepsilon}\circ
y_{\varepsilon},y_{\varepsilon})$ converging in the weak topology of
$L_{loc}^{2}((0,\infty);\mathbb{R}^{m})\times
W^{1,2}_{loc}([0,\infty);{\mathbb{R}^{d}})$.

  2. 2.

For every weak limit point $(u,y)\in
L_{loc}^{2}((0,\infty);\mathbb{R}^{m})\times
W^{1,2}_{loc}([0,\infty);\mathbb{R}^{d})$ of $(u_{\varepsilon}\circ
y_{\varepsilon},y_{\varepsilon})$ we have that $u$ is an optimal control of
(2.1) and $y$ is the associated solution of (2.2). Consequently,

$\lim_{\varepsilon\to
0^{+}}\mathcal{V}_{T_{\varepsilon,u_{\varepsilon}}}(y_{0})=V(y_{0})\mbox{ and
}\lim_{\varepsilon\to 0^{+}}V(y(T_{\varepsilon};y_{0},u_{\varepsilon}))=0.$
(5.18)

  3. 3.

If the solution of (2.1) is unique, then $u_{\varepsilon}\circ
y_{\varepsilon}$ converges to the optimal solution of (2.1) and
$y_{\varepsilon}$ converges weakly to the solution of (2.2) in
$L_{loc}^{2}((0,\infty);\mathbb{R}^{m})$ and
$W_{loc}^{1,2}((0,\infty);\mathbb{R}^{d})$, respectively.

######  Corollary 5.1.

Consider $\omega\subset\Omega$, with $\Omega$ be bounded, $p\in[1,\infty)$,
and a sequence of feedbacks $u_{\varepsilon}\in
C^{1}(\overline{\Omega};\mathbb{R}^{m})$ and times $T_{\varepsilon}>0$ with
$\lim_{\varepsilon\to 0^{+}}T_{\varepsilon}=\infty$. If we have
$y([0,T_{\varepsilon}];\omega,u_{\varepsilon})\subset\Omega$, for all
$\varepsilon$ sufficiently small, and

$\lim_{\varepsilon\to
0^{+}}\left\lVert\mathcal{V}_{T_{\varepsilon,u_{\varepsilon}}}+V\circ
y(T_{\varepsilon};\cdot,u_{\varepsilon})-V\right\rVert_{L^{p}(\omega)}=0,$
(5.19)

then there exist a sub-sequences of $u_{\varepsilon}$ and $T_{\varepsilon}$
such that (1), (2) and (3) hold for almost every $y_{0}\in\omega$.
Additionally, if $p=\infty$, then (1), (2) and (3) hold for every
$y_{0}\in\omega$.

##  6 Error estimates

In this section we provide estimates on the approximation properties of
$\mathcal{V}_{u,T}$ towards a super-solution of the HJB equation (2.6) or an
approximation thereof. That is, we consider super-solutions of the following
equation

$\mathcal{H}(y,\nabla v(y))+g(y)=0,$ (6.1)

where $g\in C(\Omega)$. We notice that the Moreau envelope of the value
function is a super-solution of an equation of this type, see (3.14), (3.15).
Below the results are derived in a general setting, with $v$ playing the role
which will eventually be played by $V$ and an approximation $\bar{v}$ of $V$.
The results differ by the regularity assumption on $v$. We start with an
estimate for super-solutions of class $C^{1}$. For the convenience of the
reader we recall that
$u_{\bar{v}}(y)=-\frac{1}{\beta}B^{\top}(\bar{y})\nabla\bar{v}(\bar{y})$.

######  Lemma 6.1.

Let $v\in C^{1}(\Omega)$ be a super-solution of (6.1) with $g\in
C(\overline{\Omega})$. Consider $\bar{v}\in C^{1,1}(\Omega)$,
$y_{0}\in\Omega$, and $T$ such that $y(\cdot;y_{0},u_{\bar{v}})$ exists on
$[0,T]$ and satisfies $y([0,T];y_{0},u_{\bar{v}})\subset\Omega$. Then we have

$\mathcal{V}_{u_{\bar{v}},T}(y_{0})+v(y(T;y_{0},u_{\bar{v}}))-v(y_{0})\leqslant\int_{0}^{T}g(y(t;y_{0},u_{\bar{v}}))dt+\beta\int_{0}^{T}|u_{v}(y(t;y_{0},u_{\bar{v}}))-u_{\bar{v}}(y(t;y_{0},u_{\bar{v}}))|^{2}dt$
(6.2)

###### Proof.

To abbreviate the notation, we set $\bar{y}(t)=y(t;y_{0},u_{\bar{v}})$,
$\bar{u}(t)=u_{\bar{v}}(\bar{y}(t))$, and $u(t)=u_{v}(\bar{y}(t))$. In
particular, $\bar{y}$ satisfies
$\bar{y}^{\prime}=f(\bar{y})+B(\bar{y})\bar{u}$ and $\bar{y}(0)=y_{0}$. Here
and below the dependence of the state and control variables on $t$ is not
indicated. By (4.4) we find

$\nabla\bar{v}(\bar{y})(f(\bar{y})+B(\bar{y})\bar{u})+\ell(\bar{y})+\frac{\beta}{2}|\bar{u}|^{2}\leqslant\nabla\bar{v}(\bar{y})(f(\bar{y})+B(\bar{y})u)+\ell(\bar{y})+\frac{\beta}{2}|u|^{2}\text{
for all }u\in\mathbb{R}^{m}.$

Using that $v$ is a super-solution of (6.1) on the right-hand side of the
previous inequality we obtain

$\nabla\bar{v}(\bar{y})(f(\bar{y})+B(\bar{y})\bar{u})+\ell(\bar{y})+\frac{\beta}{2}|\bar{u}|^{2}\leqslant(\nabla\bar{v}(\bar{y})-\nabla
v(\bar{y}))(f(\bar{y})+B(\bar{y})u)+g(\bar{y}).$ (6.3)

Here we used that $u=-\frac{1}{\beta}B^{\top}(\bar{y})\nabla v(\bar{y})$. This
and $\bar{u}=-\frac{1}{\beta}B^{\top}(\bar{y})\nabla\bar{v}(\bar{y})$, will be
used in the following equality

$\begin{array}[]{l}\displaystyle(\nabla\bar{v}(\bar{y})-\nabla
v(\bar{y})))(f(\bar{y})+B(\bar{y})u)=\frac{1}{\beta}|B(\bar{y})^{\top}(\nabla
v(\bar{y})-\nabla\bar{v}(\bar{y}))|^{2}+(\nabla\bar{v}(\bar{y})-\nabla
v(\bar{y}))(f(\bar{y})+B(\bar{y})\bar{u})\\\ \vskip 6.0pt plus 2.0pt minus
2.0pt\cr\displaystyle=\beta|u-\bar{u}|^{2}+(\nabla\bar{v}(\bar{y})-\nabla
v(\bar{y}))(f(\bar{y})+B(\bar{y})\bar{u}).\end{array}$

Together with (6.3) we arrive at

$\nabla
v(\bar{y})(f(\bar{y})+B(\bar{y})\bar{u})+\ell(\bar{y})+\frac{\beta}{2}|\bar{u}|^{2}\leqslant\beta|u-\bar{u}|^{2}+g(\bar{y}).$

Integrating from $0$ to $T$ in the previous inequality we arrive at (6.2). ∎

The next theorem is a direct consequence of this lemma. It gives an
$L^{\infty}$ estimation of the approximation property of
$\mathcal{V}_{T,\bar{v}}$, for $\bar{v}$ of class $C^{1,1}$, and a super
solution of (6.1) of class $C^{1}$.

######  Theorem 6.1.

Let $v\in C^{1}(\overline{\Omega})$ be a super-solution of (6.1) in $\Omega$
with $g\in C(\overline{\Omega})$ and let 4.1 hold with $u=u_{\bar{v}}$ for
$\bar{v}\in C^{1,1}(\overline{\Omega})$. Then we have

$\sup_{y_{0}\in\omega}\left\\{\mathcal{V}_{u_{\bar{v}},T}(y_{0})+v(y(T;\cdot,u_{\bar{v}})(y_{0}))-v(y_{0})\right\\}\leqslant
T\left(\sup_{x\in\Omega}g(x)^{+}+\beta\left\lVert
u_{\bar{v}}-u_{v}\right\rVert^{2}_{L^{\infty}(\Omega)}\right).$

Our next estimation is of vital importance for the case when the value
function is not differentiable. It will be used with
$\Omega_{1}=\omega_{\delta}.$

######  Theorem 6.2.

Let ${\Omega_{1}}\subset\Omega$ be an open and bounded set with $C^{1}$
boundary, $v\in Lip(\overline{\Omega_{1}})$ be a super-solution of (6.1) in
$\Omega_{1}$ with $g\in C(\overline{\Omega_{1}})$, and let $\bar{v}\in
C^{2}(\overline{\Omega_{1}})$ be such that for some constant $C>0$

$-tr(B(y)Du_{\bar{v}}(y))\leqslant C\mbox{ for all }y\in\Omega_{1}.$ (6.4)

If 4.1 holds true with $u=u_{\bar{v}}$, then for all $p\in[1,\infty)$ the
following inequality holds

$\left\lVert\left(\mathcal{V}_{u_{\bar{v}},T}+v(y(T;\cdot,u_{\bar{v}}))-v\right)^{+}\right\rVert_{L^{p}(\omega)}\leqslant
T|\omega|^{\frac{1}{p}}\left\lVert
g\right\rVert_{L^{\infty}(\Omega_{1})}+T^{\frac{p-1}{p}}\beta\left(\frac{e^{KT}-1}{K}\right)^{\frac{1}{p}}\left\lVert
u_{v}-u_{\bar{v}}\right\rVert^{2}_{L^{2p}(\Omega_{1};\mathbb{R}^{m})},$

where

$K=\left(C+d\left\lVert
f\right\rVert_{Lip(\overline{\Omega_{1}};\mathbb{R}^{d})}+dm\left\lVert
B\right\rVert_{Lip(\overline{\Omega_{1}};\mathbb{R}^{d\times m})}\left\lVert
u\right\rVert_{L^{\infty}(\Omega_{1};\mathbb{R}^{m})}\right).$ (6.5)

This result is based on the following technical lemma which concerns the
integration along the trajectories of the closed loop problem (4.1) for a
$C^{1}$ feedback law. The proof of 6.2 is given after the proof of this lemma.

######  Lemma 6.2.

Let $\omega\Subset\Omega_{1}$, let $u\in
C^{2}(\overline{\Omega_{1}};\mathbb{R}^{M})$ be such that there exist a
positive constant $C>0$ satisfying

$-tr(B(y)Du(y))\leqslant C\mbox{ in }\Omega_{1},$ (6.6)

and let 4.1 with $u=u_{\bar{v}}$ hold true. Then for all $\phi\in
C(\overline{\Omega_{1}};\mathbb{R}^{+})$

$\int_{\omega}\int_{0}^{T}\phi(y(t;y_{0},u_{\bar{v}}))dtdy_{0}\leqslant\frac{e^{KT}-1}{K}\int_{\Omega_{1}}\phi(z)dz$

holds, where

$K=\left(C+d\left\lVert
f\right\rVert_{Lip(\overline{\Omega_{1}};\mathbb{R}^{d})}+dm\left\lVert
B\right\rVert_{Lip(\overline{\Omega_{1}};\mathbb{R}^{d\times m})}\left\lVert
u_{\bar{v}}\right\rVert_{L^{\infty}(\Omega_{1};\mathbb{R}^{m})}\right),$

and $\mathbb{R}^{m}$ is endowed with the $\ell^{\infty}$-norm.

###### Proof of Lemma 6.2.

We know that for all $t\in[0,T]$ the mapping $y_{0}\mapsto y(t;y_{0},u)$ is
$C^{1}$ in $\omega$, and its differential satisfies

$\frac{d}{dt}D_{y_{0}}y(t;y_{0},u)=A(y(t;y_{0},u))\cdot
D_{y_{0}}y(t;y_{0},u),\ D_{y_{0}}y(0;y_{0},u)=I_{d\times d},$

where $u=u_{\bar{v}}$, and the components of $A(t,y_{0})$ are given by

$A_{i,r}(y)=\frac{\partial f_{i}}{\partial
y_{r}}(y)+\sum_{j=1}^{m}\left(\frac{\partial B_{i,j}}{\partial
y_{r}}(y)u_{j}(y)+B_{i,j}(y)\frac{\partial u_{j}}{\partial y_{r}}(y)\right).$

By classical ODE theory we know that

$det(D_{y_{0}}y(t;y_{0},u))=\exp\left(\int_{0}^{t}tr(A(y(s;y_{0},u)))ds\right).$

Using (6.6) in the previous expression we get

$det(D_{y_{0}}y(t;y_{0},u))\geqslant exp\left(-tK\right)\mbox{ for all
}t\in[0,T].$ (6.7)

By the Fubini theorem and using the change of variable $z=y(t;y_{0},u)$ we
have

$\int_{\omega}\int_{0}^{T}\phi(y(t;y_{0},u))dtdy_{0}=\int_{0}^{T}\int_{y(t;\omega,u)}\frac{\phi(z)}{det(D_{y_{0}}y(t;y^{-1}(t;z,u),u))}dzdt,$

where $z\mapsto y^{-1}(t;z,u)$ denotes the inverse function of $y_{0}\mapsto
y(t;y_{0},u)$. Now, using (6.7) we get

$\int_{\omega}\int_{0}^{T}\phi(y(t;y_{0},v))dtdy_{0}\leqslant\int_{0}^{T}\int_{y(t;\omega,u)}\phi(z)e^{Kt}dzdt=\frac{e^{KT}-1}{K}\int_{\Omega_{1}}\phi(z)dz,$

which concludes the proof. ∎

###### Proof of 6.2.

Since by assumption $\partial\Omega_{1}$ is $C^{1}$ regular, there exists a
family of functions $v_{\varepsilon}$ in $C^{1}(\overline{\Omega_{1}})$ such
that

$\lim_{\varepsilon\to 0}\left\lVert
v-v_{\varepsilon}\right\rVert_{W^{1,2p}(\Omega_{1})}=0\mbox{ and }\
\lim_{\varepsilon\to 0}\left\lVert
v-v_{\varepsilon}\right\rVert_{C(\overline{\Omega_{1}})}=0,$ (6.8)

see [53, Theorem 3, Section 5.3.3]. Then, by (2.7), (6.8), and boundedness of
$\Omega_{1}$ we get

$\begin{array}[]{l}\displaystyle\lim_{\varepsilon\to 0^{+}}H(y,\nabla
v_{\varepsilon}(y),u_{v_{\varepsilon}}(y))=\lim_{\varepsilon\to
0^{+}}\mathcal{H}(y,\nabla v_{\varepsilon}(y))=\mathcal{H}(y,\nabla
v(y))\mbox{ in }L^{p}(\Omega_{1}).\end{array}$ (6.9)

Moreover we have

$H(y,\nabla
v_{\varepsilon}(y),u_{v_{\varepsilon}}(y))+h_{\varepsilon}(y)+g(y)=\mathcal{H}(y,\nabla
v_{\varepsilon}(y))+h_{\varepsilon}(y)+g(y)\geqslant 0,$ (6.10)

where $h_{\varepsilon}\in C(\overline{\Omega_{1}})$ is given by
$h_{\varepsilon}(y)=-\min\\{0,\mathcal{H}(y,\nabla
v_{\varepsilon}(y))+g(y)\\}$ . Thanks to the assumption that $v$ is a super-
solution of (6.1) and by (6.9), we have

$\lim_{\varepsilon\to 0^{+}}h_{\varepsilon}(y)=0\mbox{ in }L^{p}(\Omega_{1}).$
(6.11)

To simplify notation, we now denote $y(t;y_{0},u_{\bar{v}})$ by $y(t)$.
Applying Lemma 6.1, with $v=v_{\epsilon}$, and using (6.10) we find that

$\begin{array}[]{l}\displaystyle\mathcal{V}_{u_{\bar{v}},T}(y_{0})+v_{\varepsilon}(y(T))-v_{\varepsilon}(y_{0})\leqslant\int_{0}^{T}(h_{\varepsilon}(y(t))+g(y(t))+\beta|u_{v_{\varepsilon}}(y(t))-u_{\bar{v}}(y(t))|^{2})dt.\end{array}$
(6.12)

To finish the proof we need to apply the ${L^{p}(\Omega_{1})}$-norm on both
sides of the inequality. By the Minkowsky inequality and the Jensen inequality
we have

$\begin{array}[]{l}\displaystyle\left(\int_{\omega}\left|\int_{0}^{T}(h_{\varepsilon}(y(t))+g(y(t))+\beta|u_{v_{\varepsilon}}(y(t))-u_{\bar{v}}(y(t))|^{2})dt\right|^{p}dy_{0}\right)^{\frac{1}{p}}\\\
\vskip 6.0pt plus 2.0pt minus
2.0pt\cr\displaystyle\leqslant\left(\int_{\omega}\left|\int_{0}^{T}g(y(t))dt\right|^{p}dy_{0}\right)^{\frac{1}{p}}+\left(\int_{\omega}\left|\int_{0}^{T}h_{\varepsilon}(y(t))dt\right|^{p}dy_{0}\right)^{\frac{1}{p}}+\left(\int_{\omega}\left|\int_{0}^{T}\beta|u_{v_{\varepsilon}}(y(t))-u_{\bar{v}}(y(t))|^{2}dt\right|^{p}dy_{0}\right)^{\frac{1}{p}}\\\
\vskip 6.0pt plus 2.0pt minus 2.0pt\cr\displaystyle\leqslant
T|\omega|^{\frac{1}{p}}\left\lVert
g\right\rVert_{L^{\infty}(\Omega_{1})}+T^{\frac{p-1}{p}}\left(\int_{\omega}\int_{0}^{T}|h_{\varepsilon}(y(t))|^{p}dtdy_{0}\right)^{\frac{1}{p}}+\beta
T^{\frac{p-1}{p}}\left(\int_{\omega}\int_{0}^{T}|u_{v_{\varepsilon}}(y(t))-u_{\bar{v}}(y(t))|^{2p}dtdy_{0}\right)^{\frac{1}{p}}.\end{array}$

Then, using Lemma 6.2 we obtain

$\begin{array}[]{l}\displaystyle\left(\int_{\omega}\left(\int_{0}^{T}(h_{\varepsilon}(y(t))+g(y(t))+\beta|u_{v_{\varepsilon}}(t)-u_{\bar{v}}(t)|^{2})dt\right)^{p}dy_{0}\right)^{\frac{1}{p}}\\\
\vskip 6.0pt plus 2.0pt minus 2.0pt\cr\displaystyle\leqslant
T|\omega|^{\frac{1}{p}}\left\lVert
g\right\rVert_{L^{\infty}(\Omega_{1})}+T^{\frac{p-1}{p}}\left(\frac{e^{KT}-1}{K}\right)^{\frac{1}{p}}\left(\left\lVert
h_{\varepsilon}\right\rVert_{L^{p}(\Omega_{1})}+\beta\left\lVert
u_{v_{\varepsilon}}-u_{\bar{v}}\right\rVert^{2}_{L^{2p}(\Omega_{1})}\right).\end{array}$

By using this in (6.12) we get

$\begin{array}[]{l}\displaystyle\left\lVert\left(\mathcal{V}_{u_{\bar{v}},T}+v_{\varepsilon}\circ
y(T;\cdot,u_{\bar{v}})-v_{\varepsilon}\right)^{+}\right\rVert_{L^{p}(\omega)}\\\
\vskip 6.0pt plus 2.0pt minus 2.0pt\cr\displaystyle\leqslant
T|\omega|^{\frac{1}{p}}\left\lVert
g\right\rVert_{L^{\infty}(\Omega_{1})}+T^{\frac{p-1}{p}}\left(\frac{e^{KT}-1}{K}\right)^{\frac{1}{p}}\left(\left\lVert
h_{\varepsilon}\right\rVert_{L^{p}(\Omega_{1})}+\beta\left\lVert
u_{v_{\varepsilon}}-u_{\bar{v}}\right\rVert^{2}_{L^{2p}(\Omega_{1})}\right)\end{array}$
(6.13)

Letting $\varepsilon\to 0^{+}$ and thanks to (6.13), (6.8) and (6.11) we get
the desired result. ∎

######  Remark 6.1.

It is worth to mentioning that these error bounds could be applied to study
the convergence of data-driven approaches. Consider $\Omega_{1}$, $T$, and
$\omega$ as in (6.2), and $v_{\theta}\in C^{2}(\Omega)$ obtained by some data-
driven approach as an approximation of the value function $V$ where $\theta$
are the parameters of this method. Setting $u_{\theta}=u_{v_{\theta}}$, 6.2
leads to an $L^{p}(\omega)$ error bound between $\mathcal{V}_{u_{\theta}}$ and
$V$ depending on the semi-concavity of $v_{\theta}$ as long as 4.1 is met with
$u=u_{\theta}$. Hence, in order to use this error bound for data-driven
approaches it would be important to study the capability of these approaches
ensuring 4.1 and of controlling the semi-concavity of $v_{\theta}$.

##  7 Escape time estimates

In Section 6 we have assumed that the state is bounded until some time $T>0$,
see 4.1. Following the notation of 4.2, here we give an estimate from below
for the escape time of the trajectories of (4.1) for a feedback law given by
an approximation of $\phi$. We consider three cases depending on the
regularity of $\phi$. The first one corresponds to $\phi\in C^{1}(\Omega)$,
the second one the semi-concavity of $\phi$ and the last one concerns the
general case with $\phi$ continuous. In the context of proving the results of
Section 5, we aim to apply the results of this section with $V$ playing the
role of $\phi$.

######  Lemma 7.1.

Let hypothesis 4.2 be satisfied.

  1. 1.

If $\phi\in C^{1}(\Omega)$, consider $u\in Lip(\Omega;\mathbb{R}^{m})$, and
let $\hat{T}$ be the maximum $T>0$ such that
$y([0,T];\omega,u)\subset\omega_{\delta}$. Then the following holds

$\hat{T}\left(\left\lVert
B(u-u_{\phi})\right\rVert_{C(\omega_{\delta};\mathbb{R}^{M})}\left\lVert\nabla
w\right\rVert_{C(\omega_{\delta};\mathbb{R}^{d})}+\max_{x\in\omega_{\delta}}g(x)\right)\geqslant\delta.$
(7.1)

  2. 2.

If $\phi\in Lip(\Omega)$, set $u_{\varepsilon}=u_{\phi_{\varepsilon}}$ with
$\phi_{\varepsilon}=\phi\ast\rho_{\varepsilon}$ a mollification of $\phi$, and
let $T_{\varepsilon}$ be the maximum $T$ such that
$y([0,T];\omega,u_{\varepsilon})\subset\omega_{\delta}$. Then there exists
$\varepsilon_{0}$ such that all $\varepsilon\in(0,\varepsilon_{0})$ we have

$T_{\varepsilon}\left(\sigma^{1}_{\varepsilon}+\frac{\sigma^{2}_{\varepsilon}}{\beta}\left\lVert
B^{\top}\nabla\phi\right\rVert_{L^{\infty}(\omega_{\delta};\mathbb{R}^{d})}\right)\geqslant\delta.$
(7.2)

where $\sigma_{\varepsilon}^{1}$ is defined in (4.5) and
$\sigma_{\varepsilon}^{2}$ in (4.6).

  3. 3.

Let $\phi\in C(\Omega)$ and set
$u_{\varepsilon,\lambda}=u_{\phi_{\varepsilon,\lambda}}$, where
$\phi_{\varepsilon,\lambda}=\mathcal{M}_{\lambda}\phi\ast\rho_{\varepsilon}$
is a mollification of $\mathcal{M}_{\lambda}\phi$, and let
$T_{\varepsilon,\lambda}$ be the maximum $T$ such that
$y(t;\omega,u_{\varepsilon,\lambda})\subset\omega_{\delta}$ for all
$t\in[0,T]$. Then there exists $\lambda_{0}>0$ such that for all
$\lambda\in(0,\lambda_{0})$ there exits $\varepsilon(\lambda)$ satisfying that
for all $\varepsilon\in(0,\varepsilon(\lambda))$

$T_{\varepsilon,\lambda}\cdot\left(\sigma^{1}_{\varepsilon,\lambda}+\frac{\sigma^{2}_{\varepsilon}}{\beta}\left\lVert
B^{\top}\nabla\mathcal{M}_{\lambda}\phi\right\rVert_{L^{\infty}{(\omega_{\delta})}}\right)\geqslant\delta,$
(7.3)

where $\sigma^{2}_{\varepsilon}$ is given in (4.6) and
$\sigma^{1}_{\varepsilon,\lambda}$ is given by (4.7).

###### Proof.

  1. 1.

_Proof of (a)_ Let $y_{0}\in\omega$ be arbitrary and denote
$y(t)=y(t;y_{0},u)$ for $t\in[0,\hat{T}]$. By 4.2 we deduce

$\begin{array}[]{l}\displaystyle\nabla
w(y(t))^{\top}(f(y(t))+B(y(t))u(y(t)))\\\ \vskip 6.0pt plus 2.0pt minus
2.0pt\cr\displaystyle\leqslant\nabla
w(y(t))^{\top}(f(y(t))+B(y(t))u_{\phi}(y(t)))+\nabla
w(y(t))^{\top}B(y(t))(u(y(t))-u_{\phi}(y(t)))\\\ \vskip 6.0pt plus 2.0pt minus
2.0pt\cr\displaystyle\leqslant g(y(t))+\nabla
w(y(t))^{\top}B(y(t))(u(y(t))-u_{\phi}(y(t))).\end{array}$

Integrating from $0$ to $t$ with $t\in[0,\hat{T}]$ we obtain

$\begin{array}[]{l}\displaystyle
w(y(t))-w(y_{0})\leqslant\int_{0}^{t}\left(g(y(s))+\nabla
w(y(s))^{\top}B(y(s))(u(y(s))-u_{\phi}(y(s)))\right)ds\\\ \vskip 6.0pt plus
2.0pt minus 2.0pt\cr\displaystyle\leqslant
t\left(\sup_{x\in\omega_{\delta}}g(x)+\left\lVert
B(u-u_{\phi})\right\rVert_{L^{\infty}(\omega_{\delta};\mathbb{R}^{d})}\left\lVert\nabla
w\right\rVert_{L^{\infty}(\omega_{\delta};\mathbb{R}^{d})}\right).\end{array}$

If $\hat{T}$ does not satisfy (7.1), then we have $\hat{T}<+\infty$ and by the
previous inequality there exists $y_{0}\in\omega$ such that

$w(y(\hat{T}))-w(y_{0})<\delta.$

Thus, by continuity there exists $\tilde{T}>\hat{T}$ such that
$w(y(\tilde{T}))-w(y_{0})\leqslant\delta$ and
$y([0,\tilde{T}];y_{0},u)\subset\omega_{\delta}$. Since this contradicts the
definition of $\hat{T}$, we obtain that (7.1) holds.

  2. 2.

_Proof of ( 2)_ Since $\omega_{\delta}\Subset\Omega$, there exists
$\varepsilon_{0}$ such that for all $\varepsilon<\varepsilon_{0}$ we have
$\omega_{\delta}+B(0,\varepsilon)\subset\Omega$. Consider $\rho_{\varepsilon}$
a family of mollifiers. Since $\rho_{\varepsilon}$ are positive and (4.2) we
have

$\nabla w^{\top}(f+Bu_{\phi})\ast\rho_{\varepsilon}(x)\leqslant
g\ast\rho_{\varepsilon}(x)\mbox{ for all }x\in\Omega_{\varepsilon},$

with $\Omega_{\varepsilon}$ defined as in 3.3. Then for $x\in\omega_{\delta}$
we can write

$\begin{array}[]{l}\displaystyle\nabla
w(x)^{\top}(f(x)+B(x)u_{\varepsilon}(x))\leqslant\nabla
w(x)^{\top}B(x)u_{\varepsilon}(x)-\left(\nabla
w^{\top}Bu_{\phi}\right)\ast\rho_{\varepsilon}(x)\\\ \vskip 6.0pt plus 2.0pt
minus 2.0pt\cr\displaystyle+g\ast\rho_{\varepsilon}(x)+\nabla
w^{\top}(x)f(x)-\nabla w^{\top}f\ast\rho_{\varepsilon}(x).\end{array}$ (7.4)

We can bound the first two terms in the right-hand side of (7.4) as follows

$\begin{array}[]{l}\displaystyle\nabla
w(x)^{\top}B(x)u_{\varepsilon}(x)+\frac{1}{\beta}\int_{B(x,\varepsilon)}\nabla
w(y)^{\top}B(y)B^{\top}(y)\nabla\phi(y)\rho_{\varepsilon}(x-y)dy\\\ \vskip
6.0pt plus 2.0pt minus
2.0pt\cr\displaystyle=-\frac{1}{\beta}\int_{B(x,\varepsilon)}\left(\nabla
w(x)^{\top}B(x)B^{\top}(y)-\nabla
w(y)^{\top}B(y)B^{\top}(y)\right)\nabla\phi(y)\rho_{\varepsilon}(x-y)dy\\\
\vskip 6.0pt plus 2.0pt minus
2.0pt\cr\displaystyle\leqslant\frac{1}{\beta}\sup_{y\in
B(x,\varepsilon)}\left|B^{\top}(x)\nabla w(x)^{\top}-B(x)^{\top}(y)\nabla
w(y)\right|\cdot\left\lVert
B^{\top}\nabla\phi\right\rVert_{L^{\infty}(\omega_{\delta};\mathbb{R}^{d})}=\frac{\sigma_{\varepsilon}^{2}}{\beta}\left\lVert
B^{\top}\nabla\phi\right\rVert_{L^{\infty}(\omega_{\delta};\mathbb{R}^{d})}.\end{array}$
(7.5)

For the remaining terms of the right-hand side of (7.4) we have

$g\ast\rho_{\varepsilon}(x)+\nabla w^{\top}(x)f(x)-\nabla
w^{\top}f\ast\rho_{\varepsilon}(x)\leqslant\sigma^{1}_{\varepsilon}.$ (7.6)

Then by using (7.5) and (7.6) in (7.4) we to arrive to

$\nabla
w(x)^{\top}(f(x)+B(x)u_{\varepsilon}(x))\leqslant(\sigma^{1}_{\varepsilon}+\frac{\sigma_{\varepsilon}^{2}}{\beta}\left\lVert
B^{\top}\nabla\phi\right\rVert_{L^{\infty}(\omega_{\delta};\mathbb{R}^{d})}).$

Integrating from 0 to $t\in(0,T_{\varepsilon})$ we obtain

$w(y(t;y_{0},u_{\varepsilon}))\leqslant
w(y_{0})+t(\sigma^{1}_{\varepsilon}+\frac{\sigma_{\varepsilon}^{2}}{\beta}\left\lVert
B^{\top}\nabla\phi\right\rVert_{L^{\infty}(\omega_{\delta};\mathbb{R}^{d})})$

for all $t\in[0,T_{\varepsilon}]$ and all $y_{0}\in\omega$. In particular, if
$T_{\varepsilon}$ does not satisfy (7.2), then $T_{\varepsilon}<\infty$ and
there exists $y_{0}\in\omega$ such that

$w(y(T_{\varepsilon};y_{0},u_{\varepsilon}))<w(y_{0})+\delta.$

From this, (7.2) follows as in the end of the proof of (1).

  3. 3.

_Proof of ( 3)_ Since $\omega_{\delta}\subset\Omega$ there exists
$\lambda_{0}$ such that for all $\lambda\in(0,\lambda_{0})$ we have that
$\omega_{\delta}\Subset\Omega_{\phi}^{\lambda}$ which in turn implies that
there exists $\varepsilon(\lambda)$ satisfying that for all
$\varepsilon\in(0,\varepsilon(\lambda))$ it holds that
$\omega_{\delta}+B(0,\varepsilon)\Subset\Omega_{\phi}^{\lambda}$. Aiming for
an application of Proposition 3.5 we define for $x\in\overline{\Omega}$ and
$p\in\mathbb{R}^{d}$ we define

$F(x,p)=-\nabla w(x)\left(f(x)-\frac{1}{\beta}B(x)B^{\top}(x)p\right).$

We estimate for $x_{1},x_{2}\in\Omega$ and $p\in\mathbb{R}^{d}$

$\begin{array}[]{l}\displaystyle|F(x_{1},p)-F(x_{2},p)|\leqslant\Big{\\{}h_{w}(|x_{1}-x_{2}|)\left(\left\lVert
f\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d})}+\frac{1}{\beta}|p|\left\lVert
B\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d\times m})}^{2}\right)\\\ \vskip
6.0pt plus 2.0pt minus 2.0pt\cr\displaystyle+\left\lVert
f\right\rVert_{Lip(\Omega;\mathbb{R}^{d})}\left\lVert\nabla
w\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d})}|x_{1}-y_{2}|+\frac{2}{\beta}|x_{1}-x_{2}|\cdot|p|\left\lVert
B\right\rVert_{Lip(\Omega);\mathbb{R}^{d\times M}}^{2}\left\lVert\nabla
w\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d})}\Big{\\}}.\end{array}$

For $s\geqslant 0$ we define $h_{1}(s)=0$,

$h_{2}(s)=\frac{1}{\beta}h_{w}(s)\left\lVert
B\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d\times
n})}^{2}+\frac{2s}{\beta}\left\lVert
B\right\rVert_{Lip(\Omega;\mathbb{R}^{d\times m})}^{2}\left\lVert\nabla
w\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d})}$

and

$h_{3}(s)=h_{\omega}(s)\left\lVert
f\right\rVert_{L^{\infty}(\Omega);\mathbb{R}^{d}}+s\left\lVert
f\right\rVert_{Lip(\Omega;\mathbb{R}^{d})}\left\lVert\nabla
w\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d})}.$

We notice that $F$ and $h_{1}$, $h_{2}$ and $h_{3}$ satisfy (3.8). Combining
this with the fact that $\phi$ and $w$ satisfy 4.2 and using 3.5 we obtain
that $\mathcal{M}_{\lambda}\phi$ is a viscosity super-solution of

$-\nabla
w(y)^{\top}(f(y)-\frac{1}{\beta}B(y)B^{\top}(y)\nabla\mathcal{M}_{\lambda}\phi(y))+g_{\lambda}(y)=0\mbox{
in }\omega_{\delta}.$

Therefore $\mathcal{M_{\lambda}\phi}$ and $w$ satisfy 4.2 with
$g=g_{\lambda}$, which enables us to use (2) to conclude (7.3).

∎

######  Remark 7.1.

In Lemma 7.1(3), we notice that in view of 3.4, $g_{\lambda}$ converges to $g$
in $C(\omega_{\delta})$ if $\nabla w$ is $\sigma-$Hölder continuous and $\phi$
is $\alpha$-Hölder continuous with $\sigma\in\left(1-\alpha,1\right]$. Indeed

$\begin{array}[]{l}\displaystyle|g_{\lambda}(x)-g(x)|\leqslant\sup_{y\in\mathcal{A}_{\lambda}\phi(x)}\Big{\\{}C\left(\left\lVert
f\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d})}|x-y|^{\sigma}+\frac{1}{\beta}\frac{|x-y|^{\sigma+1}}{\lambda}\left\lVert
B\right\rVert^{2}_{L^{\infty}(\Omega;\mathbb{R}^{d\times m})}\right)\\\ \vskip
6.0pt plus 2.0pt minus 2.0pt\cr\displaystyle+\left\lVert
f\right\rVert_{Lip(\Omega;\mathbb{R}^{d})}\left\lVert\nabla
w\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d})}|x-y|+\frac{2}{\beta}\frac{|x-y|^{2}}{\lambda}\left\lVert
B\right\rVert_{Lip(\Omega);\mathbb{R}^{d\times M}}^{2}\left\lVert\nabla
w\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d})}\Big{\\}}.\end{array}$

By (3.5) we have $|x-y|\leqslant C\lambda^{\frac{1}{2-\alpha}}$, then using
this in the previous inequality we have

$\begin{array}[]{l}\displaystyle|g_{\lambda}(x)-g(x)|\leqslant\Big{\\{}C\left(\left\lVert
f\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d})}\lambda^{\frac{\sigma}{2-\alpha}}+\frac{1}{\beta}\lambda^{\frac{\sigma+\alpha-1}{2-\alpha}}\left\lVert
B\right\rVert^{2}_{L^{\infty}(\Omega;\mathbb{R}^{d\times m})}\right)\\\ \vskip
6.0pt plus 2.0pt minus 2.0pt\cr\displaystyle+\left\lVert
f\right\rVert_{Lip(\Omega;\mathbb{R}^{d})}\left\lVert\nabla
w\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d})}\lambda^{\frac{2}{2-\alpha}}+\frac{2}{\beta}\lambda^{\frac{\alpha}{2-\alpha}}\left\lVert
B\right\rVert_{Lip(\Omega);\mathbb{R}^{d\times M}}^{2}\left\lVert\nabla
w\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d})}\Big{\\}},\end{array}$ (7.7)

Then, if $\sigma\in(1-\alpha,1]$ we obtain that $|g_{\lambda}(x)-g(x)|$ tends
to 0 as $\lambda$ goes to $0^{+}$ uniformly in $\omega_{\delta}$. Moreover, if
$\phi$ is Lipschitz continuous, then we do not need to impose further
regularity on $w$. The reason for this to hold is that by (3.5) we have
$\frac{|x-y|}{\lambda}\leqslant
2\left\lVert\nabla\phi\right\rVert_{L^{\infty(\omega_{\delta};\mathbb{R}^{d})}},$
which implies

$\begin{array}[]{l}\displaystyle|g_{\lambda}(x)-g(x)|\leqslant\sup_{y\in\mathcal{A}_{\lambda}\phi(x)}h_{w}(|x-y|)\Big{\\{}C\left(\left\lVert
f\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d})}+\frac{1}{\beta}\left\lVert\nabla\phi\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d})}\left\lVert
B\right\rVert^{2}_{L^{\infty}(\Omega;\mathbb{R}^{d\times m})}\right)\\\ \vskip
6.0pt plus 2.0pt minus 2.0pt\cr\displaystyle+\left\lVert
f\right\rVert_{Lip(\Omega;\mathbb{R}^{d})}\left\lVert\nabla
w\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d})}|x-y|+\frac{2}{\beta}\frac{|x-y|^{2}}{\lambda}\left\lVert
B\right\rVert_{Lip(\Omega);\mathbb{R}^{d\times M}}^{2}\left\lVert\nabla
w\right\rVert_{L^{\infty}(\Omega;\mathbb{R}^{d})}\Big{\\}}.\end{array}$

Then by (3.4) and (3.6) we get that $|g_{\lambda}(x)-g(x)|$ tends to 0 as
$\lambda$ tends to $0^{+}$ uniformly in $\omega_{\delta}$. These estimates
will be used in the proofs of Theorems 5.2 and 5.4 below.

##  8 Proof of the main results

###### Proof of 5.1.

Let $\tilde{T}_{\varepsilon}$ be the largest time such that
$y_{\varepsilon}([0,\tilde{T}_{\varepsilon}];\omega,u_{\varepsilon})\subset\omega_{\delta}.$
By Lemma 7.1 (a) with $g=0$ we know that

$\tilde{T}_{\varepsilon}\geqslant\frac{\delta}{\left\lVert
B(u_{\varepsilon}-u_{V})\right\rVert_{L^{\infty}(\omega_{\delta};\mathbb{R}^{d})}\left\lVert\nabla
w\right\rVert_{L^{\infty}(\omega_{\delta};\mathbb{R}^{d})}}.$

Consequently, by (5.1) and (5.2) there exists $\varepsilon_{0}>0$ such that
for all $\varepsilon\leqslant\varepsilon_{0}$ we have

$\tau_{\varepsilon}\leqslant\frac{\delta}{\left\lVert
B(u_{\varepsilon}-u_{V})\right\rVert_{L^{\infty}(\omega_{\delta};\mathbb{R}^{d})}\left\lVert\nabla
w\right\rVert_{L^{\infty}(\omega_{\delta};\mathbb{R}^{d})}}$

and thus $\tau_{\varepsilon}\leqslant\tilde{T}_{\varepsilon}$ and 4.1 holds.
Hence, we can apply 6.1 and by using (5.2) we obtain (5.3). ∎

###### Proof of 5.2.

Since $\omega_{\delta}\Subset\Omega$ and $V$ is semi-convex, then by 3.4 there
exists $\lambda_{0}>0$ such that $\omega_{\delta}\subset\Omega^{\lambda}_{V}$
and $\mathcal{M}_{\lambda}V$ is $C^{1,1}$ in $\Omega^{\lambda}_{V}$. Using a
diagonalization argument and the properties of mollification we have get
(5.5). Since $V$ is semi-convex in $\Omega$, it is Lipschitz continuous in
$\omega_{\delta}$. By 7.1 we have that $\lim_{\lambda\to 0^{+}}g_{\lambda}=0$,
where $g_{\lambda}$ was defined in (4.8). Together with the definitions of
$\sigma^{1}_{\varepsilon(\lambda),\lambda}$ and
$\sigma^{2}_{\varepsilon(\lambda)}$ we obtain

$\lim_{\lambda\to
0^{+}}\left(\sigma^{1}_{\varepsilon(\lambda),\lambda}+\frac{\sigma^{2}_{\varepsilon(\lambda)}}{\beta}\left\lVert\nabla\mathcal{M}_{\lambda}V\right\rVert_{L^{\infty}(\omega_{\delta})}\right)=0.$
(8.1)

Combining this with (5.6), we get $\lim_{\lambda\to
0^{+}}\tau_{\lambda}=\infty$. Moreover, by Lemma 7.1 (3) we obtain
$y([0,\tau_{\lambda}];\omega,u_{\lambda})\subset\omega_{\delta}$ for
$\lambda\in(0,\lambda_{0})$, where
$u_{\lambda}=u_{V_{\varepsilon(\lambda),\lambda}}$.

For the second part of (5.8) we observe that $\mathcal{M}_{\lambda}V$
satisfies (3.14). Hence, the hypotheses of 6.1 are satisfied by
$V_{\varepsilon(\lambda),\lambda}$ and $\mathcal{M}_{\lambda}V$, which implies
that

$\begin{array}[]{l}\displaystyle\left\lVert\left(\mathcal{V}_{u_{\lambda},\tau_{\lambda}}(y_{0})+\mathcal{M}_{\lambda}V\circ
y(\tau_{\lambda};\cdot,u_{{\lambda}})-\mathcal{M}_{\lambda}V\right)^{+}\right\rVert_{L^{\infty}(\omega)}\leqslant\tau_{\lambda}\left(\left\lVert
h_{\lambda}\right\rVert_{L^{\infty}(\omega_{\delta})}+\beta\left\lVert
u_{\lambda}-u_{\mathcal{M}_{\lambda}V}\right\rVert^{2}_{L^{\infty}(\omega_{\delta};\mathbb{R}^{m})}\right),\end{array}$
(8.2)

where $h_{\lambda}$ is defined in (3.15). As was pointed out in 3.5, since $V$
is Lipschitz continuous in $\omega_{\delta}$, then $\left\lVert
h_{\lambda}\right\rVert_{L^{\infty}(\omega_{\delta})}\leqslant K\lambda$ for
some constant $K$ independent of $\lambda$. Applying this estimate and (5.5)
in (8.2), we obtain

$\begin{array}[]{l}\displaystyle\left\lVert\left(\mathcal{V}_{u_{\lambda},\tau_{\lambda}}(y_{0})+\mathcal{M}_{\lambda}V\circ
y(\tau_{\lambda};\cdot,u_{{\lambda}})-\mathcal{M}_{\lambda}V\right)^{+}\right\rVert_{L^{\infty}(\omega)}\leqslant\tau_{\lambda}\left(K+\frac{\left\lVert
B\right\rVert^{2}_{L^{\infty}(\omega_{\delta};\mathbb{R}^{d\times
m})}}{\beta}\right)\lambda.\end{array}$ (8.3)

This together with the fact that $\mathcal{M}_{\lambda}V$ converges to $V$ in
$C_{loc}(\Omega)$ and (5.6) imply (5.8). ∎

###### Proof of 5.3.

Let $\varepsilon_{0}>0$ be such that
$\omega_{\delta}+B(0,\varepsilon)\subset\Omega$ for all
$\varepsilon\in(0,\varepsilon_{0})$. Then by the properties of mollification
we find

$\lim_{\varepsilon\to 0^{+}}\left\lVert
u_{\varepsilon}-u_{V}\right\rVert_{L^{p}(\omega_{\delta};\mathbb{R}^{M})}=0,$
(8.4)

where $u_{\varepsilon}=u_{V_{\varepsilon}}$, and by Proposition 3.3 we have

$\begin{array}[]{l}\displaystyle-
tr(B(y)Du_{\varepsilon}(y))\leqslant\frac{1}{\beta}tr(B(y)B(y)^{\top}\nabla
V_{\varepsilon}^{2})+\frac{md^{2}}{\beta}\left\lVert
B\right\rVert_{Lip(\omega_{\delta})}^{2}\left\lVert\nabla
V\right\rVert_{L^{\infty}(\omega_{\delta})}\\\ \vskip 6.0pt plus 2.0pt minus
2.0pt\cr\displaystyle\leqslant C\frac{1}{\beta}\left(md\left\lVert
B\right\rVert_{L^{\infty}(\omega_{\delta};\mathbb{R}^{d\times
m})}^{2}+md^{2}\left\lVert
B\right\rVert_{Lip(\omega_{\delta};\mathbb{R}^{d\times
m})}^{2}\left\lVert\nabla
V\right\rVert_{L^{\infty}(\omega_{\delta};\mathbb{R}^{d})}\right),\end{array}$
(8.5)

for all $y\in\omega_{\delta}$ and $\varepsilon\in(0,\varepsilon_{0})$.
Therefore (6.4) is satisfied with $\Omega_{1}=\omega_{\delta}$ and
$\bar{v}=V_{\varepsilon}$.

By Lemma 7.1 (2) and (5.10) we have that
$y([0,\tau_{\varepsilon}];\omega,u_{\varepsilon})\subset\omega_{\delta}$.
Further, by (5.9), (5.10) and the definition of $\sigma^{1}_{\varepsilon}$ and
$\sigma^{2}_{\varepsilon}$ we obtain $\lim_{\varepsilon\to
0^{+}}\tau_{\varepsilon}=\infty.$ Moreover, all the hypotheses of 6.2 are met
with $\Omega_{1}=\omega_{\delta}$,
$\bar{v}=u_{\varepsilon}=u_{V_{\varepsilon}}$, and $v=V$, and thus we have

$\left\lVert\left(\mathcal{V}_{u_{\varepsilon},\tau_{\varepsilon}}(y_{0})+V\circ
y(\tau_{\varepsilon};\cdot,u_{\varepsilon})-V\right)^{+}\right\rVert_{L^{p}(\omega)}\leqslant\tau_{\varepsilon}^{\frac{p-1}{p}}\beta\left(\frac{e^{K\tau_{\varepsilon}}-1}{K}\right)^{\frac{1}{p}}\left\lVert
u_{\varepsilon}-u_{V}\right\rVert^{2}_{L^{2p}(\omega_{\delta};\mathbb{R}^{m})}.$
(8.6)

By virtue of (5.9) and (5.10), then (5.11) follows from (8.6). ∎

###### Proof of 5.4.

Since $\omega_{\delta}\Subset\Omega$ there exists $\lambda_{0}\in(0,1)$ such
that for all $\lambda\in(0,\lambda_{0})$ we have
$\omega_{\delta}\subset\Omega_{V}^{\lambda}$ which in turns implies that there
exists $\varepsilon(\lambda)$ satisfying that for all
$\varepsilon\in(0,\varepsilon_{0}(\lambda))$ it holds that
$\omega_{\delta}+B(0,\varepsilon)\Subset\Omega_{\phi}^{\lambda}$. Let us
denote by $V_{\varepsilon}$ as a mollification of $V$, where $\varepsilon>0$.
We choose $\varepsilon(\lambda)$ such that
$\varepsilon(\lambda)\leqslant\min(\lambda^{\frac{1}{2-\alpha}},\varepsilon_{0}(\lambda))$.
Since $\mathcal{M}_{\lambda}V$ is semi-concave with constant
$\frac{1}{\lambda}$ in $\Omega^{\lambda}_{V}$ we obtain by 3.4 (3), (3.5) ,
and 3.3 applied to the mollification of $\mathcal{M}_{\lambda}$

$\left\lVert\nabla
V_{\varepsilon(\lambda),\lambda}\right\rVert_{L^{\infty}(\omega_{\delta})}\leqslant
C\lambda^{\frac{\alpha-1}{2-\alpha}}\,\text{ and
}\,y^{\top}\nabla^{2}V_{\varepsilon(\lambda),\lambda}(x)y\leqslant\frac{1}{\lambda}|y|^{2},\text{
for all }y\in\mathbb{R}^{d},\,x\in\omega_{\delta},$ (8.7)

where $C>0$ is a constant depending only on $V$ and $\Omega$. Applying these
bounds to

$-tr(B(y)Du_{\lambda}(y))=\frac{1}{\beta}\sum_{i=1}^{d}\sum_{j=1}^{m}B_{i,j}(y)\left(\sum_{k=1}^{d}\frac{\partial
B_{k,j}}{\partial x_{i}}(y)\frac{\partial
V_{\varepsilon(\lambda),\lambda}}{\partial
x_{k}}(y)+B_{k,j}(y)\frac{\partial^{2}V_{\varepsilon(\lambda),\lambda}}{\partial
x_{k}\partial x_{i}}(y)\right),$

we obtain that for $\lambda\in(0,\lambda_{0})$

$-tr(B(y)Du_{\lambda}(y))\leqslant\frac{md(d+1)}{\lambda\beta}C\left\lVert
B\right\rVert_{Lip(\omega_{\delta};\mathbb{R}^{d\times m})}^{2}\mbox{ for all
}y\in\omega_{\delta}$ (8.8)

for a constant $C>0$ which only depends on $\Omega$ and $V$. Here
$u_{\lambda}=u_{V_{\varepsilon(\lambda),\lambda}}$ and we used that
$\lambda^{\frac{\alpha-1}{2-\alpha}}<\frac{1}{\lambda}$ for $\lambda\in(0,1)$.

We now prove the first part of (5.16). For this, we need to verify that

$\lim_{\lambda\to
0^{+}}\left(\sigma^{1}_{\varepsilon(\lambda),\lambda}+\frac{\sigma^{2}_{\varepsilon(\lambda)}}{\beta}\left\lVert\nabla
V_{\varepsilon(\lambda),\lambda}\right\rVert_{L^{\infty}(\omega_{\delta})}\right)=0.$
(8.9)

Since $B$ is Lipschitz continuous and $\nabla w$ are Hölder continuous, and
$\varepsilon(\lambda)\leqslant\lambda^{\frac{1}{2-\alpha}}$ we have the
following estimate for $\sigma^{2}_{\varepsilon(\lambda)}$

$\sigma^{2}_{\varepsilon(\lambda)}\leqslant\left\lVert
B\right\rVert_{Lip(\omega_{\delta};\mathbb{R}^{d\times m})}\left\lVert\nabla
w\right\rVert_{L^{\infty}(\omega_{\delta};\mathbb{R}^{d})}\lambda^{\frac{1}{2-\alpha}}+C\left\lVert
B\right\rVert_{L^{\infty}(\omega_{\delta};\mathbb{R}^{d\times
m})}\lambda^{\frac{\sigma}{2-\alpha}},$

which implies that

$\sigma^{2}_{\varepsilon(\lambda)}\left\lVert\nabla
V_{\varepsilon(\lambda),\lambda}\right\rVert_{L^{\infty}(\omega_{\delta})}\leqslant
C\left(\lambda^{\frac{\sigma+\alpha-1}{2-\alpha}}+\lambda^{\frac{\alpha}{2-\alpha}}\right),$
(8.10)

for a constant $C>0$ which only depends on $B$, $f$, $\omega_{\delta}$, $w$
and $V$. Let us recall the definition of $g_{\lambda}$ from (4.8). For
$\lambda\in(0,\lambda_{0})$ we know that
$\omega_{\delta}\subset\Omega_{V}^{\lambda}$ which combined with (7.7) permits
us to find a constant $C>0$ depending $f$, $B$, $w$ and $\Omega$ such that

$g^{1}_{\lambda}(y)\leqslant
C\left(\lambda^{\frac{\sigma}{2-\alpha}}+\lambda^{\frac{\sigma+\alpha-1}{2-\alpha}}+\lambda^{\frac{1}{2-\alpha}}+\lambda^{\frac{\alpha}{2-\alpha}}\right).$
(8.11)

Using the above inequality, the Hölder continuity of $w$, the Lipschitz
continuity of $B$ in $\Omega$ and the fact that
$\varepsilon(\lambda)\leqslant\lambda^{\frac{1}{2-\alpha}}$ we obtain

$\sigma^{1}_{\varepsilon(\lambda)\lambda}\leqslant
C\left(\lambda^{\frac{\sigma}{2-\alpha}}+\lambda^{\frac{\sigma+\alpha-1}{2-\alpha}}+\lambda^{\frac{1}{2-\alpha}}+\lambda^{\frac{\alpha}{2-\alpha}}\right),$
(8.12)

where $C>0$ is a constant which only depends on $f$, $B$, $w$ and $\Omega$.
Combining (8.10) and (8.12) we obtain that (8.9) holds. This together with
(5.13) implies that $\lim_{\lambda\to 0^{+}}\tau_{\lambda}=\infty$, see
(5.15).

Moreover, by (5.15) and Lemma 7.1(3), we have
$y([0,\tau_{\lambda}];\omega,u_{\lambda})\subset\omega_{\delta}$.

For proving the second claim in (5.16) we notice that $\mathcal{M}_{\lambda}V$
satisfies (3.14) with $v=\mathcal{M}_{\lambda}V$ in $\Omega_{V}^{\lambda}$.
Hence, the hypotheses of 6.2 are satisfied with $v=\mathcal{M}_{\lambda}V$ and
$\overline{v}=V_{\varepsilon(\lambda),\lambda}$, which implies that

$\begin{array}[]{l}\displaystyle\left\lVert\left(\mathcal{V}_{u_{\lambda},\tau_{\lambda}}(y_{0})+\mathcal{M}_{\lambda}V\circ
y(\tau_{\lambda};\cdot,u_{{\lambda}})-\mathcal{M}_{\lambda}V\right)^{+}\right\rVert_{L^{p}(\omega)}\\\
\vskip 6.0pt plus 2.0pt minus
2.0pt\cr\displaystyle\leqslant\tau_{\lambda}|\omega|^{\frac{1}{p}}\left\lVert
h_{\lambda}\right\rVert_{L^{\infty}(\omega_{\delta})}+\tau_{\lambda}^{\frac{p-1}{p}}\beta\left(\frac{e^{K(\lambda)\tau_{\lambda}}-1}{K(\lambda)}\right)^{\frac{1}{p}}\left\lVert
u_{\lambda}-u_{\mathcal{M}_{\lambda}V}\right\rVert^{2}_{L^{2p}(\omega_{\delta};\mathbb{R}^{m})},\end{array}$
(8.13)

where $h_{\lambda}$ is defined in (3.15). As was pointed out in 3.5, for
$\alpha\in\left(\frac{1}{2},1\right]$ and $\lambda$ small enough we have that
$h_{\lambda}\leqslant C\lambda^{\frac{2\alpha-1}{2-\alpha}}$ for a constant
$C>0$ depending only on $\Omega$, $\ell$, $f$, $B$ and $V$. Using this and
(5.13) in (8.13) we get

$\lim_{\lambda\to
0^{+}}\left\lVert\left(\mathcal{V}_{u_{\lambda},\tau_{\lambda}}(y_{0})+\mathcal{M}_{\lambda}V\circ
y(\tau_{\lambda};\cdot,u_{{\lambda}})-\mathcal{M}_{\lambda}V\right)^{+}\right\rVert_{L^{p}(\omega)}=0.$

This estimate and the fact that $\mathcal{M}_{\lambda}V$ converges uniformly
to $V$ on compact subsets by 3.4 (2) imply that (5.16) holds. ∎

###### Proof of 5.5.

Let us prove (1) and choose $T>0$. Due to the fact that $\lim_{\varepsilon\to
0^{+}}T_{\varepsilon}=\infty$ there exists $\varepsilon_{0}(T)$ such that
$T<T_{\varepsilon}$ for all $\varepsilon\in(0,\varepsilon_{0}(T))$. Then for
$\varepsilon\in(0,\varepsilon_{0}(T))$ we have that
$y([0,T];y_{0},u_{\varepsilon})\subset\Omega$ which by the boundedness of
$\Omega$ implies that $\\{\left\lVert
y_{\varepsilon}\right\rVert_{L^{\infty}((0,T);\mathbb{R}^{d})}\\}_{\varepsilon\in(0,\varepsilon_{0})}$
is bounded. Further, ${\\{u_{\varepsilon}\circ
y_{\varepsilon}\\}}_{\varepsilon\in(0,\varepsilon_{0})}$ is bounded in
$L^{2}((0,T);\mathbb{R}^{m})$ since $\left\lVert u_{\varepsilon}\circ
y_{\varepsilon}\right\rVert^{2}_{L^{2}((0,T);\mathbb{R}^{m})}\leqslant\frac{1}{\beta}\mathcal{V}_{u_{\varepsilon},T_{\varepsilon}}(y_{0})$
and
$\\{\mathcal{V}_{T_{\varepsilon},u_{\varepsilon}}(y_{0})\\}_{\varepsilon\in(0,\varepsilon_{0})}$
is bounded. since This, together with the fact that
$\\{y_{\varepsilon}\\}_{\varepsilon\in(0,\varepsilon_{0})}$ is bounded in
$L^{\infty}((0,T);\mathbb{R}^{d})$, and that $f$ is a locally Lipschitz
function, implies that
$\\{y_{\varepsilon}\\}_{\varepsilon\in(0,\varepsilon_{0})}$ is bounded in
$W^{1,2}((0,T);\mathbb{R}^{d})$.

Therefore, there exist $y^{*}\in W^{1,2}((0,T);\mathbb{R}^{d})$ and $u^{*}\in
L^{2}((0,T);\mathbb{R}^{m})$, and a sub-sequence of $(u_{\varepsilon}\circ
y_{\varepsilon},y_{\varepsilon})$ which converges to $(u^{*},y^{*})$ weakly in
$L^{2}((0,T);\mathbb{R}^{m})\times W^{1,2}((0,T);\mathbb{R}^{d})$. Given that
the previous holds for each $T$, by a diagonal argument we have that $u^{*}\in
L_{loc}^{2}((0,\infty);\mathbb{R}^{m})$ and $y^{*}\in
W_{loc}^{1,2}((0,\infty);\mathbb{R}^{d})$, and a sub-sequence of
$(u_{\varepsilon}\circ y_{\varepsilon},y_{\varepsilon})$ converges to
$(u^{*},y^{*})$ weakly in $L_{loc}^{2}((0,\infty);\mathbb{R}^{m})\times
W_{loc}^{1,2}((0,T);\mathbb{R}^{d}).$ This proves the existence of at least
one weakly convergent sub-sequence.

We now prove (2), i.e., the optimality for every accumulation point of
$(u_{\varepsilon}\circ y_{\varepsilon},y_{\varepsilon})$. Consider an
arbitrary pair $(u,y)\in L_{loc}^{2}((0,\infty);\mathbb{R}^{m})\times
W_{loc}^{1,2}((0,T);\mathbb{R}^{d})$ and a sub-sequence of
$(u_{\varepsilon}\circ y_{\varepsilon},y_{\varepsilon})$, still denoted by
$(u_{\varepsilon}\circ y_{\varepsilon},y_{\varepsilon})$, such that
$(u_{\varepsilon}\circ y_{\varepsilon},y_{\varepsilon})$ converges to $(u,y)$
weakly in $L_{loc}^{2}((0,\infty);\mathbb{R}^{m})\times
W_{loc}^{1,2}((0,\infty);\mathbb{R}^{d}).$ Then, by the compact injection of
$W^{1,2}_{loc}((0,\infty);\mathbb{R}^{d})$ in
$C_{loc}([0,\infty);\mathbb{R}^{d})$ passing to a sub-sequence we have that
$y_{\varepsilon}$ converges to $y$ in $C_{loc}([0,\infty);\mathbb{R}^{d})$. In
particular, this implies that $y$ is a solution of (2.2). Moreover, due to the
continuity of $\ell$, the $L^{2}((0,T);\mathbb{R}^{m})$ norm lower-semi
continuity for all $T>0$, and the non-negativity of $V$ we have the inequality
in the following statements, the equality follows from assumption (5.17):

$\int_{0}^{T}\left(\ell(y)+\frac{\beta}{2}|u|^{2}\right)dt\leqslant\lim_{\varepsilon\to
0^{+}}\mathcal{V}_{u_{\varepsilon},T_{\varepsilon}}(y_{0})+V(y_{\varepsilon}(T_{\varepsilon}))=V(y_{0}).$

Since this holds for every $T>0$, we obtain that $u\in
L^{2}((0,\infty);\mathbb{R}^{m})$ and $J(u)\leqslant V(y_{0})$. Hence, $u^{*}$
is an optimal solution of (2.1). Since this is true for any accumulation point
of $(u_{\varepsilon},y_{\varepsilon})$, then (5.18) holds.

To prove (3) we note that if the the solution of (2.1) is unique, then by (2)
every sub-sequence of $u_{\varepsilon}\circ y_{\varepsilon}$ has a convergent
sub-sequence which converges to the optimal solution, since this solution is
unique, the whole sequence $u_{\varepsilon}\circ y_{\varepsilon}$ converges to
the optimal solution.∎

###### Proof of 5.1.

By (5.19) there exists a sub-sequence
${\mathcal{V}}_{u_{\varepsilon},T_{\varepsilon}}+V\circ
y({T_{\varepsilon}};\cdot,u_{\varepsilon})$ which converges almost everywhere
in $\omega$, and thus (5.17) holds almost everywhere in $\omega$, and (a)-(c)
of Theorem 5.5 hold. Furthermore, if $p=\infty$, using the continuity of $V$
and $\mathcal{V}_{u_{\varepsilon},T_{\varepsilon}}$ the whole sequence
converges everywhere in $\omega$ and therefore (1), (2) and (3) hold for every
$y_{0}\in\omega$. ∎

##  9 Example

In this section we provide an example of a control problem of the form (2.1)
with a locally Lipschitz continuous value function which satisfies 4.2 for a
smooth function $w$ with $g=0$. Additionally, we identify a set of initial
conditions where the optimal control of (2.1) is not unique, which will imply
that the value function is not differentiable in that set.

Let us start by defining the running cost for the state variable by

$\ell_{\alpha}(y)=\frac{1}{2}|y|^{2}\left(1+\alpha\psi\left(\frac{|y-z|}{\sigma}\right)\right),$

where

$\psi(s)=\left\\{\begin{array}[]{ll}\exp\left(-\frac{1}{1-s^{2}}\right)&\mbox{
if }|s|<1\\\ 0&\mbox{ if }|s|\geqslant 1\end{array}\right.,$

and $z\in\mathbb{R}^{2}$ satisfies $z_{1}<-\sigma$, $z_{2}=0$, for some
$\sigma>0$. For $\alpha\in[0,\infty)$, we consider the following control
problem

$\min_{\begin{array}[]{c}u\in L^{2}((0,\infty);\mathbb{R}^{2}),\\\
y^{\prime}=u,\
y(0)=y_{0}\end{array}}\int_{0}^{\infty}\ell_{\alpha}(y(t))dt+\frac{\beta}{2}\int_{0}^{\infty}|u(t)|^{2}dt.$
(9.1)

The value function of this problem is denoted by $V_{\alpha}$. There are two
cases of interest. They are $\alpha=0$ and $\alpha$ tending to infinity. In
the first case the value function is a quadratic function given by
$V_{0}(y)=\frac{\sqrt{\beta}}{2}|y|^{2}$ and it therefore is of class
$C^{\infty}(\mathbb{R}^{2})$. In the second case we are going to prove that as
$\alpha$ tends to infinity the value function tends to the value function
$V_{\infty}$ of the following state constrained problem

$\min_{\begin{array}[]{c}u\in L^{2}((0,\infty);\mathbb{R}^{2}),\\\
y^{\prime}=u,\ y(0)=y_{0},\\\ y(t)\in\mathbb{R}^{2}\setminus B(z,\sigma)\mbox{
for all }t\geqslant
0.\end{array}}\int_{0}^{\infty}\frac{1}{2}|y(t)|^{2}dt+\frac{\beta}{2}\int_{0}^{\infty}|u(t)|^{2}dt.$
(9.2)

on compact subsets of $\mathbb{R}^{2}\setminus B(z,\sigma)$. Here we set
$B(z,\sigma)=\\{u\in\mathbb{R}^{2}:|u-z|<\sigma\\}$. Accordingly we are going
to prove that $V_{\alpha}$ is not smooth for $\alpha$ large enough. This will
be due to the lack of uniqueness of solutions for $y_{0}$ in
$\\{(y_{1},y_{2})\in\mathbb{R}^{2}:\ y_{2}=0,\ y_{1}<z_{1}-\sigma\\}$ in
problem (9.2).

######  Proposition 9.1.

Let $\alpha\in(0,\infty)$. For each $y_{0}\in\mathbb{R}^{2}$ the control
problem (9.1) has an optimal solution and the associated value function is
bounded as follows

$\frac{\sqrt{\beta}}{2}|y_{0}|^{2}\leqslant
V_{\alpha}(y_{0})\leqslant\frac{\sqrt{\beta}}{2}\left(1+\frac{\alpha}{2}\right)|y_{0}|^{2}.$
(9.3)

Further, for every $y_{0}\in\mathbb{R}^{2}$ and every optimal control
$u^{*}\in L^{2}((0,\infty);\mathbb{R}^{2})$ of (9.1) with corresponding
trajectory $y^{*}$ it holds that $u^{*},y^{*}\in
C^{\infty}([0,\infty);\mathbb{R}^{2})$, and

$\frac{du^{*}}{dt}(t)=\frac{d^{2}y^{*}}{dt^{2}}(t)=\frac{1}{\beta}\nabla\ell_{\alpha}(y^{*}(t))\mbox{
for }t>0,$ (9.4) $\ell_{\alpha}(y^{*}(t))=\frac{\beta}{2}|u^{*}(t)|^{2}\mbox{
for all }t>0$ (9.5)

and in particular

$|u^{*}(0)|\leqslant\left(\frac{1+\alpha}{\beta}\right)^{\frac{1}{2}}|y_{0}|.$
(9.6)

###### Proof.

Since the control system appearing in (9.1) is controllable the existence of
an optimal control-state pair $(u^{*},y^{*}$ can easily be established. By the
Lagrange multiplier’s theorem there exists $p^{*}\in
L^{2}((0,\infty);\mathbb{R}^{2})$ satisfying

$p^{*}(t)^{\prime}=\nabla\ell_{\alpha}(y^{*}(t))\mbox{ and
}u^{*}=\frac{1}{\beta}p^{*}(t).$ (9.7)

Utilizing this first order optimality system it is straightforward to see that
the optimal control of (9.1) for $\alpha=0$ and $y_{0}\in\mathbb{R}^{2}$ is
given by $u^{0}=-\frac{1}{\sqrt{\beta}}y_{0}e^{-\frac{t}{\sqrt{\beta}}}$. The
trajectory associated to this control is given by
$y^{0}=y_{0}e^{-\frac{t}{\sqrt{\beta}}}$. Turning to the case $\alpha>0$ we
observe that $\psi\leqslant(1+\alpha)$, and thus $u^{0}$ is feasible for (9.1)
with $\alpha>0$. We obtain that $V_{\alpha}$ is bounded from above in the
following manner

$V_{\alpha}(y_{0})\leqslant\frac{\sqrt{\beta}}{2}\left(1+\frac{\alpha}{2}\right)|y_{0}|^{2}\mbox{
for all }y_{0}\in\mathbb{R}^{d}.$ (9.8)

Furthermore, since every feasible control for (9.1) with $\alpha>0$ is
feasible for $\alpha=0$ and since $\psi$ is bounded from below by zero, we
arrive at

$\frac{\sqrt{\beta}}{2}|y_{0}|^{2}=V_{0}(y_{0})\leqslant
V_{\alpha}(y_{0})\mbox{ for all }y_{0}\in\mathbb{R}^{d}.$ (9.9)

This proves (9.3). In order to continue the analysis of this problem, we need
the first order optimality condition of (9.1).

From (9.7) we now deduce that

$\frac{d^{2}y^{*}}{dt}(t)=\frac{du^{*}}{dt}(t)=\frac{1}{\beta}\nabla\ell_{\alpha}(y^{*}(t)),$
(9.10)

which in turns proves (9.4). We further obtain that $y\in
H^{2}((0,\infty);\mathbb{R}^{2}),\,u^{*}\in
H^{1}((0,\infty);\mathbb{R}^{2}),\,p^{*}\in H^{1}((0,\infty);\mathbb{R}^{2})$,
from which
$\lim_{t\to\infty}y^{*}(t)=\lim_{t\to\infty}u^{*}(t)=\lim_{t\to\infty}p^{*}(t)=0$
follows. By iteration and the regularity of $\psi$ we obtain that $u^{*}$ and
$y^{*}$ are elements of $C^{\infty}([0,\infty);\mathbb{R}^{2})$. Multiplying
the right hand side of (9.10) by $u^{*}(t)$, using that
$u^{*}(t)=\frac{d}{dt}y^{*}(t)$, and integrating from $t$ to infinity we
obtain (9.5). Using (9.5), the continuity of $u^{*}$ and $y^{*}$ at 0, and the
definition of $\ell_{\alpha}$ we get (9.6). ∎

######  Lemma 9.1.

If for $y_{0}\in\mathbb{R}^{2}\setminus\overline{B(z,\sigma)}$ and
$u_{0}\in\mathbb{R}^{2}$ the solution $y\in
C^{\infty}([0,\infty),\mathbb{R}^{2})$ of

$y^{\prime\prime}(t)=\frac{1}{\beta}\nabla\ell_{\alpha}(y)\mbox{ for all
}t>0\mbox{ and }y(0)=y_{0},\ y^{\prime}(0)=u_{0}$

satisfies $\lim_{t\to\infty}y(t)=0$, then $u_{0}\cdot y_{0}<0$.

###### Proof.

For $y_{0},u_{0}$ as in the statement of the lemma, we shall prove that
$u_{0}\cdot y_{0}<0$. Using the continuity of $y$ and that
$y_{0}\notin\overline{B(z,\sigma)}$ there exists $T>0$ such that
$y(t)\notin\overline{B(z,\sigma)}$ for $t\in[0,T]$. Let $\hat{T}$ be the
largest $T$ such that $y(t)\notin\overline{B(z,\sigma)}$ for
$t\in[0,\hat{T}]$. Then $y$ satisfies

$y^{\prime\prime}(t)=\frac{1}{\beta}y(t)\text{ for }t\in(0,\hat{T}).$

Therefore we obtain the following expression for $y(t)$ with $t\in(0,\hat{T})$

$y(t)=y_{0}\cosh\left(\frac{t}{\sqrt{\beta}}\right)+u_{0}\sqrt{\beta}\sinh\left(\frac{t}{\sqrt{\beta}}\right).$
(9.11)

Differentiating this expression we have

$y^{\prime}(t)=\frac{1}{\sqrt{\beta}}y_{0}\sinh\left(\frac{t}{\sqrt{\beta}}\right)+u_{0}\cosh\left(\frac{t}{\sqrt{\beta}}\right).$
(9.12)

Multiplying $y(t)$ and $y^{\prime}(t)$ for $t\in(0,\hat{T})$ we arrive at

$\frac{d}{dt}\frac{1}{2}|y(t)|^{2}=y(t)\cdot
y^{\prime}(t)=\sinh\left(\frac{t}{\beta}\right)\cosh\left(\frac{t}{\beta}\right)\left(\frac{1}{\sqrt{\beta}}|y_{0}|^{2}+\sqrt{\beta}|u_{0}|^{2}\right)+y_{0}\cdot
u_{0}\cosh\left(2\frac{t}{\beta}\right).$

If $y_{0}\cdot u_{0}\geqslant 0$ then the previous inequality implies that
$y^{\prime}(t)\cdot y(t)\geqslant 0$ which in turns allow us to deduce that
$|y(t)|^{2}\geqslant|y_{0}|^{2}$ for all $t\in[0,\hat{T}]$. If
$\hat{T}<\infty$ then by the continuity of $y$ and since
$y_{0}\notin\overline{B(z,\sigma)}$ we know that there exists $\delta>0$ such
that $y(t)\notin\overline{B(z,\sigma)}$ for all $t\in(\hat{T},\hat{T}+\delta)$
which contradicts the definition of $\hat{T}$. Therefore $\hat{T}=\infty$
which contradicts the fact that $\lim_{t\to\infty}y(t)=0$. This permits us to
conclude that $u_{0}\cdot y_{0}<0$. ∎

In the following we present the necessary optimality conditions arising from
the dynamic programming principle which are important for proving the non-
differentiability of the value function. We first prove that in the ball
$B(0,|z|-\sigma)$ the value function coincides with $V_{0}$ for every
$\alpha>0$. This together with the stability of the optimal trajectories will
allow us to get a necessary optimality condition which involves
$D^{+}V_{\alpha}(y_{0})$. This optimality condition also implies that
$D^{+}V_{\alpha}(y_{0})$ is bounded for every $y_{0}$. Combining this with
some technical results we shall prove that $V_{\alpha}$ is Lipschitz
continuous.

######  Lemma 9.2.

Let $y_{0}\in B(0,|z|-\sigma)$ and $\alpha>0$. Then
$V_{\alpha}(y_{0})=V_{0}(y_{0})=\frac{\sqrt{\beta}}{2}|y_{0}|^{2}$.

###### Proof.

By (9.4) and Lemma 9.1 we have that
$\frac{d}{dt}\frac{1}{2}|y^{*}(t)|^{2}=y^{*}(t)\cdot u^{*}(t)<0$ for all
$t\geqslant 0$. In particular we have that $y^{*}(t)\in B(0,|z|-\sigma)$ for
all $t>0$. This implies that

$\int_{0}^{\infty}\ell_{\alpha}(y^{*}(t))dt=\int_{0}^{\infty}\frac{1}{2}|y^{*}(t)|^{2}dt$

which permits us to conclude that $u^{*}$ is an optimal solution of (9.1) with
$\alpha=0$ and therefore $V_{\alpha}(y_{0})=V(y_{0})$. ∎

######  Proposition 9.2.

For every $\alpha>0$ the value function of problem (9.1) is Lipschitz
continuous on compact subsets of $\mathbb{R}^{2}$.

###### Proof.

Let $y_{0}\in\mathbb{R}^{2}$, $y_{1}\in\mathbb{R}^{2}$, and $T>0$ arbitrary.
Consider $u(t)=\frac{1}{T}(y_{0}-y_{1})$ and $y(t)=\frac{t}{T}\cdot
y_{0}+(1-\frac{t}{T})\cdot y_{1}$ for $t\in[0,T]$. We notice that
$y^{\prime}(t)=u$, $y(0)=y_{1}$ and $y(T)=y_{0}$. Then by the dynamic
programming principle we have

$V_{\alpha}(y_{1})-V_{\alpha}(y_{0})\leqslant\int_{0}^{T}\left(\ell_{\alpha}(y(t))+\frac{\beta}{2}|u(t)|^{2}\right)dt\leqslant\int_{0}^{T}\left(\frac{1+\alpha}{2}\left|\frac{t}{T}(y_{0}-y_{1})+y_{1}\right|^{2}+\frac{\beta}{2T^{2}}|y_{0}-y_{1}|^{2}\right)dt.$

By direct calculations on the right-hand side of the previous expression we
get

$V_{\alpha}(y_{1})-V_{\alpha}(y_{0})\leqslant\frac{1+\alpha}{2}\left(2T|y_{1}|^{2}+T|y_{0}-y_{1}|^{2}\right)+\frac{\beta}{2T}|y_{0}-y_{1}|^{2}.$

Since this is for an arbitrary $T$ we can choose $T=|y_{0}-y_{1}|$ in the
previous expression to conclude the Lipschitz continuity of $V$ on compact
subsets of $\mathbb{R}^{2}$. ∎

######  Lemma 9.3.

Let $\alpha\in(0,\infty)$ and $y_{0}\in\mathbb{R}^{2}$. Consider $u^{*}$ a
solution of (9.1) and $y^{*}$ its associated trajectory. Then $y^{*}$ is
exponentially asymptotically stable:

$|y^{*}(t)|\leqslant(1+\alpha)^{1/2}\exp\left(\frac{-1}{(1+\alpha)\sqrt{\beta}}t\right)|y_{0}|\mbox{
for all }t>0.$ (9.13)

###### Proof.

Due to the local Lipschitz continuity of $V_{\alpha}$ and the dynamic
programming principle we have that $V_{\alpha}\circ y^{*}$ is differentiable
for almost all $t>0$ and satisfies

$\frac{d}{dt}V_{\alpha}\circ
y^{*}(t)+\ell_{\alpha}(y(t))+\frac{\beta}{2}|u^{*}(t)|^{2}=0\mbox{ for almost
all }t>0.$ (9.14)

Using (9.5) in (9.14) we obtain

$\frac{d}{dt}V_{\alpha}\circ y^{*}(t)+2\ell_{\alpha}(y(t))=0.$

Since $\ell_{\alpha}(y)\geqslant\frac{1}{2}|y|^{2}$ for all
$y\in\mathbb{R}^{2}$ and using (9.3) in the previous equality, we get (9.13)
which proves the asymptotic exponential stability of $y^{*}$. ∎

Below $\overline{co}\\{\omega\\}$ denotes the convex closure of a set $\omega$
in $\mathbb{R}^{2}$. Further for $v\in C(\omega)$ with
$\omega\subset\mathbb{R}^{2}$ open, the set valued function $D^{+}v$ stands
for the super-differential of $v$ (see [50, Chaper 2, Section 1]).

######  Proposition 9.3.

Let $y_{0}\in\mathbb{R}^{2}$, $\alpha>0$ and set
$\mathcal{I}(y_{0})=\overline{co}\\{-\beta u^{*}(0):\ u^{*}\mbox{ is an
optimal solution of }\eqref{Example:controlproblem}\\}$. Then we have
$D^{+}V_{\alpha}(y_{0})=\mathcal{I}(y_{0})$.

######  Remark 9.1.

For our purpose, the relevance of $D^{+}V_{\alpha}(y)$ relies on the fact that
if $V_{\alpha}$ is differentiable at $y$, then
$D^{+}V_{\alpha}(y_{0})=\\{\nabla V_{\alpha}(y_{0})\\}$ (see Lemma 1.8 in [50,
Chaper 2, Section 1]).

###### Proof.

Step 1. Let $R>0$ be fixed. By 9.1 we know that there exists a constant
$C_{R}>0$ such that for all $y_{0}\in\overline{B}(0,R)$ and all solutions
$u^{*}\in L^{2}((0,\infty);\mathbb{R}^{2})$ of (9.1) we have

$\left\lVert u^{*}\right\rVert_{H^{1}((0,\infty);\mathbb{R}^{2})}\leqslant
C_{R}.$

Moreover, since $\overline{B}(0,R)$ is compact, Lemma 9.3 implies the
existence of $T_{R}>0$ such that $y^{*}(t)\in B(0,|z|-\sigma)$ for all
$t\geqslant T_{R}$, where $y^{*}$ is an optimal trajectory of (9.1) with
$y_{0}\in\overline{B}(0,R)$. Combining these statements, the dynamic
programming principle, and Lemma 9.2, we have that for all
$y_{0}\in\overline{B}(0,R)$ the value function $V_{\alpha}$ is equal to

$\min_{\begin{array}[]{c}u\in H^{1}((0,T_{R});\mathbb{R}^{2}),\ \left\lVert
u\right\rVert_{H^{1}((0,T_{R});\mathbb{R}^{2})}\leqslant C_{R},\\\
y^{\prime}=u\mbox{ in }(0,T),\
y(0)=y_{0},\end{array}}\int_{0}^{T_{R}}\ell_{\alpha}(y(t))dt+\frac{\beta}{2}\int_{0}^{T_{R}}|u(t)|^{2}dt+V_{0}(y(T_{R})).$
(9.15)

Moreover there is a one-to-one relationship between solutions to (9.1) and
(9.15) by proper restriction of optimal controls of (9.1) to $[0,T_{R}]$
respectively by extension. Utilizing the fact that the optimal controls are
different from zero inside $B(0,|z|-\sigma)$ it can be argued that for the
optimal controls of (9.15) the constraint is not active. By the first order
optimality conditions for (9.15) we obtain that
$\sqrt{\beta}u^{*}(T_{R})=-y^{*}(T_{R})$ for each optimal control-state pair.

Step 2. To call upon a general result on the sensitivity of marginal functions
below, we now endow the set of feasible controls $\overline{B}_{C_{R}}=\\{u\in
H^{1}((0,T_{R});\mathbb{R}^{2}):\left\lVert
u\right\rVert_{H^{1}((0,T_{R});\mathbb{R}^{2})}\leqslant C_{R},\\}$ of (9.15)
by the weak topology induced by $H^{1}((0,T_{R});\mathbb{R}^{2})$. For $u\in
H^{1}((0,T_{R});\mathbb{R}^{2})$ and $y_{0}\in\overline{B}(0,R)$ we define the
reduced cost-functional associated to (9.15) by
$J_{R}(y_{0},u)=\int_{0}^{T_{R}}\ell_{\alpha}(y(t))dt+\frac{\beta}{2}\int_{0}^{T_{R}}|u(t)|^{2}dt+V_{0}(y(T_{R}))$
where $y$ is the unique solution of $y^{\prime}=u$ and $y(0)=y_{0}$. In the
following we argue the continuity of
$J_{R}:\mathbb{R}^{2}\times\overline{B}_{C_{R}}\to\mathbb{R}$, as well as the
differentiability with respect to $y_{0}$ and the continuity of the gradient
considered on $\mathbb{R}^{2}\times\overline{B}_{C_{R}}$. For the continuity
of $J_{R}$ let us consider $y_{0}^{n}$ converging to $y_{0}$ and $u_{n}$
converging to $u$ in the weak topology of $H^{1}((0,T_{R});\mathbb{R}^{2})$
with $\left\lVert u_{n}\right\rVert_{H^{1}((0,T_{R});\mathbb{R}^{2})}\leqslant
C_{R}$. By the compact embedding of $H^{1}((0,T_{R});\mathbb{R}^{2})$ in
$C([0,T_{R}];\mathbb{R}^{2})$ we have that passing to a sub-sequence $u_{n}$
converges to $u$ in $C([0,T_{R}];\mathbb{R}^{2})$. This and the convergence of
$y_{0}^{n}$ to $y_{0}$ implies that for the same sub-sequence the states
$y(u_{n})$ converge to $y$ in $C([0,T_{R}];\mathbb{R}^{2})$, and that
$y^{\prime}=u$ and $y(0)=y_{0}$. Due to the fact that $\ell_{\alpha}$ is
continuous, we get that $J_{R}(y_{0}^{n},u_{n})$ converges to $J_{R}(y_{0},u)$
through this sub-sequence. Since this holds for each convergent sub-sequence
we get that the whole sequence converges and hence $J_{R}$ is continuous. The
existence of the gradient of $J_{R}$ with respect to $y_{0}$ is a direct
consequence of the classic ODE theory and it is given by

$\nabla_{y_{0}}J_{R}(y_{0},u)=\int_{0}^{T_{R}}\nabla\ell_{\alpha}(y(t))dt+\sqrt{\beta}y(T_{R}).$
(9.16)

The continuity of $\nabla_{y_{0}}J_{R}(y_{0},u)$ with respect to $(y_{0},u)$
can be argued similarly as the continuity of $J_{R}$ by using the compact
embedding of $H^{1}((0,T_{R});\mathbb{R}^{2})$ in
$C([0,T_{R}];\mathbb{R}^{2})$.

Step 3. We are now in a position to apply Proposition 4.4, and hence
Proposition 2.1 in [50, Chapter 2], to the value function associated to
(9.15), and hence to $V_{\alpha}$, for each $y_{0}\in\overline{B}(0,R)$. This
asserts that

$D^{+}V_{\alpha}(y_{0})=\overline{co}(\\{\nabla_{y_{0}}J_{R}(y_{0},u^{*}):\
u^{*}\mbox{ is an optimal solution of }\eqref{Example:controlproblem}\\}).$

Combining this fact, together with (9.16), (9.4), and
$\sqrt{\beta}u^{*}(T_{R})=-y^{*}(T_{R})$ from Step 1, we obtain
$D^{+}V_{\alpha}(y_{0})=\mathcal{I}(y_{0})$. Since $R>0$ was chosen
arbitrarily, this equality holds for all $y_{0}\in\mathbb{R}^{2}$. ∎

######  Remark 9.2.

As a consequence of (9.7) the assertion of Proposition 9.3 can equivalently be
expressed as $D^{+}V_{\alpha}(y_{0})=\overline{co}\\{-p^{*}(0):\ p^{*}\mbox{
is the adjoint state associated to an optimal solution $u^{*}$ of
}\eqref{Example:controlproblem}\\}$.

######  Lemma 9.4.

For every $\alpha\in(0,\infty)$ we have $y\cdot\nabla V_{\alpha}(y)>0$ for
almost all $y\in\mathbb{R}^{2}\setminus\overline{B(z,\sigma)}$. Further,
define $w:\mathbb{R}^{2}\to\mathbb{R}$ by

$w(y)=\left\\{\begin{array}[]{ll}0&\mbox{ if }|y|\leqslant R\\\
(|y|^{2}-R^{2})^{2}&\mbox{ if }|y|>R\end{array}\right.$ (9.17)

with $R=|z|+\sigma$. Then $w$ satisfy 4.2 for $\phi=V_{\alpha}$, with g=0, and
$\Omega$ any open bounded subset of $\mathbb{R}^{2}$ containing the ball
$B(0,R)$ and $\omega\Subset\Omega$ an open set such that
$B(0,R)\subset\omega$.

###### Proof.

Let $y_{0}\in\mathbb{R}^{2}\setminus\overline{B(z,\sigma)}$ be such that
$V_{\alpha}$ is differentiable at $y_{0}$. Then
$D^{+}V_{\alpha}(y_{0})=\\{\nabla V_{\alpha}(y_{0})\\}$. This together with
9.3 implies that $u^{*}(0)=-\frac{1}{\beta}\nabla V_{\alpha}(y_{0})$. By Lemma
9.1 and (9.4) we obtain that $u^{*}(0)\cdot y_{0}<0$. Therefore we have
$\nabla V_{\alpha}(y_{0})\cdot y_{0}>0$. This along with the fact that
$V_{\alpha}$ is differentiable almost everywhere allow us to conclude that
$\nabla V_{\alpha}(y)\cdot y>0$ for almost every
$y\in\mathbb{R}^{2}\setminus\overline{B(z,\sigma)}$.

For the rest of the statement it is enough to notice that $\nabla w(y)=0$ in
$\overline{B(0,R)}$ and $\nabla V_{\alpha}(y)\nabla w(y)>0$ for all
$y\in\mathbb{R}^{2}\setminus\overline{B(0,R)}$ since
$B(z,\sigma)\subset\overline{B(0,R)}$. ∎

Next we turn to proving the non-differentiability of $V_{\alpha}$. For this
purpose we shall establish that there exists $\alpha^{*}$ such that for all
$\alpha>\alpha^{*}$ there exists $\hat{y}_{1,\alpha}\in(-\infty,z_{1}-\sigma)$
such that for each initial condition of the form $(y_{1},0)$ with
$\hat{y}_{1}\leqslant y_{1,\alpha}$ there exists at least two optimal
solutions of (9.1). In the following we denote $e_{1}=(1,0)$ and
$e_{2}=(0,1)$.

######  Lemma 9.5.

Let $y_{0,1}\in(-\infty,z_{1}-\sigma)$ and $\alpha\in(0,\infty)$. Assume that
$y^{*}\in H^{1}((0,\infty);\mathbb{R}^{2})$ is an optimal trajectory of (9.1)
with $y_{0}=(y_{0,1},0)$. If $\frac{d}{dt}y^{*}(0)\cdot e_{2}=0$, then
$y^{*}(t)=(y_{1}^{*}(t),0)$ for all $t>0$, with $y_{1}^{*}$ the unique
solution of

$y_{1}^{\prime}=-y_{1}\frac{1}{\sqrt{\beta}}\sqrt{\left(1+\alpha\psi\left(\frac{|y_{1}-z_{1}|}{\sigma}\right)\right)},\,y_{1}(0)=y_{0,1},\mbox{
for all }t>0,$ (9.18)

and $y_{1}^{*}(t)<0$ for all $t\in(0,\infty)$.

###### Proof.

Since $y^{*}$ satisfies (9.4), we know that $y^{*}\in
C^{1}([0,\infty);\mathbb{R}^{2})$. Moreover, since $y^{*}\in
H^{1}((0,\infty);\mathbb{R}^{2})$ we can apply Lemma 9.1 to deduce that
$\frac{dy^{*}}{dt}(0)\cdot e_{1}>0$. Combining this, (9.5) and the continuity
of $\frac{d}{dt}y^{*}$ at $0$ we deduce that $\frac{dy^{*}}{dt}(0)\cdot
e_{1}=-y_{0,1}\frac{1}{\sqrt{\beta}}\sqrt{\left(1+\alpha\psi\left(\frac{|y_{0,1}-z_{1}|}{\sigma}\right)\right)}$.
Consider $\hat{y}(t)=(\hat{y}_{1}(t),0)$ for $t>0$ with $\hat{y}_{1}(t)$ the
unique solution of (9.18). Differentiating $\hat{y}$ twice and using (9.18) we
get

$\hat{y}^{\prime\prime}(t)=\frac{1}{\beta}\nabla\ell_{\alpha}(\hat{y}),\mbox{
for all }t>0,\ \hat{y}(0)=y_{0},\
\hat{y}^{\prime}(0)=\left(-y_{0,1}\frac{1}{\sqrt{\beta}}\sqrt{\left(1+\alpha\psi\left(\frac{|y_{0,1}-z_{1}|}{\sigma}\right)\right)},0\right)=\frac{dy^{*}}{dt}(0).$

Thus, by uniqueness $\hat{y}=y^{*}$ which concludes the proof. ∎

######  Lemma 9.6.

There exists $\bar{\alpha}>0$ such that for each $\alpha>\bar{\alpha}$ there
exists $\hat{y}_{\alpha,1}\in(-\infty,z_{1}-\sigma)$ such that for all
$y_{1}<\hat{y}_{\alpha,1}$, each optimal trajectory $y^{*}$ associated to an
optimal solution $u^{*}$ of (9.1) with $y^{*}(0):=y_{0}=(y_{1},0)$, there
exists $t^{*}\in(0,\infty)$ fulfilling $y^{*}(t^{*})\cdot e_{2}\neq 0$.

###### Proof.

By contradiction, let us assume that the lemma does not hold. Then there exist
sequences $\alpha_{n}>0$ and $y_{n,1}<z_{1}-\sigma$ satisfying
$y_{n+1,1}\leqslant y_{n,1}$ and

$\lim_{n\to\infty}\alpha_{n}=\infty,\ \lim_{n\to\infty}y_{n,1}=-\infty$

and there exist optimal trajectories $y_{n}^{*}$ of (9.1) with
$y^{*}_{n}(0)=(y_{n,1},0)$ and $\alpha=\alpha_{n}$ such that
$y_{n}^{*}(t)\cdot e_{2}=0$ for all $t>0$. Since $y_{n,1}$ is monotone and
diverges to minus infinity, we obtain that every $y_{n}^{*}$ coincides with
$y_{0,1}$ at a time $t_{n}\in(0,\infty)$. Defining
$\tilde{y}_{n}(t)=y_{n}^{*}(t+t_{n}),\tilde{u}_{n}(t)=u_{n}^{*}(t+t_{n})$ we
have by the dynamic programming principle that $(\tilde{y}_{n},\tilde{u}_{n})$
is an optimal trajectory-control pair of (9.1) with $\alpha=\alpha_{n}$ and
$y_{0}=(y_{0,1},0)$. Let $u_{\infty}$ be an optimal solution of (9.2) with
$y_{\infty}$ its associated trajectory. By the optimality of $\tilde{u}_{n}$
and the fact that $y_{\infty}(t)\notin B(z,\sigma)$ we have

$\int_{0}^{\infty}\ell_{\alpha_{n}}(\tilde{y}_{n}(t))dt+\frac{\beta}{2}\int_{0}^{\infty}|\tilde{u}_{n}(t)|^{2}dt\leqslant\int_{0}^{\infty}\frac{1}{2}|y_{\infty}(t)|^{2}dt+\frac{\beta}{2}\int_{0}^{\infty}|u_{\infty}(t)|^{2}dt$

Since $\ell_{\alpha_{n}}(y)\geqslant\frac{1}{2}|y|^{2}$ for all
$y\in\mathbb{R}^{2}$ and $\tilde{y}_{n}^{\prime}=\tilde{u}_{n}$, the former
inequality implies that $\\{(\tilde{y}_{n},\tilde{u}_{n})\\}_{n=1}^{\infty}$
is a bounded family in $H^{1}((0,\infty);\mathbb{R}^{2})\times
L^{2}((0,\infty);\mathbb{R}^{2})$. Passing to a sub-sequence if necessary,
there exists $\tilde{u}\in L^{2}((0,\infty);\mathbb{R}^{2})$ and $\tilde{y}\in
H^{1}((0,\infty);\mathbb{R}^{2})$ such that

$\tilde{u}_{n}\rightharpoonup\tilde{u}\mbox{ in
}L^{2}((0,\infty);\mathbb{R}^{2}),\
\tilde{y}_{n}\rightharpoonup\tilde{y}\mbox{ in
}H^{1}((0,\infty);\mathbb{R}^{2}),$

and for all $T\in(0,\infty)$

$\tilde{y}_{n}\to\tilde{y}\mbox{ in }C([0,T];\mathbb{R}^{2}).$

By the lower semi-continuity of the $L^{2}((0,\infty);\mathbb{R}^{2})$ norm
with respect to the weak topology of $L^{2}((0,\infty);\mathbb{R}^{2})$ we get

$\int_{0}^{\infty}\frac{1}{2}|\tilde{y}(t)|^{2}dt+\frac{\beta}{2}\int_{0}^{\infty}|\tilde{u}(t)|^{2}dt\leqslant\int_{0}^{\infty}\frac{1}{2}|y_{\infty}(t)|^{2}dt+\frac{\beta}{2}\int_{0}^{\infty}|u_{\infty}(t)|^{2}dt.$

Moreover by the definition of $\ell_{\alpha}$ we get for all $T\in(0,\infty)$

$\frac{1}{2}\int_{0}^{T}|\tilde{y}_{n}(t)|^{2}\psi\left(\frac{|\tilde{y}_{n}(t)-z|}{\sigma}\right)dt\leqslant\frac{1}{\alpha_{n}}\left(\int_{0}^{\infty}\frac{1}{2}|y_{\infty}(t)|^{2}dt+\frac{\beta}{2}\int_{0}^{\infty}|u_{\infty}(t)|^{2}dt\right).$

Utilizing the uniform convergence of $\tilde{y}_{n}$ one obtains that for all
$T\in(0,\infty)$ one can take the limit as $n$ goes to infinity to obtain

$\int_{0}^{T}|\tilde{y}(t)|^{2}\psi\left(\frac{|\tilde{y}(t)-z|}{\sigma}\right)dt\leqslant
0,$

from where we deduce that $\psi\left(\frac{\tilde{y}(t)-z}{\sigma}\right)=0$
for almost all $t\in(0,T)$. Since the former holds for all $T>0$ and
$B(z,\sigma)=\\{y\in\mathbb{R}^{2}:\psi(y)\neq 0\\}$ we get that
$\tilde{y}(t)\notin B(z,\sigma)$ for all $t>0$. In particular this proves that
$\tilde{u}$ is an optimal solution of (9.2). Since $\tilde{y}(t)\notin
B(z,\sigma)$ for all $t>0$, there exists $\tilde{t}>0$ such that
$dist(\tilde{y}(\tilde{t}),\mathbb{R}\times\\{0\\})\geqslant\sigma$. On the
other hand, we have that $\tilde{y}_{n}(t)e_{2}=0$ for all $t>0$, which gives
the desired contradiction. ∎

######  Remark 9.3.

With the technique of the proof to the previous lemma it can also be argued
that every sequence of solutions $u^{*}_{\alpha_{n}}$ to (9.1) with initial
condition $y_{0}\in\mathbb{R}^{2}\setminus B(z,\sigma)$ and
$\lim_{n\to\infty}\alpha_{n}=\infty$, contains a convergent subsequence and
every such subsequence converges to a solution of (9.2).

.

######  Theorem 9.1.

Let $\bar{\alpha}>0$ and $\hat{y}_{\alpha,1}$ be as in Lemma 9.6. Then every
optimal solution $u^{*}\in L^{2}((0,\infty);\mathbb{R}^{2})$ of (9.1) with
$y_{0}=(y_{0,1},0)$ satisfying $y_{0,1}<\hat{y}_{\alpha,1}$ is such
$u^{*}(0)\cdot e_{2}\neq 0$ and (9.1) admits at least two solutions. Moreover
for $\alpha>\bar{\alpha}$ the value function $V_{\alpha}$ is not
differentiable in $(y_{1},0)$ for each $y_{1}\in(-\infty,\hat{y}_{\alpha,1})$.

###### Proof.

Let $\alpha>\bar{\alpha}$, $y_{0,1}<y_{\alpha,1}$ and $u^{*}\in
L^{2}((0,\infty);\mathbb{R}^{2})$ be an optimal solution of (9.1) with
associated state $y^{*}\in H^{1}((0,\infty);\mathbb{R}^{2})$. By (9.1) we know
that $u^{*},y^{*}\in C^{\infty}([0,\infty);\mathbb{R}^{2})$. By contradiction,
let assume that $u^{*}(0)\cdot e_{2}=0$. Then by Lemma 9.5 we have
$y^{*}(t)\cdot e_{2}=0$, for all $t>0$. Nevertheless, by Lemma 9.6 there
exists $\bar{t}>0$ such that $y^{*}(\bar{t})\cdot e_{2}\neq 0$ which is a
contradiction. Hence $u^{*}(0)\cdot e_{2}\neq 0$. Moreover, defining
$\bar{u}\in L^{2}((0,\infty);\mathbb{R}^{2})$

$\bar{u}(t)=\left(\begin{array}[]{cc}1&0\\\ 0&-1\end{array}\right)u^{*}(t),$

and $\bar{y}\in H^{1}((0,\infty);\mathbb{R}^{2})$ as the unique solution of
$y^{\prime}=\bar{u}$ with $y(0)=y_{0}$, we notice that
$\ell_{\alpha}(\bar{y}(t))=\ell_{\alpha}(y^{*}(t))$ and
$|\bar{u}(t)|^{2}=|u^{*}(t)|^{2}$. Consequently $\bar{u}$ is an optimal
solution of (9.1) as well, different from $u^{*}$.

If the value function $V_{\alpha}$ were differentiable at $y_{0}$ we would
have that $D^{+}V_{\alpha}(y_{0})$ would be a singleton. However, by 9.3 we
have that $-\beta\bar{u}(0)$ and $-\beta u^{*}(0)$ are contained in
$D^{+}V_{\alpha}(y_{0})$ which is a contradiction to the fact that
$\bar{u}(0)\neq u^{*}(0)$ and therefore the value function cannot be
differentiable at $y_{0}$. ∎

##  10 Concluding remarks

In this work error bounds the convergence of feedback laws for infinite
horizon problems with non-differentiable value functions were presented. These
error bounds together with mollifications and the Moreau envelope permitted us
to construct smooth sequences of feedback laws with the property that the
associated value functions converge to the one of the original problem. This
result is important for both proving existence of solutions and convergence
for the techniques developed in [45, 16, 46, 47, 43], which train a feedback
law by minimizing learning type cost functionals over a set of initial
conditions. Further, 5.1 says that the accumulation points of the controls
obtained by this techniques are optimal controls for almost all initial
condition.

As was seen, 4.2 is important in order to ensure that the trajectories
resulting from the feedback law approximation do not escape from the domain
where the value function is approximated. Indeed, this hypothesis allowed us
to estimate the escape time of the feedback from the approximation region
depending on the regularity of the value function. In this respect, a stronger
assumption on the stabilizability of the dynamics of (2.1) could lead to
strengthen the estimates in Section 7. For instance the, in the case of
exponential stabilizability, the escape time of the trajectories resulting
from the feedback law approximation is expected to be infinity in a similar
way as was done in [45, Section 4, Proposition 1] for the smooth case.

The results of Section 6 could also be applied to bound the error of feedback
laws constructed from data driven approaches for the case of non-
differentiable value function. However, in order to do this the constructed
feedback laws must satisfy (6.4), which of course will depend on the
particularities of the method (e.g. neural network, polynomials, etc) and the
control problem under study. Moreover, a stability assumptions like 4.2 needs
to be satisfied by the closed loop problem obtained from the constructed
feedback law. In this regards, it could be of interest to study under which
conditions the results in Section 6 could be applied to prove the convergence
of data driven approaches.

The example presented in section Section 9 proves that even if all the data of
the control problem are smooth, the value function can be non-differentiable.
On the other hand, for this example the value function was proved to be
Lipschitz continuous and 4.2 was also proved to hold. This underline the
importance of the results presented in this work concerning the syntheses of
smooth feedback laws. It is also worth to mention that the non-uniqueness
result is part due to the symmetry of both the objective function and the
dynamics, which enable us to find a transformation of the optimal control
which is still optimal. Further, the particular characteristics of this
problem made it possible to demonstrate that the controls which are invariable
under this transformation are not optimal solutions for $\alpha$ large enough.
We expect that many interesting generalizations and modifications of this
example are possible.

To conclude this work we mention some interesting extensions. We order them
according to their apparent complexity and connection with this work. The
error bound derived in Section 6 and the escape time estimation in Section 7
could be easily extend to the finite horizon case with some modification on
the hypotheses. Additionally, the case of restrictions on the state could be
treated by first approximating the problem by a penalty method and followed by
the construction of a sequence of approximating feedback laws. Further, a key
condition in this work was the availability of an expression of the feedback
law as a function of the gradient of the value function. This is also the case
for problems with convex restrictions on the control, where this expression is
given by the projection onto the restrictions set. Nevertheless, since the
projection on a closed convex set is just Lipschitz continuous, following the
methods of the present work will lead to a Lipschitz sequence of feedback
laws. Finally, since we proved error estimates in Section 6 with respect to
$L_{p}$ norms with $p\in(1,\infty)$, we believe that it could be possible to
apply similar techniques to analyze the case of control problems with
discontinuous value functions, for instance problems with final cost and
escape time.

## References

  * [1] F. Bonnans, P. Chartier, H. Zidani, Discrete approximation for a class of the Hamilton-Jacobi equation for an optimal control problem of a differential-algebraic system, Control and Cybernetics 32 (1) (2003) 33–55.
  * [2] M. Falcone, R. Ferretti, Semi-Lagrangian approximation schemes for linear and Hamilton-Jacobi equations, SIAM, Philadelphia, 2013.
  * [3] A. Alla, M. Falcone, D. Kalise, An efficient policy iteration algorithm for dynamic programming equations, SIAM J. Control Optim. 37 (1) (2015) A181–A200.
  * [4] R. W. Beard, G. N. Saridis, J. T. Wen, Galerkin approximation of the generalized Hamilton-Jacobi-Bellman equation, Automatica 33 (12) (1997) 2159–2177.
  * [5] M. Puterman, S. Brumelle, On the convergence of policy iteration in stationary dynamic programming, Math. Oper. Res. 4 (1) (1979) 60–69.
  * [6] M. Santos, J. Rust, Convergence properties of policy iteration, SIAM J. Control Optim. 42 (6) (2004) 2094–2115.
  * [7] Y. T. Chow, J. Darbon, S. Osher, W. Yin, Algorithm for overcoming the curse of dimensionality for time-dependent non-convex Hamilton– Jacobi equations arising from optimal control and differential games problems, J. Sci. Comput. 73 (2017) 617–643.
  * [8] Y. T. Chow, J. Darbon, S. Osher, W. Yin, Algorithm for overcoming the curse of dimensionality for state-dependent Hamilton–Jacobi equations, J. Comput. Phys 387 (2019) 376–409.
  * [9] Y. T. Chow, W. Li, S. Osher, W. Yin, Algorithm for Hamilton–Jacobi equations in density space via a generalized Hopf formula, J. Sci. Comput. 80 (2019) 1195–1239.
  * [10] J. Darbon, S. Osher, Algorithms for overcoming the curse of dimensionality for certain Hamilton–Jacobi equations arising in control theory and elsewhere, Res. Math. Sci. 3 (2016) 19.
  * [11] J. Han, A. Jentzen, E. Weinan, Solving high-dimensional partial differential equations using deep learning, Proc. Nat. Acad. Sci. USA 115 (34) (2018) 8505–8510.
  * [12] J. Darbon, G. P. Langlois, T. Meng, Overcoming the curse of dimensionality for some Hamilton–Jacobi partial differential equations via neural network architectures, Res. Math. Sci. 7 (2020) 1–50.
  * [13] N. Nüsken, L. Richter, Solving high-dimensional Hamilton–Jacobi– Bellman pdes using neural networks: Perspectives from the theory of controlled diffusions and measures on path space (2020).
URL https://arxiv.org/abs/2005.05409

  * [14] D. Onken, L. Nurbekyan, X. Li, S. W. Fung, S. Osher, L. Ruthotto, A neural network approach applied to multi-agent optimal control, in: 2021 European Control Conference (ECC), 2021, pp. 1036–1041.
  * [15] K. Ito, C. Reisinger, Y. Zhang, A neural network-based policy iteration algorithm with global h2-superlinear convergence for stochastic games on domains, Found. Comput. Math. 21 (2) (2021) 331–374.
  * [16] K. Kunisch, D. Walter, Semiglobal optimal feedback stabilization of autonomous systems via deep neural network approximation, ESAIM: Control, Optimisation and Calculus of Variations 27 (2021) 16.
  * [17] L. Ruthotto, S. J. Oshera, W. Li, L. Nurbekyan, S. W. Fung, A machine learning framework for solving high-dimensional mean field game and mean field control problems, Proc. Natl. Acad. Sci. 117 (17) (2020) 9183–9193.
  * [18] G. Chen, Deep neural network approximations for the stable manifolds of the hamilton-jacobi equations (2023).  arXiv:2007.15350.
  * [19] Y. Zhao, J. Han, Offline supervised learning v.s. online direct policy optimization: A comparative study and a unified training paradigm for neural network-based optimal feedback control (2023).  arXiv:2211.15930.
  * [20] T. Nakamura-Zimmerer, Q. Gong, W. Kang, Qrnet: Optimal regulator design with LQR-augmented neural networks, IEEE Control Systems Letters 5 (4) (2021) 1303–1308.
  * [21] T. Nakamura-Zimmerer, Q. Gong, W. Kang, Adaptive deep learning for high-dimensional Hamilton–Jacobi–Bellman equations, SIAM J. Sci. Comput. 43 (2) (2021) A1221––A1247.
  * [22] B. Azmi, D. Kalise, K. Kunisch, Optimal feedback law recovery by gradient-augmented sparse polynomial regression, J. Mach. Learn. Res. 22 (48) (2021) 1––32.
  * [23] W. Kang, Q. Gong, T. Nakamura-Zimmerer, Algorithms of data development for deep learning and feedback design (2019).
URL https://arxiv.org/abs/1912.00492

  * [24] G. Albi, S. Bicego, D. Kalise, Gradient-augmented supervised learning of optimal feedback laws using state-dependent Riccati equations, IEEE Control Systems Letters 6 (2022) 836–841.
  * [25] T. Ehring, B. Haasdonk, Hermite kernel surrogates for the value function of high-dimensional nonlinear optimal control problems (2023).  arXiv:2305.06122.
  * [26] M. Akian, S. Gaubert, A. Lakhoua, The max-plus finite element method for solving deterministic optimal control problems: Basic properties and convergence analysis, SIAM J. Control Optim. 47 (2) (2008) 817–848.
  * [27] S. Gaubert, W. McEneaney, Z. Qu, Curse of dimensionality reduction in max-plus based approximation methods: Theoretical estimates and improved pruning algorithms, in: 2011 50th IEEE Conference on Decision and Control and European Control Conference, 2011, pp. 1054–1061.
  * [28] P. M. Dower, W. M. McEneaney, H. Zhang, Max-plus fundamental solution semigroups for optimal control problems, in: 2015 Proceedings of the Conference on Control and its Applications, 2015, pp. 368––375.
  * [29] D. Kalise, K. Kunisch, Polynomial approximation of high-dimensional Hamilton–Jacobi–Bellman equations and applications to feedback control of semilinear parabolic pdes, SIAM J. Sci. Comput. 40 (2) (2018) A629–A652.
  * [30] D. Kalise, S. Kundu, K. Kunisch, Robust feedback control of nonlinear pdes by numerical approximation of high-dimensional Hamilton–Jacobi–Isaacs equations, SIAM J. Appl. Dyn. Syst. 19 (2) (2020) 1496–1524.
  * [31] M. B. Horowitz, A. Damle, J. W. Burdick, Linear Hamilton Jacobi Bellman equations in high dimensions, in: 2014 53rd IEEE Conf. Decis. Control, 2014, pp. 5880–5887.
  * [32] E. Stefansson, Y. P. Leong, Sequential alternating least squares for solving high dimensional linear Hamilton–Jacobi–Bellman equation, in: 2016 IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 2016, pp. 3757––3764.
  * [33] A. Gorodetsky, S. Karaman, Y. Marzouk, High-dimensional stochastic optimal control using continuous tensor decompositions, Int. J. Robot. Res. 37 (2-3) (2018) 340–377.
  * [34] S. Dolgov, D. Kalise, K. Kunisch, Tensor decomposition methods for high-dimensional Hamilton–Jacobi–Bellman equations, SIAM J. Sci. Comput. 43 (3) (2021) A1625––A1650.
  * [35] M. Oster, L. Sallandt, R. Schneider, Approximating the stationary Hamilton–Jacobi–Bellman equation by hierarchical tensor products, arXiv (2019).
URL https://arxiv.org/abs/1911.00279

  * [36] M. Oster, L. Sallandt, R. Schneider, Approximating optimal feedback controllers of finite horizon control problems using hierarchical tensor formats, SIAM Journal on Scientific Computing 44 (3) (2022) B746–B770.
  * [37] A. Alla, M. Falcone, S. Volkwein, Error analysis for pod approximations of infinite horizon problems via the dynamic programming approach, SIAM Journal on Control and Optimization 55 (5) (2017) 3091––3115.
  * [38] K. Kunisch, S. Volkwein, L. Xie, Hjb-pod-based feedback design for the optimal control of evolution problems, SIAM Journal on Applied Dynamical Systems 3 (4) (2004) 701–722.
  * [39] A. Alla, M. Falcone, L. Saluzzi, An efficient dp algorithm on a tree-structure for finite horizon optimal control problems, SIAM J. Sci. Comput. 41 (4) (2019) A2384––A2406.
  * [40] Bokanowski, J. Garcke, M. Griebel, I. Klompmaker, An adaptive sparse grid semi- Lagrangian scheme for first order Hamilton-Jacobi Bellman equations, J. Sci. Comput. 55 (3) (2013) 575–605.
  * [41] J. Garcke, A. Kröner, Suboptimal feedback control of pdes by solving hjb equations on adaptive sparse grids, J. Sci. Comput. 70 (2017) 1–28.
  * [42] W. Kang, L. C. Wilcox, Mitigating the curse of dimensionality: sparse grid characteristics method for optimal feedback control and HJB equations, Computational Optimization and Applications 68 (2017) 289–315.
  * [43] O. Bokanowski, X. Warin, A. Prost, Neural networks for first order hjb equations and application to front propagation with obstacle terms (2022).  arXiv:2210.04300.
  * [44] D. Kalise, K. Kunisch, Z. Rao, Hamilton-Jacobi-Bellman Equations: Numerical Methods and Applications in Optimal Control, Vol. 21 De Gruyter - Radon Series on Computational and Applied Mathematics, De Gruyter, Berlin, 2018\.
  * [45] K. Kunisch, D. Vásquez-Varas, D. Walter, Learning optimal feedback operators and their polynomial approximation (2022).
URL https://arxiv.org/abs/2208.14120

  * [46] K. Kunisch, D. Walter, Optimal feedback control of dynamical systems via value-function approximation, Tech. rep. (2022).
  * [47] K. Kunisch, D. Vásquez-Varas, Optimal polynomial feedback laws for finite horizon control problems (2023).  arXiv:2302.09878.
  * [48] W. H. Fleming, H. M. Soner, Controlled Markov Processes and Viscosity Solutions, second ed. Edition, Stochastic Modelling and Applied Probability, Springer, New York, 2006.
  * [49] P. Cannarsa, C. Sinestrari, Semiconcave Functions, Hamilton—Jacobi Equations, and Optimal Control, Birkhäuser Boston, 2004.
  * [50] M. Bardi, I. Capuzzo-Dolcetta, Optimal Control and Viscosity Solutions of Hamilton-Jacobi-Bellman Equations, Birkhäuser Boston, 1997.
  * [51] J. M. Lasry, P. L. Lions, A remark on regularization in hilbert spaces, Israel Journal of Mathematics 55 (3) (1986) 257–266.
  * [52] F. Clarke, Y. Ledyaev, R. Stern, Asymptotic stability and smooth lyapunov functions, Journal of Differential Equations 149 (1) (1998) 69–114.
  * [53] L. Evans, Partial Differential Equations, Graduate studies in mathematics, American Mathematical Society, 2010.