remarkRemark hypothesisHypothesis claimClaim Optimal Control of Hybrid
SystemsR. Pytlak and D. Suski

# Algorithms for optimal control of hybrid systems with sliding motion

RadosŁaw Pytlak Faculty of Mathematics and Information Science, Warsaw
University of Technology, 00-665 Warsaw, Poland<EMAIL_ADDRESS>Damian Suski Institute of Automatic Control and Robotics, Warsaw University of
Technology, 02-525 Warsaw, Poland<EMAIL_ADDRESS>
###### Abstract

This paper concerns two algorithms for solving optimal control problems with
hybrid systems. The first algorithm aims at hybrid systems exhibiting sliding
modes. The first algorithm has several features which distinguishes it from
the other algorithms for problems described by hybrid systems. First of all,
it can cope with hybrid systems which exhibit sliding modes. Secondly, the
systems motion on the switching surface is described by index 2
differential–algebraic equations and that guarantees accurate tracking of the
sliding motion surface. Thirdly, the gradients of the problems functionals are
evaluated with the help of adjoint equations. The adjoint equations presented
in the paper take into account sliding motion and exhibit jump conditions at
transition times. We state optimality conditions in the form of the weak
maximum principle for optimal control problems with hybrid systems exhibiting
sliding modes and with piecewise differentiable controls. The second algorithm
is for optimal control problems with hybrid systems which do not exhibit
sliding motion. In the case of this algorithm we assume that control functions
are ${\mathcal{L}}^{\infty}$ measurable functions. For each algorithm, we show
that every accumulation point of the sequence generated by the algorithm
satisfies the weak maximum principle.

###### keywords:

optimal control, hybrid systems, sliding modes, the weak maximum principle,
algorithm for optimal control problem.

49J15, 49K15, 65K10, 34K34

##  1 Introduction

Hybrid systems are systems with mixed discrete-continuous dynamics ([27],[3]).
The set of discrete states $\mathcal{Q}$ consist of finite number of elements
denoted by $q$. The admissible controls set $\mathcal{U}$ consists of
measurable functions $u:I\rightarrow U$ defined on a closed interval $I$ with
the values in $U\in\mathbb{R}^{m}$. The continuous dynamics in each discrete
state is described by ordinary differential equations (ODEs)

(1) $\displaystyle{\displaystyle x^{\prime}=f(x,u)}$

or more generally by differential-algebraic equation (DAEs)

(2) $\displaystyle{\displaystyle 0=F(x^{\prime},x,u),}$

where $x\in\mathbb{R}^{n}$, $f:\mathbb{R}^{n}\times
U\rightarrow\mathbb{R}^{n}$, $F:\mathbb{R}^{n}\times\mathbb{R}^{n}\times
U\rightarrow\mathbb{R}^{n}$. The transitions between discrete states are
triggered when the condition of the form

(3) $\eta(x,u)\leq 0$

stops to be satisfied, where $\eta:\mathbb{R}^{n}\times
U\rightarrow\mathbb{R}$. The functions $\eta(x,u)$ are called guards. We
restrict our analysis to systems with autonomous transitions and without state
jumps during transitions.

The optimal control problem with a hybrid system has been considered in many
papers. The necessary optimality conditions for a class of hybrid systems
without state jumps have been first formulated in [28]. In [4] the variational
methods have been used to formulate adjoint equations for systems with state
jumps. The Pontryagin maximum principle for hybrid systems with state jumps
has been formulated for several classes of hybrid systems in [23], [19], [20],
[21], [24], [25], [26], [11]. In papers [19], [20], [21], [24], [26] also
algorithms based on the hybrid maximum principle are discussed. Our paper does
not consider optimal control problems in which switching times are decision
variables ([18]).

In [13] we have introduced an algorithm for optimal control problems with
hybrid systems described by higher index DAEs. In the paper we have assumed
that we deal with discrete time differential–algebraic equations resulting
from the system equations discretization by an implicit Runge–Kutta method.

In none of these papers an optimal control problem with hybrid systems
exhibiting sliding modes has been considered. This paper fills that gap by
considering hybrid systems whose motion can be on the switching manifold
(transition guard). For these optimal control problems we formulate a weak
version of the maximum principle provided that controls are represented by
piecewise smooth controls. Furthermore, we propose an algorithm which
generates a sequence of approximating optimal controls which converges to a
control which satisfies optimality conditions stated by the weak maximum
principle. The convergence is established under the assumption that continuous
time system equations are given. In the accompanying papers ([14],[15]) we
show that the algorithm convergence can still be attained if state
trajectories are obtained by an implicit Runge–Kutta method provided that the
integration procedure step sizes go to zero.

The main focus of the paper is on the problems with hybrid systems which can
exhibit sliding motion. For these problems we propose algorithms which have
global convergence properties. One algorithm is a first order method which is
stated for problems with piecewise smooth controls. We also discuss the method
which is intended for control problems with hybrid systems whose trajectories
do not experience sliding motion. In the case of these optimal control
problems we assume that control functions are elements of the space
${\mathcal{L}}^{\infty}$.

Our convergence results are based on the trajectory sensitivity analysis of
hybrid systems presented in the paper [16].

The computational approach to hybrid systems we advocate in this paper (and in
[13], [14], [15]) assumes that for a given control function we attempt to
follow true systems trajectories as close as possible by using integration
procedures with a high order of convergence. It is in contrast to the approach
which uses approximations to the discontinuous right-hand side for the
differential equations or inclusions by a smooth right–hand side ([22], [8],
[10]). The smoothing approach demonstrates that it can be used to find
approximate solutions to optimal control problems with systems which exhibit
sliding modes however there is still a lack of evidence that it is capable of
efficiently finding optimal controls and state trajectories with high accuracy
by choosing proper values of smoothing parameters.

##  2 Calculating the optimal control

In our work we consider hybrid systems with sliding modes. For the sake of
simplicity let us consider a hybrid system with two discrete states collected
in a set $\mathcal{Q}=\\{1,2\\}$. Let us assume that the invariant sets are
$\mathcal{I}(1)=\left\\{x\in\mathbb{R}^{n}:h(x)\leq 0\right\\}$ and
$\mathcal{I}(2)=\left\\{x\in\mathbb{R}^{n}:h(x)\geq 0\right\\}$ where
$h:\mathbb{R}^{n}\rightarrow\mathbb{R}$. If the hybrid system starts its
evolution from a discrete state $q=1$ the continuous state evolves according
to an equation $x^{\prime}=f_{1}(x,u)$. At a transition time $t_{t}$ the
continuous state trajectory reaches the boundary of an invariant set and we
have $h(x(t_{t}))=0$. The first order condition which guarantees that the
continuous state trajectory will leave the invariant set $\mathcal{I}(1)$ is
([9])

(4) $h_{x}(x(t_{t}))f_{1}(x(t_{t}),u(t_{t}))>0$

where $h_{x}^{T}(x)$ is the normal vector to a surface

(5) $\Sigma=\\{x\in\mathbb{R}^{n}:h(x)=0\\}$

at $x$. If at a transition time we also have

(6) $h_{x}(x(t_{t}))f_{2}(x(t_{t}),u(t_{t}))>0$

then the discrete state changes from $q=1$ to $q=2$ and the continuous state
continues the evolution according to the equation $x^{\prime}=f_{2}(x,u)$. If
at a transition time we have

(7) $h_{x}(x(t_{t}))f_{2}(x(t_{t}),u(t_{t}))<0$

then both vector fields $f_{1}(x,u)$ and $f_{2}(x,u)$ point towards the
surface $\Sigma$ and we face the sliding motion phenomenon ([6]).

The sliding motion can be handled with the concept of Filippov solutions
([7],[6]). We say that a continuous state trajectory is a Filippov solution of
the considered hybrid system if

(11) $\displaystyle{\displaystyle
x^{\prime}=\left\\{\begin{array}[]{ll}f_{1}(x,u)&{\rm if}\ h(x)<0\\\
f_{F}(x,u)&{\rm if}\ h(x)=0\\\ f_{2}(x,u)&{\rm if}\ h(x)>0\end{array}\right.}$

where $f_{F}(x,u)$ is a convex combination of $f_{1}(x,u)$ and $f_{2}(x,u)$

(12) $f_{F}(x,u)=(1-\alpha)f_{1}(x,u)+\alpha f_{2}(x,u),\ \alpha\in[0,1].$

During the sliding motion the continuous state trajectory must stay in the
surface $\Sigma$, so the condition

(13) $h_{x}(x)f_{F}(x,u)=0$

must be satisfied. From (12) and (13) it is easy to find the formula for
$\alpha$ coefficient

(14)
$\alpha(x,u)=\frac{h_{x}(x)f_{1}(x,u)}{h_{x}(x)\left(f_{1}(x,u)-f_{2}(x,u)\right)}.$

When dealing with hybrid systems we have to pay special attention to systems
behavior at switching times $t_{t}$, on the left and the right of these
points. To this end we define

(15) $\displaystyle{\displaystyle u(t_{t}^{-})=\lim_{t\rightarrow
t_{t},t<t_{t}}u(t),\ u(t_{t}^{+})=\lim_{t\rightarrow t_{t},t>t_{t}}u(t),}$

and we assume that $u$ is continuous from the left which means that

(16) $\displaystyle{\displaystyle u(t_{t}^{-})=u(t_{t})\ \ {\rm and}\
u(t_{t}^{+})\neq u(t_{t})}$

in general.

This notation at a neighborhood of time $t_{t}$ apply also to other functions,
for example

(17) $\displaystyle{\displaystyle x(t_{t}^{-})=\lim_{t\rightarrow
t_{t},t<t_{t}}x(t),\ x(t_{t}^{+})=\lim_{t\rightarrow t_{t},t>t_{t}}x(t).}$

In this setting general conditions (4)-(6) for changing discrete states must
be stated as follows:

(18) $\displaystyle h_{x}(x(t_{t}^{-}))f_{1}(x(t_{t}^{-}),u(t_{t}^{-}))$
$\displaystyle>$ $\displaystyle 0$ (19) $\displaystyle
h_{x}(x(t_{t}^{+}))f_{2}(x(t_{t}^{+}),u(t_{t}^{+}))$ $\displaystyle>$
$\displaystyle 0,$

and, in particular case when $h$ is defined through the function $\alpha(x,u)$
and, for example, the system transits to state $q=1$

(20) $\displaystyle\alpha(x(t_{t}^{-}),u(t_{t}^{-}))$ $\displaystyle=$
$\displaystyle 0$ (21) $\displaystyle
h_{x}(x(t_{t}^{+}))f_{1}(x(t_{t}^{+}),u(t_{t}^{+}))$ $\displaystyle<$
$\displaystyle 0$ (22) $\displaystyle
h_{x}(x(t_{t}^{+}))f_{2}(x(t_{t}^{+}),u(t_{t}^{+}))$ $\displaystyle<$
$\displaystyle 0$

The introduced convention is also applied to integrals, for example

$\displaystyle{\displaystyle\int_{t_{0}}^{t_{t}^{-}}p(x(t),u(t))dt=\lim_{t\rightarrow
t_{t},t<t_{t}}\int_{t_{0}}^{t}p(x(\tau),u(\tau))d\tau.}$

But taking piecewise smooth functions as controls requires also checking at
each point $t_{j}$ (see (3)) whether the system is still at a sliding mode
after control changes at time $t_{j}$, if not a new discrete state at this
time must be determined.

During the sliding motion the continuous state trajectory should stay in a set
$\Sigma$. To keep this condition satisfied during the numerical integration,
we follow the approach proposed in [1] and integrate the differential-
algebraic equations (DAEs)

(23) $\displaystyle x^{\prime}$ $\displaystyle=$ $\displaystyle
f_{F}(x,u)+h_{x}(x)^{T}z$ (24) $\displaystyle 0$ $\displaystyle=$
$\displaystyle h(x)$

instead of ODEs

(25) $x^{\prime}=f_{F}(x,u).$

It can be shown that at a sliding motion equations (23)–(24) and (25) are
equivalent ([16]). We prefer using (23)–(24) instead of (25) for the reason of
getting more stable numerical solution on the switching surface.

The sliding motion can be treated as the third discrete state with $q=3$.
Treating the sliding motion as another discrete state implies that guards
functions $h$ should depend also on control variables $u$ and system equations
should be given, in general, by implicit equations (23)-(24).

From now on the invariant set at a discrete state $q$ will be defined as
follows

(26) $\mathcal{I}(q)=\left\\{(x,z,u)\in\mathbb{R}^{n+1+m}:\eta_{q}(x,u)\leq
0\right\\},\ \eta_{q}:\mathbb{R}^{n}\times
U\rightarrow\mathbb{R}^{n_{\mathcal{I}_{q}}}$

Furthermore, the function ${\mathcal{F}}$ is related to the functions $f_{q}$
and $h_{q}$ defining, for a given discrete state $q$, DAEs:

$\displaystyle x^{\prime}$ $\displaystyle=$ $\displaystyle
f_{q}(x,u)+\left(h_{q}\right)_{x}(x)^{T}z$ $\displaystyle 0$ $\displaystyle=$
$\displaystyle h_{q}(x).$

In particular, we can have $h_{q}(x)\equiv 0$ for some $q$ and then the
guarding function $\eta_{q}$ is not dependent on $u$.

If we assume that a switching surface function $\eta$ depends on $u$ we need
continuity of function $\eta(x(t),u(t))$ to locate uniquely a switching time
$t_{t}$. For that reason we assume that control functions $u$ are piecewise
smooth—another justification for the chosen admissible controls is given in
the next Section 3 by referring to properties of the Filippov’s solution
stated in [7].

Another implication of using switching functions dependent on $u$ is the
necessity of updating the discrete state $q$ whenever control functions $u$
exhibit jumps.

Consider the system

(27) $\displaystyle x^{\prime}$ $\displaystyle=$ $\displaystyle
f(x,u)+h_{x}(x)^{T}z=f_{r}(x,z,u)$ (28) $\displaystyle 0$ $\displaystyle=$
$\displaystyle h(x),$

on $[t_{0},t_{f}]$. We assume that on some subintervals $t^{0},t^{1}]$ we can
have $h(x)\equiv 0$ and in that case system (27)–(28) reduces to ODEs. Since
we consider, for the simplicity of presentation, that our hybrid system can be
only in three states $q=1,2,3$ (and the third state reserved for a sliding
motion) our state trajectory on the entire horizon $[t_{0},t_{f}]$ will
consist of trajectories of type $(x,z)$ on subintervals $A^{s}_{i}$, $i\in
I_{s}$, and of type $x$ on subintervals $A^{1}_{i}$, $i\in I_{1}$ (if
trajectory is defined by the function $f_{1}$) and subintervals $A^{2}_{i}$,
$i\in I_{2}$ (if trajectory is defined by the function $f_{2}$). We have
$\cup_{i\in I^{s}_{i}}A^{s}_{i}\cup_{i\in I_{1}}A^{1}_{i}\cup_{i\in
I_{2}}A^{2}_{i}=[t_{0},t_{f}]$.

Our approach is based on sensitivity analysis stated in [16] where it is shown
that solutions to linearized equations of DAEs have properties similar to
those exhibited by linearized equations to ordinary differential equations,
provided that some conditions are met. Namely, if we denote by $(x,z)$ the
solution of (27)–(28) for given $u$, by $(x^{d},z^{d})$ the solution for the
control $u+d$, and by $(y^{x,d},y^{z,d})$ the solution to the linearized
equations:

(29) $\displaystyle\left(y^{x}\right)^{\prime}$ $\displaystyle=$
$\displaystyle
f_{x}(x,u)y^{x}+\left(h_{x}(x)^{T}z\right)_{x}y^{x}+h_{x}(x)^{T}y^{z}+f_{u}(x,u)d$
(30) $\displaystyle 0$ $\displaystyle=$ $\displaystyle h_{x}(x)y^{x},$

then the following will be satisfied

(31) $\displaystyle\|(x,z)\|_{{\mathcal{L}}^{\infty}}$ $\displaystyle\leq$
$\displaystyle c_{1}$ (32)
$\displaystyle\|(x^{d},z^{d})-(x,z)\|_{{\mathcal{L}}^{\infty}}$
$\displaystyle\leq$ $\displaystyle c_{2}\|d\|_{{\mathcal{L}}^{\infty}}$ (33)
$\displaystyle\|(y^{x,d},y^{z,d})\|_{{\mathcal{L}}^{\infty}}$
$\displaystyle\leq$ $\displaystyle c_{3}\|d\|_{{\mathcal{L}}^{1}}$

for some positive constants $c_{1}$, $c_{2}$, $c_{3}$. Furthermore, for any
$\varepsilon$ satisfying $\varepsilon_{t}>\varepsilon>0$, where
$\varepsilon_{t}$ is as specified in ${\bf(H3)}$, there exists function
$o:([0,\infty]\rightarrow(0,\infty)$ such that $\lim_{s\rightarrow
0^{+}}o(s)/s=0$ and

$\displaystyle{\displaystyle\|x^{d}(t)-\left(x(t)+y^{x,d}(t)\right)\|\leq
o\left(\|d\|_{{\mathcal{L}}^{\infty}}\right),}$ (34)
$\displaystyle{\displaystyle\hskip 128.0374pt\forall
t\in[t_{0},t_{t}^{1}-\varepsilon]\bigcup[t_{f}-\varepsilon,t_{f}]\bigcup_{i\in
I_{t}(u)\backslash\\{N_{t}(u)\\}}[t_{t}^{i},t_{t}^{i+1}-\varepsilon].}$

Here, the points $\\{t_{t}^{i}\\}_{i=1}^{N_{t}(u)}$ are as specified in the
assumption ${\bf(H3)}$.

In order to specify the relations for differentials of $x$ at a switching
point $t_{t}$, denoted by $dx^{d}$, additional notation has to be introduced.
Suppose that a control $u$ is perturbed by $d$, then a new switching point
$t_{t}^{d}$ will occur. We assume that before switching of discrete states the
system evolves according to equations (27–(28) with $f$ and $h$ replaced by
$f_{1}$ and $h_{1}$ respectively and after the switching the system is
described by (27–(28) with $f_{2}$ and $h_{2}$ instead of $f$ and $h$. Then,
we have

(35)
$\displaystyle\left\|x^{d}_{1}(t_{t}^{d-})-x_{1}(t_{t}^{-})-d\left(x_{1}\right)^{d}_{-}\right\|$
$\displaystyle\leq$ $\displaystyle o(\|d\|_{{\mathcal{L}}^{\infty}}),$ (36)
$\displaystyle\left\|x^{d}_{2}(t_{t}^{d+})-x_{2}(t_{t}^{+})-d\left(x_{2}\right)^{d}_{+}\right\|$
$\displaystyle\leq$ $\displaystyle o(\|d\|_{{\mathcal{L}}^{\infty}}),$

for all $u$ and $d$ such that $u+d\in{\mathcal{U}}$ where
$d\left(x_{1}\right)^{d}_{-}=y^{x,d}_{1}(t_{t}^{-})+f_{1r}(x_{1}(t_{t}^{-}),$
$z_{1}(t_{t}^{-}),u(t_{t}^{-}))$ $\times dt_{t}^{d}$, $x_{1}$, $x^{d}_{1}$,
$z_{1}$ are solutions to equations (27)–(28) with $f=f_{1}$, $h=h_{1}$,
$y^{x,d}_{1}$ are solutions to the corresponding linearized equations
associated with equations (27)–(28),
$d\left(x_{2}\right)^{d}_{+}=y^{x,d}_{2}(t_{t}^{+})+f_{2r}(x_{2}(t_{t}^{+}),z_{2}(t_{t}^{+}),u(t_{t}^{+}))dt_{t}^{d}$,
$x_{2}$, $x^{d}_{2}$, $z_{2}$ are solutions to equations (27)–(28) with
$f=f_{2}$, $h=h_{2}$ and $y^{x,d}_{2}$ are solutions to the corresponding
linearized equations associated with equations (27)–(28). Here, $dt_{t}^{d}$
is the differential of the switching point $t_{t}$ and $o$ are such that
$\lim_{s\rightarrow 0^{+}}|o(s)|/s=0$; $dx^{d}_{-}$ and $dx^{d}_{+}$
differentials correspond to systems behavior before and after the switching
($dx^{d}_{-}=dx^{d}_{+}$).

In order to present the conjectures under which (31)–(34) hold we introduce
the function $e(x,z,u)=h_{x}(x)f(x,u)+\|h_{x}(x)\|^{2}z$. Then under (H1),
(H2) and (H3) the relations (31)–(34) are satisfied.

${\bf(H1)}$

$U$ is a compact convex set in ${\mathbb{R}}^{m}$,
$f:{\mathbb{R}}^{n}\times{\mathbb{R}}^{m}\rightarrow{\mathbb{R}}^{n}$,
$h:{\mathbb{R}}^{n}\rightarrow{\mathbb{R}}$,
i) function $f(\cdot,\cdot)$ is differentiable, $f_{x}$ $f_{u}$ are continuous
and there exists $0<k_{f}^{1}<+\infty$ such that

(37) $\displaystyle{\displaystyle\|f_{x}(x,u)\|\leq k_{f}^{1}}$

for all $(x,u)\in{\mathbb{R}}^{n}\times U$,
ii) if $h(x)\not\equiv 0$ then $(x(t_{0}),z(t_{0}))$ are such that

(38) $\displaystyle{\displaystyle h(x(t_{0}))=0,\ z(t_{0})=0,\
h_{x}(x(t_{0}))f(x(t_{0}),u(t_{0}))=0,}$

$h(\cdot)$ is twice differentiable and $\nabla^{2}h$ is continuous. Moreover,
there exist $\varepsilon>0$, $0<k_{hf}<+\infty$, $0<k_{e}^{1}<+\infty$,
$0<k_{f}^{2}<+\infty$, $0<k_{f}^{3}<+\infty$, $0<k_{h}^{1}<k_{h}^{2}<+\infty$,
$0<k_{h}^{3}<+\infty$ with the properties

(39) $\displaystyle{\displaystyle\|f(x,u)\|\leq k_{f}^{2}}$ (40)
$\displaystyle{\displaystyle\|f_{u}(x,u)\|\leq k_{f}^{3}}$ (41)
$\displaystyle{\displaystyle\|h_{x}(x)f(x,u)\|\leq k_{hf}}$ (42)
$\displaystyle{\displaystyle\|e_{x,z,u}(\hat{x},\hat{z},\hat{u})-e_{x,z,u}(x,z,u)\|\leq
k_{e}^{1}\|(\hat{x},\hat{z},\hat{u})-(x,z,u)\|}$ (43)
$\displaystyle{\displaystyle k_{h}^{1}\leq\|h_{x}(x)\|\leq k_{h}^{2}}$ (44)
$\displaystyle{\displaystyle\|\nabla^{2}h(x)\|\leq k_{h}^{3}}$

for all $(x,z,u)$, $(\hat{x},\hat{z},\hat{u})$ in
$\mathbb{R}^{n}\times\mathbb{R}\times U$ such that $|h(x)|\leq\varepsilon$,
$|h(\hat{x})|\leq\varepsilon$.

In ${\bf(H1)}$ it is assumed that $U$ is a compact convex set. In order to
show (31)–(34) it is sufficient to assume that $U$ is bounded, but the
convergence analysis which is based on (31)–(34) requires the stronger
assumption.

${\bf(H2)}$

i) Function $\eta(\cdot,\cdot)$ is differentiable and there exist
$0<L_{1}<+\infty$ and $0<L_{2}<+\infty$ such that

(45) $\displaystyle\left\|\eta_{x,u}(\hat{x},\hat{u})-\eta_{x,u}(x,u)\right\|$
$\displaystyle\leq$ $\displaystyle L_{1}\|(\hat{x},\hat{u})-(x,u)\|$ (46)
$\displaystyle\left|f_{r}(\hat{x},\hat{z},\hat{u})-f_{r}(x,z,u)\right|$
$\displaystyle\leq$ $\displaystyle L_{2}\|(\hat{x},\hat{z},\hat{u})-(x,z,u)\|$

for all $(x,z,u)$, $(\hat{x},\hat{z},\hat{u})$ in
$\mathbb{R}^{n}\times\mathbb{R}\times U$.

ii) There exists $\varepsilon>0$ and $0<L_{3}<+\infty$ such that for all
switching points $t_{t}$ (each switching time corresponds to some
$u\in{\mathcal{U}}$) and for all their perturbations $t_{t}^{d}$ triggered by
perturbations $d$ such that $u+d\in{\mathcal{U}}$,
$\|d\|_{{\mathcal{L}}^{\infty}}\leq\varepsilon$, and for all $\theta\in[0,1]$
we have

$\displaystyle{\displaystyle\left|\eta_{x}(x(\tau^{\theta,d}(t_{t})),u(\tau^{\theta,d}(t_{t})))f_{r}(x(\tau^{\theta,d}(t_{t})),z(\tau^{\theta,d}(t_{t})),u(\tau^{\theta,d}(t_{t})))\right.}$
(47)
$\displaystyle{\displaystyle\left.+\eta_{u}(x(\tau^{\theta,d}(t_{t})),u(\tau^{\theta,d}(t_{t})))u^{\prime}(\tau^{\theta,d}(t_{t}))\right|\geq
L_{3}}$

where

(48) $\displaystyle\tau^{\theta,d}(t_{t})$ $\displaystyle=$ $\displaystyle
t_{t}+\theta(t^{d}_{t}-t_{t}).$

iii) Perturbations $d\in{\mathcal{U}}-u$ are such that

(49) $\displaystyle{\displaystyle\|d\|_{{\mathcal{L}}^{\infty}}\rightarrow
0\implies\|d^{\prime}\|_{{\mathcal{L}}^{\infty}}\rightarrow 0.}$

Notice that if system leaves discrete state $q=1$, or $q=2$ then $\eta=h$ and
does not depend on $u$.

We denote by $N_{t}(u)$ the number of switching times triggered by a control
$u\in{\mathcal{U}}$ and by $I_{t}(u)$ the set of indices which indicates all
switching points.

We make the hypothesis

(H3)

There exists a nonnegative integer number $I_{t}^{{\rm max}}<+\infty$ and
$\varepsilon_{t}>0$ such that

(50) $\displaystyle N_{t}(u)$ $\displaystyle\leq$ $\displaystyle I_{t}^{{\rm
max}},$ (51) $\displaystyle t_{t}^{i}$ $\displaystyle\leq$ $\displaystyle
t_{t}^{i+1}-\varepsilon_{t},\ \forall i\in I_{t}(u)\
\backslash\\{N_{t}(u)\\},$ (52) $\displaystyle t_{t}^{N(u)}$
$\displaystyle\leq$ $\displaystyle t_{f}-\varepsilon_{t}$

for all $u\in{\mathcal{U}}$.

##  3 Optimal control problem with sliding modes and piecewise smooth controls

Taking into account the considerations and definitions presented in the
previous section, in particular including the possibility of the system
entering (and spending some time) in the sliding mode, we now turn attention
to the optimal control problem of interest—(P):

(53) $\displaystyle\min_{u}\phi(x(t_{f}))$

subject to the constraints

$\displaystyle\begin{array}[]{lc}x^{\prime}=f_{1}(x,u)&\ \ {\rm if}\ q=1\\\
x^{\prime}=f_{2}(x,u)&\ \ {\rm if}\ q=2\\\
\left\\{\begin{array}[]{lll}x^{\prime}&=&f_{F}(x,u)+h_{x}(x)^{T}z\\\
0&=&h(x)\end{array}\right.&\ \ {\rm if}\ q=3\end{array}$ (56)

and the terminal constraints

(57) $\displaystyle g^{1}_{i}(x(t_{f}))=0\ \forall i\in E$ (58) $\displaystyle
g^{2}_{j}(x(t_{f}))\leq 0\ \forall j\in I.$

We also assume that the initial state $x(t_{0})=x_{0}$ is fixed.

Crucial assumption concerns the class of control functions. Our algorithm
requires that control functions are piecewise smooth functions. The
justification for that is given [16] by referring to properties of Filippov’s
solutions.

Thus the admissible control $u$ is a function $u$ from the set
${\mathcal{U}}$, where

$\displaystyle{\displaystyle{\mathcal{U}}=\left\\{u\in{\mathcal{C}}^{N}_{m}[T]:\
u(t)\in U\subset\mathbb{R}^{m},\ t\in[t_{0},t_{f}]\right\\}}$
$\displaystyle{\displaystyle{\mathcal{C}}_{m}^{N}[T]=\left\\{u\in{\mathcal{L}}^{2}_{m}[T]:\
u(t)=\sum_{j=1}^{N}\phi_{j}(t)u^{N}_{j}(t),\right.}$
$\displaystyle{\displaystyle\left.u_{j}^{N}\in\mathcal{C}_{m}^{1}[t_{j-1},t_{j}],\
j=1,\ldots,N,\ t\in[t_{0},t_{f}]\right\\},}$
$\displaystyle{\displaystyle\phi_{j}(t)=\left\\{\begin{array}[]{ll}1,&{\rm
if}\ t\in[t_{j-1},t_{j})\\\ 0,&{\rm if}\
t\not\in[t_{j-1},t_{j})\end{array}\right.,\ t_{j}=jt_{f}/N,\
j\in\\{0,1,\ldots,N\\}.}$

The method we propose for solving the problem ${\bf(P)}$ is based on an exact
penalty function. By using an exact penalty function approach, instead of
solving the problem ${\bf(P)}$ we solve the problem ${\bf P_{c}}$

(60) $\displaystyle{\displaystyle\min_{u\in{\mathcal{U}}}\bar{F}_{c}(u)}$

in which the exact penalty function $\bar{F}_{c}(u)$ is defined as follows

$\displaystyle{\displaystyle\bar{F}_{c}(u)=\bar{F}(u)+c\max\left[0,\max_{i\in
E}\left|\bar{g}^{1}_{i}(u)\right|,\max_{j\in I}\bar{g}^{2}_{j}(u)\right]}$
(61)

For fixed $c$ and $u$ the direction finding subproblem, ${\bf P_{c}(u)}$, for
the problem ${\bf(P_{c})}$ is:

$\displaystyle{\displaystyle\min_{d\in{\mathcal{D}}_{u},\beta\in\mathbb{R}}\left[\left\langle\nabla\bar{F}_{0}(u),d\right\rangle+c\beta+1/2\|d\|^{2}_{{\mathcal{L}}^{2}}\right]}$

subject to

$\displaystyle\left|\bar{g}^{1}_{i}(u)+\left\langle\nabla\bar{g}^{1}_{i}(u),d\right\rangle\right|$
$\displaystyle\leq$ $\displaystyle\beta\ \ \forall i\in E$
$\displaystyle\bar{g}^{2}_{j}(u)+\left\langle\nabla\bar{g}^{2}_{j}(u),d\right\rangle$
$\displaystyle\leq$ $\displaystyle\beta\ \ \forall j\in I.$

Here,

$\displaystyle{\displaystyle{\mathcal{D}}_{u}\left\\{d\in{\mathcal{C}}^{N}_{m}[T]:\
d\in{\mathcal{U}}-u,\ \|d\|_{\mathcal{L}}^{2}\rightarrow
0\implies\|d^{\prime}\|_{{\mathcal{L}}^{2}}\rightarrow 0\right\\}.}$

The direction finding subproblem is based on first order approximations to the
problem functionals, defined as follows

(62) $\displaystyle\langle\nabla\bar{F}_{0}(u),d\rangle$ $\displaystyle=$
$\displaystyle\phi_{x}(x^{u}(t_{f}))y^{x,d}(t_{f})$ (63)
$\displaystyle\langle\nabla\bar{g}_{i}^{1}(u),d\rangle$ $\displaystyle=$
$\displaystyle\left(g_{i}^{1}\right)_{x}(x^{u}(t_{f}))y^{x,d}(t_{f}),\ i\in E$
(64) $\displaystyle\langle\nabla\bar{g}_{j}^{2}(u),d\rangle$ $\displaystyle=$
$\displaystyle\left(g_{j}^{2}\right)_{x}(x^{u}(t_{f}))y^{x,d}(t_{f}),\ j\in
I.$

The subproblem can be reformulated as an optimization problem with the
objective function which is strictly convex. The problem therefore has the
unique solution $(\bar{d},\bar{\beta})$. Since this solution depends on $c$
and $u$, we may define descent function $\sigma_{c}(u)$ and penalty test
function $t_{c}(u)$ (see [12] for details), to be used to test optimality of a
control $u$ and to adjust $c$, respectively, as

(65)
$\displaystyle{\displaystyle\sigma_{c}(u)=\left\langle\nabla\bar{F}_{0}(u),\bar{d}\right\rangle+c\left[\bar{\beta}-M(u)\right]}$

and

(66) $\displaystyle{\displaystyle t_{c}(u)=\sigma_{c}(u)+M(u)/c}$

for given $c>0$ and $u\in{\mathcal{U}}$. Here,

$\displaystyle{\displaystyle M(u)=\max\left[0,\max_{i\in
E}\left|\bar{g}^{1}_{i}(u)\right|,\max_{j\in I}\bar{g}^{2}_{j}(u)\right],}$

A general algorithm is as follows.
Algorithm Fix parameters: $\gamma,\ \eta\in(0,1)$, $c^{0}>0$, $\kappa>1$.

  1. 1.

Choose the initial control $u_{0}\in{\cal U}$. Set $k=0$, $c_{-1}=c^{0}$.

  2. 2.

Let $c_{k}$ be the smallest number chosen from $\\{c_{k-1},\kappa
c_{k-1},\kappa^{2}c_{k-1},\ldots\\}$ such that the solution
$(d_{k},\beta_{k})$ to the direction finding subproblem ${\bf
P_{c_{k}}(u_{k})}$ satisfies

(67) $\displaystyle{\displaystyle t_{c_{k}}(u_{k})\leq 0.}$

If $\sigma_{c_{k}}(u_{k})=0$ then STOP.

  3. 3.

Let $\alpha_{k}$ be the largest number chosen from the set
$\\{1,\eta,\eta^{2},\ldots,\\}$ such that

$\displaystyle{\displaystyle u_{k+1}=u_{k}+\alpha_{k}d_{k}}$

satisfies the relation

$\displaystyle\bar{F}_{c_{k}}(u_{k+1})-\bar{F}_{c_{k}}(u_{k})$
$\displaystyle\leq$ $\displaystyle\gamma\alpha_{k}\sigma_{c_{k}}(u_{k}).$

Increase k by one. Go to Step 2.

Since

$\displaystyle{\displaystyle\left\langle\nabla\bar{F}_{0}(u_{k}),d\right\rangle+c_{k}\beta+1/2\|d\|^{2}_{{\mathcal{L}}^{2}}\leq
c_{k}M(u_{k}),}$

which holds because $0\in{\mathcal{U}}-u_{k}$, we also have

(68)
$\displaystyle{\displaystyle\left\langle\nabla\bar{F}_{0}(u_{k}),d\right\rangle+c_{k}\left[\beta-M(u_{k})\right]\leq-1/2\|d\|^{2}_{{\mathcal{L}}^{2}}\leq
0}$

which implies that the descent function $\sigma_{c_{k}}(u_{k})$ is nonpositive
valued at each iteration.

Algorithm generates a sequence of controls $\\{u_{k}\\}$ and the corresponding
sequence of penalty parameters $\\{c_{k}\\}$ such that $\\{c_{k}\\}$ is
bounded and any accumulation point of $\\{u_{k}\\}$ satisfies optimality
conditions in the form of the weak maximum principle for the problem
${\bf(P)}$, i.e. $\sigma_{\bar{c}}(\bar{u})=0$ for the limit point $\bar{u}$
and the limit point of the sequence $\\{c_{k}\\}$. But
$\sigma_{\bar{c}}(\bar{u})=0$ implies that $M(\bar{u})=0$ due to the
definition (66) and since (67) holds.

It can be shown that the set ${\mathcal{U}}$ is compact (since we assume that
$U$ is compact) and this fact can be used to prove the convergence of
Algorithm without referring to relaxed controls (cf. [12]). However, we still
need to introduce two additional hypotheses. The first one concerns the
functions defining the objective and constraints:

(H4)

$\phi$, $g_{i}^{1}$, $i\in E$, $g_{j}^{2}$, $j\in I$ are continuously
differentiable functions.

The second one is related to a constraint qualification which is needed to
prove the convergence of Algorithm. To this end we first introduce the set

$\displaystyle{\displaystyle{\mathcal{D}}=\left\\{d\in{\mathcal{C}}^{N}_{m}[T]:\
\|d\|_{{\mathcal{L}}^{2}}\rightarrow
0\implies\|d^{\prime}\|_{{\mathcal{L}}^{2}}\rightarrow 0,\ \exists
u\in{\mathcal{U}}\ {\rm such\ that}\ u+d\in{\mathcal{U}}\right\\}}$

and the set

$\displaystyle{\displaystyle{\mathcal{F}}(u)=\left\\{d\in{\mathcal{D}}:\
\max_{j\in I}\left\langle\nabla\bar{g}_{j}^{2}(u),d\right\rangle<0\right\\}.}$

Then the constraint qualification condition takes the form

(CQ)

for each $u\in{\mathcal{U}}$, ${\mathcal{F}}(u)\neq\emptyset$, and in the case
$E\neq\emptyset$ we have

(69) $\displaystyle{\displaystyle 0\in{\rm interior}{\mathcal{E}}(u)}$

where

$\displaystyle{\displaystyle{\mathcal{E}}(u)=\left\\{\left\\{\langle\nabla\bar{g}^{1}_{i},d\rangle\right\\}_{i\in
E}\in\mathbb{R}^{|E|}:\ d\in{\mathcal{F}}(u)\right\\}.}$

Under stated assumptions Algorithm is globally convergent in the following
sense.

######  Theorem 3.1.

Assume that data for (P) satisfies hypotheses ${\bf(H1)}$, ${\bf(H2)}$,
${\bf(H3)}$, ${\bf(H4)}$ and ${\bf(CQ)}$. Let $\\{u_{k}\\}$ be a sequence of
controls generated by Algorithm and let $\\{c_{k}\\}$ be a sequence of the
corresponding penalty parameters. Then

  1. i)

$\\{c_{k}\\}$ is a bounded sequence

  2. ii)

(70)
$\displaystyle{\displaystyle\lim_{k\rightarrow\infty}\sigma_{c_{k}}(u_{k})=0,\
\ \lim_{k\rightarrow\infty}M(u_{k})=0.}$

  3. iii)

Let $\bar{u}$ be the limit point of the sequence $\\{u_{k}\\}$ and $\bar{x}$
the trajectory corresponding to $\bar{u}$, then the necessary optimality
conditions hold:

(NC)

:

(71) $\displaystyle{\displaystyle
0\leq\min_{d\in{\mathcal{D}}_{\bar{u}}}\phi_{x}(\bar{x}(t_{f}))y^{\bar{x},d}(t_{f})}$

subject to the constraints

(72) $\displaystyle
g_{i}^{1}(\bar{x}(t_{f}))+\left(g_{i}^{1}\right)_{x}(\bar{x}(t_{f}))y^{\bar{x},d}(t_{f})$
$\displaystyle=$ $\displaystyle 0,\ i\in E$ (73) $\displaystyle
g_{j}^{2}(\bar{x}(t_{f}))+\left(g_{j}^{2}\right)_{x}(\bar{x}(t_{f}))y^{\bar{x},d}(t_{f})$
$\displaystyle\leq$ $\displaystyle 0,\ j\in I_{0,\bar{u}}$

together with $g_{i}^{1}(\bar{x}(t_{f}))=0$, $i\in E$,
$g_{j}^{2}(\bar{x}(t_{f}))\leq 0$, $j\in I$. Here,

$\displaystyle{\displaystyle I_{\varepsilon,u}=\left\\{j\in I:\
\bar{g}_{j}^{2}(u)\geq\max_{j\in I}\bar{g}_{j}^{2}(u)-\varepsilon\right\\}.}$

The proof of Theorem 3.1 follows the lines of the proof of Theorem 5.1 stated
in [17], however for the completeness of presentation it is provided in
Appendix C. The proof heavily uses the implications of the constraint
qualification (CQ) stated in the following lemma and on the properties of
solutions to the system equations and solutions to their linearized equations
$(x,z)$, $(y^{x,d},y^{z,d})$ expressed by (31)–(34). The proof Lemma 3.1 is
given in Appendix A.

######  Lemma 3.1.

Assume ${\bf(H1)}$, ${\bf(H2)}$, ${\bf(H3)}$, ${\bf(H4)}$ and ${\bf(CQ)}$. For
any control $\tilde{u}\in{\mathcal{U}}$ there exist a neighborhood
${\mathcal{O}}(\tilde{u})$ of $\tilde{u}$, $K_{1}>0$ and $K_{2}>0$ with the
following properties: given any $u\in{\mathcal{U}}$ such that
$u\in{\mathcal{O}}(\tilde{u})$ there exists $v\in{\cal U}$ such that

(74) $\displaystyle{\displaystyle\max_{i\in
E}\left|\bar{g}^{1}_{i}(u)+\left\langle\nabla\bar{g}^{1}_{i}(u),v-u\right\rangle\right|-\max_{i\in
E}\left|\bar{g}^{1}_{i}(u)\right|\leq-K_{1}M(u)}$ (75)
$\displaystyle{\displaystyle\max_{j\in
I}\left\langle\nabla\bar{g}^{2}_{j}(u),v-u\right\rangle\leq-K_{1}M(u)}$ (76)
$\displaystyle{\displaystyle{\rm and}\ \
\left\|v-u\right\|_{{\mathcal{L}}^{\infty}}\leq\ \ K_{2}M(u).}$

The role of (CQ) is to ensure uniform boundedness of penalty parameter values.
In particular, if (CQ) is satisfied one can prove the following proposition
which is needed in order to demonstrate the feasibilities of Steps 2–3 of
Algorithm.

######  Proposition 3.1.

  1. (i)

Assume that hypotheses ${\bf(H1)}$, ${\bf(H2)}$, ${\bf(H3)}$, ${\bf(H4)}$ and
${\bf(CQ)}$ are satisfied. Then for any $u\in{\cal U}$ there exists
$\bar{c}>0$ such that for all $c>\bar{c}$

$\displaystyle{\displaystyle t_{c}(u)\leq 0.}$

  2. (ii)

Assume that hypotheses ${\bf(H1)}$, ${\bf(H2)}$, ${\bf(H3)}$ and ${\bf(H4)}$
are satisfied. Then for any $u\in{\cal U}$ and $c>0$ such that
$\sigma_{c}(u)<0$ there exists $\bar{\alpha}>0$ such that if
$\alpha\in[0,\bar{\alpha})$ then

$\displaystyle\tilde{F}_{c}(\tilde{u})-\tilde{F}_{c}(u)$ $\displaystyle\leq$
$\displaystyle\gamma\alpha\sigma_{c}(u)$

where $\tilde{u}=u+\alpha d$, $(d,\beta)$ is the solution to the direction
finding subproblem corresponding to $c$ and $u$ and $\sigma_{c}(u)$ is defined
by (65).

The proofs of Lemma 3.1 closely follows the lines of the proof of Lemma 6.1
stated in [17], so it is omitted here. The proof of Proposition 3.1 is given
in Appendix B.

##  4 Adjoint equations

The efficient implementation of Algorithm needs computationally tractable
algorithm for solving the direction finding subproblem ${\bf P_{c}(u)}$ which
at every iteration requires the evaluation of
$\langle\nabla\bar{F}_{0}(u),d\rangle$,
$\langle\nabla\bar{g}_{i}^{1}(u),d\rangle$, $i\in E$,
$\langle\nabla\bar{g}_{j}^{2}(u),d\rangle$, $j\in I$. If we do that by using
solutions to the linearized equations then each update of $d$ would request
solving these equations. If we use adjoint equations the evaluation comes down
to the calculations of scalar products since $\nabla\bar{F}_{0}(u)$,
$\nabla\bar{g}_{i}^{1}(u)$, $i\in E$, $\nabla\bar{g}_{j}^{2}(u)$, $j\in I$ are
available prior to solving ${\bf P_{c}(u)}$. It is sufficient to show how
$\nabla\bar{F}_{0}(u)$ could be calculated with the help of adjoint equations
since the adjoint equations for the other functionals will differ only by
terminal conditions.

We formulate the adjoint equations for two cases, they are derived on the
basis of a variational approach presented in [4] and by taking into account
our results on solutions to linearized equations of ordinary differential
equations and differential–algebraic equations of the special form arising
when a system remains in the sliding mode.

Case 1). In the first case we assume that in the time interval $[t_{0},t_{t}]$
the system evolves according to

(77) $x^{\prime}=f_{1}(x,u).$

At a transition time $t_{t}$ the continuous state trajectory meets the
switching surface such that the following condition holds

(78) $h(x(t_{t}))=0.$

After the transition the system evolves according to DAEs

(79) $\displaystyle x^{\prime}$ $\displaystyle=$ $\displaystyle
f_{F}(x,u)+h_{x}^{T}(x)z$ (80) $\displaystyle 0$ $\displaystyle=$
$\displaystyle h(x)$

up to an ending time $t_{f}$.

To derive the adjoint equations we construct the following augmented
functional

$\displaystyle{\displaystyle\Phi(x,z,u,\lambda,\pi)=\phi(x(t_{f}))+\pi
h(x(t_{t}))+\int_{t_{0}}^{t_{t}^{-}}\lambda_{f}^{T}(t)\left(x^{\prime}(t)-f_{1}(x(t),u(t))\right)dt}$
$\displaystyle{\displaystyle+\int_{t_{t}^{+}}^{t_{f}}\left[\lambda_{f}^{T}(t)\left(x^{\prime}(t)-f_{F}(x(t),u(t))-h_{x}^{T}(x(t))z(t)\right)+\lambda_{h}^{T}(t)h(x(t))\right]dt.}$

Suppose now that we calculate the variation of the augmented functional with
respect to $(x,z)$ only.

$\displaystyle{\displaystyle
d\Phi(x,z,u,\lambda,\pi)=\phi_{x}(x(t_{f}))y^{x,d}(t_{f})+\pi
h_{x}(x(t_{t}))dx(t_{t})+}$
$\displaystyle{\displaystyle\lambda_{f}^{T}(t_{t}^{-})\left(x^{\prime}(t_{t}^{-})-f_{1}(x(t_{t}^{-}),u(t_{t}^{-}))\right)dt_{t}}$
$\displaystyle{\displaystyle+d\left[\int_{t_{0}}^{t_{t}^{-}}\lambda_{f}^{T}(t)\left(x^{\prime}(t)-f_{1}(x(t),u(t))\right)dt\right]}$
$\displaystyle{\displaystyle-\lambda_{f}^{T}(t_{t}^{+})\left(x^{\prime}(t_{t}^{+})-f_{F}(x(t_{t}^{+}),u(t_{t}^{+}))-h_{x}^{T}(x(t_{t}^{+}))z(t_{t}^{+})\right)dt_{t}-\lambda_{h}^{T}(t_{t}^{+})h(x(t_{t}^{+}))dt_{t}}$
$\displaystyle{\displaystyle+d\left[\int_{t_{t}^{+}}^{t_{f}}\left[\lambda_{f}^{T}(t)\left(x^{\prime}(t)-f_{F}(x(t),u(t))-h_{x}^{T}(x(t))z(t)\right)+\lambda_{h}^{T}(t)h(x(t))\right]dt\right].}$

By integrating by parts the formulas $\int\lambda_{f}(t)x(t)dt$ (since
$\lambda_{f}$ is sufficiently smooth) we obtain

$\displaystyle{\displaystyle
d\Phi(x,z,u,\lambda,\pi)=\phi_{x}(x(t_{f}))y^{x,d}(t_{f})+\pi
h_{x}(x(t_{t}))dx(t_{t})}$
$\displaystyle{\displaystyle+\lambda_{f}^{T}(t_{t}^{-})\left(x^{\prime}(t_{t}^{-})-f_{1}(x(t_{t}^{-}),u(t_{t}^{-}))\right)dt_{t}+d\left[\left[\lambda_{f}^{T}(t)x(t)\right]_{t_{0}}^{t_{t}^{-}}\right]-}$
$\displaystyle{\displaystyle
d\left[\int_{t_{0}}^{t_{t}^{-}}\left[(\lambda_{f}^{T})^{\prime}(t)x(t)+\lambda_{f}^{T}(t)f_{1}(x(t),u(t))\right]dt\right]}$
$\displaystyle{\displaystyle-\lambda_{f}^{T}(t_{t}^{+})\left(x^{\prime}(t_{t}^{+})-f_{F}(x(t_{t}^{+}),u(t_{t}^{+}))-h_{x}^{T}(x(t_{t}^{+}))z(t_{t}^{+})\right)dt_{t}}$
$\displaystyle{\displaystyle-\lambda_{h}^{T}(t_{t}^{+})h(x(t_{t}^{+}))dt_{t}+d\left[\left[\lambda_{f}^{T}(t)x(t)\right]_{t_{t}^{+}}^{t_{f}}\right]-}$
$\displaystyle{\displaystyle
d\left[\int_{t_{t}^{+}}^{t_{f}}\left[(\lambda_{f}^{T})^{\prime}(t)x(t)+\lambda_{f}^{T}(t)\left(f_{F}(x(t),u(t))+h_{x}^{T}(x(t))z(t)\right)-\lambda_{h}^{T}(t)h(x(t))\right]dt\right].}$

Expanding further the variations (also with respect to u) and taking into
account the initial conditions of the linearized equations we obtain

$\displaystyle{\displaystyle
d\Phi(x,z,u,\lambda,\pi)=\phi_{x}(x(t_{f}))y^{x,d}(t_{f})+\pi
h_{x}(x(t_{t}))dx(t_{t})+}$
$\displaystyle{\displaystyle\lambda_{f}^{T}(t_{t}^{-})x^{\prime}(t_{t}^{-})dt_{t}-\lambda_{f}^{T}(t_{t}^{-})f_{1}(x(t_{t}^{-}),u(t_{t}^{-}))dt_{t}+\lambda_{f}^{T}(t_{t}^{-})y^{x,d}(t_{t}^{-})}$
$\displaystyle{\displaystyle-\int_{t_{0}}^{t_{t}^{-}}\left[(\lambda_{f}^{T})^{\prime}(t)y^{x,d}(t)+\lambda_{f}^{T}(t)(f_{1})_{x}(x(t),u(t))y^{x,d}(t)\right.}$
$\displaystyle{\displaystyle\left.+\lambda_{f}^{T}(t)(f_{1})_{u}(x(t),u(t))d(t)\right]dt-\lambda_{f}^{T}(t_{t}^{+})x^{\prime}(t_{t}^{+})dt_{t}+}$
$\displaystyle{\displaystyle\lambda_{f}^{T}(t_{t}^{+})f_{F}(x(t_{t}^{+}),u(t_{t}^{+}))dt_{t}+\lambda_{f}^{T}(t_{t}^{+})h_{x}^{T}(x(t_{t}^{+}))z(t_{t}^{+})dt_{t}-}$
$\displaystyle{\displaystyle\lambda_{h}^{T}(t_{t}^{+})h(x(t_{t}^{+}))dt_{t}+\lambda_{f}^{T}(t_{f})y^{x,d}(t_{f})-\lambda_{f}^{T}(t_{t}^{+})y^{x,d}(t_{t}^{+})-}$
$\displaystyle{\displaystyle\int_{t_{t}^{+}}^{t_{f}}\left[(\lambda_{f}^{T})^{\prime}(t)y^{x,d}(t)+\lambda_{f}^{T}(t)(f_{F})_{x}(x(t),u(t))y^{x,d}(t)\right.}$
$\displaystyle{\displaystyle+\lambda_{f}^{T}(t)(f_{F})_{u}(x(t),u(t))d(t)+\lambda_{f}^{T}(t)h_{x}^{T}(x(t))y^{z,d}(t)+}$
$\displaystyle{\displaystyle\left.\lambda_{f}^{T}(t)(h_{x}^{T}(x(t))z(t))_{x}y^{x,d}(t)-\lambda_{h}^{T}(t)h_{x}(x(t))y^{x,d}(t)\right]dt.}$

Now, if we take into account the formula (35)–(36) for the differential of
$x(t_{t})$, and rearrange the components with respect to differentials
$dx(t_{t}),\ dt_{t}$ and variations $y^{x,d}(t_{f}),\ y^{x,d}(t),\
y^{z,d}(t),\ d(t)$

$\displaystyle{\displaystyle
d\Phi(x,z,u,\lambda,\pi)=\left(\phi_{x}(x(t_{f}))+\lambda_{f}^{T}(t_{f})\right)y^{x,d}(t_{f})+\left(\pi
h_{x}(x(t_{t}))+\right.}$
$\displaystyle{\displaystyle\left.\lambda_{f}^{T}(t_{t}^{-})-\lambda_{f}^{T}(t_{t}^{+})\right)dx(t_{t})+\left(-\lambda_{f}^{T}(t_{t}^{-})f_{1}(x(t_{t}^{-}),u(t_{t}^{-}))+\right.}$
$\displaystyle{\displaystyle\left.\lambda_{f}^{T}(t_{t}^{+})f_{F}(x(t_{t}^{+}),u(t_{t}^{+}))+\lambda_{f}^{T}(t_{t}^{+})h_{x}^{T}(x(t_{t}^{+}))z(t_{t}^{+})-\lambda_{h}^{T}(t_{t}^{+})h(x(t_{t}^{+}))\right)dt_{t}}$
$\displaystyle{\displaystyle-\int_{t_{0}}^{t_{t}^{-}}\left((\lambda_{f}^{T})^{\prime}(t)+\lambda_{f}^{T}(t)(f_{1})_{x}(x(t),u(t))y^{x,d}(t)+\lambda_{f}^{T}(t)(f_{1})_{u}(x(t),u(t))d(t)\right)dt}$
$\displaystyle{\displaystyle-\int_{t_{t}^{+}}^{t_{f}}\left[\left((\lambda_{f}^{T})^{\prime}(t)+\lambda_{f}^{T}(t)(f_{F})_{x}(x(t),u(t))+\lambda_{f}^{T}(t)(h_{x}^{T}(x(t))z(t))_{x}-\right.\right.}$
$\displaystyle{\displaystyle\left.\left.\lambda_{h}^{T}(t)h_{x}(x(t))\right)y^{x,d}(t)+\lambda_{f}^{T}(t)(f_{F})_{u}(x(t),u(t))d(t)+\lambda_{f}^{T}(t)h_{x}^{T}(x(t))y^{z,d}(t)\right]dt.}$

We can now state conditions for adjoint equations such that the expressions
with differentials $dx(t_{t}),\ dt_{t}$ and variations $y^{x,d}(t_{f}),\
y^{x,d}(t),\ y^{z,d}(t)$ disappear and only the coefficients with variation
$d(t)$ remain. These components will state a formula to efficiently calculate
the gradient of $\phi(x(t_{f}))$ with respect to controls.

Let us start with components that are present under the integrals. We want to
zero the following components

$\displaystyle{\displaystyle\left((\lambda_{f}^{T})^{\prime}(t)+\lambda_{f}^{T}(t)(f_{1})_{x}(x(t),u(t))\right)y^{x,d}(t),\
t\in[t_{0},t_{t}^{-}],}$
$\displaystyle{\displaystyle\left((\lambda_{f}^{T})^{\prime}(t)+\lambda_{f}^{T}(t)(f_{F})_{x}(x(t),u(t))+\right.}$
$\displaystyle{\displaystyle\left.\lambda_{f}^{T}(t)(h_{x}^{T}(x(t))z(t))_{x}-\lambda_{h}^{T}(t)h_{x}(x(t))\right)y^{x,d}(t),\
t\in[t_{t}^{+},t_{f}],}$
$\displaystyle{\displaystyle\lambda_{f}^{T}(t)h_{x}^{T}(x(t))y^{z,d}(t),\
t\in[t_{t}^{+},t_{f}].}$

We achieve that by assuming that

(81)
$(\lambda_{f}^{T})^{\prime}(t)=-\lambda_{f}^{T}(t)(f_{1})_{x}(x(t),u(t)),\
t\in[t_{0},t_{t}^{-}]$

and

(82) $\displaystyle(\lambda_{f}^{T})^{\prime}(t)$ $\displaystyle=$
$\displaystyle-\lambda_{f}^{T}(t)(f_{F})_{x}(x(t),u(t))-\lambda_{f}^{T}(t)(h_{x}^{T}(x(t))z(t))_{x}$
$\displaystyle+\lambda_{h}(t)h_{x}(x(t))$ (83) $\displaystyle 0$
$\displaystyle=$ $\displaystyle\lambda_{f}^{T}(t)h_{x}^{T}(x(t)),\
t\in[t_{t}^{+},t_{f}].$

(81) is a system of ODEs with respect to the adjoint variable
$\lambda_{f}(t)$. (82)–(83) form a system of DAEs with respect to the adjoint
variables $\lambda_{f}(t)$ and $\lambda_{h}(t)$. The DAEs are of index 2 which
can be shown by considering the differentiation of (83)

$\displaystyle 0$ $\displaystyle=$
$\displaystyle\left(\lambda_{f}^{T}(t)h_{x}^{T}(x(t))\right)^{\prime}=(\lambda_{f}^{T})^{\prime}(t)h_{x}^{T}(x(t))+\lambda_{f}^{T}(t)\left(h_{x}^{T}(x(t))\right)^{\prime}$
$\displaystyle=$
$\displaystyle\left(-\lambda_{f}^{T}(t)(f_{F})_{x}(x(t),u(t))-\lambda_{f}^{T}(t)(h_{x}^{T}(x(t))z(t))_{x}\right.$
$\displaystyle\left.+\lambda_{h}(t)h_{x}(x(t))\right)h_{x}^{T}(x(t))+\lambda_{f}^{T}(t)\left(h_{x}^{T}(x(t))\right)^{\prime}$

The algebraic variable $\lambda_{h}^{T}(t)$ is multiplied by
$h_{x}(x(t))h_{x}^{T}(x(t))$ which is nonzero by the assumption (H1). This
means that we can evaluate $\lambda_{h}(t)$ by using the above equation which
proves that the DAEs are index 2 equations. One can also show that under (H1)
a solution to the equations (82)-(83), and to the equations (81), are such
that $\lambda_{f}$ is an absolutely continuous functions. This fact justifies
the use of the integration by parts formula in our derivations.

Let us consider the component

$\displaystyle{\displaystyle\left(\phi_{x}(x(t_{f}))+\lambda_{f}^{T}(t_{f})\right)y^{x,d}(t_{f}).}$

At the end of the time interval the trajectory evolves according to DAEs
(82)-(83). The trajectory, after variation of controls, stays within the
hyperplane defined by $h(x(t))=0$ which means that the variation
$y^{x,d}(t_{f})$ is always orthogonal to $h_{x}(x(t_{f}))$. Therefore, the
component disappears if the following condition is satisfied:

$\displaystyle{\displaystyle\phi_{x}(x(t_{f}))+\lambda_{f}^{T}(t_{f})=\nu
h_{x}(x(t_{f})).}$

for some real number $\nu$.

The DAEs (82)-(83) have to be consistently initialized. To provide the
consistent endpoint conditions for the adjoint variables $\lambda_{f},\
\lambda_{h}$ we follow the approach presented in [5] and solve the following
system of equations at time $t_{f}$ for the variables
$\lambda_{f},\lambda_{h},\nu$

(84) $\displaystyle\nu h_{x}^{T}(x(t_{f}))$ $\displaystyle=$
$\displaystyle\phi_{x}^{T}(x(t_{f}))+\lambda_{f}(t_{f})$ $\displaystyle 0$
$\displaystyle=$ $\displaystyle h_{x}(x(t_{f}))\lambda_{f}(t_{f})$ (85)
$\displaystyle 0$ $\displaystyle=$
$\displaystyle\left(h_{x}(x(t_{f}))\right)^{\prime}\lambda_{f}(t_{f})-h_{x}(x(t_{f}))\left(f_{F}\right)_{x}^{T}(x(t_{f}),u(t_{f}))\lambda_{f}(t_{f})-$
$\displaystyle
h_{x}(x(t_{f}))\left(h_{x}^{T}(x(t_{f}))z(t_{f})\right)_{x}^{T}\lambda_{f}(t_{f})+$
$\displaystyle h_{x}(x(t_{f}))h_{x}^{T}(x(t_{f}))\lambda_{h}(t_{f}).$

The solution to these equations is (due to (H1)):
$\nu=h_{x}\phi_{x}^{T}/\|h_{x}\|^{2}$,
$\lambda_{f}=h_{x}\phi_{x}^{T}h_{x}^{T}/\|h_{x}\|^{2}-\phi_{x}^{T}$,
$\lambda_{h}=\lambda_{f}^{T}\left(\left(f_{F}\right)_{x}+\left(h_{x}z\right)_{x}+h_{x}^{T}\right)/\|h_{x}\|^{2}$,
where all functions are evaluated at $(x(t_{f}),z(t_{f}),u(t_{f}))$.

At the transition time $t_{t}$ the adjoint variable $\lambda_{f}$ undergoes a
jump. To calculate the value of $\lambda_{f}(t_{t}^{-})$ the following system
of equations have to be solved for the variables $\lambda_{f}(t_{t}^{-}),\
\pi$

$\displaystyle\lambda_{f}(t_{t}^{-})$ $\displaystyle=$
$\displaystyle\lambda_{f}(t_{t}^{+})-\pi h_{x}^{T}(x(t_{t}))$ (86)
$\displaystyle\lambda_{f}^{T}(t_{t}^{-})f_{1}(x(t_{t}^{-}),u(t_{t}^{-}))$
$\displaystyle=$
$\displaystyle\lambda_{f}^{T}(t_{t}^{+})f_{F}(x(t_{t}^{+}),u(t_{t}^{+}))+\lambda_{f}^{T}(t_{t}^{+})h_{x}^{T}(x(t_{t}^{+}))z(t_{t}^{+})-$
$\displaystyle\lambda_{h}^{T}(t_{t}^{+})h(x(t_{t}^{+})).$

The solution to these equations is (due to (H2):
$\pi=((\lambda_{f}^{+})^{T}f_{1}(-)-(\lambda_{f}^{+})^{T}f_{F}(+)-$
$(\lambda_{f}^{+})^{T}h_{x}(+)z^{+}+\lambda_{h}^{+}h(+))/h_{x}(+)f_{1}(-)$,
$\lambda_{f}^{-}=\lambda_{f}^{+}-\pi h_{x}(+)$. where $\lambda_{f}^{+}$, etc.
is aa a function evaluated at $t_{t}^{+}$ (according to our notation (15)),
$\lambda_{f}^{-}$ at $t_{t}^{-}$, function $f_{F}(+)$, etc. is as evaluated at
$(x(t_{t}^{+}),u(t_{t}^{+}))$, function $f_{1}(-)$, etc. at
$(x(t_{t}^{-}),u(t_{t}^{-}))$.

Once we solve the adjoint equations we obtain the adjoint variables
$\lambda_{f}$ and $\lambda_{h}$. Eventually we are in a position to calculate
the first variation of the cost function $\phi(t_{f})$ with respect to a
control function variation $d$ as follows

$\displaystyle{\displaystyle
d\phi(x(t_{f}))=d\Phi(x,z,u,\lambda,\pi)=\left\langle\bar{F}_{0}(u),d\right\rangle=}$
$\displaystyle{\displaystyle-\int_{t_{0}}^{t_{t}^{-}}\lambda_{f}^{T}(t)(f_{1})_{u}(x(t),u(t))d(t)dt-\int_{t_{t}^{+}}^{t_{f}}\lambda_{f}^{T}(t)(f_{F})_{u}(x(t),u(t))d(t)dt.}$

##  5 The weak maximum principle

On the basis of the defined adjoint equations we can formulate the weak
maximum principle for the considered problem. Suppose that $\bar{u}$ is the
problem solution. Then, as shown in Theorem 3.1, $\bar{u}$ will satisfy
conditions (71)–(73) together with $g_{i}^{1}(x^{\bar{u}}(t_{f}))=0$, $i\in
E$, $g_{j}^{2}(x^{\bar{u}}(t_{f}))\leq 0$, $j\in I$.

The weak maximum principle for the problem (P) can assume a quite complicated
form depending on the number of switching points triggered by the optimal
control $\bar{u}$ In order to exemplify the possible conditions stated by the
weak maximum principle consider the case when optimal control $\bar{u}$ forces
the system behavior denoted as Case 1) in Section 4. We call the necessary
optimality conditions for that case ${\bf(NC^{13})}$

${\bf(NC^{13})}$: There exist: nonnegative numbers $\alpha_{j}^{2}$, $j\in I$,
numbers $\alpha_{i}^{1}$, $i\in E$ such that $\sum_{i\in
E}\left|\alpha_{i}^{1}\right|+\sum_{j\in I}\alpha_{j}^{2}\neq 0$; piecewise
differentiable function $\lambda_{f}$; piecewise continuous function
$\lambda_{h}$, such that the following hold:
(i) terminal conditions ($\lambda_{f}(t_{f})$, $\lambda_{h}(t_{f})$ and $\nu$
are evaluated from equations below)

$\displaystyle-\lambda_{f}(t_{f})$ $\displaystyle=$
$\displaystyle\phi_{x}^{T}(\bar{x}(t_{f}))+\nu
h_{x}^{T}(\bar{x}(t_{f}))+\sum_{i\in
E}\alpha_{i}^{1}\left(g_{i}^{1}\right)_{x}^{T}(\bar{x}(t_{f}))+\sum_{j\in
I}\alpha_{j}^{2}\left(g_{j}^{2}\right)_{x}^{T}(\bar{x}(t_{f}))$ $\displaystyle
0$ $\displaystyle=$ $\displaystyle h_{x}(\bar{x}(t_{f}))\lambda_{f}(t_{f})$
$\displaystyle 0$ $\displaystyle=$
$\displaystyle\left(h_{x}(\bar{x}(t_{f}))\right)^{\prime}\lambda_{f}(t_{f})-h_{x}(\bar{x}(t_{f}))\left(f_{F}\right)_{x}^{T}(\bar{x}(t_{f}),\bar{u}(t_{f}))\lambda_{f}(t_{f})-$
$\displaystyle
h_{x}(\bar{x}(t_{f}))\left(h_{x}^{T}(\bar{x}(t_{f}))\bar{z}(t_{f})\right)_{x}^{T}\lambda_{f}(t_{f})+h_{x}(\bar{x}(t_{f}))h_{x}^{T}(\bar{x}(t_{f}))\lambda_{h}(t_{f});$

(ii) adjoint equations
for $t\in[t_{t},t_{f})$

$\displaystyle\lambda_{f}^{\prime}$ $\displaystyle=$
$\displaystyle-\left(f_{F}\right)_{x}^{T}(\bar{x},\bar{u})\lambda_{f}-\left(h_{x}^{T}(\bar{x})z\right)_{x}^{T}\lambda_{f}+h_{x}^{T}(\bar{x})\lambda_{h}$
$\displaystyle 0$ $\displaystyle=$ $\displaystyle h_{x}(\bar{x})\lambda_{f}$

for $t\in[t_{0},t_{t})$

$\displaystyle\lambda_{f}^{\prime}=-\left(f_{1}\right)_{x}^{T}(\bar{x},\bar{u})\lambda_{f};$

(iii) jump conditions ($\lambda_{f}(t_{t}^{-})$ and $\pi$ are evaluated from
the equations below)

$\displaystyle\lambda_{f}(t_{t}^{-})$ $\displaystyle=$
$\displaystyle\lambda_{f}(t_{t}^{+})-\pi h_{x}^{T}(\bar{x}(t_{t}^{-}))$
$\displaystyle\lambda_{f}^{T}(t_{t}^{-})f_{1}(\bar{x}(t_{t}^{-}),\bar{u}(t_{t}^{-}))$
$\displaystyle=$
$\displaystyle\lambda_{f}^{T}(t_{t}^{+})f_{F}(\bar{x}(t_{t}^{+}),\bar{u}(t_{t}^{+}))+$
$\displaystyle\lambda_{f}^{T}(t_{t}^{+})h_{x}^{T}(\bar{x}(t_{t}^{+}))\bar{z}(t_{t}^{+})-\lambda_{h}^{T}(t_{t}^{+})h(\bar{x}(t_{t}^{+}));$

(iv) the weak maximum principle
for $t\in[t_{t},t_{f})$

$\displaystyle\lambda_{f}^{T}(t)\left(f_{F}\right)_{u}(\bar{x}(t),\bar{u}(t))u\leq\lambda_{f}^{T}(t)\left(f_{F}\right)_{u}(\bar{x}(t),\bar{u}(t))\bar{u}(t)$

for $t\in[t_{0},t_{t}]$

$\displaystyle\lambda_{f}^{T}(t)\left(f_{1}\right)_{u}(\bar{x}(t),\bar{u}(t))u\leq\lambda_{f}^{T}(t)\left(f_{1}\right)_{u}(\bar{x}(t),\bar{u}(t))\bar{u}(t)$

for all $u\in U$;
(v) complementarity conditions

$\displaystyle\alpha_{j}^{2}=0,\ \ {\rm if}\ \ j\not\in I_{0,\bar{u}}.$

The necessary optimality conditions ${\bf(NC^{13})}$ are stated for the case
when the optimal trajectory $(\bar{x},\bar{z})$ corresponding to the optimal
control $\bar{u}$ is such that to the switching point $t_{t}$ the system
evolution is according to $f_{1}$, and then the movement takes on the sliding
surface until $t_{f}$. For that case we can prove the theorem which is the
restatement of part iii) of Theorem 3.1.

######  Theorem 5.1.

Assume that data for (P) satisfies hypotheses ${\bf(H1)}$, ${\bf(H2)}$,
${\bf(H3)}$, ${\bf(H4)}$ and ${\bf(CQ)}$. Let $\\{u_{k}\\}$ be a sequence of
controls generated by Algorithm and let $\\{c_{k}\\}$ be a sequence of the
corresponding penalty parameters. Then

  1. iii)

if $\bar{u}$ is the limit point of $\\{u_{k}\\}$, $\bar{x}$ its state
corresponding trajectory and suppose that for $\bar{u}$ Case 1) applies, then
the pair $(\bar{x},\bar{u})$ satisfies the necessary optimality conditions
${\bf(NC^{13})}$.

###### Proof.

As shown in the proof of Theorem 5.1 (Stage 2–Dualization) in [17] the
conclusions of the proof of Theorem 3.1 can be expressed by

$\displaystyle{\displaystyle\min_{d\in{\mathcal{D}}_{\bar{u}}}\max_{\gamma\in{\mathcal{K}}}\Psi(d,\gamma)=0}$

where

$\displaystyle{\displaystyle{\mathcal{K}}=\left\\{\gamma=\left(\alpha_{0},\\{\alpha^{1}_{i}\\}_{i\in
E},\\{\alpha^{2}_{j}\\}_{j\in I}\right)\in\mathbb{R}^{1+|E|+|I|}:\
\alpha_{0}\geq 0,\ \alpha^{2}_{j}\geq 0,\ j\in I,\right.}$
$\displaystyle{\displaystyle\left.\ \alpha_{0}+\sum_{i\in
E}\left|\alpha^{1}_{i}\right|+\sum_{j\in I}\alpha^{2}_{j}=1,\
\alpha^{2}_{j}=0\ \ {\rm if}\ j\not\in I_{0,\bar{u}}\right\\}}$

and

$\displaystyle{\displaystyle\Psi(d,\gamma):=\alpha_{0}\left\langle\nabla\bar{F}_{0}(\bar{u}),d\right\rangle+c\left(\sum_{i\in
E}\alpha^{1}_{i}\left\langle\nabla\bar{g}^{1}_{i}(\bar{u}),d\right\rangle+\sum_{j\in
I_{0,\bar{u}}}\alpha^{2}_{j}\left\langle\nabla\bar{g}^{2}_{j}(\bar{u}),d\right\rangle\right).}$

$\Psi(\cdot,\gamma)$ is a linear function on ${\mathcal{C}}^{N}_{m}[T]$ of
which ${\mathcal{D}}_{\bar{u}}$ is a convex subset. $\Psi(d,\cdot)$ is a
bounded linear map and ${\mathcal{K}}$ is a compact convex set with respect to
the product topology of $\mathbb{R}^{1+|E|+|I|}$. It follows from the minimax
theorem ([2]) that there exists some nonzero $\bar{\gamma}\in{\mathcal{K}}$
such that

(87)
$\displaystyle{\displaystyle\min_{d\in{\mathcal{D}}_{\bar{u}}}\max_{\gamma\in{\mathcal{K}}}\Psi(d,\gamma)=\min_{d\in{\mathcal{D}}_{\bar{u}}}\Psi(d,\bar{\gamma})=0,}$

with $\bar{\gamma}=(\bar{\alpha}_{0},\\{\bar{\alpha}^{1}_{i}\\}_{i\in
E},\\{\bar{\alpha}^{2}_{j}\\}_{j\in I})$.

Since the constraint qualification ${\bf(CQ)}$ holds we can easily show that
$\bar{\alpha}_{0}\neq 0$.

The adjoint equations have been derived for the functional $\bar{F}_{0}(u)$,
however similar analysis could be carried out for the functional

$\displaystyle{\displaystyle H(u)=\bar{F}_{0}(u)+c\left(\sum_{i\in
E}\bar{\alpha}_{i}^{1}\bar{g}_{i}^{1}(u)+\sum_{j\in
I}\bar{\alpha}_{j}^{2}\bar{g}_{j}^{2}(\bar{u})\right),}$

then by taking $\alpha_{i}^{1}=c\bar{\alpha}_{i}^{1}$, $i\in E$,
$\alpha_{j}^{2}=c\bar{\alpha}_{j}^{2}$, $j\in I$ (notice that $c>0$).

The other cases than Case 1) can be analyzed, with respect to necessary
optimality conditions, in a similar way. We do not enumerate these cases
because they are numerous, e.g.: the system behaves first according to
function $f_{2}$, then, until the end it is in the sliding mode; the system
starts in the sliding mode before moving to the mode with dynamics described
by $f_{1}$ or $f_{2}$; there are several discrete modes the system assumes on
the interval $[t_{0},t_{f}]$.

In particular, in Case 2)—the system starts in the sliding mode and at the
time $t_{t}$ it switches to the discrete state $q=1$, the necessary optimality
conditions will shape as follows (we assume that the switching function for
leaving the sliding mode is given by $\eta(x,u)$).

${\bf(NC^{31})}$: There exist: nonnegative numbers $\alpha_{j}^{2}$, $j\in I$,
numbers $\alpha_{i}^{1}$, $i\in E$ such that $\sum_{i\in
E}\left|\alpha_{i}^{1}\right|+\sum_{j\in I}\alpha_{j}^{2}\neq 0$; piecewise
smooth function $\lambda_{f}$; piecewise continuous function $\lambda_{h}$,
such that the following hold:
(i) terminal conditions

$\displaystyle{\displaystyle-\lambda_{f}(t_{f})=\phi_{x}^{T}(\bar{x}(t_{f}))+\sum_{i\in
E}\alpha_{i}^{1}\left(g_{i}^{1}\right)_{x}^{T}(\bar{x}(t_{f}))+\sum_{j\in
I}\alpha_{j}^{2}\left(g_{j}^{2}\right)_{x}^{T}(\bar{x}(t_{f}))}$

(ii) adjoint equations
for $t\in[t_{t},t_{f})$

$\displaystyle\lambda_{f}^{\prime}=-\left(f_{1}\right)_{x}^{T}(\bar{x},\bar{u})\lambda_{f};$

for $t\in[t_{0},t_{t})$

$\displaystyle\lambda_{f}^{\prime}$ $\displaystyle=$
$\displaystyle-\left(f_{F}\right)_{x}^{T}(\bar{x},\bar{u})\lambda_{f}-\left(h_{x}^{T}(\bar{x})z\right)_{x}^{T}\lambda_{f}+h_{x}^{T}(\bar{x})\lambda_{h}$
$\displaystyle 0$ $\displaystyle=$ $\displaystyle h_{x}(\bar{x})\lambda_{f}$

(iii) jump conditions ($\lambda_{f}(t_{t}^{-})$, $\nu_{t}$ and $\pi$ are
evaluated from the equations below)

(88)
$\displaystyle\lambda_{f}(t_{t}^{-})-\lambda_{f}(t_{t}^{+})+\pi\eta_{x}^{T}(x(t_{t}),u(t_{t}))$
$\displaystyle=$ $\displaystyle\nu_{t}h_{x}^{T}(x(t_{t}^{-}))$
$\displaystyle\lambda_{f}^{T}(t_{t}^{-})f_{F}(x(t_{t}^{-}),u(t_{t}^{-}))+\lambda_{f}^{T}(t_{t}^{-})h_{x}^{T}(x(t_{t}^{-}))z(t_{t}^{-})$
(89)
$\displaystyle-\pi\eta_{u}(x(t_{t}),u(t_{t}))u^{\prime}(t_{t}^{-})-\lambda_{h}^{T}(t_{t}^{-})h(x(t_{t}^{-}))$
$\displaystyle=$
$\displaystyle\lambda_{f}^{T}(t_{t}^{+})f_{1}(x(t_{t}^{+}),u(t_{t}^{+}))$

and

(90) $\displaystyle 0$ $\displaystyle=$ $\displaystyle
h_{x}(x(t_{t}^{-}))\lambda_{f}(t_{t}^{-})$ (91) $\displaystyle 0$
$\displaystyle=$
$\displaystyle\left(h_{x}(x(t_{t}^{-}))\right)^{\prime}\lambda_{f}(t_{t}^{-})-$
$\displaystyle
h_{x}(x(t_{t}^{-}))\left(f_{F}\right)_{x}^{T}(x(t_{t}^{-}),u(t_{t}^{-}))\lambda_{f}(t_{t}^{-})-$
$\displaystyle
h_{x}(x(t_{t}^{-}))\left(h_{x}^{T}(x(t_{t}^{-}))z(t_{t}^{-})\right)_{x}^{T}\lambda_{f}(t_{t}^{-})+$
$\displaystyle
h_{x}(x(t_{t}^{-}))h_{x}^{T}(x(t_{t}^{-}))\lambda_{h}(t_{t}^{-}).$

(iv) the weak maximum principle
for $t\in[t_{t},t_{f})$

$\displaystyle\lambda_{f}^{T}(t)\left(f_{1}\right)_{u}(\bar{x}(t),\bar{u}(t))u\leq\lambda_{f}^{T}(t)\left(f_{1}\right)_{u}(\bar{x}(t),\bar{u}(t))\bar{u}(t)$

for $t=t_{t}$

$\displaystyle{\displaystyle\left(\lambda_{f}^{T}(t_{t}^{-})\left(f_{F}\right)_{u}(\bar{x}(t_{t}^{-}),\bar{u}(t_{t}^{-}))+\pi\eta_{u}(x(t_{t}^{-}),u(t_{t}^{-}))\right)u\leq}$
$\displaystyle\left(\lambda_{f}^{T}(t_{t}^{-})\left(f_{F}\right)_{u}(\bar{x}(t_{t}^{-}),\bar{u}(t_{t}^{-}))+\pi\eta_{u}(x(t_{t}^{-}),u(t_{t}^{-}))\right)\bar{u}(t_{t}^{-})$

for $t\in[t_{0},t_{t})$

$\displaystyle\lambda_{f}^{T}(t)\left(f_{F}\right)_{u}(\bar{x}(t),\bar{u}(t))u\leq\lambda_{f}^{T}(t)\left(f_{F}\right)_{u}(\bar{x}(t),\bar{u}(t))\bar{u}(t)$

for all $u\in U$;
(v) complementarity conditions

$\displaystyle\alpha_{j}^{2}=0,\ \ {\rm if}\ \ j\not\in I_{0,\bar{u}}.$

##  6 Hybrid systems without sliding motion

The process of finding optimal solutions to control problems with hybrid
systems significantly simplifies when trajectories generated during the
process do contain sections with sliding modes. In this case we assume that
our state trajectory on the entire horizon $[t_{0},t_{f}]$ will consist of
trajectories of type of type $x$ only, i.e. we have

$\displaystyle{\displaystyle[t_{0},t_{f}]=\cup_{i\in I_{1}}A^{1}_{i}\cup_{i\in
I_{2}}A^{2}_{i}.}$

For the simplicity of presentation consider the case when the system on time
subinterval $[t_{0},t_{t})$ evolves according to equations

(92) $\displaystyle{\displaystyle x^{\prime}=f_{1}(x,u),}$

and then, on the subinterval $[t_{t},t_{f}]$, according to

(93) $\displaystyle{\displaystyle x^{\prime}=f_{2}(x,u),}$

Due to our previous remarks we do not have to restrict the class of admissible
control functions to piecewise smooth functions, and we can can take as
${\mathcal{U}}$ the set

(94)
$\displaystyle{\displaystyle{\mathcal{U}}=\left\\{u\in{\mathcal{L}}_{m}^{2}[T]:\
u(t)\in U,\ {\rm a.e.\ on}\ T\right\\},}$

For this set of admissible controls, and due to our assumption that sliding
mode does not occur, the relations (31)–(34) hold for $x$, $x^{d}$ and
$y^{x,d}$ on the subintervals $[t_{0},t_{t}]$ and $[t_{t},t_{f}]$.

The adjoint equations for the equations (92)–(93) can be derived in a way
described in Section 4. To this end we define the functional

$\displaystyle{\displaystyle\Phi(x,z,u,\lambda,\pi)=\phi(x(t_{f}))+\pi
h(x(t_{t}))+}$
$\displaystyle{\displaystyle\int_{t_{0}}^{t_{t}^{-}}\lambda_{f}^{T}(t)\left(x^{\prime}(t)-f_{1}(x(t),u(t))\right)dt+\int_{t_{t}^{+}}^{t_{f}}\lambda_{f}^{T}(t)\left(x^{\prime}(t)-f_{2}(x(t),u(t))\right)dt,}$

and after applying arguments as in Section 4, we arrive at the equations for
adjoint variable $\lambda_{f}$:

(95)
$(\lambda_{f}^{T})^{\prime}(t)=-\lambda_{f}^{T}(t)(f_{1})_{x}(x(t),u(t)),\
t\in[t_{0},t_{t}]$

and

(96)
$\displaystyle{\displaystyle(\lambda_{f}^{T})^{\prime}(t)=\lambda_{f}^{T}(t)(f_{2})_{x}(x(t),u(t)),\
t\in(t_{t},t_{f}].}$

The transversality condition for the equations (95)–(96) is

(97) $\displaystyle{\displaystyle\phi_{x}^{T}(x(t_{f}))+\lambda_{f}(t_{f})=0}$

and the jump conditions at the switching time $t_{t}$ are

$\displaystyle\lambda_{f}(t_{t}^{-})$ $\displaystyle=$
$\displaystyle\lambda_{f}(t_{t}^{+})-\pi h_{x}^{T}(x(t_{t}))$ (98)
$\displaystyle\lambda_{f}^{T}(t_{t}^{-})f_{1}(x(t_{t}^{-}),u(t_{t}^{-}))$
$\displaystyle=$
$\displaystyle\lambda_{f}^{T}(t_{t}^{+})f_{2}(x(t_{t}^{+}),u(t_{t}^{+})).$

Then the first variation of the cost function $\phi(t_{f})$ with respect to a
control function variation $d$ is follows

$\displaystyle{\displaystyle d\phi(x(t_{f}))=d\Phi(x,z,u,\lambda,\pi)=}$
$\displaystyle{\displaystyle-\int_{t_{0}}^{t_{t}^{-}}\lambda_{f}^{T}(t)(f_{1})_{u}(x(t),u(t))d(t)dt-\int_{t_{t}^{+}}^{t_{f}}\lambda_{f}^{T}(t)(f_{2})_{u}(x(t),u(t))d(t)dt.}$

Having adjoint equations for the simplest possible case of control problems
which hybrid systems do not exhibit sliding modes we can formulate necessary
optimality conditions for the case—${\bf(NC^{12})}$.

${\bf(NC^{12})}$: There exist: nonnegative numbers $\alpha_{j}^{2}$, $j\in I$,
numbers $\alpha_{i}^{1}$, $i\in E$ such that $\sum_{i\in
E}\left|\alpha_{i}^{1}\right|+\sum_{j\in I}\alpha_{j}^{2}\neq 0$; number
$\pi$; absolutely continuous function $\lambda_{f}$, such that the following
hold:
(i) terminal conditions

$\displaystyle\lambda_{f}(t_{f})$ $\displaystyle=$
$\displaystyle-\phi_{x}^{T}(\bar{x}(t_{f}))$

(ii) adjoint equations
for $t\in(t_{t},t_{f})$

$\displaystyle\lambda_{f}^{\prime}$ $\displaystyle=$
$\displaystyle-\left(f_{2}\right)_{x}^{T}(\bar{x},\bar{u})\lambda_{f}$

for $t\in[t_{0},t_{t}]$

$\displaystyle\lambda_{f}^{\prime}=-\left(f_{1}\right)_{x}^{T}(\bar{x},\bar{u})\lambda_{f};$

(iii) jump conditions ($\lambda_{f}(t_{t}^{-})$ and $\pi$ are the solutions to
the algebraic equations stated below)

$\displaystyle\lambda_{f}(t_{t}^{-})$ $\displaystyle=$
$\displaystyle\lambda_{f}(t_{t}^{+})-\pi h_{x}^{T}(\bar{x}(t_{t}^{-}))$
$\displaystyle\lambda_{f}^{T}(t_{t}^{-})f_{1}(\bar{x}(t_{t}^{-}),\bar{u}(t_{t}^{-}))$
$\displaystyle=$
$\displaystyle\lambda_{f}^{T}(t_{t}^{+})2_{F}(\bar{x}(t_{t}^{+}),\bar{u}(t_{t}^{+}))$

(iv) the weak maximum principle
for $t\in(t_{t},t_{f})$

$\displaystyle\lambda_{f}^{T}(t)\left(f_{2}\right)_{u}(\bar{x}(t),\bar{u}(t))u\leq\lambda_{f}^{T}(t)\left(f_{2}\right)_{u}(\bar{x}(t),\bar{u}(t))\bar{u}(t)$

for $t\in[t_{0},t_{t}]$

$\displaystyle\lambda_{f}^{T}(t)\left(f_{1}\right)_{u}(\bar{x}(t),\bar{u}(t))u\leq\lambda_{f}^{T}(t)\left(f_{1}\right)_{u}(\bar{x}(t),\bar{u}(t))\bar{u}(t)$

for all $u\in U$;
(v) complementarity conditions

$\displaystyle\alpha_{j}^{2}=0,\ \ {\rm if}\ \ j\not\in I_{0,\bar{u}}.$

Since we elaborate on the case of optimal control problems with hybrid systems
which do not involve sliding modes we have to introduce the assumption which
excludes the possibility of system going into the sliding mode. The assumption
concerns switching points $t_{t}$ and is postulated instead of ${\bf(H2)}$.

${\bf(H2^{\prime})}$

Function $h(\cdot)$ is differentiable and there exist $0<L_{1}<+\infty$ and
$0<L_{2}<+\infty$ such that

(99) $\displaystyle{\displaystyle\left\|h_{x}(\hat{x})-h_{x}(x)\right\|\leq
L_{1}\|\hat{x}-x\|,}$ (100)
$\displaystyle{\displaystyle\left|h_{x}(\hat{x})f_{i}(\hat{x},\hat{u})-h_{x}(x)f_{i}(x,u)\right|\leq
L_{2}\|(\hat{x},\hat{u})-(x,u)\|,}$

for all $(x,u)$, $(\hat{x},\hat{u})$ in $\mathbb{R}^{n}\times U$.

For any $u\in{\mathcal{U}}$ and any switching point $t_{t}$ the following
limits exist

(101) $\displaystyle{\displaystyle\lim_{t\rightarrow t_{t},t<t_{t}}u(t),\
\lim_{t\rightarrow t_{t},t>t_{t}}u(t)}$

(and are denoted by $u(t_{t}^{-})$ and $u(t_{t}^{+})$ respectively).

There exists $\varepsilon>0$ and $0<L_{3}<+\infty$ such that for all switching
points $t_{t}$ (each switching time corresponds to some $u\in{\mathcal{U}}$)
and for all their perturbations $t_{t}^{d}$ triggered by perturbations $d$
such that $u+d\in{\mathcal{U}}$, $\|d\|_{{\mathcal{L}}^{2}}\leq\varepsilon$,
and for all $\theta\in[0,1]$ we have

(102) $\displaystyle{\displaystyle
h_{x}(x(\tau^{\theta,d}(t_{t})))f_{i}(x(\tau^{\theta,d}(t_{t})),u(\tau^{\theta,d}(t_{t})))\geq
L_{3},}$

or

(103) $\displaystyle{\displaystyle
h_{x}(x(\tau^{\theta,d}(t_{t})))f_{i}(x(\tau^{\theta,d}(t_{t})),u(\tau^{\theta,d}(t_{t})))\leq-
L_{3},}$

for $i=1,2$, where

(104) $\displaystyle\tau^{\theta,d}(t_{t})$ $\displaystyle=$ $\displaystyle
t_{t}+\theta(t^{d}_{t}-t_{t}).$

The hypothesis (H2’) which replaces (H2) does not request the condition (49)
for perturbations $d$. This is the consequence of having the switching
function $\eta=h$ which does not depend on controls. That simplification
spreads through the definitions which concern perturbations $d$, for example,
the definitions of ${\mathcal{D}}_{u}$ and ${\mathcal{D}}$, which should be
updated in order to work with the broader sets of perturbations, do not
contain the requirement (49). Theorem 6.1 can be proved along the lines of the
proof of Theorem 3.1, Theorem 5.1 and with sets ${\mathcal{D}}_{u}$,
${\mathcal{D}}$ modified in this way.

######  Theorem 6.1.

Assume that data for (P) satisfies hypotheses ${\bf(H1)}$,
${\bf(H2^{\prime})}$, ${\bf(H3)}$, ${\bf(H4)}$ and ${\bf(CQ)}$. Let
$\\{u_{k}\\}$ be a sequence of controls generated by Algorithm and let
$\\{c_{k}\\}$ be a sequence of the corresponding penalty parameters. Then

  1. i)

$\\{c_{k}\\}$ is a bounded sequence

  2. ii)

$\displaystyle{\displaystyle\lim_{k\rightarrow\infty}M(u_{k})=0.}$

  3. iii)

if $\bar{u}$ is the limit point of $\\{u_{k}\\}$, $\bar{x}$ its state
corresponding trajectory and suppose that for $\bar{u}$ the case of this
section applies, then the pair $(\bar{x},\bar{u})$ satisfies the necessary
optimality conditions ${\bf(NC^{12})}$.

##  7 Conclusions

The paper presents the computational approach to hybrid optimal control
problems with sliding modes. It seems to be the first method for optimal
control problems with hybrid systems which can exhibit sliding modes. We show
that under the assumption that controls are represented by piecewise constant
functions the presented method is globally convergent in the sense that every
accumulation point of a sequence generated by the method satisfies necessary
optimality conditions in the form of the weak maximum principle. In the
accompanying papers ([14],[15]) we present the implementable version of our
computational method by relaxing the condition that continuous state
trajectories are available. In these papers we assume that system equations
are integrated by an implicit Runge–Kutta method and we show that the global
convergence of the implementable version of our algorithm is preserved
provided that the integration procedure step sizes converge to zero.

The algorithms presented in the paper are first order methods since they are
based on linearizations of functions defining our optimal control problem. We
can provide second order versions of these algorithms by applying an SQP
approach in which second order terms in the functions expansions are placed in
a quadratic term of the objective function of ${\bf P_{c}(u)}$—then instead of
using the term $1/2\|d\|_{{\mathcal{L}}^{2}}$ we use $1/2d^{T}Hd$. $H$
represents the approximation to the Hessian of the Lagrange function of the
problem. Of course the second order approach can be applied if the optimal
control problem can be stated as the problem in the Euclidean space, for
example when control functions are piecewise constant functions ([12]). The
convergence analysis presented in this paper can be used to prove global
convergence of the second order method provided that matrices $H$ used at
iterations are uniformly positive definite and bounded.

##  Appendix A Proof of Lemma 3.1

###### Proof.

Take any $u\in{\cal U}$. Let $r>0$ be a number such that

$\displaystyle{\displaystyle M(u)<r}$

for all $u\in{\cal U}$. We deduce from ${\bf(CQ)}$ that there is a simplex in
${\cal E}(u)\subset{\cal R}^{n_{E}}$ ($n_{E}=|E|$) with vertices
$\\{e_{j}\\}_{j=0}^{n_{E}}$ which contains $0$ as an interior point. By
definition of ${\cal E}(u)$, there exist $d_{0},\ldots,d_{n_{E}}\in{\cal D}$
and $\delta>0$ such that for $j=0,\ldots,n_{E}$

$\displaystyle{\displaystyle\left\\{\left\langle\nabla\bar{g}^{1}_{i}(u),d_{j}\right\rangle\right\\}_{i\in
E}=e_{j},\ \ \max_{i\in
I}\left\langle\nabla\bar{g}^{2}_{i}(u),d_{j}\right\rangle\leq-\delta.}$

Let $(\lambda_{0},\lambda_{1},\ldots,\lambda_{n_{E}})$ be the barycentric
coordinates of $0$ w.r.t. the vertices $e_{j}$ of the simplex, i.e.

$\displaystyle{\displaystyle 0\
\left(=\sum_{j=0}^{n_{E}}\lambda_{j}e_{j}\right)=\nabla\bar{g}^{1}(u)\circ\sum_{j=0}^{n_{E}}\lambda_{j}d_{j}.}$

Here

$\displaystyle\nabla\bar{g}^{1}(u\circ d$ $\displaystyle:=$
$\displaystyle\left\\{\left\langle\nabla\bar{g}^{1}_{i}(u),d\right\rangle\right\\}_{i\in
E}.$

We shall also write

$\displaystyle\bar{g}^{1}(u)$ $\displaystyle:=$
$\displaystyle\left\\{\bar{g}^{1}_{i}(u)\right\\}_{i\in E}.$

Since the vertices are in general position and $0$ is an interior point, the
$\lambda_{i}$’s are all positive and we may find $\delta_{1}>0$ such that

$\displaystyle{\displaystyle\left(\lambda_{0}-\sum_{j=1}^{n_{E}}\alpha_{j},\lambda_{1}+\alpha_{1},\ldots,\lambda_{n_{E}}+\alpha_{n_{E}}\right)}$
$\displaystyle{\displaystyle\in\left\\{\gamma\in{\cal R}^{n_{E}+1}:\
\gamma_{j}\geq 0\ \forall j,\ \sum_{j=0}^{n_{E}}\gamma_{j}=1\right\\}}$

whenever $\alpha\in{\cal B}(0,\delta_{1})\subset{\cal R}^{n_{E}}$. (${\cal
B}(0,\delta_{1})$ is a ball with radius $\delta_{1}$.) Furthermore, the
$n_{E}\times n_{E}$ matrix $P(u)$ defined by

(105) $\displaystyle{\displaystyle
P(u)\alpha:=\sum_{j=1}^{n_{E}}\nabla\bar{g}^{1}(u)\circ\alpha_{j}(d_{j}-d_{0})}$

is invertible for $u=\tilde{u}$, from the definition of $d_{j},\
j=1,\ldots,n_{E}$.

In consequence of hypothesis ${\bf(CQ)}$ and in view of the continuity
properties of the mapping $u\rightarrow y^{x,d}$ for fixed $d$ we may choose a
neighborhood ${\cal O}(\tilde{u})$ of $\tilde{u}$ in ${\cal U}$ and numbers
$\hat{r}\geq r$ and $\delta_{2}\in(0,\hat{r}^{-1}]$ such that for any
$u\in{\cal U}$ satisfying $u\in{\cal O}(\tilde{u})$

$\displaystyle(i)\hskip 14.22636pt$ $\displaystyle\max_{i\in
I}\left\langle\nabla\bar{g}^{2}_{i}(u),v_{j}-u\right\rangle\leq-\delta/2\ \
\forall j$ $\displaystyle(ii)\hskip 14.22636pt$ $\displaystyle P(u)\ \ {\rm
is\ invertible}$ $\displaystyle(iii)\hskip 14.22636pt$
$\displaystyle\left\|P(u)^{-1}\nabla\bar{g}^{1}(u)\circ\left(\left(\sum_{j=0}^{n_{E}}\lambda_{j}v_{j}\right)-u\right)\right\|\leq\delta_{1}/2$
$\displaystyle(iv)\hskip 14.22636pt$
$\displaystyle\delta_{2}\left\|P(u)^{-1}\right\|n_{E}^{1/2}\leq\delta_{1}/2.$

(In (iv) the norm is the Frobenius norm. Here the controls $v_{j}\in{\cal U}$,
$j=0,\ldots,n_{E}$ are defined to be

$\displaystyle{\displaystyle v_{j}(t):=u(t)+d_{j}(t,u(t)).}$

Now suppose that the control $u$ is not feasible. Set

$\displaystyle{\displaystyle\alpha=P(u)^{-1}\left[-\nabla\bar{g}^{1}(u)\circ\left(\left(\sum_{j=0}^{n_{E}}\lambda_{j}v_{j}\right)-u\right)-\delta_{2}M(u)^{-1}\bar{g}^{1}(u)\right].}$
(106)

Notice that, by properties (iii) and (iv), $\|\alpha\|\leq\delta_{1}$. Also
set

$\displaystyle{\displaystyle\hat{v}=v_{0}+\sum_{j=1}^{n_{E}}(\lambda_{j}+\alpha_{j})(v_{j}-v_{0}).}$

Because $\|\alpha\|\leq\delta_{1}$ we have that $\hat{v}\in{\cal U}$. Finally
we define $v$ to be

$\displaystyle{\displaystyle v=u+\left(M(u)/\hat{r}\right)(\hat{v}-u).}$

Since $M(u)/\hat{r}\leq 1$, it follows that $v\in{\cal U}$. We now verify that
this control function has the required properties.

Notice first that

(107)
$\displaystyle{\displaystyle\left\|v-u\right\|_{{\mathcal{L}}^{\infty}}\leq(2d/\hat{r})M(u),}$

where $d$ is a bound on the norms of elements in ${\cal U}$.

We have from (105) and (106) that

$\displaystyle P(u)\alpha$ $\displaystyle=$
$\displaystyle\nabla\bar{g}^{1}(u)\circ\sum_{j=1}^{n_{E}}\alpha_{j}(v_{j}-v_{0})$
$\displaystyle=$
$\displaystyle-\nabla\bar{g}^{1}(u)\circ\left(\sum_{j=1}^{n_{E}}\lambda_{j}\left(v_{j}-v_{0}\right)+v_{0}-u\right)-\delta_{2}M(u)^{-1}\bar{g}^{1}(u).$

By definition of $\hat{v}$,

$\displaystyle{\displaystyle\nabla\bar{g}^{1}(u)\circ(\hat{v}-u)=-\delta_{2}M(u)^{-1}\bar{g}^{1}(u).}$

But then

$\displaystyle{\displaystyle\nabla\bar{g}^{1}(u)\circ(v-u)=-(\delta_{2}/\hat{r})\bar{g}^{1}(u).}$

Since $\delta_{2}/\hat{r}\leq 1$, it follows that

(108) $\displaystyle{\displaystyle\max_{i\in
E}\left|\bar{g}^{1}_{i}(u)+\left\langle\nabla\bar{g}^{1}_{i}(u),v-u\right\rangle\right|-\max_{i\in
E}\left|\bar{g}^{1}_{i}(u)\right|\leq-(\delta_{2}/\hat{r})M(u).}$

We deduce from property (i) that

$\displaystyle{\displaystyle\left\langle\nabla\bar{g}^{2}_{j}(u),v_{0}+\sum_{i=1}^{n_{E}}\left(\lambda_{i}+\alpha_{i}\right)\left(v_{i}-v_{0}\right)-u\right\rangle\leq-\delta/2,\
\forall j\in I.}$

It follows that

$\displaystyle{\displaystyle\left\langle\nabla\bar{g}^{2}_{j}(u),v-u\right\rangle\leq\left(M(u)\right)\left(-\delta/2\right)\
\forall j\in I.}$

Since $M(u)/\hat{r}\leq 1$ we deduce that

$\displaystyle{\displaystyle\min\left[0,\bar{g}^{2}_{j}(u)\right]+\left\langle\nabla\bar{g}^{2}_{j}(u),v-u\right\rangle\leq-\left(\delta/\left(2\hat{r}\right)\right)M(u)\
\forall j\in I.}$ (109)

Surveying inequalities (107)–(109), we see that $v$ satisfies all relevant
conditions for completion of the proof, when we set
$K_{1}=\min\\{\delta_{2},\delta/(2\hat{r})\\}$ and $K_{2}=2d/\hat{r}$, numbers
whose magnitudes do not depend on our choice of $u$.

##  Appendix B Proof of Proposition 3.1

###### Proof.

(i) Fix $u\in{\mathcal{U}}$, we must find $\hat{c}>0$ such that, if
$c>\hat{c}$ then $t_{c}(u)\leq 0$. If $M(u)=0$ then of course $t_{c}(u)\leq 0$
for any $c>0$. If $M(u)>0$ then according to Lemma 3.1 there exists
$\hat{d}\in{\cal U}-u$ such that, if we set $\varepsilon=K_{1}M(u)>0$ with
$K_{1}$ as in Lemma 3.1, then

$\displaystyle{\displaystyle\max_{i\in
E}\left[\left|\bar{g}^{1}_{i}(u)+\left\langle\nabla\bar{g}^{1}_{i}(u),\hat{d}\right\rangle\right|-M(u)\right]\leq\max_{i\in
E}\left[\left|\bar{g}^{1}_{i}(u)+\left\langle\nabla\bar{g}^{1}_{i}(u),\hat{d}\right\rangle\right|-\left|\bar{g}_{i}^{1}(u)\right|\right]\leq-\varepsilon,}$
$\displaystyle{\displaystyle\max_{j\in
I}\left[\bar{g}^{2}_{j}(u)+\left\langle\nabla\bar{g}^{2}_{j}(u),\hat{d}\right\rangle-M(u)\right]\leq\max_{j\in
I}\left[\bar{g}^{2}_{j}(u)+\left\langle\nabla\bar{g}^{2}_{j}(u),\hat{d}\right\rangle-\bar{g}^{2}_{i}(u)\right]\leq-\varepsilon}$

and

$\displaystyle{\displaystyle\theta(u)<-\varepsilon.}$

Here

$\displaystyle{\displaystyle\theta(u)=\max\left[\max_{i\in
E}\left|\bar{g}^{1}_{i}(u)+\left\langle\nabla\bar{g}^{1}_{i}(u),\hat{d}\right\rangle\right|-M(u),\right.}$
$\displaystyle{\displaystyle\left.\max_{j\in
I}\left[\bar{g}^{2}_{j}(u)+\left\langle\nabla\bar{g}^{2}_{j}(u),\hat{d}\right\rangle-M(u)\right]\right].}$

Because
$\sigma_{c}(u)=\left\langle\nabla\bar{F}_{0}(u),\bar{d}\right\rangle+c\left[\bar{\beta}-M(u)\right]$,
from the definition of $t_{c}$ and (31)–(33), we get

$\displaystyle{\displaystyle t_{c}(u)\leq W+c\theta(u)+M(u)/c,}$

where $W=\max\ [0,\langle\nabla\bar{F}_{0}(u),\hat{d}\rangle]$. It follows
that $t_{c}(u)\leq 0$ for any $c>\hat{c}$ where

$\displaystyle{\displaystyle\hat{c}:=\max\left[1,\frac{W+M(u)}{-\theta(u)}\right].}$

(ii) Take $u\in{\mathcal{U}}$ and $c>0$ such that $\sigma_{c}(u)<0$. Let
$(d,\beta)$ be the solution to ${\bf P_{c}(u)}$. Since $\sigma_{c}(u)\neq 0$,
it follows that $d\neq 0$.

We deduce from the differentiability properties of $\phi$, $g^{1}_{i}$,
$g^{2}_{j}$, and (34) that there exists $o:[0,\infty)\rightarrow[0,\infty)$
such that $s^{-1}o(s)\rightarrow 0$ as $s\downarrow 0$ and the following three
inequalities are valid for any $\alpha\in[0,1]$:

$\displaystyle{\displaystyle\bar{F}_{c}(u+\alpha
d)-\bar{F}_{c}(u)\leq\alpha\left\langle\nabla\bar{F}_{0}(u),d\right\rangle+c\max\left[\max_{i\in
E}\left|\bar{g}^{1}_{i}(u)+\alpha\left\langle\nabla\bar{g}^{1}_{i}(u),d\right\rangle\right|,\right.}$
(110) $\displaystyle{\displaystyle\left.\max_{j\in
I}\left(\bar{g}^{2}_{j}(u)+\alpha\left\langle\nabla\bar{g}^{2}_{j}(u),d\right\rangle\right)\right]-\max\left[\max_{i\in
E}\left|\bar{g}^{1}_{i}(u)\right|,\max_{j\in
I}\bar{g}^{2}_{j}(u)\right]+o(\alpha).}$

By convexity of the functions $e\rightarrow\max_{i\in
E}|\bar{g}^{1}_{i}(u)+\langle\nabla\bar{g}^{1}_{i}(u),e\rangle|$,
$e\rightarrow\max_{j\in I}(\bar{g}^{2}_{j}(u)$
$+\langle\nabla\bar{g}^{2}_{j}(u),e\rangle)$:

$\displaystyle{\displaystyle\max_{i\in
E}\left|\bar{g}^{1}_{i}(u)+\alpha\left\langle\nabla\bar{g}^{1}_{i}(u),d\right\rangle\right|-M(u_{k})\leq\alpha\left[\max_{i\in
E}\left|\bar{g}^{1}_{i}(u)+\left\langle\nabla\bar{g}^{1}_{i}(u),d\right\rangle\right|-M(u_{k})\right],}$
$\displaystyle{\displaystyle\max_{j\in
I}\left(\bar{g}^{2}_{j}(u)+\alpha\left\langle\nabla\bar{g}^{2}_{j}(u),d\right\rangle\right)-M(u_{k})\leq\alpha\left[\max_{j\in
I}\left(\bar{g}^{2}_{j}(u)+\left\langle\nabla\bar{g}^{2}_{j}(u),d\right\rangle\right)-M(u_{k})\right].}$

From inequality (110) then

$\displaystyle{\displaystyle\bar{F}_{c}(u+\alpha
d)-\bar{F}_{c}(u)\leq\alpha\left\\{\left\langle\nabla\bar{F}_{0}(u),d\right\rangle+c\max[\max_{i\in
E}\left|\bar{g}^{1}_{i}(u)+\left\langle\nabla\bar{g}^{1}_{i}(u),d\right\rangle\right|,\right.}$
$\displaystyle{\displaystyle\left.\left.\max_{j\in
I}\left(\bar{g}^{2}_{j}(u)+\left\langle\nabla\bar{g}^{2}_{j}(u),d\right\rangle\right)\right]-cM(u)\right\\}+o(\alpha)\leq\alpha\sigma_{c}(u)+o(\alpha).}$

It follows that

(111) $\displaystyle{\displaystyle\bar{F}_{c}(u+\alpha
d)-\bar{F}_{c}(u)\leq\alpha\gamma\sigma_{c}(u)\ \
\forall\alpha\in[0,\alpha_{1}],}$

where $\alpha_{1}>0$ is such that $o(\beta)\leq\beta(\gamma-1)\sigma_{c}(u)$
for all $\beta\in[0,\alpha_{1}]$.

##  Appendix C Proof of Theorem 3.1

###### Proof.

(i) Let $\\{u_{k}\\}$ be the sequence generated by Algorithm and let
$\\{c_{k}\\}$ be the corresponding penalty parameters. Let $\\{k_{l}\\}$ be
the sequence of index values at which the penalty parameter increases. By
extracting a further subsequence (we do not relabel) we can arrange that the
sequence $\\{u_{k_{l}}\\}$ has a limit point $\bar{u}\in{\mathcal{U}}$ because
${\mathcal{U}}$ is compact. We shall find a number $\hat{c}<\infty$ such that
for sufficiently large $k_{l}$ ‘$c_{k_{l}}>\hat{c}$’ implies
‘$\sigma_{c_{k_{l}}}(u_{k_{l}})\leq-M(u_{k_{l}})|/c_{k_{l}}$’. This
contradicts our assumption that the penalty parameter increases along the
subsequence. So we may conclude that $\\{c_{k}\\}$ is bounded.

Fix $k_{l}$ such that $u_{k_{l}}\in{\mathcal{O}}(\bar{u})$, where
${\mathcal{O}}(\bar{u})$ is the neighborhood of $\bar{u}$ as specified in
Lemma 3.1. From the minimizing property of $\sigma_{c_{k_{l}}}(u_{k_{l}})$ we
deduce

(112) $\displaystyle\sigma_{c_{k_{l}}}(u_{k_{l}})$ $\displaystyle\leq$
$\displaystyle\left\langle\nabla\bar{F}_{0}(u_{k_{l}}),v_{k_{l}}-u_{k_{l}}\right\rangle+$
$\displaystyle c_{k_{l}}\max\left[\max_{i\in
E}\left|\bar{g}^{1}_{i}(u_{k_{l}})+\left\langle\nabla\bar{g}^{1}_{i}(u_{k_{l}}),v_{k_{l}}-u_{k_{l}}\right\rangle\right|-M(u_{k_{l}}),\right.$
$\displaystyle\left.\max_{j\in
I}\left[\bar{g}^{2}_{j}(u_{k_{l}})+\left\langle\nabla\bar{g}^{2}_{j}(u_{k_{l}}),v_{k_{l}}-u_{k_{l}}\right\rangle\right]-M(u_{k_{l}})\right]$

for a control function $v_{k_{l}}$ satisfying conditions (74)–(76) of Lemma
3.1 in which $v_{k_{l}}$, $u_{k_{l}}$ replace $v$, $u$ respectively. It
follows

$\displaystyle\sigma_{c_{k_{l}}}(u_{k_{l}})$ $\displaystyle\leq$
$\displaystyle\left|\left\langle\nabla\bar{F}_{0}(u_{k_{l}}),v_{k_{l}}-u_{k_{l}}\right\rangle\right|-c_{k_{l}}K_{1}M(u_{k_{l}}).$

Since the control constraint $U$ is bounded and in view of (31)–(33) there
exists $r>0$ (independent of $k_{l}$) such that

$\displaystyle{\displaystyle\left|\left\langle\nabla\bar{F}_{0}(u_{k_{l}}),v_{k_{l}}-u_{k_{l}}\right\rangle\right|\leq
r\|v_{k_{l}}-u_{k_{l}}\|_{{\mathcal{L}}^{\infty}}.}$

Hence

$\displaystyle{\displaystyle\sigma_{c_{k_{l}}}(u_{k_{l}})\leq-(c_{k_{l}}K_{1}-rK_{2})M(u_{k_{l}}).}$

We conclude that

$\displaystyle{\displaystyle\sigma_{c_{k_{l}}}(u_{k_{l}})\leq-M(u_{k_{l}})/c_{k_{l}}}$

if $c_{k_{l}}\geq\hat{c}$, where $\hat{c}$ is the smallest positive number $c$
which satisfies the equation: $c^{2}K_{1}-c(rK_{2})\geq 1$.

(ii) and (iii) Let $\\{u_{k}\\}$ be an infinite sequence generated by
Algorithm. We must show that
$\lim_{k\rightarrow\infty}\sigma_{c_{k}}(u_{k})=0$ and, if a convergent
subsequence of $\\{u_{k}\\}$ has a limit point $\bar{u}\in{\mathcal{U}}$, that
conditions ${\bf(NC^{13})}$ are satisfied at $\bar{u}$.

Stage 1. (Convergence analysis) Since the $c_{k}$’s are bounded and can
increase only by multiplies of $c^{0}$, we must have $c_{k}=c$ for all $k\geq
k_{0}$, for some $k_{0}$ and $c>0$. In view of the manner in which $u_{k}$’s
are constructed, we have

$\displaystyle{\displaystyle\tilde{F}_{c}(u_{k+1})-\tilde{F}_{c}(u_{k})\leq\gamma\alpha_{k}\sigma_{c}(u_{k})}$

for all $k\geq k_{0}$. This means that, for all $j\geq 1$, $k\geq k_{0}$

(113)
$\displaystyle{\displaystyle\bar{F}_{c}(u_{k+j})-\bar{F}_{c}(u_{k})\leq\gamma\sum_{i=0}^{j-1}\alpha_{k+i}\sigma_{c}(u_{k+i}).}$

Since $\\{\bar{F}_{c}(u_{k})\\}$ is a bounded sequence and
$\alpha_{k}\sigma_{c}(u_{k})$ are nonpositive, we conclude

(114) $\displaystyle{\displaystyle\alpha_{k}\sigma_{c}(u_{k})\rightarrow 0\ \
{\rm as}\ k\rightarrow\infty.}$

Since $\sigma_{c}(u)$ is bounded as $u$ ranges over ${\mathcal{U}}$ we can
arrange by a subsequence extraction (we do not relabel) that

$\displaystyle{\displaystyle\sigma_{c}(u_{k})\rightarrow\sigma\ {\rm for\
some}\ \sigma\leq 0.}$

We claim that $\sigma=0$. To show this, suppose to the contrary that
$\sigma<0$. Then by (114) $\alpha_{k}\rightarrow 0$.

We must have

$\displaystyle\bar{F}_{c}(u_{k}+\eta^{-1}\alpha_{k}d_{k})-\bar{F}_{c}(u_{k})$
$\displaystyle>$ $\displaystyle\gamma\eta^{-1}\alpha_{k}\sigma_{c}(u_{k})$

for all $k$ sufficiently large.

However $\|\alpha_{k}d_{k}\|_{{\mathcal{L}}^{\infty}}\rightarrow 0$ as
$k\rightarrow\infty$ (it follows from the presence of term
$1/2\|d\|^{2}_{{\mathcal{L}}^{2}}$ in the objective function of the direction
finding subproblem).

It follows now from (31)–(34) and Propositions 2.1.5–2.1.6 stated in [12] that
there exists a function $o:[0,\infty)\rightarrow[0,\infty)$ such that
$s^{-1}o(s)\rightarrow 0$ as $s\downarrow 0$ and

(115) $\displaystyle\max_{i\in
E}\left|\bar{g}^{1}_{i}(u_{k}+\eta^{-1}\alpha_{k}d_{k})\right|$
$\displaystyle\leq$ $\displaystyle\max_{i\in
E}\left|\bar{g}^{1}_{i}(u_{k})+\left\langle\nabla\bar{g}^{1}_{i}(u_{k}),\eta^{-1}\alpha_{k}d_{k}\right\rangle\right|+$
$\displaystyle
o(\eta^{-1}\alpha_{k}\left\|d_{k}\right\|_{{\mathcal{L}}^{\infty}}),$ (116)
$\displaystyle\max_{j\in I}\bar{g}^{2}_{j}(u_{k}+\eta^{-1}\alpha_{k}d_{k})$
$\displaystyle\leq$ $\displaystyle\max_{j\in
I}\left[\bar{g}^{2}_{j}(u_{k})+\left\langle\nabla\bar{g}^{2}_{j}(u_{k}),\eta^{-1}\alpha_{k}d_{k}\right\rangle\right]+$
$\displaystyle
o(\eta^{-1}\alpha_{k}\left\|d_{k}\right\|_{{\mathcal{L}}^{\infty}}),$

and

$\displaystyle{\displaystyle\bar{F}_{c}(u_{k}+\eta^{-1}\alpha_{k}d_{k})-\bar{F}_{c}(u_{k})\leq\left\langle\nabla\bar{F}_{0}(u_{k}),\eta^{-1}\alpha_{k}d_{k}\right\rangle+}$
$\displaystyle{\displaystyle c\max\left[\max_{i\in
E}\left|\bar{g}^{1}_{i}(u_{k})+\left\langle\nabla\bar{g}^{1}_{i}(u_{k}),\eta^{-1}\alpha_{k}d_{k}\right\rangle\right|\right.-M(u_{k}),}$
(117) $\displaystyle{\displaystyle\left.\max_{j\in
I}\bar{g}^{2}_{j}(u_{k})+\left\langle\nabla\bar{g}^{2}_{j}(u_{k}),\eta^{-1}\alpha_{k}d_{k}\right\rangle-M(u_{k})\right]+o(\eta^{-1}\alpha_{k}\left\|d_{k}\right\|_{{\mathcal{L}}^{\infty}}).}$

By convexity of the functions $e\rightarrow\max_{i\in
E}|\bar{g}^{1}_{i}(u)+\langle\nabla\bar{g}^{1}_{i}(u),e\rangle|$,
$e\rightarrow\max_{j\in I}(\bar{g}^{2}_{j}(u)$
$+\langle\nabla\bar{g}^{2}_{j}(u),e\rangle)$:

$\displaystyle{\displaystyle\max_{i\in
E}\left|\bar{g}^{1}_{i}(u_{k})+\left\langle\nabla\bar{g}^{1}_{i}(u_{k}),\eta^{-1}\alpha_{k}d_{k}\right\rangle\right|-M(u_{k})}$
(118) $\displaystyle{\displaystyle\leq\eta^{-1}\alpha_{k}\left(\max_{i\in
E}\left|\bar{g}^{1}_{i}(u_{k})+\left\langle\nabla\bar{g}^{1}_{i}(u_{k}),d_{k}\right\rangle\right|-M(u_{k})\right),}$
$\displaystyle{\displaystyle\max_{j\in
I}\bar{g}^{2}_{j}(u_{k})+\left\langle\nabla\bar{g}^{2}_{j}(u_{k}),\eta^{-1}\alpha_{k}d_{k}\right\rangle-M(u_{k})}$
(119) $\displaystyle{\displaystyle\leq\eta^{-1}\alpha_{k}\left(\max_{j\in
I}\bar{g}^{2}_{j}(u_{k})+\left\langle\nabla\bar{g}^{2}_{j}(u_{k}),d_{k}\right\rangle-M(u_{k})\right),}$

Combining inequalities (115)–(119), noting the definition of
$\sigma_{c}(u_{k})$ and the fact that $(d_{k},\beta_{k})$ solves ${\bf
P_{c}(u_{k})}$, and dividing across the resulting inequality by $\alpha_{k}$
we arrive at

$\displaystyle{\displaystyle\eta^{-1}\sigma_{c}(u_{k})+\alpha_{k}^{-1}o(\eta^{-1}\alpha_{k}\left\|d_{k}\right\|_{{\mathcal{L}}^{\infty}})\geq\eta^{-1}\gamma\sigma_{c}(u_{k}).}$

We get $\eta^{-1}\sigma\geq\eta^{-1}\gamma\sigma$ in the limit. But this
implies $\gamma\geq 1$, since $\sigma<0$ by assumption. From this
contradiction we conclude the validity of $\sigma=0$. Assertion (ii) of the
theorem follows from the definition of $t_{c}$ and part (i).

Let $\\{u_{k}\\}$ be a convergent subsequence with the limit point
$\bar{u}\in{\mathcal{U}}$. We must show that conditions ${\bf(NC^{13})}$ are
satisfied at $\bar{u}$. First we establish that $\bar{u}$ is feasible for
${\bf(P)}$ and, for some $c>0$,

$\displaystyle{\displaystyle
0\leq\left[\left\langle\nabla\bar{F}_{0}(\bar{u}),d\right\rangle+c\max\left[\max_{i\in
E}\left|\left\langle\nabla\bar{g}^{1}_{i}(\bar{u}),d\right\rangle\right|,\max_{j\in
I}\left\langle\nabla\bar{g}^{2}_{j}(\bar{u}),d\right\rangle\right]\right]}$

for all $d\in{\mathcal{D}}_{\bar{u}}$.

Since $\\{u_{k}\\}\rightarrow\bar{u}$ we know that $x_{k}=x^{u_{k}}\rightarrow
x^{\bar{u}}=\bar{x}$ uniformly. Because the penalty parameter is not updated
for $k$ sufficiently large we have

$\displaystyle{\displaystyle\sigma_{c}(u_{k})\leq-M(u_{k})/c.}$

But we have shown that $\sigma_{c}(u_{k})\rightarrow 0$ as
$k\rightarrow\infty$. It follows now from the fact that
$u_{k}\rightarrow\bar{u}\in{\mathcal{U}}$ that in the limit

(120) $\displaystyle{\displaystyle\max_{i\in
E}\left|\bar{g}^{1}_{i}(\bar{u})\right|=0,\ \max_{j\in
I}\bar{g}^{2}_{j}(\bar{u})\leq 0.}$

We have established that $\bar{u}$ is feasible for ${\bf(P)}$.

Since $\sigma_{c}(u_{k})\rightarrow 0$ and $M(u_{k})\rightarrow 0$ we must
also have $d_{k}\rightarrow 0$. If it does not happen then we would come to
the contradiction with (68). That implies that we also must have
$\beta_{k}\rightarrow 0$ and that means that at $\bar{u}$ (72)–(73) must hold.

Now choose any sequence $\rho_{k}\downarrow 0$, $\rho_{k}\leq 1$ for all $k$
such that

(121) $\displaystyle{\displaystyle\rho_{k}^{-1}\sigma_{c}(u_{k})\rightarrow 0\
\ {\rm as}\ k\rightarrow\infty.}$

Choose any $d\in{\mathcal{D}}_{\bar{u}}$. By the convexity of ${\mathcal{U}}$,
$\rho_{k}d_{k}\in{\mathcal{D}}_{u_{k}}$ for each $k$.

By definition of $\sigma_{c}$ then

$\displaystyle\sigma_{c}(u_{k})$ $\displaystyle\leq$
$\displaystyle\phi_{x}(x^{u_{k}}(t_{f}))y^{x_{k},\rho_{k}d_{k}}(t_{f})+$
$\displaystyle c\max\left[\max_{i\in
E}\left|g^{1}_{i}(x^{u_{k}}(t_{f}))+(g^{1}_{i})_{x}(x^{u_{k}}(t_{f}))y^{x_{k},\rho_{k}d_{k}}(t_{f})\right|-M(u_{k}),\right.$
$\displaystyle\left.\max_{j\in
I}\left[g^{2}_{j}(x^{u_{k}}(t_{f}))+(g^{2}_{j})_{x}(x^{u_{k}}(t_{f}))y^{x_{k},\rho_{k}d_{k}}(t_{f})\right]-M(u_{k})\right].$

Fix $\hat{\varepsilon}>0$. Since $\rho_{k}\downarrow 0$ (and consequently
$y^{x_{k},\rho_{k}d_{k}}\rightarrow 0$ uniformly), we have:

$\displaystyle{\displaystyle\max_{j\in
I}\left[g^{2}_{j}(x^{u_{k}}(t_{f}))+(g^{2}_{j})_{x}(x^{u_{k}}(t_{f}))y^{x_{k},\rho_{k}d_{k}}(t_{f})\right]}$
$\displaystyle{\displaystyle\leq\max_{I_{\hat{\varepsilon},\bar{u}}}\left[\
g^{2}_{j}(x^{u_{k}}(t_{f}))+(g^{2}_{j})_{x}(x^{u_{k}}(t_{f}))y^{x_{k},\rho_{k}d_{k}}(t_{f})\right],}$

for all $k$ sufficiently large. Here, $I_{\varepsilon,u}=\\{i\in I:\
\bar{g}_{i}^{2}(u)\geq\max_{j\in I}[\bar{g}_{j}^{2}(u)]-\varepsilon\\}$.
Inserting these inequalities into (LABEL:l524), noting that
$y^{x_{k},\rho_{k}d_{k}}=\rho_{k}y^{x_{k},d_{k}}$, dividing across by
$\rho_{k}$ and passing to the limit with the help of (121) and continuity of
$\bar{F}_{0}(\cdot)$, $u\rightarrow\langle\nabla\bar{F}_{0}(u),d\rangle$, etc,
we obtain

$\displaystyle{\displaystyle
0\leq\phi_{x}(x^{\bar{u}}_{r}(t_{f}))y^{\bar{x},d}(t_{f})+c\max\left[\max_{i\in
E}\left|(g^{1}_{i})_{x}(x^{\bar{u}}(t_{f}))y^{\bar{x},d}(t_{f})\right|,\right.}$
(123) $\displaystyle{\displaystyle\left.\max_{j\in
I_{\hat{\varepsilon},\bar{u}}}(g^{2}_{j})_{x}(x^{\bar{u}}(t_{f}))y^{\bar{x},d}(t_{f})\right].}$

This inequality is valid for each $\hat{\varepsilon}>0$ and
$d\in{\mathcal{D}}_{\bar{u}}$.

Again choose arbitrary $d\in{\cal D}_{\bar{u}}$ and take
$\varepsilon_{k}\downarrow 0$. For each $k$ let
$(g^{2}_{j})_{x}(x^{\bar{u}}(t_{f}))\cdot$ $y^{\bar{x},d}(t_{f})$ achieves its
maximum over $I_{\varepsilon_{k},\bar{u}}$ at $j=j_{k}$. Then

$\displaystyle{\displaystyle
0\leq\phi_{x}(x^{\bar{u}}(t_{f}))y^{\bar{x},d}(t_{f})+c\left[\max_{i\in
E}\left|(g^{1}_{i})_{x}(x^{\bar{u}}(t_{f}))y^{\bar{x},d}(t_{f})\right|,\right.}$
(124)
$\displaystyle{\displaystyle\left.(g^{2}_{j_{k}})_{x}(x^{\bar{u}}(t_{f}))y^{\bar{x},d}(t_{f})\right].}$

Extract a subsequence (we do not relabel) such that $j_{k}=\bar{j}$ for all
$k$ and $t_{k}\rightarrow\bar{t}$ for some index value $\bar{j}$. By
continuity of the functions involved $\bar{j}\in I_{0,\bar{u}}$ and (124) is
valid with $\bar{j}$.

We have arrived at

$\displaystyle{\displaystyle
0\leq\phi_{x}(x^{\bar{u}}(t_{f}))y^{\bar{x},d}(t_{f})+c\max\left[\max_{i\in
E}\left|(g^{1}_{i})_{x}(x^{\bar{u}}(t_{f}))y^{\bar{x},d}(t_{f})\right|,\right.}$
(125) $\displaystyle{\displaystyle\left.\max_{j\in
I_{0,\bar{u}}}(g^{2}_{j})_{x}(x^{\bar{u}}(t_{f}))y^{\bar{x},d}(t_{f})\right].}$

Since $\sigma_{c_{k}}(u_{k})\rightarrow 0$, from (68) we must also have
$d_{k}\rightarrow 0$. $\\{c_{k}\\}$ is bounded and $M(u_{k})\rightarrow 0$
then (68) also implies that $\beta_{k}\rightarrow 0$. Therefore, the
inequality (125), which holds for all $d\in{\mathcal{D}}_{\bar{u}}$, in
particular for $(d\equiv 0)\in{\mathcal{D}}_{\bar{u}}$, is what we set out to
prove.

Finally, we must attend to the case when Algorithm generates a finite sequence
which terminates at a control $u_{\bar{k}}=\bar{u}$ satisfying the stopping
criterion. This case is dealt with by applying the preceding arguments to the
infinite sequence of controls obtained by ‘filling in’ with repetitions of the
following control $u_{k}$.

## References

  * [1] U. Ascher and L. Petzold, Computer methods for ordinary differential equations and differential-algebraic equations, Society for Industrial and Applied Mathematics, Philadelphia, 1998.
  * [2] J. Aubin and I. Ekeland, Applied Nonlinear Analysis, Wiley-Interscience, New York, 1984.
  * [3] M. S. Branicky, V. S. Borkar, and S. K. Mitter, A unified framework for hybrid control: model and optimal control theory, IEEE Trans. on Autom. Control, 43 (1998), pp. 31–65.
  * [4] A. Bryson and Y. Ho, Applied Optimal Control, Hemisphere, New York, 1975\.
  * [5] Y. Cao, S. Li, L. Petzold, and R. Serben, Adjoint sensitivity analysis for differential–algebraic equations: The adjoint DAE system and its numerical solution, SIAM J. Sci. Comp., 24 (2000), pp. 1076–1089.
  * [6] L. Dieci and L. Lopez, Sliding motion in Filippov differential systems: Theoretical results and a computational approach, SIAM J. Numer. Anal., 47 (2009), pp. 2023–2051.
  * [7] A. F. Filippov, Differential Equations with Discontinuous Righthand Sides, Kluwer Academic Publishers, 1988.
  * [8] M. A. H. Schiller, Convergence of continuous approximations for discontinuous odes, Applied Numerical Mathematics, 62 (2012), pp. 1503–1514.
  * [9] J. Lygeros, K. H. Johansson, S. S. Sastry, and M. Egerstedt, On the existence of executions of hybrid automata, in Proceed. of the 38th IEEE CDC, Phoenix, Arizona, 1999, pp. 2249–2254.
  * [10] I. Mynttinen, A. Hoffmann, E. Runge, and P. Li, Smoothing and regularization strategies for optimization of hybrid dynamic systems, Optim. Eng., 16 (2015), pp. 541–569.
  * [11] A. Pakniyat and P. Caines, The hybrid minimum principle in the presence of switching costs, in Proceed. of the 38th IEEE CDC, December 10-13, Florence, Italy, 2013, pp. 3831–3836.
  * [12] R. Pytlak, Numerical Methods for Optimal Control Problems with State Constraints. Lecture Notes in Mathematics 1707, Springer–Verlag, Berlin, Heidelberg, 1999.
  * [13] R. Pytlak and D. Suski, On solving hybrid optimal control problems with higher index DAEs, Optimization Methods & Software, 32 (2017), pp. 940–962.
  * [14] R. Pytlak and D. Suski, Numerical procedure for optimal control of hybrid systems with sliding modes, Part 1, submitted to arXiv, (2021).
  * [15] R. Pytlak and D. Suski, Numerical procedure for optimal control of hybrid systems with sliding modes, Part 2, submitted to arXiv, (2021).
  * [16] R. Pytlak and D. Suski, Trajectory sensitivity analysis of hybrid systems with sliding motion, submitted for publication, (2021).
  * [17] R. Pytlak and R. Vinter, A feasible directions algorithm for optimal control problems with state and control constraints: convergence analysis, SIAM J. on Control and Optimization, 36 (1998), pp. 1999–2019.
  * [18] C. Seatzu, D. Corona, A. Giua, and A. Bemporad, Optimal control of continuous time switched affine systems, IEEE Transactions on AC, 51 (2006), pp. 726–741.
  * [19] M. S. Shaikh, Optimal control of hybrid systems: theory and algorithms, Ph.D. dissertation, McGill University, Montreal, 2004\.
  * [20] M. S. Shaikh and P. Caines, On the hybrid optimal control problem: Theory and algorithms, IEEE Transactions on AC, 52 (2007), pp. 1587–1603.
  * [21] M. S. Shaikh and P. Caines, Correction to: On the hybrid optimal control problem: Theory and algorithms, IEEE Transactions on AC, 54 (2009), p. 1.
  * [22] D. Stewart and M. Anitescu, Optimal control of systems with discontinuous differential equations, Num. Math., 114 (2010), pp. 653–695.
  * [23] H. Sussmann, A maximum principle for hybrid optimal control problems, in Proceed. of the 38th IEEE CDC, Phoenix, Arizona, 1999, pp. 425–430.
  * [24] F. Taringoo and P. Caines, Gradient-geodesic HMP algorithms for the optimization of hybrid systems based on the geometry of switching manifolds, in Proceed. of the 38th IEEE CDC, December 15-17, Atlanta, GA, 2010, pp. 1534–1539.
  * [25] F. Taringoo and P. Caines, On the extension of the hybrid minimum principle to Riemannian manifolds, in Proceed. of the 38th IEEE CDC, December 12-15, Orlando, Fl, 2011, pp. 3301–3306.
  * [26] F. Taringoo and P. Caines, On the optimal control of hybrid systems on Lie groups and the exponential gradient HMP algorithm, in Proceed. of the 38th IEEE CDC, December 10-13, Florence, Italy, 2013, pp. 2653–2658.
  * [27] A. van der Schaft and H. Schumacher, An Introduction to Hybrid Dynamical Systems, Springer–Verlag, London, 2000.
  * [28] H. Witsenhausen, A class of hybrid-state continuous-time dynamic systems, IEEE Transactions on AC, 11 (1966), pp. 161–167.