# Robust Action Governor for Uncertain Piecewise Affine Systems
with Non-convex Constraints and Safe Reinforcement Learning

Yutong Li<EMAIL_ADDRESS>Nan Li<EMAIL_ADDRESS>H. Eric Tseng<EMAIL_ADDRESS>Anouck Girard<EMAIL_ADDRESS>Dimitar Filev<EMAIL_ADDRESS>Ilya Kolmanovsky
<EMAIL_ADDRESS>Department of Aerospace Engineering, University of Michigan,
Ann Arbor, MI 48105, USA  Ford Motor Company, Dearborn, MI, 48126 USA

###### Abstract

The action governor is an add-on scheme to a nominal control loop that
monitors and adjusts the control actions to enforce safety specifications
expressed as pointwise-in-time state and control constraints. In this paper,
we introduce the Robust Action Governor (RAG) for systems the dynamics of
which can be represented using discrete-time Piecewise Affine (PWA) models
with both parametric and additive uncertainties and subject to non-convex
constraints. We develop the theoretical properties and computational
approaches for the RAG. After that, we introduce the use of the RAG for
realizing safe Reinforcement Learning (RL), i.e., ensuring all-time constraint
satisfaction during online RL exploration-and-exploitation process. This
development enables safe real-time evolution of the control policy and
adaptation to changes in the operating environment and system parameters (due
to aging, damage, etc.). We illustrate the effectiveness of the RAG in
constraint enforcement and safe RL using the RAG by considering their
applications to a soft-landing problem of a mass-spring-damper system.

###### keywords:

Action governor; Constrained control; Reinforcement learning; Piecewise affine
systems; Uncertainties.

††thanks: This paper was not presented at any IFAC meeting. Corresponding
author Yutong Li. The first two authors contributed equally to this work.

, , , , ,

##  1 Introduction

Cyber-physical systems (CPS) are becoming increasingly adopted in a broad
range of applications. Examples of CPS include self-driving cars [36],
manufacturing systems [19], and medical devices [30]. Piecewise-affine (PWA)
models are capable of representing the interaction of continuous and discrete
dynamics, which frequently arise in CPS, and, thereby, have gained increasing
research interest [51, 20, 9].

Safety is a crucial consideration for many CPS. Many safety requirements can
be expressed as state and control constraints. One route to handle safety
constraints is through incorporating them into the controller design. Model
predictive control (MPC) is a popular approach due to its ability to
explicitly enforce constraints [12, 9]. MPC for PWA models has been studied
in, e.g., [29] and [8]. Another route to handle constraints is through add-on,
supervisory schemes that complement the existing/legacy controller. The
reference governor (RG) is one of such add-on schemes, which enforces
constraints by monitoring and manipulating the reference inputs to the closed-
loop system [16]. RG design strategies based on PWA models are presented in
[10]. The control barrier function (CBF) provides another way to handle
constraints via a supervisory scheme [55]: A CBF and a Control Lyapunov
Function (CLF) are integrated into a quadratic program (QP) to promote
stability and enforce constraints [1].

Enforcing state and control constraints is more challenging when there exist
discrepancies between the real-world system and its control-oriented model.
Such discrepancies may arise from unknown model parameters, unmodeled
dynamics, external disturbances, etc. These are collectively referred to as
uncertainties.

Figure 1: Schematic diagram of the RAG.

Robust model predictive control (RMPC) is a variant of MPC for addressing
uncertainties [3]. A popular RMPC approach is the tube MPC, where uncertain
trajectories are bounded by a tube and constraints are tightened so that
enforcing the tightened constraints for a nominal trajectory implies
constraint satisfaction by all trajectories admissible under the bounded
uncertainty. Another approach is the min-max MPC, where control performance
under the worst-case uncertainty is optimized using a min-max approach.
Typical approaches based on RMPC require the (re-)design of the controller.
Robust versions of RG and CBF to enforce constraints in the presence of
uncertainties have also been proposed, for linear systems in [25] and for
nonlinear systems with known input-to-state stability (ISS) Lyapunov functions
in [17, 23, 13].

In this paper, we focus on another add-on, supervisory scheme, called the
Robust Action Governor (RAG), for handling safety constraints of CPS. The RAG
is a scheme which monitors and modifies the output of the nominal control
policy, as illustrated in Fig.1. The RAG monitors the control action generated
by the nominal policy and, when necessary, minimally modifies it to ensure
that the plant response to the modified control action satisfies prescribed
constraints. The action governor (AG) scheme was first introduced for linear
systems in [32] and then extended to PWA models subject to only additive
disturbances in [33]. In this paper, we further develop the AG theory and
computational approaches to enable AG design based on PWA models with both
parametric and additive uncertainties.

More specifically, this paper differentiates itself from our previous work in
[33] and the work of [49] by considering both parametric and additive
uncertainties in the PWA model, while [33] and [49] only consider additive
disturbances. On the one hand, incorporating both parametric and additive
uncertainties significantly enlarges the class of systems to which the RAG can
be applied. This is because many uncertainties, such as uncertainties in
system parameters, can be straightforwardly represented as parametric
uncertainties in the model, but can be difficult, if not impossible, to be
non-conservatively account for with additive disturbances [37, 6]. On the
other hand, in order to address parametric uncertainties, new methods for
computation and analysis of the RAG are needed, which necessitates the
developments in this paper, including the new proofs for the RAG properties.

We note that the RAG addressed in this paper focuses on enforcing non-convex
constraints, i.e., the region within which the system state needs to be
maintained to ensure safety is generally non-convex. Such as in the obstacle
avoidance problems frequently encountered in autonomous vehicle/mobile robot
applications [15, 54]. Many previous approaches to PWA systems, including the
RMPC approach in [28], the approaches based on RG in [10] and based on CBF in
[52, 43], have focused on convex constraints. Although non-convex constraints
are addressed in [45, 40, 47, 41], by either RG or CBF, these references do
not deal with PWA systems, which is the focus of this work. Furthermore, to
the best of our knowledge, no methods using CBFs to treat PWA models with both
parametric and additive uncertainties currently exist, partly because CBF for
a PWA system with parametric uncertainties is not easy to compute [7]. Some
other differences of the AG approach as compared to the RG and the CBF
approaches for general systems are highlighted in [32, 33].

In this paper, after introducing the RAG and describing its properties, we
illustrate its use for safe RL. Specifically, we propose a safe RL framework
where the RAG is used as a safety supervisor to monitor the actions selected
by a nominal RL agent and correct the unsafe actions to safe ones. This way,
one can augment an arbitrary RL algorithm with the RAG after which the
combined algorithm ensures all-time safety during the entire learning process.
Along these lines, in this paper we demonstrate the integration of Q-learning
with the RAG to realize safe Q-learning. In addition, we demonstrate that the
development of an explicit safe RL policy to reduce the online computational
cost of the safe RL is feasible using the imitation learning. Although similar
approaches have been proposed in the model-based RL literature [2, 14, 27, 50,
35], none of these references deals with PWA systems that have both parametric
and additive uncertainties.

The main contributions of this paper are as follows:

  * •

We extend the RAG scheme developed in [34, 33] to PWA systems subject to both
additive and parametric uncertainties. This is the first time that the
parametric uncertainties are accommodated in the RAG design.

  * •

We analyze theoretical properties of the proposed RAG scheme, including robust
constraint satisfaction and recursive feasibility. New safe set computational
approaches that handle parametric uncertainties are developed. These
theoretical properties and computational approaches are based on the PWA
system model subject to non-convex constraints and both additive and
parametric uncertainties, which distinguish our RAG scheme versus existing
robust control techniques. For instance, while robust variants of the CBF have
been proposed, finding CBFs and CLFs for PWA systems is nontrivial [7],
especially in the presence of both additive and parametric uncertainties.

  * •

We extend the safe RL framework in [34] based on the proposed RAG scheme. Both
additive and parametric uncertainties are taken into account within the safe
exploration region design for the RL agent. Moreover, in order to reduce the
online computational cost, an explicit safe RL policy is obtained by using
imitation learning.

  * •

We demonstrate the effectiveness of the proposed RAG scheme and safe RL
framework using an example of soft-landing control for a mass-spring-damper
system. Such an example is relevant to applications in the ones of aircraft
flight control [44], robotic manipulation [26, 39] and cooperative navigation
[53].

The remainder of the paper is organized as follows: In Section 2 the main
assumptions and problem setup are introduced. In Section 3 the main
theoretical results and computational approaches of the RAG are presented. In
Section 4 the safe RL framework is introduced. Numerical examples and results
are reported in Section 5. Section 6 contains concluding remarks and
directions for future work.

Definitions and Notation: Given two sets $A$ and $B$, then $A\oplus
B=\\{a+b:a\in A,b\in B\\}$ (Minkowski sum), and $A\sim B=\\{a:a\oplus
B\subseteq A\\}$ (Pontryagin difference). $\text{Proj}_{x}(A)$ represents the
projection of the set $A$ onto $x$ space. A polyhedron (polyhedral set) is the
intersection of a finite number of closed and/or open halfspaces, a polytope
is a closed and bounded (equivalently, compact) polyhedron and a polygon is
the union of a finite number of polyhedra (and is thus not necessarily
convex).

##  2 Problem formulation

In this paper, we focus on systems that can be described by the following PWA
model,

$\displaystyle\begin{split}x(t+1)&=f_{q(t)}(x(t),u(t),w(t))\\\
&=A_{q(t)}(w^{\text{p}}(t))x(t)+B_{q(t)}(w^{\text{p}}(t))u(t)\\\
&\quad+f_{q(t)}(w^{\text{p}}(t))+E_{q(t)}(w^{\text{p}}(t))w^{\text{a}}(t),\end{split}$
(1)

where $x(t)\in\mathbb{R}^{n}$ represents the system state at the discrete time
instant $t\in\mathbb{Z}_{\geq 0}$, $u(t)$ represents the control input,
$w(t)=(w^{\text{p}}(t),w^{\text{a}}(t))$ represents unmeasured uncertainties,
including both parametric uncertainty $w^{\text{p}}(t)$ and additive
uncertainty $w^{\text{a}}(t)$, and $q(t)\in Q=\\{1,2,...,n_{q}\\}$ represents
the current mode of the system. We assume the mode $q(t)$ is state-dependent,
in particular,

$q(t)=q\quad\text{if}\quad x(t)\in P_{q},$ (2)

where $P_{q}\subset\mathbb{R}^{n}$ is a polyhedral set and the collection
$\\{P_{q}\\}_{q=1}^{n_{q}}$ forms a partition of $\mathbb{R}^{n}$. This means
the dynamics (1) hold for all $x\in P_{q(t)}$. Given a mode $q\in Q$, we
assume the uncertain matrices
$(A_{q}(w^{\text{p}}),B_{q}(w^{\text{p}}),f_{q}(w^{\text{p}}),E_{q}(w^{\text{p}}))$
can be parameterized by $w^{\text{p}}$ as

$\displaystyle\begin{split}&A_{q}(w^{\text{p}})=\sum_{j=1}^{n^{\text{p}}_{q}}w^{\text{p}}_{j}A_{q,j},\,\,B_{q}(w^{\text{p}})=\sum_{j=1}^{n^{\text{p}}_{q}}w^{\text{p}}_{j}B_{q,j},\\\
&f_{q}(w^{\text{p}})=\sum_{j=1}^{n^{\text{p}}_{q}}w^{\text{p}}_{j}f_{q,j},\,\,E_{q}(w^{\text{p}})=\sum_{j=1}^{n^{\text{p}}_{q}}w^{\text{p}}_{j}E_{q,j},\end{split}$
(3)

where $(A_{q,j},B_{q,j},f_{q,j},E_{q,j})$, for $j=1,...,n^{\text{p}}_{q}$, are
known constant matrices. Given the current mode $q(t)\in Q$, we assume the
uncertainties $w(t)$ take values according to

$w(t)=(w^{\text{p}}(t),w^{\text{a}}(t))\in W^{\text{p}}_{q(t)}\times
W^{\text{a}}_{q(t)}=W_{q(t)},$ (4)

where $W^{\text{a}}_{q(t)}$ is a mode-dependent polytope, and
$W^{\text{p}}_{q(t)}$ is a mode-dependent unit simplex, written as

$W^{\text{p}}_{q(t)}=\\{w^{\text{p}}\in\mathbb{R}^{n^{\text{p}}_{q(t)}}:\sum_{j=1}^{n^{\text{p}}_{q(t)}}w^{\text{p}}_{j}=1,\
w^{\text{p}}_{j}\geq 0\\}.$ (5)

We note that the above equations, (1)-(5), define a state‐dependent switching
system with both parametric and additive uncertainties.

We now consider the following safety requirements,

$x(t)\in X,\quad u(t)\in U_{q(t)},\quad\forall t\in\mathbb{Z}_{\geq 0},$ (6)

where $X\subset\mathbb{R}^{n}$ is a mode-independent safe region for the
state, and $U_{q(t)}$ is a mode-dependent safe region for the control input
and is assumed to be a polytope. In particular, we consider nonconvex safety
requirements in the form of

$x(t)\in X=\bigcup_{i=1}^{r_{0}}X^{i},$ (7)

where $X^{i}$ is a polyhedron for each $i=1,...,r_{0}$, and thus $X$ is a
polygon (not necessarily convex).

Oftentimes, it is easier to design a nominal control policy that achieves
liveness properties such as stabilization and reference tracking than to
develop one that ensures both liveness and the strict satisfaction of safety
requirements. For instance, for linear systems, a variety of tools exist for
stabilization and reference tracking, such as pole placement, linear-quadratic
regulator, etc., but these conventional tools cannot strictly handle
constraints. Along these lines, suppose a nominal control policy, $\phi$, has
been defined for the system (1),

$u_{\phi}(t)=\phi(x(t),r(t),t),$ (8)

where $r(t)\in\mathbb{R}^{r}$ denotes a reference command signal corresponding
to the nominal control objective (such as tracking). We note that there is no
further assumption on the nominal control policy $\phi$ and it can be
nonlinear and/or time-varying. For instance, $\phi$ may represent a neural
network controller that evolves over time via online learning, which will be
further discussed in Sections 4 and 5. Note also that for the PWA system (1),
such a control policy is typically mode-dependent. However, since the mode
$q(t)$ is determined by the state $x(t)$ according to (2), the dependence on
$q(t)$ eventually becomes the dependence on $x(t)$. So we have chosen not to
include $q(t)$ as an input to the control policy $\phi$ in (8).

In this paper, we consider a scenario where the safety requirement (7) has not
been incorporated or cannot be guaranteed by the nominal policy (8). The
objective of this paper is to develop an add-on control scheme for the system
(1) to strictly handle the safety requirements (6)-(7) in the presence of
parametric and additive uncertainties.

##  3 Robust action governor

The solution we propose is an add-on, supervisory scheme, called the Robust
Action Governor (RAG), illustrated in Fig. 1. The RAG monitors the nominal
control input $u_{\phi}$ and, if necessary, minimally modifies $u_{\phi}$ to
ensure that the system state $x$ stays inside $X$ at all time instants even in
the presence of model uncertainties/disturbances.

In particular, the RAG operates by solving the following constrained
optimization problem at each time instant $t$ under the assumption that the
current state $x(t)$ and mode $q(t)$ are both known,

$\displaystyle\begin{split}&u(t)\in\operatorname*{arg\,min}_{u\in U_{q(t)}}\
{\|u-u_{\phi}(t)\|}_{S}^{2}\\\ &\text{ s.t.
}\,A_{q(t)}(w^{\text{p}}_{q(t)}(t))x(t)+B_{q(t)}(w^{\text{p}}_{q(t)}(t))u+f_{q(t)}(w^{\text{p}}_{q(t)}(t))\\\
&\quad\quad+E_{q(t)}(w^{\text{p}}_{q(t)}(t))w^{\text{a}}_{q(t)}(t)\in\chi_{\text{safe}},\
\forall w^{\text{p}}_{q(t)}(t)\in W^{\text{p}}_{q(t)},\\\ &\quad\quad\forall
w^{\text{a}}_{q(t)}(t)\in W^{\text{a}}_{q(t)},\end{split}$ (9)

where $\chi_{\text{safe}}\subset X$ is a “safe set” which will be introduced
in the next section. The function $\|\cdot\|_{S}=\sqrt{(\cdot)^{T}S(\cdot)}$,
where $S$ is a positive-definite penalty matrix, is used to promote
minimization of the difference between the nominal control $u_{\phi}$ and the
modified control $u$.

###  3.1 Safe set

To ensure robust satisfaction of the non-convex safety requirement (7) at all
future time instants $t$, the safe set $\chi_{\text{safe}}$ should have the
following property: Suppose $x(0)\in\chi_{\text{safe}}$, then there exists a
state-feedback control law, $u(x)$, such that

  * •

the control inputs $u(t)=u(x(t))$ take values necessarily in $U_{q(t)}$ for
all $t=0,1,...$

  * •

under $u(x)$, the states $x(t)$ are necessarily inside $X$ for all $t=1,2,...$

under any uncertainty realizations $\\{w(0),w(1),...\\}$ that satisfy $w(t)\in
W_{q(t)}$ for all $t=0,1,...$

We first introduce the following lemma to facilitate the definition of
$\chi_{\text{safe}}$.

Lemma 1 Let $J$ be an affine function of $w\in W$, $W$ be a polytope with
vertices $\\{\bar{w}_{i}\\}_{i=1}^{n_{W}}$, and $Z$ be a polyhedron. Then,

$J(w)z\in Z,\quad\forall w\in W,$ (10)

if and only if

$J(\bar{w}_{i})z\in Z,\quad i=1,...,n_{W}.$ (11)

Proof: The proof follows from Lemma 11.1 of [9]. $\blacksquare$

To define $\chi_{\text{safe}}$, we first consider a sequence of sets,
$\chi_{\text{safe},k}$, defined recursively as follows,

$\displaystyle\begin{split}\chi_{\text{safe},0}=X,\end{split}$ (12)

and for $k\in\mathbb{Z}_{\geq 1}$

$\displaystyle\chi_{\text{safe},k}=\text{Proj}_{x}({\Lambda_{\text{safe},k-1}})$
(13a) where $\displaystyle\Lambda_{\text{safe},k}$
$\displaystyle=\bigcup_{q=1}^{n_{q}}\bigcup_{i=1}^{r_{k}}\\{(x,u):x\in\chi_{\text{safe},k}\cap{P}_{q},u\in
U_{q},\text{s.t.}\,\forall w\in W_{q},$ $\displaystyle
A_{q}(w^{\text{p}})x+B_{q}(w^{\text{p}})u+f_{q}(w^{\text{p}})+E_{q}(w^{\text{p}})w^{\text{a}}\in\chi_{\text{safe},k}^{i}\\}$
$\displaystyle=\bigcup_{q=1}^{n_{q}}\bigcup_{i=1}^{r_{k}}\\{(x,u):x\in\chi_{\text{safe},k}\cap{P}_{q},u\in
U_{q},\text{s.t.}\,\forall w^{\text{a}}\in W_{q}^{\text{a}},$ $\displaystyle
A_{q}(w^{\text{p}})x+B_{q}(w^{\text{p}})u+f_{q}(w^{\text{p}})+E_{q}(w^{\text{p}})w^{\text{a}}\in\chi^{i}_{\text{safe},k},\forall
w^{\text{p}}\in W^{\text{p}}_{q}\\}$
$\displaystyle=\bigcup_{q=1}^{n_{q}}\bigcup_{i=1}^{r_{k}}\\{(x,u):x\in\chi_{\text{safe},k}\cap{P}_{q},u\in
U_{q},\text{s.t.}\,\forall w^{\text{a}}\in W_{q}^{\text{a}},$ $\displaystyle
A_{q,j}x+B_{q,j}u+f_{q,j}+E_{q,j}w^{\text{a}}\in\chi^{i}_{\text{safe},k},j=1,2,...,n^{\text{p}}_{q}\\}$
$\displaystyle=\bigcup_{q=1}^{n_{q}}\bigcup_{i=1}^{r_{k}}\\{(x,u):x\in\chi_{\text{safe},k}\cap{P}_{q},u\in
U_{q},\text{s.t.}\,$ $\displaystyle
j=1,2,...,n^{\text{p}}_{q},A_{q,j}x+B_{q,j}u\in\chi^{i}_{\text{safe},k}\sim
E_{q,j}W_{q}^{\text{a}}-f_{q,j}\\}$
$\displaystyle=\bigcup_{q=1}^{n_{q}}\bigcup_{i=1}^{r_{k}}\\{(x,u):x\in\chi_{\text{safe},k}\cap{P}_{q},u\in
U_{q},\text{s.t.}$
$\displaystyle\begin{bmatrix}\bar{A}_{q}&\bar{B}_{q}\end{bmatrix}\begin{bmatrix}x\\\
u\end{bmatrix}\in\prod_{j=1}^{n^{\text{p}}_{q}}(\chi^{i}_{\text{safe},k}\sim
E_{q,j}W_{q}^{\text{a}}-f_{q,j})\\}.$ (13b)

where $\chi_{\text{safe},k}$ is assumed to be a polygon, written as
$\chi_{\text{safe},k}=\bigcup_{i=1}^{r_{k}}\chi^{i}_{\text{safe},k}$ with
polyhedra $\chi^{i}_{\text{safe},k}$, $i=1,...,r_{k}$. $\prod$ represents the
Cartesian product where $\prod_{j=1}^{s}S_{j}=\\{(x_{1},...,x_{s}):x_{j}\in
S_{j},\forall j\in\\{1,...,s\\}\\}$, and
$\begin{bmatrix}\bar{A}_{q}&\bar{B}_{q}\end{bmatrix}$ are defined as

$\displaystyle\begin{split}&\begin{bmatrix}\bar{A}_{q}&\bar{B}_{q}\end{bmatrix}=\begin{bmatrix}{A}_{q,1}&{B}_{q,1}\\\
\vdots&\vdots\\\
{A}_{q,n_{q}^{\text{p}}}&{B}_{q,n_{q}^{\text{p}}}\end{bmatrix}.\end{split}$
(14)

Note that in (13), between the third line and the forth line, we use the
result from Lemma 1. In what follows we first show that if
$\chi_{\text{safe},k-1}$ is indeed a polygon, then $\chi_{\text{safe},k}$
defined according to (13) is also a polygon. A direct consequence of this fact
is that if $\chi_{\text{safe},0}=X$ is a polygon, then the sequence of sets,
$\\{\chi_{\text{safe},k}\\}_{k=0}^{\infty}$, defined recursively according to
(13) are all polygons. Then, it can be shown that if
$x(0)\in\chi_{\text{safe},k}$, $k\in\mathbb{Z}_{\geq 0}$, then it is possible
to keep $x(t)$ within $X$ for at least $k$ steps. These properties are
elaborated in the following propositions.

Proposition 2 If
$\chi_{\text{safe},k-1}=\bigcup_{i=1}^{r_{k-1}}\chi^{i}_{\text{safe},k-1}$,
for some $k\in\mathbb{Z}_{\geq 1}$, is indeed a polygon with polyhedra
$\chi^{i}_{\text{safe},k-1}$, $j=1,...,r_{k-1}$, then $\chi_{\text{safe},k}$
defined according to (13) is also a polygon.

Proof: Let
$\chi_{\text{safe},k-1}=\bigcup_{i=1}^{r_{k-1}}\chi^{i}_{\text{safe},k-1}$ be
a polygon for some $k\in\mathbb{Z}_{\geq 1}$, recall that the Pontryagin
difference of a polyhedron and a polytope is a polyhedron, and the Cartesian
product of polyhedron is a (higher-dimensional) polyhedron, thus
$\prod_{j=1}^{n^{\text{p}}_{q}}(\chi^{i}_{\text{safe},k-1}\sim
E_{q,j}W_{q}^{\text{a}}-f_{q,j})$ is a polyhedron. Furthermore, because the
preimage of a polyhedron under a linear transformation is a polyhedron,
$\chi_{\text{safe},k-1}\cap P_{q}$ is a polygon, $U_{q}$ is a polytope, and
the intersection of a polygon and a polytope is a polygon, hence
$\\{(x,u):x\in\chi_{\text{safe},k-1}\cap{P}_{q},u\in
U_{q},\text{s.t.}\begin{bmatrix}\bar{A}_{q}&\bar{B}_{q}\end{bmatrix}\begin{bmatrix}x\\\
u\end{bmatrix}\in\prod_{j=1}^{n^{\text{p}}_{q}}(\chi^{i}_{\text{safe},k-1}\sim
E_{q,j}W_{q}^{\text{a}}-f_{q,j})\\}$ is a polygon, so as
$\Lambda_{\text{safe},k-1}$. Finally, as the projection of a polygon onto a
subspace is a (lower-dimensional) polygon [46], and according to (13a), we
have $\chi_{\text{safe},k}=\text{Proj}_{x}(\Lambda_{\text{safe},k-1})$ is a
polygon. $\blacksquare$

Proposition 3 If $x(0)\in\chi_{\text{safe},k}$, then there exists a state-
feedback control sequence $\\{u_{0}(x(0)),...,u_{k-1}(x(k-1))\\}\in
U_{q(0)}\times\cdots\times U_{q(k-1)}$, such that for any disturbance sequence
$\\{w(0),...,w(k-1)\\}\in W_{q(0)}\times\cdots\times W_{q(k-1)}$, $x(j)\in X$
for all $0\leq j\leq k$.

Proof: The proof is by induction. For $k=1$, $x(0)\in\chi_{\text{safe},1}$
implies that there exists a control $u(x(0))$, such that
$(x(0),u(x(0)))\in\Lambda_{\text{safe},0}$ from (13a). Thus, by the definition
of $\Lambda_{\text{safe},0}$, we have that $x(0)\in\chi_{\text{safe},0}=X$,
and there exist $q_{0}\in\\{1,...,n_{q}\\}$ and $i_{0}\in\\{1,...,r_{0}\\}$,
such that there exists $u_{0}(x(0))\in U_{q_{0}}$, for all $w_{0}\in
W_{q_{0}}$,
$x(1)=f_{q_{0}}(x(0),u_{0},w_{0})\in\chi^{i_{0}}_{\text{safe},0}\subset\bigcup_{i=1}^{r_{0}}\chi^{i}_{\text{safe},0}=\chi_{\text{safe},0}=X$.
Suppose the statement has been proven for all $\chi_{\text{safe},j}$, $1\leq
j\leq k$. If $x(0)\in\chi_{\text{safe},k+1}$, then from (13a) we have that
there exists $u(x(0))$, such that $(x(0),u(x(0)))\in\Lambda_{\text{safe},k}$.
Again, from the definition of $\Lambda_{\text{safe},k+1}$ in (13), there exist
$q_{0}\in\\{1,...,n_{q}\\}$ and $i_{0}\in\\{1,...,r_{k}\\}$ such that there
exists $u_{0}(x(0))\in U_{q_{0}}$, for all $w_{0}\in W_{q_{0}}$,
$x(1)=f_{q_{0}}(x(0),u_{0},w_{0})\in\chi^{i_{0}}_{\text{safe},k}\subset\bigcup_{i=1}^{r_{k}}\chi^{i}_{\text{safe},k}=\chi_{\text{safe},k}$.
Since $x(1)\in\chi_{\text{safe},k}$, by our induction hypothesis above, there
exists $\\{u_{1}(x(1)),...,u_{k}(x(k))\\}\in U_{q(1)}\times\cdots\times
U_{q(k)}$, such that for all $\\{w_{1},...,w_{k}\\}\in
W_{q(1)}\times\cdots\times W_{q(k)}$, $x(j+1)\in X$ for all $0\leq j\leq k$.
That is, there exists $\\{u_{0}(x(0)),u_{1}(x(1)),...,u_{k}(x(k))\\}\in
U_{q(0)}\times U_{q(1)}\times\cdots\times U_{q(k)}$, such that for all
$\\{w_{0},w_{1},...,w_{k}\\}\in W_{q(0)}\times W_{q(1)}\times\cdots\times
W_{q(k)}$, $x(j+1)\in X$, for $0\leq j+1\leq k+1$. This proves the statement
for $k+1$ and hence completes the induction step. $\blacksquare$

Proposition 4 For each $k=0,1,...$, it holds that
$\chi_{\text{safe},k+1}\subset\chi_{\text{safe},k}$, i.e.,
$\chi_{\text{safe},k}$ is a decreasing sequence of sets. Thereby,
$\chi_{\text{safe},\infty}=\lim_{k\to\infty}\chi_{\text{safe},k}$ exists111In
the set-theoretic sense [48]. and satisfies
$\chi_{\text{safe},\infty}\subset\chi_{\text{safe},k}$ for all $k$.

Proof: Suppose $x\in\chi_{\text{safe},k+1}$ for some $k\in\mathbb{Z}_{\geq
0}$. According to (13a), there exists $u(x)$ such that
$(x,u(x))\in\Lambda_{\text{safe},k}$. Then, by the definition (13), there
exist $q_{k}\in\\{1,...,n_{q}\\}$ and $i_{k}\in\\{1,...,r_{k}\\}$ such that
$x\in\chi_{\text{safe},k}$. This implies
$\chi_{\text{safe},k+1}\subset\chi_{\text{safe},k}$, and thus
$\chi_{\text{safe},k}$ is decreasing. Thus, the sequence of sets
$\chi_{\text{safe},k}$ converges and the limit exists [48]. $\blacksquare$

On the basis of Proposition 4, we now define $\chi_{\text{safe}}$ as
$\chi_{\text{safe}}=\lim_{k\to\infty}\chi_{\text{safe},k}=\bigcap_{k=0}^{\infty}\chi_{\text{safe},k}$.
With this definition, we have the following propositions, which verify that
the desired property stated at the beginning of Section 3.1 is fulfilled by
$\chi_{\text{safe}}$.

Proposition 5 Suppose at a given time instant $t\in\mathbb{Z}_{\geq 0}$, (9)
is feasible and $u(t)$ is determined as the solution of (9), then at $t+1$,
$x(t+1)\in X$.

Proof: As (9) is feasible and $u(t)$ is determined as the solution of (9) at
$t$, we have $x(t+1)\in\chi_{\text{safe}}$ as the constraint of (9) is
satisfied. Additionally,
$\chi_{\text{safe}}=\bigcap_{k=0}^{\infty}\chi_{\text{safe},k}\subset\chi_{\text{safe},0}=X$,
thus $x(t+1)\in X$. $\blacksquare$

Proposition 6 Suppose $x\in\chi_{\text{safe}}$ and the following condition
holds:

$\displaystyle\begin{split}\text{Proj}_{x}\Bigg{[}\bigcap_{k=0}^{\infty}\Lambda_{\text{safe},k}\Bigg{]}=\bigcap_{k=0}^{\infty}\chi_{\text{safe},k}.\end{split}$
(15)

Then, there exists $u\in U_{q(x)}$ such that
$f_{q(x)}(x,u,w)\in\chi_{\text{safe}}$ for any $w\in W_{q(x)}$.

Proof: Given
$x\in\chi_{\text{safe}}=\bigcap_{k=0}^{\infty}\chi_{\text{safe},k}$, suppose
(15) holds, then there exists a control $u(x)$, such that
$(x,u(x))\in\bigcap_{k=0}^{\infty}\Lambda_{\text{safe},k}$, i.e.,
$(x,u(x))\in\Lambda_{\text{safe},k}$ for all $k\in\mathbb{Z}_{\geq 0}$. For
each $k$, by the definition (13), there exists $i_{k}\in\\{1,...,r_{k}\\}$
such that $x\in P_{q(x)}$, $u(x)\in U_{q(x)}$, and
$f_{q(x)}(x,u(x),W_{q(x)})\subset\chi^{i_{k}}_{\text{safe},k}\subset\bigcup_{i=1}^{r_{k}}\chi^{i}_{\text{safe},k}=\chi_{\text{safe},k}$.
Since this holds for all $k\in\mathbb{Z}_{\geq 0}$, this implies
$f_{q(x)}(x,u(x),W_{q(x)})\subset\bigcap_{k=0}^{\infty}\chi_{\text{safe},k}=\chi_{\text{safe}}$.
$\blacksquare$

Corollary 7 Suppose (15) holds, $x(0)\in\chi_{\text{safe}}$ and the RAG
operates based on (9), then (9) is recursively feasible at all
$t\in\mathbb{Z}_{\geq 0}$ and the safety requirements in (6) are satisfied for
all $t\in\mathbb{Z}_{\geq 0}$.

Proof: This result follows directly from Propositions 4 and 5. $\blacksquare$

Note that (15) does not always hold (in general,
$\text{Proj}_{x}[\bigcap_{k=0}^{\infty}\Lambda_{\text{safe},k}]\\\
\subset\bigcap_{k=0}^{\infty}\chi_{\text{safe},k}$), it does hold in many
cases. The following proposition classifies the circumstances where (15) is
guaranteed to hold.

Proposition 8 Suppose there exists $k_{0}$ such that the sets
$\Lambda_{\text{safe},k}$ are nonempty and compact for all $k\geq k_{0}$, then
$\text{Proj}_{x}\Bigg{[}\bigcap_{k=0}^{\infty}\Lambda_{\text{safe},k}\Bigg{]}=\bigcap_{k=0}^{\infty}\chi_{\text{safe},k}$.

Proof: The proof follows from Proposition 4 in [5]. $\blacksquare$

Note that the assumption in Proposition 8 holds in many examples, e.g.,
continuous systems with additive disturbances, where $X$, $U$ and $W$ are
compact sets. See [5] for more examples.

###  3.2 Offline and online computations

According to the definition of
$\chi_{\text{safe}}=\bigcap_{k=0}^{\infty}\chi_{\text{safe},k}$, the exact
determination of $\chi_{\text{safe}}$ relies on the iterative computation of
set $\chi_{\text{safe},k}$ according to (13) with $k\to\infty$. In practice,
as $\chi_{\text{safe},k}$ is a decreasing sequence of sets, we will use a
compact approximation of $\chi_{\text{safe}}$ in the RAG, which is denoted as
$\tilde{\chi}_{\text{safe},k^{\prime}}$, where $k^{\prime}$ being sufficiently
large.

To facilitate numerical computations, we introduce a compact set
$\mathcal{X}=\\{x\in\mathbb{R}^{n}:\tilde{H}x\leq\tilde{h}\\}$ that covers the
operating range of the system. This way, the safety requirements in (7) can be
represented in the form of $x(t)\in X\cap\mathcal{X}$. We note that in
practice, $\mathcal{X}$ can be chosen as a sufficiently large set and only $X$
is used to represent actual safety requirements. We also consider a collection
of polyhedral sets $\\{\tilde{P}_{q}\\}_{q=1}^{n_{q}}$ such that
$\tilde{P}_{q}=P_{q}\cap\mathcal{X}$ for all $q=1,...,n_{q}$.

With $\mathcal{X}$ and $\tilde{P}_{q}$, $\tilde{\chi}_{\text{safe},k}$ as the
approximation of ${\chi}_{\text{safe},k}$ can be numerically computed using
Algorithm 1. The set operations in Algorithm 1, including set
intersection/union, Minkowski sum $\oplus$, Pontryagin difference $\sim$ and
projection of polytopes (line 4), can be computed using the Multi-Parametric
Toolbox 3 (MPT3) [21].

Algorithm 1 Offline computation of $\tilde{\chi}_{\text{safe},k}$

Input: $A_{q,j},B_{q,j},f_{q,j},E_{q,j},U_{q},W^{\text{a}}_{q},\tilde{P}_{q}\
(\forall q\in\\{1,...,n_{q}\\},\forall
j\in\\{1,...,n^{\text{p}}_{q}\\}),\tilde{\chi}_{\text{safe},k-1}=\bigcup_{i=1}^{r_{k-1}}\tilde{\chi}_{\text{safe},k-1}^{i}$
Output:$\ \tilde{\chi}_{\text{safe},k}$

1:$\chi_{\text{temp}}\leftarrow\emptyset$

2:for each mode $q\in\\{1,...,n_{q}\\}$ do

3: for each $i\in\\{1,...,r_{k-1}\\}$ do

4:
$R\leftarrow\text{Proj}_{x}\\{(x,u):x\in\tilde{\chi}_{\text{safe},k-1}\cap\tilde{P}_{q},u\in
U_{q},\text{s.t.}\begin{bmatrix}\bar{A}_{q}&\bar{B}_{q}\end{bmatrix}\begin{bmatrix}x\\\
u\end{bmatrix}\in\prod_{j=1}^{n^{\text{p}}_{q}}(\tilde{\chi}^{i}_{\text{safe},k-1}\sim
E_{q,j}W_{q}^{\text{a}}-f_{q,j})\\}$

5: $\chi_{\text{temp}}\leftarrow\chi_{\text{temp}}\cup R$

6: end for

7:end for

8:$\tilde{\chi}_{\text{safe},k}\leftarrow\chi_{\text{temp}}$

After $\tilde{\chi}_{\text{safe},k^{\prime}}$ has been computed using
Algorithm 1, the constraints in the RAG online optimization problem (9) are
approximated as follows,

$\displaystyle\begin{split}&A_{q(t)}(w^{\text{p}}_{q(t)})x(t)+B_{q(t)}(w^{\text{p}}_{q(t)})u+E_{q(t)}(w^{\text{p}}_{q(t)})w^{\text{a}}_{q(t)}+f_{q(t)}(w^{\text{p}}_{q(t)})\\\
&\in\tilde{\chi}_{\text{safe},k^{\prime}},\ \forall w^{\text{p}}_{q(t)}\in
W^{\text{p}}_{q(t)},\forall w^{\text{a}}_{q(t)}\in
W^{\text{a}}_{q(t)},\tilde{\chi}_{\text{safe},k^{\prime}}=\bigcup_{i=1}^{r_{k}^{\prime}}\tilde{\chi}_{\text{safe},k^{\prime}}^{i}.\end{split}$
(16)

In order to deal with the parametric uncertainties, we further tighten
(LABEL:equ:OriginalCtr) as follows,

$\displaystyle\begin{split}&A_{q(t)}(w^{\text{p}}_{q(t)})x(t)+B_{q(t)}(w^{\text{p}}_{q(t)})u+E_{q(t)}(w^{\text{p}}_{q(t)})w^{\text{a}}_{q(t)}\\\
&+f_{q(t)}(w^{\text{p}}_{q(t)})\in\tilde{\chi}^{i}_{\text{safe},k^{\prime}},\
\forall w^{\text{p}}_{q(t)}\in W^{\text{p}}_{q(t)},\forall
w^{\text{a}}_{q(t)}\in W^{\text{a}}_{q(t)},\\\ &\text{for some
}i\in\\{1,...,r_{k^{\prime}}\\}.\end{split}$ (17)

Note that (LABEL:equ:tightenCtr1) requires there exists control input $u(t)$
such that
$f_{q(t)}(x(t),u(t),W_{q(t)})\subset\tilde{\chi}^{i}_{\text{safe},k^{\prime}}\subset\tilde{\chi}_{\text{safe},k^{\prime}}$
for some $i\in\\{1,...,r_{k}^{\prime}\\}$, which is more strict than the
constraints (LABEL:equ:OriginalCtr), where $u(t)$ is required to make
$f_{q(t)}(x(t),u(t),W_{q(t)})\subset\tilde{\chi}_{\text{safe},k^{\prime}}=\bigcup_{i=1}^{r_{k}^{\prime}}\tilde{\chi}_{\text{safe},k^{\prime}}^{i}$
for any $i\in\\{1,...,r_{k}^{\prime}\\}$. This way the feasible solution of
the optimization problem in (9) with the tightened constraint
(LABEL:equ:tightenCtr1) satisfies the constraint (7). Using Lemma 1, we can
write (LABEL:equ:tightenCtr1) in the form as follows,

$\displaystyle\begin{split}&A_{q(t),j}x(t)+B_{q(t),j}u+f_{q(t),j}\in\tilde{\chi}^{i}_{\text{safe},k^{\prime}}\sim
E_{q(t),j}W^{\text{a}}_{q(t)},\\\ &\text{for all
}j=\\{1,...,n^{\text{p}}_{q(t)}\\},\text{for some
}i\in\\{1,...,r_{k^{\prime}}\\}.\end{split}$ (18)

Since the Pontryagin difference of a polytope
$\tilde{\chi}^{i}_{\text{safe},k^{\prime}}$ and a polytope
$E_{q(t),j}W^{\text{a}}_{q(t)}$ is still a polytope [25], and can be expressed
as follows:

$\displaystyle\begin{split}&\tilde{\chi}^{i}_{\text{safe},k^{\prime}}\sim
E_{q(t),j}W^{\text{a}}_{q(t)}=\left\\{x\in\mathbb{R}^{n}:H^{i}_{j}x\leq
h^{i}_{j}\right\\}.\end{split}$ (19)

Therefore, the set-inclusion constraints in (LABEL:equ:tightenCtr2) can be
expressed using the following set of constraints:

$\displaystyle\begin{split}&H^{i}_{j}\left(A_{q(t),j}x(t)+B_{q(t),j}u+f_{q(t),j}\right)\leq
h^{i}_{j}+M(1-\delta_{i}),\\\ &\forall
j\in\\{1,...,n^{\text{p}}_{q(t)}\\},\\\\[4.0pt]
&\sum_{i=1}^{r_{k}}\delta_{i}\geq 1,\,\delta_{i}\in\\{0,1\\}.\,\end{split}$
(20)

Hence, the RAG online optimization problem (9) can be solved as a Mixed-
Integer Quadratic Programming (MIQP) problem using standard MIQP solvers.

##  4 Safe reinforcement learning using RAG

###  4.1 Safe Online Learning

In this section, we integrate the proposed RAG with a RL algorithm to achieve
safe learning that enforces all-time safety constraint satisfaction. The
comparison between conventional RL and proposed safe RL scheme is illustrated
in Fig. 2. As shown in Fig. 2a, conventional RL agent continuously interacts
with the environment to optimize its action via trial-and-error by maximizing
its accumulated reward (or minimizing accumulated cost). Specifically, at each
time instant $t\in\mathbb{Z}_{\geq 0}$, the agent first takes a measurement of
the state $x(t)\in\mathbb{R}^{n}$, then applies a control action
$u_{\phi}(t)\in\mathbb{R}^{m}$ based on the current control policy
$\pi:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$, and finally collects a reward
$R(t)\in\mathbb{R}$. The control policy $\pi$ is learned and keeps evolving
from the collected experience $\\{x(t),u_{\phi}(t),R(t)\\}$ to maximize the
long-term reward $\tilde{R}=\sum_{t=0}^{\infty}\gamma^{t}R(t)$, where
$\gamma\in(0,1)$ is a discount factor. Apparently, the RL agent needs to
explore within the action space to maximize the long-term reward, which may
cause unsafe behaviors (i.e., violation of certain safety constraints) during
the learning process. This feature hinders the application of RL to online
policy learning/evolving of actual engineering systems, which motivates us to
propose a safe RL framework.

The proposed safe RL framework is illustrated in Fig. 2b. The RAG is
introduced between the RL agent and the environment where the control policy
acts on, e.g., the plant in a conventional control setting. This way, the RAG
is able to monitor the control signal $u_{\phi}(t)$ generated by the RL agent
and corrects the ones that may cause unsafe behaviors, i.e. violate safety
constraints, to the safe action $u^{\text{safe}}(t)$. As the RAG is an add-on
module, no restriction is imposed for the base RL module to be combined with
the RAG to achieve safe RL. In this paper, we consider the Neural-Fitted
Q-learning (NFQ) as the base RL module to demonstrate the ability to achieve
safe learning with RAG.

Figure 2: Schematic diagram of conventional RL (a) and safe RL with RAG (b).

In NFQ, we use a neural network to approximate the Q-function, and we update
this Q-function approximation using the following formulas,

$\displaystyle Q(x(t),u(t))\leftarrow$
$\displaystyle\lambda\tilde{Q}(x(t),u(t))+(1-\lambda)\big{(}R(x(t),u(t))$
$\displaystyle+\gamma\tilde{V}(x(t+1))\big{)},$ (21a)
$\tilde{V}(x)=\max_{u\in{U}}\tilde{Q}(x,u),$ (21b)

where $\lambda\in(0,1)$ is the learning rate, $\gamma\in(0,1)$ is the discount
factor, and $\tilde{Q}(x,u)$ represents the approximated Q-values extracted
from the neural network. We formalize the proposed safe RL scheme in Algorithm
2. An $\epsilon$-greedy action selection rule is used by the RL agent to
balance exploration and exploitation in lines 7-11. The control action is
modified in line 12 where any action that may lead to constraint violation
will be modified to $u^{\text{safe}}$ by the RAG via solving (9). The reward
is collected in line 14 and the Q-value is updated in lines 15-17, new
experience is stored into the replay buffer $B$ in line 18, which is used for
training the Q-function NN $\tilde{Q}(x,u)$ in line 23. In particular, in line
17 the Q-value of the current state $x(t)$ and nominal control $u(t)$,
$Q(x(t),u(t))$ is updated with the reward $R(x(t),u^{\text{safe}}(t))$ brought
by the safe action $u^{\text{safe}}(t)$. Furthermore, in line 18, we store the
tuple of current state, nominal control (not the safe action
$u^{\text{safe}}(t)$) and updated Q-value, $(x(t),u(t),Q(x(t),u(t)))$, to the
replay buffer. This way, the agent is able to explore all possible actions in
the action space $U$ as action modification by the RAG is not perceived by the
agent. This may be beneficial as this will potentially improve convergence to
the optimal control policy [22].

Algorithm 2 Safe RL algorithm

Input Initialized Q-value NN $\tilde{Q}(x,u)$, empty replay buffer $B$, and
the maximum trajectory number within a training episode $N$.

1:for each training episode do

2: $n_{t}\leftarrow 0$

3: while $n_{t}<N$ do

4: Randomly generate an initial state $x(0)\in\chi_{\text{safe}}$

5: $t\leftarrow 0$

6: while $t<T$ do

7: if $rand()<\epsilon$ then

8: $u(t)$ takes a random value within $U$

9: else

10: $u(t)\in\operatorname*{arg\,max}_{u\in U}\tilde{Q}(x(t),u)$

11: end if

12: Solve (9) to modify $u(t)$ to $u^{\text{safe}}(t)$

13: Apply $u^{\text{safe}}(t)$ to system

14: Observe next state $x(t+1)$ and reward $R(x(t),u^{\text{safe}}(t))$

15: Update Q value:

16: $\tilde{V}(x(t+1))=\max_{u\in U}\tilde{Q}(x(t+1),u)$

17:
$Q(x(t),u(t))\leftarrow\lambda\tilde{Q}(x(t),u(t))+(1-\lambda)\big{(}R(x(t),u^{\text{safe}}(t))+\gamma\tilde{V}(x(t+1))\big{)}$

18: Store $(x(t),u(t),Q(x(t),u(t)))$ to replay buffer $B$

19: $t\leftarrow t+1$

20: end while

21: $n_{t}\leftarrow n_{t}+1$

22: end while

23: Train Q-value NN $\tilde{Q}(x,u)$ using sampled data from replay buffer
$B$

24:end for

###  4.2 Explicit safe RL policy

The safe RL policies obtained from Algorithm 2 require implementing together
with the RAG to guarantee safety, i.e., numerically solving on-line MIQP
optimization problem (9) at each time instant $t$, based on the value of the
current state $x(t)$ and the nominal control action selected by the safe RL
policy $u_{\phi}(t)$. The computational cost of solving MIQP problems has been
progressively reduced by recent advances in microcontroller and optimization
algorithm [4, 38, 24]. However, solving online optimization problem still
prevents the application of the RAG in many contexts, especially in safety
critical applications, where the software certification may be required and
conducting one with optimization solver is nontrivial.

The idea of the explicit safe RL policy is to learn the mapping from all
states $x(t)\in\chi_{\text{safe}}$ to $u^{\text{safe}}(t)$, which is solved by
(9), via imitation learning [42]. In particular, we define a policy as a map
from states $x(t)$ to the safe action $u^{\text{safe}}(t)$, i.e.

$\pi:x(t)\mapsto u^{\text{safe}}(t).$ (22)

This map is determined by numerically solving (9). In what follows, we pursue
an explicit approximation of $\pi$, denoted by $\tilde{\pi}$, exploiting the
approach called imitation learning.

The objective of imitation learning is to imitate expert’s behavior via
learning a control policy from expert’s behavior. In this paper, we treat the
map $\pi$ as the expert. The imitation learning can be formulated as the
following standard supervised learning problem,

$\tilde{\pi}\in\operatorname*{arg\,min}_{\pi_{\theta}}\mathbb{E}_{\bar{\mathbb{s}}\sim\mathbb{P}(\bar{\mathbb{s}}|\pi)}[\mathcal{L(\pi(\bar{\mathbb{s}}),\tilde{\pi}(\bar{\mathbb{s}}})],$
(23)

where $\bar{\mathbb{s}}$ represents the pair $(x,u^{\text{safe}})$,
$\pi_{\theta}$ presents a policy parameterized by the optimization variable
$\theta$ (e.g., the weights of a neural network), $\mathcal{L}$ is a loss
function penalizing the difference between expert’s policy $\pi$ and its
approximation $\tilde{\pi}$. The notation
$\mathbb{E}_{\bar{\mathbb{s}}\sim\mathbb{P}(\bar{\mathbb{s}}|\pi)}(\cdot)$ is
defined as

$\mathbb{E}_{\bar{\mathbb{s}}\sim\mathbb{P}(\bar{\mathbb{s}}|\pi)}(\cdot)=\int(\cdot)\mathbb{d}\mathbb{P}(\bar{\mathbb{s}}|\pi).$
(24)

Note that the expectation in procedure (23) is with respect to the probability
distribution $\mathbb{P}(\bar{\mathbb{s}}|\pi)$ of the data $\bar{\mathbb{s}}$
that is determined by the expert policy $\pi$, which is essentially the
empirical distribution of $\bar{\mathbb{s}}$ in the pre-collected dataset. The
training of $\tilde{\pi}$ can be conducted either offline or online, for
instance, a neural network based $\tilde{\pi}$ may evolve over time by
collecting data $(x(t),u^{\text{safe}}(t))$ online.

The key feature of applying $\tilde{\pi}$ as a control policy is that the
computational cost will be reduced by avoiding solving MIQP in (9). Using
$\tilde{\pi}$ as a control policy has a drawback in that constraint violation
may occur. However, in many applications a slight constraint violation is
tolerable [31], in such cases, a small error between $\pi$ and $\tilde{\pi}$
may be sufficient enough to ensure the system safety.

##  5 Numerical examples

In this section, we apply the proposed RAG, safe RL and explicit safe RL
schemes to address a soft-landing control problem of a mass-spring-damper
system. We first demonstrate the effectiveness of the RAG enforcing constraint
satisfaction with a nominal control policy in Section 5.1. Then, the safe RL
framework proposed in Section 4.1 is validated in Section 5.2. The
performances, e.g. control performances and online computational time, of the
explicit safe RL policy are reported in Section 5.3.

###  5.1 Soft-landing control of a mass-spring-damper system using the RAG

Consider a mass-spring-damper system (MSD) in Fig. 3 where the mass moves
within a bounded region with the controlled external force $F$. We impose
velocity constraints to the mass when it approaches the boundaries to achieve
soft-landing. This soft-landing control of the MSD can be used to model motion
control of aircarft systems [44], manipulation of robot arms [39, 26] and
interaction of mobile robots [53]. In particular, in this paper we consider an
automated tool-tray transfer systems delivering tools/materials between
working stations within an automated assembly line.

Taking the $x$-axis origin at the neutral position of the spring. The dynamics
of the mass-spring-damper system can be modeled as

$m(w^{\text{p}})\ddot{x}=F-c\dot{x}-F_{\text{s}}+w^{\text{a}},$ (25a)
$F_{\text{s}}=\begin{cases}k_{1}x,&\text{if $-x_{\text{m}}\leq x\leq
x_{\text{m}}$ }\\\
k_{2}x+(k_{1}-k_{2})x_{\text{m}},&\text{$x>x_{\text{m}}$}\\\
k_{2}x-(k_{1}-k_{2})x_{\text{m}},&\text{$x<-x_{\text{m}}$}\end{cases}$ (25b)

where $x$ represents the position of the mass $m$ and $c=0.8\ \text{Ns/m}$
represents the damping coefficient. The nonlinearity of the spring force
$F_{\text{s}}$ is approximated by a piecewise affine model as (25b), where
$x_{\text{m}}=1.75\ \text{m}$ is the spring force mode separation point,
$k_{1}=1\ \text{N/m}$ and $k_{2}=0.8\ \text{N/m}$ are spring stiffness for
corresponding mode, respectively. In particular, we assume that the controlled
external force can only attract but not repel the mass, i.e. $F\geq 0$. This
represents the typical scenarios where $F$ is generated by an electromagnetic
system [11]. $w^{\text{a}}\in W^{\text{a}}=\\{w^{\text{a}}\in\mathbb{R}:\
-1\leq w^{\text{a}}\leq 1\\}$ represents a unknown but bounded external
control input disturbance. We also assume the mass
$m\in[m_{\text{min}},m_{\text{max}}]$ is unknown but bounded, taking into
account of the fact that the mass of the materials/tools may vary. The
uncertainty of $m$ is modeled as parametric disturbances of the system, i.e.
$w^{\text{p}}=[w^{\text{p}}_{1},w^{\text{p}}_{2}]^{T}\in
W^{\text{p}}=\\{w^{\text{p}}_{1}+w^{\text{p}}_{2}=1,w^{\text{p}}_{1},w^{\text{p}}_{2}\geq
0\\}$ and
$\frac{1}{m}=w^{\text{p}}_{1}\frac{1}{m^{\text{min}}}+w^{\text{p}}_{2}\frac{1}{m^{\text{max}}}$.

By using Euler discretization method with sampling time step
$T_{\text{s}}=0.1\ \text{s}$, equation (25) becomes

$\displaystyle\begin{split}&z(t+1)=\\\
&\begin{cases}\begin{bmatrix}1&T_{\text{s}}\\\
-w^{\text{p}}_{1}\frac{k_{1}T_{\text{s}}}{m^{\text{min}}}-w^{\text{p}}_{2}\frac{k_{1}T_{\text{s}}}{m^{\text{max}}}&1-w^{\text{p}}_{1}\frac{cT_{\text{s}}}{m^{\text{min}}}-w^{\text{p}}_{2}\frac{cT_{\text{s}}}{m^{\text{max}}}\end{bmatrix}z(t)+\\\
\begin{bmatrix}0\\\
w^{\text{p}}_{1}\frac{T_{\text{s}}}{m^{\text{min}}}+w^{\text{p}}_{2}\frac{T_{\text{s}}}{m^{\text{max}}}\end{bmatrix}u(t)+\begin{bmatrix}0\\\
w^{\text{p}}_{1}\frac{T_{\text{s}}}{m^{\text{min}}}+w^{\text{p}}_{2}\frac{T_{\text{s}}}{m^{\text{max}}}\end{bmatrix}w^{\text{a}}(t),\hfill\\\
\text{if $-x_{\text{m}}\leq x\leq x_{\text{m}}$ }\\\ \\\
\begin{bmatrix}1&T_{\text{s}}\\\
-w^{\text{p}}_{1}\frac{k_{2}T_{\text{s}}}{m^{\text{min}}}-w^{\text{p}}_{2}\frac{k_{2}T_{\text{s}}}{m^{\text{max}}}&1-w^{\text{p}}_{1}\frac{cT_{\text{s}}}{m^{\text{min}}}-w^{\text{p}}_{2}\frac{cT_{\text{s}}}{m^{\text{max}}}\end{bmatrix}z(t)+\\\
\begin{bmatrix}0\\\
w^{\text{p}}_{1}\frac{T_{\text{s}}}{m^{\text{min}}}+w^{\text{p}}_{2}\frac{T_{\text{s}}}{m^{\text{max}}}\end{bmatrix}u(t)+\begin{bmatrix}0\\\
w^{\text{p}}_{1}\frac{T_{\text{s}}}{m^{\text{min}}}+w^{\text{p}}_{2}\frac{T_{\text{s}}}{m^{\text{max}}}\end{bmatrix}w^{\text{a}}(t)+\\\
\begin{bmatrix}0\\\
-w^{\text{p}}_{1}\frac{T_{\text{s}}(k_{1}-k_{2})x_{\text{m}}}{m^{\text{min}}}-w^{\text{p}}_{2}\frac{T_{\text{s}}(k_{1}-k_{2})x_{\text{m}}}{m^{\text{max}}}\end{bmatrix},\hfill\\\
\text{if $x>x_{\text{m}}$ }\\\ \\\ \begin{bmatrix}1&T_{\text{s}}\\\
-w^{\text{p}}_{1}\frac{k_{2}T_{\text{s}}}{m^{\text{min}}}-w^{\text{p}}_{2}\frac{k_{2}T_{\text{s}}}{m^{\text{max}}}&1-w^{\text{p}}_{1}\frac{cT_{\text{s}}}{m^{\text{min}}}-w^{\text{p}}_{2}\frac{cT_{\text{s}}}{m^{\text{max}}}\end{bmatrix}z(t)+\\\
\begin{bmatrix}0\\\
w^{\text{p}}_{1}\frac{T_{\text{s}}}{m^{\text{min}}}+w^{\text{p}}_{2}\frac{T_{\text{s}}}{m^{\text{max}}}\end{bmatrix}u(t)+\begin{bmatrix}0\\\
w^{\text{p}}_{1}\frac{T_{\text{s}}}{m^{\text{min}}}+w^{\text{p}}_{2}\frac{T_{\text{s}}}{m^{\text{max}}}\end{bmatrix}w^{\text{a}}(t)+\\\
\begin{bmatrix}0\\\
w^{\text{p}}_{1}\frac{T_{\text{s}}(k_{1}-k_{2})x_{\text{m}}}{m^{\text{min}}}+w^{\text{p}}_{2}\frac{T_{\text{s}}(k_{1}-k_{2})x_{\text{m}}}{m^{\text{max}}}\end{bmatrix},\hfill\\\
\text{if $x<-x_{\text{m}}$ }\end{cases}\end{split}$ (26)

where $z=[x,\dot{x}]^{T}$ and $u=F$.

Figure 3: Schematics of a mass-spring-damper system controlled by an external
attractive force $F$.

To avoid high velocity contacts which lead noise and wear, as the mass
approaches the boundary ($x=d$), which represents the position of a working
station, it is desirable to reduce the velocity of the mass gradually to
achieve soft-landing, which can be represented by imposing the following
constraint

$\displaystyle\begin{split}\begin{cases}-\epsilon-\beta(d-x)\leq\dot{x}\leq\epsilon+\beta(d-x),&\text{if
$x_{\text{c}}\leq x\leq d$ }\\\ -v_{\text{max}}\leq\dot{x}\leq
v_{\text{max}},&\text{if $x\leq x_{\text{c}}$ }\end{cases}\end{split}$ (27)

The constraint (27) enforces that when the mass contacts the boundary ($x=d$),
the velocity is within $[-\epsilon,\epsilon]$. This range is gradually relaxed
by $\beta$ as the mass moves away from the boundary. Note that the soft-
landing constraint is not required when $x\leq x_{\text{c}}$, which leads (27)
to be a non-convex constraint in general.

We also enforce the following constraint to prevent the mass moving too far
away from the working station which can be represented as

$\displaystyle\begin{split}x\geq 0.\end{split}$ (28)

In addition, the constraint on the control input $F$ is defined as

$\displaystyle\begin{split}0\leq F\leq F_{\text{max}}.\end{split}$ (29)

A RL control policy is employed as the nominal control
$u_{\phi}=\phi(x,\dot{x},x^{\text{ref}})$, the reward function of the RL agent
$R$ is designed as follows,

$R=w_{1}R_{1}+w_{2}R_{2},$ (30a) $R_{1}=-(x-x^{\text{ref}})^{2}$ (30b)
$R_{2}=\begin{cases}-(\dot{x}-v_{\text{b}})^{2},&\text{if
\eqref{equ:SoftLandingConstr} is violated}\\\ 0,&\text{if
\eqref{equ:SoftLandingConstr} is satisfied}\end{cases}$ (30c)

where $w_{1}$ and $w_{2}$ are tuning weightings. $x^{\text{ref}}$ is the
reference position for the mass, in particular, we assume
$x^{\text{ref}}\in\\{0,d\\}$. $v_{\text{b}}$ represents the closest velocity
constraint value determined by (27) corresponding to the current position $x$,
which can be defined as follows,

$\displaystyle\begin{split}v_{\text{b}}=\begin{cases}-\epsilon-\beta(d-x),&\text{if
$x_{\text{c}}\leq x\leq d$ and $\dot{x}\leq 0$}\\\
\epsilon+\beta(d-x),&\text{if $x_{\text{c}}\leq x\leq d$ and $\dot{x}>0$}\\\
-v_{\text{max}},&\text{if $x\leq x_{\text{c}}$ and $\dot{x}\leq 0$}\\\
v_{\text{max}},&\text{if $x\leq x_{\text{c}}$ and
$\dot{x}>0$}\end{cases}\end{split}$ (31)

The rest of parameters used in the system model and constraints are selected
as $d=5\ \text{m}$, $v_{\text{max}}=5\ \text{m/s}$, $F_{\text{max}}=10\
\text{N}$, $x_{\text{c}}=3.3\ [\text{m}]$, $\epsilon=0.5\ \text{m/s}$,
$\beta=-2.95\ \text{s}^{-1}$, $m_{\text{min}}=0.5\ \text{kg}$ and
$m_{\text{max}}=1.5\ \text{kg}$.

Apparently, the satisfactions of constraints (27) and (28) are not guaranteed
with the nominal RL control policy $u_{\phi}$. To enforce such safety
constraints, the RAG is designed considering the safe set $X$ based on (27)
and (28), which is in general non-convex and shown in Fig. 7. In this paper,
$\chi_{\text{safe}}$ is computed with MPT3 toolbox offline [21]. The online
MIQP optimization problem in (9) is solved by Gurobi [18].

We consider the initial condition $[x(0),\dot{x}(0)]^{T}=[0,0]^{T}$ and the
desired position of the mass is shown as a periodical signal between 0 and 5
(black solid line) in Fig. 5(a). We set $k=60$ and compute the safe set
$\chi_{\text{safe}}$ using Algorithm 1. The set $\chi_{\text{safe}}$ and state
trajectory using the RAG are shown in Fig. 4. We can observe that the RAG
keeps the state trajectory always within $\chi_{\text{safe}}$ while the
trajectory with the nominal RL control policy gets out of
$\chi_{\text{safe}}$, which leads constraint violations shown in Fig. 5.

Figure 4: Comparison of state trajectory with the RAG and nominal control
policy under adversarial additive disturbances $w^{\text{a}}$ and
computational results of $\chi_{\text{safe}}$. Figure 5: Simulation results of
states and constraints with the RAG and nominal control. (a) The displacement
of the mass. (b) The velocity of the mass. Figure 6: Control inputs
comparisons of the RAG and nominal control policy, results of mode transitions
and disturbances. (a) Control inputs. (b) Mode. (c) Parametric disturbances.
(d) Additive disturbances.

The comparisons of states and control inputs using the RAG and that using the
nominal RL control policy are presented in Figs. 5 and 6. The safety
constraints in (27) and (28) (red dash-dotted line in Fig. 5) are violated
with the nominal control policy. In contrast to the results with nominal
control, as shown in Figs. 5 and 6(a), the RAG enforces constraint
satisfaction by modifying the nominal control $u_{\phi}$ in advance to
decrease the mass velocity when it approaches the boundary ($x=5\ \text{[m]}$)
and origin. This way the mass can contact the boundary with a smoother way and
also will not move away cross the origin. The mode transition can be safely
handled by using the RAG, which is shown in Fig. 6(b). The parametric and
additive disturbances applied during simulation are shown in Figs. 6(c) and
6(d). To validate the performances of the RAG, in this test, the adversarial
additive disturbances are imposed where the disturbance tries its best to
violate constraints, i.e., apply maximum disturbance in the same moving
direction of the mass.

Figure 7: Constraint violation tests over 500 experiments with randomly
generated parametric and additive disturbances.

To verify that the RAG can guarantee constraint satisfaction in the presence
of any disturbance realizations, we run $500$ simulation experiments with
randomized parametric and additive disturbance inputs $w(k)\in W$. The
simulated trajectories are shown in Fig. 7. It can be observed that no
constraint violation occurs in these 500 experiments.

###  5.2 Safe reinforcement learning with the RAG

In this section, we demonstrate the effectiveness of the proposed safe RL
framework by applying it to achieve safe online training for the nominal
control policy used in the last section. We assume that the parameter value of
the actual mass $m$ and distance between the origin and boundary $d$ are
different from the ones used to design the nominal RL control policy $\phi$.
This is common in the real implementations as either $m$ and $d$ are difficult
to measure precisely or their values vary during the system operation. Our
goal is to online evolve the control policy $\phi$ safely, i.e., without
violating constraints (27) and (28), to adapt to the new parameters $m=1.3\
\text{kg}$ and $d=5.5\ \text{m}$ via RL.

We use the same reward function in (30) to conduct the online training. The
comparison between training histories of conventional and safe RL algorithms
are shown in Fig. 8. We can observe from 8(b) that the constraint violation
rate of conventional RL keeps decreasing as the learning proceeds. In
contrast, no constraint violation is exhibited for the safe RL with the RAG
during the entire training process. Moreover, by using the RAG, the second
term in the reward function $R_{2}$ is always equal to zero, which leads to
smaller expectation and variation values compared with the ones with the
conventional RL, as shown in Figure 8(a).

Figure 8: Training histories of conventional RL and safe RL. (a) Constraint
violation rates of each episode. (b) Average reward values of each episode.
Solid lines represent average values and shaded areas represent the standard
deviation values over 20 experiments.

The validation results of trained policies are shown in Figs. 9 to 11.
Although the tracking performances of reference are both satisfactory with the
conventional RL (green dash line in Fig. 10) and safe RL (blue solid line in
Fig. 10), constraint satisfaction is enforced by the control policy with the
RAG, while under the control of conventional RL policy, there are still
occasional constraint violations. This can also be verified by Fig. 9 as the
state trajectory is always within $\chi_{\text{safe}}$. Similar results as in
section 5.1 of control inputs and mode transitions can be observed in Fig. 11,
where the RAG modifies the nominal control actions preventing constraint
violation from happening.

We remark that the fact that our safe RL algorithm based on the use of the RAG
guarantees no safety constraint violation during both the training and the
operating phases yields that it can be used for onboard applications. For
instance, it can be used to continuously improve the performance of a
controller during its onboard operation.

Figure 9: Comparisons of trajectories with controls of baseline RL control
policy (Baseline), updated RL control policy (RL) and updated RL control
policy with the RAG (RL+RAG). Figure 10: Simulation results of states and
constraints with the RAG. (a) The displacement of the mass. (b) The velocity
of the mass. Figure 11: Control inputs comparisons of the RAG and nominal
control policy, results of mode transitions and disturbances. (a) Control
inputs. (b) Mode. (c) Parametric disturbances. (d) Additive disturbances.

###  5.3 Results of explicit safe RL policy

In this section, we compare the performances of the explicit safe RL policy
(23) and the conventional RAG (9) using the example in Section 5.1. A trained
RL policy together with the RAG is served as expert. We use a neural network
to approximate the expert’s action, which is the exact mapping between states
$x$ and safe control inputs $u^{\text{safe}}$ obtained by solving the RAG
optimization problem (9), via the imitation learning in (23). After training,
we apply the explicit safe RL policy as shown in Fig. 12 directly to the MSD
system to track reference mass position while trying to keep the constraints
(27) being satisfied.

The control performances using the explicit safe RL policy (ESafeRL) and safe
RL policy with the RAG (SafeRL) are compared in Fig. 13 and 14. Similar to the
SafeRL, the mass tracks the reference position and achieves soft-landing with
the control of ESafeRL. Meanwhile, as reported in Fig. 15, the ESafeRL reduces
the online computational time by over 95% on average compared with one with
SafeRL by avoiding solving MIQP (9) online.

Figure 12: ERAG control policy. (a)-(b) Control policy with
$x^{\text{ref}}=5$. (c)-(d) Control policy with $x^{\text{ref}}=0$.  Figure
13: Simulation results of states and constraints with the ERAG and RAG. (a)
The displacement of the mass. (b) The velocity of the mass. Figure 14: Control
inputs comparisons of the ERAG and RAG, results of mode transitions and
disturbances. (a) Control inputs. (b) Mode. (c) Parametric disturbances. (d)
Additive disturbances. Figure 15: Comparisons of online computational time
between explicit safe RL and safe RL policy over 500 experiments with randomly
generated initial conditions and disturbances. The solid line represents the
average value and shaded area represents the standard deviation value.

##  6 Conclusions

In this paper, we introduced an extension of an add-on scheme referred to as
the Robust Action Governor (RAG), to the case where discrete-time piecewise
affine (PWA) models subject to both additive and parametric uncertainties and
non-convex constraints are treated. The RAG modifies the nominal control input
when it becomes necessary to enforce constraints. Theoretical properties of
robust constraint satisfaction and recursive feasibility have been
established. A safe reinforcement learning (RL) framework is established based
on the RAG. Applications of the RAG and safe RL to a soft-landing control
problem of a mass-spring-damper system have been successfully demonstrated in
simulations. Future work will focus on extensions of the RAG to systems with
state-dependent uncertainties and on computation efficiency improvements.

This work is supported by Ford Motor Company.

## References

  * [1] Aaron D Ames, Xiangru Xu, Jessy W Grizzle, and Paulo Tabuada.  Control barrier function based quadratic programs for safety critical systems.  IEEE Transactions on Automatic Control, 62(8):3861–3876, 2016.
  * [2] Anil Aswani, Humberto Gonzalez, S Shankar Sastry, and Claire Tomlin.  Provably safe and robust learning-based model predictive control.  Automatica, 49(5):1216–1226, 2013.
  * [3] Alberto Bemporad and Manfred Morari.  Robust model predictive control: A survey.  In Robustness in Identification and Control, pages 207–226. Springer, 1999.
  * [4] Alberto Bemporad and Vihangkumar V Naik.  A numerically robust mixed-integer quadratic programming solver for embedded hybrid model predictive control.  IFAC-PapersOnLine, 51(20):412–417, 2018.
  * [5] Dimitri Bertsekas.  Infinite time reachability of state-space regions by using feedback control.  IEEE Transactions on Automatic Control, 17(5):604–613, 1972.
  * [6] SP Bhattacharyya.  Robust control under parametric uncertainty: An overview and recent results.  Annual Reviews in Control, 44:45–77, 2017.
  * [7] Pratik Biswas, Pascal Grieder, Johan Löfberg, and Manfred Morari.  A survey on stability analysis of discrete-time piecewise affine systems.  IFAC Proceedings Volumes, 38(1):283–294, 2005.
  * [8] Francesco Borrelli, Mato Baotić, Alberto Bemporad, and Manfred Morari.  Dynamic programming for constrained optimal control of discrete-time linear hybrid systems.  Automatica, 41(10):1709–1721, 2005.
  * [9] Francesco Borrelli, Alberto Bemporad, and Manfred Morari.  Predictive control for linear and hybrid systems.  Cambridge University Press, 2017.
  * [10] Francesco Borrelli, Paolo Falcone, Jaroslav Pekar, and Greg Stewart.  Reference governor for constrained piecewise affine systems.  Journal of Process Control, 19(8):1229–1237, 2009.
  * [11] S Di Cairano, Alberto Bemporad, Ilya V Kolmanovsky, and Davor Hrovat.  Model predictive control of magnetically actuated mass spring dampers for automotive applications.  International Journal of Control, 80(11):1701–1716, 2007.
  * [12] Eduardo F Camacho, Daniel R Ramírez, Daniel Limón, D Muñoz De La Peña, and Teodoro Alamo.  Model predictive control techniques for hybrid systems.  Annual Reviews in Control, 34(1):21–31, 2010.
  * [13] R. Cheng, M. J. Khojasteh, A. D. Ames, and J. W. Burdick.  Safe multi-agent interaction through robust control barrier functions with learned uncertainties.  In 2020 IEEE Conference on Decision and Control (CDC), pages 777–783, 2020.
  * [14] Jaime F Fisac, Anayo K Akametalu, Melanie N Zeilinger, Shahab Kaynama, Jeremy Gillula, and Claire J Tomlin.  A general safety framework for learning-based control in uncertain robotic systems.  IEEE Transactions on Automatic Control, 64(7):2737–2752, 2018.
  * [15] Joseph Funke, Matthew Brown, Stephen M Erlien, and J Christian Gerdes.  Collision avoidance and stabilization for autonomous vehicles in emergency scenarios.  IEEE Transactions on Control Systems Technology, 25(4):1204–1216, 2016.
  * [16] Emanuele Garone, Stefano Di Cairano, and Ilya Kolmanovsky.  Reference and command governors for systems with constraints: A survey on theory and applications.  Automatica, 75:306–328, 2017.
  * [17] Elmer Gilbert and Ilya Kolmanovsky.  Nonlinear tracking control in the presence of state and control constraints: a generalized reference governor.  Automatica, 38(12):2063–2073, 2002.
  * [18] Gurobi Optimization, LLC.  Gurobi optimizer reference manual.  2021\.
  * [19] S Ehsan Hashemi-Petroodi, Simon Thevenin, Sergey Kovalev, and Alexandre Dolgui.  Operations management issues in design and control of hybrid human-robot collaborative manufacturing systems: a survey.  Annual Reviews in Control, 49:264–276, 2020.
  * [20] Wilhemus PMH Heemels, Bart De Schutter, and Alberto Bemporad.  Equivalence of hybrid dynamical models.  Automatica, 37(7):1085–1091, 2001.
  * [21] M. Herceg, M. Kvasnica, C. Jones, and M. Morari.  Multi-Parametric Toolbox 3.0.  In 2013 European Control Conference (ECC), pages 502–510, Zürich, Switzerland, July 17–19 2013.  http://control.ee.ethz.ch/~mpt.
  * [22] Tommi Jaakkola, Michael I Jordan, and Satinder P Singh.  On the convergence of stochastic iterative dynamic programming algorithms.  Neural Computation, 6(6):1185–1201, 1994.
  * [23] Mrdjan Jankovic.  Robust control barrier functions for constrained stabilization of nonlinear systems.  Automatica, 96:359–367, October 2018.
  * [24] Juan L Jerez, Paul J Goulart, Stefan Richter, George A Constantinides, Eric C Kerrigan, and Manfred Morari.  Embedded predictive control on an fpga using the fast gradient method.  In 2013 European Control Conference (ECC), pages 3614–3620. IEEE, 2013.
  * [25] Ilya Kolmanovsky and Elmer G Gilbert.  Theory and computation of disturbance invariant sets for discrete-time linear systems.  Mathematical Problems in Engineering, 4(4):317–367, 1998.
  * [26] Karel Kraus, Zbyněk Šika, Petr Beneš, Jan Krivošej, and Tomáš Vyhlídal.  Mechatronic robot arm with active vibration absorbers.  Journal of Vibration and Control, 26(13-14):1145–1156, 2020.
  * [27] Rie B Larsen, Andrea Carron, and Melanie N Zeilinger.  Safe learning for distributed systems with bounded uncertainties.  IFAC-PapersOnLine, 50(1):2536–2542, 2017.
  * [28] Mircea Lazar, WPMH Heemels, Siep Weiland, and Alberto Bemporad.  Stabilization conditions for model predictive control of constrained pwa systems.  In 2004 IEEE Conference on Decision and Control (CDC)(IEEE Cat. No. 04CH37601), volume 5, pages 4595–4600. IEEE, 2004.
  * [29] Mircea Lazar, WPMH Heemels, Siep Weiland, and Alberto Bemporad.  Stabilizing model predictive control of hybrid systems.  IEEE Transactions on Automatic Control, 51(11):1813–1818, 2006\.
  * [30] Insup Lee and Oleg Sokolsky.  Medical cyber physical systems.  In 2010 IEEE Design Automation Conference, pages 743–748. IEEE, 2010.
  * [31] Nan Li, Anouck Girard, and Ilya Kolmanovsky.  Chance-constrained controller state and reference governor.  Automatica, 133:109864, 2021.
  * [32] Nan Li, Kyoungseok Han, Anouck Girard, H Tseng, Dimitar Filev, and Ilya Kolmanovsky.  Action governor for discrete-time linear systems with non-convex constraints.  IEEE Control Systems Letters, 5(1):121–126, 2020.
  * [33] Yutong Li, Nan Li, H Eric Tseng, Anouck Girard, Dimitar Filev, and Ilya Kolmanovsky.  Robust action governor for discrete-time piecewise affine systems with additive disturbances.  IEEE Control Systems Letters, 2021.
  * [34] Yutong Li, Nan Li, Hongtei Tseng, Anouck Girard, Dimitar Filev, and Ilya Kolmanovsky.  Safe reinforcement learning using robust action governor.  2021 Learning for Dynamics and Control Conference (L4DC), 144:1093–1104, 2021.
  * [35] Zhaojian Li, Uroš Kalabić, and Tianshu Chu.  Safe reinforcement learning: Learning with supervision using a constraint-admissible set.  In 2018 American Control Conference (ACC), pages 6390–6395. IEEE, 2018.
  * [36] Todd Litman.  Autonomous vehicle implementation predictions: Implications for transport planning.  2020\.
  * [37] Jiechao Liu, Paramsothy Jayakumar, Jeffrey L Stein, and Tulga Ersal.  Improving the robustness of an mpc-based obstacle avoidance algorithm to parametric uncertainty using worst-case scenarios.  Vehicle System Dynamics, 57(6):874–913, 2019.
  * [38] Tobia Marcucci and Russ Tedrake.  Warm start of mixed-integer programs for model predictive control of hybrid systems.  IEEE Transactions on Automatic Control, 2020.
  * [39] Vinh Nguyen, Joshua Johnson, and Shreyes Melkote.  Active vibration suppression in robotic milling using optimal control.  International Journal of Machine Tools and Manufacture, 152:103541, 2020.
  * [40] Marco M Nicotra, Dominic Liao-McPherson, Laurent Burlion, and Ilya V Kolmanovsky.  Spacecraft attitude control with nonconvex constraints: An explicit reference governor approach.  IEEE Transactions on Automatic Control, 65(8):3677–3684, 2019.
  * [41] Gennaro Notomista and Matteo Saveriano.  Safety of dynamical systems with multiple non-convex unsafe sets using control barrier functions.  IEEE Control Systems Letters, 6:1136–1141, 2021.
  * [42] Dean A Pomerleau.  Efficient training of artificial neural networks for autonomous navigation.  Neural Computation, 3(1):88–97, 1991.
  * [43] Stephen Prajna and Ali Jadbabaie.  Safety verification of hybrid systems using barrier certificates.  In International Workshop on Hybrid Systems: Computation and Control, pages 477–492. Springer, 2004.
  * [44] Kiran Rajaram, Melvin Rafi, James E Steck, and Animesh Chakravarthy.  Laplace-based predictive estimation of loss-of-control boundaries on a transport aircraft.  In AIAA Guidance, Navigation, and Control Conference, page 1022, 2017.
  * [45] Sasa Raković, Franco Blanchini, Eva Cruck, and Manfred Morari.  Robust obstacle avoidance for constrained linear discrete time systems: A set-theoretic approach.  In 2007 IEEE Conference on Decision and Control (CDC), pages 188–193, 2007.
  * [46] Sasa Raković, Eric Kerrigan, David Mayne, and John Lygeros.  Reachability analysis of discrete-time systems with disturbances.  IEEE Transactions on Automatic Control, 51(4):546–561, 2006.
  * [47] Raffaele Romagnoli, Luis D Couto, Alejandro Goldar, Michel Kinnaert, and Emanuele Garone.  A feedback charge strategy for li-ion battery cells based on reference governor.  Journal of process control, 83:164–176, 2019.
  * [48] Walter Rudin.  Functional analysis.  McGraw-Hill Science, Engineering & Mathematics, 1991.
  * [49] Bastian Schürmann, Riccardo Vignali, Maria Prandini, and Matthias Althoff.  Set-based control for disturbed piecewise affine systems with state and actuation constraints.  Nonlinear Analysis: Hybrid Systems, 36:100826, 2020.
  * [50] Christoffer Sloth, George J Pappas, and Rafael Wisniewski.  Compositional safety analysis using barrier certificates.  In 2012 ACM International Conference on Hybrid Systems: Computation and Control, pages 15–24, 2012.
  * [51] Eduardo D. Sontag.  Nonlinear regulation: The piecewise linear approach.  IEEE Transactions on Automatic Control, 26(2):346–358, 1981.
  * [52] Andrew J Taylor and Aaron D Ames.  Adaptive safety with control barrier functions.  In 2020 American Control Conference (ACC), pages 1399–1405. IEEE, 2020.
  * [53] Pablo Urcola, Luis Riazuelo, Maria Teresa Lazaro, and Luis Montano.  Cooperative navigation using environment compliant robot formations.  In 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2789–2794. IEEE, 2008.
  * [54] Christos K Verginis and Dimos V Dimarogonas.  Adaptive robot navigation with collision avoidance subject to 2nd-order uncertain dynamics.  Automatica, 123:109303, 2021.
  * [55] Peter Wieland and Frank Allgöwer.  Constructive safety using control barrier functions.  IFAC Proceedings Volumes, 40(12):462–467, 2007.