# Stochastic receding horizon control with bounded control inputs: a vector
space approach

Debasish Chatterjee  ,  Peter Hokayem  and  John Lygeros  Automatic Control
Laboratory
Physikstrasse 3
ETH Zürich
8092 Zürich
Switzerland<EMAIL_ADDRESS>
###### Abstract.

We design receding horizon control strategies for stochastic discrete-time
linear systems with additive (possibly) unbounded disturbances, while obeying
hard bounds on the control inputs. We pose the problem of selecting an
appropriate optimal controller on vector spaces of functions and show that the
resulting optimization problem has a tractable convex solution. Under the
assumption that the zero-input and zero-noise system is asymptotically stable,
we show that the variance of the state is bounded when enforcing hard bounds
on the control inputs, for any receding horizon implementation. Throughout the
article we provide several examples that illustrate how quantities needed in
the formulation of the resulting optimization problems can be calculated off-
line, as well as comparative examples that illustrate the effectiveness of our
control strategies.

This research was partially supported by the Swiss National Science Foundation
under grant 200021-122072, and the FeedNetBack project FP7-ICT-223866
(www.feednetback.eu).

###### Contents

  1. 1 Introduction
  2. 2 Problem Statement
  3. 3 Main Result
  4. 4 Various Cases of Constrained Controls
    1. 4.1 Bounded controls, unbounded noise, and $p=\infty$
    2. 4.2 Bounded controls, bounded noise, and $p=2$
    3. 4.3 Constraints on control energy
  5. 5 Stability Analysis
    1. 5.1 Mean-square boundedness
    2. 5.2 Input-to-state stability
  6. 6 Numerical Examples
  7. 7 Conclusion and Future Directions
  8. A.1 Some identities
  9. A.2 Proof of mean-square boundedness

##  1\. Introduction

Receding horizon control is a popular paradigm for designing control policies.
In the context of deterministic systems it has received a considerable amount
of attention over the last two decades, and significant advancements have been
made in terms of its theoretical foundations as well as industrial
applications. The motivation comes primarily from the fact that receding
horizon control yields tractabile control laws for deterministic systems in
the presence of constraints, and has consequently become popular in the
industry. The counterpart in the context of stochastic systems, however, is a
relatively recent development. In this article we solve the problem of
stochastic receding horizon control for linear systems subject to additive
(possibly) unbounded disturbances and hard norm bounds on the control inputs,
over a class of feedback policies. Methods for guaranteeing hard bounds on the
control inputs, within our context, while ensuring tractability of the
underlying optimization problem are, to our knowledge, not available in the
current literature. Preliminary results in this direction were reported in
[HCL09].

In the deterministic setting, the receding horizon control scheme is dominated
by worst-case analysis relying on robust control and robust optimization
methods, see, for example, [Ber05, MRRS00, BM99, LHBW07, Mac01, Bla99, FB05,
YB09, RH05] and the references therein. The central idea is to synthesize a
controller based on the bounds of the noise such that a certain target set
becomes invariant with respect to the closed-loop dynamics. However, such an
approach tends to yield rather conservative controllers and large
infeasibility regions. Moreover, assigning an a priori bound to the noise
seems to demand considerable insight. A stochastic model of the noise is a
natural alternative approach to this problem: the conservativeness of worst-
case controllers may be reduced, and one may not need to impose any a priori
bounds on the maximum magnitude of the noise. In [BB07], the authors
reformulate the stochastic programming problem as a deterministic one with
bounded noise and solve a robust optimization problem over a finite horizon,
followed by estimating the performance when the noise is unbounded but takes
high values with low probability (as in the Gaussian case). In [PS09] a
slightly different problem is addressed in which the noise enters in a
multiplicative manner, and hard constraints on the states and control inputs
are relaxed to constraints resembling the integrated chance constraints of
[Han83] or risk measures in mathematical finance. Similar relaxations of hard
constraints to soft probabilistic ones have also appeared in [CKW08] for both
multiplicative and additive noise inputs, as well as in [OJM08] for additive
noise inputs. There are also other approaches, e.g., those employing
randomized algorithms as in [BW07, Bat04, MLL05]. Related lines of research
can be found in [vHB03, vHB06] dealing with constrained model predictive
control (MPC) for stochastic systems motivated by industrial applications, in
[RCMA+09, BSW02, SSW06] dealing with stochastic stability, in [SB09b] dealing
with Q-design, in, e.g., [LH07, LHC03] dealing with alternative approaches to
control under actuator constraints and neural-network approximation. The
articles [ACCL09, CACL09] deal with a formulation that allows probabilistic
state constraints but not hard input constraints, and is hence complementary
to the approach in the present article, and [HCCL10] treats the case of output
feedback. . Finally, note that probabilistic constraints on the controllers
naturally raise difficult questions on what actions to take when such
constraints are violated, see [CCCL08] and [CP09] for partial solutions to
these issues.

The main contributions of the article are as follows: We give a tractable,
convex, and globally feasible solution to the finite-horizon stochastic linear
quadratic (LQ) problem for linear systems with possibly unbounded additive
noise and hard constraints on the elements of the control policy. Within this
framework one has two directions to pursue in terms of controller design,
namely, a posteriori bounding the standard LQG controller, or employing
certainty-equivalent receding horizon controller. While the former direction
explicitly incorporates some aspects of feedback, the synthesis of the latter
involves control constraints and implicitly incorporates the notion of
feedback. Our choice of feedback policies explores the middle ground between
these two choices: we explicitly incorporate both the control bounds and
feedback at the design phase. More specifically, we adopt a policy that is
affine in certain bounded functions of the past noise inputs. The optimal
control problem is lifted onto general vector spaces of candidate control
functions from which the controller can be selected algorithmically by solving
a convex optimization problem. Our novel approach does not require
artificially relaxing the hard constraints on the control input to soft
probabilistic ones (to ensure large feasible sets), and still provides a
globally feasible solution to the problem. Minimal assumptions of the noise
sequence being i.i.d and having finite second moment are imposed. The effect
of the noise appears in the convex optimization problem as certain fixed
cross-covariance matrices, which may be computed off-line and stored.

Once tractability of the optimization problem is ensured, we employ the
resulting control policy in a receding horizon scheme. Under our policies the
closed-loop system is in general not necessarily Markovian, and as a result
stability of the closed-loop system is not immediate. In fact, we can no
longer appeal directly to standard Foster-Lyapunov methods. We establish that
our receding horizon control scheme provides stability under the assumption
that the zero-input and zero-noise system is asymptotically stable. We provide
examples that demonstrate the effectiveness of our policies with respect to
standard methods such as certainty-equivalent MPC, standard unconstrained LQG
and saturated LQG control. These examples show that our policies perform no
worse than the standard unconstrained LQG controller in the absence of control
constraints, and outperform the certainty-equivalent MPC as well as the
saturated LQG control by a significant margin.

Our mechanism for selection of a policy consists of two steps: The first
concerns the structure of our policies, and is motivated by preceding work in
robust optimization and MPC [Löf03, BTGGN04, GKM06]. The second concerns the
procedure for selection of an optimal policy from a general vector space of
candidate control functions, and is inspired by approximate dynamic
programming techniques [BT96, LR06, SS85, dFR03, Pow07]. With respect to the
first step, our policies are more general compared to those in [Löf03,
BTGGN04, GKM06]. With respect to the second, the selection procedure of our
policies consists of a one-step tractable static optimization program.

The rest of this article is organized as follows. In Section 2 we state the
main problem to be solved in the most general form. In Section 3 we provide a
tractable solution to the finite horizon optimization problem on general
vector spaces. This result is specialized to various classes of noise and
input constraint sets in Section 4. Stability of receding horizon
implementations of the obtained closed-loop policy is shown in Section 5, and
input-to-state stability properties are discussed in Section 5.2. We provide a
host of numerical examples that illustrate the effectiveness of our approach
in Section 6. Finally, we conclude in Section 7 with a discussion on future
research directions.

### Notation

Hereafter, $\mathbb{N}\coloneqq\\{1,2,\ldots\\}$ is the set of natural
numbers, $\mathbb{N}_{0}\coloneqq\mathbb{N}\cup\\{0\\}$, $\mathbb{Z}$ is the
set of all the integers, $\mathbb{R}_{\geqslant 0}$ is the set of nonnegative
real numbers, and $\mathbb{C}$ denotes the set of complex numbers. We let
$\mathbf{1}_{A}(\cdot)$ denote the indicator function of a set $A$, and
$\mathbf{I}_{n\times n}$ and $\mathbf{0}_{n\times m}$ denote the
$n$-dimensional identity matrix and $n\times m$-dimensional zeros matrix,
respectively. Let $\left\lVert{\cdot}\right\rVert$ denote the standard
Euclidean norm, and $\left\lVert{\cdot}\right\rVert_{p}$ denote the usual
$\ell_{p}$ norms. Also, let $\mathbb{E}_{x_{0}}[\cdot]$ denote the expected
value given $x_{0}$, and $\mathbf{tr}\\!\left(\cdot\right)$ denote the trace
of a matrix. If $M_{1}$ and $M_{2}$ are two matrices with the same number of
rows, we employ the standard notation $[M_{1}\mid M_{2}]$ for the matrix
obtained by stacking the columns of $M_{1}$ followed by the columns of
$M_{2}$. For a given symmetric $n$-dimensional matrix $M$ with real entries,
let $\\{\lambda_{i}(M)\mid i=1,\ldots,n\\}$ be the set of eigenvalues of $M$,
and let $\lambda_{\rm max}(M)\coloneqq\max_{i}\lambda_{i}(M)$ and
$\lambda_{\text{min}}(M)\coloneqq\min_{i}\lambda_{i}(M)$. Finally, for a
random vector $X$ let $\Sigma_{X}$ denote the matrix
$\mathbb{E}\bigl{[}XX^{\mathsf{T}}\bigr{]}$ and $\mu_{X}$ denote the vector
$\mathbb{E}\bigl{[}X\bigr{]}$.

##  2\. Problem Statement

Consider the following discrete-time stochastic dynamical system:

(2.1) $x_{t+1}=\bar{A}x_{t}+\bar{B}u_{t}+w_{t},\qquad t\in\mathbb{N}_{0},$

where $x_{t}\in\mathbb{R}^{n}$ is the state, $u_{t}$ is the control input
taking values in some given control set
$\bar{\mathbb{U}}\subseteq\mathbb{R}^{m}$ to be defined later,
$\bar{A}\in\mathbb{R}^{n\times n}$, $\bar{B}\in\mathbb{R}^{n\times m}$, and
$(w_{t})_{t\in\mathbb{N}_{0}}$ is a sequence of stochastic noise vectors with
$w_{t}\in\mathbb{W}\subseteq\mathbb{R}^{n}$. We assume that the initial
condition $x_{0}$ is given and that, at any time $t$, $x_{t}$ is observed
perfectly. We do not assume that the components of the noise $w_{t}$ are
uncorrelated, nor that they have zero mean; this effectively means that
$w_{t}$ may be of the form $\bar{F}w_{t}^{\prime}+b$ for some noise
$w_{t}^{\prime}\in\mathbb{R}^{p}$ whose components are uncorrelated or
mutually independent, $F\in\mathbb{R}^{n\times p}$, and $b\in\mathbb{R}^{n}$.
Without loss of generality we shall stick to the simpler notation of (2.1)
throughout this article. The results readily extend to the general case of
$w_{t}=\bar{F}w_{t}^{\prime}+b$, as can be seen in [HCL09].

Generally, a _control policy_ $\pi$ is a sequence
$(\pi_{0},\pi_{1},\pi_{2},\ldots)$ of Borel measurable maps
$\pi_{t}:\underbrace{\mathbb{R}^{n}\times\cdots\times\mathbb{R}^{n}}_{k(t)-\text{
times}}\to\bar{\mathbb{U}},\;t\in\mathbb{N}_{0}$. Policies of finite length
such as $(\pi_{t},\pi_{t+1},\ldots,\pi_{t+N-1})$ will be denoted in the sequel
by $\pi_{t:t+N-1}$.

Fix an optimization horizon $N\in\mathbb{N}$ and let us consider the following
objective function at time $t$ given the state $x_{t}$:

(2.2)
$V_{t}\coloneqq\mathbb{E}\Biggl{[}\sum_{k=0}^{N-1}\bigl{(}x_{t+k}^{\mathsf{T}}Q_{k}x_{t+k}+u_{t+k}^{\mathsf{T}}R_{k}u_{t+k}\bigr{)}+x_{t+N}^{\mathsf{T}}Q_{N}x_{t+N}\,\Bigg{|}\,x_{t}\Biggr{]},$

where $Q_{t}>0,R_{t}>0,Q_{N}>0$ are some given symmetric matrices of
appropriate dimension. At each time instant $t$, we are interested in
minimizing (2.2) over the class of causal state feedback strategies $\Pi$
defined as:

(2.3) $\left[\begin{matrix}u_{t}\\\ u_{t+1}\\\ \vdots\\\
u_{t+N-1}\end{matrix}\right]=\left[\begin{array}[]{l}\pi_{t}(x_{t})\\\
\pi_{t+1}(x_{t},x_{t+1})\\\ \vdots\\\
\pi_{t+N-1}(x_{t},x_{t+1},\cdots,x_{t+N-1})\end{array}\right],$

for some measurable functions
$\pi_{t:t+N-1}\coloneqq\\{\pi_{t},\cdots,\pi_{t+N-1}\\}\in\Pi$, while
satisfying $u_{t}\in\bar{\mathbb{U}}$ for each $t$. The _receding horizon
control_ procedure for a given control horizon $N_{c}\in\\{1,\ldots,N\\}$ and
time $t$ can be described as follows:

  * (a)

measure the state $x_{t}$;

  * (b)

determine an admissible optimal feedback control policy, say
$\pi^{*}_{t:t+N-1}\in\Pi$, that minimizes the $N$-stage cost function (2.2)
starting from time $t$, given the measured initial condition $x_{t}$;

  * (c)

apply the first $N_{c}$ elements $\pi^{*}_{t:t+N_{c}-1}$ of the policy
$\pi^{*}_{t:t+N-1}$;

  * (d)

increase $t$ to $t+N_{c}$, and go back to step (a).

In this context, if $N_{c}=1$ then this is usual MPC, and if $N_{c}=N$, then
it is usually known as rolling horizon control.

Since both the system (2.1) and cost (2.2) are time-invariant, it is enough to
consider the problem of minimizing the cost for $t=0$. In view of the above we
consider the problem:

(2.4)
$\displaystyle\min_{\pi_{0:N-1}\in\Pi}\bigl{\\{}V_{0}\,\big{|}\,\text{dynamics
\eqref{eq:system}},\text{ and }u_{t}\in\bar{\mathbb{U}}\text{ for each
}t\bigr{\\}}.$

If feasible, the problem (2.4) generates an optimal sequence of feedback
control laws $\pi^{*}=\left\\{\pi^{*}_{0},\cdots,\pi^{*}_{N-1}\right\\}$.

The evolution of the system (2.1) over a single optimization horizon $N$ can
be described in a compact form as follows:

(2.5) $x=Ax_{0}+Bu+Dw,$

where

$x\coloneqq\begin{bmatrix}x_{0}\\\ x_{1}\\\ \vdots\\\
x_{N}\end{bmatrix},\qquad u\coloneqq\begin{bmatrix}u_{0}\\\ u_{1}\\\ \vdots\\\
u_{N-1}\end{bmatrix},\qquad w\coloneqq\begin{bmatrix}w_{0}\\\ w_{1}\\\
\vdots\\\ w_{N-1}\end{bmatrix},\qquad
A\coloneqq\begin{bmatrix}\mathbf{I}_{n\times n}\\\ \bar{A}\\\ \vdots\\\
\bar{A}^{N}\end{bmatrix},$ $B\coloneqq\begin{bmatrix}\mathbf{0}_{n\times
m}&\cdots&\cdots&\mathbf{0}_{n\times m}\\\ \bar{B}&\ddots&&\vdots\\\
\bar{A}\bar{B}&\bar{B}&\ddots&\vdots\\\ \vdots&&\ddots&\mathbf{0}_{n\times
m}\\\ \bar{A}^{N-1}\bar{B}&\cdots&\bar{A}\bar{B}&\bar{B}\end{bmatrix},\qquad
D\coloneqq\begin{bmatrix}\mathbf{0}_{n\times
n}&\cdots&\cdots&\mathbf{0}_{n\times n}\\\ \mathbf{I}_{n\times
n}&\ddots&&\vdots\\\ \bar{A}&\mathbf{I}_{n\times n}&\ddots&\vdots\\\
\vdots&&\ddots&\mathbf{0}_{n\times n}\\\
\bar{A}^{N-1}&\cdots&\bar{A}&\mathbf{I}_{n\times n}\end{bmatrix}.$

Using the compact notation above, the optimization Problem (2.4) can be
rewritten as follows:

(2.6) $\displaystyle\min_{\pi_{0:N-1}\in\Pi}$
$\displaystyle\bigl{\\{}\mathbb{E}_{x_{0}}\bigl{[}x^{\mathsf{T}}Qx+u^{\mathsf{T}}Ru\bigr{]}\,\big{|}\,\text{dynamics
\eqref{eq:compactdyn}},u\in\mathbb{U}\bigr{\\}},$

where $Q=\operatorname{diag}\\{Q_{0},\ldots,Q_{N}\\}$,
$R=\operatorname{diag}\\{R_{0},\ldots,R_{N-1}\\}$, and
$\mathbb{U}\coloneqq\underbrace{\bar{\mathbb{U}}\times\ldots\times\bar{\mathbb{U}}}_{N-\text{times}}$.

##  3\. Main Result

We require that our controller is selected from a vector space of candidate
controllers spanned by a given set of “simple” basis functions. The precise
algorithmic selection procedure is based on the solution to an optimization
problem. The basis functions may represent particular types of control
functions that are easy or inexpensive to implement, e.g., minimum attention
control [Bro97], or may be the only ones available for a specific application.
For instance, piecewise constant policy elements with finitely many elements
in their range may be viewed as controllers that can provide only finitely
many values; this may be viewed as an extended version of a bang-bang
controller, or as a hybrid controller with a finite control alphabet.

More formally, let $\mathcal{H}$ be a nonempty separable vector space of
functions with the control set $\mathbb{U}$ as their range, i.e.,
$\mathcal{H}$ is the linear span of measurable functions
${\mathfrak{e}}^{\nu}:\mathbb{W}\to\mathbb{U}$, where $\nu\in\mathcal{I}$ \-
an ordered countable index set (see [Lue69] for more details). As mentioned
above, the elements of $\mathcal{H}$ may be linear combinations of typical
“simple” controller functions for $t=0,1,\ldots,N-1$. We are interested in
policies of the form $u_{t}=\eta_{t}+\sum_{i=0}^{t-1}\psi_{t,i}(w_{i})$, where
$\eta_{t}$ is an $m$-dimensional vector and each component of the
$m$-dimensional vector-valued function $\psi_{t,i}$ is a member of
$\mathcal{H}$. Although this feedback function is directly from the noise,
since the state is assumed to be perfectly measured, from the system dynamics
(2.1) it follows at once that this controller $u_{t}$ is actually a feedback
from all the states $x_{0},\ldots,x_{t}$. Indeed, in the spirit of [Löf03,
BTGGN04, GKM06, SB09a] we have

$\displaystyle u_{0}$ $\displaystyle=\eta_{0},$ $\displaystyle u_{1}$
$\displaystyle=\eta_{1}+\psi_{1,0}(x_{1}-\bar{A}x_{0}-\bar{B}\eta_{0}),$
$\displaystyle u_{2}$
$\displaystyle=\eta_{2}+\psi_{2,0}(x_{1}-\bar{A}x_{0}-\bar{B}\eta_{0})+\psi_{2,1}\bigl{(}x_{2}-\bar{A}x_{1}-\bar{B}\bigl{(}\eta_{1}+\psi_{1,0}(x_{1}-\bar{A}x_{0}-\bar{B}\eta_{0})\bigr{)}\bigr{)},$
$\displaystyle\vdots$

In other words, by construction, $u_{t}$ is generally a nonlinear feedback
controller depending on the past $t$ states.111Note that the controller input
at time $t$ is non-Markovian as it is a function of the state vectors at all
the previous times and not just on $x_{t-1}$. Also by construction, it is
causal.

Our general control policy can now be expressed as the vector

(3.1) $u=\eta+\varphi(w)\coloneqq\begin{bmatrix}\eta_{0}\\\ \eta_{1}\\\
\vdots\\\ \eta_{N-1}\end{bmatrix}+\begin{bmatrix}\varphi_{0}\\\
\varphi_{1}(w_{0})\\\ \vdots\\\
\varphi_{N-1}(w_{0},w_{1},\ldots,w_{N-2})\end{bmatrix},$

where,

  * •

$\varphi_{0}=0$,

  * •

$w_{t}$ for $t=0,\ldots,N-1$ is the $t$-th random noise vector,

  * •

$\eta_{t}$ is an $m$-dimensional vector for $t=0,\ldots,N-1$,

  * •

$\varphi_{t}(w_{0},\ldots,w_{t-1})=\sum_{i=0}^{t-1}\varphi_{t,i}(w_{i})$ for
$t=1,\ldots,N-1$ is an $m$-dimensional vector, and

  * •

each function $\varphi_{t,i}$ belongs to the linear span of the basis elements
$({\mathfrak{e}}^{\nu})_{\nu\in\mathcal{I}}$, and thus has a representation as
a linear combination
$\varphi_{t,i}(\cdot)=\sum_{\nu\in\mathcal{I}}\theta_{t,i}^{\nu}{\mathfrak{e}}^{\nu}(\cdot)$,
$t=1,\ldots,N-1$, $i=0,\ldots,t-1$, where $\theta_{t,i}^{\nu}$ are matrices of
coefficients of appropriate dimension.

Analogous to Fourier coefficients in harmonic analysis, we call the
$\theta_{t,i}^{\nu}$ the $\nu$-th Fourier coefficient of the function
$\varphi_{t,i}$. Therefore, whenever $|\mathcal{I}|<\infty$ for every
$t=1,\ldots,N-1$, we have the finite representation

(3.2)
$\varphi_{t}(w_{0},\ldots,w_{t-1})=\begin{bmatrix}\theta_{t,0}&\theta_{t,1}&\ldots&\theta_{t,t-1}&\boldsymbol{0}&\ldots&\boldsymbol{0}\end{bmatrix}_{R^{m\times
n|\mathcal{I}|(N-1)}}\begin{bmatrix}{\mathfrak{e}}(w_{0})\\\
{\mathfrak{e}}(w_{1})\\\ \vdots\\\
{\mathfrak{e}}(w_{N-2})\end{bmatrix}_{\mathbb{R}^{n|\mathcal{I}|(N-1)\times
1}},$

where $\theta_{t,i}\in\mathbb{R}^{m\times n|\mathcal{I}|}$,
$\boldsymbol{0}\in\mathbb{R}^{m\times n|\mathcal{I}|}$,

$\theta_{t,i}\coloneqq\left[\begin{matrix}\theta^{1}_{t,i}\,\;\cdots\,\,\theta^{|\mathcal{I}|}_{t,i}\end{matrix}\right],\;\;\theta_{t,i}^{\nu}\in\mathbb{R}^{m\times
n},\quad{\rm
and}\quad{\mathfrak{e}}(w_{i})\coloneqq\left[\begin{matrix}{\mathfrak{e}}^{1}(w_{i})\\\
\vdots\\\
{\mathfrak{e}}^{|\mathcal{I}|}(w_{i})\end{matrix}\right],\quad\forall\,i=0,1,\cdots,N-2.$

In this notation the policy (3.1) can be written as

(3.3)
$u=\eta+\varphi(w)=\eta+\begin{bmatrix}\boldsymbol{0}&\boldsymbol{0}&\cdots&\boldsymbol{0}\\\
\theta_{1,0}&\boldsymbol{0}&\cdots&\boldsymbol{0}\\\
\theta_{2,0}&\theta_{2,1}&\cdots&\boldsymbol{0}\\\
\vdots&\vdots&\ddots&\vdots\\\
\theta_{N-1,0}&\theta_{N-1,1}&\cdots&\theta_{N-1,N-2}\end{bmatrix}\begin{bmatrix}{\mathfrak{e}}(w_{0})\\\
{\mathfrak{e}}(w_{1})\\\ \vdots\\\
{\mathfrak{e}}(w_{N-2})\end{bmatrix}\eqqcolon\eta+\Theta{\mathfrak{e}}(w),$

where $\Theta$ is now the matrix of Fourier coefficients having dimension
$Nm\times\bigl{(}n(N-1)|\mathcal{I}|\bigr{)}$. This Fourier coefficient matrix
$\Theta$ and the vector $\eta$ play the role of the optimization parameters in
our search for an optimal policy. Note that ${\mathfrak{e}}(w)$ does not
include the noise vector $w_{N-1}$, and that $\Theta$ is strictly lower block
triangular to enforce causality. In what follows, as a matter of notation, by
$\Theta_{t}$ we shall denote the formal $t$-th block-row of the matrix
$\Theta$ in (3.3), i.e.,
$\Theta_{t}\coloneqq\begin{bmatrix}\theta_{t,0}&\cdots&\theta_{t,t-1}&0&\cdots&0\end{bmatrix}$,
for $t=0,\cdots,N-1$, with $\Theta_{0}$ being the identically $0$ row. We make
the following assumption:

######  Assumption 3.1.

The sequence $(w_{t})_{t\in\mathbb{N}_{0}}$ of noise vectors is i.i.d with
$\Sigma=\mathbb{E}\bigl{[}w_{t}w_{t}^{\mathsf{T}}\bigr{]}$.$\diamondsuit$

So far we have not stipulated any boundedness properties on the elements of
the vector space $\mathcal{H}$. This means that the control policy elements
may be unbounded maps. First we stipulate the following structure on the
control sets:

For a given $p\in[1,\infty]$, the control input vector $u_{t}$ is bounded in
$p$-norm at each instant of time $t$, i.e., for $p\in[1,\infty]$ let
$U_{\max}^{(p)}>0$ be given, with

(3.4) $\displaystyle u_{t}\in\bar{\mathbb{U}}_{p}$
$\displaystyle\coloneqq\bigl{\\{}\xi\in\mathbb{R}^{m}\big{|}\left\lVert{\xi}\right\rVert_{p}\leqslant
U_{\max}^{(p)}\bigr{\\}}\quad\forall\,t\in\mathbb{N}_{0},\quad\text{and}$
$\displaystyle\mathbb{U}_{p}$
$\displaystyle\coloneqq\underbrace{\bar{\mathbb{U}}_{p}\times\ldots\times\bar{\mathbb{U}}_{p}}_{N-\text{times}}.$

One could easily include more general constraint sets $\mathbb{U}_{p}$, for
instance, to capture bounds on the rate of change of inputs.

Our basic result is the next Theorem.

######  Theorem 3.2.

Consider the system (2.1). Suppose that Assumption 3.1 holds, $\mathcal{H}$ is
finite-dimensional ($|\mathcal{I}|<\infty$), and every component of the basis
functions ${\mathfrak{e}}^{\nu}$ is bounded by $\mathcal{E}>0$ in absolute
value. Then the problem (2.6) under the policy (3.1) and control sets (3.4)
for $p\in[1,\infty]$ is convex with respect to the decision variables
$(\eta,\Theta)$ defined in (3.3). For $p=1,2$, and $\infty$ it admits convex
tractable versions with tighter domains of $(\eta,\Theta)$, given by

(3.5) $\displaystyle\underset{(\eta,\Theta)}{\text{minimize}}$
$\displaystyle\operatorname{\mathbf{tr}}\Bigl{(}\Theta^{\mathsf{T}}\bigl{(}{B}^{\mathsf{T}}QB+R\bigr{)}\Theta\Sigma_{{\mathfrak{e}}}\Bigr{)}+2\operatorname{\mathbf{tr}}\Bigl{(}\Theta^{\mathsf{T}}{B}^{\mathsf{T}}QD\Sigma_{{\mathfrak{e}}}^{\prime}\Bigr{)}+\eta^{\mathsf{T}}\bigl{(}{B}^{\mathsf{T}}QB+R\bigr{)}\eta$
$\displaystyle\quad+2\bigl{(}x_{0}^{\mathsf{T}}{A}^{\mathsf{T}}QB\eta+\eta^{\mathsf{T}}{B}^{\mathsf{T}}QD\mu_{w}+x_{0}^{\mathsf{T}}A^{\mathsf{T}}QB\Theta\mu_{\mathfrak{e}}\bigr{)}$
$\displaystyle\quad+2\eta^{\mathsf{T}}\bigl{(}R+B^{\mathsf{T}}QB\bigr{)}\Theta\mu_{\mathfrak{e}}+c$
subject to $\displaystyle\text{$\Theta$ strictly lower block triangular as
in~{}\eqref{e:Thetadef}},$
$\displaystyle\begin{cases}p=1:&\left\lVert{\eta_{t}}\right\rVert_{1}+\mathcal{E}t\left\lVert{\Theta_{t}}\right\rVert_{1}\leqslant
U_{\max}^{(1)},\quad\forall\,t=0,1,\ldots,N-1,\\\
p=\infty:&\left\lVert{\eta_{t}}\right\rVert_{\infty}+\mathcal{E}\left\lVert{\Theta_{t}}\right\rVert_{\infty}\leqslant
U_{\max}^{(\infty)},,\quad\forall\,t=0,1,\ldots,N-1,\\\
p=2:&\left\lVert{\begin{bmatrix}\eta_{t}&\Theta_{t}\end{bmatrix}}\right\rVert_{2}\mathchoice{{\hbox{$\displaystyle\sqrt{1+\mathcal{E}t\,}$}\lower
0.4pt\hbox{\vrule
height=6.83331pt,depth=-5.46667pt}}}{{\hbox{$\textstyle\sqrt{1+\mathcal{E}t\,}$}\lower
0.4pt\hbox{\vrule
height=6.83331pt,depth=-5.46667pt}}}{{\hbox{$\scriptstyle\sqrt{1+\mathcal{E}t\,}$}\lower
0.4pt\hbox{\vrule
height=4.78333pt,depth=-3.82668pt}}}{{\hbox{$\scriptscriptstyle\sqrt{1+\mathcal{E}t\,}$}\lower
0.4pt\hbox{\vrule height=3.41666pt,depth=-2.73334pt}}}\leqslant
U_{\max}^{(2)},\quad\forall\,t=0,1,\ldots,N-1,\end{cases}$

where

$\displaystyle\Sigma_{{\mathfrak{e}}}$
$\displaystyle\coloneqq\mathbb{E}\bigl{[}{\mathfrak{e}}(w){\mathfrak{e}}(w)^{\mathsf{T}}\bigr{]},$
$\displaystyle\Sigma_{{\mathfrak{e}}}^{\prime}$
$\displaystyle\coloneqq\mathbb{E}\bigl{[}w{\mathfrak{e}}(w)^{\mathsf{T}}\bigr{]},$
$\displaystyle\mu_{w}$
$\displaystyle\coloneqq\mathbb{E}[w],\quad\quad\mu_{\mathfrak{e}}\coloneqq\mathbb{E}[{\mathfrak{e}}(w)],$
$\displaystyle c$ $\displaystyle\coloneqq
x_{0}^{\mathsf{T}}{A}^{\mathsf{T}}QAx_{0}+2x_{0}^{\mathsf{T}}{A}^{\mathsf{T}}QD\mu+\operatorname{\mathbf{tr}}\bigl{(}{D}^{\mathsf{T}}QD\Sigma_{w}\bigr{)}.$

###### Proof of Theorem 3.2.

It is easy to see that $x^{\mathsf{T}}Qx+u^{\mathsf{T}}Ru$ is convex
nondecreasing, and both $x$ and $u$ are affine functions of the design
parameters $(\eta,\Theta)$ for any realization of the noise $w$. Hence,
$V_{0}$ is convex in $(\eta,\Theta)$ since taking expectations of a convex
function retains convexity [BV04, Section 3.2]. Moreover, the control
constraint sets in (3.4) are convex in $(\eta,\Theta)$. This settles the first
claim.

The objective function (2.2) is given by

$\displaystyle\mathbb{E}_{x_{0}}$
$\displaystyle\bigl{[}\bigl{(}Ax_{0}+Bu+Dw\bigr{)}^{\mathsf{T}}Q\bigl{(}Ax_{0}+Bu+Dw\bigr{)}\bigr{]}+\mathbb{E}_{x_{0}}\bigl{[}{u}^{\mathsf{T}}Ru\bigr{]}$
$\displaystyle=\mathbb{E}_{x_{0}}\bigl{[}\bigl{(}Ax_{0}+B(\eta+\Theta{\mathfrak{e}}(w))+Dw\bigr{)}^{\mathsf{T}}Q\bigl{(}Ax_{0}+B(\eta+\Theta{\mathfrak{e}}(w))+Dw\bigr{)}\bigr{]}$
$\displaystyle\quad+\mathbb{E}_{x_{0}}\bigl{[}(\eta+\Theta{\mathfrak{e}}(w))^{\mathsf{T}}R(\eta+\Theta{\mathfrak{e}}(w))\bigr{]}$
$\displaystyle=x_{0}^{\mathsf{T}}{A}^{\mathsf{T}}QAx_{0}+2x_{0}^{\mathsf{T}}{A}^{\mathsf{T}}QB\eta+\eta^{\mathsf{T}}\bigl{(}{B}^{\mathsf{T}}QB+R\bigr{)}\eta$
$\displaystyle\quad+2\bigl{(}Ax_{0}+B\eta\bigr{)}^{\mathsf{T}}Q\bigl{(}B\Theta\mathbb{E}_{x_{0}}[{\mathfrak{e}}(w)]+D\mathbb{E}_{x_{0}}[w]\bigr{)}$
$\displaystyle\quad+\mathbb{E}_{x_{0}}\bigl{[}\bigl{(}B\Theta{\mathfrak{e}}(w)+Dw\bigr{)}^{\mathsf{T}}Q\bigl{(}B\Theta{\mathfrak{e}}(w)+Dw\bigr{)}\bigr{]}+\mathbb{E}_{x_{0}}\bigl{[}(\Theta{\mathfrak{e}}(w))^{\mathsf{T}}R\Theta{\mathfrak{e}}(w)\bigr{]}$
$\displaystyle=x_{0}^{\mathsf{T}}{A}^{\mathsf{T}}QAx_{0}+2x_{0}^{\mathsf{T}}{A}^{\mathsf{T}}QB\eta+\eta^{\mathsf{T}}\bigl{(}{B}^{\mathsf{T}}QB+R\bigr{)}\eta+2\eta^{\mathsf{T}}R\Theta\mathbb{E}[{\mathfrak{e}}(w)]$
$\displaystyle\quad+2\bigl{(}Ax_{0}+B\eta\bigr{)}^{\mathsf{T}}Q\bigl{(}D\mathbb{E}_{x_{0}}[w]+B\Theta\mathbb{E}[{\mathfrak{e}}(w)]\bigr{)}+\operatorname{\mathbf{tr}}\Bigl{(}{D}^{\mathsf{T}}QD\mathbb{E}_{x_{0}}\bigl{[}w^{\mathsf{T}}\bigr{]}\Bigr{)}$
$\displaystyle\quad+\operatorname{\mathbf{tr}}\Bigl{(}\Theta^{\mathsf{T}}\bigl{(}{B}^{\mathsf{T}}QB+R\bigr{)}\Theta\mathbb{E}_{x_{0}}\bigl{[}{\mathfrak{e}}(w){\mathfrak{e}}(w)^{\mathsf{T}}\bigr{]}\Bigr{)}+2\operatorname{\mathbf{tr}}\Bigl{(}\Theta^{\mathsf{T}}{B}^{\mathsf{T}}QD\mathbb{E}_{x_{0}}\bigl{[}w{\mathfrak{e}}(w)^{\mathsf{T}}\bigr{]}\Bigr{)}.$

Incorporating the definitions $\Sigma_{{\mathfrak{e}}}$,
$\Sigma_{{\mathfrak{e}}}^{\prime}$, $\mu_{w}$, $\mu_{\mathfrak{e}}$, and $c$,
the right-hand side above equals

$\displaystyle\operatorname{\mathbf{tr}}\Bigl{(}\Theta^{\mathsf{T}}\bigl{(}{B}^{\mathsf{T}}QB+R\bigr{)}\Theta\Sigma_{{\mathfrak{e}}}\Bigr{)}+2\operatorname{\mathbf{tr}}\Bigl{(}\Theta^{\mathsf{T}}{B}^{\mathsf{T}}QD\Sigma_{{\mathfrak{e}}}^{\prime}\Bigr{)}+\eta^{\mathsf{T}}\bigl{(}{B}^{\mathsf{T}}QB+R\bigr{)}\eta$
$\displaystyle\quad+2\bigl{(}x_{0}^{\mathsf{T}}{A}^{\mathsf{T}}QB\eta+\eta^{\mathsf{T}}{B}^{\mathsf{T}}QD\mu_{w}+x_{0}^{\mathsf{T}}A^{\mathsf{T}}QB\Theta\mu_{\mathfrak{e}}\bigr{)}+2\eta^{\mathsf{T}}\bigl{(}R+B^{\mathsf{T}}QB\bigr{)}\Theta\mu_{\mathfrak{e}}$
$\displaystyle\quad+\bigl{(}x_{0}^{\mathsf{T}}{A}^{\mathsf{T}}QAx_{0}+2x_{0}^{\mathsf{T}}{A}^{\mathsf{T}}QD\mu_{w}+\operatorname{\mathbf{tr}}\bigl{(}{D}^{\mathsf{T}}QD\Sigma_{w}\bigr{)}\bigr{)}$
$\displaystyle=\operatorname{\mathbf{tr}}\Bigl{(}\Theta^{\mathsf{T}}\bigl{(}{B}^{\mathsf{T}}QB+R\bigr{)}\Theta\Sigma_{{\mathfrak{e}}}\Bigr{)}+2\operatorname{\mathbf{tr}}\Bigl{(}\Theta^{\mathsf{T}}{B}^{\mathsf{T}}QD\Sigma_{{\mathfrak{e}}}^{\prime}\Bigr{)}+\eta^{\mathsf{T}}\bigl{(}{B}^{\mathsf{T}}QB+R\bigr{)}\eta$
$\displaystyle\quad+2\bigl{(}x_{0}^{\mathsf{T}}{A}^{\mathsf{T}}QB\eta+\eta^{\mathsf{T}}{B}^{\mathsf{T}}QD\mu_{w}+x_{0}^{\mathsf{T}}A^{\mathsf{T}}QB\Theta\mu_{\mathfrak{e}}\bigr{)}+2\eta^{\mathsf{T}}\bigl{(}R+B^{\mathsf{T}}QB\bigr{)}\Theta\mu_{\mathfrak{e}}+c.$

Since the matrix $\Sigma_{{\mathfrak{e}}}$ is positive semidefinite, it can be
expressed as a finite nonnegative linear combination of matrices of the type
$\sigma\sigma^{\mathsf{T}}$, for vectors $\sigma$ of appropriate dimension
[BSM03, Theorem 1.10]. Accordingly, if
$\Sigma_{{\mathfrak{e}}}=\sum_{i=1}^{k}\sigma_{i}\sigma_{i}^{\mathsf{T}}$,
then

$\displaystyle\operatorname{\mathbf{tr}}\Bigl{(}\Theta^{\mathsf{T}}\bigl{(}{B}^{\mathsf{T}}QB+R\bigr{)}\Theta\Sigma_{{\mathfrak{e}}}\Bigr{)}$
$\displaystyle=\sum_{i=1}^{k}\operatorname{\mathbf{tr}}\bigl{(}\Theta^{\mathsf{T}}\bigl{(}{B}^{\mathsf{T}}QB+R\bigr{)}\Theta\sigma_{i}\sigma_{i}^{\mathsf{T}}\bigr{)}$
$\displaystyle=\sum_{i=1}^{k}\Bigl{(}\sigma_{i}^{\mathsf{T}}\Theta^{\mathsf{T}}\bigl{(}{B}^{\mathsf{T}}QB+R\bigr{)}\Theta\sigma_{i}\Bigr{)}.$

Defining $\widehat{\Theta}_{i}\coloneqq\Theta\sigma_{i}$ and adjoining these
equalities to the constraints of the optimization program (3.5), we arrive at
the optimization program

(3.6)
$\displaystyle\underset{(\Theta,\widehat{\Theta}_{1},\ldots,\widehat{\Theta}_{k})}{\text{minimize}}$
$\displaystyle\sum_{i=1}^{k}\widehat{\Theta}_{i}^{\mathsf{T}}\bigl{(}{B}^{\mathsf{T}}QB+R\bigr{)}\widehat{\Theta}_{i}+2\operatorname{\mathbf{tr}}\Bigl{(}\Theta^{\mathsf{T}}{B}^{\mathsf{T}}QD\Sigma_{{\mathfrak{e}}}^{\prime}\Bigr{)}+\eta^{\mathsf{T}}\bigl{(}{B}^{\mathsf{T}}QB+R\bigr{)}\eta$
$\displaystyle\qquad+2\bigl{(}x_{0}^{\mathsf{T}}{A}^{\mathsf{T}}QB\eta+\eta^{\mathsf{T}}{B}^{\mathsf{T}}QD\mu+x_{0}^{\mathsf{T}}A^{\mathsf{T}}QB\Theta\mu^{\mathfrak{e}}\bigr{)}$
$\displaystyle\qquad+2\eta^{\mathsf{T}}\bigl{(}R+B^{\mathsf{T}}QB\bigr{)}\Theta\mu^{\mathfrak{e}}+c$
subject to $\displaystyle\Theta\text{ strictly lower block triangular as
in~{}\eqref{e:Thetadef}},$
$\displaystyle\widehat{\Theta}_{i}=\Theta\sigma_{i}\quad\text{for all
}i=1,\ldots,k.$

We see immediately that (3.6) is a convex program in the parameters $\eta$,
$\Theta$ and $\widehat{\Theta}_{i}$, and is equivalent to the cost in (3.5).

It only remains to consider the last constraint in (3.5). First we consider
the cases of $p=1,\infty$. Using the notation above, an application of the
triangle inequality immediately shows that the constraints can be written as

(3.7)
$\displaystyle\begin{cases}p=1:&\left\lVert{\eta_{t}}\right\rVert_{1}+\mathcal{E}t\left\lVert{\Theta_{t}}\right\rVert_{1}\leqslant
U_{\max}^{(1)},\quad\forall\,t=0,1,\ldots,N-1,\\\
p=\infty:&\left\lVert{\eta_{t}}\right\rVert_{\infty}+\mathcal{E}\left\lVert{\Theta_{t}}\right\rVert_{\infty}\leqslant
U_{\max}^{(\infty)},\quad\forall\,t=0,1,\ldots,N-1.\end{cases}$

It follows that the objective function in (3.6) is quadratic and the
constraints in (3.6)-(3.7) are affine in the optimization parameters $\eta$,
$\Theta$, and $\widehat{\Theta}$. As such, for $p=1,\infty$ our problem is a
quadratic program.

For the case of $p=2$, note that
$\eta_{t}+\Theta_{t}{\mathfrak{e}}(w)=\begin{bmatrix}\eta_{t}&\Theta_{t}\end{bmatrix}\begin{bmatrix}1\\\
{\mathfrak{e}}(w)\end{bmatrix}$, and by definition of $\mathcal{E}$ it is
clear that
$\left\lVert{\begin{bmatrix}\eta_{t}&\Theta_{t}\end{bmatrix}\begin{bmatrix}1\\\
{\mathfrak{e}}(w)\end{bmatrix}}\right\rVert_{2}\leqslant\left\lVert{\begin{bmatrix}\eta_{t}&\Theta_{t}\end{bmatrix}}\right\rVert_{2}\mathchoice{{\hbox{$\displaystyle\sqrt{1+\mathcal{E}t\,}$}\lower
0.4pt\hbox{\vrule
height=6.83331pt,depth=-5.46667pt}}}{{\hbox{$\textstyle\sqrt{1+\mathcal{E}t\,}$}\lower
0.4pt\hbox{\vrule
height=6.83331pt,depth=-5.46667pt}}}{{\hbox{$\scriptstyle\sqrt{1+\mathcal{E}t\,}$}\lower
0.4pt\hbox{\vrule
height=4.78333pt,depth=-3.82668pt}}}{{\hbox{$\scriptscriptstyle\sqrt{1+\mathcal{E}t\,}$}\lower
0.4pt\hbox{\vrule height=3.41666pt,depth=-2.73334pt}}}$. This immediately
translates to
$\left\lVert{\begin{bmatrix}\eta_{t}&\Theta_{t}\end{bmatrix}}\right\rVert_{2}\mathchoice{{\hbox{$\displaystyle\sqrt{1+\mathcal{E}t\,}$}\lower
0.4pt\hbox{\vrule
height=6.83331pt,depth=-5.46667pt}}}{{\hbox{$\textstyle\sqrt{1+\mathcal{E}t\,}$}\lower
0.4pt\hbox{\vrule
height=6.83331pt,depth=-5.46667pt}}}{{\hbox{$\scriptstyle\sqrt{1+\mathcal{E}t\,}$}\lower
0.4pt\hbox{\vrule
height=4.78333pt,depth=-3.82668pt}}}{{\hbox{$\scriptscriptstyle\sqrt{1+\mathcal{E}t\,}$}\lower
0.4pt\hbox{\vrule height=3.41666pt,depth=-2.73334pt}}}\leqslant
U_{\max}^{(2)}$, which is the third constraint in Problem 3.5 and it is a
quadratic constraint in the optimization parameters $(\eta,\Theta)$.
Therefore, for $p=2$ our problem is a quadratically constrained quadratic
program. ∎

The optimization problem (3.5) simplifies if we assume that
$\mu^{\mathfrak{e}}=\mathbb{E}[{\mathfrak{e}}(w)]=0$. Note that
$\mathbb{E}[{\mathfrak{e}}(w)]=0$ if and only if
$\mathbb{E}\bigl{[}{\mathfrak{e}}_{t,i}^{\nu}(w_{t,i})\bigr{]}=0$ for all
$\nu\in\mathcal{I}$. At an intuitive level this translates to the condition
that the functions ${\mathfrak{e}}_{t,i}^{\nu}\in\mathcal{H}$ should be
“centered” with respect to the random variables $w_{t,i}$. In particular, this
simply means that for noise distributions that are symmetric about $0$, the
functions ${\mathfrak{e}}^{\nu}$ should be centered at $0$ and be
antisymmetric. For example, if the noise is Gaussian with mean $0$ and
diagonal covariance matrix (uncorrelated components), each component of the
functions ${\mathfrak{e}}^{\nu}$ should be an odd function.

The matrices $\Sigma_{{\mathfrak{e}}}$, $\Sigma_{{\mathfrak{e}}}^{\prime}$,
the vector $v$, and the number $c$ in Theorem 3.2 are all constants
independent of $x_{0}$, and can be computed off-line. As such, even if closed-
form expressions for the entries of the matrices do not exist, they can be
numerically computed to desired precision. The optimization problem (3.5) is a
quadratic program [BV04, p. 152] for $p=1,\infty$, and a quadratically
constrained quadratic program [BV04, p. 152] for $p=2$, in the optimization
parameters
$\bigl{\\{}\eta,\Theta,\bigl{\\{}\widehat{\Theta}_{i},i=1,\ldots,k\bigr{\\}}\bigr{\\}}$,
and can be easily coded in standard software packages such as cvx [GB00] or
YALMIP [Löf04]. Note that the optimization problem (3.5) is always feasible
(simply set $\Theta=0$ and $\eta=0$ to see this). This is not a surprise,
since there are no constraints on the state, and by construction
$0\in\mathbb{U}$. Finally, note that the third constraint in Problem (3.5) for
various values of $p$, is a result of robustly satisfying the constraints
posed by the various control sets (3.4) for any realization of the noise $w$.

In general, the total number of decision variables in the optimization program
(3.5) is $mN\bigl{(}1+\tfrac{1}{2}n(N-1)|\mathcal{I}|\bigr{)}$. The number of
decision variables can be substantially reduced, e.g., by choosing
$\mathcal{H}$ to be $1$-dimensional, or by fixing certain (block) elements of
the Fourier coefficient matrix $\Theta$ to $0$.

##  4\. Various Cases of Constrained Controls

We examine in this section several special cases of Theorem 3.2 under various
restrictions on the classes of noise and control inputs.

###  4.1. Bounded controls, unbounded noise, and $p=\infty$

Let the noise take values in $\mathbb{R}^{n}$. We provide tractable convex
programs to design a policy that by construction respects the control
constraint sets (3.4), with $p=\infty$. Starting from (3.1) let

(4.1) $u=\eta+\Theta\varphi(w),$

where

  * •

$\varphi(w)\coloneqq\left[\begin{matrix}\varphi_{0}\\\ \varphi_{1}(w_{0})\\\
\vdots\\\ \varphi_{N-1}(w_{0},\ldots,{w}_{N-2})\end{matrix}\right]$,

  * •

$\varphi_{0}=0$,
$\varphi_{t}(w_{0},\ldots,w_{t-1})=\sum_{j=0}^{t-1}\theta_{t}^{j}\varphi_{t,j}(w_{j})$
for $t=1,\ldots,N-2$, and

  * •

$\varphi_{t,j}(w_{j})=\bigl{[}\widetilde{\varphi}(w_{j,1}),\ldots,\widetilde{\varphi}(w_{j,n})\bigr{]}^{\mathsf{T}}$
for some function $\widetilde{\varphi}$ such that
$\sup\limits_{s\in\mathbb{R}}\widetilde{\varphi}(s)=\phi_{\max}<\infty$, and
$\varphi_{t,j}:\mathbb{W}\to\mathbb{U}_{\infty}$.

In other words, we saturate the measurements that we obtain from the noise
input vector before inserting them into our control vector. This way we allow
that the noise distribution is supported over the entire $\mathbb{R}^{n}$,
which is an advantage over other approaches [BB07, GKM06]. Moreover, the
choice of the component saturation function $\widetilde{\varphi}$ is left open
as long as the noise sequence satisfies Assumption 3.1. For example, we can
accommodate standard saturation, piecewise linear, and sigmoidal functions to
name a few.

Our choice of saturating the measurement from the noise vectors, as we shall
see below, renders the resulting optimization problem tractable as opposed to
calculating the entire control input vector $u$ and then saturating it a
posteriori; one can see that the latter approach tends to lead to an
intractable optimization problem. Note also that the choice of control inputs
in (4.1) yields a possibly non-Markovian feedback.

######  Corollary 4.1.

Consider the system (2.1). Suppose that Assumption 3.1 holds, and
$\mathbb{E}[{\mathfrak{e}}(w)]=0$ with ${\mathfrak{e}}(w)=\varphi(w)$, where
$\varphi$ is defined in (4.1). Then for $p=\infty$ the problem (2.6) under the
control policy (4.1) is a convex optimization program with respect to the
decision variables $(\eta,\Theta)$, given by

(4.2) $\displaystyle\underset{(\eta,\Theta)}{\text{minimize}}$
$\displaystyle\mathbf{tr}\\!\left(\Theta^{\mathsf{T}}\bigl{(}R+B^{\mathsf{T}}QB\bigr{)}\Theta\Gamma_{1}\right)+2\mathbf{tr}\\!\left(DQB\Theta\Gamma_{2}\right)$
$\displaystyle+\eta^{\mathsf{T}}\bigl{(}R+B^{\mathsf{T}}QB\bigr{)}\eta+b^{\mathsf{T}}\eta+c$
subject to
$\displaystyle\max\limits_{i=1,\cdots,m}\left(|\eta_{t,i}|+\left\lVert{\Theta_{t,i}}\right\rVert_{1}\phi_{\rm
max}\right)\leqslant U_{\max}^{(\infty)},\quad t=0,\ldots,N-1,$
$\displaystyle\text{and $\Theta$ strictly lower block triangular as
in~{}\eqref{e:Thetadef}},$

where $\eta_{t,i}$ and $\Theta_{t,i}$ are the $i$-th rows of $\eta_{t}$ and
$\Theta_{t}$, respectively,

$\displaystyle c$
$\displaystyle=x_{0}^{\mathsf{T}}AQAx_{0}+\mathbf{tr}\\!\left(D^{\mathsf{T}}QD\Sigma_{\bar{w}}\right),$
$\displaystyle b$ $\displaystyle=2B^{\mathsf{T}}QAx_{0},$
$\displaystyle\Gamma_{1}$
$\displaystyle=\mathrm{diag}\bigl{\\{}\mathbb{E}\bigl{[}\varphi_{0}(w_{0})\varphi_{0}(w_{0})^{\mathsf{T}}\bigr{]},\cdots,\mathbb{E}\bigl{[}\varphi_{N-1}(w_{N-1})\varphi_{N-1}(w_{N-1})^{\mathsf{T}}\bigr{]}\bigr{\\}},$
$\displaystyle\Gamma_{2}$
$\displaystyle=\mathrm{diag}\bigl{\\{}\mathbb{E}\bigl{[}\varphi_{0}(w_{0})w_{0}^{\mathsf{T}}\bigr{]},\cdots,\mathbb{E}\bigl{[}\varphi_{N-1}(w_{N-1})w_{N-1}^{\mathsf{T}}\bigr{]}\bigr{\\}}.$

The resulting policy is guaranteed to satisfy the control constraint set (3.4)
for $p=\infty$.

A complete proof may be found in [HCL09]; it proceeds along the lines of the
proof of Theorem 3.2. Note that the program (4.2) exactly solves (2.6) under
the policy (4.1) and is neither a restriction nor a relaxation.

Problem (4.2) is a quadratic program in the optimization parameters
$(\eta,\Theta)$ (see the discussion following Theorem 3.2). The matrices
$\Gamma_{1}$ and $\Gamma_{2}$ capture the statistics of the noise in the
presence of the functions $\varphi$ and can be computed numerically _off-line_
using Monte Carlo techniques [RC04, Section 3.2]. This method will be utilized
in the examples in Section 6. However, in some instances it is actually
possible to compute these matrices in closed form; this is shown in the next
three examples.

######  Example 4.2.

Let us consider (2.1) when the noise process $(w_{t})_{t\in\mathbb{N}_{0}}$ is
an i.i.d sequence of Gaussian random vectors of mean $0$ and covariance
$\Sigma$ and standard sigmoidal policy functions $\widetilde{\varphi}$, i.e.,
$\widetilde{\varphi}(t)\coloneqq
t/\mathchoice{{\hbox{$\displaystyle\sqrt{1+t^{2}\,}$}\lower 0.4pt\hbox{\vrule
height=7.95523pt,depth=-6.36421pt}}}{{\hbox{$\textstyle\sqrt{1+t^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=7.95523pt,depth=-6.36421pt}}}{{\hbox{$\scriptstyle\sqrt{1+t^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=5.59444pt,depth=-4.47557pt}}}{{\hbox{$\scriptscriptstyle\sqrt{1+t^{2}\,}$}\lower
0.4pt\hbox{\vrule height=4.36427pt,depth=-3.49144pt}}}$. Assume further that
the components of $w_{t}$ are mutually independent, which implies that
$\Sigma$ is a diagonal matrix
$\operatorname{diag}\\{\sigma_{1}^{2},\ldots,\sigma_{n}^{2}\\}$. Then from the
identities in Fact 1 in §A.1, we have for $i=1,\ldots,n$ and $j=0,\ldots,N-1$,

$\displaystyle\mathbb{E}\bigl{[}\widetilde{\varphi}(w_{j,i})^{2}\bigr{]}$
$\displaystyle=\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{i}}\int_{-\infty}^{\infty}\widetilde{\varphi}(t)^{2}\mathrm{e}^{-\frac{t^{2}}{2\sigma_{i}^{2}}}\mathrm{d}t=2\cdot\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{i}}\int_{0}^{\infty}\frac{t^{2}}{1+t^{2}}\mathrm{e}^{-\frac{t^{2}}{2\sigma_{i}^{2}}}$
$\displaystyle=\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{i}-\pi\mathrm{e}^{-\frac{1}{2\sigma_{i}^{2}}}\operatorname{erfc}\Bigl{(}\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule height=3.22221pt,depth=-2.57779pt}}}\sigma_{i}}\Bigr{)}.$

This shows that the matrix $\Gamma_{1}$ in Corollary 4.1 is
$\operatorname{diag}\\{\Sigma^{\prime},\ldots,\Sigma^{\prime}\\}$, where

$\Sigma^{\prime}\coloneqq\operatorname{diag}\left\\{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{1}-\pi\mathrm{e}^{-\frac{1}{2\sigma_{1}^{2}}}\operatorname{erfc}\Bigl{(}\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{1}}\Bigr{)},\ldots,\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{n}-\pi\mathrm{e}^{-\frac{1}{2\sigma_{n}^{2}}}\operatorname{erfc}\Bigl{(}\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{n}}\Bigr{)}\right\\}.$

Similarly, since

$\displaystyle\mathbb{E}\bigl{[}\widetilde{\varphi}(w_{j,i})w_{j,i}\bigr{]}$
$\displaystyle=\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{i}}\int_{-\infty}^{\infty}t\widetilde{\varphi}(t)\mathrm{e}^{-\frac{t^{2}}{2\sigma_{i}^{2}}}\mathrm{d}t=2\cdot\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{i}}\int_{-\infty}^{\infty}\frac{t^{2}}{\mathchoice{{\hbox{$\displaystyle\sqrt{1+t^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=7.95523pt,depth=-6.36421pt}}}{{\hbox{$\textstyle\sqrt{1+t^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=7.95523pt,depth=-6.36421pt}}}{{\hbox{$\scriptstyle\sqrt{1+t^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=5.59444pt,depth=-4.47557pt}}}{{\hbox{$\scriptscriptstyle\sqrt{1+t^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=4.36427pt,depth=-3.49144pt}}}}\mathrm{e}^{-\frac{t^{2}}{2\sigma_{i}}}\mathrm{d}t$
$\displaystyle=\frac{\sigma_{i}}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}}U\Bigl{(}\frac{1}{2},0,\frac{1}{2\sigma_{i}^{2}}\Bigr{)},$

where $U$ is the confluent hypergeometric function (defined in the Appendix),
the matrix $\Gamma_{2}$ in Corollary 4.1 is
$\operatorname{diag}\\{\Sigma^{\prime\prime},\ldots,\Sigma^{\prime\prime}\\}$,
where

$\Sigma^{\prime\prime}\coloneqq\operatorname{diag}\left\\{\frac{\sigma_{1}}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}}U\Bigl{(}\frac{1}{2},0,\frac{1}{2\sigma_{1}^{2}}\Bigr{)},\ldots,\frac{\sigma_{n}}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}}U\Bigl{(}\frac{1}{2},0,\frac{1}{2\sigma_{n}^{2}}\Bigr{)}\right\\}.$

Therefore, given the system (2.1), the control policy (4.6), and the
description of the noise input as above, the matrices $\Gamma_{1}$ and
$\Gamma_{2}$ derived above complete the set of hypotheses of Corollary 4.1.
The problem (2.4) can now be solved as the quadratic program (4.2).$\triangle$

######  Example 4.3.

Consider the setting of Example 4.2 (with $\widetilde{\varphi}$ a standard
sigmoid) under the assumption that $\Sigma$ is a not necessarily diagonal
matrix. To wit, the components of $w_{t}$ may be correlated at each time
$t\in\mathbb{N}_{0}$; however, the random vector sequence
$(w_{t})_{t\in\mathbb{N}_{0}}$ is assumed to be i.i.d. This is equivalent to
the knowledge of the correlations between the random variables
$\bigl{\\{}w_{t,i}\big{|}i=1,\ldots,n\bigr{\\}}$, which are constant over $t$.
Then $\mathbb{E}[\varphi(\bar{w})\varphi(\bar{w})^{\mathsf{T}}]$ is a block
diagonal matrix. Indeed, we have with
$\Sigma_{i,j}\coloneqq\begin{bmatrix}\sigma_{i}^{2}&\rho_{i,j}^{2}\\\
\rho_{i,j}^{2}&\sigma_{j}^{2}\end{bmatrix}$,

$\displaystyle\mathbb{E}\bigl{[}$
$\displaystyle\widetilde{\varphi}(w_{t,i})\widetilde{\varphi}(w_{t,j})\bigr{]}$
$\displaystyle=\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\det{\Sigma_{i,j}}\,}$}\lower
0.4pt\hbox{\vrule
height=6.94444pt,depth=-5.55559pt}}}{{\hbox{$\textstyle\sqrt{2\pi\det{\Sigma_{i,j}}\,}$}\lower
0.4pt\hbox{\vrule
height=6.94444pt,depth=-5.55559pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\det{\Sigma_{i,j}}\,}$}\lower
0.4pt\hbox{\vrule
height=4.8611pt,depth=-3.8889pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\det{\Sigma_{i,j}}\,}$}\lower
0.4pt\hbox{\vrule
height=3.47221pt,depth=-2.77779pt}}}}\iint_{\mathbb{R}^{2}}\frac{t_{1}t_{2}}{\mathchoice{{\hbox{$\displaystyle\sqrt{(1+t_{1}^{2})(1+t_{2}^{2})\,}$}\lower
0.4pt\hbox{\vrule
height=7.95523pt,depth=-6.36421pt}}}{{\hbox{$\textstyle\sqrt{(1+t_{1}^{2})(1+t_{2}^{2})\,}$}\lower
0.4pt\hbox{\vrule
height=7.95523pt,depth=-6.36421pt}}}{{\hbox{$\scriptstyle\sqrt{(1+t_{1}^{2})(1+t_{2}^{2})\,}$}\lower
0.4pt\hbox{\vrule
height=5.59444pt,depth=-4.47557pt}}}{{\hbox{$\scriptscriptstyle\sqrt{(1+t_{1}^{2})(1+t_{2}^{2})\,}$}\lower
0.4pt\hbox{\vrule
height=4.36427pt,depth=-3.49144pt}}}}\exp\biggl{(}-\frac{1}{2}\begin{bmatrix}t_{1}&t_{2}\end{bmatrix}\Sigma_{i,j}^{-1}\begin{bmatrix}t_{1}\\\
t_{2}\end{bmatrix}\biggr{)}\;\mathrm{d}t_{1}\mathrm{d}t_{2},$

and

$\displaystyle\mathbb{E}\bigl{[}$
$\displaystyle\widetilde{\varphi}(w_{t,i})w_{t,j}\bigr{]}$
$\displaystyle=\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\det\Sigma_{i,j}\,}$}\lower
0.4pt\hbox{\vrule
height=6.94444pt,depth=-5.55559pt}}}{{\hbox{$\textstyle\sqrt{2\pi\det\Sigma_{i,j}\,}$}\lower
0.4pt\hbox{\vrule
height=6.94444pt,depth=-5.55559pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\det\Sigma_{i,j}\,}$}\lower
0.4pt\hbox{\vrule
height=4.8611pt,depth=-3.8889pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\det\Sigma_{i,j}\,}$}\lower
0.4pt\hbox{\vrule
height=3.47221pt,depth=-2.77779pt}}}}\iint_{\mathbb{R}^{2}}\frac{t_{1}t_{2}}{\mathchoice{{\hbox{$\displaystyle\sqrt{1+t_{1}^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=7.95523pt,depth=-6.36421pt}}}{{\hbox{$\textstyle\sqrt{1+t_{1}^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=7.95523pt,depth=-6.36421pt}}}{{\hbox{$\scriptstyle\sqrt{1+t_{1}^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=5.59444pt,depth=-4.47557pt}}}{{\hbox{$\scriptscriptstyle\sqrt{1+t_{1}^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=4.36427pt,depth=-3.49144pt}}}}\exp\biggl{(}-\frac{1}{2}\begin{bmatrix}t_{1}&t_{2}\end{bmatrix}\Sigma_{i,j}^{-1}\begin{bmatrix}t_{1}\\\
t_{2}\end{bmatrix}\biggr{)}\;\mathrm{d}t_{1}\mathrm{d}t_{2}.$

Note that the computations of the integrals above can be carried out off-line.
We define the matrices $\Sigma_{t}$ and $\Sigma_{t}^{\prime}$ with the
$(i,j)$-th entry of $\Sigma_{t}$ being
$\mathbb{E}\bigl{[}\widetilde{\varphi}(w_{t,i})\widetilde{\varphi}(w_{t,j})\bigr{]}$
and the $(i,j)$-th entry of $\Sigma_{t}^{\prime}$ being
$\mathbb{E}\bigl{[}\widetilde{\varphi}(w_{t,i})w_{t,j}\bigr{]}$, and it
follows that the matrices
$\Gamma_{1}=\operatorname{diag}\bigl{\\{}\Sigma_{0},\ldots,\Sigma_{N-2}\bigr{\\}}$,
and
$\Gamma_{2}=\operatorname{diag}\bigl{\\{}\Sigma_{0}^{\prime},\ldots,\Sigma_{N-2}^{\prime}\bigr{\\}}$.
$\triangle$

######  Example 4.4.

Consider the system (2.1) as in Example 4.2, and with $\widetilde{\varphi}$
the standard saturation function defined as
$\widetilde{\varphi}(t)=\operatorname{sgn}(t)\min\\{|t|,1\\}$. From Corollary
4.1 we have for $i=1,\ldots,n$ and $j=0,\ldots,N-1$, using the identities in
Fact 1 in §A.1,

$\displaystyle\mathbb{E}\bigl{[}\widetilde{\varphi}(w_{j,i})^{2}\bigr{]}$
$\displaystyle=\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{i}}\int_{-\infty}^{\infty}\widetilde{\varphi}(t)^{2}\mathrm{e}^{-\frac{t^{2}}{2\sigma_{i}^{2}}}\mathrm{d}t$
$\displaystyle=\frac{2}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{i}}\int_{0}^{1}t^{2}\mathrm{e}^{-\frac{t^{2}}{2\sigma_{i}^{2}}}\mathrm{d}t+\frac{2}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{i}}\int_{1}^{\infty}\mathrm{e}^{-\frac{t^{2}}{2\sigma_{i}^{2}}}\mathrm{d}t$
$\displaystyle=\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{i}^{3}\operatorname{erf}\Bigl{(}\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{i}}\Bigr{)}-2\sigma_{i}^{2}\mathrm{e}^{-\frac{1}{2\sigma_{i}^{2}}}+1+\operatorname{erf}\Bigl{(}\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule height=3.22221pt,depth=-2.57779pt}}}\sigma_{i}}\Bigr{)}$
$\displaystyle\eqqcolon\xi_{i}^{\prime}\text{ (say)},$

and

$\displaystyle\mathbb{E}\bigl{[}\widetilde{\varphi}(w_{j,i})w_{j,i}\bigr{]}$
$\displaystyle=\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{i}}\int_{-\infty}^{\infty}t\widetilde{\varphi}(t)\mathrm{e}^{-\frac{t^{2}}{2\sigma_{i}^{2}}}\mathrm{d}t$
$\displaystyle=\frac{2}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{i}}\int_{0}^{1}t^{2}\mathrm{e}^{-\frac{t^{2}}{2\sigma_{i}^{2}}}\mathrm{d}t+\frac{2}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{i}}\int_{1}^{\infty}t\mathrm{e}^{-\frac{t^{2}}{2\sigma_{i}^{2}}}\mathrm{d}t$
$\displaystyle=\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{i}^{3}\operatorname{erf}\Bigl{(}\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma_{i}}\Bigr{)}-2\sigma_{i}^{2}\mathrm{e}^{-\frac{1}{2\sigma_{i}^{2}}}+\mathchoice{{\hbox{$\displaystyle\sqrt{\frac{2}{\pi}\,}$}\lower
0.4pt\hbox{\vrule
height=8.59721pt,depth=-6.8778pt}}}{{\hbox{$\textstyle\sqrt{\frac{2}{\pi}\,}$}\lower
0.4pt\hbox{\vrule
height=6.01805pt,depth=-4.81447pt}}}{{\hbox{$\scriptstyle\sqrt{\frac{2}{\pi}\,}$}\lower
0.4pt\hbox{\vrule
height=4.2986pt,depth=-3.4389pt}}}{{\hbox{$\scriptscriptstyle\sqrt{\frac{2}{\pi}\,}$}\lower
0.4pt\hbox{\vrule
height=4.2986pt,depth=-3.4389pt}}}\sigma_{i}\operatorname{Gamma}(2\sigma_{i}^{2},1)$
$\displaystyle\eqqcolon\xi_{i}^{\prime\prime}\text{ (say)}.$

Therefore, in this case the matrix $\Gamma_{1}$ in Corollary 4.1 is
$\operatorname{diag}\\{\Sigma^{\prime},\ldots,\Sigma^{\prime}\\}$ with
$\Sigma^{\prime}\coloneqq\operatorname{diag}\\{\xi_{1}^{\prime},\ldots,\xi_{n}^{\prime}\\}$,
and the matrix $\Gamma_{2}$ is
$\operatorname{diag}\\{\Sigma^{\prime\prime},\ldots,\Sigma^{\prime\prime}\\}$
with
$\Sigma^{\prime\prime}\coloneqq\operatorname{diag}\\{\xi_{1}^{\prime\prime},\ldots,\xi_{n}^{\prime\prime}\\}$.
These information complete the set of hypotheses of Corollary 4.1, and the
problem (2.4) can now be solved as a quadratic program (4.2).$\triangle$

###  4.2. Bounded controls, bounded noise, and $p=2$

In this subsection we specialize to the case of the noise being drawn from a
compact subset of $\mathbb{R}^{n}$, and the control inputs set
$\mathbb{U}_{2}$. We make the following assumption:

######  Assumption 4.5.

The noise takes values in a compact set
$\mathbb{W}\subseteq\mathbb{R}^{n}$.$\diamondsuit$

Under Assumption 4.5 Hilbert space techniques may be effectively employed in
our basic controller synthesis framework of Section 3 in the following way.
Let $(\mathcal{H},\left\langle{\cdot},{\cdot}\right\rangle_{\mathcal{H}})$ be
a separable Hilbert space of measurable maps
${\mathfrak{e}}:\mathbb{W}\to\mathbb{U}_{2}$ supported on the compact set
$\mathbb{W}$. The inner product is defined as
$\left\langle{\varphi_{1}},{\varphi_{2}}\right\rangle_{\mathcal{H}}\coloneqq\sum_{i=1}^{n}\left\langle{\varphi_{1,i}},{\varphi_{2,i}}\right\rangle$
where $\left\langle{\cdot},{\cdot}\right\rangle$ is the standard inner product
on real-valued functions on $\mathbb{W}$. Fix a complete orthonormal basis
$({\mathfrak{e}}^{\nu})_{\nu\in\mathcal{I}}\subseteq\mathcal{H}$. Since
$\mathcal{H}$ is separable, the set $\mathcal{I}$ is at most countable. Just
as in (3.1) we let our candidate control policies be of the form

(4.3) $u=\begin{bmatrix}\eta_{0}\\\ \eta_{1}\\\ \vdots\\\
\eta_{N-1}\end{bmatrix}+\begin{bmatrix}\mathbf{0}&\mathbf{0}&\cdots&\mathbf{0}\\\
\theta_{1,0}&\mathbf{0}&\cdots&\mathbf{0}\\\
\theta_{2,0}&\theta_{2,1}&\cdots&\mathbf{0}\\\ \vdots&\vdots&\ddots&\vdots\\\
\theta_{N-1,0}&\theta_{N-1,1}&\cdots&\theta_{N-1,N-2}\end{bmatrix}\begin{bmatrix}{\mathfrak{e}}(w_{0})\\\
{\mathfrak{e}}(w_{1})\\\ \vdots\\\
{\mathfrak{e}}(w_{N-2})\end{bmatrix}\eqqcolon\eta+\Theta{\mathfrak{e}}(w),$

where the vector ${\mathfrak{e}}(\cdot)$ is the formal vector formed by
concatenating the (ordered) basis elements
$({\mathfrak{e}}^{\nu})_{\nu\in\mathcal{I}}$, the various $\theta$-s are
formal matrices as in Section 3, and $\eta_{t}$ is an $m$-dimensional vector
for $t=0,\ldots,N-1$. This takes us back to the setting of Section 3.

The following Corollary illustrates the technique explained above; its proof
will only be sketched—it is similar to the proof of Theorem 3.2. Note that for
finite-dimensional Hilbert spaces, depending on the choice of the orthonormal
basis, the matrix $\Theta$ may have complex or real entries.

######  Corollary 4.6.

Consider the system (2.1). Suppose that Assumptions 3.1 and 4.5 hold. Then for
$p=2$ and corresponding control set $\mathbb{U}_{2}$ problem (2.6) under the
policy (4.3) admits convex tractable reformulation with tighter domains of the
decision variables $(\eta,\Theta)$ defined in (4.3), and is equivalent to the
following program:

(4.4) the minimization problem (3.5) $\displaystyle\text{subject
to}\quad\left\lVert{\eta_{t}}\right\rVert+\mathchoice{{\hbox{$\displaystyle\sqrt{N-1\,}$}\lower
0.4pt\hbox{\vrule
height=6.83331pt,depth=-5.46667pt}}}{{\hbox{$\textstyle\sqrt{N-1\,}$}\lower
0.4pt\hbox{\vrule
height=6.83331pt,depth=-5.46667pt}}}{{\hbox{$\scriptstyle\sqrt{N-1\,}$}\lower
0.4pt\hbox{\vrule
height=4.78333pt,depth=-3.82668pt}}}{{\hbox{$\scriptscriptstyle\sqrt{N-1\,}$}\lower
0.4pt\hbox{\vrule
height=3.41666pt,depth=-2.73334pt}}}\left\lVert{\Theta_{t}}\right\rVert\leqslant
U_{\max}^{(2)},\quad\text{for }t=0,\ldots,N-1,$
$\displaystyle\qquad\qquad\qquad\text{and $\Theta$ strictly lower block
triangular as in~{}\eqref{e:Thetadef}}.$

Moreover, if $\hat{\mathcal{H}}$ is a finite-dimensional subspace of
$\mathcal{H}$ spanned by $({\mathfrak{e}}^{\nu})_{\nu\in\mathcal{J}}$ for some
finite $\mathcal{J}\subseteq\mathcal{I}$, then the problem (4.4) admits a
reformulation as a quadratically constrained quadratic program with respect to
the new decision variables $\bigl{(}\eta,\hat{\Theta}\bigr{)}$ corresponding
to $\hat{\mathcal{H}}$, given by

(4.5) the minimization problem (3.5) $\displaystyle\text{subject
to}\quad\left\lVert{\begin{bmatrix}\eta_{t}&\hat{\Theta}_{t}\end{bmatrix}}\right\rVert\leqslant
U_{\max}^{(2)}/\mathchoice{{\hbox{$\displaystyle\sqrt{N\,}$}\lower
0.4pt\hbox{\vrule
height=6.83331pt,depth=-5.46667pt}}}{{\hbox{$\textstyle\sqrt{N\,}$}\lower
0.4pt\hbox{\vrule
height=6.83331pt,depth=-5.46667pt}}}{{\hbox{$\scriptstyle\sqrt{N\,}$}\lower
0.4pt\hbox{\vrule
height=4.78333pt,depth=-3.82668pt}}}{{\hbox{$\scriptscriptstyle\sqrt{N\,}$}\lower
0.4pt\hbox{\vrule height=3.41666pt,depth=-2.73334pt}}}\quad\text{for
}t=0,\ldots,N-1,$ $\displaystyle\qquad\qquad\qquad\text{and $\Theta$ strictly
lower block triangular as in~{}\eqref{e:Thetadef}},$

where the vector ${\hat{\mathfrak{e}}}(\cdot)$ is the vector formed by
concatenating the (ordered) basis elements
$({\mathfrak{e}}^{\nu})_{\nu\in\mathcal{J}}$,
${\hat{\mathfrak{e}}}(w)\coloneqq\bigl{[}{\hat{\mathfrak{e}}}(w_{0})^{\mathsf{T}},\ldots,{\hat{\mathfrak{e}}}(w_{N-2})^{\mathsf{T}}\bigr{]}^{\mathsf{T}}$,
$\hat{\Sigma}_{{\mathfrak{e}}}\coloneqq\mathbb{E}\bigl{[}{\hat{\mathfrak{e}}}(w){\hat{\mathfrak{e}}}(w)^{\mathsf{T}}\bigr{]}$,
$\hat{\Sigma}_{{\mathfrak{e}}}^{\prime}\coloneqq\mathbb{E}\bigl{[}w{\hat{\mathfrak{e}}}(w)^{\mathsf{T}}\bigr{]}$.
In both the above cases the resulting policies are guaranteed to satisfy the
control constraint set (3.4) for $p=2$.

###### Proof.

(Sketch.) Evaluating the objective function in (2.6) gives the objective
function in (3.5). Recall that $\Theta_{t}$ is the $t$-th block row of the
formal matrix $\Theta$, and $\Theta_{t,i}$ is the $i$th sub-row of the block
row $\Theta_{t}$, where $t=0,\ldots,N-1$ and $i=1,\ldots,n$. Applying the
triangle inequality for any $t=0,\ldots,N-1$, we get

$\displaystyle\left\lVert{\eta_{t}+\Theta_{t}{\mathfrak{e}}(w)}\right\rVert$
$\displaystyle\leqslant\left\lVert{\eta_{t}}\right\rVert+\left\lVert{\Theta_{t}{\mathfrak{e}}(w)}\right\rVert=\left\lVert{\eta_{t}}\right\rVert+\mathchoice{{\hbox{$\displaystyle\sqrt{\left\langle{\Theta_{t}{\mathfrak{e}}(w)},{\Theta_{t}{\mathfrak{e}}(w)}\right\rangle_{\mathcal{H}}\,}$}\lower
0.4pt\hbox{\vrule
height=7.5pt,depth=-6.00003pt}}}{{\hbox{$\textstyle\sqrt{\left\langle{\Theta_{t}{\mathfrak{e}}(w)},{\Theta_{t}{\mathfrak{e}}(w)}\right\rangle_{\mathcal{H}}\,}$}\lower
0.4pt\hbox{\vrule
height=7.5pt,depth=-6.00003pt}}}{{\hbox{$\scriptstyle\sqrt{\left\langle{\Theta_{t}{\mathfrak{e}}(w)},{\Theta_{t}{\mathfrak{e}}(w)}\right\rangle_{\mathcal{H}}\,}$}\lower
0.4pt\hbox{\vrule
height=7.5pt,depth=-6.00003pt}}}{{\hbox{$\scriptscriptstyle\sqrt{\left\langle{\Theta_{t}{\mathfrak{e}}(w)},{\Theta_{t}{\mathfrak{e}}(w)}\right\rangle_{\mathcal{H}}\,}$}\lower
0.4pt\hbox{\vrule height=7.5pt,depth=-6.00003pt}}}$
$\displaystyle=\left\lVert{\eta_{t}}\right\rVert+\mathchoice{{\hbox{$\displaystyle\sqrt{\sum_{i=1}^{n}\left\langle{\Theta_{t,i}{\mathfrak{e}}(w)},{\Theta_{t,i}{\mathfrak{e}}(w)}\right\rangle\,}$}\lower
0.4pt\hbox{\vrule
height=7.5pt,depth=-6.00003pt}}}{{\hbox{$\textstyle\sqrt{\sum_{i=1}^{n}\left\langle{\Theta_{t,i}{\mathfrak{e}}(w)},{\Theta_{t,i}{\mathfrak{e}}(w)}\right\rangle\,}$}\lower
0.4pt\hbox{\vrule
height=7.5pt,depth=-6.00003pt}}}{{\hbox{$\scriptstyle\sqrt{\sum_{i=1}^{n}\left\langle{\Theta_{t,i}{\mathfrak{e}}(w)},{\Theta_{t,i}{\mathfrak{e}}(w)}\right\rangle\,}$}\lower
0.4pt\hbox{\vrule
height=7.5pt,depth=-6.00003pt}}}{{\hbox{$\scriptscriptstyle\sqrt{\sum_{i=1}^{n}\left\langle{\Theta_{t,i}{\mathfrak{e}}(w)},{\Theta_{t,i}{\mathfrak{e}}(w)}\right\rangle\,}$}\lower
0.4pt\hbox{\vrule
height=7.5pt,depth=-6.00003pt}}}=\left\lVert{\eta_{t}}\right\rVert+\mathchoice{{\hbox{$\displaystyle\sqrt{(N-1)\sum_{i=1}^{n}\left\lVert{\Theta_{t,i}}\right\rVert^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=9.30444pt,depth=-7.44359pt}}}{{\hbox{$\textstyle\sqrt{(N-1)\sum_{i=1}^{n}\left\lVert{\Theta_{t,i}}\right\rVert^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=9.30444pt,depth=-7.44359pt}}}{{\hbox{$\scriptstyle\sqrt{(N-1)\sum_{i=1}^{n}\left\lVert{\Theta_{t,i}}\right\rVert^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=8.78888pt,depth=-7.03114pt}}}{{\hbox{$\scriptscriptstyle\sqrt{(N-1)\sum_{i=1}^{n}\left\lVert{\Theta_{t,i}}\right\rVert^{2}\,}$}\lower
0.4pt\hbox{\vrule height=8.78888pt,depth=-7.03114pt}}}$
$\displaystyle=\left\lVert{\eta_{t}}\right\rVert+\mathchoice{{\hbox{$\displaystyle\sqrt{N-1\,}$}\lower
0.4pt\hbox{\vrule
height=6.83331pt,depth=-5.46667pt}}}{{\hbox{$\textstyle\sqrt{N-1\,}$}\lower
0.4pt\hbox{\vrule
height=6.83331pt,depth=-5.46667pt}}}{{\hbox{$\scriptstyle\sqrt{N-1\,}$}\lower
0.4pt\hbox{\vrule
height=4.78333pt,depth=-3.82668pt}}}{{\hbox{$\scriptscriptstyle\sqrt{N-1\,}$}\lower
0.4pt\hbox{\vrule
height=3.41666pt,depth=-2.73334pt}}}\left\lVert{\Theta_{t}}\right\rVert$

by orthogonality of the basis elements
$({\mathfrak{e}}^{\nu})_{\nu\in\mathcal{I}}$. The right-hand side of the last
equality appears as the constraint in (4.4).

For the finite-dimensional case (4.5), we note that the objective function is
identical to the one in (4.4), and the constraint in (4.5) follows from the
fact that
$\left\lVert{\eta_{t}+\hat{\Theta}_{t}{\hat{\mathfrak{e}}}(w)}\right\rVert=\left\lVert{\begin{bmatrix}\eta_{t}&\hat{\Theta}_{t}\end{bmatrix}\begin{bmatrix}1\\\
{\hat{\mathfrak{e}}}(w)\end{bmatrix}}\right\rVert$, and
$\left\lVert{\begin{bmatrix}1\\\
{\hat{\mathfrak{e}}}(w)\end{bmatrix}}\right\rVert=\mathchoice{{\hbox{$\displaystyle\sqrt{1+\sum_{i=0}^{N-2}\left\langle{{\hat{\mathfrak{e}}}(w_{i})},{{\hat{\mathfrak{e}}}(w_{i})}\right\rangle\,}$}\lower
0.4pt\hbox{\vrule
height=7.5pt,depth=-6.00003pt}}}{{\hbox{$\textstyle\sqrt{1+\sum_{i=0}^{N-2}\left\langle{{\hat{\mathfrak{e}}}(w_{i})},{{\hat{\mathfrak{e}}}(w_{i})}\right\rangle\,}$}\lower
0.4pt\hbox{\vrule
height=7.5pt,depth=-6.00003pt}}}{{\hbox{$\scriptstyle\sqrt{1+\sum_{i=0}^{N-2}\left\langle{{\hat{\mathfrak{e}}}(w_{i})},{{\hat{\mathfrak{e}}}(w_{i})}\right\rangle\,}$}\lower
0.4pt\hbox{\vrule
height=7.5pt,depth=-6.00003pt}}}{{\hbox{$\scriptscriptstyle\sqrt{1+\sum_{i=0}^{N-2}\left\langle{{\hat{\mathfrak{e}}}(w_{i})},{{\hat{\mathfrak{e}}}(w_{i})}\right\rangle\,}$}\lower
0.4pt\hbox{\vrule
height=7.5pt,depth=-6.00003pt}}}=\mathchoice{{\hbox{$\displaystyle\sqrt{N\,}$}\lower
0.4pt\hbox{\vrule
height=6.83331pt,depth=-5.46667pt}}}{{\hbox{$\textstyle\sqrt{N\,}$}\lower
0.4pt\hbox{\vrule
height=6.83331pt,depth=-5.46667pt}}}{{\hbox{$\scriptstyle\sqrt{N\,}$}\lower
0.4pt\hbox{\vrule
height=4.78333pt,depth=-3.82668pt}}}{{\hbox{$\scriptscriptstyle\sqrt{N\,}$}\lower
0.4pt\hbox{\vrule height=3.41666pt,depth=-2.73334pt}}}$. This leads to a
quadratically constrained quadratic program in the finite- dimensional
decision variables $\bigl{(}\eta,\hat{\Theta}\bigr{)}$. ∎

Let us illustrate the usage of Corollary 4.6 through the following example.

######  Example 4.7.

Consider the system (2.1), and suppose that the $n$ components of the noise
vector $w_{t}$ are independent uniform random variables taking values in
$[-a,a]$ for some $a>1$. Therefore, $\mathbb{W}=[-a,a]^{n}$. It is a standard
fact in Fourier analysis that the system
$\bigl{\\{}\mathrm{e}^{2\pi\mathrm{i}\nu(t/(2a))}\,\big{|}\,\nu\in\mathbb{Z}\bigr{\\}}$
is an orthonormal basis for the Hilbert space of square-integrable functions
on $[-a,a]$ equipped with the standard inner product
$\left\langle{f},{g}\right\rangle\coloneqq\frac{1}{2a}\int_{-a}^{a}f(t)g(t){\mathrm{d}t}$.
We let

$\displaystyle\hat{\mathcal{H}}$
$\displaystyle\coloneqq\operatorname{span}\Biggl{\\{}\biggl{[}\frac{\sin(\pi\nu
t_{1}/a)}{\mathchoice{{\hbox{$\displaystyle\sqrt{n\,}$}\lower
0.4pt\hbox{\vrule
height=4.30554pt,depth=-3.44446pt}}}{{\hbox{$\textstyle\sqrt{n\,}$}\lower
0.4pt\hbox{\vrule
height=4.30554pt,depth=-3.44446pt}}}{{\hbox{$\scriptstyle\sqrt{n\,}$}\lower
0.4pt\hbox{\vrule
height=3.01389pt,depth=-2.41113pt}}}{{\hbox{$\scriptscriptstyle\sqrt{n\,}$}\lower
0.4pt\hbox{\vrule
height=2.15277pt,depth=-1.72223pt}}}},\ldots,\frac{\sin(\pi\nu
t_{n}/2)}{\mathchoice{{\hbox{$\displaystyle\sqrt{n\,}$}\lower
0.4pt\hbox{\vrule
height=4.30554pt,depth=-3.44446pt}}}{{\hbox{$\textstyle\sqrt{n\,}$}\lower
0.4pt\hbox{\vrule
height=4.30554pt,depth=-3.44446pt}}}{{\hbox{$\scriptstyle\sqrt{n\,}$}\lower
0.4pt\hbox{\vrule
height=3.01389pt,depth=-2.41113pt}}}{{\hbox{$\scriptscriptstyle\sqrt{n\,}$}\lower
0.4pt\hbox{\vrule
height=2.15277pt,depth=-1.72223pt}}}}\biggr{]}^{\mathsf{T}}\,\Bigg{|}\,t_{i}\in[-a,a],i=1,\ldots,n,\nu=1,\ldots,M\Biggr{\\}}.$

Let
${\mathfrak{e}}^{\nu}(t_{1},\ldots,t_{n})\coloneqq\mathchoice{{\hbox{$\displaystyle\sqrt{\frac{2}{n}\,}$}\lower
0.4pt\hbox{\vrule
height=8.59721pt,depth=-6.8778pt}}}{{\hbox{$\textstyle\sqrt{\frac{2}{n}\,}$}\lower
0.4pt\hbox{\vrule
height=6.01805pt,depth=-4.81447pt}}}{{\hbox{$\scriptstyle\sqrt{\frac{2}{n}\,}$}\lower
0.4pt\hbox{\vrule
height=4.2986pt,depth=-3.4389pt}}}{{\hbox{$\scriptscriptstyle\sqrt{\frac{2}{n}\,}$}\lower
0.4pt\hbox{\vrule height=4.2986pt,depth=-3.4389pt}}}\bigl{[}\sin(\pi\nu
t_{1}/a),\ldots,\sin(\pi\nu t_{n}/a)\bigr{]}^{\mathsf{T}}$, $t_{i}\in[-a,a]$.
It is clear that the $\mathbb{R}^{n}$-valued functions
$\bigl{\\{}{\mathfrak{e}}^{\nu},\;\nu=1,\ldots,M\bigr{\\}}$ form an
orthonormal set. Indeed,

$\displaystyle\left\langle{{\mathfrak{e}}_{\nu_{1}}},{{\mathfrak{e}}_{\nu_{2}}}\right\rangle_{\hat{\mathcal{H}}}$
$\displaystyle=\sum_{i=1}^{n}\left\langle{{\mathfrak{e}}_{\nu_{1},i}},{{\mathfrak{e}}_{\nu_{2},i}}\right\rangle=\frac{2}{n}\sum_{i=1}^{n}\frac{1}{2a}\int_{-a}^{a}\sin(\pi\nu_{1}t_{i}/a)\sin(\pi\nu_{2}t_{i}/a)\mathrm{d}t_{i}$
$\displaystyle=\frac{2}{n}\sum_{i=1}^{n}\frac{1}{4}\int_{-1}^{1}\bigl{(}\cos((\nu_{1}-\nu_{2})\pi
s_{i})-\cos((\nu_{1}+\nu_{2})\pi s_{i})\bigr{)}\mathrm{d}s_{i}$
$\displaystyle=\begin{cases}\frac{1}{2n}\sum_{i=1}^{n}2=1&\text{if
}\nu_{1}=\nu_{2},\\\ 0&\text{otherwise}.\end{cases}$

We define
$u_{t}\coloneqq\eta_{t}+\Theta_{t}{\mathfrak{e}}(w)=\eta_{t}+\sum_{j=0}^{t-1}\theta_{t,j}{\mathfrak{e}}(w_{j})=\eta_{t}+\sum_{j=0}^{t-1}\sum_{\nu=1}^{M}\theta_{t,j}^{\nu}{\mathfrak{e}}^{\nu}(w_{j})$
for appropriate matrices $\theta_{t,j}^{\nu}$. Now finding policies of the
form (4.3) that minimize the objective function in (2.6) becomes
straightforward in the setting of Corollary 4.6. The matrices
$\Sigma_{{\mathfrak{e}}}$ and $\Sigma_{{\mathfrak{e}}}^{\prime}$ in Corollary
4.6 are now easy to derive from Euler’s identity
$\mathrm{e}^{\mathrm{i}\theta}=\cos\theta+\mathrm{i}\sin\theta$, and the fact
that the characteristic function of a uniform random variable $\zeta$
supported on $[-a,a]$ is given by
$\mathbb{E}\bigl{[}\mathrm{e}^{2\pi\mathrm{i}v\zeta}\bigr{]}=\frac{1}{2a}\int_{-a}^{a}\mathrm{e}^{2\pi\mathrm{i}vt}\,\mathrm{d}t=\operatorname{sinc}(2\pi
va)$ for some $v\in\mathbb{R}$, where the function $\operatorname{sinc}$ is
defined as $\operatorname{sinc}(\xi)\coloneqq\sin(\xi)/\xi$ if $\xi\neq 0$ and
$1$ otherwise.

An alternative representation of the various matrices may be obtained by
looking at each component of the policy elements separately. In this approach
we define

$\displaystyle{\hat{\mathfrak{e}}}(w_{t,i})$
$\displaystyle\coloneqq\begin{bmatrix}{\mathfrak{e}}_{0}(w_{t,i})&{\mathfrak{e}}_{1}(w_{t,i})&\cdots&{\mathfrak{e}}_{M}(w_{t,i})\end{bmatrix}^{\mathsf{T}},$
$\displaystyle{\hat{\mathfrak{e}}}(w_{t})$
$\displaystyle\coloneqq\begin{bmatrix}{\hat{\mathfrak{e}}}(w_{t,1})^{\mathsf{T}}&\cdots&{\hat{\mathfrak{e}}}(w_{t,n})^{\mathsf{T}}\end{bmatrix}^{\mathsf{T}},\quad{\hat{\mathfrak{e}}}(w)\coloneqq\begin{bmatrix}{\hat{\mathfrak{e}}}(w_{0})^{\mathsf{T}}&\cdots&{\hat{\mathfrak{e}}}(w_{N-2})^{\mathsf{T}}\end{bmatrix}^{\mathsf{T}}.$

In the above notation
$\eta_{t,i}+\sum_{j=0}^{t-1}\theta_{j,i}{\hat{\mathfrak{e}}}(w_{j,i})$ is of
course the $i$-th entry of the input $u_{t}$ at time $t$, where
$t=0,\ldots,N-1$ and $i=1,\ldots,n$. $\triangle$

###  4.3. Constraints on control energy

Some applications require constraints on the total control energy expended
over a finite horizon. In the framework that we have established so far, such
constraints are easy to incorporate. Indeed, if we require that
$u^{\mathsf{T}}Su\leqslant\beta^{2}$ for some preassigned $\beta>0$ and
positive definite matrix $S$, then in the setting of Theorem 3.2 this can be
ensured by adjoining the condition
$\left\lVert{\eta}\right\rVert_{S}+\left\lVert{\Theta}\right\rVert_{S}\left\lVert{S}\right\rVert_{\infty}\mathcal{E}\leqslant\beta$
to the constraints, where
$\left\lVert{\eta}\right\rVert_{M}\coloneqq\mathchoice{{\hbox{$\displaystyle\sqrt{\eta^{\mathsf{T}}M\eta\,}$}\lower
0.4pt\hbox{\vrule
height=6.83331pt,depth=-5.46667pt}}}{{\hbox{$\textstyle\sqrt{\eta^{\mathsf{T}}M\eta\,}$}\lower
0.4pt\hbox{\vrule
height=6.83331pt,depth=-5.46667pt}}}{{\hbox{$\scriptstyle\sqrt{\eta^{\mathsf{T}}M\eta\,}$}\lower
0.4pt\hbox{\vrule
height=4.78333pt,depth=-3.82668pt}}}{{\hbox{$\scriptscriptstyle\sqrt{\eta^{\mathsf{T}}M\eta\,}$}\lower
0.4pt\hbox{\vrule height=3.51944pt,depth=-2.81557pt}}}$ is the standard
weighted $2$-norm for a positive definite matrix $M$.

### Comparison with affine policies

As pointed out earlier affine feedback policies from the noise have been
previously treated in [Löf03, BTGGN04, GKM06, GK08], where the following
feedback policy was considered:

(4.6) $u_{t}=\sum_{i=0}^{t-1}\theta_{t,i}w_{i}+\eta_{t}.$

In the deterministic setting it was shown in [GKM06] that there exists a one-
to-one (nonlinear) mapping between control policies in the form (4.6) and the
class of affine state feedback policies. That is, provided one is interested
in affine state feedback policies, the parametrization (4.6) constitutes no
loss of generality. In fact, we shall illustrate in the examples, in the
unconstrained inputs case, that the performance of this strategy with
${\mathfrak{e}}(w_{i})$ in place of $w_{i}$ is almost as good as the standard
LQG controller if not equally good. However, in the constrained inputs case
this choice is suboptimal in the class of measurable control policies, but it
ensures tractability of a large class of optimal control problems. It can be
seen that the solution to the optimization problem (2.4) is tractable with
this parametrization [GKM06]. However, if the elements of the noise vector $w$
are unbounded, the control input (4.6) does not have an upper bound. For the
case of bounded inputs, the control policy (4.6) under unbounded noise will in
general not satisfy the control constraint sets (3.4). This unboundedness is a
potential problem in practical applications, and has been usually circumvented
by assuming that the noise input lies within a compact set [BB07, GKM06] and
designing a worst-case min-max type controller under this assumption.

It is important to point out that our result in Section 4.2 differs from that
in [GKM06] in two aspects. First, we are solving the problem on finite-
dimensional Hilbert spaces with general basis functions as opposed to a finite
collection of affine functions in [GKM06]. Second, the feasibility of our
problem is maintained for any bound on the elements of $\mathbb{W}$, as our
constraint in (4.5) could still produce a feedback gain matrix $\Theta$ that
has norm substantially different that $0$, whereas if there are elements in
$\mathbb{W}$ with large enough norm and we take the control input to be
$u=\eta+\Theta w$, the constraints produce always a solution $\Theta$ with
norm very close to $0$, hence practically only the open-loop term remains in
the case of [GKM06].

##  5\. Stability Analysis

The main result in Theorem 3.2 asserts that the finite horizon optimization
problem (2.6) is convex and tractable using the policy (3.1). To apply this
result in a receding horizon fashion, it is imperative to further study some
qualitative stability properties of the proposed policy. Under this policy the
closed-loop system is not necessarily Markovian, and as such, standard Foster-
Lyapunov methods cannot be directly applied. In what follows, we treat the
stability problem for $p=\infty$ and $U_{\max}\coloneqq U_{\max}^{(\infty)}$.
However, this is without any loss of generality, for the same results hold
(with minor modifications in the proofs) for $p=1,2$ as well. We impose the
following assumption:

######  Assumption 5.1.

The matrix $A$ in (2.1) is Schur stable, i.e., the absolute value of the
eigenvalues of $A$ are all strictly less than $1$.$\diamondsuit$

At a first glance this assumption on $A$ might seem restrictive. Indeed, in
the deterministic setting we know [YSS97] that for discrete-time controlled
systems it is possible to achieve global asymptotic stability with bounded
control inputs if and only if the pair $(A,B)$ is stabilizable with arbitrary
controls, and the spectral radius of $A$ is at most $1$. However, the problem
of ensuring bounded variance of linear stochastic systems with bounded control
inputs is to our knowledge still largely open; see, however, the recent
manuscript [RCMA+09] for partial results as well as in [BSW02, SSW06].

###  5.1. Mean-square boundedness

We shall show that the variance of the state is uniformly bounded under
receding horizon application of the strategy (3.1), for any control horizon
$N_{c}\leqslant N$. The receding horizon implementation is iterative in
nature: the optimization problem is solved every $kN_{c}$ steps, where
$k\in\mathbb{N}_{0}$. The resulting optimal control policy (applied over a
horizon $N_{c}$) is given by

$\pi_{kN_{c}:(k+1)N_{c}-1}^{*}(x_{kN_{c}})\coloneqq\left[\begin{matrix}\pi^{*}_{kN_{c}}(x_{kN_{c}})\\\
\pi^{*}_{kN_{c}+1}(x_{kN_{c}})\\\ \vdots\\\
\pi^{*}_{(k+1)N_{c}-1}(x_{kN_{c}})\end{matrix}\right]=\left[\begin{matrix}\eta_{0}^{*}(x_{kN_{c}})\\\
\eta_{1}^{*}(x_{kN_{c}})+\Theta_{1}^{*}(x_{kN_{c}}){\mathfrak{e}}(w)\\\
\vdots\\\
\eta_{N_{c}-1}^{*}(x_{kN_{c}})+\Theta^{*}_{N_{c}-1}(x_{kN_{c}}){\mathfrak{e}}(w)\end{matrix}\right]$

where the control gains depend explicitly on the initial condition
$x_{kN_{c}}$. For $\ell=1,\cdots,N_{c}$, the resulting closed-loop system over
horizon $N_{c}$ is given by:

(5.1)
$x_{kN_{c}+\ell}=A^{\ell}x_{kN_{c}}+B_{\ell}\pi_{kN_{c}:kN_{c}+\ell-1}^{*}(x_{kN_{c}})+D_{\ell}\tilde{w}_{kN_{c}:kN_{c}+\ell-1},\qquad
k\in\mathbb{N}_{0},$

where
$B_{\ell}\coloneqq\left[\begin{matrix}{\bar{A}}^{\ell-1}\bar{B}&\cdots&\bar{A}\bar{B}&\bar{B}\end{matrix}\right]$,
$D_{\ell}\coloneqq\left[\begin{matrix}{\bar{A}}^{\ell-1}&\cdots&\bar{A}&\mathbf{I}_{n\times
n}\end{matrix}\right]$, and
$\tilde{w}_{kN_{c}:kN_{c}+\ell-1}\coloneqq\left[\begin{matrix}w_{kN_{c}}^{\mathsf{T}}&\cdots&w_{kN_{c}+\ell-1}^{\mathsf{T}}\end{matrix}\right]^{\mathsf{T}}$.

Suppose that the above $N_{c}$-horizon optimal policy is computed as in
Corollary 4.1. We define the receding horizon policy corresponding to the
consecutive concatenation of this $N_{c}$-horizon optimal policy as

(5.2)
$\pi^{*}\coloneqq\bigl{(}\pi_{0:N_{c}-1}^{*}(x_{0}),\;\pi_{N_{c}:2N_{c}-1}^{*}(x_{N_{c}}),\;\pi_{2N_{c}:3N_{c}-1}^{*}(x_{2N_{c}}),\cdots\bigr{)}.$

######  Proposition 5.2.

Consider the system (2.1), and suppose that Assumptions 3.1 and 5.1 hold. For
$p=\infty$ and any control horizon $1\leqslant N_{c}\leqslant N$ the receding
horizon control policy $\pi^{*}$ renders the closed loop system (5.1) mean-
square bounded, i.e.,
$\sup_{t\in\mathbb{N}_{0}}\mathbb{E}_{x_{0}}\bigl{[}\left\lVert{x_{t}}\right\rVert^{2}\bigr{]}<\infty$
for every initial condition $x_{0}\in\mathbb{R}^{n}$.

The proof of this Proposition is postponed to §A.2 in the Appendix.

###  5.2. Input-to-state stability

Input-to-state stability (iss) is an interesting and important qualitative
property of input-output behavior of dynamical systems. In the deterministic
discrete-time setting [JW01], iss generalizes the well-known bounded-input
bounded-output (BIBO) property of linear systems [AM06, p. 490] to the setting
of nonlinear systems. iss provides a description of the behavior of a system
subjected to bounded inputs, and as such it may be viewed as an
$\mathcal{L}_{\infty}$ to $\mathcal{L}_{\infty}$ gain of a given nonlinear
system. In this section we are interested in a useful stochastic variant of
input-to-state stability; see e.g., [Bor00, ST03] for other possible
definitions and ideas (primarily in continuous-time).

######  Definition 5.3.

The system (2.1) is _input-to-state stable in $\mathcal{L}_{1}$_ if there
exist functions $\beta\in\mathcal{KL}$ and
$\alpha,\gamma_{1},\gamma_{2}\in\mathcal{K}_{\infty}$ such that for every
initial condition $x_{0}\in\mathbb{R}^{n}$ we have

(5.3)
$\mathbb{E}_{x_{0}}\bigl{[}\alpha(\left\lVert{x_{t}}\right\rVert)\bigr{]}\leqslant\beta(\left\lVert{x_{0}}\right\rVert,t)+\gamma_{1}\Bigl{(}\sup_{s\in\mathbb{N}_{0}}\left\lVert{u_{s}}\right\rVert_{\infty}\Bigr{)}+\gamma_{2}\bigl{(}\left\lVert{\Sigma}\right\rVert^{\prime}\bigr{)}\qquad\forall\,t\in\mathbb{N}_{0},$

where $\left\lVert{\cdot}\right\rVert^{\prime}$ is an appropriate matrix
norm.$\Diamond$

One evident difference of iss in $\mathcal{L}_{1}$ with the deterministic
definition of iss is the presence of the function $\alpha$ inside the
expectation in (5.3). It turns out that often it is more natural to arrive at
an estimate of $\mathbb{E}_{x_{0}}[\alpha(\left\lVert{x_{t}}\right\rVert)]$
for some $\alpha\in\mathcal{K}_{\infty}$ than an estimate of
$\mathbb{E}_{x_{0}}[\left\lVert{x_{t}}\right\rVert]$. Moreover, in case
$\alpha$ is convex, Jensen’s inequality [Dud02, p. 348] implies that such an
estimate is stronger than an estimate of
$\mathbb{E}_{x_{0}}[\left\lVert{x_{t}}\right\rVert]$.

The property expressed by (5.3) is one possible iss-type property for
stochastic systems. One can come up with alternative stochastic analogs of the
iss property, such as the following: $\forall\,\varepsilon\in\;]0,1[$
$\exists\,\beta\in\mathcal{KL}$ and
$\exists\,\gamma_{1},\gamma_{2}\in\mathcal{K}_{\infty}$ such that
$\mathbb{P}\bigl{(}\left\lVert{x_{t}}\right\rVert\leqslant\beta(\left\lVert{x_{0}}\right\rVert,t)+\gamma(\sup_{s\in\mathbb{N}_{0}}\left\lVert{u_{s}}\right\rVert)+\gamma_{2}(\left\lVert{\Sigma}\right\rVert^{\prime})\;\forall\,t\in\mathbb{N}_{0}\bigr{)}\geqslant
1-\varepsilon$. Intuitively this means that for $1-\varepsilon$ proportion of
the sample paths the deterministic iss property holds uniformly. However, in
an additive i.i.d unbounded noise setting as in (2.1), this property fails to
hold because almost surely the states undergo excursions outside any bounded
set infinitely often; in this case the weaker version:
$\forall\,\varepsilon\in\;]0,1[$ $\exists\,\beta\in\mathcal{KL}$ and
$\exists\,\gamma_{1},\gamma_{2}\in\mathcal{K}_{\infty}$ such that
$\mathbb{P}\bigl{(}\left\lVert{x_{t}}\right\rVert\leqslant\beta(\left\lVert{x_{0}}\right\rVert,t)+\gamma(\sup_{s\in\mathbb{N}_{0}}\left\lVert{u_{s}}\right\rVert)+\gamma_{2}(\left\lVert{\Sigma}\right\rVert^{\prime})\bigr{)}\geqslant
1-\varepsilon\;\forall\,t\in\mathbb{N}_{0}$ is comparatively better suited. We
shall however stick with the iss in $\mathcal{L}_{1}$ property in this
article.

The following Proposition can be established with the aid of Proposition 5.2
for $p=\infty$; the proofs for $p=1$ and $2$ are also similar in spirit.

######  Proposition 5.4.

Consider the system (2.1), and suppose that Assumptions 3.1 and 5.1 hold. Then
the closed-loop system (5.1) is iss in $\mathcal{L}_{1}$ under the policy
$\pi^{*}$ in (5.2) for any $1\leqslant N_{c}\leqslant N$.

##  6\. Numerical Examples

In this section we present several numerical examples to illustrate the
theoretical results in the preceding sections. We start in Example 6.1 by
comparing the performance of our policy (3.3) to that of the standard finite
horizon LQG controller whenever the control inputs set
$\bar{\mathbb{U}}\equiv\mathbb{R}^{m}$, i.e., there are no bounds on the norm
of the inputs. Then we compare the performance of our policy (3.3) against a
saturated LQG controller in Example 6.2. Finally, in Example 6.3 we illustrate
the effectiveness of our policy (3.3) compared to the certainty-equivalent
receding horizon control.

######  Example 6.1 (Unconstrained Inputs).

A natural question that may arise whenever the control inputs in our setup are
not constrained, i.e., $\bar{\mathbb{U}}\equiv\mathbb{R}^{m}$, is the
following: How does the policy (3.3) compare to the globally optimal
controller, which in this case is the standard finite-horizon LQG controller?
One would expect our policy to perform worse on the average since we restrict
to a class of feedback policies that may not contain the globally optimal
controller.

We compared our policy against that of the LQG problem in simulation for two
controllable $3$-dimensional single-input linear systems. In each case we
solved an unconstrained finite-horizon LQ optimal control problem
corresponding to state and control weights $Q_{t}=3\,\mathbf{I}_{3\times 3}$
and $R_{t}=1$ for every $t$. We selected an optimization horizon $N=50$, and
simulated the system responses starting from $10^{3}$ different initial
conditions $x_{0}$ selected at random uniformly from the cube
$[-100,100]^{3}$, and noise sequences $w_{t}$ corresponding to i.i.d Gaussian
noise of mean $0$ and (randomly chosen) variance

$\Sigma_{w}=\begin{bmatrix}2.830399255&5.491512606&3.612257417\\\
5.491512606&11.554870229&6.896706270\\\
3.612257417&6.896706270&4.625993264\end{bmatrix}.$

We selected the nonlinear bounded term ${\mathfrak{e}}(w)$ in our policy
$u=\eta+\Theta{\mathfrak{e}}(w)$ to be a vector of scalar sigmoidal functions
$\varphi(\xi)\coloneqq
0.2\xi/\mathchoice{{\hbox{$\displaystyle\sqrt{1+0.04\xi^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=8.74889pt,depth=-6.99915pt}}}{{\hbox{$\textstyle\sqrt{1+0.04\xi^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=8.74889pt,depth=-6.99915pt}}}{{\hbox{$\scriptstyle\sqrt{1+0.04\xi^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=6.14998pt,depth=-4.92001pt}}}{{\hbox{$\scriptscriptstyle\sqrt{1+0.04\xi^{2}\,}$}\lower
0.4pt\hbox{\vrule height=4.7611pt,depth=-3.8089pt}}}$ applied to each
coordinate of the vector $w$. The covariance matrices
$\Sigma_{{\mathfrak{e}}}$ and $\Sigma_{{\mathfrak{e}}^{\prime}}$ that are
required to solve the optimization problem (3.3) were computed empirically via
classical Monte Carlo methods [RC04, Section 3.2] using $10^{6}$ i.i.d
samples.

The first system is described by:

(6.1) $x_{k+1}=\begin{bmatrix}0&1&0\\\ 0&0&1\\\
0.4&0.5&-0.25\end{bmatrix}x_{k}+\begin{bmatrix}0\\\ 0\\\
1\end{bmatrix}u_{k}+w_{k}.$

The system pair $(A,B)$ is in Brunovsky canonical form, and $A$ has
eigenvalues at $0.8642$, and $-0.5571\pm\mathrm{i}0.3905$. The test results
showed that the mean of the ratio of the cost corresponding to LQG to the cost
corresponding to our policy is $0.99916$, and the standard deviation of this
ratio is $0.003619$.

The second system is described by:

(6.2) $x_{k+1}=\begin{bmatrix}1&1&0\\\ 0&1&1\\\
0&0&1\end{bmatrix}x_{k}+\begin{bmatrix}0\\\ 0\\\ 1\end{bmatrix}u_{k}+w_{k}.$

This particular system matrix $A$ is in Jordan canonical form and has three
eigenvalues at $1$. The test results showed that the mean of the ratio of the
cost of LQG against the cost of our policy is $0.99673$ and the corresponding
standard deviation is $0.008045$.

Computations for determining our policy in the above two cases were carried
out in the MATLAB-based software package cvx. In the case of the system (6.2)
the solver utilized by cvx reported numerical problems in five different runs,
for which it gave values of the aforementioned ratio below $0.96$. Note that
we have not discarded these five cases from the mean and variance figures
reported above.

The close-to-optimal performance of our policy is surprising in view of the
fact that the vector-space $\mathcal{H}$ is the linear span of one bounded
function, and does not contain the theoretically optimal linear (in the
current state) controller. We conjecture that this is due to injectivity of
the mapping ${\mathfrak{e}}$, due to which ${\mathfrak{e}}(w_{t})$ retains all
information generated by $w_{t}$. Of course, in the absence of control
constraints our solution is much more computationally demanding than the LQG
controller, and would not be used in practice in this case.$\triangle$

######  Example 6.2 (Saturated LQG and Receding Horizon).

We compare the performance of saturated LQG against our policy (3.3) for the
system (6.2) in this example. We fixed the optimization horizon $N=2$, the
control horizon $N_{c}=1$, and the weight matrices for the states and the
control to be $Q_{t}=\mathbf{I}_{3\times 3}$ and $R_{t}=0.01$ for all $t$,
respectively. The control bounds in both cases was $[-2,2]$, the nonlinear
bounded term ${\mathfrak{e}}(w_{t})$ in our policy
$u=\eta+\Theta{\mathfrak{e}}(w)$ was a vector of scalar standard saturation
functions applied to each coordinate of the vector $w_{t}$, and the LQG
control input was saturated at $\pm 2$. The covariance matrices
$\Sigma_{{\mathfrak{e}}}$ and $\Sigma_{{\mathfrak{e}}^{\prime}}$ required to
solve the optimization problem (3.5) were computed empirically via classical
Monte Carlo integration methods [RC04, Section 3.2] using $10^{6}$ i.i.d
samples.

We simulated the system (6.2) starting from the same initial condition
$x_{0}=\left[\begin{matrix}0&0&0\end{matrix}\right]^{\mathsf{T}}$ for $100$
different independent realizations of the noise sequence $w_{t}$ over a
horizon of $200$. The behavior of the average (over the $100$ realizations)
cost corresponding to the two scenarios is shown in Figure 1. The simulations
were coded in MATLAB and the optimization programs were coded in the software
package cvx. The average total cost incurred at the end of the simulation
horizon when using the saturated LQG scheme above was $1.790\times 10^{12}$
units, whereas the average total cost incurred at the end of the simulation
horizon ($t=200$) using our policy (3.3) in a receding horizon fashion was
$4.486\times 10^{8}$ units.$\triangle$

Figure 1. Plots of average costs corresponding to saturated LQG and our
receding horizon scheme for $N_{c}=1$ in Example 6.2.

######  Example 6.3 (Constrained Inputs).

Consider the 2-dimensional linear stochastic system:

(6.3) $x_{t+1}=\begin{bmatrix}1.23&-0.15\\\
0.25&1\end{bmatrix}x_{t}+\begin{bmatrix}0.14\\\ 0.12\end{bmatrix}u_{t}+w_{t},$

where $(w_{t})_{t\in\mathbb{N}_{0}}$ is a sequence of i.i.d Gaussian noise
with zero mean and (randomly generated) variance
$\begin{bmatrix}2.722030613&4.975999693\\\
4.975999693&9.102559685\end{bmatrix}$. Let the weight matrices corresponding
to the states and control be $Q_{t}=\mathbf{I}_{2\times 2}$ and $R_{t}=0.8$
for each $t$. The covariance matrices $\Sigma_{{\mathfrak{e}}}$ and
$\Sigma_{{\mathfrak{e}}^{\prime}}$ that are required to solve the optimization
problem (3.3) were computed empirically via classical Monte Carlo integration
methods [RC04, Section 3.2] using $10^{6}$ samples.

We fixed the optimization horizon $N=7$, the nonlinear saturation
${\mathfrak{e}}(w_{t})$ to be a vector of scalar sigmoidal functions
$\varphi(\xi)\coloneqq
0.2\xi/\mathchoice{{\hbox{$\displaystyle\sqrt{1+0.04\xi^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=8.74889pt,depth=-6.99915pt}}}{{\hbox{$\textstyle\sqrt{1+0.04\xi^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=8.74889pt,depth=-6.99915pt}}}{{\hbox{$\scriptstyle\sqrt{1+0.04\xi^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=6.14998pt,depth=-4.92001pt}}}{{\hbox{$\scriptscriptstyle\sqrt{1+0.04\xi^{2}\,}$}\lower
0.4pt\hbox{\vrule height=4.7611pt,depth=-3.8089pt}}}$ applied to each
coordinate of the vector $w_{t}$, and compared the certainty-equivalent MPC
strategy ($N_{c}=1$, $\Theta\equiv 0$, $w_{t}\equiv 0$) against our receding
horizon strategy (3.3) with control horizon $N_{c}=4$. The control constraints
in both cases were $u_{t}\in[-200,200]$. We simulated the system in both cases
starting from the same initial condition
$x_{0}=\left[\begin{matrix}0&0\end{matrix}\right]^{\mathsf{T}}$, for $60$
different realizations of the noise sequence $w_{t}$; plots of states, average
cost, and standard deviation are shown in Figures 2 and 3. The average cost
incurred when using the certainty-equivalent MPC scheme was $7.893\times
10^{5}$ units, whereas the average cost incurred when using our policy (3.3)
in a receding horizon fashion was $3.141\times 10^{5}$ units. Therefore,
applying our policy in a receding horizon fashion one saves $60.2\%$ of the
cost corresponding to the certainty-equivalent MPC controller on the average.
This example illustrates that there may be cases where open-loop certainty-
equivalent MPC, in the absence of state-constraints, is outperformed by a
large margin by a judiciously selected receding-horizon strategy. The
simulations were coded in YALMIP and were solved using SDPT-3; the solver-time
statistics (in sec.) for the certainty-equivalent MPC and receding horizon
schemes were as follows:

| certainty-equivalent MPC | receding horizon
---|---|---
Mean | $32.127$ | $59.615$
Standard deviation | $4.610$ | $21.675$
Maximum | $50.590$ | $90.036$
Minimum | $20.240$ | $20.466$

These statistics correspond to the above simulations carried out on an
$\text{x}86\\_64$ octa-core machine with 24GB RAM, each processor of which was
an Intel${}^{\text{\textregistered}}$ Xeon${}^{\text{\textregistered}}$ CPU
E5540 2.53GHz with cache size 8192 KB, running GNU/Linux.

Figure 2. Plots of states corresponding to: certainty-equivalent MPC with
$N_{c}=1$ (left) and our receding horizon control scheme with $N_{c}=4$
(right) in Example 6.3.

(a) Plot of average costs (b) Plot of standard deviations

Figure 3. Plots of average cost (left) and standard deviations (right)
corresponding to: certainty-equivalent MPC with $N_{c}=1$ and our receding
horizon control scheme with $N_{c}=4$ in Example 6.3.

We also applied the first four control values of the certainty-equivalent
scheme and compared it against our receding horizon scheme using policy (3.3),
i.e., $N_{c}=4$ for both controllers. We simulated the system in both cases
starting from the same initial condition
$x_{0}=\left[\begin{matrix}0&0\end{matrix}\right]^{\mathsf{T}}$, for $60$
different realizations of the noise sequence $w_{t}$; plots of the states,
average cost, and standard deviation are shown in Figures 4 and 5. The average
cost incurred when using the certainty-equivalent with control horizon
$N_{c}=4$ was $4.211\times 10^{5}$ units, whereas the average cost incurred
when using our policy (3.3) in a receding horizon fashion was $3.295\times
10^{5}$ units. We see that by applying our policy in a receding horizon
fashion one saves $21.7\%$ of the cost corresponding to the certainty
equivalence controller on the average. The simulations were coded in YALMIP
and were solved using SDPT-3; the solver-time statistics (in sec.) for the
certainty-equivalent and receding horizon schemes were as follows:

| certainty-equivalent | receding horizon
---|---|---
Mean | $7.537$ | $67.494$
Standard deviation | $0.812$ | $11.845$
Maximum | $9.776$ | $85.232$
Minimum | $6.101$ | $43.601$

These statistics correspond to the above simulations carried out on an
$\text{x}86\\_64$ octa-core machine with 24GB RAM, each processor of which was
an Intel${}^{\text{\textregistered}}$ Xeon${}^{\text{\textregistered}}$ CPU
E5540 2.53GHz with cache size 8192 KB, running GNU/Linux.$\triangle$

Figure 4. Plots of states corresponding to: certainty-equivalent with
$N_{c}=4$ (left) and our receding horizon control scheme with $N_{c}=4$
(right) in Example 6.3.

(a) Plot of average costs (b) Plot of standard deviations

Figure 5. Plots of average cost (left) and standard deviations (right)
corresponding to: certainty-equivalent with $N_{c}=4$ and our receding horizon
control scheme with $N_{c}=4$ in Example 6.3.

##  7\. Conclusion and Future Directions

We provided tractable solutions to a variety of finite-horizon stochastic
optimal control problems with quadratic cost, hard control constraints, and
unbounded additive noise. These problems arise as parts of solutions to the
stochastic receding horizon problems (2.4). The control policy obtained as a
result of the finite-horizon optimal control sub-problems may be nonlinear
with respect to the previous states, and the policy elements are chosen from a
vector space that is largely up to the designer. One of the key features of
our approach is that the variance-like matrices employed in the finite-horizon
optimal control sub-problems may be computed off-line, and we illustrated this
feature with several examples. We demonstrated that applying our obtained
policies in a receding horizon fashion results in bounded state variance.
Finally, we provided several numerical examples that illustrate the
effectiveness of our method with respect to the commonly used certainty-
equivalent MPC controllers.

The development in this article affords extensions in several directions. One
is the incorporation of state constraints. As discussed in §1, hard state
constraints do not make sense in the stochastic with additive unbounded noise
setting unless one is prepared to artificially relax them once infeasibility
is encountered. Probabilistic constraints and integrated chance constraints
[Han83] constitute popular alternative methods to impose constraints on the
state that are more probabilistic in nature. It will be interesting to see how
the approach introduced in this article reacts to state-constraints. A second
direction is to consider specific kinds of nonlinear models, particularly
those which involve multiplicative noise, in our framework, and a third is to
consider different objective functions such as affine functions given by the
$\ell_{\infty}$ and the $\ell_{1}$ norms.

## Acknowledgments

We are indebted to Soumik Pal for pointing out the possibility of representing
policies as elements of a vector space. We thank Colin Jones for some useful
discussions on convexity of some of the optimization programs, and the three
anonymous reviewers for their valuable suggestions that have led to
substantial improvements of the original manuscript.

## References

  * [ACCL09] M. Agarwal, E. Cinquemani, D. Chatterjee, and J. Lygeros, _On convexity of stochastic optimization problems with constraints_ , European Control Conference, 2009, pp. 2827–2832.
  * [AM06] P. J. Antsaklis and A. N. Michel, _Linear Systems_ , Birkhäuser Boston Inc., Boston, MA, 2006.
  * [AS64] M. Abramowitz and I. A. Stegun, _Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables_ , National Bureau of Standards Applied Mathematics Series, vol. 55, Superintendent of Documents, U.S. Government Printing Office, Washington D.C., 1964.
  * [Bat04] I. Batina, _Model predictive control for stochastic systems by randomized algorithms_ , Ph.D. thesis, Technische Universiteit Eindhoven, 2004.
  * [BB07] D. Bertsimas and D. B. Brown, _Constrained stochastic LQC: a tractable approach_ , IEEE Transactions on Automatic Control 52 (2007), no. 10, 1826–1841.
  * [Ber05] D. P. Bertsekas, _Dynamic programming and suboptimal control: a survey from ADP to MPC_ , European Journal of Control 11 (2005), no. 4-5, 310–334.
  * [Ber09] D. S. Bernstein, _Matrix Mathematics_ , 2 ed., Princeton University Press, 2009.
  * [Bla99] F. Blanchini, _Set invariance in control_ , Automatica 35 (1999), no. 11, 1747–1767.
  * [BM99] A. Bemporad and M. Morari, _Robust model predictive control: a survey_ , Robustness in Identification and Control 245 (1999), 207–226.
  * [Bor00] V. S. Borkar, _Uniform stability of controlled Markov processes_ , System theory: modeling, analysis and control (Cambridge, MA, 1999), Kluwer International Series in Engineering Computer Science, vol. 518, Kluwer Academic Publishers, Boston, MA, 2000, pp. 107–120.
  * [Bro97] R. W. Brockett, _Minimum attention control_ , Proceedings of the 36th IEEE Conference on Decision and Control, vol. 3, 1997, pp. 2628–2632.
  * [BSM03] A. Berman and N. Shaked-Monderer, _Completely Positive Matrices_ , World Scientific Publishing Co. Inc., River Edge, NJ, 2003.
  * [BSW02] I. Batina, A. A. Stoorvogel, and S. Weiland, _Optimal control of linear, stochastic systems with state and input constraints_ , Proceedings of the 41st IEEE Conference on Decision and Control, vol. 2, 2002, pp. 1564–1569.
  * [BT96] D. Bertsekas and J. Tsitsiklis, _Neuro-Dynamic Programming_ , Athena Scientific, 1996.
  * [BTGGN04] A. Ben-Tal, A. Goryashko, E. Guslitzer, and A. Nemirovski, _Adjustable robust solutions of uncertain linear programs_ , Mathematical Programming 99 (2004), no. 2, 351–376.
  * [BV04] S. Boyd and L. Vandenberghe, _Convex Optimization_ , Cambridge University Press, Cambridge, 2004, Sixth printing with corrections, 2008.
  * [BW07] L. Blackmore and B. C. Williams, _Optimal, robust predictive control of nonlinear systems under probabilistic uncertainty using particles_ , Proceedings of the American Control Conference, 2007, pp. 1759–1761.
  * [CACL09] E. Cinquemani, M. Agarwal, D. Chatterjee, and J. Lygeros, _On convex problems in chance-constrained stochastic model predictive control_ , http://arxiv.org/abs/0905.3447, 2009.
  * [CCCL08] D. Chatterjee, E. Cinquemani, G. Chaloulos, and J. Lygeros, _Stochastic optimal control up to a hitting time: optimality and rolling-horizon implementation_ , http://arxiv.org/abs/0806.3008, 2008.
  * [CKW08] M. Cannon, B. Kouvaritakis, and X. Wu, _Probabilistic constrained MPC for systems with multiplicative and additive stochastic uncertainty_ , IFAC World Congress (Seoul, Korea), 2008.
  * [CP09] D. Chatterjee and S. Pal, _An excursion-theoretic view of stability of stochastic hybrid systems_ , http://arxiv.org/abs/0901.2269, 2009.
  * [dFR03] D. P. de Farias and B. Van Roy, _The linear programming approach to approximate dynamic programming_ , Operations Research 51 (2003), no. 6, 850–865.
  * [Dud02] R. M. Dudley, _Real Analysis and Probability_ , Cambridge Studies in Advanced Mathematics, vol. 74, Cambridge University Press, Cambridge, 2002, Revised reprint of the 1989 original.
  * [FB05] H. Fukushima and R. R. Bitmead, _Robust constrained predictive control using comparison model_ , Automatica 41 (2005), no. 1, 97–106.
  * [GB00] M. Grant and S. Boyd, _CVX: Matlab software for disciplined convex programming (web page and software)_ , http://stanford.edu/~boyd/cvx, 2000\.
  * [GK08] P. J. Goulart and E. C. Kerrigan, _Input-to-state stability of robust receding horizon control with an expected value cost_ , Automatica 44 (2008), no. 4, 1171–1174.
  * [GKM06] P. J. Goulart, E. C. Kerrigan, and J. M. Maciejowski, _Optimization over state feedback policies for robust control with constraints_ , Automatica 42 (2006), no. 4, 523–533.
  * [Han83] W. K. Klein Haneveld, _On integrated chance constraints_ , Stochastic programming (Gargnano), Lecture Notes in Control and Inform. Sci., vol. 76, Springer, Berlin, 1983, pp. 194–209.
  * [HCCL10] P. Hokayem, E. Cinquemani, D. Chatterjee, and J. Lygeros, _Stochastic MPC with output feedback and bounded control inputs_ , 2010, Submitted to the American Control Conference.
  * [HCL09] P. Hokayem, D. Chatterjee, and J. Lygeros, _On stochastic model predictive control with bounded control inputs_ , http://arxiv.org/abs/0902.3944, 2009\.
  * [JW01] Z.-P. Jiang and Y. Wang, _Input-to-state stability for discrete-time nonlinear systems_ , Automatica 37 (2001), no. 6, 857–869.
  * [LH07] E. Lavretsky and N. Hovakimyan, _Stable adaptation in the presence of actuator constraints with flight control applications_ , Journal of Guidance Control and Dynamics 30 (2007), no. 2, 337.
  * [LHBW07] M. Lazar, W. P. M. H. Heemels, A. Bemporad, and S. Weiland, _Discrete-time non-smooth nonlinear MPC: stability and robustness_ , Lecture Notes in Control and Information Sciences, vol. 358, Springer-Verlag, 2007, pp. 93–103.
  * [LHC03] E. Lavretsky, N. Hovakimyan, and A. J. Calise, _Upper bounds for approximation of continuous-time dynamics using delayed outputs and feedforward neural networks_ , IEEE Transactions on Automatic Control 48 (2003), no. 9, 1606–1610.
  * [Löf03] J. Löfberg, _Minimax Approaches to Robust Model Predictive Control_ , Ph.D. thesis, Linköpings Universitet, 2003.
  * [Löf04] by same author, _YALMIP : A Toolbox for Modeling and Optimization in MATLAB_ , Proceedings of the CACSD Conference (Taipei, Taiwan), 2004.
  * [LR06] B. Lincoln and A. Rantzer, _Relaxing dynamic programming_ , IEEE Transactions on Automatic Control 51 (2006), no. 8, 1249–1260.
  * [Lue69] D. G. Luenberger, _Optimization by Vector Space Methods_ , J. Wiley & Sons, 1969.
  * [Mac01] J. M. Maciejowski, _Predictive Control with Constraints_ , Prentice Hall, 2001.
  * [MLL05] J. M. Maciejowski, A. Lecchini, and J. Lygeros, _NMPC for complex stochastic systems using Markov Chain Monte Carlo_ , International Workshop on Assessment and Future Directions of Nonlinear Model Predictive Control (Stuttgart, Germany), Lecture Notes in Control and Information Sciences, vol. 358/2007, Springer, 2005, pp. 269–281.
  * [MRRS00] D. Q. Mayne, J. B. Rawlings, C. V. Rao, and P. O. M. Scokaert, _Constrained model predictive control: stability and optimality_ , Automatica 36 (2000), no. 6, 789–814.
  * [OJM08] F. Oldewurtel, C.N. Jones, and M. Morari, _A tractable approximation of chance constrained stochastic MPC based on affine disturbance feedback_ , Proceedings of the 47th IEEE Conference on Decision and Control, 2008, pp. 4731–4736.
  * [Pow07] W. B. Powell, _Approximate Dynamic Programming_ , Wiley Series in Probability and Statistics, Wiley-Interscience [John Wiley & Sons], Hoboken, NJ, 2007.
  * [PS09] J. A. Primbs and C. H. Sung, _Stochastic receding horizon control of constrained linear systems with state and control multiplicative noise_ , IEEE Transactions on Automatic Control 54 (2009), no. 2, 221–230.
  * [RC04] C. P. Robert and G. Casella, _Monte Carlo Statistical Methods_ , 2 ed., Springer, 2004.
  * [RCMA+09] F. Ramponi, D. Chatterjee, A. Milias-Argeitis, P. Hokayem, and J. Lygeros, _Attaining mean square boundedness of a marginally stable noisy linear system with a bounded control input_ , http://arxiv.org/abs/0907.1436, 2009\.
  * [RH05] A. Richards and J. How, _Robust model predictive control with imperfect information_ , Proceedings of the American Control Conference, 2005, pp. 268–273.
  * [SB09a] J. Skaf and S. Boyd, _Design of affine controllers via convex optimization_ , http://www.stanford.edu/~boyd/papers/affine_contr.html, 2009, To appear in IEEE Transactions on Automatic Control.
  * [SB09b] by same author, _Nonlinear Q-design for convex stochastic control_ , IEEE Transactions on Automatic Control 54 (2009), no. 10, 2426–2430.
  * [SS85] P. J. Schweitzer and A. Seidmann, _Generalized polynomial approximations in Markovian decision processes_ , Journal of Mathematical Analysis and Applications 110 (1985), no. 2, 568–582.
  * [SSW06] A. A. Stoorvogel, A. Saberi, and S. Weiland, _On external semi-global stochastic stabilization of linear systems with input saturation_ , http://homepage.mac.com/a.a.stoorvogel/subm03.pdf, 2006.
  * [ST03] J. Spiliotis and J. Tsinias, _Notions of exponential robust stochastic stability, ISS and their Lyapunov characterization_ , International Journal of Robust and Nonlinear Control 13 (2003), no. 2, 173–187.
  * [vHB03] D. H. van Hessem and O. H. Bosgra, _A full solution to the constrained stochastic closed-loop MPC problem via state and innovations feedback and its receding horizon implementation_ , Proceedings of the 42nd IEEE Conference on Decision and Control, vol. 1, 2003, pp. 929–934.
  * [vHB06] by same author, _Stochastic closed-loop model predictive control of continuous nonlinear chemical processes_ , Journal of Process Control 16 (2006), no. 3, 225–241.
  * [YB09] J. Yan and R. Bitmead, _A constrained model-predictive approach to coordinated control_ , To Appear in Automatica, 2009.
  * [YSS97] Y. D. Yang, E. D. Sontag, and H. J. Sussmann, _Global stabilization of linear discrete-time systems with bounded feedback_ , Systems and Control Letters 30 (1997), no. 5, 273–281.

## Appendix

###  A.1. Some identities

Recall the following standard special mathematical functions: the _standard
error function_
$\operatorname{erf}(z)\coloneqq\frac{2}{\mathchoice{{\hbox{$\displaystyle\sqrt{\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.01389pt,depth=-2.41113pt}}}{{\hbox{$\textstyle\sqrt{\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.01389pt,depth=-2.41113pt}}}{{\hbox{$\scriptstyle\sqrt{\pi\,}$}\lower
0.4pt\hbox{\vrule
height=2.10971pt,depth=-1.68779pt}}}{{\hbox{$\scriptscriptstyle\sqrt{\pi\,}$}\lower
0.4pt\hbox{\vrule
height=1.50694pt,depth=-1.20557pt}}}}\int_{0}^{z}\mathrm{e}^{-\frac{t^{2}}{2}}\mathrm{d}t$
and the _complementary error function_ [AS64, p. 297] defined by
$\operatorname{erfc}(z)\coloneqq 1-\operatorname{erf}(z)$ for
$z\in\mathbb{R}$, the _incomplete Gamma function_ [AS64, p. 260] defined by
$\Gamma(a,z)\coloneqq\int_{z}^{\infty}t^{a-1}\mathrm{e}^{-t}\mathrm{d}t$ for
$z,a>0$, the _confluent hypergeometric function_ [AS64, p. 505] defined by
$U(a,b,z)\coloneqq\frac{1}{\Gamma(a)}\int_{0}^{\infty}\mathrm{e}^{-zt}t^{a-1}(1+t)^{b-a-1}\mathrm{d}t$
for $a,b,z>0$, and $\Gamma$ is the standard Gamma function. All of these are
implemented as standard functions in Mathematica. The following facts can be
found in [AS64] and are collected here for completeness.

###### Facts about Special Functions.

For $\sigma^{2}>0$ we have

  * •

$\displaystyle{\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma}\int_{z}^{\infty}\mathrm{e}^{-\frac{t^{2}}{2\sigma^{2}}}\mathrm{d}t=\frac{1}{2}\Bigl{(}1+\operatorname{erf}\Bigl{(}\frac{z}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma}\Bigr{)}\Bigr{)}}$

  * •

$\displaystyle{\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma}\int_{0}^{\infty}\frac{t^{2}}{1+t^{2}}\mathrm{e}^{-\frac{t^{2}}{2\sigma^{2}}}\mathrm{d}t=\frac{1}{2}\Bigl{(}\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma-\pi\mathrm{e}^{-\frac{1}{2\sigma^{2}}}\operatorname{erfc}\Bigl{(}\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma}\Bigr{)}\Bigr{)}}$

  * •

$\displaystyle{\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma}\int_{0}^{1}t^{2}\mathrm{e}^{-\frac{t^{2}}{2\sigma^{2}}}\mathrm{d}t=\mathchoice{{\hbox{$\displaystyle\sqrt{\frac{\pi}{2}\,}$}\lower
0.4pt\hbox{\vrule
height=7.52776pt,depth=-6.02223pt}}}{{\hbox{$\textstyle\sqrt{\frac{\pi}{2}\,}$}\lower
0.4pt\hbox{\vrule
height=5.26944pt,depth=-4.21558pt}}}{{\hbox{$\scriptstyle\sqrt{\frac{\pi}{2}\,}$}\lower
0.4pt\hbox{\vrule
height=3.76387pt,depth=-3.01111pt}}}{{\hbox{$\scriptscriptstyle\sqrt{\frac{\pi}{2}\,}$}\lower
0.4pt\hbox{\vrule
height=3.76387pt,depth=-3.01111pt}}}\sigma^{3}\operatorname{erf}\Bigl{(}\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma}\Bigr{)}-\sigma^{2}\mathrm{e}^{-\frac{1}{2\sigma^{2}}}}$;

  * •

$\displaystyle{\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma}\int_{1}^{\infty}t\mathrm{e}^{-\frac{t^{2}}{2\sigma^{2}}}\mathrm{d}t=\frac{\sigma}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}}\operatorname{Gamma}(2\sigma^{2},1)}$

  * •

$\displaystyle{\frac{1}{\mathchoice{{\hbox{$\displaystyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\pi\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}\sigma}\int_{0}^{\infty}\frac{t^{2}}{\mathchoice{{\hbox{$\displaystyle\sqrt{1+t^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=7.95523pt,depth=-6.36421pt}}}{{\hbox{$\textstyle\sqrt{1+t^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=7.95523pt,depth=-6.36421pt}}}{{\hbox{$\scriptstyle\sqrt{1+t^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=5.59444pt,depth=-4.47557pt}}}{{\hbox{$\scriptscriptstyle\sqrt{1+t^{2}\,}$}\lower
0.4pt\hbox{\vrule
height=4.36427pt,depth=-3.49144pt}}}}\mathrm{e}^{-\frac{t^{2}}{2\sigma^{2}}}\mathrm{d}t=\frac{\sigma}{2\mathchoice{{\hbox{$\displaystyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\textstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=6.44444pt,depth=-5.15558pt}}}{{\hbox{$\scriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=4.51111pt,depth=-3.6089pt}}}{{\hbox{$\scriptscriptstyle\sqrt{2\,}$}\lower
0.4pt\hbox{\vrule
height=3.22221pt,depth=-2.57779pt}}}}U\Bigl{(}\frac{1}{2},0,\frac{1}{2\sigma^{2}}\Bigr{)}}$.

###  A.2. Proof of mean-square boundedness

###### Proof of Proposition 5.2.

Fix $x_{0}\in\mathbb{R}^{n}$. For any $n\times n$ matrix $P=P^{\mathsf{T}}>0$,
using (5.1) and the fact that $\mathbb{E}\left[{\mathfrak{e}}(w)\right]=0$, we
see that for every $\ell=1,\cdots,N_{c}$

$\displaystyle\mathbb{E}_{x_{kN_{c}}}\bigl{[}x_{kN_{c}+\ell}^{\mathsf{T}}Px_{kN_{c}+l}\bigr{]}$
$\displaystyle=x_{kN_{c}}^{\mathsf{T}}(A^{\ell})^{\mathsf{T}}PA^{\ell}x_{kN_{c}}+2x_{kN_{c}}^{\mathsf{T}}(A^{\ell})^{\mathsf{T}}PB_{\ell}\mathbb{E}_{x_{kN_{c}}}\bigl{[}\pi^{*}_{kN_{c}:kN_{c}+\ell-1}(x_{kN_{c}})\bigr{]}$
$\displaystyle\quad+\mathbb{E}_{x_{kN_{c}}}\bigl{[}\left\lVert{B_{\ell}\pi^{*}_{kN_{c}:kN_{c}+\ell-1}(x_{kN_{c}})+D_{\ell}\tilde{w}_{kN_{c}:kN_{c}+\ell-1}}\right\rVert_{P}^{2}\bigr{]},$

where
$\left\lVert{\xi}\right\rVert_{P}\coloneqq\mathchoice{{\hbox{$\displaystyle\sqrt{\xi^{\mathsf{T}}P\xi\,}$}\lower
0.4pt\hbox{\vrule
height=8.85777pt,depth=-7.08626pt}}}{{\hbox{$\textstyle\sqrt{\xi^{\mathsf{T}}P\xi\,}$}\lower
0.4pt\hbox{\vrule
height=8.85777pt,depth=-7.08626pt}}}{{\hbox{$\scriptstyle\sqrt{\xi^{\mathsf{T}}P\xi\,}$}\lower
0.4pt\hbox{\vrule
height=6.22777pt,depth=-4.98224pt}}}{{\hbox{$\scriptscriptstyle\sqrt{\xi^{\mathsf{T}}P\xi\,}$}\lower
0.4pt\hbox{\vrule height=4.83888pt,depth=-3.87112pt}}}$. Using the fact that
$\left\lVert{\pi^{*}_{kN_{c}:kN_{c}+\ell-1}(x_{kN_{c}})}\right\rVert_{\infty}\leqslant
U_{\rm max}$ by construction, we obtain the following bound:

(A.1)
$\displaystyle\mathbb{E}_{x_{kN_{c}}}\bigl{[}x_{kN_{c}+\ell}^{\mathsf{T}}Px_{kN_{c}+\ell}\bigr{]}\leqslant
x_{kN_{c}}^{\mathsf{T}}(A^{\ell})^{\mathsf{T}}PA^{\ell}x_{kN_{c}}+2c_{1\ell}\left\lVert{x_{kN_{c}}}\right\rVert_{\infty}+c_{2\ell},$

where

$\displaystyle c_{1\ell}$ $\displaystyle\coloneqq
m\left\lVert{(A^{\ell})^{\mathsf{T}}PB_{\ell}}\right\rVert_{\infty}U_{\rm
max},$ $\displaystyle c_{2\ell}$ $\displaystyle\coloneqq
m\left\lVert{B_{\ell}^{\mathsf{T}}PB_{\ell}}\right\rVert_{\infty}U_{\max}^{2}+\mathbf{tr}\\!\left(D_{\ell}^{\mathsf{T}}PD_{\ell}\Sigma_{w}\right)$
$\displaystyle\quad+\max_{\tiny{\left\lVert{\Upsilon(x_{kN_{c}})}\right\rVert_{\infty}\leqslant
U_{\max}/\phi_{\max}}}\big{[}\mathbf{tr}\\!\left(\Upsilon(x_{kN_{c}})^{\mathsf{T}}B_{\ell}^{\mathsf{T}}PB_{\ell}\Upsilon(x_{kN_{c}})\Lambda_{1}\right)+2\mathbf{tr}\\!\left(\Upsilon(x_{kN_{c}})^{\mathsf{T}}B_{\ell}^{\mathsf{T}}PD_{\ell}\Lambda_{2}\right)\big{]},$
$\displaystyle\text{and}\quad\Upsilon(x_{kN_{c}})\coloneqq\left[\begin{matrix}\Theta^{*}_{1}(x_{kN_{c}})\\\
\cdots\\\ \Theta^{*}_{N_{c}-1}(x_{kN_{c}})\end{matrix}\right].$

Since $A$ is a Schur stable matrix (and hence so is $A^{\ell}$) there exists
[Ber09, Proposition 11.10.5] a matrix $P_{\ell}=P_{\ell}^{\mathsf{T}}>0$ with
real-valued entries that satisfies
$(A^{\ell})^{\mathsf{T}}P_{\ell}A^{\ell}-P_{\ell}=-\mathbf{I}_{n\times n}$; in
particular, we have
$x_{kN_{c}}^{\mathsf{T}}(A^{\ell})^{\mathsf{T}}P_{\ell}A^{\ell}x_{kN_{c}}\leqslant
x_{kN_{c}}^{\mathsf{T}}P_{\ell}x_{kN_{c}}-x_{kN_{c}}^{\mathsf{T}}x_{kN_{c}}$.
Therefore, with $P=P_{\ell}$ in (A.1) we arrive at

(A.2)
$\displaystyle\mathbb{E}_{x_{kN_{c}}}\bigl{[}x_{kN_{c}+\ell}^{\mathsf{T}}P_{\ell}x_{kN_{c}+\ell}\bigr{]}\leqslant
x_{kN_{c}}^{\mathsf{T}}P_{\ell}x_{kN_{c}}-\left\lVert{x_{kN_{c}}}\right\rVert^{2}+2c_{1\ell}\left\lVert{x_{kN_{c}}}\right\rVert_{\infty}+c_{2\ell}.$

For $\zeta_{\ell}\in\;]\max\\{0,1-\lambda_{\max}(P_{\ell})\\},1[$ let
$r_{\ell}\coloneqq\frac{1}{\zeta_{\ell}}\bigl{(}c_{1\ell}+\mathchoice{{\hbox{$\displaystyle\sqrt{c_{1\ell}^{2}+c_{2\ell}\zeta_{\ell}\,}$}\lower
0.4pt\hbox{\vrule
height=6.94444pt,depth=-5.55559pt}}}{{\hbox{$\textstyle\sqrt{c_{1\ell}^{2}+c_{2\ell}\zeta_{\ell}\,}$}\lower
0.4pt\hbox{\vrule
height=6.94444pt,depth=-5.55559pt}}}{{\hbox{$\scriptstyle\sqrt{c_{1\ell}^{2}+c_{2\ell}\zeta_{\ell}\,}$}\lower
0.4pt\hbox{\vrule
height=4.8611pt,depth=-3.8889pt}}}{{\hbox{$\scriptscriptstyle\sqrt{c_{1\ell}^{2}+c_{2\ell}\zeta_{\ell}\,}$}\lower
0.4pt\hbox{\vrule height=3.47221pt,depth=-2.77779pt}}}\bigr{)}$. Then
elementary properties of the quadratic function
$g(y)\coloneqq-\zeta_{\ell}y^{2}+2c_{1\ell}y+c_{2\ell}$ show that

$\displaystyle-\zeta_{\ell}\left\lVert{x_{kN_{c}}}\right\rVert_{\infty}^{2}+2c_{1\ell}\left\lVert{x_{kN_{c}}}\right\rVert_{\infty}+c_{2\ell}\leqslant
0\quad\text{whenever }\left\lVert{x_{kN_{c}}}\right\rVert_{\infty}>r_{\ell},$

In view of the above fact, simple manipulations in (A.2) lead to

$\displaystyle\mathbb{E}_{x_{kN_{c}}}\bigl{[}x_{kN_{c}+\ell}^{\mathsf{T}}P_{\ell}x_{kN_{c}+\ell}\bigr{]}\leqslant
x_{kN_{c}}^{\mathsf{T}}P_{\ell}x_{kN_{c}}-(1-\zeta_{\ell})\left\lVert{x_{kN_{c}}}\right\rVert^{2}\quad\text{whenever
}\left\lVert{x_{kN_{c}}}\right\rVert_{\infty}>r_{\ell},$

from which, letting
$\rho_{\ell}\coloneqq\Bigl{(}1-\frac{1-\zeta_{\ell}}{\lambda_{\text{max}}(P_{\ell})}\Bigr{)}$,
we arrive at

(A.3)
$\displaystyle\mathbb{E}_{x_{kN_{c}}}\bigl{[}x_{kN_{c}+\ell}^{\mathsf{T}}P_{\ell}x_{kN_{c}+\ell}\bigr{]}\leqslant\rho_{\ell}x_{kN_{c}}^{\mathsf{T}}P_{\ell}x_{kN_{c}}\quad\text{whenever
}\left\lVert{x_{kN_{c}}}\right\rVert_{\infty}>r_{\ell}.$

Let us define

$\displaystyle\rho$
$\displaystyle\coloneqq\max\limits_{\ell=1,\cdots,N_{c}}\rho_{\ell},$
$\displaystyle r^{\prime}$
$\displaystyle\coloneqq\max\limits_{\ell=1,\cdots,N_{c}}r_{\ell},$
$\displaystyle\overline{\lambda}$
$\displaystyle\coloneqq\max\limits_{\ell=1,\dots,N_{c}}\lambda_{\max}(P_{\ell}),$
$\displaystyle\underline{\lambda}$
$\displaystyle\coloneqq\min\limits_{\ell=1,\dots,N_{c}}\lambda_{\min}(P_{\ell}).$

Then we can obtain using (A.3) the conservative bound for every
$\ell=1,\ldots,N_{c}$:

$\mathbb{E}_{x_{kN_{c}}}\bigl{[}x_{kN_{c}+\ell}^{\mathsf{T}}P_{N_{c}}x_{kN_{c}+\ell}\bigr{]}\leqslant\rho^{\prime}x_{kN_{c}}^{\mathsf{T}}P_{N_{c}}x_{kN_{c}}\quad\text{whenever
}\left\lVert{x_{kN_{c}}}\right\rVert_{\infty}>r^{\prime},$

where
$\rho^{\prime}\coloneqq\rho\frac{\overline{\lambda}\lambda_{\max}(P_{N_{c}})}{\underline{\lambda}\lambda_{\min}(P_{N_{c}})}$.
It follows immediately that

(A.4)
$\mathbb{E}_{x_{kN_{c}}}\bigl{[}x_{kN_{c}+\ell}^{\mathsf{T}}P_{N_{c}}x_{kN_{c}+\ell}\bigr{]}\leqslant\rho^{\prime}x_{kN_{c}}^{\mathsf{T}}P_{N_{c}}x_{kN_{c}}+b^{\prime}\mathbf{1}_{K^{\prime}}(x_{kN_{c}}),$

where
$K^{\prime}\coloneqq\bigl{\\{}\xi\in\mathbb{R}^{n}\big{|}\left\lVert{\xi}\right\rVert_{\infty}\leqslant
r^{\prime}\bigr{\\}}$.

Let us define the function $V(\xi)\coloneqq\xi^{\mathsf{T}}P_{N_{c}}\xi$, and
fix $k\in\mathbb{N}$ and $\ell=1,\dots,N_{c}$. Let
$K_{N_{c}}\coloneqq\bigl{\\{}\xi\in\mathbb{R}^{n}\big{|}\left\lVert{\xi}\right\rVert_{\infty}\leqslant
r_{N_{c}}\bigr{\\}}$, $b\coloneqq\sup\limits_{x\in
K}\mathbb{E}_{x}\bigl{[}V(x_{N_{c}})\bigr{]}$, and
$b^{\prime}\coloneqq\max\limits_{\ell=1,\ldots,N_{c}}\sup\limits_{x\in
K^{\prime}}\mathbb{E}_{x}\bigl{[}V(x_{\ell})\bigr{]}$. From (A.4) we get

$\displaystyle\mathbb{E}_{x_{0}}\bigl{[}V(x_{kN_{c}+\ell})\bigr{]}$
$\displaystyle=\mathbb{E}_{x_{0}}\bigl{[}\mathbb{E}\bigl{[}V(x_{kN_{c}+\ell})\,\big{|}\,x_{kN_{c}}\bigr{]}\bigr{]}\leqslant\mathbb{E}_{x_{0}}\bigl{[}\rho^{\prime}V(x_{kN_{c}})+b^{\prime}\mathbf{1}_{K^{\prime}}(x_{kN_{c}})\bigr{]}$