# A New Dissipativity Condition for Asymptotic Stability of Discounted
Economic MPC

Mario Zanon<EMAIL_ADDRESS>Sébastien Gros  IMT School for Advanced
Studies Lucca, Piazza San Francesco 19, 55100, Lucca, Italy  NTNU, Gløshaugen,
Trondheim, Norway

###### Abstract

Economic Model Predictive Control has recently gained popularity due to its
ability to directly optimize a given performance criterion, while enforcing
constraint satisfaction for nonlinear systems. Recent research has developed
both numerical algorithms and stability analysis for the undiscounted case.
The introduction of a discount factor in the cost, however, can be desirable
in some cases of interest, e.g., economics, stochastically terminating
processes, Markov decision processes, etc. Unfortunately, the stability theory
in this case is still not fully developed. In this paper we propose a new
dissipativity condition to prove asymptotic stability in the infinite horizon
case and we connect our results with existing ones in the literature on
discounted economic optimal control. Numerical examples are provided to
illustrate the theoretical results.

###### keywords:

Discounted Optimal Control, Economic MPC, Asymptotic Stability

††thanks: This paper was not presented at any IFAC meeting. Corresponding
author M. Zanon. This paper was partially supported by the Italian Ministry of
University and Research under the PRIN’17 project “Data-driven learning of
constrained control systems” , contract no. 2017J89ARP; by ARTES 4.0 Advanced
Robotics and enabling digital Technologies & Systems 4.0, CUP:
B81J18000090008; and by the Norwegian Research Council project “Safe
Reinforcement-Learning using MPC” (SARLEM).

, ,

##  1 Introduction

Model Predictive Control (MPC) has become popular thanks to its ability to
control nonlinear systems while explicitly imposing constraint satisfaction
and optimizing a given objective [11, 19]. However, the ability to optimize a
given objective is only partially exploited in _tracking MPC_ , where the cost
penalizes the distance from a given (possibly optimal) reference. On the
contrary, in so-called _economic MPC_ a given performance criterion is
directly optimized, with the aim to achieve optimal performance not only at
the reference, but also during transients.

The main drawback of economic MPC is the increased difficulty in guaranteeing
asymptotic stability, compared to the tracking MPC case. This difficulty stems
from the fact that the stage cost is generic and not necessarily positive-
definite with respect to a given steady-state. Recent research has developed a
sound stability theory for undiscounted economic MPC based on the concept of
_strict dissipativity_ [1, 3, 4, 5, 7, 12, 14, 21, 22, 23, 24, 25, 26].

While MPC is commonly formulated in an undiscounted setting, in a number of
cases it makes sense to introduce a discount factor. The discount factor can
carry several meanings, e.g., as a model of the interest rate in case the
problem consists in maximizing a given capital; or as a model of processes
having an uncertain lifetime [18]. Additionally, a discount factor is often
the preferred approach to formulate well-posed Markov Decision Processes
(MDPs), which offer one of the most generic formulation for optimal control.
In particular, MDPs are at the core of Reinforcement Learning techniques [20],
which are most developed for the discounted case, while the undiscounted
setting has received less attention.

Unfortunately, the stability theory for discounted MPC formulations is more
involved than the one for the undiscounted setting. As proven in [17], even
the case of a positive-definite stage cost over an infinite horizon poses
additional difficulties and is nontrivial to analyze. The case of an economic
cost over an infinite horizon has been analyzed in, e.g., [6, 8, 10]. In [9]
the authors propose a local stability analysis which makes it possible to
characterize cases in which the optimal steady-state depends on both the
initial state and the discount factor. Unfortunately, a tight condition for
stability is still not available in the literature, as we will discuss more in
detail in Section 3.

In this paper, we propose a new strict dissipativity concept for discounted
infinite-horizon MPC which is stronger than existing strict dissipativity
concepts but allows us to prove asymptotic stability without additional
conditions. Moreover, we provide a characterization of the optimal steady-
state to which the system is stabilized. Finally, we connect the discounted
case to the undiscounted one by proving that the discount factor can be
eliminated, provided that the cost is suitably modified. Our assumptions allow
us to provide a different insight than the one provided in [6], though we will
show that the two sets of assumption share many common points. A thorough
investigation of the necessity of our assumptions for asymptotic stability
will be the subject of future research.

This paper is organized as follows. We introduce the problem and briefly
recall the main strict dissipativity definitions in Section 2. We define the
new strict dissipativity concept, prove asymptotic stability and characterize
the optimal steady-state in Section 3, where we also compare our assumptions
to those of [6], which studies the same setting. We provide some examples in
Section 4 and conclude with Section 5.

##  2 Preliminaries

We consider nonlinear discrete-time systems with dynamics and stage cost given
respectively by

$\displaystyle{\boldsymbol{{x}}}_{+}={\boldsymbol{{f}}}\left({\boldsymbol{{x}}},{\boldsymbol{{u}}}\right),$
and $\displaystyle L({\boldsymbol{{x}}},{\boldsymbol{{u}}}),$ (1)

where ${\boldsymbol{{x}}}\in\mathbb{R}^{n_{\boldsymbol{{x}}}}$,
${\boldsymbol{{u}}}\in\mathbb{R}^{n_{\boldsymbol{{u}}}}$ denote the states and
inputs respectively. Since we are interested in a discounted optimal control
setting, we further introduce the discount factor $\gamma\in]0,1[$.
Furthermore, the system is subject to the state and input constraints

$\displaystyle\left({\boldsymbol{{x}}},{\boldsymbol{{u}}}\right)\in\mathbb{Z}:=\left\\{\,\left({\boldsymbol{{x}}},{\boldsymbol{{u}}}\right)\,|\,{\boldsymbol{{h}}}\left({\boldsymbol{{x}}},{\boldsymbol{{u}}}\right)\leq
0\,\right\\}.$

For ease of notation and without loss of generality, we will assume that
$L({\boldsymbol{{x}}},{\boldsymbol{{u}}})=\infty$ for all
$\left({\boldsymbol{{x}}},{\boldsymbol{{u}}}\right)\notin\mathbb{Z}$.

For a given policy ${\boldsymbol{{\pi}}}({\boldsymbol{{x}}})$, we will denote
closed-loop trajectories as

$\displaystyle{\boldsymbol{{x}}}_{k+1}^{\boldsymbol{{\pi}}}={\boldsymbol{{f}}}\left({\boldsymbol{{x}}}_{k}^{\boldsymbol{{\pi}}},{\boldsymbol{{\pi}}}\left({\boldsymbol{{x}}}_{k}^{\boldsymbol{{\pi}}}\right)\right),$
$\displaystyle{\boldsymbol{{x}}}_{0}^{\boldsymbol{{\pi}}}={\boldsymbol{{x}}}_{0},$
(2)

where we omit the dependence on ${\boldsymbol{{x}}}_{0}$, since the initial
state is independent of the policy and no confusion can arise. We define the
sets of admissible initial conditions and policies as

$\displaystyle\mathbb{X}_{0}$
$\displaystyle:=\left\\{\,{\boldsymbol{{x}}}_{0}\,|\,\exists\,{\boldsymbol{{\pi}}}\
\mathrm{s.t.}\
{\boldsymbol{{h}}}\left({\boldsymbol{{x}}}_{k}^{{\boldsymbol{{\pi}}}},{\boldsymbol{{\pi}}}\left({\boldsymbol{{x}}}_{k}^{{\boldsymbol{{\pi}}}}\right)\right)\leq
0,\forall\,k\geq 0\,\right\\},$ $\displaystyle\Pi$
$\displaystyle:=\left\\{\,{\boldsymbol{{\pi}}}\,|\,{\boldsymbol{{h}}}\left({\boldsymbol{{x}}}_{k}^{{\boldsymbol{{\pi}}}},{\boldsymbol{{\pi}}}\left({\boldsymbol{{x}}}_{k}^{{\boldsymbol{{\pi}}}}\right)\right)\leq
0,\forall\,{\boldsymbol{{x}}}_{0}\in\mathbb{X}_{0},\,k\geq 0\,\right\\}.$

######  Assumption 1.

Sets $\mathbb{Z}$ and $\mathbb{X}_{0}$ are compact. Moreover,
$|L({\boldsymbol{{x}}},{\boldsymbol{{u}}})|<\infty,\
\forall\,({\boldsymbol{{x}}},{\boldsymbol{{u}}})\in\mathbb{Z}$.

The assumption on boundedness of $L$ is without loss of generality, since one
can exclude all points for which it becomes unbounded by suitably defining
${\boldsymbol{{h}}}$ and, consequently, $\mathbb{Z}$. The role of the
compactness assumption on $\mathbb{Z}$ and $\mathbb{X}_{0}$ is to guarantee
that the functions we will use when defining the cost remain bounded on the
system trajectories. While this assumption can be relaxed, we adopt it here in
order to avoid additional technicalities.

For a given discount factor $\gamma\in]0,1[$, we define the optimal value
function of the MPC problem as

$\displaystyle
V_{\star}^{\gamma}\left({\boldsymbol{{x}}}_{0}\right):=\min_{{\boldsymbol{{\pi}}}\in\Pi}\
\sum_{k=0}^{\infty}\gamma^{k}L\left({\boldsymbol{{x}}}_{k}^{\boldsymbol{{\pi}}},{\boldsymbol{{\pi}}}\left({\boldsymbol{{x}}}_{k}^{\boldsymbol{{\pi}}}\right)\right),$
(3)

and denote any optimal policy as ${\boldsymbol{{\pi}}}_{\star}^{\gamma}$. We
recall that the optimal value function (3) satisfies the Bellman equation

$\displaystyle
V_{\star}^{\gamma}\left({\boldsymbol{{x}}}\right)=\min_{{\boldsymbol{{\pi}}}\in\Pi}\
L({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}({\boldsymbol{{x}}}))+\gamma
V_{\star}^{\gamma}\left({\boldsymbol{{f}}}\left({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}({\boldsymbol{{x}}})\right)\right).$
(4)

In this paper we will compare several infinite-horizon MPC formulations of the
form (3), which differ by the used discount factor or stage cost. With slight
abuse of notation, though ${\boldsymbol{{\pi}}}_{\star}^{\gamma}$ refers to
one of the possibly many policies which are optimal for (3), we will write,
e.g., ${\boldsymbol{{\pi}}}={\boldsymbol{{\pi}}}_{\star}^{\gamma}$ to denote
that a given policy ${\boldsymbol{{\pi}}}$ belongs to the set of optimal
policies of (3). Moreover, throughout the paper we will use the following
definition.

######  Definition 1 (MPC Equivalence).

Two MPC formulations are _equivalent_ if their value functions coincide and
any policy which is optimal for one MPC formulation is also optimal for the
other MPC formulation.

In the following, we will make use of the comparison functions defined by the
function classes $\mathcal{K}$, $\mathcal{K}_{\infty}$, $\mathcal{L}$,
$\mathcal{KL}$. We define $\mathbb{R}_{\geq 0}:=\\{\ x\in\mathbb{R}\ |\ x\geq
0\ \\}$. Function $\alpha:\mathbb{R}_{\geq 0}\to\mathbb{R}_{\geq 0}$ satisfies
$\alpha\in\mathcal{K}$ if it is continuous, zero at zero and strictly
increasing. If additionally $\alpha\in\mathcal{K}$ is radially unbounded, then
$\alpha\in\mathcal{K}_{\infty}$. Function $\delta:\mathbb{R}_{\geq
0}\to\mathbb{R}_{\geq 0}$ satisfies $\delta\in\mathcal{L}$ if it is continuous
and strictly decreasing with $\lim_{t\to\infty}\delta(t)=0$. Finally,
$\beta:\mathbb{R}_{\geq 0}\times\mathbb{R}_{\geq 0}\to\mathbb{R}_{\geq 0}$
satisfies $\beta\in\mathcal{KL}$ if it is continuous,
$\beta(\cdot,t)\in\mathcal{K}$, and $\beta(x,\cdot)\in\mathcal{L}$.

In this paper, we aim at characterizing the tuples
$\left({\boldsymbol{{f}}},L,\gamma\right)$ that yield optimal policies
${\boldsymbol{{\pi}}}_{\star}^{\gamma}$ achieving asymptotic stability to a
given steady state. To that end, we need the following definition.

######  Definition 2.

The steady state

$\displaystyle{\boldsymbol{{x}}}^{\mathrm{s}}={\boldsymbol{{f}}}\left({\boldsymbol{{x}}}^{\mathrm{s}},{\boldsymbol{{u}}}^{\mathrm{s}}\right)$
(5)

is asymptotically stable if there exists a function $\beta\in\mathcal{KL}$
such that all closed-loop trajectories satisfy

$\displaystyle\left\|{\boldsymbol{{x}}}_{k}^{{\boldsymbol{{\pi}}}_{\star}^{\gamma}}-{\boldsymbol{{x}}}^{\mathrm{s}}\right\|\leq\beta\left(\left\|{\boldsymbol{{x}}}_{0}-{\boldsymbol{{x}}}^{\mathrm{s}}\right\|,k\right).$
(6)

Given a steady state
$\left({\boldsymbol{{x}}}^{\mathrm{s}},{\boldsymbol{{u}}}^{\mathrm{s}}\right)$
candidate for proving asymptotic stability, without loss of generality and for
simplicity of notation we will assume throughout the paper that
$L({\boldsymbol{{x}}}^{\mathrm{s}},{\boldsymbol{{u}}}^{\mathrm{s}})=0$.

Since our analysis will be constructed based on a newly defined dissipativity
concept, we summarize next existing dissipativity definitions used to prove
stability in the MPC context.

###  2.1 A Brief Summary on Dissipativity and Stability

In order to study the stability properties of problems for which
$L({\boldsymbol{{x}}},{\boldsymbol{{u}}})\ngeq\alpha(\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\|)$
for some ${\boldsymbol{{x}}}^{\mathrm{s}}$, $\alpha\in\mathcal{K}$, the
following concept of strict dissipativity has been introduced in [1, 3] for
the case of $\gamma=1$:

$\displaystyle\exists\,\lambda:\
L({\boldsymbol{{x}}},{\boldsymbol{{u}}})+\lambda({\boldsymbol{{x}}})-\lambda({\boldsymbol{{f}}}({\boldsymbol{{x}}},{\boldsymbol{{u}}}))\geq\rho(\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\|),$
(7)

for $\rho\in\mathcal{K}$. Under this condition, if $\lambda$ is bounded,
asymptotic stability has been proven in [1]. Necessity of strict dissipativity
has been proven in [14] under an additional controllability assumption, and
provided that
$({\boldsymbol{{x}}}^{\mathrm{s}},{\boldsymbol{{u}}}^{\mathrm{s}})\in\mathrm{int}(\mathbb{Z})$.
This dissipativity concept has been extended in several directions, including
considering the periodic case [16, 25], and approximate economic MPC schemes
[22, 23, 24]. Additionally, some results require strictness also in the
inputs, i.e., $\rho(\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\|)$
is replaced by $\rho(\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}},\
{\boldsymbol{{u}}}-{\boldsymbol{{u}}}^{\mathrm{s}}\|)$. For more details on
the topic we refer to the excellent survey [4] and references therein.

In the context of discounted optimal control, the conditions for stability are
harder to derive. Relevant work includes [17], where the stage cost is assumed
to be positive-definite, and [6, 8, 10], which rely on the following
_discounted strict dissipativity_ concept:

$\displaystyle\exists\,\lambda:\
L({\boldsymbol{{x}}},{\boldsymbol{{u}}})+\lambda({\boldsymbol{{x}}})-\gamma\lambda({\boldsymbol{{f}}}({\boldsymbol{{x}}},{\boldsymbol{{u}}}))\geq\rho(\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\|).$
(8)

In this case, the discounted strict dissipativity condition is not sufficient
to prove stability. We will discuss further in Section 3.4 the required
additional assumptions, and observe that in [6] practical stability is proven,
rather than asymptotic stability.

We propose next a new dissipativity concept which can be interpreted as the
joint conditions (7)-(8), where the stage cost in the first condition needs to
be modified. Since the term discounted strict dissipativity is in use to
define condition (8), we will refer to our new condition as _Strong Discounted
Strict Dissipativity_ (SDSD), since it implies (8) but the converse is in
general not true.

##  3 Strong Discounted Strict Dissipativity

In this section, we consider the stabilization to an arbitrary steady-state
$({\boldsymbol{{x}}}_{\mathrm{s}},{\boldsymbol{{u}}}_{\mathrm{s}})$. To that
end, we will first define SDSD; then, assuming that it holds for
$({\boldsymbol{{x}}}_{\mathrm{s}},{\boldsymbol{{u}}}_{\mathrm{s}})$, we will
prove that it entails asymptotic stability of the closed-loop system to
$({\boldsymbol{{x}}}_{\mathrm{s}},{\boldsymbol{{u}}}_{\mathrm{s}})$.
Afterwards, we will further specify the set of steady-states to which the
system can be stabilized, and describe that set in terms of a steady-state
optimality criterion. To that end, we will further establish an equivalence
between the discounted and undiscounted case. Finally, we will compare our
assumption to an alternative one used in the literature.

Strong Discounted Strict Dissipativity holds if the tuple
$\left({\boldsymbol{{f}}},L,\gamma\right)$ satisfies the following assumption.

######  Assumption 2 (SDSD).

There exist a function $\lambda({\boldsymbol{{x}}})$, continuous at
${\boldsymbol{{x}}}^{\mathrm{s}}$, bounded for bounded ${\boldsymbol{{x}}}$,
satisfying $\lambda({\boldsymbol{{x}}}^{\mathrm{s}})=0$, and a function
$\rho\in\mathcal{K}$ such that:

$\displaystyle(i)\qquad$ $\displaystyle
L({\boldsymbol{{x}}},{\boldsymbol{{u}}})+\lambda({\boldsymbol{{x}}})-\gamma\lambda({\boldsymbol{{f}}}({\boldsymbol{{x}}},{\boldsymbol{{u}}}))\geq\rho(\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\|),$
(9a) $\displaystyle(ii)\qquad$ $\displaystyle
L({\boldsymbol{{x}}},{\boldsymbol{{u}}})+\lambda({\boldsymbol{{x}}})-\lambda({\boldsymbol{{f}}}({\boldsymbol{{x}}},{\boldsymbol{{u}}}))$
$\displaystyle\hskip
32.50005pt+(\gamma-1)V_{\star}^{\gamma}({\boldsymbol{{f}}}({\boldsymbol{{x}}},{\boldsymbol{{u}}}))\geq\rho(\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\|),$
(9b)

hold for all $({\boldsymbol{{x}}},{\boldsymbol{{u}}})\in\mathbb{Z}$.

In the limit $\gamma\to 1$, the two conditions coincide and we recover the
standard definition of strict dissipativity for the undiscounted case (7).
Furthermore, condition (9a) coincides with discounted strict dissipativity
(8).

In the more general case in which the states can also take unbounded values,
additional assumptions need to be introduced to guarantee boundedness of
$\lambda({\boldsymbol{{x}}})$, and the stronger requirement
$\rho\in\mathcal{K}_{\infty}$ is needed. Since the remainder of our analysis
remains valid in that case, we avoid these technicalities for the sake of
simplicity.

Finally, we introduce a standard assumption which can be interpreted as a weak
controllability assumption for cost functions which are sufficiently regular
[19].

######  Assumption 3.

The value function $V_{\star}^{\gamma}({\boldsymbol{{x}}})$ is continuous at
${\boldsymbol{{x}}}^{\mathrm{s}}$, and bounded on $\mathbb{X}_{0}$.

We are now ready to deliver our main results.

###  3.1 Asymptotic Stability

In order to prove asymptotic stability, we will first define a modified stage
cost and recall an existing result which will be exploited in the proof.

We define the following modified cost111in this case we do not adopt the
nomenclature “rotated cost” used in the undiscounted case, which we reserve
for a different definition of stage cost used later., together with the
associated optimal policy and value function:

$\displaystyle\hat{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{u}}})$
$\displaystyle:=L({\boldsymbol{{x}}},{\boldsymbol{{u}}})+\lambda({\boldsymbol{{x}}})-\gamma\lambda({\boldsymbol{{f}}}({\boldsymbol{{x}}},{\boldsymbol{{u}}})),$
(10)
$\displaystyle{\boldsymbol{{\hat{\pi}}}}_{\star}^{\gamma}\left({\boldsymbol{{x}}}_{0}\right)$
$\displaystyle:=\arg\min_{{\boldsymbol{{\pi}}}\in\Pi}\
\sum_{k=0}^{\infty}\gamma^{k}\hat{L}^{\gamma}\left({\boldsymbol{{x}}}_{k}^{\boldsymbol{{\pi}}},{\boldsymbol{{\pi}}}\left({\boldsymbol{{x}}}_{k}^{\boldsymbol{{\pi}}}\right)\right).$
(11)
$\displaystyle\hat{V}_{{\boldsymbol{{\pi}}}}^{\gamma}\left({\boldsymbol{{x}}}_{0}\right)$
$\displaystyle:=\sum_{k=0}^{\infty}\gamma^{k}\hat{L}^{\gamma}\left({\boldsymbol{{x}}}_{k}^{\boldsymbol{{\pi}}},{\boldsymbol{{\pi}}}\left({\boldsymbol{{x}}}_{k}^{\boldsymbol{{\pi}}}\right)\right),\
\
\hat{V}_{\star}^{\gamma}({\boldsymbol{{x}}}):=\hat{V}_{{\boldsymbol{{\hat{\pi}}}}_{\star}^{\gamma}}^{\gamma}({\boldsymbol{{x}}}).$
(12)

Similarly to the undiscounted case [1, 3], the cost modification that yields
$\hat{L}^{\gamma}$ does not affect the primal solution, as stated in the next
theorem.

######  Theorem 1 ([8], [10]).

Suppose that $\lambda$ is bounded. Then, for all
${\boldsymbol{{x}}}_{0}\in\mathbb{X}_{0}$, we have
${\boldsymbol{{\hat{\pi}}}}_{\star}^{\gamma}({\boldsymbol{{x}}})={\boldsymbol{{\pi}}}_{\star}^{\gamma}({\boldsymbol{{x}}})$,
and

$\displaystyle\hat{V}_{\star}^{\gamma}({\boldsymbol{{x}}})=V_{\star}^{\gamma}({\boldsymbol{{x}}})+\lambda({\boldsymbol{{x}}}).$
(13)

We are now ready to prove our main result.

######  Theorem 2 (Asymptotic Stability).

Suppose that Assumptions 1, 2, 3 hold. Then, for all
${\boldsymbol{{x}}}_{0}\in\mathbb{X}_{0}$, $\hat{V}_{\star}^{\gamma}$ is a
Lyapunov function and the system in closed-loop with policy
${\boldsymbol{{\hat{\pi}}}}_{\star}^{\gamma}({\boldsymbol{{x}}})={\boldsymbol{{\pi}}}_{\star}^{\gamma}({\boldsymbol{{x}}})$
is asymptotically stabilized to the steady state
${\boldsymbol{{x}}}^{\mathrm{s}}$.

Using (9a), we obtain

$\displaystyle\hat{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{u}}})\geq\rho(\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\|),$
(14)

such that
$\hat{V}_{\star}^{\gamma}({\boldsymbol{{x}}})\geq\rho(\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\|)$.

Because by assumption $V_{\star}^{\gamma}$ and $\lambda$ are bounded, zero and
continuous at ${\boldsymbol{{x}}}^{\mathrm{s}}$, then also
$\hat{V}_{\star}^{\gamma}$ is bounded, zero and continuous at
${\boldsymbol{{x}}}^{\mathrm{s}}$. Therefore, it can be upper-bounded by a
$\mathcal{K}$ function $\alpha$ such that [19, Proposition B.25]:

$\displaystyle\hat{V}_{\star}^{\gamma}({\boldsymbol{{x}}})\leq\alpha(\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\|).$
(15)

Using (10) and (13), Condition (9b) equivalently reads as

$\displaystyle\hat{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{u}}})+(\gamma-1)\left(V_{\star}^{\gamma}({\boldsymbol{{f}}}({\boldsymbol{{x}}},{\boldsymbol{{u}}}))+\lambda({\boldsymbol{{f}}}({\boldsymbol{{x}}},{\boldsymbol{{u}}}))\right)$
$\displaystyle\hskip
20.00003pt\overset{\eqref{eq:V_Vhat}}{=}\hat{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{u}}})+(\gamma-1)\hat{V}_{\star}^{\gamma}({\boldsymbol{{f}}}({\boldsymbol{{x}}},{\boldsymbol{{u}}}))\geq\rho(\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\|),$
(16)

which entails that $\hat{V}_{\star}^{\gamma}$ satisfies the decrease condition

$\displaystyle\hat{V}_{\star}^{\gamma}\left({\boldsymbol{{f}}}\left({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}_{\star}^{\gamma}\left({\boldsymbol{{x}}}\right)\right)\right)-\hat{V}_{\star}^{\gamma}({\boldsymbol{{x}}})$
$\displaystyle\hskip
20.00003pt=-\left(\hat{L}^{\gamma}\left({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}_{\star}^{\gamma}\left({\boldsymbol{{x}}}\right)\right)+(\gamma-1)\hat{V}_{\star}^{\gamma}\left({\boldsymbol{{f}}}\left({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}_{\star}^{\gamma}\left({\boldsymbol{{x}}}\right)\right)\right)\right)$
$\displaystyle\hskip
20.00003pt\leq-\rho\left(\left\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\right\|\right).$
(17)

$\hfill\qed$

After having proved asymptotic stability, we now turn to the characterization
of the set of optimal steady states, to which the system is stabilized if SDSD
holds. To that end, we will first establish an equivalence between the
discounted MPC and a suitably defined undiscounted MPC.

###  3.2 Equivalence with the Undiscounted Case

In this section, we formulate an undiscounted formulation which yields the
same policy and value function as the discounted one. To that end, we define a
new modified stage cost, which differs from (10) and, as we will discuss
later, is related to condition (9b):

$\displaystyle\tilde{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{u}}})$
$\displaystyle:=L({\boldsymbol{{x}}},{\boldsymbol{{u}}})+(\gamma-1)V_{\star}^{\gamma}({\boldsymbol{{f}}}({\boldsymbol{{x}}},{\boldsymbol{{u}}})),$
(18)

with corresponding undiscounted value function and optimal policy

$\displaystyle{\boldsymbol{{\tilde{\pi}}}}_{\star}^{\gamma}\left({\boldsymbol{{x}}}_{0}\right)$
$\displaystyle:=\arg\min_{{\boldsymbol{{\pi}}}\in\Pi}\
\sum_{k=0}^{\infty}\tilde{L}^{\gamma}\left({\boldsymbol{{x}}}_{k}^{\boldsymbol{{\pi}}},{\boldsymbol{{\pi}}}\left({\boldsymbol{{x}}}_{k}^{\boldsymbol{{\pi}}}\right)\right),$
(19)
$\displaystyle\tilde{V}_{{\boldsymbol{{\pi}}}}^{\gamma}\left({\boldsymbol{{x}}}_{0}\right)$
$\displaystyle:=\sum_{k=0}^{\infty}\tilde{L}^{\gamma}\left({\boldsymbol{{x}}}_{k}^{\boldsymbol{{\pi}}},{\boldsymbol{{\pi}}}\left({\boldsymbol{{x}}}_{k}^{\boldsymbol{{\pi}}}\right)\right),\
\
\tilde{V}_{\star}^{\gamma}({\boldsymbol{{x}}}):=\tilde{V}_{{\boldsymbol{{\tilde{\pi}}}}_{\star}^{\gamma}}^{\gamma}({\boldsymbol{{x}}}).$
(20)

######  Theorem 3.

Suppose that Assumptions 1, 2, 3 hold. Then, for all
${\boldsymbol{{x}}}_{0}\in\mathbb{X}_{0}$,

$\displaystyle{\boldsymbol{{\tilde{\pi}}}}_{\star}^{\gamma}({\boldsymbol{{x}}})={\boldsymbol{{\pi}}}_{\star}^{\gamma}({\boldsymbol{{x}}}),$
$\displaystyle\tilde{V}_{\star}^{\gamma}({\boldsymbol{{x}}})=V_{\star}^{\gamma}({\boldsymbol{{x}}}).$
(21)

We use (18) and (4) to write

$\displaystyle\tilde{L}^{\gamma}\left({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}_{\star}^{\gamma}({\boldsymbol{{x}}})\right)$
$\displaystyle=V_{\star}^{\gamma}({\boldsymbol{{x}}})-V_{\star}^{\gamma}\left({\boldsymbol{{f}}}\left({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}_{\star}^{\gamma}({\boldsymbol{{x}}})\right)\right),$

which we use in (20) to obtain, for all
${\boldsymbol{{x}}}_{0}\in\mathbb{X}_{0}$,

$\displaystyle\tilde{V}_{{\boldsymbol{{\pi}}}_{\star}^{\gamma}}^{\gamma}({\boldsymbol{{x}}}_{0})$
$\displaystyle=\sum_{k=0}^{\infty}V_{\star}^{\gamma}\left({\boldsymbol{{x}}}_{k}^{{\boldsymbol{{\pi}}}_{\star}^{\gamma}}\right)-V_{\star}^{\gamma}\left({\boldsymbol{{f}}}\left({\boldsymbol{{x}}}_{k}^{{\boldsymbol{{\pi}}}_{\star}^{\gamma}},{\boldsymbol{{\pi}}}_{\star}^{\gamma}\left({\boldsymbol{{x}}}_{k}^{{\boldsymbol{{\pi}}}_{\star}^{\gamma}}\right)\right)\right)$
$\displaystyle=V_{\star}^{\gamma}\left({\boldsymbol{{x}}}_{0}\right),$ (22)

where we simplified the terms in the telescopic sum, which are all finite
since ${\boldsymbol{{x}}}_{0}\in\mathbb{X}_{0}$ and Assumptions 1-2 hold, and
we exploited Theorem 2,
$\lambda\left({\boldsymbol{{x}}}^{\mathrm{s}}\right)=0$ and (13) to establish
$\lim_{k\to\infty}V_{\star}^{\gamma}\left({\boldsymbol{{x}}}_{k}^{{\boldsymbol{{\pi}}}_{\star}^{\gamma}}\right)=0$.
Then, we apply the Dynamic Programming recursion, and exploit (18), (3.2) to
derive

$\displaystyle\arg\min_{{\boldsymbol{{\pi}}}\in\Pi}\
\tilde{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}({\boldsymbol{{x}}}))+\tilde{V}_{{\boldsymbol{{\pi}}}_{\star}^{\gamma}}^{\gamma}({\boldsymbol{{f}}}({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}({\boldsymbol{{x}}})))$
$\displaystyle\hskip 10.00002pt=\arg\min_{{\boldsymbol{{\pi}}}\in\Pi}\
L({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}({\boldsymbol{{x}}}))+\gamma
V_{\star}^{\gamma}({\boldsymbol{{f}}}({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}({\boldsymbol{{x}}})))={\boldsymbol{{\pi}}}_{\star}^{\gamma}({\boldsymbol{{x}}}),$

which proves that ${\boldsymbol{{\pi}}}_{\star}^{\gamma}$ is optimal for cost
$\tilde{L}^{\gamma}$ in the undiscounted setting with prediction horizon $1$
and terminal cost
$\tilde{V}_{{\boldsymbol{{\pi}}}_{\star}^{\gamma}}^{\gamma}$. Using (20), we
obtain that $\tilde{V}_{{\boldsymbol{{\pi}}}_{\star}^{\gamma}}^{\gamma}$ is
the resulting value function, such that, by propagating the reasoning over
several steps, by induction we obtain

${\boldsymbol{{\pi}}}_{\star}^{\gamma}\left({\boldsymbol{{x}}}_{0}\right)=\arg\min_{{\boldsymbol{{\pi}}}\in\Pi}\,\lim_{N\to\infty}\sum_{k=0}^{N-1}\tilde{L}^{\gamma}\left({\boldsymbol{{x}}}_{k}^{\boldsymbol{{\pi}}},{\boldsymbol{{\pi}}}\left({\boldsymbol{{x}}}_{k}^{\boldsymbol{{\pi}}}\right)\right)+\tilde{V}_{{\boldsymbol{{\pi}}}_{\star}^{\gamma}}^{\gamma}\left({\boldsymbol{{x}}}_{N}^{\boldsymbol{{\pi}}}\right).$

Note that
$\tilde{V}_{{\boldsymbol{{\pi}}}_{\star}^{\gamma}}^{\gamma}({\boldsymbol{{x}}})=V_{\star}^{\gamma}({\boldsymbol{{x}}})$
is continuous at ${\boldsymbol{{x}}}_{\mathrm{s}}$ by Assumption 3 and
$\tilde{V}_{{\boldsymbol{{\pi}}}_{\star}^{\gamma}}^{\gamma}({\boldsymbol{{x}}}_{\mathrm{s}})=0$.
Moreover, by Theorem 2 policy ${\boldsymbol{{\pi}}}_{\star}^{\gamma}$ yields
asymptotic stability. Therefore, we also have

$\displaystyle{\boldsymbol{{\pi}}}_{\star}^{\gamma}\left({\boldsymbol{{x}}}_{0}\right)=\arg\min_{{\boldsymbol{{\pi}}}\in\Pi}\
$
$\displaystyle\sum_{k=0}^{\infty}\tilde{L}^{\gamma}\left({\boldsymbol{{x}}}_{k}^{\boldsymbol{{\pi}}},{\boldsymbol{{\pi}}}\left({\boldsymbol{{x}}}_{k}^{\boldsymbol{{\pi}}}\right)\right)$
(23a) $\displaystyle\mathrm{s.t.}\ $
$\displaystyle\lim_{k\to\infty}{\boldsymbol{{{\boldsymbol{{x}}}_{k}^{\boldsymbol{{\pi}}}}}}={\boldsymbol{{x}}}_{\mathrm{s}}.$
(23b)

We now observe that, by Assumption 2, Equation (9b) entails that the
undiscounted strict dissipativity criterion (7) holds for the undiscounted
problem (19). This directly entails that
${\boldsymbol{{\tilde{\pi}}}}_{\star}^{\gamma}$, solution of (19), must
asymptotically stabilize the system to the steady state
${\boldsymbol{{x}}}^{\mathrm{s}}$. Consequently,
${\boldsymbol{{\tilde{\pi}}}}_{\star}^{\gamma}$ is a feasible solution of
(23), which entails that, by optimality of
${\boldsymbol{{\pi}}}_{\star}^{\gamma}$ for (23), we must have
$\tilde{V}_{{\boldsymbol{{\tilde{\pi}}}}_{\star}^{\gamma}}^{\gamma}({\boldsymbol{{x}}})\geq\tilde{V}_{{\boldsymbol{{\pi}}}_{\star}^{\gamma}}^{\gamma}({\boldsymbol{{x}}})$.
This proves that ${\boldsymbol{{\pi}}}_{\star}^{\gamma}({\boldsymbol{{x}}})$
must also be optimal for (19) and, consequently,

$\displaystyle\tilde{V}_{\star}^{\gamma}({\boldsymbol{{x}}})=\tilde{V}_{{\boldsymbol{{\pi}}}_{\star}^{\gamma}}^{\gamma}({\boldsymbol{{x}}})\overset{\eqref{eq:Vhat_V}}{=}V_{\star}^{\gamma}({\boldsymbol{{x}}}).$

$\hfill\qed$

By exploiting this equivalence, Condition (9b) can be interpreted as the
standard strict dissipativity condition on the rotated stage cost [1], i.e.,:

$\displaystyle\tilde{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{u}}})+\lambda({\boldsymbol{{x}}})-\lambda({\boldsymbol{{f}}}({\boldsymbol{{x}}},{\boldsymbol{{u}}}))\geq\rho(\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\|).$
(24)

Note, however, that condition (9b) alone is not sufficient for asymptotic
stability in the discounted case, since condition (9a) is necessary in order
to establish the equivalence proven in Theorem 3 and the lower bound for
$\hat{V}_{\star}^{\gamma}$.

###  3.3 Discounted Optimal Steady-State

We define next the optimal steady state to which the closed-loop system
converges, provided that the SDSD Assumption 2 is met. To that end, we use
stage cost $\tilde{L}^{\gamma}$ to formulate the optimization problem

$\displaystyle({\boldsymbol{{x}}}^{\mathrm{s}}_{\star},{\boldsymbol{{u}}}^{\mathrm{s}}_{\star}):=\arg\min_{{\boldsymbol{{x}}},{\boldsymbol{{u}}}}\
$ $\displaystyle\tilde{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{u}}})$
(25a) $\displaystyle\mathrm{s.t.}\ $
$\displaystyle{\boldsymbol{{x}}}={\boldsymbol{{f}}}({\boldsymbol{{x}}},{\boldsymbol{{u}}}).$
(25b)

Note that, in general,
$({\boldsymbol{{x}}}^{\mathrm{s}}_{\star},{\boldsymbol{{u}}}^{\mathrm{s}}_{\star})$
needs not be unique. We prove in the next theorem that, if the SDSD Assumption
2 holds for some steady state, then it must hold for steady-state (25).

######  Theorem 4.

Suppose that Assumptions 1, 2, 3 hold for a given
$({\boldsymbol{{x}}}^{\mathrm{s}},{\boldsymbol{{u}}}^{\mathrm{s}})$. Then,
$({\boldsymbol{{x}}}^{\mathrm{s}},{\boldsymbol{{u}}}^{\mathrm{s}})$ must be a
solution of (25):

$\displaystyle({\boldsymbol{{x}}}^{\mathrm{s}},{\boldsymbol{{u}}}^{\mathrm{s}})\in({\boldsymbol{{x}}}^{\mathrm{s}}_{\star},{\boldsymbol{{u}}}^{\mathrm{s}}_{\star}).$
(26)

By Theorem 2, we have that, from any initial state, the system in closed-loop
with policy
${\boldsymbol{{\hat{\pi}}}}_{\star}^{\gamma}({\boldsymbol{{x}}})={\boldsymbol{{\pi}}}_{\star}^{\gamma}({\boldsymbol{{x}}})$
is stabilized to
${\boldsymbol{{x}}}^{\mathrm{s}}={\boldsymbol{{f}}}({\boldsymbol{{x}}}^{\mathrm{s}},{\boldsymbol{{u}}}^{\mathrm{s}})$.
Assume by contradiction that
$({\boldsymbol{{x}}}^{\mathrm{s}},{\boldsymbol{{u}}}^{\mathrm{s}})\notin({\boldsymbol{{x}}}^{\mathrm{s}}_{\star},{\boldsymbol{{u}}}^{\mathrm{s}}_{\star})$.
By definition of
$({\boldsymbol{{x}}}^{\mathrm{s}}_{\star},{\boldsymbol{{u}}}^{\mathrm{s}}_{\star})$,
this entails that

$\displaystyle
0=\tilde{L}^{\gamma}({\boldsymbol{{x}}}^{\mathrm{s}},{\boldsymbol{{u}}}^{\mathrm{s}})>\tilde{L}^{\gamma}({\boldsymbol{{\bar{x}}}},{\boldsymbol{{\bar{u}}}}),$
$\displaystyle\forall\
({\boldsymbol{{\bar{x}}}},{\boldsymbol{{\bar{u}}}})\in({\boldsymbol{{x}}}^{\mathrm{s}}_{\star},{\boldsymbol{{u}}}^{\mathrm{s}}_{\star}).$
(27)

The trajectory ${\boldsymbol{{x}}}_{k}={\boldsymbol{{\bar{x}}}}$,
${\boldsymbol{{u}}}_{k}={\boldsymbol{{\bar{u}}}}$, $k=0,\ldots,\infty$ is
feasible for Problem (19) and, by (27), yields the cost

$\displaystyle\sum_{k=0}^{\infty}\tilde{L}^{\gamma}({\boldsymbol{{\bar{x}}}},{\boldsymbol{{\bar{u}}}})=-\infty,$

such that
$\tilde{V}_{{\boldsymbol{{\pi}}}_{\star}^{\gamma}}^{\gamma}({\boldsymbol{{\bar{x}}}})=-\infty$.

However, because
$\left|\tilde{L}^{\gamma}({\boldsymbol{{\bar{x}}}},{\boldsymbol{{\bar{u}}}})\right|<\infty$,
we have that ${\boldsymbol{{\bar{x}}}}\in\mathbb{X}_{0}$. By Theorem 3 we have
$\tilde{V}_{{\boldsymbol{{\pi}}}_{\star}^{\gamma}}^{\gamma}({\boldsymbol{{\bar{x}}}})=V_{\star}^{\gamma}({\boldsymbol{{\bar{x}}}})$
and Assumption 3 guarantees that $V_{\star}^{\gamma}({\boldsymbol{{x}}})$ is
bounded for all ${\boldsymbol{{x}}}\in\mathbb{X}_{0}$, which yields the
desired contradiction. $\hfill\qed$ This theorem entails that in the general
case the optimal steady-state depends on the discount factor $\gamma$, i.e.,
since by (18) $\tilde{L}^{\gamma}$ does depend on $\gamma$, different $\gamma$
can yield different
$({\boldsymbol{{x}}}^{\mathrm{s}}_{\star},{\boldsymbol{{u}}}^{\mathrm{s}}_{\star})$,
as had been conjectured in [6] and discussed in [10].

######  Remark 1.

We observe that, if (9) holds for
$({\boldsymbol{{x}}}^{\mathrm{s}},{\boldsymbol{{u}}}^{\mathrm{s}})$, then
$\tilde{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{u}}})\geq\rho\left(\left\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\right\|\right)$,
such that ${\boldsymbol{{x}}}^{\mathrm{s}}_{\star}$ must be unique. However,
without stronger assumptions, e.g., using
$\rho(\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}},\
{\boldsymbol{{u}}}-{\boldsymbol{{u}}}^{\mathrm{s}}\|)$ in (9), the optimal
input ${\boldsymbol{{u}}}^{\mathrm{s}}_{\star}$ need not be unique.

Note that the result of Theorem 4 is not surprising in the light of the
equivalence with the undiscounted setting stated in Theorem 3. Since
undiscounted MPC with stage cost $\tilde{L}^{\gamma}$ is equivalent to
discounted MPC with stage cost $L$, the steady-state they converge to must
also match. Indeed, (25) yields the optimal steady state for the standard case
of undiscounted economic MPC, such that the proof of Theorem 4 can
equivalently be obtained by combining Theorem 3 with the standard results for
the undiscounted case.

After having analyzed the properties that stem from SDSD, we comment next on
the relationship between SDSD and other conditions used in the literature.

###  3.4 Relationship with Other Conditions for Stability

Since the setting of this paper has been analyzed in [6] using different
assumptions, we discuss next the main differences between our assumptions and
the ones therein. In both cases it is assumed that (9a) holds, i.e.,

$\displaystyle\hat{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{u}}})\geq\rho\left(\left\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\right\|\right).$

The main difference is that we assume (9b), while in [6] it is required that,
given two constants $0\leq\varphi<\Phi$, for all

$\displaystyle\varphi\leq\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\|\leq\Phi,$

there exists another constant $1\leq C<(1-\gamma)^{-1}$ such that

$\displaystyle\hat{V}_{\star}^{\gamma}({\boldsymbol{{x}}})\leq
C\inf_{{\boldsymbol{{u}}}}\hat{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{u}}}).$
(28)

An alternative less conservative condition proposed in [6] is

$\displaystyle\hat{V}_{\star}^{\gamma}({\boldsymbol{{x}}})\leq
C\hat{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}_{\star}^{\gamma}({\boldsymbol{{x}}})),$
(29)

with $1\leq C<(1-\gamma)^{-1}$. In order to be able to compare (9b) with (29),
we first rewrite both in equivalent expressions which allow us to better
highlight the similarities and differences. By selecting $\varphi>0$
arbitrarily small, the strict inequality holds with equality only for
${\boldsymbol{{x}}}={\boldsymbol{{x}}}_{\mathrm{s}}$. By further selecting the
most favorable constant $C=(1-\gamma)^{-1}$ one can replace (29) with

$\displaystyle(1-\gamma)\hat{V}_{\star}^{\gamma}({\boldsymbol{{x}}})\leq\hat{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}_{\star}^{\gamma}({\boldsymbol{{x}}}))-\rho^{\prime}(\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\|),$
(30)

with $\rho^{\prime}\in\mathcal{K}$. Note that we exploited the fact that
$\hat{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}({\boldsymbol{{x}}}))>0$
for all ${\boldsymbol{{x}}}\neq{\boldsymbol{{x}}}_{\mathrm{s}}$, while in [6]
the inequality was applied to the original value function $V_{\star}^{\gamma}$
instead of the modified $\hat{V}_{\star}^{\gamma}$. Finally, we observe that,
using the Bellman equation, i.e., Equation (4) written for stage cost
$\hat{L}^{\gamma}$, in (30) we have

$\displaystyle(1-\gamma)\left(\hat{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}_{\star}^{\gamma}({\boldsymbol{{x}}}))+\gamma\hat{V}_{\star}^{\gamma}({\boldsymbol{{f}}}({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}_{\star}^{\gamma}({\boldsymbol{{x}}})))\right)$
$\displaystyle\hskip
100.00015pt\leq\hat{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}_{\star}^{\gamma}({\boldsymbol{{x}}}))-\rho^{\prime}(\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\|),$

which can be rewritten as

$\displaystyle(1-\gamma)\hat{V}_{\star}^{\gamma}({\boldsymbol{{f}}}({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}_{\star}^{\gamma}({\boldsymbol{{x}}})))\leq\hat{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{\pi}}}_{\star}^{\gamma}({\boldsymbol{{x}}}))-\rho(\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\|),$
(31)

where $\gamma\rho(\cdot)=\rho^{\prime}(\cdot)$.

Condition (9b), using (10) and (13), equivalently reads

$\displaystyle(1-\gamma)\hat{V}_{\star}^{\gamma}({\boldsymbol{{f}}}({\boldsymbol{{x}}},{\boldsymbol{{u}}}))\leq\hat{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{u}}})-\rho(\|{\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}}\|).$
(32)

We can now compare our assumption with that of [6], since (31) closely
resembles (32), except for the fact that the functions in (31) are evaluated
using the optimal policy. Indeed, in Equation (17) in the proof of Theorem 2
we only need that (31) holds for the optimal policy and not for all possible
control inputs. Consequently, Assumption 2 could be relaxed to only hold for
${\boldsymbol{{u}}}={\boldsymbol{{\pi}}}_{\star}^{\gamma}({\boldsymbol{{x}}})$,
therefore obtaining a condition which closely resembles the one used in [6].

A similar situation is found in the undiscounted case, where, though the
stability proof only requires strict dissipativity to hold for the optimal
policy, it is commonly assumed that strict dissipativity holds for all
feasible inputs. However, this is not restrictive: it has been proven in [14]
that, under a mild controllability assumption, if
$({\boldsymbol{{x}}}^{\mathrm{s}},{\boldsymbol{{u}}}^{\mathrm{s}})\in\mathrm{int}\mathbb{Z}$,
strict dissipativity is also necessary for asymptotic stability. Investigating
whether this also applies to the discounted case has been partially done in
[15] and will be the subject of future research. In this paper, we limit
ourselves to the next discussion, which exploits the equivalence with the
undiscounted case.

In order to provide further insight about Assumption (9b), we state the
following lemma.

######  Lemma 1.

Suppose that Assumption 1 and

$\displaystyle\lim_{N\to\infty}V_{\star}^{\gamma}\left({\boldsymbol{{x}}}_{N}^{{\boldsymbol{{\pi}}}_{\star}^{\gamma}}\right)=0,$
$\displaystyle\left|V_{\star}^{\gamma}\left({\boldsymbol{{x}}}_{k}^{{\boldsymbol{{\pi}}}_{\star}^{\gamma}}\right)\right|<\infty,\quad\forall\
k\geq 0$ (33)

hold. Then,

$\displaystyle{\boldsymbol{{\tilde{\pi}}}}_{\star}^{\gamma}({\boldsymbol{{x}}})={\boldsymbol{{\pi}}}_{\star}^{\gamma}({\boldsymbol{{x}}}),$
$\displaystyle\tilde{V}_{\star}^{\gamma}({\boldsymbol{{x}}})=V_{\star}^{\gamma}({\boldsymbol{{x}}}).$
(34)

The proof follows the same arguments used in Theorem 3. $\hfill\qed$

We observe that, if (33) does not hold, then the closed-loop system cannot be
asymptotically stabilized to
$({\boldsymbol{{x}}}^{\mathrm{s}},{\boldsymbol{{u}}}^{\mathrm{s}})$ by policy
${\boldsymbol{{\pi}}}_{\star}^{\gamma}$, such that this assumption is not
conservative in this context. Note however, that (33) is not sufficient for
asymptotic stability, since we do not assume that $V_{\star}^{\gamma}$ neither
its evolution in time is upper or lower bounded by class $\mathcal{K}$
functions.

By Lemma 1, there exists an undiscounted formulation which yields the same
optimal policy and value function as the discounted formulation. This
equivalent formulation can be obtained by modifying stage cost
$\hat{L}^{\gamma}$ and, consequently, the corresponding value function
$\hat{V}_{\star}^{\gamma}$, to obtain the undiscounted stage cost

$\displaystyle\hat{\tilde{L}}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{u}}}):=\hat{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{u}}})+(\gamma-1)\hat{V}_{\star}^{\gamma}({\boldsymbol{{f}}}({\boldsymbol{{x}}},{\boldsymbol{{u}}})),$

yielding value function $\hat{\tilde{V}}_{\star}^{\gamma}$, obtained by
replacing ${\tilde{L}}^{\gamma}$ with $\hat{\tilde{L}}^{\gamma}$ in (19)-(20).
Note that in the literature on undiscounted economic MPC,
$\hat{\tilde{L}}^{\gamma}$ is called a rotated cost (with respect to
$\tilde{L}^{\gamma}$) and used to prove asymptotic stability [1], similarly to
what we did using $\hat{L}^{\gamma}$ in the discounted setting. We exploit
this equivalence to discuss next possible relaxations of Assumption 2 in the
undiscounted setting.

If both (33) and Assumptions 1, 2(i), 3 hold, then the equivalent undiscounted
formulation yields a value function which is both upper and lower bounded by
class $\mathcal{K}$ functions, but the stage cost $\hat{\tilde{L}}^{\gamma}$
is not guaranteed to be positive definite, such that we cannot directly prove
the decrease condition for the Lyapunov function candidate
$\hat{\tilde{V}}_{\star}^{\gamma}$. The missing condition, i.e., positive-
definiteness of the stage cost is precisely Assumption 2(ii).

On the other hand, if both (33) and Assumptions 1, 2(ii), 3 hold, then the
stage cost $\hat{\tilde{L}}^{\gamma}$ is positive definite, which, in turn,
implies the Lyapunov lower bound on the value function
$\hat{\tilde{V}}_{\star}^{\gamma}$. Since the Lyapunov upper bound is granted
by Assumption 3, in this case we do have that
$\hat{\tilde{V}}_{\star}^{\gamma}$ is a Lyapunov function. Since (33) and
Assumption 1 guarantee that the discounted and undiscounted formulations are
equivalent and the remaining assumptions are standard in the undiscounted
setting, this is possibly the least restrictive set of assumptions to prove
asymptotic stability in the discounted setting. However, as discussed above,
(33) is already a weak stability assumption in itself.

##  4 Examples

We provide next two examples which we intentionally select as simple enough in
order to be able to provide insight in the theoretical results of this paper.

###  4.1 Linear Quadratic Regulator (LQR)

Consider the simple example from [6], with linear dynamics

$\displaystyle{\boldsymbol{{x}}}_{+}$
$\displaystyle=A{\boldsymbol{{x}}}+B{\boldsymbol{{u}}},$ $\displaystyle A$
$\displaystyle=\left[\begin{array}[]{cc}2&0\\\ 1&2\end{array}\right],$
$\displaystyle B$ $\displaystyle=\left[\begin{array}[]{cc}1&0\\\
0&1\end{array}\right],$

and quadratic stage cost

$\displaystyle L({\boldsymbol{{x}}},{\boldsymbol{{u}}})$
$\displaystyle={\boldsymbol{{x}}}^{\top}Q{\boldsymbol{{x}}}+{\boldsymbol{{u}}}^{\top}R{\boldsymbol{{u}}},$
$\displaystyle Q$ $\displaystyle=\left[\begin{array}[]{cc}1&0\\\
0&1\end{array}\right],$ $\displaystyle R$
$\displaystyle=\left[\begin{array}[]{cc}1&0\\\ 0&1\end{array}\right].$

In order to abide by the proposed theory, state and control constraints ought
to be added. We observe here that for any choice of such constraints which
defines a sufficiently large region having the origin in its interior, there
will be a set of initial conditions such that the constraints are never active
over the LQR trajectories. We additionally observe that this set of initial
conditions can be made arbitrarily large by increasing the size of the
feasible state and input set defined by these constraints. It follows that the
proposed theory applies within an arbitrarily large set of initial conditions,
including the origin. Due to these observations, we do not specify state and
input constraints here. Note however that, in case such constraints are
removed, it is possible that the optimal solution is not stabilizing even
though SDSD holds.

In [6] it has been noted that a constant $C$ satisfying (28) with
$\lambda({\boldsymbol{{x}}})=0$ exists only for $\gamma$ larger than $\approx
0.846$, but
$V_{\star}^{\gamma}({\boldsymbol{{x}}})={\boldsymbol{{x}}}^{\top}P{\boldsymbol{{x}}}$
is a Lyapunov function for all $\gamma$ larger than $\approx 0.3342$.
Nevertheless, the system is still stable for $\gamma=0.334$, even though
$V_{\star}^{\gamma}$ is not a Lyapunov function.

In this example, since the dynamics are linear and the cost is quadratic, we
are able to compute a quadratic function
$\lambda({\boldsymbol{{x}}})={\boldsymbol{{x}}}^{\top}\Lambda{\boldsymbol{{x}}}$
satisfying conditions (9) by solving a Semidefinite Program (SDP), which we
solve using Yalmip [13] and Mosek [2]. For $\gamma=0.334$, this yields (up to
the fifth significant digit),

$\displaystyle\Lambda=-\left[\begin{array}[]{cc}3.9511&1.2702\\\
1.2702&2.7738\end{array}\right],$

with corresponding modified stage cost

$\displaystyle\hat{L}^{\gamma}({\boldsymbol{{x}}},{\boldsymbol{{u}}})=\left[\begin{array}[]{c}{\boldsymbol{{x}}}\\\
{\boldsymbol{{u}}}\end{array}\right]^{\top}\hat{H}^{\gamma}\left[\begin{array}[]{c}{\boldsymbol{{x}}}\\\
{\boldsymbol{{u}}}\end{array}\right],$
$\displaystyle\hat{H}^{\gamma}=\left[\begin{array}[]{cccc}4.3127&1.9906&2.8167&1.6304\\\
1.9906&1.6905&0.7643&1.7322\\\ 2.8167&0.7643&2.2173&0.3822\\\
1.6304&1.7322&0.3822&1.8661\end{array}\right]\succ 0.023I.$

Then, we obtain a Lyapunov function as

$\displaystyle\hat{V}_{\star}^{\gamma}({\boldsymbol{{x}}})=V_{\star}^{\gamma}({\boldsymbol{{x}}})+\lambda({\boldsymbol{{x}}})={\boldsymbol{{x}}}^{\top}\hat{P}^{\gamma}{\boldsymbol{{x}}},$

with

$\displaystyle\hat{P}^{\gamma}=P+\Lambda=\left[\begin{array}[]{cc}0.1537&0.0632\\\
0.0632&0.0907\end{array}\right]\succ 0.0516I,$

and $P$ the matrix defining
$V_{\star}^{\gamma}({\boldsymbol{{x}}})={\boldsymbol{{x}}}^{\top}P{\boldsymbol{{x}}}$.

Finally, the solution is stabilizing for $\gamma\geq\underline{\gamma}$, with
$\underline{\gamma}\approx 0.3109$. For ${\gamma}=0.3109$ we can still compute
a matrix $\Lambda$ which satisfies SDSD and yields

$\displaystyle\hat{H}^{\gamma}\succ 1.325\cdot 10^{-8}I,$
$\displaystyle\hat{P}^{\gamma}\succ 2.459\cdot 10^{-4}I,$

such that our conditions are tight for this example.

Note that, since $H\succ 0$, condition (9a) is satisfied for $\Lambda=0$, but
this is not enough to conclude asymptotic stability. Indeed, condition (9b) is
not satisfied for $\Lambda=0$ for $\gamma\leq\gamma_{1}$ with
$\gamma_{1}\approx 0.93507$, even though, as pointed out above,
$V_{\star}^{\gamma}$ is a Lyapunov function for $\gamma$ larger than $\approx
0.3342$.

Finally, we stress that for $\gamma<\underline{\gamma}\approx 0.3109$, if we
impose either only (9a) or only (9b), we are always able to compute a storage
function, even though the closed-loop system is not asymptotically stable.
This fact highlights the well-known insufficiency of discounted strict
dissipativity for proving stability, unless further conditions are met.

###  4.2 A Simple Nonlinear Example

Consider the scalar nonlinear system defined by

$\displaystyle{\boldsymbol{{x}}}_{+}$
$\displaystyle=0.01\,{\boldsymbol{{u}}}(1-{\boldsymbol{{x}}})+0.96\,{\boldsymbol{{x}}},$
$\displaystyle L({\boldsymbol{{x}}},{\boldsymbol{{u}}})$
$\displaystyle=-1.5\,{\boldsymbol{{u}}}+2\,{\boldsymbol{{u}}}{\boldsymbol{{x}}}+0.1\,({\boldsymbol{{u}}}-4)^{2},$

with ${\boldsymbol{{x}}}\in[0,1]\subset\mathbb{R}$,
${\boldsymbol{{u}}}\in[0,20]\subset\mathbb{R}$.

Since the problem is of sufficiently small dimension, we can solve it by
dynamic programming. We verified numerically that the storage function

$\displaystyle\lambda({\boldsymbol{{x}}})=-\nabla_{\boldsymbol{{x}}}V_{\star}^{\gamma}({\boldsymbol{{x}}}^{\mathrm{s}})\,({\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}})+50\,({\boldsymbol{{x}}}-{\boldsymbol{{x}}}^{\mathrm{s}})^{2},$

satisfies the SDSD assumption for all discount factors $\gamma\in\ ]0,1]$.
Indeed, in this case the policy ${\boldsymbol{{\pi}}}_{\star}^{\gamma}$ is
stabilizing for all discount factors $\gamma$ in that interval.

We display in Figure 1 the optimal steady state as a function of the discount
factor $\gamma$, where we evaluated numerically that the optimal policy indeed
stabilizes the system to the solution of the steady-state problem (25), which
we solved numerically.

Figure 1: Optimal steady state as a function of $\gamma$.

##  5 Conclusions

In this paper we have defined a new concept of dissipativity for discounted
MPC, which allowed us to prove asymptotic stability, to characterize the
optimal steady-state and to establish an equivalence between discounted and
undiscounted problems. We have compared our assumptions to similar ones used
in the literature and we have provided two examples to illustrate our
findings.

Future work will consider further investigating the necessity of our
assumption. Furthermore, the finite-horizon case presents some challenges
which impede a direct application of the available stability theory.
Similarly, the stochastic case, which is particularly relevant in the context
of reinforcement learning, is far from being fully understood.

## References and Notes

  * [1] R. Amrit, J. Rawlings, and D. Angeli.  Economic optimization using model predictive control with a terminal cost.  Annual Reviews in Control, 35:178–186, 2011.
  * [2] MOSEK ApS.  MOSEK webpage.  http://www.mosek.com, 2011.
  * [3] M. Diehl, R. Amrit, and J.B. Rawlings.  A Lyapunov Function for Economic Optimizing Model Predictive Control.  IEEE Trans. of Automatic Control, 56(3):703–707, March 2011.
  * [4] T. Faulwasser, L. Grüne, and M. Müller.  Economic nonlinear model predictive control: Stability, optimality and performance.  Foundations and Trends in Systems and Control, 5(1):1–98, 2018\.
  * [5] T. Faulwasser and M. Zanon.  Asymptotic Stability of Economic NMPC: The Importance of Adjoints.  In Proceedings of the IFAC Nonlinear Model Predictive Control Conference, 2018.
  * [6] V. Gaitsgory, L. Grüne, M. Höger, C. M. Kellett, and S. R. Weller.  Stabilization of strictly dissipative discrete time systems with discounted optimal control.  Automatica, 93:311 – 320, 2018.
  * [7] L. Grüne.  Economic receding horizon control without terminal constraints.  Automatica, 49:725–734, 2013.
  * [8] L. Grüne, C. M. Kellett, and S. R. Weller.  On a discounted notion of strict dissipativity.  IFAC-PapersOnLine, 49(18):247 – 252, 2016.  10th IFAC Symposium on Nonlinear Control Systems NOLCOS 2016.
  * [9] L. Grüne and l. Krügel.  Local turnpike analysis using local dissipativity for discrete time discounted optimal control.  arXiv, 2020.  https://arxiv.org/abs/2101.11295.
  * [10] L. Grüne, M. A. Müller, C. M. Kellett, and S. R. Weller.  Strict dissipativity for discrete time discounted optimal control problems.  Mathematical Control & Related Fields, 2020.
  * [11] L. Grüne and J. Pannek.  Nonlinear Model Predictive Control.  Springer, London, 2011.
  * [12] L. Grüne and M. Stieler.  A Lyapunov function for economic MPC without terminal conditions.  In Proc. of the 53rd IEEE Conference on Decision and Control, pages 2740–2745, December 2014.
  * [13] J. Löfberg.  YALMIP: A toolbox for modeling and optimization in MATLAB.  In Proceedings of the CACSD Conference, Taipei, Taiwan, 2004.
  * [14] M. A. Müller, D. Angeli, and F. Allgöwer.  On necessity and robustness of dissipativity in economic model predictive control.  IEEE Transactions on Automatic Control, 60(6):1671–1676, June 2015\.
  * [15] Matthias Müller and Lars Grüne.  On the relation between dissipativity and discounted dissipativity.  In 2017 IEEE 56th Annual Conference on Decision and Control (CDC), pages 5570–5575, 2017.
  * [16] Matthias A. Müller and Lars Grüne.  Economic model predictive control without terminal constraints for optimal periodic behavior.  Automatica, 70:128 – 139, 2016.
  * [17] R. Postoyan, L. Buşoniu, D. Nešić’, and J. Daafouz.  Stability of infinite-horizon optimal control with discounted cost.  In 53rd IEEE Conference on Decision and Control, pages 3903–3908, 2014.
  * [18] M. L. Puterman.  Markov Decision Processes: Discrete Stochastic Dynamic Programming.  John Wiley & Sons, Inc., USA, 1st edition, 1994.
  * [19] J. B. Rawlings, D. Q. Mayne, and M. Diehl.  Model Predictive Control: Theory, Computation, and Design.  Nob Hill Publishing, 2 edition, 2017.
  * [20] R. S. Sutton and A. G. Barto.  Reinforcement Learning: An Introduction.  The MIT Press, Cambridge, MA, USA, 2nd edition, 2018.
  * [21] M. Zanon, S. Gros, and M. Diehl.  A Lyapunov Function for Periodic Economic Optimizing Model Predictive Control.  In Proceedings of the 52nd Conference on Decision and Control (CDC), pages 5107–5112, 2013.
  * [22] M. Zanon, S. Gros, and M. Diehl.  Indefinite Linear MPC and Approximated Economic MPC for Nonlinear Systems.  Journal of Process Control, 24:1273–1281, 2014.
  * [23] M. Zanon, S. Gros, and M. Diehl.  A Tracking MPC Formulation that is Locally Equivalent to Economic MPC.  Journal of Process Control, 45:30 – 42, 2016.
  * [24] M. Zanon, S. Gros, and M. Diehl.  A Periodic Tracking MPC that is Locally Equivalent to Periodic Economic MPC.  In Proceedings of the 2017 IFAC World Congress, 2017.
  * [25] M. Zanon, L. Grüne, and M. Diehl.  Periodic optimal control, dissipativity and MPC.  IEEE Transactions on Automatic Control, 62(6):2943–2949, 2017.
  * [26] Mario Zanon and Timm Faulwasser.  Economic MPC without terminal constraints: Gradient-correcting end penalties enforce asymptotic stability.  Journal of Process Control, 63:1 – 14, 2018.