# Absorbing Markov Decision Processes

François Dufour111 Institut Polytechnique de Bordeaux; INRIA Bordeaux Sud
Ouest, Team: ASTRAL; IMB, Institut de Mathématiques de Bordeaux, Université de
Bordeaux, France<EMAIL_ADDRESS>Tomás Prieto-
Rumeau222Statistics Department, UNED, Madrid, Spain. e-mail:
<EMAIL_ADDRESS>(Author for correspondence)

###### Abstract

In this paper, we study discrete-time absorbing Markov Decision Processes
(MDP) with measurable state space and Borel action space with a given initial
distribution. For such models, solutions to the characteristic equation that
are not occupation measures may exist. Several necessary and sufficient
conditions are provided to guarantee that any solution to the characteristic
equation is an occupation measure. Under the so-called continuity-compactness
conditions, it is shown that the set of occupation measures is compact in the
weak-strong topology if and only if the model is uniformly absorbing. Finally,
it is shown that the occupation measures are characterized by the
characteristic equation and an additional condition. Several examples are
provided to illustrate our results.

Keywords: Markov decision processes; absorbing model; occupation measures,
characteristic equation, phantom mesures, compactness of the set of occupation
measures

AMS 2020 Subject Classification: 91A10, 91A15.

##  1 Introduction

In this work, we consider a discrete-time absorbing Markov Decision Process
(MDP) with measurable state space $\mathbf{X}$ and Borel action space
$\mathbf{A}$ with a given initial distribution denoted by $\eta$. We consider
general measurable state spaces to cover models that are studied in game
theory, see for example [15] and [7] for an analysis of absorbing non-zero sum
Markov games. An absorbing MDP is a special type of MDP where some measurable
subset $\Delta$ of the state space is considered as absorbing and lead to
termination of the process. This means that once the system enters in
$\Delta$, it remains there indefinitely without any further transitions with a
null reward or cost. The underlying assumption is that for any policy, the
average time to enter $\Delta$ is finite. For more details on this type of
MDP, we refer to the following papers and books [2, 9, 10, 18] and their
references therein.

The objective of this paper is to study the properties of the solution set of
the characteristic equation and of the set of occupation measures. The notion
of occupation measure is particularly important and plays a central role in
solving constraint Markov decision processes, see for example the references
[2, 6, 13, 14, 17]. It describes the expected amount of time spends by the
state and action processes in any measurable subset of
$\mathbf{X}\times\mathbf{A}$ and under any policy. Any occupation measure
satisfies the so-called characteristic equation which is of the form

$\displaystyle\mu^{\mathbf{X}}(\cdot)=\eta(\cdot)+\int_{X\times
A}R(\cdot|x,a)\mu(dx,da)$ (1)

(see equation (3) for a more precise statement) where $\mu^{\mathbf{X}}$ is
the marginal of the measure $\mu$ on $\mathbf{X}$, $\eta$ is the initial
distribution of the MDP and $R$ is a kernel depending on the transition kernel
of the MDP and on the absorbing set $\Delta$.

We will show that although the model is absorbing, there may exist solutions
to the characteristic equation that are not occupation measures (see example
3.4). These are called phantom solution to the characteristic equation (or
simply phantom measures). In order to avoid this pathological phenomenon, we
will provide in Theorem 3.6 several necessary and sufficient conditions to
ensure that there are no phantom measures, thereby guaranteeing that any
solution to the characteristic equation is an occupation measure. By
strengthening our hypotheses, that is, assuming the so-called continuity-
compactness conditions introduced by Schäl in [19] (see Condition (S) below),
we will show in our second main result (see Theorem 4.9) that the set of
occupation measures is compact in the weak-strong topology (see [4]) if and
only if the model is uniformly absorbing. Finally, it will be shown in our
last main result (see Theorem 4.5) that under Condition (S) the occupation
measures are characterized by the characteristic equation and an additional
condition. This is the first notable difference with the discounted model,
where the occupation measures are essentially characterized by a single
equation of type (1).

#### Notation and terminology.

On a measurable space $(\mathbf{\Omega},\mathcal{F})$ we will consider the set
of finite signed measures $\boldsymbol{\mathcal{M}}(\mathbf{\Omega})$, the set
of finite nonnegative measures
$\boldsymbol{\mathcal{M}}^{+}(\mathbf{\Omega})$, and the set of probability
measures $\boldsymbol{\mathcal{P}}(\mathbf{\Omega})$. For a set
$\Gamma\in\mathcal{F}$, we denote by
$\mathbf{I}_{\Gamma}:\Omega\rightarrow\\{0,1\\}$ the indicator function of the
set $\Gamma$, that is, $\mathbf{I}_{\Gamma}(\omega)=1$ if and only if
$\omega\in\Gamma$. For $\omega\in\mathbf{\Omega}$, we write
$\delta_{\\{\omega\\}}$ for the Dirac probability measure at $\omega$ defined
on $(\mathbf{\Omega},\mathcal{F})$ by
$\delta_{\\{\omega\\}}(B)=\mathbf{I}_{B}(\omega)$ for any $B\in\mathcal{F}$.
If $\mu\in\boldsymbol{\mathcal{M}}(\mathbf{\Omega})$ and
$\Gamma\in\mathcal{F}$, we denote by $\mu_{\Gamma}$ the measure on
$(\mathbf{\Omega},\mathcal{F})$ defined by $\mu_{\Gamma}(B)=\mu(\Gamma\cap B)$
for $B\in\mathcal{F}$. The trace $\sigma$-algebra of a set
$\Gamma\subseteq\mathbf{\Omega}$ is denoted by $\mathcal{F}_{\Gamma}$. On
$\boldsymbol{\mathcal{P}}(\mathbf{\Omega})$, the $s$-topology is the coarsest
topology that makes $\mu\mapsto\mu(D)$ continuous for every $D\in\mathcal{F}$.

Given a measurable space $(\mathbf{\Omega},\mathcal{F})$ and
$\lambda\in\boldsymbol{\mathcal{P}}(\mathbf{\Omega})$, we will denote by
$L^{1}(\mathbf{\Omega},\mathcal{F},\lambda)$ the family of measurable
functions (identifying those which are $\lambda$-a.s. equal)
$f:\mathbf{\Omega}\rightarrow\mathbb{R}$ which are $\lambda$-integrable, i.e.,
$\|f\|_{1}=\int_{\mathbf{X}}|f(x)|\lambda(dx)<\infty$.

Let $(\mathbf{\Omega},\mathcal{F})$ and
$(\widetilde{\mathbf{\Omega}},\widetilde{\mathcal{F}})$ be two measurable
spaces. A kernel on $\widetilde{\mathbf{\Omega}}$ given $\mathbf{\Omega}$ is a
mapping
$Q:\mathbf{\Omega}\times\widetilde{\mathcal{F}}\rightarrow\mathbb{R}^{+}$ such
that $\omega\mapsto Q(B|\omega)$ is measurable on
$(\mathbf{\Omega},\mathcal{F})$ for every $B\in\widetilde{\mathcal{F}}$, and
$B\mapsto Q(B|\omega)$ is in
$\boldsymbol{\mathcal{M}}^{+}(\widetilde{\mathbf{\Omega}})$ for every
$\omega\in\mathbf{\Omega}$. If $Q(\widetilde{\mathbf{\Omega}}|\omega)=1$ for
all $\omega\in\mathbf{\Omega}$ then we say that $Q$ is a stochastic kernel. We
write $\mathbb{I}_{\Gamma}$ for the kernel on $\mathbf{\Omega}$ given
$\mathbf{\Omega}$ defined by
$\mathbb{I}_{\Gamma}(B|\omega)=\mathbf{I}_{\Gamma}(\omega)\delta_{\\{\omega\\}}(B)$
for $\omega\in\mathbf{\Omega}$ and $B\in\mathcal{F}$. Let $Q$ be a stochastic
kernel on $\widetilde{\mathbf{\Omega}}$ given $\mathbf{\Omega}$. For a bounded
measurable function $f:\widetilde{\mathbf{\Omega}}\rightarrow\mathbb{R}$, we
will denote by $Qf:\mathbf{\Omega}\rightarrow\mathbb{R}$ the measurable
function

$Qf(\omega)=\int_{\mathbf{\Omega^{\prime}}}f(\widetilde{\omega})Q(d\widetilde{\omega}|\omega)\quad\hbox{for
$\omega\in\mathbf{\Omega}$}.$

For a measure $\mu\in\boldsymbol{\mathcal{M}}^{+}(\mathbf{\Omega})$, we denote
by $\mu Q$ the finite measure on
$(\widetilde{\mathbf{\Omega}},\widetilde{\mathcal{F}})$ given by

$B\mapsto\mu
Q\,(B)=\int_{\mathbf{\Omega}}Q(B|\omega)\mu(d\omega)\quad\hbox{for
$B\in\widetilde{\mathcal{F}}$}.$

The product of the $\sigma$-algebras $\mathcal{F}$ and
$\widetilde{\mathcal{F}}$ is denoted by
$\mathcal{F}\otimes\widetilde{\mathcal{F}}$ and consists of the
$\sigma$-algebra generated by the measurable rectangles, that is, the sets of
the form $\Gamma\times\widetilde{\Gamma}$ for $\Gamma\in\mathcal{F}$ and
$\widetilde{\Gamma}\in\widetilde{\mathcal{F}}$. We denote by $\mu\otimes Q$
the unique probability measure (or finite measure) on the product space
$(\mathbf{\Omega}\times\widetilde{\mathbf{\Omega}},\mathcal{F}\otimes\widetilde{\mathcal{F}})$
satisfying

$(\mu\otimes
Q)(\Gamma\times\widetilde{\Gamma})=\int_{\Gamma}Q(\widetilde{\Gamma}|\omega)\mu(d\omega)\quad\hbox{for
$\Gamma\in\mathcal{F}$ and $\widetilde{\Gamma}\in\widetilde{\mathcal{F}}$,}$

see Proposition III-2-1 in [16] for a proof of existence and uniqueness of
such measure. Let $(\overline{\mathbf{\Omega}},\overline{\mathcal{F}})$ be a
third measurable space and $R$ a stochastic kernel on
$\overline{\mathbf{\Omega}}$ given $\widetilde{\mathbf{\Omega}}$. Then we will
denote by $QR$ the stochastic kernel on $\overline{\mathbf{\Omega}}$ given
$\mathbf{\Omega}$ given by

$QR(\Gamma|\omega)=\int_{\widetilde{\mathbf{\Omega}}}R(\Gamma|\tilde{\omega})Q(d\tilde{\omega}|\omega)\quad\hbox{for
$\Gamma\in\overline{\mathcal{F}}$ and $\omega\in\mathcal{F}$}.$

Given
$\mu\in\boldsymbol{\mathcal{M}}(\mathbf{\Omega}\times\widetilde{\mathbf{\Omega}})$,
the marginal measures are
$\mu^{\mathbf{\Omega}}\in\boldsymbol{\mathcal{M}}(\mathbf{\Omega})$ and
$\mu^{\widetilde{\mathbf{\Omega}}}\in\boldsymbol{\mathcal{M}}(\widetilde{\mathbf{\Omega}})$
defined by
$\mu^{\mathbf{\Omega}}(\cdot)=\mu(\cdot\times\widetilde{\mathbf{\Omega}})$ and
$\mu^{\widetilde{\mathbf{\Omega}}}(\cdot)=\mu(\mathbf{\Omega}\times\cdot)$. If
$\pi$ is a kernel on
$\widetilde{\mathbf{\Omega}},\times\overline{\mathbf{\Omega}}$ given
$\mathbf{\Omega}$ the marginal kernels are $\pi^{\widetilde{\mathbf{\Omega}}}$
and $\pi^{\overline{\mathbf{\Omega}}}$, respectively defined by
$\pi^{\widetilde{\mathbf{\Omega}}}=\pi(\cdot\times\overline{\mathbf{\Omega}}|\omega)$
and
$\pi^{\overline{\mathbf{\Omega}}}=\pi(\widetilde{\mathbf{\Omega}}\times\cdot|\omega)$
for $\omega\in\mathbf{\Omega}$.

We say that $f:\mathbf{\Omega}\times\mathbf{S}\rightarrow\mathbf{S}^{\prime}$,
where $\mathbf{S}^{\prime}$ is a metric space, is a Carathéodory function if
$f(\cdot,s)$ is measurable on $\mathbf{\Omega}$ for every $s\in\mathbf{S}$ and
$f(\omega,\cdot)$ is continuous on $\mathbf{S}$ for every
$\omega\in\mathbf{\Omega}$. The family of the so-defined Carathéodory
functions is denoted by
${\mathcal{C}ar}(\mathbf{\Omega}\times\mathbf{S},\mathbf{S}^{\prime})$. The
family of Carathéodory functions which, in addition, are bounded is denoted by
${\mathcal{C}ar}_{b}(\mathbf{\Omega}\times\mathbf{S},\mathbf{S}^{\prime})$.
When the metric space $\mathbf{S}$ is separable then any
$f\in{\mathcal{C}ar}(\mathbf{\Omega}\times\mathbf{S},\mathbf{S}^{\prime})$ is
a jointly measurable function on
$(\mathbf{\Omega}\times\mathbf{S},\mathcal{F}\otimes\boldsymbol{\mathfrak{B}}(\mathbf{S}))$;
see [1, Lemma 4.51].

If $\mathbf{S}$ is a Polish space (a complete and separable metric space), on
$\boldsymbol{\mathcal{M}}(\mathbf{\Omega}\times\mathbf{S})$ we will consider
the $ws$-topology (weak-strong topology) which is the coarsest topology for
which the mappings

$\mu\mapsto\int_{\mathbf{\Omega}\times\mathbf{S}}f(\omega,s)\mu(d\omega,ds)$

for $f\in{\mathcal{C}ar}_{b}(\mathbf{\Omega}\times\mathbf{S},\mathbb{R})$ are
continuous. There are other equivalent definitions of this topology as
discussed, for instance, in [11, Section 3.3].

The next disintegration lemma will be useful in the forthcoming (see Theorem 1
in [20]).

######  Lemma 1.1 (Disintegration lemma)

Let $(\mathbf{\Omega},\mathcal{F})$ be a measurable space and let $\mathbf{S}$
be a Polish space. Let $\varphi:\mathbf{\Omega}\twoheadrightarrow\mathbf{S}$
be a weakly measurable correspondence with nonempty closed values, and let
$\mathbf{K}$ be the graph of the correspondence. For every
$\mu\in\boldsymbol{\mathcal{M}}^{+}(\mathbf{\Omega}\times\mathbf{S})$ such
that $\mu(\mathbf{K}^{c})=0$ there exists a stochastic kernel $Q$ on
$\mathbf{S}$ given $\mathbf{\Omega}$ such that

$\mu=\mu^{\mathbf{\Omega}}\otimes Q$ (2)

and such that $Q(\varphi(\omega)|\omega)=1$ for each
$\omega\in\mathbf{\Omega}$. Moreover, $Q$ is unique
$\mu^{\mathbf{\Omega}}$-almost surely, meaning that if $Q$ and $Q^{\prime}$
are two stochastic kernels that satisfy (2) then for all $\omega$ in a set of
$\mu^{\mathbf{\Omega}}$-probability one, the probability measures
$Q(\cdot|\omega)$ and $Q^{\prime}(\cdot|\omega)$ coincide.

##  2 The absorbing control model

The main goal of this section is to introduce the parameters defining the
model with a brief presentation of the construction of the controlled process.
We also describe the notions of (uniformly) absorbing PDM and give the
definition of an occupation measure, providing a first elementary property
(see lemma 2.4).

###  2.1 The control model.

We consider a stationary Markov controlled process
$(\mathbf{X},\mathbf{A},\\{\mathbf{A}(x):x\in\mathbf{X}\\},Q,\eta)$ consisting
of:

  * •

A measurable state space $\mathbf{X}$ endowed with a $\sigma$-algebra
$\boldsymbol{\mathfrak{X}}$.

  * •

A Borel space $\mathbf{A}$, representing the action space.

  * •

A family of nonempty measurable sets $\mathbf{A}(x)\subseteq\mathbf{A}$ for
$x\in\mathbf{X}$. The set $\mathbf{A}(x)$ gives the available actions in state
$x$. Let
$\mathbf{K}=\\{(x,a)\in\mathbf{X}\times\mathbf{A}:a\in\mathbf{A}(x)\\}$ be the
family of feasible state-action pairs. We assume that $\mathbf{K}$ is a
measurable subset of $\mathbf{X}\times\mathbf{A}$ endowed with the
$\sigma$-algebra
$\boldsymbol{\mathfrak{X}}\otimes\boldsymbol{\mathfrak{B}}(\mathbf{A})$.

  * •

A stochastic kernel $Q$ on $\mathbf{X}$ given $\mathbf{X}\times\mathbf{A}$,
which stands for the transition probability function.

  * •

An initial distribution given by
$\eta\in\boldsymbol{\mathcal{P}}(\mathbf{X})$.

Additionally, we assume that a measurable set
$\Delta\in\boldsymbol{\mathfrak{X}}$ is given. As we shall see later in
Defintion 2.2, $\Delta$ will play the role of the _absorbing set_. The so-
defined control model is denoted by $\boldsymbol{\mathfrak{M}}(\eta,\Delta)$
where we make explicit the dependence on the initial distribution and the
absorbing set.

The space of admissible histories of the controlled process up to time
$n\in\mathbb{N}$ is denoted by $\mathbf{H}_{n}$. It is defined recursively by

$\mathbf{H}_{0}=\mathbf{X}\quad\hbox{and}\quad\mathbf{H}_{n}=\mathbf{H}_{n-1}\times\mathbf{A}\times\mathbf{X}\quad\hbox{for
$n\geq 1$},$

all endowed with their corresponding product $\sigma$-algebras. A control
policy $\pi$ is a sequence $\\{\pi_{n}\\}_{n\geq 0}$ of stochastic kernels on
$\mathbf{A}$ given $\mathbf{H}_{n}$, denoted by $\pi_{n}(da|h_{n})$, such that

$\pi_{n}(\mathbf{A}(x_{n})|h_{n})=1\quad\hbox{for each $n\geq 0$ and
$h_{n}=(x_{0},a_{0},\ldots,x_{n})\in\mathbf{H}_{n}$}.$

The set of all policies is denoted by $\mathbf{\Pi}$.

Let us denote by $\mathbf{M}$ the set of stochastic kernels $\varphi$ on
$\mathbf{A}$ given $\mathbf{X}$ satisfying $\varphi(\mathbf{A}(x)|x)=1$ for
any $x\in\mathbf{X}$. A policy
$\pi=\\{\pi_{n}\\}_{n\in\mathbb{N}}\in\mathbf{\Pi}$ is called a _stationary
randomized policy_ if there exists $\varphi\in\mathbf{M}$ satisfying

$\pi_{n}(\cdot|h_{n})=\varphi(\cdot|x_{n})\quad\hbox{ for any
$h_{n}=(x_{0},a_{0},\ldots,x_{n})\in\mathbf{H}_{n}$ and $n\in\mathbb{N}$}.$

In such a case, we will write $\varphi$ instead of $\pi$ to emphasize that the
corresponding stationary randomized policy $\pi$ is generated by $\varphi$. We
denote by $\mathbf{\Pi}^{s}$ the set of all stationary randomized policies.

A policy $\pi=\\{\pi_{n}\\}_{n\in\mathbb{N}}\in\mathbf{\Pi}$ is called a
_Markovian randomized policy_ if there exists a sequence
$\\{\varphi_{n}\\}_{n\in\mathbb{N}}$ in $\mathbf{M}$ satisfying

$\pi_{n}(\cdot|h_{n})=\varphi_{n}(\cdot|x_{n})\quad\hbox{ for any
$h_{n}=(x_{0},a_{0},\ldots,x_{n})\in\mathbf{H}_{n}$ and $n\in\mathbb{N}$}.$

Let $\mathbf{\Pi}^{m}$ be the set of all Markovian randomized policies. We
have $\mathbf{\Pi}^{s}\subseteq\mathbf{\Pi}^{m}\subseteq\mathbf{\Pi}$.

The canonical space of all possible sample paths of the state-action process
is $\mathbf{\Omega}=(\mathbf{X}\times\mathbf{A})^{\infty}$ endowed with the
product $\sigma$-algebra $\boldsymbol{\mathcal{F}}$. The coordinate projection
functions from $\mathbf{\Omega}$ to the state space $\mathbf{X}$, the action
space $\mathbf{A}$, and $\mathbf{H}_{n}$ for $n\geq 0$ are respectively
denoted by $X_{n}$, $A_{n}$, and $H_{n}$. We will refer to
$\\{X_{n}\\}_{n\in\mathbb{N}}$ as to state process and
$\\{A_{n}\\}_{n\in\mathbb{N}}$ as the action process. It is a well known
result that for every policy $\pi\in\mathbf{\Pi}$ and any initial probability
measure $\lambda$ on $(\mathbf{X},\boldsymbol{\mathfrak{X}})$ there exists a
unique probability measure $\mathbb{P}_{\lambda,\pi}$ on
$(\mathbf{\Omega},\boldsymbol{\mathcal{F}})$ such that
$\mathbb{P}_{\lambda,\pi}(\mathbf{K}^{\infty})=1$ and such that for every
$n\in\mathbb{N}$, $\Gamma\in\boldsymbol{\mathfrak{X}}$, and
$\Lambda\in\boldsymbol{\mathfrak{B}}(\mathbf{A})$ we have
$\mathbb{P}_{\lambda,\pi}(X_{0}\in\Gamma)=\lambda(\Gamma)$,

$\mathbb{P}_{\lambda,\pi}(X_{n+1}\in\Gamma\mid H_{n},A_{n})=Q(\Gamma\mid
X_{n},A_{n})\quad\hbox{and}\quad\mathbb{P}_{\lambda,\pi}(A_{n}\in\Lambda\mid
H_{n})=\pi_{n}(\Lambda\mid H_{n})$

with $\mathbb{P}_{\lambda,\pi}$-probability one. The expectation with respect
to $\mathbb{P}_{\lambda,\pi}$ is denoted by $\mathbb{E}_{\lambda,\pi}$.

###### Definition 2.1

The hitting time $T_{\Delta}$ of the set $\Delta$ is given by
$T_{\Delta}:\mathbf{\Omega}\rightarrow\mathbb{N}\cup\\{\infty\\}$ defined as

$T_{\Delta}(x_{0},a_{0},x_{1},a_{1},\ldots)=\min\\{n\geq 0:x_{n}\in\Delta\\},$

where the $\min$ over the empty set is defined as $+\infty$.

Next we propose the definition of an absorbing control model.

###### Definition 2.2

Given an initial distribution $\lambda\in\boldsymbol{\mathcal{P}}(\mathbf{X})$
and $\Delta\in\boldsymbol{\mathfrak{X}}$, we say that the control model
$\boldsymbol{\mathfrak{M}}(\lambda,\Delta)$ is absorbing if the conditions (a)
and (b) below are satisfied, and we say that it is uniformly absorbing if,
additionally, condition (c) holds.

  1. (a).

For every $(x,a)\in\Delta\times\mathbf{A}$ we have $Q(\Delta|x,a)=1$.

  2. (b).

For any $\pi\in\mathbf{\Pi}$ the expected hitting time
$\mathbb{E}_{\lambda,\pi}[T_{\Delta}]$ is finite.

  3. (c).

The following limit holds:

$\lim_{n\rightarrow\infty}\sup_{\pi\in\mathbf{\Pi}^{s}}\sum_{t=n}^{\infty}\mathbb{P}_{\lambda,\pi}\\{T_{\Delta}>t\\}=0.$

We define now the occupation measures of an absorbing control model
$\boldsymbol{\mathfrak{M}}(\lambda,\Delta)$.

###### Definition 2.3

The occupation measure
$\mu_{\lambda,\pi}\in\boldsymbol{\mathcal{M}}_{+}(\mathbf{X}\times\mathbf{A})$
of the control policy $\pi\in\mathbf{\Pi}$ is

$\displaystyle\mu_{\lambda,\pi}(\Gamma)$ $\displaystyle=$
$\displaystyle\mathbb{E}_{\lambda,\pi}\Big{[}\sum_{t=0}^{\infty}\mathbf{I}_{\\{T_{\Delta}>t\\}}\cdot\mathbf{I}_{\\{(X_{t},A_{t})\in\Gamma\\}}\Big{]}\quad\hbox{for
$\Gamma\in\boldsymbol{\mathfrak{X}}\otimes\mathfrak{B}(\mathbf{A})$.}$

We note that the occupation measure $\mu_{\lambda,\pi}$ takes into account the
state-action process up to time $T_{\Delta}$ and it does not count the time
spent in $\Delta$: indeed, $\mu_{\lambda,\pi}(\Delta\times\mathbf{A})=0$. We
will consider the following sets of occupation measures
$\boldsymbol{\mathcal{O}}_{\lambda}=\\{\mu_{\lambda,\pi}:\pi\in\mathbf{\Pi}\\}$,
$\boldsymbol{\mathcal{O}}^{m}_{\lambda}=\\{\mu_{\lambda,\pi}:\pi\in\mathbf{\Pi}^{m}\\}$
and also
$\boldsymbol{\mathcal{O}}^{s}_{\lambda}=\\{\mu_{\lambda,\pi}:\pi\in\mathbf{\Pi}^{s}\\}$.

###### Lemma 2.4

For an absorbing control model $\boldsymbol{\mathfrak{M}}(\lambda,\Delta)$,
the set $\boldsymbol{\mathcal{O}}_{\lambda}$ is bounded.

Proof. Note that the total mass of the occupation measure $\mu_{\lambda,\pi}$
is

$\mu_{\lambda,\pi}(\mathbf{X}\times\mathbf{A})=\mathbb{E}_{\lambda,\pi}\Big{[}\sum_{t=0}^{\infty}\mathbf{I}_{\\{T_{\Delta}>t\\}}\Big{]}=\sum_{t=0}^{\infty}\mathbb{P}_{\lambda,\pi}\\{T_{\Delta}>t\\}=\mathbb{E}_{\lambda,\pi}[T_{\Delta}],$

which is finite as a consequence of item (b) in Definition 2.2. We also have
$\sup_{\pi\in\mathbf{\Pi}}\mathbb{E}_{\lambda,\pi}[T_{\Delta}]<\infty$
according to [8, Sections 4.4 and 5.5] for the special case of a Borel state
space or Proposition 2.4(i) in [7] for the general case of a measurable state
space. This shows that the set $\boldsymbol{\mathcal{O}}_{\lambda}$ is
bounded. $\Box$

##  3 The characteristic equations

In this section, we define the notion of phantom measure. We show that for
absorbing MDP, phantom measures may exist. In order to avoid this pathological
phenomenon, we establish our first main result (see Theorem 3.6) that provides
several necessary and sufficient conditions to ensure that there are no
phantom measures.

###### Definition 3.1

Given the absorbing control model $\boldsymbol{\mathfrak{M}}(\eta,\Delta)$, we
say $\mu\in\boldsymbol{\mathcal{M}}_{+}(\mathbf{X}\times\mathbf{A})$ is a
solution of the characteristic equations when

$\mu(\mathbf{K}^{c})=0\quad\hbox{and}\quad\mu^{\mathbf{X}}=(\eta+\mu
Q)\mathbb{I}_{\Delta^{c}}.$ (3)

We denote by $\boldsymbol{\mathcal{C}}_{\eta}$ the family of all solutions of
the characteristic equations.

###### Lemma 3.2

Suppose that the control model $\boldsymbol{\mathfrak{M}}(\eta,\Delta)$ is
absorbing. Given any $\pi\in\mathbf{\Pi}$ and $\sigma\in\mathbf{M}$, we have

$\mathbb{P}_{x,\sigma}\\{T_{\Delta}=\infty\\}=0\text{ for
}\mu_{\eta,\pi}^{\mathbf{X}}\text{-almost all }x\in\mathbf{X}.$

Proof. For notational convenience, let us introduce the functions $h_{t}$ and
$h$ on $\mathbf{X}$ taking values in $[0,1]$ as
$h_{t}(x)=\mathbb{P}_{x,\sigma}\\{T_{\Delta}>t\\}$ and
$h(x)=\mathbb{P}_{x,\sigma}\\{T_{\Delta}=\infty\\}$ respectively. Note that
$h_{t}$ vanishes on $\Delta$ and also that $h_{t}$ is a decreasing sequence of
measurable functions which converges pointwise to
$h(x)=\mathbb{P}_{x,\sigma}\\{T_{\Delta}=\infty\\}$. To prove the result, we
proceed by contradiction and suppose that $\int
hd\mu^{\mathbf{X}}_{\eta,\pi}>0$. By definition of the occupation measure
$\mu_{\eta,\pi}$ and since $h$ vanishes on $\Delta$, this implies that there
exist some $s\geq 0$ with $\mathbb{E}_{\eta,\pi}[h(X_{s})]>0$ and, in
particular, since $h_{t}\downarrow h$,

$\sum_{t=0}^{\infty}\mathbb{E}_{\eta,\pi}[h_{t}(X_{s})]=\infty.$ (4)

Define the strategy $\gamma\in\mathbf{\Pi}$ as follows: $\gamma_{k}=\pi_{k}$
for $0\leq k\leq s-1$ and $\gamma_{k}=\sigma$ for every $k\geq s$ (in case
that $s=0$, this definition reduces to $\gamma=\sigma$). Consequently, the
distribution of $X_{s}$ is the same under $\mathbb{P}_{\eta,\pi}$ and
$\mathbb{P}_{\eta,\gamma}$, and we have

$\mathbb{E}_{\eta,\pi}[h_{t}(X_{s})]=\mathbb{E}_{\eta,\gamma}[h_{t}(X_{s})]=\mathbb{P}_{\eta,\gamma}\\{T_{\Delta}>t+s\\}.$

Recalling (4), this implies that the series

$\sum_{t=0}^{\infty}\mathbb{P}_{\eta,\gamma}\\{T_{\Delta}>t+s\\}=\infty,$

in contradiction with $\mathbb{E}_{\eta,\gamma}[T_{\Delta}]$ being finite.
This gives the result. $\Box$

###### Proposition 3.3

Suppose that the control model $\boldsymbol{\mathfrak{M}}(\eta,\Delta)$ is
absorbing.

  * (i).

Given any $\pi\in\mathbf{\Pi}$, its occupation measure $\mu_{\eta,\pi}$
satisfies the characteristic equations. Therefore, we have the inclusion
$\boldsymbol{\mathcal{O}}_{\eta}\subseteq\boldsymbol{\mathcal{C}}_{\eta}$.

  * (ii).

Given any $\pi\in\mathbf{\Pi}$, there exists $\sigma\in\mathbf{M}$ such that
$\mu_{\eta,\pi}=\mu_{\eta,\sigma}$. Hence,
$\boldsymbol{\mathcal{O}}^{s}_{\eta}=\boldsymbol{\mathcal{O}}_{\eta}$.

Proof. (i). We have already shown that $\mu_{\eta,\pi}$ is a finite measure on
$\mathbf{X}\times\mathbf{A}$. To prove the stated result, note that for any
$B\in\boldsymbol{\mathfrak{X}}$ we have

$\displaystyle\mu^{\mathbf{X}}(B)=\sum_{t=0}^{\infty}\mathbb{P}_{\eta,\pi}\\{T_{\Delta}>t,X_{t}\in
B\\}=\eta(B-\Delta)+\sum_{t=1}^{\infty}\mathbb{E}_{\eta,\pi}\big{[}\mathbb{P}_{\eta,\pi}\\{T_{\Delta}>t,X_{t}\in
B\mid H_{t-1},A_{t-1}\\}\big{]}.$

Observe now that for each $t\geq 1$ the conditional probability within
brackets vanishes on the set $\\{T_{\Delta}\leq t-1\\}$, and so

$\displaystyle\mu^{\mathbf{X}}(B)$ $\displaystyle=$
$\displaystyle\eta(B-\Delta)+\sum_{t=1}^{\infty}\mathbb{E}_{\eta,\pi}\big{[}Q(B-\Delta\mid
X_{t-1},A_{t-1})\cdot\mathbf{I}_{\\{T_{\Delta}>t-1\\}}\big{]}$
$\displaystyle=$
$\displaystyle\eta(B-\Delta)+\int_{\mathbf{X}\times\mathbf{A}}Q(B-\Delta|x,a)\mu(dx,da),$

which can be equivalently written precisely as $\mu^{\mathbf{X}}=(\eta+\mu
Q)\mathbb{I}_{\Delta^{c}}$. By construction of the state-action process, it is
clear that $\mu(\mathbf{K}^{c})=0$. This shows that $\mu_{\eta,\pi}$ indeed
verifies the characteristic equations.
(ii). We will show that for any $\pi\in\mathbf{\Pi}$ there is some
$\sigma\in\mathbf{M}$ with $\mu_{\eta,\pi}=\mu_{\eta,\sigma}$. Since
$\mu_{\eta,\pi}$ is in
$\boldsymbol{\mathcal{M}}_{+}(\mathbf{X}\times\mathbf{A})$ with
$\mu_{\eta,\pi}(\mathbf{K}^{c})=0$, the measure $\mu_{\eta,\pi}$ can be
disintegrated as

$\mu_{\eta,\pi}=\mu_{\eta,\pi}^{\mathbf{X}}\otimes\sigma\quad\hbox{for some
$\sigma\in\mathbf{M}$}.$ (5)

It follows that the characteristic equation can be written

$\mu_{\eta,\pi}^{\mathbf{X}}=(\eta+\mu_{\eta,\pi}Q)\mathbb{I}_{\Delta^{c}}=(\eta+\mu_{\eta,\pi}^{\mathbf{X}}Q_{\sigma})\mathbb{I}_{\Delta^{c}}.$

Iterating this equation we obtain that for any $t\geq 0$

$\displaystyle\mu_{\eta,\pi}^{\mathbf{X}}=\eta\mathbb{I}_{\Delta^{c}}\sum_{k=0}^{t-1}(Q_{\sigma}\mathbb{I}_{\Delta^{c}})^{k}+\mu_{\eta,\pi}^{\mathbf{X}}(Q_{\sigma}\mathbb{I}_{\Delta^{c}})^{t}.$
(6)

We have, for any $x\in\mathbf{X}$ and $t\geq 0$,

$(Q_{\sigma}\mathbb{I}_{\Delta^{c}})^{t}(\mathbf{X}|x)=Q_{\sigma}^{t}(\Delta^{c}|x)=\mathbb{P}_{x,\sigma}\\{T_{\Delta}>t\\}$

and
$\mathbb{P}_{x,\sigma}\\{T_{\Delta}>t\\}\downarrow\mathbb{P}_{x,\sigma}\\{T_{\Delta}=\infty\\}$
as $t\rightarrow\infty$ for any $x\in\mathbf{X}$. Applying Lemma 3.2, we
obtain that
$\lim_{t\rightarrow\infty}\mu_{\eta,\pi}^{\mathbf{X}}(Q_{\sigma}\mathbb{I}_{\Delta^{c}})^{t}(\mathbf{X})=0$.
Taking the limit as $t\rightarrow\infty$ in (6) we obtain

$\mu_{\eta,\pi}^{\mathbf{X}}=\eta\mathbb{I}_{\Delta^{c}}\sum_{k=0}^{\infty}(Q_{\sigma}\mathbb{I}_{\Delta^{c}})^{k}=\mu_{\eta,\sigma}^{\mathbf{X}}.$

Once we have shown that the $\mathbf{X}$-marginals of the occupation measures
of $\mu_{\eta,\pi}$ and $\mu_{\eta,\sigma}$ coincide, we conclude the result
from (5). $\Box$
The result in Proposition 3.3 does not exclude the possibility that there are
measures in $\boldsymbol{\mathcal{C}}_{\eta}$ but not in
$\boldsymbol{\mathcal{O}}_{\eta}$. Such measures are called _phantom
measures_. In other words, a phantom measure
$\mu\in\boldsymbol{\mathcal{C}}_{\eta}$ satisfies the characteristic equations
but it is not the occupation measure of any policy in $\mathbf{\Pi}$. The
following simple example illustrates the existence of phantom measures.

###### Example 3.4

Consider on $\mathbf{X}=\mathbb{N}\mathop{\cup}\\{-1,-2\\}$ the model given by
$Q(\\{0\\}|0,a)=Q(\\{0\\}|1,a)=1$ and for some $\beta\in(0,1)$,
$Q(\\{1\\}|i,a)=1-\beta$ and $Q(\\{i+1\\}|i,a)=\beta$ for $i\geq 2$ and
$Q(\\{j\\}|j,a)=1-\beta$ for $j\in\\{-1,-2\\}$ and
$Q(\\{-1\\}|-2,a)=Q(\\{-2\\}|-1,a)=\beta$ where $a$ is the only action. Let us
consider $\eta=1/2\delta_{1}+1/2\delta_{2}$ as initial distribution. A mesure
$\nu$ on $\mathbf{X}$ is a solution of (3) if and only if it satisfies
$\nu(0)=0$, $\displaystyle\nu(1)=\eta(1)+(1-\beta)\sum_{j=2}^{\infty}\nu(j)$,
$\nu(2)=\eta(2)$ and $\nu(i+1)=\beta\nu(i)$ for $i\geq 2$ and
$\nu(-1)=(1-\beta)\nu(-1)+\beta\nu(-2)$,
$\nu(-2)=(1-\beta)\nu(-2)+\beta\nu(-1)$. Therefore, this implies that any
measure $\nu_{K}$ for $K>0$ defined by $\nu_{K}(0)=0$,
$\displaystyle\nu_{K}(1)=1$ and $\nu_{K}(n)=1/2\beta^{n-2}$ for $n\geq 2$ and
$\nu_{K}(-1)=\nu_{K}(-2)=K$ is a phantom measure. It will be shown in Theorem
4.5 that the occupation measures are characterized by (3) and an additional
condition. This is the first notable difference with the discounted model,
where the occupation measures are essentially characterized by a single
equation of type (3), that is, $\mu(\mathbf{K}^{c})=0$ and
$\mu^{\mathbf{X}}=\eta+\alpha\mu Q$ (where $\alpha$ is the discount factor)
excluding the existence of phantom measures.

###### Definition 3.5

Let $\boldsymbol{\mathfrak{M}}(\eta,\Delta)$ be an absorbing control model.
Given a measure
$\vartheta\in\boldsymbol{\mathcal{M}}_{+}(\mathbf{X}\times\mathbf{A})$ we say
that $\vartheta$ is invariant for the kernel $Q\mathbb{I}_{\Delta^{c}}$ when
$\vartheta(\mathbf{K}^{c})=0$ and $\vartheta^{\mathbf{X}}=\vartheta
Q\mathbb{I}_{\Delta^{c}}$.

Let us make some comments on this definition. Note that we are not excluding
that $\vartheta$ is the null measure. We call such a measure invariant
because, by disintegration, there exists $\sigma\in\mathbf{M}$ satisfying
$\vartheta^{\mathbf{X}}=\vartheta^{\mathbf{X}}Q_{\sigma}\mathbb{I}_{\Delta^{c}}$
which can be written as

$\vartheta^{\mathbf{X}}(\Delta)=0\quad\hbox{and}\quad\vartheta^{\mathbf{X}}(B)=\int_{\mathbf{X}}Q_{\sigma}(B|x)\vartheta^{\mathbf{X}}(dx)$

for measurable $B\subseteq\Delta^{c}$, and so $\vartheta^{\mathbf{X}}$ is an
invariant measure for the substochastic kernel $Q_{\sigma}$ on $\Delta^{c}$.

###### Theorem 3.6

Let $\boldsymbol{\mathfrak{M}}(\eta,\Delta)$ be absorbing.

  * (i).

A measure is in $\boldsymbol{\mathcal{C}}_{\eta}$ if and only if it can be
decomposed as the sum of an occupation measure $\mu_{\eta,\sigma}$ in
$\boldsymbol{\mathcal{O}}_{\eta}$ and an invariant measure $\nu\otimes\sigma$
for $Q\mathbb{I}_{\Delta^{c}}$ with $\sigma\in\mathbf{M}$ and
$\nu\in\boldsymbol{\mathcal{M}}_{+}(\mathbf{X})$.

  * (ii).

The following statements are equivalent.

    * (a).

The unique invariant measure for $Q\mathbb{I}_{\Delta^{c}}$ is the null
measure on $\mathbf{X}\times\mathbf{A}$.

    * (b).

$\boldsymbol{\mathcal{C}}_{\eta}=\boldsymbol{\mathcal{O}}_{\eta}$, i.e., there
are no phantom measures.

    * (c).

The set $\boldsymbol{\mathcal{C}}_{\eta}$ is bounded.

Proof. (i). Suppose that
$\mu\in\boldsymbol{\mathcal{M}}^{+}(\mathbf{X}\times\mathbf{A})$ is a solution
of the characteristic equations. Proceeding as in the proof of Proposition
3.3, we derive the existence of $\sigma\in\mathbf{M}$ such that
$\mu=\mu^{\mathbf{X}}\otimes\sigma$ and

$\displaystyle\mu^{\mathbf{X}}=\eta\mathbb{I}_{\Delta^{c}}\sum_{k=0}^{t-1}(Q_{\sigma}\mathbb{I}_{\Delta^{c}})^{k}+\mu^{\mathbf{X}}(Q_{\sigma}\mathbb{I}_{\Delta^{c}})^{t}$
(7)

for any $t\geq 1$. It follows that by taking the limit as $t\rightarrow\infty$
that

$\nu=\lim_{t\rightarrow\infty}\mu^{\mathbf{X}}(Q_{\sigma}\mathbb{I}_{\Delta^{c}})^{t}=\mu^{\mathbf{X}}-\mu_{\eta,\sigma}^{\mathbf{X}}$

defines a measure in
$\boldsymbol{\mathcal{M}}_{+}(\mathbf{X}\times\mathbf{A})$ satisfying
$\mu^{\mathbf{X}}=\mu_{\eta,\sigma}^{\mathbf{X}}+\nu$. Recalling that

$\mu^{\mathbf{X}}=\big{(}\eta+\mu^{\mathbf{X}}Q_{\sigma}\big{)}\mathbb{I}_{\Delta^{c}}\quad\hbox{and}\quad\mu_{\eta,\sigma}^{\mathbf{X}}=\big{(}\eta+\mu_{\eta,\sigma}^{\mathbf{X}}Q_{\sigma}\big{)}\mathbb{I}_{\Delta^{c}},$

we get that $\nu=\nu Q_{\sigma}\mathbb{I}_{\Delta^{c}}$. The measure
$\vartheta=\nu\otimes\sigma$ satisfies the conditions in the statement of this
proposition: $\mu=\mu_{\eta,\sigma}+\vartheta$.

Conversely, it is straightforward to check that the sum of an invariant
measure and an occupation measure (which satisfies the characteristic
equations) satisfies itself the characteristic equations and, hence, belongs
to $\boldsymbol{\mathcal{C}}_{\eta}$. In fact, the sum of any measure in
$\boldsymbol{\mathcal{C}}_{\eta}$ and an invariant measure lies in
$\boldsymbol{\mathcal{C}}_{\eta}$.
(ii). The implication $(a)\Rightarrow(b)$ follows directly from item (i),
while $(b)\Rightarrow(c)$ is derived from Lemma 2.4. We prove
$(c)\Rightarrow(a)$ by contradiction and, hence, we suppose that there exists
a non-null invariant measure $\vartheta$. For any $K>0$ we have that
$K\vartheta$ is also an invariant measure and so for any
$\mu\in\boldsymbol{\mathcal{C}}_{\eta}$ and every $K>0$ we have
$\mu+K\vartheta\in\boldsymbol{\mathcal{C}}_{\eta}$, which is not compatible
with $\boldsymbol{\mathcal{C}}_{\eta}$ being bounded. $\Box$
Our next result gives some insight on the behavior of phantom measures.

###### Proposition 3.7

Suppose that $\boldsymbol{\mathfrak{M}}(\eta,\Delta)$ is absorbing and let
$\vartheta$ be a non-null invariant measure for $Q\mathbb{I}_{\Delta^{c}}$.
For every $\pi\in\mathbf{\Pi}$ we have
$\mu_{\eta,\pi}^{\mathbf{X}}\perp\vartheta^{\mathbf{X}}$.

Proof. By disintegration of $\vartheta$ there exists $\sigma\in\mathbf{M}$
with
$\vartheta^{\mathbf{X}}=\vartheta^{\mathbf{X}}Q_{\sigma}\mathbb{I}_{\Delta^{c}}$.
Iterating this equation we obtain
$\vartheta^{\mathbf{X}}=\vartheta^{\mathbf{X}}Q^{t}_{\sigma}\mathbb{I}_{\Delta^{c}}$
for every $t\geq 1$ and taking the limit as $t\rightarrow\infty$ yields

$\vartheta^{\mathbf{X}}(\mathbf{X})=\vartheta^{\mathbf{X}}(\Delta^{c})=\int_{\mathbf{X}}\mathbb{P}_{x,\sigma}\\{T_{\Delta}=\infty\\}\vartheta^{\mathbf{X}}(dx)$

and so the set
$B=\\{x\in\mathbf{X}:\mathbb{P}_{x,\sigma}\\{T_{\Delta}=\infty\\}<1\\}$
satisfies $\vartheta^{\mathbf{X}}(B)=0$. By the Lebesgue decomposition theorem
there exist two finite measures
$\vartheta_{1}^{\mathbf{X}},\vartheta_{2}^{\mathbf{X}}\in\boldsymbol{\mathcal{M}}_{+}(\mathbf{X})$
with

$\vartheta^{\mathbf{X}}=\vartheta_{1}^{\mathbf{X}}+\vartheta_{2}^{\mathbf{X}}\quad\hbox{with
$\vartheta_{1}^{\mathbf{X}}\ll\mu_{\eta,\pi}^{\mathbf{X}}$ and
$\vartheta_{2}^{\mathbf{X}}\perp\mu_{\eta,\pi}^{\mathbf{X}}$}.$

Therefore,

$\displaystyle\vartheta^{\mathbf{X}}(\mathbf{X})$ $\displaystyle=$
$\displaystyle\int_{\mathbf{X}}\mathbb{P}_{x,\sigma}\\{T_{\Delta}=\infty\\}\vartheta^{\mathbf{X}}_{1}(dx)+\int_{\mathbf{X}}\mathbb{P}_{x,\sigma}\\{T_{\Delta}=\infty\\}\vartheta^{\mathbf{X}}_{2}(dx)$
$\displaystyle=$
$\displaystyle\int_{\mathbf{X}}\mathbb{P}_{x,\sigma}\\{T_{\Delta}=\infty\\}\cdot\frac{d\vartheta^{\mathbf{X}}_{1}}{d\mu^{\mathbf{X}}_{\eta,\pi}}\
\mu^{\mathbf{X}}_{\eta,\pi}(dx)+\int_{\mathbf{X}}\mathbb{P}_{x,\sigma}\\{T_{\Delta}=\infty\\}\vartheta^{\mathbf{X}}_{2}(dx).$

Applying Lemma 3.2, we obtain that
$\mathbb{P}_{x,\sigma}\\{T_{\Delta}=\infty\\}$ vanishes
$\mu^{\mathbf{X}}_{\eta,\pi}$-a.s. and so

$\vartheta^{\mathbf{X}}(\mathbf{X})=\int_{\mathbf{X}}\mathbb{P}_{x,\sigma}\\{T_{\Delta}=\infty\\}\vartheta^{\mathbf{X}}_{2}(dx).$

Observe that $\vartheta^{\mathbf{X}}_{2}(B)=0$ and so

$\vartheta^{\mathbf{X}}(\mathbf{X})=\int_{B^{c}}\mathbb{P}_{x,\sigma}\\{T_{\Delta}=\infty\\}\vartheta^{\mathbf{X}}_{2}(dx)=\vartheta^{\mathbf{X}}_{2}(B^{c})\
=\ \vartheta^{\mathbf{X}}_{2}(\mathbf{X}).$

This shows that the measure $\vartheta^{\mathbf{X}}_{1}$ is null and,
therefore, $\vartheta^{\mathbf{X}}=\vartheta^{\mathbf{X}}_{2}$, showing the
result. $\Box$
This result establishes, loosely speaking, that the state process
$\\{X_{t}\\}$ under any policy $\pi\in\mathbf{\Pi}$ never visits the support
of a non-null invariant measure.

##  4 Compactness of occupation measures

In section, we introduce an additional hypothesis. It is assumed the so-called
continuity-compactness conditions (see Condition (S) below). In this context,
we show in our second main result (see Theorem 4.9) that the set of occupation
measures is compact in the weak-strong topology if and only if the model is
uniformly absorbing. Finally, it is shown that the occupation measures are
characterized by the characteristic equation and an additional condition see
our last main result (see Theorem 4.5). This represents a distinguishing
factor from the discounted model, where occupation measures are fundamentally
defined by a single characteristic equation.

Condition (S)

  * $(\mathrm{S}_{1})$

The action set $\mathbf{A}$ is compact and the correspondence from
$\mathbf{X}$ to $\mathbf{A}$ given by $x\mapsto\mathbf{A}(x)$ is weakly
measurable with nonempty compact values.

  * $(\mathrm{S}_{2})$

For any $x\in\mathbf{X}$ and $\Gamma\in\boldsymbol{\mathfrak{X}}$, the mapping
$a\mapsto Q(\Gamma|x,\cdot)$ is continuous on $\mathbf{A}$.

###### Lemma 4.1

Suppose that the Conditions $(\mathrm{S}_{1})$-$(\mathrm{S}_{2})$ are
satisfied. There exists $\xi^{*}\in\mathbf{M}$ such that

$\displaystyle Q(\cdot|x,a)\ll Q_{\xi^{*}}(\cdot|x)\quad\hbox{for every
$(x,a)\in\mathbf{K}$}.$ (8)

Proof. The multifunction from $\mathbf{X}$ to $\mathbf{A}$ defined by
$x\rightarrow\mathbf{A}(x)$ is weakly measurable. By $(\mathrm{S}_{1})$ and
Corollary 18.15 in [1], we obtain the existence of a sequence
$\\{\xi_{n}\\}_{n\in\mathbb{N}}$ of measurable selectors for the multifunction
$x\mapsto\mathbf{A}(x)$ satisfying

$\mathbf{A}(x)=\overline{\\{\xi_{n}(x):n\in\mathbb{N}\\}}\quad\hbox{for any
$x\in\mathbf{X}$.}$ (9)

Define now $\xi^{*}\in\mathbf{\Pi}^{s}$ by means of

$\xi^{*}(da|x)=\sum_{k\in\mathbb{N}}\frac{1}{2^{k+1}}\delta_{\xi_{k}(x)}(da)\quad\hbox{so
that}\quad
Q_{\xi^{*}}(dy|x)=\sum_{k\in\mathbb{N}}\frac{1}{2^{k+1}}Q(dy|x,\xi_{k}(x)).$

To prove the result, fix arbitrary $(x,a)\in\mathbf{K}$ and
$B\in\boldsymbol{\mathfrak{X}}$ such that $Q_{\xi^{*}}(B|x)=0$. This implies
that $Q(B|x,\xi^{*}_{k}(x))=0$ for all $k\in\mathbb{N}$. Using
$(\mathrm{S}_{2})$ and (9) we obtain that $Q(B|x,a)=0$. $\Box$
Based on this result, we define
$\lambda^{\beta}\in\boldsymbol{\mathcal{P}}(\mathbf{X})$ as

$\lambda^{\beta}=(1-\beta)\sum_{k\in\mathbb{N}}\beta^{k}\eta Q_{\xi^{*}}^{k}$
(10)

where $\beta$ is some fixed parameter with $0<\beta<1$.

###### Lemma 4.2

Suppose that $\boldsymbol{\mathfrak{M}}(\eta,\Delta)$ is absorbing and that
the Conditions $(\mathrm{S}_{1})$-$(\mathrm{S}_{2})$ are satisfied. For every
$\pi\in\mathbf{\Pi}$ we have

$\displaystyle\mu^{\mathbf{X}}_{\eta,\pi}\ll\mu^{\mathbf{X}}_{\eta,\xi^{*}}\sim\lambda^{\beta}_{\Delta^{c}}\ll\lambda^{\beta}.$

Proof: By Proposition 3.3(ii), there exists $\sigma\in\mathbf{M}$ such that

$\mu_{\eta,\pi}^{\mathbf{X}}=\eta\sum_{k=0}^{\infty}Q_{\sigma}^{k}\mathbb{I}_{\Delta^{c}}.$
(11)

We are going to show that for every $k\geq 1$ and $x\in\mathbf{X}$ we have
$Q^{k}_{\sigma}(\cdot|x)\ll Q^{k}_{\xi^{*}}(\cdot|x)$. The proof is by
induction.

For the case $k=1$ suppose that $Q_{\xi^{*}}(B|x)=0$. By Lemma 4.1 this
implies that $Q(B|x,a)=0$ for all $a\in\mathbf{A}(x)$ and $Q_{\sigma}(B|x)=0$
follows. Assuming the result true for some $k$ and, for $k+1$, suppose that
$Q^{k+1}_{\xi^{*}}(B|x)=0$. Note that

$Q^{k+1}_{\xi^{*}}(B|x)=\int_{\mathbf{X}}Q^{k}_{\xi^{*}}(B|y)Q_{\xi^{*}}(dy|x)=0$

implies that $Q^{k}_{\xi^{*}}(B|y)=0$ for all $y\in C$ where $C$ is such that
$Q_{\xi^{*}}(C|x)=1$. By the induction hypothesis, we have that
$Q^{k}_{\sigma}(B|y)=0$ for all $y\in C$ with $Q_{\sigma}(C|x)=1$. This shows
that, indeed, $Q^{k+1}_{\sigma}(B|x)=0$.

As a direct consequence we have that

$\eta\sum_{k=0}^{\infty}Q^{k}_{\sigma}\ll\eta\sum_{k=0}^{\infty}Q^{k}_{\xi^{*}}\sim(1-\beta)\eta\sum_{k=0}^{\infty}\beta^{k}Q^{k}_{\xi^{*}}=\lambda^{\beta}$

and the stated result follows from (11). $\Box$

###### Proposition 4.3

Under Conditions $(\mathrm{S}_{1})$-$(\mathrm{S}_{2})$, if the control model
$\boldsymbol{\mathfrak{M}}(\eta,\Delta)$ is absorbing then
$\boldsymbol{\mathfrak{M}}(\lambda^{\beta},\Delta)$ is also absorbing.

Proof. Consider arbitrary $\pi\in\mathbf{\Pi}$. We have the following
equalities

$\displaystyle\mathbb{E}_{\lambda^{\beta},\pi}[T_{\Delta}]$ $\displaystyle=$
$\displaystyle\sum_{j=0}^{\infty}\mathbb{E}_{\lambda^{\beta},\pi}[\mathbf{I}_{\Delta^{c}}(X_{j})]$
$\displaystyle=$
$\displaystyle(1-\beta)\sum_{j=0}^{\infty}\sum_{k=0}^{\infty}\beta^{k}\mathbb{E}_{\eta
Q^{k}_{\xi^{*}},\pi}[\mathbf{I}_{\Delta^{c}}(X_{j})].$

Observe that using the policy $\pi$ for the initial distribution $\eta
Q^{k}_{\xi^{*}}$ is equivalent to using the policy $\gamma^{k}\in\mathbf{\Pi}$
given by

$\gamma^{k}_{j}(da|x_{0},a_{0},\ldots,x_{j})=\xi^{*}(da|x_{j})\quad\hbox{for
$0\leq j<k$}$

and

$\gamma^{k}_{j}(da|x_{0},a_{0},\ldots,x_{j})=\pi_{j-k}(da|x_{k},a_{k},\ldots,x_{j})\quad\hbox{for
$j\geq k$}$

for the initial distribution $\eta$ just by make a shift of $k$ time units.
Therefore,

$\mathbb{E}_{\eta
Q^{k}_{\xi^{*}},\pi}[\mathbf{I}_{\Delta^{c}}(X_{j})]=\mathbb{E}_{\eta,\gamma^{k}}[\mathbf{I}_{\Delta^{c}}(X_{j+k})].$

It follows that

$\displaystyle\mathbb{E}_{\lambda^{\beta},\pi}[T_{\Delta}]$ $\displaystyle=$
$\displaystyle(1-\beta)\sum_{j=0}^{\infty}\sum_{k=0}^{\infty}\beta^{k}\mathbb{E}_{\eta,\gamma^{k}}[\mathbf{I}_{\Delta^{c}}(X_{j+k})]$
$\displaystyle=$
$\displaystyle(1-\beta)\sum_{k=0}^{\infty}\beta^{k}\sum_{j=0}^{\infty}\mathbb{E}_{\eta,\gamma^{k}}[\mathbf{I}_{\Delta^{c}}(X_{j+k})]$
$\displaystyle\leq$
$\displaystyle(1-\beta)\sum_{k=0}^{\infty}\beta^{k}\mathbb{E}_{\eta,\gamma^{k}}[T_{\Delta}].$

The model $\boldsymbol{\mathfrak{M}}(\eta,\Delta)$ being absorbing, we have
that

$\sup_{\pi\in\mathbf{\Pi}}\mathbb{E}_{\eta,\pi}[T_{\Delta}]=\mathbf{c}<\infty$

and so $\mathbb{E}_{\lambda^{\beta},\pi}[T_{\Delta}]\leq\mathbf{c}$, which
shows that $\boldsymbol{\mathfrak{M}}(\lambda^{\beta},\Delta)$ is absorbing as
well. $\Box$

###### Proposition 4.4

Suppose that $\boldsymbol{\mathfrak{M}}(\eta,\Delta)$ is absorbing and that
the Conditions $(\mathrm{S}_{1})$-$(\mathrm{S}_{2})$ are satisfied. Let
$\boldsymbol{\Gamma}$ be an arbitrary subset of $\mathbf{\Pi}^{s}$ and let
$\\{h_{\pi}\\}_{\pi\in\boldsymbol{\Gamma}}$ be a family of non-negative
functions in $L^{1}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda^{\beta})$
which are uniformly $\lambda^{\beta}$-integrable. Under these conditions,

$\lim_{t\rightarrow\infty}\sup_{\pi\in\boldsymbol{\Gamma}}\int_{\mathbf{X}}Q^{t}_{\pi}(\Delta^{c}|x)h_{\pi}(x)\lambda^{\beta}(dx)=0.$

Proof. Consider a fixed arbitrary $\epsilon>0$. By the uniform integrability
hypothesis, there exists $c_{\epsilon}>0$ such that

$\sup_{\pi\in\boldsymbol{\Gamma}}\int_{\\{x\in\mathbf{X}:h_{\pi}(x)>c_{\epsilon}\\}}h_{\pi}(x)\lambda^{\beta}(dx)\leq\epsilon.$

Therefore, for any $\pi\in\mathbf{\Gamma}$ and $t\geq 1$

$\int_{\mathbf{X}}Q^{t}_{\pi}(\Delta^{c}|x)h_{\pi}(x)\lambda^{\beta}(dx)\leq\epsilon+c_{\epsilon}\mathbb{P}_{\lambda^{\beta},\pi}\\{T_{\Delta}>t\\}\leq\epsilon+\frac{c_{\epsilon}}{t}\cdot\mathbb{E}_{\lambda^{\beta},\pi}[T_{\Delta}]\leq\epsilon+\frac{c_{\epsilon}}{t}\sup_{\pi\in\mathbf{\Pi}}\mathbb{E}_{\lambda^{\beta},\pi}[T_{\Delta}]$

By Proposition 4.3 we have that the above supremum is finite. Hence, for $t$
sufficiently large we obtain that

$\sup_{\pi\in\boldsymbol{\Gamma}}\int_{\mathbf{X}}Q^{t}_{\pi}(\Delta^{c}|x)h_{\pi}(x)\lambda^{\beta}(dx)<2\epsilon$

and the result follows. $\Box$
Our next result gives a characterization of $\boldsymbol{\mathcal{O}}_{\eta}$
based on the probability measure $\lambda^{\beta}$.

###### Theorem 4.5

Let $\boldsymbol{\mathfrak{M}}(\eta,\Delta)$ be an absorbing model satisfying
the conditions $(\mathrm{S}_{1})$-$(\mathrm{S}_{2})$. Given a measure
$\mu\in\boldsymbol{\mathcal{C}}_{\eta}$, we have
$\mu\in\boldsymbol{\mathcal{O}}_{\eta}$ if and only if
$\mu^{\mathbf{X}}\ll\lambda^{\beta}$.

Proof. Clearly, if $\mu\in\boldsymbol{\mathcal{O}}_{\eta}$ then
$\mu^{\mathbf{X}}\ll\lambda^{\beta}$ as a direct consequence of Lemma 4.2.

Conversely, let us assume that $\mu\in\boldsymbol{\mathcal{C}}_{\eta}$ is such
that $\mu^{\mathbf{X}}\ll\lambda^{\beta}$. Using Theorem 3.6(i), we can find
$\sigma\in\mathbf{M}$ and an invariant measure
$\vartheta\in\boldsymbol{\mathcal{M}}_{+}(\mathbf{X}\times\mathbf{A})$ for the
kernel $Q\mathbb{I}_{\Delta^{c}}$ such that

$\mu=\mu_{\eta,\sigma}+\vartheta\quad\hbox{and}\quad\vartheta^{\mathbf{X}}=\vartheta^{\mathbf{X}}Q_{\sigma}\mathbb{I}_{\Delta^{c}}.$

This implies that $\vartheta^{\mathbf{X}}\ll\lambda^{\beta}$ and so for every
$t\geq 1$

$\displaystyle\vartheta^{\mathbf{X}}(\mathbf{X})=\vartheta^{\mathbf{X}}(Q_{\sigma}\mathbb{I}_{\Delta^{c}})^{t}(\mathbf{X})=\vartheta^{\mathbf{X}}Q^{t}_{\sigma}(\Delta^{c})=\int_{\mathbf{X}}Q^{t}_{\sigma}(\Delta^{c}|x)\frac{d\vartheta^{\mathbf{X}}}{d\lambda^{\beta}}(x)\lambda^{\beta}(dx).$

Applying Proposition 4.4 to the set $\mathbf{\Gamma}=\\{\sigma\\}$ and the
function $d\vartheta^{\mathbf{X}}/d\lambda^{\beta}$, we can take the limit as
$t\rightarrow\infty$ in the last expression to obtain that
$\vartheta^{\mathbf{X}}(\mathbf{X})=0$. This shows that, indeed,
$\mu\in\boldsymbol{\mathcal{O}}_{\eta}$. $\Box$

###### Lemma 4.6

Assume that the Conditions $(\mathrm{S}_{1})$-$(\mathrm{S}_{2})$ hold. Let
$\\{\nu_{\pi}\\}_{\pi\in\mathbf{\Pi}^{s}}$ be a relatively $s$-compact subset
of $\boldsymbol{\mathcal{M}}_{+}(\mathbf{X})$. The set
$\\{\nu_{\pi}Q_{\pi}\\}_{\pi\in\mathbf{\Pi}^{s}}$ is relatively $s$-compact
subset of $\boldsymbol{\mathcal{M}}_{f}(\mathbf{X})$.

Proof. Define

$\mathbf{\Lambda}=\\{\nu_{\pi}\otimes\pi\\}_{\pi\in\mathbf{\Pi}^{s}}\subseteq\boldsymbol{\mathcal{M}}_{f}(\mathbf{X}\times\mathbf{A}).$

By hypothesis, $\mathbf{\Lambda}^{\mathbf{X}}$ is relatively $s$-compact and
$\mathbf{\Lambda}^{\mathbf{A}}$ is tight since $\mathbf{A}$ is a compact
metric space. Therefore, $\mathbf{\Lambda}$ is relatively sequentially
$ws$-compact by using Theorem 2.5 in [4]. So, there exists a sequence
$\\{\pi_{n}\\}$ in $\mathbf{\Pi}^{s}$ such that

$\\{\nu_{\pi_{n}}\otimes\pi_{n}\\}\stackrel{{\scriptstyle
ws}}{{\longrightarrow}}\mu$ (12)

for some measure
$\mu\in\boldsymbol{\mathcal{M}}_{f}(\mathbf{X}\times\mathbf{A})$. By the limit
(12) and since the function $(x,a)\mapsto Q(B|x,a)$ on
$\mathbf{X}\times\mathbf{A}$ is a bounded Carathéodory function for every
$B\in\boldsymbol{\mathfrak{X}}$, it follows that

$\nu_{\pi_{n}}Q_{\pi_{n}}(B)=\int_{\mathbf{X}\times\mathbf{A}}Q(B|x,a)d(\nu_{\pi_{n}}\otimes\pi_{n})\rightarrow\int_{\mathbf{X}\times\mathbf{A}}Q(B|x,a)d\mu=\mu
Q(B),$

which shows that

$\nu_{\pi_{n}}Q_{\pi_{n}}\stackrel{{\scriptstyle s}}{{\rightarrow}}\mu Q,$

establishing that $\\{\nu_{\pi}Q_{\pi}\\}_{\pi\in\mathbf{\Pi}^{s}}$ is
relatively sequentially $s$-compact. Finally, [4, Proposition 2.2] or [12,
Theorem 2.6] give the result on relative $s$-compactness. $\Box$
Given the initial distribution $\eta\in\boldsymbol{\mathcal{P}}(\mathbf{X})$,
a policy $\pi\in\mathbf{\Pi}$, and $k\in\mathbb{N}$ we define
$\mu_{\eta,\pi,k}$ as the distribution of $(X_{k},A_{k})$ under
$\mathbb{P}_{\eta,\pi}$. In symbols, we let

$\mu_{\eta,\pi,k}=\mathbb{P}_{\eta,\pi}\circ(X_{k},A_{k})^{-1}\in\boldsymbol{\mathcal{P}}(\mathbf{X}\times\mathbf{A}).$

We consider the following subsets of measures

$\boldsymbol{\mathcal{D}}_{\eta,k}^{s}=\\{\mu_{\eta,\pi,k}\\}_{\pi\in\mathbf{\Pi}^{s}}\quad\hbox{for
$k\in\mathbb{N}$},$

which correspond to the families of state-action distributions of the process
at time $k$ for stationary policies.

###### Lemma 4.7

Assume that the Conditions $(\mathrm{S}_{1})$-$(\mathrm{S}_{2})$ hold. For
each $k\in\mathbb{N}$, the sets $\boldsymbol{\mathcal{D}}_{\eta,k}^{s}$ are
relatively $ws$-compact and

$\lim_{n\rightarrow\infty}\sup_{\pi\in\mathbf{\Pi}^{s}}\mu^{\mathbf{X}}_{\eta,\pi,k}(\Gamma_{n})=0$
(13)

for any decreasing sequence of sets $\Gamma_{n}\in\boldsymbol{\mathfrak{X}}$
with $\Gamma_{n}\downarrow\emptyset$.

Proof. First of all we prove by induction that
$(\boldsymbol{\mathcal{D}}_{\eta,k}^{s})^{\mathbf{X}}$ is relatively
$s$-compact in $\boldsymbol{\mathcal{M}}_{+}(\mathbf{X})$ for every
$k\in\mathbb{N}$. For $k=0$ we have
$(\boldsymbol{\mathcal{D}}_{\eta,k}^{s})^{\mathbf{X}}=\\{\eta\\}$ and the
result is obvious. Assuming the result is true for some $k\geq 0$, observe
that $\mu^{\mathbf{X}}_{\eta,\pi,k+1}=\mu^{\mathbf{X}}_{\eta,\pi,k}Q_{\pi}$
and we get the claim by applying Lemma 4.6.

Once we know that
$\\{\mu_{\eta,\pi,k}^{\mathbf{X}}\\}_{\pi\in\mathbf{\Pi}^{s}}$ is relatively
$s$-compact and $\mathbf{A}$ being a compact metric space, it follows that

$\\{\mu^{\mathbf{X}}_{\eta,\pi,k}\otimes\pi\\}_{\pi\in\mathbf{\Pi}^{s}}=\\{\mu_{\eta,\pi,k}\\}_{\pi\in\mathbf{\Pi}^{s}}=\boldsymbol{\mathcal{D}}^{s}_{\eta,k}$

is relatively $ws$-compact by Theorem 5.2 in [4], showing the first claim. The
last part of the result is obtained by using Lemma 4.6.5 and Theorem 4.7.25 in
[5]. $\Box$

###### Lemma 4.8

Suppose $\boldsymbol{\mathfrak{M}}(\eta,\Delta)$ is absorbing and that the
Conditions $(\mathrm{S}_{1})$-$(\mathrm{S}_{2})$ hold. The set
$\boldsymbol{\mathcal{O}}_{\eta}$ is $ws$-closed in
$\boldsymbol{\mathcal{M}}_{+}(\mathbf{X}\times\mathbf{A})$.

Proof. Let us consider a net $\\{\nu_{\alpha}\\}_{\alpha\in\mathcal{I}}$ in
$\boldsymbol{\mathcal{O}}_{\eta}$ converging to some
$\nu\in\boldsymbol{\mathcal{M}}_{f}(\mathbf{X}\times\mathbf{A})$ in the
$ws$-topology. From Theorem 4.5 we have that for any $\alpha\in\mathcal{I}$

$\displaystyle\nu_{\alpha}(\mathbf{K}^{c})=0,\qquad\nu^{\mathbf{X}}_{\alpha}=(\eta+\nu_{\alpha}Q)\mathbb{I}_{\Delta^{c}},\quad\hbox{and}\quad\nu^{\mathbf{X}}_{\alpha}\ll\lambda^{\beta}.$

It easily follows that the limiting measure $\nu$ satisfies
$\nu^{\mathbf{X}}\ll\lambda^{\beta}$ and the equation
$\nu^{\mathbf{X}}=(\eta+\nu Q)\mathbb{I}_{\Delta^{c}}$. Let us show that we
also have $\nu(\mathbf{K}^{c})=0$.

Since the action sets $\mathbf{A}(x)$ are compact, we have that the function
$(x,a)\mapsto\mathbf{I}_{\mathbf{K}^{c}}(x,a)$ is measurable, non-negative and
lower semi-continuous on $\mathbf{A}$ for each $x\in\mathbf{X}$. Let us denote
by $\bar{\boldsymbol{\mathfrak{X}}}$ the completion of
$\boldsymbol{\mathfrak{X}}$ with respect to $\lambda_{\beta}$. By [11,
Proposition 3.9] there exists a sequence
$\\{\bar{\Psi}_{n}\\}_{n\in\mathbb{N}}$ of
$\bar{\boldsymbol{\mathfrak{X}}}\otimes\boldsymbol{\mathfrak{B}}(\mathbf{A})$-measurable
bounded Carathéodory functions such that
${\bar{\Psi}}_{n}\uparrow\mathbf{I}_{\mathbf{K}^{c}}$ as $n\rightarrow\infty$.

Applying [3, Lemma A-6] to $\bar{\Psi}_{n}$ (being both a l.s.c. and a u.s.c.
integrand), for every $n\in\mathbb{N}$ there exists a measurable set
$N_{n}\in\boldsymbol{\mathfrak{X}}$ with $\lambda^{\beta}(N_{n})=0$ and a
$\boldsymbol{\mathfrak{X}}\otimes\boldsymbol{\mathfrak{B}}(\mathbf{A})$-measurable
bounded Carathéodory function $\tilde{\Psi}_{n}$ such that

$\tilde{\Psi}_{n}(x,a)=\bar{\Psi}_{n}(x,a)\quad\hbox{for all
$x\in\mathbf{X}-N_{n}$ and $a\in\mathbf{A}$}.$

Letting $N=\cup_{n\in\mathbb{N}}N_{n}$ we define
$\Psi_{n}(x,a)=\mathbf{I}_{N^{c}}(x)\cdot\tilde{\Psi}_{n}(x,a)$ for
$(x,a)\in\mathbf{X}\times\mathbf{A}$. It should be clear that each $\Psi_{n}$
is a
$\boldsymbol{\mathfrak{X}}\otimes\boldsymbol{\mathfrak{B}}(\mathbf{A})$-measurable
bounded Carathéodory function with

$\Psi_{n}\uparrow\mathbf{I}_{(N^{c}\times\mathbf{A})\cap\mathbf{K}^{c}}.$

Hence, given $\epsilon>0$ we can find some $n_{0}$ with

$\nu\big{(}(N^{c}\times\mathbf{A})\cap\mathbf{K}^{c}\big{)}\leq\int_{\mathbf{X}\times\mathbf{A}}\Psi_{n_{0}}d\nu+\epsilon/2.$

Since $\nu^{\mathbf{X}}\ll\lambda^{\beta}$ and $\lambda^{\beta}(N)=0$ it is
$\nu^{\mathbf{X}}(N)=0$ and, in fact, we have

$\nu(\mathbf{K}^{c})\leq\int_{\mathbf{X}\times\mathbf{A}}\Psi_{n_{0}}d\nu+\epsilon/2.$

There exists some $\alpha_{0}$ such that for every $\alpha\succ\alpha_{0}$

$\nu(\mathbf{K}^{c})\leq\int_{\mathbf{X}\times\mathbf{A}}\Psi_{n_{0}}d\nu+\epsilon/2\leq\int_{\mathbf{X}\times\mathbf{A}}\Psi_{n_{0}}d\nu_{\alpha}+\epsilon\leq\nu_{\alpha}((N^{c}\times\mathbf{A})\cap\mathbf{K}^{c}\big{)}+\epsilon.$

Since the last measure equals zero we have shown that
$\nu(\mathbf{K}^{c})\leq\epsilon$. As $\epsilon>0$ was arbitrary, we conclude
that $\nu(\mathbf{K}^{c})=0$. This completes the proof that
$\nu\in\boldsymbol{\mathcal{O}}_{\eta}$ and, therefore,
$\boldsymbol{\mathcal{O}}_{\eta}$ is closed. $\Box$

###### Theorem 4.9

Suppose $\boldsymbol{\mathfrak{M}}(\eta,\Delta)$ is absorbing and that the
Conditions $(\mathrm{S}_{1})$-$(\mathrm{S}_{2})$ are satisfied. The following
statements are equivalent.

  * $(a)$

$\boldsymbol{\mathfrak{M}}(\eta,\Delta)$ is uniformly absorbing.

  * $(b)$

The set $\boldsymbol{\mathcal{O}}_{\eta}$ is relatively $ws$-compact.

  * $(c)$

The set $\boldsymbol{\mathcal{O}}_{\eta}$ is $ws$-compact.

Proof. $(a)\Rightarrow(b)$. The set of $\mathbf{A}$-marginal measures of
$\boldsymbol{\mathcal{O}}_{\eta}$ is tight since $\mathbf{A}$ is compact.
Recalling that
$\boldsymbol{\mathcal{O}}_{\eta}=\boldsymbol{\mathcal{O}}_{\eta}^{s}$, by
Proposition 3.3(ii) and applying [4, Theorem 5.2] we need to show that the set
of $\mathbf{X}$-marginal measures of $\boldsymbol{\mathcal{O}}^{s}_{\eta}$ is
relatively $s$-compact or, equivalently, that

$\lim_{n\rightarrow\infty}\sup_{\pi\in\mathbf{\Pi}^{s}}\mu^{\mathbf{X}}_{\eta,\pi}(\Gamma_{n})=0$
(14)

for any decreasing sequence of sets $\Gamma_{n}\in\boldsymbol{\mathfrak{X}}$
with $\Gamma_{n}\downarrow\emptyset$ (see Lemma 4.6.5 and Theorem 4.7.25 in
[5]). Now, observe that for every $k\in\mathbb{N}$

$\displaystyle\sup_{\pi\in\mathbf{\Pi}^{s}}\mu^{\mathbf{X}}_{\eta,\pi}(\Gamma_{n})$
$\displaystyle\leq$
$\displaystyle\sum_{t=0}^{k}\sup_{\pi\in\mathbf{\Pi}^{s}}\mu^{\mathbf{X}}_{\eta,\pi,t}(\Gamma_{n})+\sup_{\pi\in\mathbf{\Pi}^{s}}\sum_{t>k}\mathbb{P}_{\eta,\pi_{\mu}}\\{T_{\Delta}>t\\}.$
(15)

We have that
$\sup_{\pi\in\mathbf{\Pi}^{s}}\sum_{t>k}\mathbb{P}_{\eta,\pi_{\mu}}\\{T_{\Delta}>t\\}$
as $k\rightarrow\infty$ since $\boldsymbol{\mathfrak{M}}(\eta,\Delta)$ is
uniformly absorbing by hypothesis. Moreover, the leftmost term in (15)
converges to zero as $k\rightarrow\infty$ by using Lemma 4.7 showing that the
limit (14) holds. This shows that $\boldsymbol{\mathcal{O}}_{\eta}$ is
relatively compact for the $ws$-topology.
$(b)\Rightarrow(a)$. Since $\boldsymbol{\mathcal{O}}_{\eta}$ is relatively
compact for the $ws$-topology, it follows from [4, Theorem 5.2] that the set
of $\mathbf{X}$-marginal measures of
$\boldsymbol{\mathcal{O}}_{\eta}=\boldsymbol{\mathcal{O}}_{\eta}^{s}$ is
relatively s-compact. Recalling that
$\mu^{\mathbf{X}}_{\eta,\pi}\ll\lambda^{\beta}$ for any
$\pi\in\mathbf{\Pi}^{s}$, using Lemma 4.2, Proposition 2.2 in [4], and
Corollary 2.7 in [12], we get that the family
$\\{h_{\pi}\\}_{\pi\in\mathbf{\Pi}^{s}}$ of density functions
$h_{\pi}=d\mu^{\mathbf{X}}_{\eta,\pi}/d\lambda^{\beta}$ is uniformly
$\lambda^{\beta}$-integrable. Now, observe that for $\pi\in\mathbf{\Pi}^{s}$,

$\sum_{k=t}^{\infty}\mathbb{P}_{\eta,\pi}\\{T_{\Delta}>k\\}=\sum_{j=0}^{\infty}\mathbb{P}_{\eta
Q_{\pi}^{j},\pi}\\{T_{\Delta}>t\\}=\mu^{\mathbf{X}}_{\eta,\pi}Q_{\pi}^{t}(\Delta^{c})=\int_{\mathbf{X}}Q_{\pi}^{t}(\Delta^{c}|x)h_{\pi}(x)\lambda^{\beta}(dx)$

and by using Proposition 4.4 we can conclude that the rightmost term in the
previous equation converges to zero uniformly in $\pi\in\mathbf{\Pi}^{s}$ as
$t\rightarrow\infty$. This establishes that
$\boldsymbol{\mathfrak{M}}(\eta,\Delta)$ is indeed uniformly absorbing.
$(b)\Leftrightarrow(c)$. This is obvious using Lemma 4.8 and recalling that
$\boldsymbol{\mathcal{O}}_{\eta}=\boldsymbol{\mathcal{O}}_{\eta}^{s}$ $\Box$

###### Example 4.10

Consider the model defined with state and actions spaces

$\mathbf{X}=\\{(i,j)\in\mathbb{N}\times\mathbb{N}:0\leq j\leq
2^{i}-1\\}\cup\\{\bar{x}\\}\quad\hbox{and}\quad\mathbf{A}=\\{c,s\\},$

with $\mathbf{A}(x)=\\{c,s\\}$ for $x\in\mathbb{N}\times\\{0\\}$ and
$\mathbf{A}(x)=\\{s\\}$ otherwise. The transition kernel is given by:

  * •

$Q\big{(}\bar{x}\mid(i,0),c)\big{)}=\frac{1}{2}$ and
$Q\big{(}(i+1,0)\mid(i,0),c)\big{)}=\frac{1}{2}$ for $i\in\mathbb{N}$;

  * •

$Q\big{(}(i,j+1)\mid(i,j),s\big{)}=1$ for $i\in\mathbb{N}$ and $0\leq
j<2^{i}-1$;

  * •

$Q\big{(}\bar{x}\mid(i,2^{i}-1),s\big{)}=1$ for $i\in\mathbb{N}$;

  * •

$Q(\bar{x}\mid\bar{x},s)=1$.

In [9, Example 3.13], it has been shown that this model is absorbing to
$\Delta=\\{\bar{x}\\}$ with initial distribution $\eta=\delta_{(0,0)}$ but not
uniformly absorbing. Our objective is to illustrate by means of this example
that the set of occupation measures $\boldsymbol{\mathcal{O}}_{\eta}$ is not
compact for the $ws$-topology, as can be derived from Theorem 4.9.

Let us consider the sequence $\\{\gamma_{t}\\}_{t\in\mathbb{N}}$ of
nonrandomized stationary policies in $\mathbf{\Pi}^{s}$ defined as follows.
The policy $\gamma_{t}(da|x)$ takes the action $c$ for all states $(i,0)$ for
$0\leq i<t$ and takes the action $s$ for all states $(i,0)$ with $i\geq t$. If
$\boldsymbol{\mathcal{O}}_{\eta}$ were relatively compact, then for any
decreasing sequence of sets $\Gamma_{n}\downarrow\emptyset$ we would have

$\lim_{n\rightarrow\infty}\sup_{t\in\mathbb{N}}\mu^{\mathbf{X}}_{\eta,\gamma_{t}}(\Gamma_{n})=0.$
(16)

Let us consider

$\Gamma_{n}=\big{\\{}(i,j)\in\mathbf{X}:i\geq n\text{ and }1\leq j\leq
2^{i}-1\\}\big{\\}}$

which indeed satisfies $\Gamma_{n}\downarrow\emptyset$. Observe that

$\mathbb{P}_{\eta,\gamma_{n}}\\{X_{n+j}=(n,j)\\}=\frac{1}{2^{n}}\quad\hbox{for
$n\geq 1$ and $j\in\\{0,\cdots,2^{n}-1\\}$}.$

Therefore,

$\sup_{t\in\mathbb{N}}\mu^{\mathbf{X}}_{\eta,\gamma_{t}}(\Gamma_{n})\geq\mu^{\mathbf{X}}_{\eta,\gamma_{n}}(\Gamma_{n})=2^{n}\cdot\frac{1}{2^{n}}=1,$

contradicting (16). This exhibits that, for this control model which is not
uniformly absorbing, the set of occupation measures
$\boldsymbol{\mathcal{O}}_{\eta}$ is not compact.

## References

  * [1] Charalambos D. Aliprantis and Kim C. Border.  Infinite dimensional analysis.  Springer, Berlin, third edition, 2006.  A hitchhiker’s guide.
  * [2] E. Altman.  Constrained Markov decision processes.  Stochastic Modeling. Chapman & Hall/CRC, Boca Raton, FL, 1999.
  * [3] Erik J. Balder.  A general approach to lower semicontinuity and lower closure in optimal control theory.  SIAM J. Control Optim., 22(4):570–598, 1984.
  * [4] Erik J. Balder.  On ws-convergence of product measures.  Math. Oper. Res., 26(3):494–518, 2001.
  * [5] Vladimir I. Bogachev.  Measure theory. Vol. I, II.  Springer-Verlag, Berlin, 2007.
  * [6] Vivek S. Borkar.  Convex analytic methods in Markov decision processes.  In Handbook of Markov decision processes, volume 40 of Internat. Ser. Oper. Res. Management Sci., pages 347–375. Kluwer Acad. Publ., Boston, MA, 2002.
  * [7] François Dufour and Tomás Prieto-Rumeau.  Nash equilibria for total expected reward absorbing Markov games: the constrained and unconstrained cases.  arXiv:2305.04514, 2023.
  * [8] E.B. Dynkin and A.A. Yushkevich.  Controlled Markov processes, volume 235 of Grundlehren der Mathematischen Wissenschaften.  Springer-Verlag, Berlin, 1979.
  * [9] Eugene A. Feinberg and Alexey Piunovskiy.  Sufficiency of deterministic policies for atomless discounted and uniformly absorbing MDPs with multiple criteria.  SIAM J. Control Optim., 57(1):163–191, 2019.
  * [10] Eugene A. Feinberg and Uriel G. Rothblum.  Splitting randomized stationary policies in total-reward Markov decision processes.  Math. Oper. Res., 37(1):129–153, 2012.
  * [11] Liviu C. Florescu and Christiane Godet-Thobie.  Young measures and compactness in measure spaces.  De Gruyter, Berlin, 2012.
  * [12] Peter Gänssler.  Compactness and sequential compactness in spaces of measures.  Z. Wahrscheinlichkeitstheorie und Verw. Gebiete, 17:124–146, 1971\.
  * [13] Onésimo Hernández-Lerma and Jean-Bernard Lasserre.  Discrete-time Markov control processes: Basic optimality criteria, volume 30 of Applications of Mathematics.  Springer-Verlag, New York, 1996.
  * [14] Onésimo Hernández-Lerma and Jean-Bernard Lasserre.  Further topics on discrete-time Markov control processes, volume 42 of Applications of Mathematics.  Springer-Verlag, New York, 1999.
  * [15] Anna Jaśkiewicz and Andrzej S. Nowak.  Non-Zero-Sum Stochastic Games, pages 1–64.  Springer International Publishing, Cham, 2017.
  * [16] Jacques Neveu.  Bases mathématiques du calcul des probabilités.  Masson et Cie, Éditeurs, Paris, 1970.  Préface de R. Fortet, Deuxième édition, revue et corrigée.
  * [17] A. B. Piunovskiy.  Optimal control of random sequences in problems with constraints, volume 410 of Mathematics and its Applications.  Kluwer Academic Publishers, Dordrecht, 1997.  With a preface by V. B. Kolmanovskii and A. N. Shiryaev.
  * [18] A. B. Piunovskiy.  Examples in Markov decision processes, volume 2 of Imperial College Press Optimization Series.  Imperial College Press, London, 2013.
  * [19] Manfred Schäl.  On dynamic programming: compactness of the space of policies.  Stochastic Processes Appl., 3(4):345–364, 1975.
  * [20] Michel Valadier.  Désintégration d’une mesure sur un produit.  C. R. Acad. Sci. Paris Sér. A, 276:33–35, 1973.