#  risk-sensitive discounted cost criterion for Continuous-time Markov
decision processes on a general state space

Subrata Golui  Department of Mathematics
Indian Institute of Technology Guwahati
Guwahati, Assam, India<EMAIL_ADDRESS>and  Chandan Pal  Department of
Mathematics
Indian Institute of Technology Guwahati
Guwahati, Assam, India<EMAIL_ADDRESS>
###### Abstract.

In this paper, we consider risk-sensitive discounted control problem for
continuous-time jump Markov processes taking values in general state space.
The transition rates of underlying continuous-time jump Markov processes and
the cost rates are allowed to be unbounded. Under certain Lyapunov condition,
we establish the existence and uniqueness of the solution to the Hamilton-
Jacobi-Bellman (HJB) equation. Also we prove the existence of optimal risk-
sensitive control in the class of Markov control.

Keywords: Continuous-time Markov decision process; history-dependent control;
general state space; risk-sensitive discounted criterion; HJB equation;
optimal control.

##  1\. INTRODUCTION

In this paper we study the risk-sensitive discounted criterion for continuous-
time Markov decision processes (CTMDPs) with Borel state space. Risk-sensitive
or exponential of integral’ is a very popular cost criterion due to its
applications in many areas such as queueing systems and finance, for more
details see [[1], [24]] and the references therein. In the literature risk-
sensitive control problems for CTMDPs are an important class of stochastic
optimal control problems and have been widely studied under different sets of
conditions. Finite horizon risk-sensitive CTMDPs for countable sate space were
studied in [[3], [5], [8], [23]] and for infinite horizon risk-sensitive
CTMDPs we refer to [[3], [11], [15], [16], [19], [25]]. For important
contributions to the risk-sensitive control of discrete time MDP on a general
state space, see [[17], [18]]. Although risk-sensitive control of CTMDPs on a
countable state space have been studied extensively, but the corresponding
literature in the context of risk sensitive control of CTMDPs on general state
space is rather limited. Some exceptions are [10],[19].

In the paper [19], the authors studied risk-sensitive control of pure jump
processes on general state space. They considered bounded transition and cost
rates and all controls are Markovian. In [19], authors proved a HJB
characterization of the optimal risk-sensitive control. The boundedness
assumption on transition and cost rates plays a key role in the proof of the
existence of the optimal risk-sensitive control in [19]. This boundedness
requirement, however, imposes some restrictions in applications, for instance
in queueing control and population processes, where the transition and
reward/cost rates are usually unbounded. In [10], the author considered the
finite-horizon risk-sensitive control problem for CTMDPs on Borel state space
with unbounded transition and cost rates and proved the existence of optimal
control via HJB equation.

In this paper we study a much more general problem. To the best of our
knowledge, this is the first work which deals with infinite horizon discounted
risk-sensitive control for CTMDPs on general state space with unbounded cost
and transition rates and the controls can be history-dependent. The main
objective of this work is to prove the existence of solution of the HJB
equation and characterization of optimal risk-sensitive control. We first
consider for bounded transition and cost rates, and establish the existence of
a solution to the corresponding HJB equation by Banach’s fixed point theorem
as in [19]. Then we will relax the bounded hypothesis and we extend this
result to unbounded transition and cost rates. We characterize the value
function via HJB equation. Also we prove the existence of an optimal control
in the class of Markov control and HJB characterization of the optimal risk-
sensitive control.

The rest of this article is structured as follows. Section 2 deals with the
description of the problem, required notations, some Assumptions, and
preliminary results. In Section 3, we gave continuity-compactness Assumption
and we prove the stochastic representation of the solution of the HJB equation
(3.3). In Section 4, we truncate our transition and cost rates and prove the
existence of the unique solution to the HJB equation. The required optimal
control is proven in Section 5. In Section 6, we illustrate our theory and
assumptions by an illustrative example.

##  2\. The control problems

The model of CTMDP is a five-tuple which consists of the following elements:

$\mathbb{M}:=\\{S,A,(A(x)\subset A,x\in S),c(x,a),q(\cdot|x,a)\\},$

  * •

a Borel space $S$, called the state space, whose elements are referred to as
states of the system and the corresponding Borel $\sigma$-algebra is
$\mathcal{B}(S)$.

  * •

$A$ is the action set, which is assumed to be Borel space with the Borel
$\sigma$-algebra $\mathcal{B}(A)$.

  * •

for each $x\in S$, $A(x)\in\mathcal{B}(A)$ denotes the set of admissible
actions for state $x$. Let $K:=\\{(x,a)|x\in S,a\in A(x)\\}$, which is a Borel
subset of $S\times A$.

  * •

the measurable function $c:K\to\mathbb{R}_{+}$ denotes the cost rate function.
We require cost function $c(x,a)$ to measure (or evaluate) the utility of
taking action $a$ at state $x$.

  * •

given any $(x,a)\in K$, the transition rate $q(\cdot|x,a)$ is a Borel
measurable signed kernel on $S$ given $K$. That is, $q(\cdot|x,a)$ satifies
countable additivity; $q(D|x,a)\geq 0$ where $(x,a)\in K$ and $x\notin D$.
Moreover, we assume that $q(\cdot|x,a)$ satisfies the following conservative
and stable conditions: for any $x\in S,$

$\displaystyle q(S|x,a)\equiv 0~{}~{}~{}\text{and}$
$\displaystyle~{}q^{*}(x):=\sup_{a\in A(x)}q_{x}(a)<\infty,$

where $q_{x}(a):=-q(\\{x\\}|x,a)\geq 0.$ We need transition rates to specify
the random dynamic evolution of the system.

Next, we give an informal description of the evolution of the CTMDPs as
follows. The controller observes continuously the current state of the system.
When the system is in state $x\in S$ at time $t\geq 0$, he/she chooses action
$a_{t}\in A(x)$ according to some control. As a consequence of this, the
following happens:

  * •

the controller incurs an immediate cost at rate $c(x,a_{t})$; and

  * •

after a random sojourn time (i.e., the holding time at state $x$), the system
jumps to a set $B$ ($x\notin B$) of states with the transition probability
$\dfrac{q(B|x,a_{t})}{q_{x}(a_{t})}$ determined by the transition rates
$q(dy|x,a_{t})$. The distribution function of the sojourn time is
$(1-e^{-\int_{t}^{t+x}q_{x}(a_{s})ds})$. (see Proposition B.8 in [[4], p. 205]
for details).

When the state of the system transits to the new state $y\neq x$, the above
procedure is repeated. Thus, the controller tries to minimize his/her costs
with respect to some performance criterion
$\mathscr{J}_{\alpha}(\cdot,\cdot,\cdot)$, which in our present case is
defined by (2.2), below. To formalize what is described above, below we
describe the construction of continuous time Markov decision processes
(CTMDPs) under possibly history-dependent controls. To construct the
underlying CTMDPs (as in [[9], [13], [20]], [21]) we introduce some notations:
let $S_{\Delta}:=S\cup\\{\Delta\\}$ (with some $\Delta\notin S$),
$\Omega_{0}:=(S\times(0,\infty))^{\infty}$,
$\Omega_{k}:=(S\times(0,\infty))^{k}\times
S\times(\\{\infty\\}\times\\{\Delta\\})^{\infty}$ for $k\geq 1$ and
$\Omega:=\cup_{k=0}^{\infty}\Omega_{k}$. Let $\mathscr{F}$ be the Borel
$\sigma$-algebra on $\Omega$. Then we obtain the measurable space
$(\Omega,\mathscr{F})$. For some $k\geq 1$, and sample
$\omega:=(x_{0},\theta_{1},x_{1},\cdots,\theta_{k},x_{k},\cdots)\in\Omega,$
define

$\displaystyle
X_{0}(\omega):=x_{0},~{}T_{0}(\omega):=0,~{}X_{k}(w):=x_{k},~{}T_{n}(\omega):=T_{n-1}(\omega)+\theta_{n},~{}T_{\infty}(\omega):=\lim_{n\rightarrow\infty}T_{n}(\omega).$

Using $\\{T_{k}\\}$, we define the state process $\\{\xi_{t}\\}_{t\geq 0}$ as

$\xi_{t}(\omega):=\sum_{k\geq 0}I_{\\{T_{k}\leq t<T_{k+1}\\}}x_{k}+I_{\\{t\geq
T_{\infty}\\}}\Delta,\text{ for }t\geq 0~{}(\text{with}~{}T_{0}:=0).$ (2.1)

Here, $I_{E}$ denotes the indicator function of a set $E$, and we use the
convention that $0+z=:z$ and $0z=:0$ for all $z\in S_{\Delta}$. Obviously,
$\xi_{t}(\omega)$ is right-continuous on $[0,\infty)$. We denote
$\xi_{t-}(\omega):=\liminf_{s\rightarrow t-}\xi_{s}(\omega)$. From eq. (2.1),
we see that $T_{k}(\omega)$ $(k\geq 1)$ denotes the $k$-th jump moment of
$\\{\xi_{t},t\geq 0\\}$, $X_{k-1}(\omega)=x_{k-1}$ is the state of the process
on $[T_{k-1}(\omega),T_{k}(\omega))$,
$\theta_{k}=T_{k}(\omega)-T_{k-1}(\omega)$ plays the role of sojourn time at
state $x_{k-1}$, and the sample path $\\{\xi_{t}(\omega),t\geq 0\\}$ has at
most denumerable states $x_{k}(k=0,1,\cdots)$. The process after $T_{\infty}$
is regarded to be absorbed in the state $\Delta$. Thus, let
$q(\cdot|\Delta,a_{\Delta}):\equiv 0$, $A_{\Delta}:=A\cup\\{a_{\Delta}\\}$,
$A(\Delta):=\\{a_{\Delta}\\}$, $c(\Delta,a):\equiv 0$ for all $a\in
A_{\Delta}$, where $a_{\Delta}$ is isolated point.
To precisely define the criterion, we need to introduce the concept of a
control as in [[9],[6] and [14]]. Take the right-continuous $\sigma$-algebras
$\\{\mathscr{F}_{t}\\}_{t\geq 0}$ with $\mathscr{F}_{t}:=\sigma(\\{T_{k}\leq
s,X_{k}\in B\\}:B\in\mathcal{B}(S),0\leq s\leq t,k\geq 0)$. For all $t\geq 0$,
$\mathscr{F}_{s-}=:\bigvee_{0\leq t<s}\mathscr{F}_{t}$, and
$\mathscr{P}:=\sigma(\\{A\times\\{0\\},A\in\mathscr{F}_{0}\\}\cup\\{B\times(s,\infty),B\in\mathscr{F}_{s-}\\})$
which denotes the $\sigma$-algebra of predictable sets on
$\Omega\times[0,\infty)$ related to $\\{\mathscr{F}_{t}\\}_{t\geq 0}$. To
complete the specification of a stochastic optimal control problem, we need,
of course, to introduce an optimality criterion. This requires to define the
class of controls as below.

######  Definition 2.1.

A transition probability $\pi(da|\omega,t)$ from
$(\Omega\times[0,\infty),\mathscr{P})$ onto
$(A_{\Delta},\mathcal{B}(A_{\Delta}))$ such that
$\pi(A(\xi_{t-}(\omega)|\omega,t)\equiv 1$ is called a history-dependent
control. The set of all randomized history-dependent controls is denoted by
$\Pi$. A control $\pi\in\Pi$, is called a Markov if
$\pi(da|\omega,t)=\pi(da|\xi_{t-}(w),t)$ for every $w\in\Omega$ and $t\geq 0$,
where $\xi_{t-}(w):=\lim_{s\uparrow t}\xi_{s}(w)$. We denote by $\Pi^{m}$ the
family of all Markov controls. A Markov control $\pi_{t}(da|\cdot)$ is called
a deterministic Markov control whenever there exists a measurable mapping
$f:[0,\infty)\times S\rightarrow A$ such that
$\pi(da|t,x)=I_{\\{f(t,x)\\}}(da)$, which means that $\pi(da|t,x)$ is a Dirac
measure at $f(t,x)$ for every $x\in S$ and $t\geq 0$. Such a Markov control
will be denoted by $f$ for simplicity. The set of such controls is denoted by
$\Pi^{d}_{m}$.

For any compact metric space $Y$, let $P(Y)$ denote the space of probability
measures on $Y$ with Prohorov topology. Under Assumption 2.1 below, for any
initial state $x\in S$ and any control $\pi\in\Pi$, Theorem 4.27 in [14]
yields the existence of a unique probability measure denoted by $P^{\pi}_{x}$
on $(\Omega,\mathscr{F})$. Let $E^{\pi}_{x}$ be the expectation operator with
respect to $P^{\pi}_{x}$. Fix any discounted factor $\alpha>0$. For any
$\pi\in\Pi$ and $x\in S$, the risk-sensitive discounted criterion is defined
as

$\displaystyle\mathscr{J}_{\alpha}(\theta,x,\pi):=\frac{1}{\theta}\log\biggl{\\{}{E}^{\pi}_{x}\biggl{[}exp\biggl{(}\theta{\int_{0}^{\infty}e^{-\alpha
t}\biggl{(}\int_{A}c(\xi_{t}(\omega),a)\pi(da|\omega,t)\biggr{)}dt}\biggr{)}\biggr{]}\biggr{\\}},$
(2.2)

provided that the integral is well defined, where $\xi_{t}$ is the Markov
process corresponding to $\pi\in\Pi$ and $\theta\in(0,1]$ denotes a risk-
sensitive parameter and the limiting case of $\theta\rightarrow 0$ is the
risk-neutral case. For each $x\in S$, let

$\mathscr{J}^{*}_{\alpha}(\theta,x)=\inf_{\pi\in\Pi}\mathscr{J}_{\alpha}(\theta,x,\pi).$

A control $\pi^{*}\in\Pi$ is said to be optimal if
$\mathscr{J}_{\alpha}(\theta,x,\pi^{*})=\mathscr{J}^{*}_{\alpha}(\theta,x)$
for all $x\in S$. The objective of this paper is to provide conditions for the
existence of optimal control and introduce a HJB characterization of such
control.
Since logarithm is an increasing function, instead of studying
$\mathscr{J}_{\alpha}(\theta,x,\pi)$, we will consider
$\tilde{J}_{\alpha}(\theta,x,\pi)$ on $[0,1]\times S\times\Pi$ defined by

$\displaystyle\tilde{J}_{\alpha}(\theta,x,\pi):={E}^{\pi}_{x}\biggl{[}exp\biggl{(}\theta{\int_{0}^{\infty}e^{-\alpha
t}\biggl{(}\int_{A}c(\xi_{t}(\omega),a)\pi(da|\omega,t)\biggr{)}dt}\biggr{)}\biggr{]}.$
(2.3)

Obviously, $\tilde{J}_{\alpha}(\theta,x,\pi)\geq 1$ for
$(\theta,x)\in[0,1]\times S$ and $\pi\in\Pi$, and we have $\pi^{*}$ is optimal
if and only if
$\inf_{\pi\in\Pi}\tilde{J}_{\alpha}(\theta,x,\pi)=\tilde{J}(\theta,x,\pi^{*})=:\tilde{J}^{*}_{\alpha}(\theta,x)~{}\forall
x\in S.$ Since the rates $q(dy|x,a)$ and costs $c(x,a)$ are allowed to be
unbounded, we next give conditions for the non-explosion of $\\{\xi_{t},t\geq
0\\}$ and finiteness of $\mathscr{J}_{\alpha}(\theta,x,\pi)$, which had been
widely used in CTMDPs; see, for instance, [ [4], [6], [7], [9] and [22]] and
reference therein.

######  Assumption 2.1.

There exists a real-valued Borel measurable function $V_{0}\geq 1$ on $S$ and
constants $\rho_{0}>0$, $M_{0}>0$, $L_{0}\geq 0$ and
$0<\rho_{1}<\min\\{\alpha,\rho^{-1}_{0}\alpha^{2}\\}$ such that

  1. (i)

$\int_{S}V_{0}(y)q(dy|x,a)\leq\rho_{0}V_{0}(x)~{}~{}~{}\forall(x,a)\in K$;

  2. (ii)

$\sup_{a\in A(x)}q_{x}(a)\leq M_{0}V_{0}(x)~{}~{}~{}\forall x\in S$;

  3. (iii)

$\sup_{a\in A(x)}c(x,a)\leq\rho_{1}\log V_{0}(x)+L_{0}~{}~{}~{}\forall x\in
S.$

######  Proposition 2.1.

Under Assumption 2.1, for any control $\pi\in\Pi$ and
$(\theta,x)\in[0,1]\times S$, the following results are true:

  1. (a)

${P}^{\pi}_{x}(T_{\infty}=\infty)=1$, $P^{\pi}_{x}(\xi_{0}=x)=1$, and
${P}^{\pi}_{x}(\xi_{t}\in S)=1$ for all $t\geq 0$;

  2. (b)

${E}^{\pi}_{x}[V_{0}(\xi_{t})]\leq e^{\rho_{0}t}V_{0}(x)$ for all $t\geq 0;$

  3. (c)

We have

$\displaystyle\tilde{J}_{\alpha}(\theta,x,\pi)\leq\frac{\alpha^{2}}{\alpha^{2}-\rho_{0}\rho_{1}\theta}e^{{\theta
L_{0}}/{\alpha}}[V_{0}(x)]^{\frac{\rho_{1}\theta}{\alpha}}\leq\frac{\alpha^{2}}{\alpha^{2}-\rho_{0}\rho_{1}}e^{{L_{0}}/{\alpha}}V_{0}(x).$

Also, we get

$\displaystyle\mathscr{J}^{*}_{\alpha}(\theta,x)\leq\log\biggl{(}{\frac{\alpha^{2}}{\alpha^{2}-\rho_{0}\rho_{1}}}\biggr{)}+\frac{L_{0}}{\alpha}+\frac{\rho_{1}}{\alpha}\log{V_{0}(x)}~{}~{}\forall\theta\in(0,1],x\in
S.$ (2.4)

###### Proof.

For parts $(a)$ and $(b)$, see, [6] and ( [9], Theorem 3.1 ).
Proof of part (c): Let $\beta(dt)=\alpha e^{-\alpha t}dt$, which is a
probability measure on $[0,\infty).$ For any $\pi\in\Pi$ and
$(\theta,x)\in[0,1]\times S$, by (2.3) and Jensen’s inequality we have

$\displaystyle\tilde{J}_{\alpha}(\theta,x,\pi)$
$\displaystyle={E}^{\pi}_{x}\biggl{[}exp\biggl{(}\int_{0}^{\infty}\frac{\theta}{\alpha}\int_{A}c(\xi_{t}(\omega),a)\pi(da|\omega,t)\beta(dt)\biggr{)}\biggr{]}$
$\displaystyle\leq{E}^{\pi}_{x}\biggl{[}\int_{0}^{\infty}exp\biggl{(}\frac{\theta}{\alpha}\int_{A}c(\xi_{t}(\omega),a)\pi(da|\omega,t)\biggr{)}\beta(dt)\biggr{]}.$

By Assumption 2.1 and part (b) we obtain

$\displaystyle\tilde{J}_{\alpha}(\theta,x,\pi)$
$\displaystyle\leq{E}^{\pi}_{x}\biggl{[}\int_{0}^{\infty}exp\biggl{(}\frac{\theta}{\alpha}(\rho_{1}\log{V_{0}(\xi_{t})}+L_{0})\biggr{)}\beta(dt)\biggr{]}$
$\displaystyle=e^{{\theta
L_{0}}/{\alpha}}\biggl{[}\int_{0}^{\infty}{E}^{\pi}_{x}\biggl{(}V_{0}(\xi_{t})^{\frac{\rho_{1}\theta}{\alpha}}\biggr{)}\beta(dt)\biggr{]}$
$\displaystyle\leq e^{{\theta
L_{0}}/{\alpha}}\biggl{[}\int_{0}^{\infty}({E}^{\pi}_{x}[V_{0}(\xi_{t})])^{\frac{\rho_{1}\theta}{\alpha}}\beta(dt)\biggr{]}~{}~{}~{}(\text{since}~{}~{}\rho_{1}\theta<\alpha)$
$\displaystyle\leq\alpha e^{{\theta
L_{0}}/{\alpha}}[V_{0}(x)]^{\frac{\rho_{1}\theta}{\alpha}}\biggl{[}\int_{0}^{\infty}exp\biggl{(}\frac{\rho_{0}\rho_{1}\theta
t}{\alpha}-\alpha t\biggr{)}dt\biggr{]}$
$\displaystyle={\frac{\alpha^{2}}{\alpha^{2}-\rho_{0}\rho_{1}\theta}}e^{{\theta
L_{0}}/{\alpha}}[V_{0}(x)]^{\frac{\rho_{1}\theta}{\alpha}}~{}~{}~{}(\text{since}~{}~{}\rho_{0}\rho_{1}\theta<\alpha^{2}).$

Hence, we have
$\sup_{\theta\in[0,1]}\tilde{J}^{*}_{\alpha}(\theta,x)\leq\frac{\alpha^{2}}{\alpha^{2}-\rho_{0}\rho_{1}}e^{{L_{0}}/{\alpha}}V_{0}(x)$,
and

$\displaystyle\sup_{\theta\in(0,1]}\mathscr{J}^{*}_{\alpha}(\theta,x)=\sup_{\theta\in(0,1]}\frac{1}{\theta}\log{\tilde{J}^{*}_{\alpha}(\theta,x)}\leq\sup_{\theta\in(0,1]}\frac{1}{\theta}\biggl{(}\log{\frac{\alpha^{2}}{\alpha^{2}-\rho_{0}\rho_{1}\theta}}\biggr{)}+\frac{L_{0}}{\alpha}+\frac{\rho_{1}}{\alpha}\log
V_{0}(x).$

Now, by a direct calculation, one can show that (2.4) holds. ∎

Here we assume the following conditions, so that we can apply the Feynman-Kac
formula formula for a large enough class of functions, which had been widely
used in CTMDPs; see, for instance, [[5], [7], [8], [10], [23]].

######  Assumption 2.2.

There exist a Borel measurable function $V_{1}\geq 1$ on $S$, and constants
$0<\rho_{2}<\alpha$, $b_{1}\geq 0$, $M_{1}\geq 1$ such that

  1. (i)

$\int_{S}q(dy|x,a)V^{2}_{1}(y)\leq\rho_{2}V^{2}_{1}(x)+b_{1}~{}~{}\forall~{}(x,a)\in
K$,

  2. (ii)

$V_{0}^{2}(x)\leq M_{1}V_{1}(x)~{}~{}\forall~{}x\in S$ where $V_{0}$ is
introduced in Assumption 2.1.

We now introduce some frequently used notations.

  * •

For any Borel space $X$, $\mathcal{B}(X)$ denotes the corresponding Borel
$\sigma$-algebra.

  * •

$C^{\infty}_{c}(a,b)$ denotes the set of all infinitely differentiable
functions on $(a,b)$ with compact support.

  * •

Let $A_{c}([0,1]\times S)$ denote the space of all functions which are real-
valued and differentiable almost everywhere with respect to
$\theta\in[0,1]$eg. When the partial derivative (with respect to
$\theta\in[0,1]$) does not exist for some $(\theta,x)\in[0,1]\times S$, we
take $\frac{\partial\varphi_{\alpha}}{\partial\theta}(\theta,x)$ to be any
real number, and so
$\frac{\partial\varphi_{\alpha}}{\partial\theta}(\cdot,\cdot)$ is defined on
$[0,1]\times S$. Given any real-valued function $W\geq 1$ on $S$ and any Borel
set $X$, a real-valued function $u$ on $X\times S$ is called $W$ bounded if
$\|u\|^{\infty}_{W}:=\sup_{(\theta,x)\in X\times
S}\frac{|u(\theta,x)|}{W(x)}<\infty$. Denote $B_{W}(X\times S)$ the Banach
space of all $W$-bounded functions. When $W\equiv 1$, $B_{1}([0,1]\times S)$
is the space of all bounded functions on $[0,1]\times S.$
Now define $B^{1}_{W_{0},W_{1}}([0,1]\times S):=\\{\varphi_{\alpha}\in
B_{W_{0}}([0,1]\times S)\cap A_{c}([0,1]\times
S):\frac{\partial\varphi_{\alpha}}{\partial\theta}\in B_{W_{1}}([0,1]\times
S)\\}$.

##  3\. stochastic representation of a solution to the HJB equation

In this section, we prove that if the HJB equation for the cost criterion
(2.3) has a solution then we will give a stochastic representation of that
solution. Using dynamic programming heuristics, the HJB equations for the
discounted cost criterion (2.3) is given by

$\displaystyle\left\\{\begin{array}[]{ll}\alpha\theta\frac{\partial\varphi_{\alpha}}{\partial\theta}(\theta,x)&=\displaystyle{\inf_{a\in
A(x)}\biggl{[}\int_{S}q(dy|x,a)\varphi_{\alpha}(\theta,y)+\theta
c(x,a)\varphi_{\alpha}(\theta,x)\biggr{]}},\\\
&1\leq\varphi_{\alpha}(\theta,x)\leq{\frac{\alpha^{2}}{\alpha^{2}-\rho_{0}\rho_{1}\theta}}e^{{\theta
L_{0}}/{\alpha}}(V_{0}(x))^{\frac{\rho_{1}\theta}{\alpha}}~{}~{}\text{for}~{}~{}(\theta,x)\in[0,1]\times
S,\end{array}\right.$ (3.3)

for each $x\in S$ and a.e. $\theta\in[0,1]$ where the upper bound of
$\varphi_{\alpha}(\theta,x)$ is inspired by Proposition 2.1.

To ensure the existence of optimal control, in addition to Assumptions 2.1 and
2.2, we also need the following continuity and compactness conditions.

######  Assumption 3.1.

The following conditions hold:

  1. (i)

for each $x\in S$, the set $A(x)$ is compact;

  2. (ii)

for any fixed $x\in S$, $q(\cdot|x,a)$ and $c(x,a)$ are continuous in $a\in
A(x)$;

  3. (iii)

for any given $x\in S$, the function $\displaystyle\int_{S}V_{0}(y)q(dy|x,a)$
is continuous in $a\in A(x)$, where $V_{0}$ is introduced in Assumption 2.1.

In the next theorem we show that if the HJB equation has a solution then its
stochastic representation is equal to the value function corresponding to the
cost criterion (2.3).

######  Theorem 3.1.

Under Assumptions 2.1, 2.2, and 3.1 suppose that the HJB equation (3.3) has a
solution $\varphi_{\alpha}\in B^{1}_{V_{0},V_{1}}([0,1]\times S)$ satisfying
the bounds. Then, for all $(\theta,x)\in[0,1]\times S$, we have the
probabilistic representation of $\varphi_{\alpha}$ as

$\displaystyle\varphi_{\alpha}(\theta,x)=\inf_{\pi\in\Pi}E^{\pi}_{x}\biggl{[}exp\biggl{(}\theta\int_{0}^{\infty}\int_{A}e^{-\alpha
t}c(\xi_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]}$ (3.4)

which means $\varphi_{\alpha}(\theta,x)=\tilde{J}^{*}_{\alpha}(\theta,x)$ for
all $(\theta,x)\in[0,1]\times S$.

###### Proof.

First we see that

$\displaystyle\biggl{[}\theta
c(x,a)\varphi_{\alpha}(\theta,x)+\int_{S}q(dy|x,a)\varphi_{\alpha}(\theta,y)\biggr{]}$

is continuous in $a\in A(x)$ and $A(x)$ is compact. So by measurable selection
theorem, [[2],Proposition 7.33], there exists a measurable function
$f^{*}:[0,1]\times S\rightarrow A$ such that

$\displaystyle\inf_{a\in A(x)}\biggl{[}\theta
c(x,a)\varphi_{\alpha}(\theta,x)+\int_{S}q(dy|x,a)\varphi_{\alpha}(\theta,y)\biggr{]}$
$\displaystyle=\biggl{[}\theta
c(x,f^{*}(\theta,x))\varphi_{\alpha}(\theta,x)+\int_{S}q(dy|x,f^{*}(\theta,x))\varphi_{\alpha}(\theta,y)\biggr{]}.$
(3.5)

Let

$\pi^{*}:\mathbb{R}_{+}\times S\to P(A)\ $

be defined by

$\displaystyle\pi^{*}(\cdot|t,x)=I_{\\{\hat{f}^{*}(t,x)\\}}(\cdot),~{}\text{where}~{}\hat{f}^{*}:[0,1]\times
S\rightarrow A,$ $\displaystyle\quad~{}\text{be a measurable mapping, defined
by}~{}\hat{f}^{*}(t,x):=f^{*}(\theta e^{-\alpha t},x).$

When $\frac{\partial\varphi_{\alpha}}{\partial\theta}$ does not exist for some
$(\theta,x)$, we define

$\displaystyle\alpha\theta\frac{\partial\varphi_{\alpha}}{\partial\theta}(\theta,x)$
$\displaystyle=\inf_{a\in A(x)}\displaystyle\biggl{[}\theta
c(x,a)\varphi_{\alpha}(\theta,x)+\int_{S}q(dy|x,a)\varphi_{\alpha}(\theta,y)\biggr{]}.$

Then we observe from equation (3.3) that for any $(\theta,x)\in[0,1]\times S$
and $a\in A(x)$ that

$\displaystyle-\alpha\theta\frac{\partial\varphi_{\alpha}}{\partial\theta}(\theta,x)+\displaystyle{\biggl{[}\int_{S}q(dy|x,a)\varphi_{\alpha}(\theta,y)+\theta
c(x,a)\varphi_{\alpha}(\theta,x)\biggr{]}}\geq 0.$ (3.6)

For any history-dependent control $\pi\in\Pi$ and $\theta\in[0,1]$, let
$\\{\xi_{t},t\geq 0\\}$ be the corresponding process, and define
$\theta(t):=\theta e^{-\alpha t}$. Now for each $\omega\in\Omega$, by equation
(3.6), we get

$\displaystyle-\alpha\theta(s)$
$\displaystyle\frac{\partial\varphi_{\alpha}}{\partial\theta}(\theta(s),\xi_{s}(\omega))$
$\displaystyle+\displaystyle{\biggl{[}\int_{S}\int_{A}q(dy|\xi_{s}(\omega),a)\varphi_{\alpha}(\theta(s),y)\pi(da|\omega,s)+\theta(s)\int_{A}c(\xi_{s}(\omega),a)\varphi_{\alpha}(\theta(s),\xi_{s}(\omega))\pi(da|\omega,s)\biggr{]}}\geq
0$ (3.7)

and define $g:[0,\infty)\times S\times\Omega\to[0,\infty)$ by

$g(t,x,\omega):=exp\biggl{(}\int_{0}^{t}\int_{A}\theta(s)c(\xi_{s}(\omega),a)\pi(da|\omega,s)ds\biggr{)}\varphi_{\alpha}(\theta(t),x).$

Let $\beta(ds):=\alpha e^{-\alpha s}ds$. Then under Assumptions 2.1 and 2.2,
we have

$\displaystyle E^{\pi}_{x}\biggl{[}exp\biggl{(}\int_{0}^{t}\int_{A}2e^{-\alpha
s}c(\xi_{s},a)\pi(da|\omega,s)ds\biggr{)}\biggr{]}$ $\displaystyle\leq
E^{\pi}_{x}\biggl{[}exp\biggl{(}\int_{0}^{\infty}\frac{2}{\alpha}\int_{A}c(\xi_{s},a)\pi(da|\omega,s)\beta(ds)\biggr{)}\biggr{]}$
$\displaystyle\leq
E^{\pi}_{x}\biggl{[}\int_{0}^{\infty}exp\biggl{(}\frac{2}{\alpha}\int_{A}c(\xi_{s},a)\pi(da|\omega,s)\biggr{)}\beta(ds)\biggr{]}$
$\displaystyle~{}(\text{by Jensen's inequality})$ $\displaystyle\leq
E^{\pi}_{x}\biggl{[}\int_{0}^{\infty}exp\biggl{(}\frac{2}{\alpha}(\rho_{1}\log{V_{0}(\xi_{s})}+L_{0})\biggr{)}\beta(ds)\biggr{]}$
$\displaystyle~{}(\text{by Assumption \ref{assm 2.1}})$ $\displaystyle\leq
e^{{2L_{0}}/{\alpha}}\biggl{[}\int_{0}^{\infty}E^{\pi}_{x}\biggl{(}V_{0}(\xi_{s})^{\frac{4\rho_{1}}{\alpha}}\biggr{)}\beta(ds)\biggr{]}$
$\displaystyle\leq
e^{{2L_{0}}/{\alpha}}M_{1}^{2}\biggl{[}\int_{0}^{\infty}E^{\pi}_{x}\biggl{(}V_{1}(\xi_{s})^{\frac{2\rho_{1}}{\alpha}}\biggr{)}\beta(ds)\biggr{]}$
$\displaystyle\leq\alpha
e^{{2L_{0}}/{\alpha}}M_{1}^{2}\biggl{(}V_{1}^{2}(x)+\frac{b_{1}}{\rho_{2}}\biggr{)}\biggl{[}\int_{0}^{\infty}e^{\rho_{2}s-\alpha
s}ds\biggr{]}$ $\displaystyle\leq\frac{\alpha
e^{{2L_{0}}/{\alpha}}}{\alpha-\rho_{2}}M_{1}^{2}\biggl{(}V_{1}^{2}(x)+\frac{b_{1}}{\rho_{2}}\biggr{)}.$
(3.8)

Now using Assumptions 2.1 and 2.2, we obtain

$\displaystyle\biggl{|}\biggl{[}-\alpha\theta(s)\frac{\partial\varphi_{\alpha}}{\partial\theta}(\theta(s),\xi_{s}(\omega))+\int_{S}\int_{A}q(dy|\xi_{s}(\omega),a)\varphi_{\alpha}(\theta(s),y)\pi(da|\omega,s)$
$\displaystyle\quad+\theta(s)\int_{A}c(\xi_{s}(\omega),a)\varphi_{\alpha}(\theta(s),\xi_{s}(\omega))\pi(da|\omega,s)\biggr{]}\biggr{|}$
$\displaystyle\leq\biggl{[}\alpha\biggl{\|}\frac{\partial\varphi_{\alpha}}{\partial\theta}\biggr{\|}_{V_{1}}V_{1}(\xi_{s})+\|\varphi_{\alpha}\|_{V_{0}}\biggl{(}\int_{S}\int_{A}q(dy|\xi_{s}(\omega),a)V_{0}(y)\pi(da|\omega,s)+2V_{0}^{2}(\xi_{s})M_{0}\biggr{)}$
$\displaystyle\quad+\|\varphi_{\alpha}\|_{V_{0}}(\rho_{1}\log
V_{0}(\xi_{s})+L_{0})V_{0}(\xi_{s})\biggr{]}$
$\displaystyle\leq\biggl{[}\alpha\biggl{\|}\frac{\partial\varphi_{\alpha}}{\partial\theta}\biggr{\|}_{V_{1}}V_{1}(\xi_{s})+\|\varphi_{\alpha}\|_{V_{0}}(\rho_{0}V_{0}(\xi_{s})+2V_{0}^{2}(\xi_{s})M_{0})+(\rho_{1}V_{0}(\xi_{s})+L_{0})\|\varphi_{\alpha}\|_{V_{0}}V_{0}(\xi_{s})\biggr{]}$
$\displaystyle\leq\biggl{[}\alpha\biggl{\|}\frac{\partial\varphi_{\alpha}}{\partial\theta}\biggr{\|}_{V_{1}}V_{1}(\xi_{s})+\|\varphi_{\alpha}\|_{V_{0}}(\rho_{0}+2M_{0})M_{1}V_{1}(\xi_{s})+(\rho_{1}+L_{0})M_{1}V_{1}(\xi_{s})\|\varphi_{\alpha}\|_{V_{0}}\biggr{]}$
$\displaystyle=\biggl{[}\alpha\biggl{\|}\frac{\partial\varphi_{\alpha}}{\partial\theta}\biggr{\|}_{V_{1}}+\|\varphi_{\alpha}\|_{V_{0}}(\rho_{0}+2M_{0}+\rho_{1}+L_{0})M_{1}\biggr{]}V_{1}(\xi_{s}).$
(3.9)

Now by (3.8), we get

$\displaystyle\biggl{|}E^{\pi}_{x}\biggl{[}exp\biggl{(}\int_{0}^{s}\int_{A}\theta(v)c(\xi_{v}(\omega),a)\pi(da|\omega,v)dv\biggr{)}V_{1}(\xi_{s})\biggr{]}\biggr{|}$
$\displaystyle\leq\biggl{|}E^{\pi}_{x}\biggl{[}exp\biggl{(}\int_{0}^{s}\int_{A}2\theta(v)c(\xi_{v}(\omega),a)\pi(da|\omega,v)dv\biggr{)}\biggr{]}E^{\pi}_{x}[V^{2}_{1}(\xi_{s})]\biggr{|}$
$\displaystyle\leq\frac{\alpha
e^{{2L_{0}}/{\alpha}}}{\alpha-\rho_{2}}M_{1}^{2}\biggl{(}V_{1}^{2}(x)+\frac{b_{1}}{\rho_{2}}\biggr{)}e^{\rho_{2}s}\biggl{(}V_{1}^{2}(x)+\frac{b_{1}}{\rho_{2}}\biggr{)}.$

Therefore we have

$\displaystyle\biggl{|}E^{\pi}_{x}\biggl{[}\int_{0}^{t}exp\biggl{(}\int_{0}^{s}\int_{A}\theta(v)c(\xi_{v}(\omega),a)\pi(da|\omega,v)dv\biggr{)}V_{1}(\xi_{s})ds\biggr{]}\biggr{|}$
$\displaystyle\leq\frac{\alpha
e^{{2L_{0}}/{\alpha}}}{\alpha-\rho_{2}}M_{1}^{2}\biggl{(}V_{1}^{2}(x)+\frac{b_{1}}{\rho_{2}}\biggr{)}\biggl{(}V_{1}^{2}(x)+\frac{b_{1}}{\rho_{2}}\biggr{)}\int_{0}^{t}e^{\rho_{2}s}ds$
$\displaystyle\leq\frac{\alpha
e^{{2L_{0}}/{\alpha}}}{\alpha-\rho_{2}}M_{1}^{2}\biggl{(}V_{1}^{2}(x)+\frac{b_{1}}{\rho_{2}}\biggr{)}^{2}\frac{e^{\rho_{2}t}}{\rho_{2}}<\infty.$
(3.10)

So by (3.9) and (3.10), we say

$\displaystyle\biggl{|}E^{\pi}_{x}\biggl{\\{}\int_{0}^{t}exp\biggl{(}\int_{0}^{s}\int_{A}\theta(v)c(\xi_{v}(\omega),a)\pi(da|\omega,v)dv\biggr{)}$
$\displaystyle\times\biggl{[}-\alpha\theta(s)\frac{\partial\varphi_{\alpha}}{\partial\theta}(\theta(s),\xi_{s}(\omega))+\int_{S}\int_{A}q(dy|\xi_{s}(\omega),a)\varphi_{\alpha}(\theta(s),y)\pi(da|\omega,s)$
$\displaystyle+\theta(s)\int_{A}c(\xi_{s}(\omega),a)\varphi_{\alpha}(\theta(s),\xi_{s}(\omega))\pi(da|\omega,s)\biggr{]}ds\biggr{\\}}\biggr{|}<\infty.$

Thus, using the extension of Feynman-Kac formula in [ [8], Theorem 3.1] to the
function $g$, we have

$\displaystyle
E^{\pi}_{x}[g(t,\xi_{t}(\omega),\omega)]-\varphi_{\alpha}(\theta,x)$
$\displaystyle=E^{\pi}_{x}\biggl{\\{}\int_{0}^{t}exp\biggl{(}\int_{0}^{s}\int_{A}\theta(v)c(\xi_{v}(\omega),a)\pi(da|\omega,v)dv\biggr{)}$
$\displaystyle\times\biggl{[}-\alpha\theta(s)\frac{\partial\varphi_{\alpha}}{\partial\theta}(\theta(s),\xi_{s}(\omega))+\int_{S}\int_{A}q(dy|\xi_{s}(\omega),a)\varphi_{\alpha}(\theta(s),y)\pi(da|\omega,s)$
$\displaystyle+\theta(s)\int_{A}c(\xi_{s}(\omega),a)\varphi_{\alpha}(\theta(s),\xi_{s}(\omega))\pi(da|\omega,s)\biggr{]}ds\biggr{\\}}.$
(3.11)

Now from (3.7) and (3.11), we have

$\displaystyle\varphi_{\alpha}(\theta,x)\leq
E^{\pi}_{x}\biggl{[}exp\biggl{(}\int_{0}^{t}\int_{A}\theta(s)c(\xi_{s},a)\pi(da|\omega,s)ds\biggr{)}\varphi_{\alpha}(\theta(t),\xi_{t})\biggr{]}.$
(3.12)

Given any $p>1$, let $q>1$ such that $\frac{1}{p}+\frac{1}{q}=1$, by Holder’s
inequality we have

$\displaystyle\varphi_{\alpha}(\theta,x)$ $\displaystyle\leq
E^{\pi}_{x}\biggl{[}exp\biggl{(}\int_{0}^{t}\int_{A}\theta(s)c(\xi_{s},a)\pi(da|\omega,s)ds\biggr{)}\varphi_{\alpha}(\theta(t),\xi_{t})\biggr{]}$
$\displaystyle\leq\biggl{\\{}E^{\pi}_{x}\biggl{[}exp\biggl{(}p\int_{0}^{t}\int_{A}\theta(s)c(\xi_{s},a)\pi(da|\omega,s)ds\biggr{)}\biggr{]}\biggr{\\}}^{{1}/{p}}$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\times\biggl{\\{}E^{\pi}_{x}[\varphi^{q}_{\alpha}(\theta(t),\xi_{t})]\biggr{\\}}^{{1}/{q}}$
$\displaystyle=:T_{1}(p,t)\cdot T_{2}(q,t).$ (3.13)

For
$T_{2}(q,t):=\\{E^{\pi}_{x}[\varphi^{q}_{\alpha}(\theta(t),\xi_{t})]\\}^{{1}/{q}}$,
by the upper bound of $\varphi_{\alpha}$ in (3.3), we have

$\displaystyle\varphi_{\alpha}(\theta(t),\xi_{t})=\varphi_{\alpha}(\theta
e^{-\alpha t},\xi_{t})\leq\frac{\alpha^{2}}{\alpha^{2}-\theta e^{-\alpha
t}\rho_{0}\rho_{1}}exp\biggl{(}\frac{\theta e^{-\alpha
t}L_{0}}{\alpha}\biggr{)}[V_{0}(\xi_{t})]^{\frac{\rho_{1}\theta e^{-\alpha
t}}{\alpha}}.$

If $t>\alpha^{-1}\log({\theta q\rho_{1}}/{\alpha})$ then ${\theta e^{-\alpha
t}q\rho_{1}}/{\alpha}<1$. Hence, by Jensen’s inequality and Proposition
2.1(b), we obtain

$\displaystyle T_{2}(q,t)\leq$
$\displaystyle\biggl{\\{}E^{\pi}_{x}\biggl{[}\biggl{(}\frac{\alpha^{2}}{\alpha^{2}-\theta
e^{-\alpha t}\rho_{0}\rho_{1}}\biggr{)}^{q}exp\biggl{(}\frac{q\theta
e^{-\alpha t}L_{0}}{\alpha}\biggr{)}[V_{0}(\xi_{t})]^{\frac{q\rho_{1}\theta
e^{-\alpha t}}{\alpha}}\biggr{]}\biggr{\\}}^{{1}/{q}}$
$\displaystyle=\frac{\alpha^{2}}{\alpha^{2}-\theta e^{-\alpha
t}\rho_{0}\rho_{1}}exp\biggl{(}\frac{\theta e^{-\alpha
t}L_{0}}{\alpha}\biggr{)}\biggl{[}E^{\pi}_{x}[V_{0}^{\frac{q\rho_{1}\theta
e^{-\alpha t}}{\alpha}}(\xi_{t})]\biggr{]}^{\frac{1}{q}}$
$\displaystyle\leq\frac{\alpha^{2}}{\alpha^{2}-\theta e^{-\alpha
t}\rho_{0}\rho_{1}}exp\biggl{(}\frac{\theta e^{-\alpha
t}L_{0}}{\alpha}\biggr{)}[E^{\pi}_{x}(V_{0}(\xi_{t}))]^{\frac{\rho_{1}\theta
e^{-\alpha t}}{\alpha}}$ $\displaystyle\leq\frac{\alpha^{2}}{\alpha^{2}-\theta
e^{-\alpha t}\rho_{0}\rho_{1}}exp\biggl{(}\frac{\theta e^{-\alpha
t}}{\alpha}(L_{0}+\rho_{0}\rho_{1}t)\biggr{)}V^{\frac{\theta e^{-\alpha
t}\rho_{1}}{\alpha}}_{0}(x)=:T_{3}(t).$ (3.14)

By letting $t\to\infty$ we obtain

$\displaystyle
T_{1}(p,t)\rightarrow\biggl{\\{}E^{\pi}_{x}\biggl{[}exp\biggl{(}p\int_{0}^{\infty}\int_{A}\theta(s)c(\xi_{s},a)\pi(da|\omega,s)ds\biggr{)}\biggr{]}\biggr{\\}}^{{1}/{p}}$
and $\displaystyle T_{3}(t)\rightarrow 1.$ (3.15)

Combining (3.13), (3.14) and (3.15), we obtain

$\displaystyle\varphi_{\alpha}(\theta,x)\leq\biggl{\\{}E^{\pi}_{x}\biggl{[}exp\biggl{(}p\theta\int_{0}^{\infty}\int_{A}e^{-\alpha
t}c(\xi_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]}\biggr{\\}}^{{1}/{p}},$

for $p>1$. Then, passing to the limit as $p\downarrow 1$, we obtain

$\displaystyle\varphi_{\alpha}(\theta,x)\leq
E^{\pi}_{x}\biggl{[}exp\biggl{(}\theta\int_{0}^{\infty}\int_{A}e^{-\alpha
t}c(\xi_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]}.$

Since $\pi\in\Pi$ is arbitrary control, we have

$\displaystyle\varphi_{\alpha}(\theta,x)$
$\displaystyle\leq\inf_{\pi\in\Pi}E^{\pi}_{x}\biggl{[}exp\biggl{(}\theta\int_{0}^{\infty}\int_{A}e^{-\alpha
t}c(\xi_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]}.$ (3.16)

Using (3.3), (3.5) and (3.11), we can show that

$\displaystyle
E^{\pi^{*}}_{x}\biggl{[}exp\biggl{(}\int_{0}^{t}\theta(s)c(\xi_{s},\pi^{*}_{s}(da|s,\xi_{s}))ds\biggr{)}\varphi_{\alpha}(\theta(t),\xi_{t})\biggr{]}=\varphi_{\alpha}(\theta,x).$
(3.17)

Now, using the lower bound of $\varphi_{\alpha}$ in (3.3) and Fatou’s lemma,
we obtain

$\displaystyle\liminf_{t\rightarrow\infty}E^{\pi^{*}}_{x}\biggl{[}exp\biggl{(}\int_{0}^{t}\int_{A}\theta(s)c(\xi_{s},a)\pi^{*}(da|s,\xi_{s})ds\biggr{)}\varphi_{\alpha}(\theta(t),\xi_{t})\biggr{]}$
$\displaystyle\geq\liminf_{t\rightarrow\infty}E^{\pi^{*}}_{x}\biggl{[}exp\biggl{(}\int_{0}^{t}\int_{A}\theta(s)c(\xi_{s},a)\pi^{*}(da|s,\xi_{s})ds\biggr{)}\biggr{]}$
$\displaystyle\geq
E^{\pi^{*}}_{x}\biggl{[}\liminf_{t\rightarrow\infty}exp\biggl{(}\int_{0}^{t}\int_{A}\theta(s)c(\xi_{s},a)\pi^{*}(da|s,\xi_{s})ds\biggr{)}\biggr{]}$
$\displaystyle=\tilde{J}_{\alpha}(\theta,x,\pi^{*}).$ (3.18)

From (3.17) and (3.18), we have

$\displaystyle\tilde{J}_{\alpha}(\theta,x,\pi^{*})\leq\varphi_{\alpha}(\theta,x).$

Thus

$\displaystyle\inf_{\pi\in\Pi}\tilde{J}_{\alpha}(\theta,x,\pi)\leq\tilde{J}_{\alpha}(\theta,x,\pi^{*})\leq\varphi_{\alpha}(\theta,x).$
(3.19)

From (3.16) and (3.19), we have (3.4). ∎

##  4\. The existence of solution to the HJB equation

In this Section, we prove that the equation (3.3) is the HJB equation for the
$\alpha$ discounted cost (2.3) and the equation (3.3) has a solution in
$B^{1}_{V_{0},V_{1}}([0,1]\times S)$. We now proceed to make a rigorous
analysis of the above. First we truncate our transition and cost rates which
plays a crucial role to derive the HJB equations and find the solution. Fix
any $n\geq 1$, $0<\delta<1$. For each $n\geq 1$, $x\in S$, $a\in A(x)$, let
$A_{n}(x):=A(x)$, $S_{n}:=\\{x\in S|V_{0}(x)\leq n\\}$, and
$K_{n}:=\\{(x,a)|x\in S_{n},a\in A_{n}(x)\\}$. Moreover for each $x\in S$,
$a\in A_{n}(x)$ define

$\displaystyle q^{(n)}(dy|x,a):=\left\\{\begin{array}[]{ll}q(dy|x,a)~{}\text{
if}~{}x\in S_{n},\\\ 0~{}\text{ if }~{}x\notin S_{n}\end{array}\right.$ (4.3)

and

$\displaystyle c_{n}(x,a):=\left\\{\begin{array}[]{ll}c(x,a)\wedge\text{
min}\\{n,\rho_{1}\ln V_{0}(x)+L_{0}\\}~{}\text{ if}~{}x\in S_{n},\\\
0~{}\text{ if }~{}x\notin S_{n}.\end{array}\right.$ (4.6)

######  Lemma 4.1.

Suppose Assumptions 2.1, 2.2 and 3.1 are satisfied. Then, there exists a
unique function $\varphi^{(n,\delta)}_{\alpha}$ (depending on $n$, $\delta$)
in $B^{1}_{V_{0},V_{1}}([0,1]\times S)$ for which the followings are true :

  1. (1)

$\varphi^{(n,\delta)}_{\alpha}\in B_{1}([0,1]\times S)$ is a bounded solution
to the following partial differential equations (PDEs) for all $x\in S$ and
a.e. $\theta\in(\delta,1]:$

$\displaystyle\left\\{\begin{array}[]{ll}\alpha\theta\frac{\partial\varphi^{(n,\delta)}_{\alpha}}{\partial\theta}(\theta,x)&=\displaystyle{\inf_{a\in
A(x)}\biggl{[}\theta
c_{n}(x,a)\varphi^{(n,\delta)}_{\alpha}(\theta,x)+\int_{S}q^{(n)}(dy|x,a)\varphi^{(n,\delta)}_{\alpha}(\theta,y)\biggr{]}}\\\
\varphi^{(n,\delta)}_{\alpha}(\delta,x)&=e^{{n\delta}/{\alpha}}.\end{array}\right.$
(4.9)

  2. (2)

$\varphi^{(n,\delta)}_{\alpha}(\theta,x)$ has a stochastic representation as
follows: for each $x\in S$ and a.e. $\theta\in(\delta,1]$,

$\displaystyle\varphi_{\alpha}^{(n,\delta)}(\theta,x)$
$\displaystyle=\inf_{\pi\in\Pi}E^{\pi}_{x}\biggl{[}e^{{n\delta}/{\alpha}}exp\biggl{(}\theta\int_{0}^{T_{\delta}(\theta)}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]},$

where $T_{\delta}(\theta):=\alpha^{-1}\log(\theta/\delta)$ and $\xi^{(n)}_{t}$
is the process corresponding to the $q^{(n)}(\cdot|x,a)$.

###### Proof.

(1) Since $S_{n}:=\\{x\in S|V_{0}(x)\leq n\\}$, by Assumption 2.1(ii), we say
that $q^{(n)}_{x}(a):=\int_{S/\\{x\\}}q^{(n)}(dy|x,a)$ is bounded. So we can
use the Lyapunov function $V\equiv 1$ such that
$\int_{S}q^{(n)}(dy|x,a)V(y)\leq\rho_{0}V(x)$, and
$\overline{q}^{(n)}:=\sup_{(x,a)\in K}q^{(n)}_{x}(a)<\infty$. Now let us
define an nonlinear operator $T$ on $B_{1}([0,1]\times S)$ as follows:

$\displaystyle Tu(\theta,x)=$ $\displaystyle e^{{\delta
n}/{\alpha}}+\frac{1}{\alpha}\int_{\delta}^{\theta}\inf_{a\in
A(x)}\biggl{[}\frac{1}{s}\int_{S}q^{(n)}(dy|x,a)u(s,y)+c_{n}(x,a)u(s,x)\biggr{]}ds,$

where $u\in B_{1}([0,1]\times S)$ and $(\theta,x)\in[\delta,1]\times S$. By
using the Assumption 2.1 and the fact that $c_{n}$ is bounded, we obtain

$\displaystyle\sup_{\theta\in[\delta,1]}\sup_{x\in S}|Tu(\theta,x)|$
$\displaystyle\leq e^{{\delta
n}/{\alpha}}+\frac{1}{\alpha}\int_{\delta}^{1}\sup_{a\in
A(x)}\biggl{\\{}\frac{1}{s}\sup_{x\in
S}\biggl{[}\int_{S}|q^{(n)}(dy|x,a)||u(s,y)|\biggr{]}+n\sup_{x\in
S}|u(s,x)|\biggr{\\}}ds$ $\displaystyle\leq e^{{\delta
n}/{\alpha}}+\frac{\|u\|_{1}^{\infty}}{\alpha}\biggl{\\{}\int_{\delta}^{1}\sup_{a\in
A(x)}\frac{1}{s}\sup_{x\in
S}\biggl{(}2q^{(n)}_{x}(a)\biggr{)}ds+n(1-\delta)\biggr{\\}}$
$\displaystyle\leq e^{{\delta
n}/{\alpha}}+\frac{1}{\alpha}\biggl{[}(-2)\overline{q}^{(n)}\log{\delta}+n(1-\delta)\biggr{]}{\|u\|_{1}^{\infty}}.$

Therefore, $T$ is a nonlinear operator from $B_{1}([0,1]\times S)$ to
$B_{1}([0,1]\times S)$. For any $g_{1},g_{2}\in B_{1}([0,1]\times S)$ and
$\theta\in[\delta,1]$, we have

$\displaystyle\sup_{x\in S}|Tg_{1}(t,x)-Tg_{2}(t,x)|$
$\displaystyle\leq\frac{1}{\alpha}\int_{\delta}^{t}\biggl{(}2\overline{q}^{(n)}/s+n\biggr{)}\sup_{x\in
S}|g_{1}(s,x)-g_{2}(s,x)|ds$
$\displaystyle\leq\frac{1}{\alpha}\biggl{[}2\overline{q}^{(n)}(\log
t-\log\delta)+n(t-\delta)\biggr{]}\|g_{1}-g_{2}\|^{\infty}_{1}.$ (4.11)

Now, we prove the following:

$\sup_{x\in
S}|T^{l}g_{1}(t,x)-T^{l}g_{2}(t,x)|\leq\frac{\|g_{1}-g_{2}\|^{\infty}_{1}}{\alpha^{l}\cdot
l!}\biggl{[}2\overline{q}^{(n)}(\log
t-\log\delta)+n(t-\delta)\biggr{]}^{l}~{}~{}\forall~{}l\geq 1.$ (4.12)

By (4.11) and (4.12) we have

$\displaystyle\sup_{x\in S}|T^{l+1}g_{1}(t,x)-T^{l+1}g_{2}(t,x)|$
$\displaystyle\leq\frac{1}{\alpha}\int_{\delta}^{t}\biggl{(}2\overline{q}^{(n)}/s+n\biggr{)}\sup_{x\in
S}|T^{l}g_{1}(s,x)-T^{l}g_{2}(s,x)|ds$
$\displaystyle\leq\frac{\|g_{1}-g_{2}\|^{\infty}_{1}}{\alpha^{l+1}\cdot
l!}\int_{\delta}^{t}\biggl{(}2\overline{q}^{(n)}/s+n\biggr{)}\biggl{[}2\overline{q}^{(n)}(\log
s-\log\delta)+n(s-\delta)\biggr{]}^{l}ds$
$\displaystyle=\frac{\|g_{1}-g_{2}\|^{\infty}_{1}}{\alpha^{l+1}\cdot(l+1)!}\biggl{[}2\overline{q}^{(n)}(\log
t-\log\delta)+n(t-\delta)\biggr{]}^{l+1}.$

Since $\sum_{k\geq 1}\frac{1}{\alpha^{k}\cdot
k!}\biggl{[}-2\overline{q}^{(n)}\log\delta+n(1-\delta)\biggr{]}^{k}<\infty$,
there exists some $m$ such that $\beta:=\frac{1}{\alpha^{m}\cdot
m!}\biggl{[}-2\overline{q}^{(n)}\log\delta+n(1-\delta)\biggr{]}^{m}<1,$ which
implies that
$\|T^{m}g_{1}-T^{m}g_{2}\|_{1}^{\infty}\leq\beta\|g_{1}-g_{2}\|^{\infty}_{1}$.
Therefore, T is a $m$-step contraction operator on $B_{1}([0,1]\times S)$. So,
by Banach fixed point theorem, there exists a unique bounded function
$\varphi_{\alpha}^{(n,\delta)}\in B_{1}([0,1]\times S)$ (depending on
$(n,\delta)$) such that
$T\varphi^{(n,\delta)}_{\alpha}=\varphi^{(n,\delta)}_{\alpha}$; that is,

$\displaystyle\varphi^{(n,\delta)}_{\alpha}(\theta,x)=e^{{\delta
n}/{\alpha}}+\frac{1}{\alpha}\int_{\delta}^{\theta}\inf_{a\in
A(x)}\biggl{[}\frac{1}{s}\int_{S}q^{(n)}(dy|x,a)\varphi^{(n,\delta)}_{\alpha}(s,y)+c_{n}(x,a)\varphi^{(n,\delta)}_{\alpha}(s,x)\biggr{]}ds.$

Also note that $\varphi^{(n,\delta)}_{\alpha}(\delta,x)=e^{{\delta
n}/{\alpha}}$. Hence by using (4.3), (4.6) and the above equation, we have
$\varphi^{(n,\delta)}_{\alpha}\in B^{1}_{V_{0},V_{1}}([0,1]\times S)$ and it
satisfies equation (4.9).

(2) We see that

$\displaystyle\biggl{[}\theta
c_{n}(x,a)\varphi^{(n,\delta)}_{\alpha}(\theta,x)+\int_{S}q^{(n)}(dy|x,a)\varphi^{(n,\delta)}_{\alpha}(\theta,y)\biggr{]}$

is continuous in $a\in A(x)$ and $A(x)$ is compact. So by measurable selection
theorem, [[2],Proposition 7.33], there exists a measurable function
$f_{\delta}^{*}:[0,1]\times S\rightarrow A$ such that

$\displaystyle\inf_{a\in A(x)}\biggl{[}\theta
c_{n}(x,a)\varphi^{(n,\delta)}_{\alpha}(\theta,x)+\int_{S}q^{(n)}(dy|x,a)\varphi^{(n,\delta)}_{\alpha}(\theta,y)\biggr{]}$
$\displaystyle=\biggl{[}\theta
c_{n}(x,f_{\delta}^{*}(\theta,x))\varphi^{(n,\delta)}_{\alpha}(\theta,x)+\int_{S}q^{(n)}(dy|x,f_{\delta}^{*}(\theta,x))\varphi^{(n,\delta)}_{\alpha}(\theta,y)\biggr{]}.$
(4.13)

Let

$\pi_{\delta}^{*}:\mathbb{R}_{+}\times S\to P(A)\ $

be defined by

$\displaystyle\pi_{\delta}^{*}(\cdot|t,x)=I_{\\{\hat{f}_{\delta}^{*}(t,x)\\}}(\cdot),~{}\text{where}~{}\hat{f}_{\delta}^{*}:[0,1]\times
S\rightarrow A,$ $\displaystyle\quad~{}\text{be a measurable mapping, defined
by}~{}\hat{f}_{\delta}^{*}(t,x):=f_{\delta}^{*}(\theta e^{-\alpha t},x).$

Let $\theta(t):=\theta e^{-\alpha t}$ for $t\in[0,\infty)$. Since $c_{n}$ and
$\varphi_{\alpha}^{(n,\delta,k)}$ are bounded, by Dynkin’s formula we get

$\displaystyle
E^{\pi}_{x}\biggl{[}exp\biggl{(}\int_{0}^{T_{\delta}(\theta)}\int_{A}\theta(s)c_{n}(\xi^{(n)}_{s},a)\pi(da|\omega,s)ds\biggr{)}\varphi^{(n,\delta)}_{\alpha}\biggl{(}\theta(T_{\delta}),\xi_{T_{\delta}}^{(n)}\biggr{)}\biggr{]}-\varphi^{(n,\delta)}_{\alpha}(\theta,x)$
$\displaystyle=E^{\pi}_{x}\biggl{\\{}\int_{0}^{T_{\delta}(\theta)}\biggl{[}-\alpha\theta(s)\frac{\partial\varphi^{(n,\delta)}_{\alpha}}{\partial\theta}(\theta(s),\xi^{(n)}_{s})+\int_{S}\int_{A}q^{(n)}(dy|\xi^{(n)}_{s},a)\varphi^{(n,\delta)}_{\alpha}(\theta(s),y)\pi(da|\omega,s)$
$\displaystyle+\theta(s)\int_{A}c_{n}(\xi^{(n)}_{s},a)\varphi_{\alpha}^{(n,\delta)}(\theta(s),\xi^{(n)}_{s})\pi(da|\omega,s)\biggr{]}$
$\displaystyle
exp\biggl{(}\int_{0}^{s}\int_{A}\theta(v)c_{n}(\xi_{v}^{(n)},a)\pi(da|\omega,v)dv\biggr{)}ds\biggr{\\}}.$
(4.14)

By using (4.9) and (4.14), we obtain

$\displaystyle
E^{\pi}_{x}\biggl{[}exp\biggl{(}\int_{0}^{T_{\delta}(\theta)}\int_{A}\theta(s)c_{n}(\xi^{(n)}_{s},a)\pi(da|\omega,s)ds\biggr{)}\varphi^{(n,\delta)}_{\alpha}\biggl{(}\theta(T_{\delta}),\xi_{T_{\delta}}^{(n)}\biggr{)}\biggr{]}\geq\varphi^{(n,\delta)}_{\alpha}(\theta,x).$

Since $\pi\in\Pi$ is an arbitrary control and
$\varphi_{\alpha}^{(n,\delta)}(\theta(T_{\delta}(\theta)),\xi^{(n)}_{T_{\delta}})=e^{{n\delta}/{\alpha}}$,
we have

$\displaystyle\varphi^{(n,\delta)}_{\alpha}(\theta,x)\leq\inf_{\pi\in\Pi}E^{\pi}_{x}\biggl{[}e^{{n\delta}/{\alpha}}exp\biggl{(}\int_{0}^{T_{\delta}(\theta)}\int_{A}\theta(s)c_{n}(\xi^{(n)}_{s},a)\pi(da|\omega,s)ds\biggr{)}\biggr{]}.$
(4.15)

Using equations (4.9), (4.13) and (4.14), we can show that

$\displaystyle\varphi^{(n,\delta)}_{\alpha}(\theta,x)=E^{\pi_{\delta}^{*}}_{x}\biggl{[}e^{{n\delta}/{\alpha}}exp\biggl{(}\int_{0}^{T_{\delta}(\theta)}\int_{A}\theta(s)c_{n}(\xi^{(n)}_{s},a)\pi_{\delta}^{*}(da|s,\xi^{(n)}_{s})ds\biggr{)}\biggr{]}.$

Therefore

$\displaystyle\varphi^{(n,\delta)}_{\alpha}(\theta,x)\geq\inf_{\pi\in\Pi}E^{\pi}_{x}\biggl{[}e^{{n\delta}/{\alpha}}exp\biggl{(}\int_{0}^{T_{\delta}(\theta)}\int_{A}\theta(s)c_{n}(\xi^{(n)}_{s},a)\pi(da|\omega,s)ds\biggr{)}\biggr{]}.$
(4.16)

Therefore, from (4.15) and (4.16), we obtain (LABEL:eq_4.4). This completes
the proof. ∎

######  Theorem 4.1.

Suppose Assumptions 2.1, 2.2 and 3.1 hold. Then the HJB equation (3.3) has a
unique solution $\varphi_{\alpha}\in B^{1}_{V_{0},V_{1}}([0,1]\times S)$
satisfying $1\leq\varphi_{\alpha}(\theta,x)\leq{\frac{\alpha^{2}e^{{\theta
L_{0}}/{\alpha}}}{\alpha^{2}-\rho_{0}\rho_{1}\theta}}(V(x))^{\frac{\rho_{1}\theta}{\alpha}}$
for all $(\theta,x)\in[0,1]\times S.$

###### Proof.

First note that, $\varphi^{(n,\delta)}_{\alpha}$ is the solution to the
equation (4.9), which depends on two parameters $n$, $\delta$. We prove this
theorem in two steps.

Step 1. In the first step, we construct a solution
$\varphi^{(n)}_{\alpha}(\cdot,x)$ from
$\varphi^{(n,\delta)}_{\alpha}(\cdot,x)$ by passing the limit as
$\delta\rightarrow 0$, such that $\varphi^{(n)}_{\alpha}(\cdot,x)$ is
absolutely continuous function and satisfies the following DEs:

$\displaystyle\left\\{\begin{array}[]{lllll}\alpha\theta\frac{\partial\varphi^{(n)}_{\alpha}}{\partial\theta}(\theta,x)&=\displaystyle{\inf_{a\in
A(x)}}\biggl{[}\int_{S}q^{(n)}(dy|x,a)\varphi_{\alpha}^{(n)}(\theta,y)+\theta
c_{n}(x,a)\varphi^{(n)}_{\alpha}(\theta,x)\biggr{]},~{}x\in
S,~{}\text{a.e.},~{}\theta\in[0,1],\\\
1\leq\varphi^{(n)}_{\alpha}(\theta,x)&\leq{\frac{\alpha^{2}e^{{\theta
L_{0}}/{\alpha}}}{\alpha^{2}-\rho_{0}\rho_{1}\theta}}(V_{0}(x))^{\frac{\rho_{1}\theta}{\alpha}}~{}~{}~{}\forall~{}(\theta,x)\in[0,1]\times
S.\end{array}\right.$ (4.19)

Given $0<\delta<1$ and $1\leq n<\infty$ by (LABEL:eq_4.4) and $\sup_{(x,a)\in
K}c_{n}(x,a)\leq n$, we have $\varphi^{(n,\delta)}_{\alpha}(\theta,x)\leq
e^{2n/\alpha},~{}~{}x\in S,\theta\in[\delta,1]$.

Next, we extend the domain of $\varphi^{(n,\delta)}_{\alpha}$ to $[0,1]\times
S$ by

$\overline{\varphi}^{(n,\delta)}_{\alpha}(\theta,x)=\left\\{\begin{array}[]{lcl}{\varphi}^{(n,\delta)}_{\alpha}(\theta,x),&\delta\leq\theta\leq
1~{}\forall x\in S\\\ e^{{n\delta}/{\alpha}},&0\leq\theta<\delta~{}\forall
x\in S.\end{array}\right.$

We consider the following expression, for any given $\pi\in\Pi$, $x\in S$,
$\theta,\theta_{0}\in[\delta,1]$:

$\displaystyle\biggl{|}E^{\pi}_{x}$
$\displaystyle\biggl{[}e^{n\delta/\alpha}exp\biggl{(}\theta\int_{0}^{T_{\delta}(\theta)}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]}$
$\displaystyle-E^{\pi}_{x}\biggl{[}e^{n\delta/\alpha}exp\biggl{(}\theta_{0}\int_{0}^{T_{\delta}(\theta_{0})}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]}\biggr{|}$
$\displaystyle\leq P_{1}+P_{2},$

where

$\displaystyle P_{1}:=$
$\displaystyle\biggl{|}E^{\pi}_{x}\biggl{[}e^{n\delta/\alpha}exp\biggl{(}\theta\int_{0}^{T_{\delta}(\theta)}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]}$
$\displaystyle-E^{\pi}_{x}\biggl{[}e^{n\delta/\alpha}exp\biggl{(}\theta_{0}\int_{0}^{T_{\delta}(\theta)}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]}\biggr{|},$

and

$\displaystyle P_{2}:=$
$\displaystyle\biggl{|}E^{\pi}_{x}\biggl{[}e^{n\delta/\alpha}exp\biggl{(}\theta_{0}\int_{0}^{T_{\delta}(\theta)}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]}$
$\displaystyle-E^{\pi}_{x}\biggl{[}e^{n\delta/\alpha}exp\biggl{(}\theta_{0}\int_{0}^{T_{\delta}(\theta_{0})}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]}\biggr{|}.$

Fix $n\geq 1$; we have

$\displaystyle\int_{0}^{T_{\delta}(\theta)}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\leq
n\int_{0}^{T_{\delta}(\theta)}e^{-\alpha t}dt\leq\frac{n}{\alpha}$

and

$\displaystyle\int_{T_{\delta}(\theta\wedge\theta_{0})}^{T_{\delta}(\theta\vee\theta_{0})}$
$\displaystyle\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\leq
n\int_{T_{\delta}(\theta\wedge\theta_{0})}^{T_{\delta}(\theta\vee\theta_{0})}e^{-\alpha
t}dt$ $\displaystyle=\frac{n}{\alpha}[exp(-\alpha
T_{\delta}(\theta\wedge\theta_{0}))-exp(-\alpha
T_{\delta}(\theta\vee\theta_{0}))]\leq\frac{\delta
n|\theta_{0}-\theta|}{\alpha\theta\theta_{0}},$

where $c\wedge d:=min\\{c,d\\}$ and $c\vee d:=max\\{c,d\\}$. Hence, we obtain

$\displaystyle P_{1}$
$\displaystyle=e^{n\delta/\alpha}E^{\pi}_{x}\biggl{[}exp\biggl{(}(\theta\wedge\theta_{0})\int_{0}^{T_{\delta}(\theta)}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}$
$\displaystyle~{}~{}~{}~{}\times\biggl{(}exp\biggl{(}|\theta_{0}-\theta|\int_{0}^{T_{\delta}(\theta)}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}-1\biggr{)}\biggr{]}$
$\displaystyle\leq
e^{2n/\alpha}E^{\pi}_{x}\biggl{[}exp\biggl{(}|\theta_{0}-\theta|\int_{0}^{T_{\delta}(\theta)}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}-1\biggr{]}$
$\displaystyle\leq
e^{2n/\alpha}\Big{(}exp\Big{(}\frac{n}{\alpha}|\theta_{0}-\theta|\Big{)}-1\Big{)}$
$\displaystyle\leq
e^{2n/\alpha}\Big{(}e^{n/\alpha}-1\Big{)}|\theta_{0}-\theta|.$

Here, the last inequality follows from the fact that $e^{bz}-1\leq(e^{b}-1)z$
for all $z\in[0,1]$ and $b>0$. Similarly for $P_{2}$ we have

$\displaystyle P_{2}$
$\displaystyle=e^{n\delta/\alpha}E^{\pi}_{x}\biggl{[}exp\biggl{(}\theta_{0}\int_{0}^{T_{\delta}(\theta\wedge\theta_{0})}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}$
$\displaystyle~{}~{}~{}~{}\times\biggl{(}exp\biggl{(}\theta_{0}\int_{T_{\delta}(\theta\wedge\theta_{0})}^{T_{\delta}(\theta\vee\theta_{0})}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}-1\biggr{)}\biggr{]}$
$\displaystyle\leq
e^{2n/\alpha}E^{\pi}_{i}\biggl{[}exp\biggl{(}\theta_{0}\int_{T_{\delta}(\theta\wedge\theta_{0})}^{T_{\delta}(\theta\vee\theta_{0})}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}-1\biggr{]}$
$\displaystyle\leq
e^{2n/\alpha}\biggl{(}exp\biggl{(}\frac{n\delta|\theta-\theta_{0}|}{\alpha\theta}\biggr{)}-1\biggr{)}$
$\displaystyle\leq
e^{2n/\alpha}\biggl{(}e^{n/\alpha}-1\biggr{)}|\theta_{0}-\theta|.$

Hence for all $(\theta,x)\in[0,1]\times S$, we have

$\displaystyle|\overline{\varphi}^{(n,\delta)}_{\alpha}(\theta_{0},x)-\overline{\varphi}^{(n,\delta)}_{\alpha}(\theta,x)|\leq
2e^{{2n}/{\alpha}}(e^{{n}/{\alpha}}-1)|\theta-\theta_{0}|.$ (4.20)

Now we want to show that $\overline{\varphi}^{(n,\delta)}_{\alpha}$ is
decreasing as $\delta\rightarrow 0$ for any $(\theta,x)$. For a fixed
$\alpha>0$ and $\varepsilon>0$ small enough, consider
$\overline{\varphi}^{(n,\delta+\varepsilon)}_{\alpha}(\theta,x)-\overline{\varphi}^{(n,\delta)}_{\alpha}(\theta,x)$
and assume that $h_{\delta}:=e^{\frac{n\delta}{\alpha}}$. By measurable
selection theorem we get the minimizer $\pi^{*}_{\delta+\varepsilon}$ like in
equation (4.13), corresponding to
$\overline{\varphi}_{\alpha}^{(n,\delta+\varepsilon)}$ such that the
followings cases hold.
Case 1. If $\delta+\varepsilon<\theta$ then

$\displaystyle\overline{\varphi}^{(n,\delta+\varepsilon)}_{\alpha}(\theta,x)-\overline{\varphi}^{(n,\delta)}_{\alpha}(\theta,x)$
$\displaystyle=E^{\pi^{*}_{\delta+\varepsilon}}_{x}\biggl{[}h_{\delta+\varepsilon}exp\biggl{(}\theta\int_{0}^{T_{\delta+\varepsilon}}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},\pi^{*}_{\delta+\varepsilon}(da|t,\xi^{(n)}_{t}))dt\biggr{)}\biggr{]}$
$\displaystyle\quad-\inf_{\pi\in\Pi}E^{\pi}_{x}\biggl{[}h_{\delta}exp\biggl{(}\theta\int_{0}^{T_{\delta}}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},\pi(da|\omega,t))dt\biggr{)}\biggr{]}$
$\displaystyle\geq
h_{\delta}E^{\pi^{*}_{\delta+\varepsilon}}_{x}\biggl{[}exp\biggl{(}\theta\int_{0}^{T_{\delta+\varepsilon}}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},\pi^{*}_{\delta+\varepsilon}(da|t,\xi^{(n)}_{t}))dt\biggr{)}$
$\displaystyle\quad\times\biggl{\\{}h_{\varepsilon}-exp\biggl{(}\theta\int_{T_{\delta+\varepsilon}}^{T_{\delta}}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},\pi^{*}_{\delta+\varepsilon}(da|t,\xi^{(n)}_{t}))dt\biggr{)}\biggr{\\}}\biggr{]}$
$\displaystyle\geq
h_{\delta}E^{\pi^{*}_{\delta+\varepsilon}}_{x}\biggl{[}exp\biggl{(}\theta\int_{0}^{T_{\delta+\varepsilon}}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},\pi^{*}_{\delta+\varepsilon}(da|t,\xi^{(n)}_{t}))dt\biggr{)}\biggl{\\{}h_{\varepsilon}-exp\biggl{(}\theta\int_{T_{\delta+\varepsilon}}^{T_{\delta}}e^{-\alpha
t}ndt\biggr{)}\biggr{\\}}\biggr{]}$
$\displaystyle=h_{\delta}E^{\pi^{*}_{\delta+\varepsilon}}_{x}\biggl{[}exp\biggl{(}\theta\int_{0}^{T_{\delta+\varepsilon}}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},\pi^{*}_{\delta+\varepsilon}(da|t,\xi^{(n)}_{t}))dt\biggr{)}\biggl{\\{}h_{\varepsilon}-exp\biggl{(}\frac{n\theta(e^{-\alpha
T_{\delta+\varepsilon}}-e^{-\alpha
T_{\delta}})}{\alpha}\biggr{)}\biggr{\\}}\biggr{]}$ $\displaystyle=0.$

Case 2. $\delta<\theta\leq\delta+\varepsilon$

$\displaystyle\overline{\varphi}^{(n,\delta+\varepsilon)}_{\alpha}(\theta,x)-\overline{\varphi}^{(n,\delta)}_{\alpha}(\theta,x)$
$\displaystyle=h_{\delta+\varepsilon}-E^{\pi^{*}_{\delta}}_{x}\biggl{[}h_{\delta}exp\biggl{(}\theta\int_{0}^{T_{\delta}}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},\pi^{*}_{\delta}(da|t,\xi^{(n)}_{t}))dt\biggr{)}\biggr{]}$
$\displaystyle=h_{\delta}\biggl{[}h_{\varepsilon}-E^{\pi^{*}_{\delta}}_{x}\biggl{[}exp\biggl{(}\theta\int_{{0}}^{T_{\delta}}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},\pi^{*}_{\delta}(da|t,\xi^{(n)}_{t}))dt\biggr{)}\biggr{]}\biggr{]}$
$\displaystyle\geq
h_{\delta}\biggl{[}h_{\varepsilon}-exp\biggl{(}\theta\int_{{0}}^{T_{\delta}}e^{-\alpha
t}ndt\biggr{)}\biggr{]}$
$\displaystyle=h_{\delta}\biggl{[}h_{\varepsilon}-e^{n\theta\frac{(1-e^{-\alpha
T_{\delta}})}{\alpha}}\biggr{]}\geq 0.$

Case 3. $\theta\leq\delta$

$\overline{\varphi}^{(n,\delta+\varepsilon)}_{\alpha}(\theta,x)-\overline{\varphi}^{(n,\delta)}_{\alpha}(\theta,x)=h_{\delta+\varepsilon}-h_{\delta}=h_{\delta}(h_{\varepsilon}-1)=h_{\delta}(e^{\frac{n\varepsilon}{\alpha}}-1)\geq
0.$

Hence $\overline{\varphi}_{\alpha}^{(n,\delta)}(\theta,x)$ is increasing in
$\delta$ for any $(\theta,x)\in[0,1]\times S$. Now from (4.20), we know that
for each $x\in S$, $\overline{\varphi}_{\alpha}^{(n,\delta)}(\cdot,x)$ is
Lipschitz continous in $\theta\in[0,1]$. Also,
$\overline{\varphi}_{\alpha}^{(n,\delta)}(\theta,x)$ is increasing in $\delta$
for any $(\theta,x)\in[0,1]\times S$ and bounded above (since
$\overline{\varphi}^{(n,\delta)}_{\alpha}(\theta,x)\leq
e^{2n/\alpha},~{}~{}x\in S,\theta\in[\delta,1]$), therefore there exists a
function $\varphi^{(n)}_{\alpha}$ on $[0,1]\times S$ that is continuous with
respect to $\theta\in[0,1]$, such that along a subsequence
$\delta_{m}\rightarrow 0$, we have
$\lim_{m\rightarrow\infty}\overline{\varphi}_{\alpha}^{(n,\delta_{m})}(\theta,x)=\varphi_{\alpha}^{(n)}(\theta,x)$
and for any fixed $x\in S$ this convergence is uniform in $\theta\in[0,1]$.

Let $\psi\in C^{\infty}_{c}(0,1)$, then we have

$\displaystyle-\int_{0}^{1}\alpha\frac{d(\theta\psi)}{d\theta}(\theta)\overline{\varphi}_{\alpha}^{(n,\delta_{m})}(\theta,x)d\theta=\int_{0}^{1}\alpha\theta\frac{\partial\overline{\varphi}^{(n,\delta_{m})}_{\alpha}}{\partial\theta}(\theta,x)\psi(\theta)d\theta$
$\displaystyle=\int_{0}^{1}\displaystyle{\inf_{a\in A(x)}}\biggl{[}\theta
c_{n}(x,a)\overline{\varphi}^{(n,\delta_{m})}_{\alpha}(\theta,x)+\int_{S}q^{(n)}(dy|x,a)\overline{\varphi}^{(n,\delta_{m})}_{\alpha}(\theta,y)\biggr{]}\psi(\theta)d\theta$
$\displaystyle-\int_{0}^{\delta_{m}}\inf_{a\in A(x)}\biggl{[}\theta
c_{n}(x,a)\overline{\varphi}^{(n,\delta_{m})}_{\alpha}(\theta,x)+\int_{S}q^{(n)}(dy|x,a)\overline{\varphi}^{(n,\delta_{m})}_{\alpha}(\theta,y)\biggr{]}\psi(\theta)d\theta$
$\displaystyle=\int_{0}^{1}\displaystyle{\inf_{a\in A(x)}\biggl{[}\theta
c_{n}(x,a)\overline{\varphi}^{(n,\delta_{m})}_{\alpha}(\theta,x)+\int_{S}q^{(n)}(dy|x,a)\overline{\varphi}^{(n,\delta_{m})}_{\alpha}(\theta,y)\biggr{]}}\psi(\theta)d\theta$
$\displaystyle-\int_{0}^{\delta_{m}}\inf_{a\in A(x)}\biggl{[}\theta
c_{n}(x,a)\overline{\varphi}^{(n,\delta_{m})}_{\alpha}(\theta,x)\biggr{]}\psi(\theta)d\theta.$
(4.21)

Now take $\tau(x):=M_{0}V_{0}(x)$ and define

$Q^{(n)}(dy|x,a):=\delta_{x}(dy)+\frac{q^{(n)}(dy|x,a)}{\tau(x)}$

for all $(x,a)\in K$ where $\delta_{x}(\cdot)$ is the Dirac measure
concentrated at $x$. We see that under Assumption 2.1, $Q^{(n)}$ is a
stochatic kernel on $S$ given $K$. Then (4.21) can be written as

$\displaystyle-\int_{0}^{1}\biggl{\\{}\frac{\alpha}{\tau(x)}\frac{d(\theta\psi)}{d\theta}\overline{\varphi}_{\alpha}^{(n,\delta_{m})}(\theta,x)-\overline{\varphi}_{\alpha}^{(n,\delta_{m})}(\theta,x)\psi(\theta)\biggr{\\}}d\theta$
$\displaystyle=\int_{0}^{1}\displaystyle{\inf_{a\in
A(x)}}\biggl{[}\frac{\theta}{\tau(x)}c_{n}(x,a)\overline{\varphi}^{(n,\delta_{m})}_{\alpha}(\theta,x)+\int_{S}Q^{(n)}(dy|x,a)\overline{\varphi}^{(n,\delta_{m})}_{\alpha}(\theta,y)\biggr{]}\psi(\theta)d\theta$
$\displaystyle-\frac{1}{\tau(x)}\int_{0}^{\delta_{m}}\inf_{a\in
A(x)}\biggl{[}\theta
c_{n}(x,a)\overline{\varphi}^{(n,\delta_{m})}_{\alpha}(\theta,x)\biggr{]}\psi(\theta)d\theta.$
(4.22)

Now

$\displaystyle\biggl{|}\displaystyle{\inf_{a\in
A(x)}\biggl{[}\frac{\theta}{\tau(x)}c_{n}(x,a)\overline{\varphi}^{(n,\delta_{m})}_{\alpha}(\theta,x)+\int_{S}Q^{(n)}(dy|x,a)\overline{\varphi}^{(n,\delta_{m})}_{\alpha}(\theta,y)\biggr{]}}\psi(\theta)\biggr{|}$
$\displaystyle\leq|\psi(\theta)|\displaystyle{\sup_{a\in
A(x)}\biggl{[}\frac{\theta}{\tau(x)}|c_{n}(x,a)||\overline{\varphi}^{(n,\delta)}_{\alpha}(\theta,x)|+\int_{S}Q^{(n)}(dy|x,a)|\overline{\varphi}^{(n,\delta)}_{\alpha}(\theta,y)|\biggr{]}}$
$\displaystyle\leq{\frac{\alpha^{2}}{\alpha^{2}-\rho_{0}\rho_{1}\theta}}e^{{\theta
L_{0}}/{\alpha}}\displaystyle\sup_{a\in
A(x)}\biggl{[}\frac{\theta}{\tau(x)}nV_{0}^{\frac{\rho_{1}\theta}{\alpha}}(x)+\int_{S}Q(dy|x,a)V_{0}^{\frac{\rho_{1}\theta}{\alpha}}(y)\biggr{]}|\psi(\theta)|$
$\displaystyle\leq{\frac{\alpha^{2}}{\alpha^{2}-\rho_{0}\rho_{1}\theta}}e^{{\theta
L_{0}}/{\alpha}}\displaystyle\sup_{a\in
A(x)}\biggl{[}\frac{\theta}{\tau(x)}nV_{0}(x)+\int_{S}Q(dy|x,a)V_{0}(y)\biggr{]}|\psi(\theta)|$
$\displaystyle\leq{\frac{\alpha^{2}}{\alpha^{2}-\rho_{0}\rho_{1}\theta}}e^{{\theta
L_{0}}/{\alpha}}\displaystyle\biggl{[}\frac{\theta}{\tau(x)}nV_{0}(x)+V_{0}(x)+\rho_{0}\frac{V_{0}(x)}{\tau(x)}\biggr{]}|\psi(\theta)|$
$\displaystyle={\frac{\alpha^{2}}{\alpha^{2}-\rho_{0}\rho_{1}\theta}}e^{{\theta
L_{0}}/{\alpha}}\displaystyle\biggl{[}\frac{\theta}{\tau(x)}nV_{0}(x)+V_{0}(x)+\frac{\rho_{0}}{M_{0}}\biggr{]}|\psi(\theta)|.$
(4.23)

Since for each fixed $x\in S$, $A(x)$ is compact, there exists a subsequence
of $\\{m\\}$, by abuse of notation, we denote the same sequence and $a^{*}\in
A(x)$ such that $\lim_{m\rightarrow\infty}a^{*}_{m}=a^{*}$. Now, from (4.22),
for any $a\in A(x)$, we have

$\displaystyle-\int_{0}^{1}\biggl{\\{}\frac{\alpha}{\tau(x)}\frac{d(\theta\psi)}{d\theta}\overline{\varphi}_{\alpha}^{(n,\delta_{m})}(\theta,x)-\overline{\varphi}_{\alpha}^{(n,\delta_{m})}(\theta,x)\psi(\theta)\biggr{\\}}d\theta$
$\displaystyle=\int_{0}^{1}\displaystyle\biggl{[}\frac{\theta}{\tau(x)}c_{n}(x,a^{*}_{m})\overline{\varphi}^{(n,\delta_{m})}_{\alpha}(\theta,x)+\int_{S}Q^{(n)}(dy|x,a^{*}_{m})\overline{\varphi}^{(n,\delta_{m})}_{\alpha}(\theta,y)\biggr{]}\psi(\theta)d\theta$
$\displaystyle\quad-\frac{1}{\tau(x)}\int_{0}^{\delta_{m}}\inf_{a\in
A(x)}\biggl{[}\theta
c_{n}(x,a)\overline{\varphi}^{(n,\delta_{m})}_{\alpha}(\theta,x)\biggr{]}\psi(\theta)d\theta.$
(4.24)

So, by Lemma 8.3.7 in Hernandez-Lerma and Lassere (1999) [12] taking limit as
$m\rightarrow\infty$ in (4.24), we get

$\displaystyle-\int_{0}^{1}\biggl{\\{}\frac{\alpha}{\tau(x)}\frac{d(\theta\psi)}{d\theta}(\theta)\varphi_{\alpha}^{(n)}(\theta,x)-\varphi_{\alpha}^{(n)}(\theta,x)\psi(\theta)\biggr{\\}}d\theta$
$\displaystyle\geq\int_{0}^{1}\displaystyle\biggl{[}\frac{\theta}{\tau(x)}c_{n}(x,a^{*})\varphi^{(n)}_{\alpha}(\theta,x)+\int_{S}Q^{(n)}(dy|x,a^{*})\varphi^{(n)}_{\alpha}(\theta,y)\biggr{]}\psi(\theta)d\theta.$

Hence

$\displaystyle-\int_{0}^{1}\biggl{\\{}\frac{\alpha}{\tau(x)}\frac{d(\theta\psi)}{d\theta}(\theta)\varphi_{\alpha}^{(n)}(\theta,x)-\varphi_{\alpha}^{(n)}(\theta,x)\psi(\theta)\biggr{\\}}d\theta$
$\displaystyle\geq\inf_{a\in
A(x)}\int_{0}^{1}\displaystyle\biggl{[}\frac{\theta}{\tau(x)}c_{n}(x,a)\varphi^{(n)}_{\alpha}(\theta,x)+\int_{S}Q^{(n)}(dy|x,a)\varphi^{(n)}_{\alpha}(\theta,y)\biggr{]}\psi(\theta)d\theta.$
(4.25)

But

$\displaystyle-\int_{0}^{1}\biggl{\\{}\frac{\alpha}{\tau(x)}\frac{d(\theta\psi)}{d\theta}\overline{\varphi}_{\alpha}^{(n,\delta_{m})}(\theta,x)-\overline{\varphi}_{\alpha}^{(n,\delta_{m})}(\theta,x)\psi(\theta)\biggr{\\}}d\theta$
$\displaystyle\leq\int_{0}^{1}\displaystyle\biggl{[}\frac{\theta}{\tau(x)}c_{n}(x,a)\overline{\varphi}^{(n,\delta_{m})}_{\alpha}(\theta,x)+\int_{S}Q^{(n)}(dy|x,a)\overline{\varphi}^{(n,\delta_{m})}_{\alpha}(\theta,y)\biggr{]}\psi(\theta)d\theta$
$\displaystyle\quad-\frac{1}{\tau(x)}\int_{0}^{\delta_{m}}\inf_{a\in
A(x)}\biggl{[}\theta
c_{n}(x,a)\overline{\varphi}^{(n,\delta_{m})}_{\alpha}(\theta,x)\biggr{]}\psi(\theta)d\theta.$

By similar arguments, we get

$\displaystyle-\int_{0}^{1}\biggl{\\{}\frac{\alpha}{\tau(x)}\frac{d(\theta\psi)}{d\theta}(\theta)\varphi_{\alpha}^{(n)}(\theta,x)-\varphi_{\alpha}^{(n)}(\theta,x)\psi(\theta)\biggr{\\}}d\theta$
$\displaystyle\leq\inf_{a\in
A(x)}\int_{0}^{1}\displaystyle\biggl{[}\frac{\theta}{\tau(x)}c_{n}(x,a)\varphi^{(n)}_{\alpha}(\theta,x)+\int_{S}Q^{(n)}(dy|x,a)\varphi^{(n)}_{\alpha}(\theta,y)\biggr{]}\psi(\theta)d\theta.$
(4.26)

From (4.25) and (4.26), we get

$\displaystyle-\int_{0}^{1}\biggl{\\{}\frac{\alpha}{\tau(x)}\frac{d(\theta\psi)}{d\theta}(\theta)\varphi_{\alpha}^{(n)}(\theta,x)-\varphi_{\alpha}^{(n)}(\theta,x)\psi(\theta)\biggr{\\}}d\theta$
$\displaystyle=\inf_{a\in
A(x)}\int_{0}^{1}\displaystyle\biggl{[}\frac{\theta}{\tau(x)}c_{n}(x,a)\varphi^{(n)}_{\alpha}(\theta,x)+\int_{S}Q^{(n)}(dy|x,a)\varphi^{(n)}_{\alpha}(\theta,y)\biggr{]}\psi(\theta)d\theta.$
(4.27)

Thus we obtain

$\displaystyle-\int_{0}^{1}\alpha\frac{d(\theta\psi)}{d\theta}(\theta)\varphi_{\alpha}^{(n)}(\theta,x)d\theta$
$\displaystyle=\inf_{a\in
A(x)}\int_{0}^{1}\displaystyle\biggl{[}{\theta}c_{n}(x,a)\varphi^{(n)}_{\alpha}(\theta,x)+\int_{S}q^{(n)}(dy|x,a)\varphi^{(n)}_{\alpha}(\theta,y)\biggr{]}\psi(\theta)d\theta.$

Hence

$\displaystyle\alpha\theta\frac{\partial\varphi^{(n)}_{\alpha}}{\partial\theta}(\theta,x)$
$\displaystyle=\inf_{a\in A(x)}\displaystyle\biggl{[}\theta
c_{n}(x,a)\varphi^{(n)}_{\alpha}(\theta,x)+\int_{S}q^{(n)}(dy|x,a)\varphi^{(n)}_{\alpha}(\theta,y)\biggr{]}~{}\text{a.e.}~{}\theta\in[0,1]$

in the sense of distribution. When
$\frac{\partial\varphi^{(n)}_{\alpha}}{\partial\theta}$ does not exist for
some $(\theta,x)$, we define

$\displaystyle\alpha\theta\frac{\partial\varphi^{(n)}_{\alpha}}{\partial\theta}(\theta,x)$
$\displaystyle=\inf_{a\in A(x)}\displaystyle\biggl{[}\theta
c_{n}(x,a)\varphi^{(n)}_{\alpha}(\theta,x)+\int_{S}q^{(n)}(dy|x,a)\varphi^{(n)}_{\alpha}(\theta,y)\biggr{]}.$

Now for $\theta\in[\delta_{m},1]$, by using (LABEL:eq_4.4) and Proposition
2.1, we have

$\displaystyle\varphi^{(n,\delta_{m})}_{\alpha}(\theta,x)=\inf_{\pi\in\Pi}E^{\pi}_{x}\biggl{[}e^{n\delta_{m}/\alpha}\exp\biggl{(}\theta\int_{0}^{T_{\delta_{m}}(\theta)}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]}$
$\displaystyle\leq
e^{n\delta_{m}/\alpha}\inf_{\pi\in\Pi}E^{\pi}_{x}\biggl{[}exp\biggl{(}\theta\int_{0}^{\infty}\int_{B}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]}$
$\displaystyle\leq
e^{n\delta_{m}/\alpha}\inf_{\pi\in\Pi}E^{\pi}_{x}\biggl{[}exp\biggl{(}\theta\int_{0}^{\infty}\int_{B}\int_{A}e^{-\alpha
t}c(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]}$ $\displaystyle\leq
e^{n\delta_{m}/\alpha}{\frac{\alpha^{2}e^{{\theta
L_{0}}/{\alpha}}}{\alpha^{2}-\rho_{0}\rho_{1}\theta}}(V_{0}(x))^{\frac{\rho_{1}\theta}{\alpha}}.$

Note that $\varphi_{\alpha}^{(n,\delta_{m})}\rightarrow\varphi_{\alpha}^{(n)}$
as $m\rightarrow\infty$. Thus, letting $m\rightarrow\infty$ in the above
equation, we obtain

$\displaystyle
1\leq\varphi^{(n)}_{\alpha}(\theta,x)\leq{\frac{\alpha^{2}e^{{\theta
L_{0}}/{\alpha}}}{\alpha^{2}-\rho_{0}\rho_{1}\theta}}(V_{0}(x))^{\frac{\rho_{1}\theta}{\alpha}}.$
(4.28)

By using (4.3), (4.6), (4.28), and the PDE satisfied by
$\varphi^{(n)}_{\alpha}$ (that is just proven), we say that
$\varphi^{(n)}_{\alpha}\in B^{1}_{V_{0},V_{1}}([0,1]\times S)$ and it is a
solution of (4.19). Thus by closely mimicking the arguments as in Theorem 3.1,
one can easily get the stochastic representation of the solution
$\varphi_{\alpha}^{(n)}$, that is

$\displaystyle\varphi^{(n)}_{\alpha}(\theta,x)=\inf_{\pi\in\Pi}E^{\pi}_{x}\biggl{[}exp\biggl{(}\theta\int_{0}^{\infty}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]}.$ (4.29)

Step 2: In this step we prove Theorem 4.1, by passing to the limit as
$n\to\infty$. Now we will prove that for each $x\in S$,
$\\{\varphi^{(n)}_{\alpha}\\}_{n\geq 1}$ is equicontinuous on $[0,1]$. We
consider the following expression, for any given $\pi\in\Pi$, $x\in S$,
$\theta,\theta_{0}\in[0,1]$:

$\displaystyle\biggl{|}E^{\pi}_{x}$
$\displaystyle\biggl{[}exp\biggl{(}\theta\int_{0}^{\infty}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]}$
$\displaystyle-E^{\pi}_{x}\biggl{[}exp\biggl{(}\theta_{0}\int_{0}^{\infty}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]}\biggr{|}$
$\displaystyle\leq K_{1},$

where

$\displaystyle K_{1}$
$\displaystyle=E^{\pi}_{x}\biggl{[}exp\biggl{(}(\theta\wedge\theta_{0})\int_{0}^{\infty}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}$
$\displaystyle~{}~{}~{}~{}\times\biggl{(}exp\biggl{(}|\theta_{0}-\theta|\int_{0}^{\infty}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}-1\biggr{)}\biggr{]}$
$\displaystyle\leq
E^{\pi}_{x}\biggl{[}exp\biggl{(}(\theta\wedge\theta_{0})\int_{0}^{\infty}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}$
$\displaystyle~{}~{}~{}~{}\times\biggl{(}exp\biggl{(}\int_{0}^{\infty}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}-1\biggr{)}|\theta_{0}-\theta|\biggr{]}$
$\displaystyle\leq
E^{\pi}_{x}\biggl{[}exp\biggl{(}\int_{0}^{\infty}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}$
$\displaystyle~{}~{}~{}~{}\times\biggl{(}exp\biggl{(}\int_{0}^{\infty}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}|\theta_{0}-\theta|\biggr{)}\biggr{]}$
$\displaystyle=|\theta_{0}-\theta|\times
E^{\pi}_{x}\biggl{[}exp\biggl{(}2\int_{0}^{\infty}\int_{A}e^{-\alpha
t}c_{n}(\xi^{(n)}_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]}$
$\displaystyle\leq|\theta_{0}-\theta|\times\frac{\alpha
e^{{2L_{0}}/{\alpha}}}{\alpha-\rho_{2}}M_{1}^{2}\biggl{(}V_{1}^{2}(x)+\frac{b_{1}}{\rho_{2}}\biggr{)}.$

Here, the first inequality is according to $e^{bz}-1\leq(e^{b}-1)z$ for all
$z\in[0,1]$ and $b>0$ and the last inequality follows from (3.8). Therefore,
we have

$\displaystyle|\varphi^{(n)}_{\alpha}(\theta_{0},x)-\varphi^{(n)}_{\alpha}(\theta,x)|$
$\displaystyle\leq\sup_{\pi\in\Pi}|\theta_{0}-\theta|\times\frac{\alpha
e^{{2L_{0}}/{\alpha}}}{\alpha-\rho_{2}}M_{1}^{2}\biggl{(}V_{1}^{2}(x)+\frac{b_{1}}{\rho_{2}}\biggr{)}$
$\displaystyle=|\theta_{0}-\theta|\times\frac{\alpha
e^{{2L_{0}}/{\alpha}}}{\alpha-\rho_{2}}M_{1}^{2}\biggl{(}V_{1}^{2}(x)+\frac{b_{1}}{\rho_{2}}\biggr{)}.$
(4.30)

By measurable selection theorem, [[2],Proposition 7.33], there exists a
measurable function $f^{*}_{n}:[0,1]\times S\rightarrow A$ such that

$\displaystyle\inf_{a\in A(x)}\biggl{[}\theta
c_{n}(x,a)\varphi_{\alpha}(\theta,x)+\int_{S}q^{(n)}(dy|x,a)\varphi_{\alpha}(\theta,y)\biggr{]}$
$\displaystyle=\biggl{[}\theta
c_{n}(x,f^{*}_{n}(\theta,x))\varphi_{\alpha}(\theta,x)+\int_{S}q^{(n)}(dy|x,f^{*}_{n}(\theta,x))\varphi_{\alpha}(\theta,y)\biggr{]}.$
(4.31)

Let

$\pi^{*}_{n}:\mathbb{R}_{+}\times S\to P(A)\ $

be defined by

$\displaystyle\pi^{*}_{n}(\cdot|t,x)=I_{\\{\hat{f}^{*}_{n}(t,x)\\}}(\cdot),~{}\text{where}~{}\hat{f}_{n}^{*}:[0,1]\times
S\rightarrow A,$ $\displaystyle\quad~{}\text{be a measurable mapping, defined
by}~{}\hat{f}_{n}^{*}(t,x):=f_{n}^{*}(\theta e^{-\alpha t},x).$

Hence by equation (4.19), we have a.e. $\theta\in[0,1]$ and $\forall x\in S$,
we have

$\displaystyle\left\\{\begin{array}[]{lllll}\alpha\theta\frac{\partial\varphi^{(n)}_{\alpha}}{\partial\theta}(\theta,x)&=\displaystyle\biggl{[}\int_{S}q^{(n)}(dy|x,f^{*}_{n}(\theta,x))\varphi_{\alpha}^{(n)}(\theta,y)+\theta
c_{n}(x,f^{*}_{n}(\theta,x))\varphi^{(n)}_{\alpha}(\theta,x)\biggr{]}\\\
1\leq\varphi^{(n)}_{\alpha}(\theta,x)&\leq{\frac{\alpha^{2}e^{{\theta
L_{0}}/{\alpha}}}{\alpha^{2}-\rho_{0}\rho_{1}\theta}}(V_{0}(x))^{\frac{\rho_{1}\theta}{\alpha}}~{}~{}~{}\forall~{}(\theta,x)\in[0,1]\times
S.\end{array}\right.$ (4.34)

Since $c_{n}\geq 0$, by (4.29), we say $\varphi^{(n)}_{\alpha}(\theta,x)$ is
increasing in $\theta$. Also we know that $\varphi^{(n)}_{\alpha}(\theta,x)$
is differentiable a.e. with respect to $\theta\in[0,1]$. So

$\displaystyle\frac{\partial\varphi^{(n)}_{\alpha}}{\partial\theta}(\theta,x)\geq
0~{}~{}\text{for a.e.}~{}\theta.$ (4.35)

So, by (4.3), (4.6) and (4.34), for all $x\in S$ and for a.e. $\theta$, we
have

$\displaystyle\left\\{\begin{array}[]{lllll}&-\alpha\theta\frac{\partial\varphi^{(n)}_{\alpha}}{\partial\theta}(\theta,x)+\displaystyle\biggl{[}\int_{S}q^{(n-1)}(dy|x,f^{*}_{n}(\theta,x))\varphi_{\alpha}^{(n)}(\theta,y)+\theta
c_{n-1}(x,f^{*}_{n}(\theta,x))\varphi^{(n)}_{\alpha}(\theta,x)\biggr{]}\leq
0\\\ &\quad\quad\text{if}~{}x\in S_{n-1}\end{array}\right.$ (4.38)

and

$\displaystyle\left\\{\begin{array}[]{llll}&-\alpha\theta\frac{\partial\varphi^{(n)}_{\alpha}}{\partial\theta}(\theta,x)+\displaystyle\biggl{[}\int_{S}q^{(n-1)}(dy|x,f^{*}_{n}(\theta,x))\varphi_{\alpha}^{(n)}(\theta,y)+\theta
c_{n-1}(x,f^{*}_{n}(\theta,x))\varphi^{(n)}_{\alpha}(\theta,x)\biggr{]}\\\
&=-\alpha\theta\frac{\partial\varphi^{(n)}_{\alpha}}{\partial\theta}(\theta,x)\leq
0\\\ &\quad\quad\text{if}~{}x\notin S_{n-1}~{}\text{(by~{}(\ref{eq
4.25}))}.\end{array}\right.$ (4.42)

So, by Feynman-Kac formula, we get

$\displaystyle
E^{\pi^{*}_{n}}_{x}\biggl{[}exp\biggl{(}\theta\int_{0}^{\infty}\int_{A}e^{-\alpha
t}c_{n-1}(\xi^{(n-1)}_{t},a)\pi^{*}_{n}(da|t,\xi^{(n-1)}_{t})dt\biggr{)}\biggr{]}\leq\varphi^{(n)}_{\alpha}(\theta,x)~{}\text{for
all}~{}(\theta,x)\in[0,1]\times S.$ (4.43)

Also using (4.19) and Feynman-Kac formula (see (3.11) and (3.16)), we have

$\displaystyle\varphi^{(n-1)}_{\alpha}(\theta,x)\leq
E^{\pi^{*}_{n}}_{x}\biggl{[}exp\biggl{(}\theta\int_{0}^{\infty}\int_{A}e^{-\alpha
t}c_{n-1}(\xi^{(n-1)}_{t},a)\pi^{*}_{n}(da|t,\xi^{(n-1)}_{t})dt\biggr{)}\biggr{]}.$
(4.44)

By (4.43) and (4.44), we have
$\varphi^{(n-1)}_{\alpha}(\theta,x)\leq\varphi^{(n)}_{\alpha}(\theta,x).$

Hence $\varphi^{(n)}_{\alpha}(\theta,x)$ is increasing in $n$ for any
$(\theta,x)\in[0,1]\times S$. Now from (4.30), we know that for each $x\in S$,
$\varphi^{(n)}(\cdot,x)$ is Lipschitz continous in $\theta\in[0,1]$. Also,
$\varphi^{(n)}_{\alpha}(\theta,x)$ is increasing as $n\rightarrow\infty$ for
any $(\theta,x)\in[0,1]\times S$ and bounded above (by (4.28)), therefore
there exists a function $\varphi_{\alpha}$ on $[0,1]\times S$ that is
continuous with respect to $\theta\in[0,1]$, such that along a subsequence
$n_{k}\rightarrow\infty$, we have
$\lim_{n_{k}\rightarrow\infty}\varphi^{(n_{k})}_{\alpha}(\theta,x)=\varphi_{\alpha}(\theta,x)$
and this convergence is uniform in $\theta\in[0,1]$ for each fixed $x\in S$.
Moreover, by (4.28), we have

$\displaystyle 1\leq\varphi_{\alpha}(\theta,x)$
$\displaystyle\leq{\frac{\alpha^{2}e^{{\theta
L_{0}}/{\alpha}}}{\alpha^{2}-\rho_{0}\rho_{1}\theta}}(V_{0}(x))^{\frac{\rho_{1}\theta}{\alpha}}.$

As the proof of equation (4.19) in the step 1 (starting from the first
equality of (4.21)), we say that $\varphi_{\alpha}$ is a solution to the HJB
equation (3.3). Also by (3.3) and Assumption 2.1, we have

$\biggl{|}\displaystyle{\inf_{a\in
A(x)}\biggl{[}\int_{S}q(dy|x,a)\varphi_{\alpha}(\theta,y)+\theta
c(x,a)\varphi_{\alpha}(\theta,x)\biggr{]}}\biggr{|}\leq{\frac{\alpha^{2}e^{{\theta
L_{0}}/{\alpha}}}{\alpha^{2}-\rho_{0}\rho_{1}\theta}}[\rho_{0}+2M_{0}+\rho_{1}+L_{0}]M_{1}V_{1}(x).$

So,
$|\alpha\theta\frac{\partial\varphi_{\alpha}}{\partial\theta}(\theta,x)|\leq{\frac{\alpha^{2}e^{{\theta
L_{0}}/{\alpha}}}{\alpha^{2}-\rho_{0}\rho_{1}\theta}}[\rho_{0}+2M_{0}+\rho_{1}+L_{0}]M_{1}V_{1}(x)$.
Hence $\varphi_{\alpha}\in B^{1}_{V_{0},V_{1}}([0,1]\times S)$. Finally, the
uniqueness of $\varphi_{\alpha}(\theta,x)$ follows from the stochastic
representation in Theorem 3.1. ∎

##  5\. The existence of optimal control

In this section, we present the main result of this article. Here we show the
existence of an optimal control.

######  Theorem 5.1.

Suppose that Assumptions 2.1, 2.2 and 3.1 are satisfied. Then, the following
assertions hold.

  1. (1)

The HJB equation (3.3) has a unique solution $\varphi_{\alpha}\in
B^{1}_{V_{0},V_{1}}([0,1]\times S)$ and the solution admits the following
representation

$\displaystyle 1\leq\varphi_{\alpha}(\theta,x)$
$\displaystyle=\inf_{\pi\in\Pi}E^{\pi}_{x}\biggl{[}exp\biggl{(}\theta\int_{0}^{\infty}\int_{A}e^{-\alpha
t}c(\xi_{t},a)\pi(da|\omega,t)dt\biggr{)}\biggr{]}$
$\displaystyle\leq{\frac{\alpha^{2}e^{{\theta
L_{0}}/{\alpha}}}{\alpha^{2}-\rho_{0}\rho_{1}\theta}}(V_{0}(x))^{\frac{\rho_{1}\theta}{\alpha}}.$

  2. (2)

There exists a measurable function $f^{*}:[0,1]\times S\to A$ such that

$\displaystyle\alpha\theta\frac{\partial\varphi_{\alpha}}{\partial\theta}(\theta,x)$
$\displaystyle=\biggl{[}\int_{S}q(dy|x,f^{*}(\theta,x))\varphi_{\alpha}(\theta,y)+\theta
c(x,f^{*}(\theta,x))\varphi_{\alpha}(\theta,x)\biggr{]}$
$\displaystyle\text{a.e.}~{}\theta\in[0,1].$ (5.1)

  3. (3)

Furthermore an optimal Markov control for the cost criterion (2.2) exists and
is given by

$\tilde{\pi}^{*}(\cdot|t,x):=I_{\\{\hat{f}(t,x)\\}}(\cdot),\hat{f}(t,x):=f^{*}(\theta
e^{-\alpha t},x),$

where $f^{*}$ satisfies (5.1).

###### Proof.

Part (1) follows from Theorems 3.1 and 4.1. To prove (2), for each given
$(\theta,x)\in[0,1]\times S$, by [12], we have the continuity of the function

$\eta(x,\theta,a):=\int_{S}q(dy|x,a)\varphi_{\alpha}(\theta,y)+\theta
c(x,a)\varphi_{\alpha}(\theta,x)$

in $a\in A(x)$. Thus, the measurable selection theorem [[2], Proposition 7.33]
ensured the existence of a measurable function $f^{*}$ satisfying (5.1), and
so (2) follows. Moreover for any $f^{*}$ satisfying (5.1), from the proof of
Theorem 3.1, we have
$\inf_{\pi\in\Pi}\tilde{J}_{\alpha}(x,\theta,{\pi})=\tilde{J}_{\alpha}(x,\theta,\tilde{\pi}^{*})=\varphi_{\alpha}(\theta,x)$,
which together with (2.2), (2.3) and part (1), we have
$\inf_{\pi\in\Pi}\mathscr{J}_{\alpha}(x,\theta,{\pi})=\mathscr{J}_{\alpha}(x,\theta,\tilde{\pi}^{*})=\frac{1}{\theta}\ln\tilde{J}_{\alpha}(x,\theta,\tilde{\pi}^{*})=\frac{1}{\theta}\ln\varphi_{\alpha}(\theta,x).$
Hence $\tilde{\pi}^{*}$ is an optimal Markov control. ∎

##  6\. Application and example

In this section, we verify the above assumptions with one example, where the
transition and cost rates are unbounded.

######  Example 6.1.

The Gaussian Model: Suppose a hunter is hunting outside his house for his
manager. Suppose the house is at state 0. A positive state represents the
distance from the house to the right, and a negative state represents the
distance from the house to the left. Let $S=\mathbb{R}$. If the current
postion is $x\in S$, the hunter takes a action $a\in A(x)$, then after an
exponentially distributed travel time with rate $\lambda(x,a)>0$, the hunter
reaches the new position, and the travel distance follows the normal
distribution with mean $x$ and variance $\sigma$. (Or we can interpret
$\lambda(x,a)$ as the total jump intensity that is an arbitrary measurable
positive-valued function on $S\times A$, and the distribution of the state
after a jump from $x\in S$ is normal with the variance $\sigma$ and
expectation $x$.) Also assume that the hunter receives a payoff $c(x,a)$ from
his manager for each unit of time he spends there. Let us consider the model
as $A_{2}:=\\{S,(A,A(x),x\in S),c(x,a),q(dy|x,a)\\}$, where
$S=(-\infty,\infty)$. For each $D\in\mathscr{B}(S)$, the transition rate is

$\displaystyle q(D|x,a)=\lambda(x,a)\bigg{[}\int_{y\in
D}\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(y-x)^{2}}{2\sigma^{2}}}dy-\delta_{x}(D)\bigg{]},~{}x\in
S,a\in A(x).$ (6.1)

To ensure the existence of an optimal Markov control for the model, we
consider the following hypotheses.

  1. (I)

For each fixed $x\in S$, $\lambda(x,a)$ is continuous in $a\in A(x)$ and there
exists a positive constant $M$ such that $0<\sup_{a\in A(x)}\lambda(x,a)\leq
M(x^{2}+1)$ and
$M<\frac{\alpha}{3780(\sigma^{8}+\sigma^{6}+\sigma^{4}+\sigma^{2})}$.

  2. (II)

For each $x\in S$, the cost rate $c(x,a)$ is nonnegative and continuous in
$a\in A(x)$ and there exists constant
$0<\rho_{1}<\min\\{\alpha,\frac{\alpha^{2}}{M\sigma^{2}}\\}$ such that

$\sup_{a\in A(x)}c(x,a)\leq\rho_{1}\log(1+x^{2}).$

  3. (III)

For each fixed $x\in S$, $A(x)$ is a compact subset of the Borel spaces $A$.

######  Proposition 6.1.

Under conditions (I)-(III), the above controlled system satisfies the
Assumptions 2.1, 2.2, and 3.1. Hence by Theorem 5.1, there exists an optimal
Markov control for this model.

###### Proof.

We know
$\frac{1}{\sqrt{2\pi}\sigma}\int_{-\infty}^{\infty}(y-x)^{2k+1}e^{-\frac{(y-x)^{2}}{2\sigma^{2}}}dy=0$
and
$\frac{1}{\sqrt{2\pi}\sigma}\int_{-\infty}^{\infty}(y-x)^{2k}e^{-\frac{(y-x)^{2}}{2\sigma^{2}}}dy=1\cdot
3\cdots(2k-1)\sigma^{2k}$ for all $k=0,1\cdots.$
We first verify Assumption 2.1. Let $V_{0}(x)=x^{2}+1$.

$\displaystyle\int_{S}V_{0}(y)q(dy|x,a)$
$\displaystyle=\lambda(x,a)\bigg{[}\frac{1}{\sqrt{2\pi}\sigma}\int_{-\infty}^{\infty}(y^{2}+1)e^{-\frac{(y-x)^{2}}{2\sigma^{2}}}dy-(x^{2}+1)\bigg{]}$
$\displaystyle=\lambda(x,a)\sigma^{2}$ $\displaystyle\leq
M\sigma^{2}V_{0}(x).$ (6.2)

Let $\rho_{0}=M\sigma^{2}$. Then
$\int_{S}V_{0}(y)q(dy|x,a)\leq\rho_{0}V_{0}(x).$ Now

$\displaystyle q^{*}(x)=\sup_{a\in A(x)}q_{x}(a)=\sup_{a\in
A(x)}\lambda(x,a)\leq M(x^{2}+1)=MV_{0}(x)~{}\forall~{}x\in S.$

Now by condition (II), we can write

$\displaystyle\sup_{a\in A(x)}c(x,a)\leq\rho_{1}\log(1+x^{2})+M.$

Also by condition (II), $0<\rho_{1}<\min\\{\alpha,\rho_{0}^{-1}\alpha^{2}\\}$.
Hence Assumption 2.1 is verified.
Now we verify Assumption 2.2.
Let $V_{1}(x)=x^{4}+1$. Then for any $x\in S$, $a\in A(x)$,

$\displaystyle\int_{S}q(dy|x,a)V^{2}_{1}(y)$
$\displaystyle=\lambda(x,a)\biggl{[}\frac{1}{\sqrt{2\pi}\sigma}\int_{-\infty}^{\infty}(y^{4}+1)^{2}e^{-\frac{(y-x)^{2}}{2\sigma^{2}}}dy-(x^{4}+1)^{2}\biggr{]}$
$\displaystyle=\lambda(x,a)(105\sigma^{8}+420x^{2}\sigma^{6}+210x^{4}\sigma^{4}+6\sigma^{4}+12\sigma^{2}x^{2}+28x^{6}\sigma^{2})$
$\displaystyle\leq
420\lambda(x,a)(\sigma^{8}+\sigma^{6}+\sigma^{4}+\sigma^{2})(x^{6}+x^{4}+x^{2}+1)$
$\displaystyle\leq
420\lambda(x,a)(\sigma^{8}+\sigma^{6}+\sigma^{4}+\sigma^{2})(3x^{6}+3)$
$\displaystyle\leq
1260M(\sigma^{8}+\sigma^{6}+\sigma^{4}+\sigma^{2})(x^{6}+1)(1+x^{2})$
$\displaystyle\leq
3780M(\sigma^{8}+\sigma^{6}+\sigma^{4}+\sigma^{2})(x^{4}+1)^{2}$
$\displaystyle\leq
3780M(\sigma^{8}+\sigma^{6}+\sigma^{4}+\sigma^{2})V_{1}^{2}(x)+1.$

Now take $\rho_{2}=3780M(\sigma^{8}+\sigma^{6}+\sigma^{4}+\sigma^{2})$. Then
by condition (I), we have $0<\rho_{2}<\alpha$. Also, $(1+x^{2})^{2}\leq
2(1+x^{4})$ for all $x\in S$. Let $M_{1}=2$, then $V_{0}^{2}(x)\leq
M_{1}V_{1}(x)$. Hence, Assumption 2.2 is verified. Now by conditions (I) and
(II), $q(\cdot|x,a)$ and $c(x,a)$ are continuous in $a\in A(x)$. Now by (6),
$\int_{S}V_{0}(y)q(dy|x,a)$ is continuous in $a\in A(x)$. Hence Assumption 3.1
is also verified. So, by Theorem 5.1, we say that there exists an optimal
Markov control for this model. ∎

## References

  * [1] N. BAUERLE AND U. RIEDER, More risk-sensitive Markov decision processes, Math. Oper. Res., 39 (2014), pp. 105-120.
  * [2] D. BERTSEKAS AND S. SHREVE, Stochastic Optimal Control: The Discrete-Time Case, Academic Press, Inc., 1996.
  * [3] M. K. GHOSH AND S. SAHA, Risk-sensitive control of continuous-time Markov chains, Stoch., 86 (2014), pp. 655-675.
  * [4] X. P. GUO AND O. HERNANDEZ-LERMA, Continuous-Time Markov decision processes: Theory and Applications, Stochastic Modelling and Applied Probability, Springer, Berlin, 2009.
  * [5] X. GUO, X. HUANG, AND Y. HUANG, Finite-horizon optimality for continuous-time Markov decision processes with unbounded transition rates, Adv. Appl. Probab., 47 (2015), pp. 1064-1087.
  * [6] X. GUO, Y. HUANG, AND X. SONG, Linear programming and constrained average optimality for general continuous-time Markov decision processes in history-dependent polices, Siam J. Control Optim, 50 (2012), pp. 23-47.
  * [7] X. GUO AND Z. W. LIAO, Risk-sensitive discounted continuous-time Markov decision processes with unbounded rates, SIAM J. Control Optim., 57 (2019), pp. 3857-3883.
  * [8] X. GUO, Q. LIU, AND Y. ZHANG, Finite horizon risk-sensitive continuous-time Markov decision processes with unbounded transition and cost rates, 4OR, 17 (2019), pp. 427-442.
  * [9] X. GUO AND A. PIUNOVSKIY, Discounted continuous-time Markov decision processes with constraints: Unbounded transition and loss rates, Math. Oper. Res., 36 (2011), pp. 105-132.
  * [10] X. GUO AND J. ZHANG, Risk-sensitive continuous-time Markov decision processes with unbounded rates and Borel spaces, Discrete Event Dyn. Syst., 29 (2019), pp. 445-471.
  * [11] X. GUO AND Y. ZHANG, On risk-sensitive piecewise deterministic Markov decision processes, Appl. Math. Optim. 81 (2020), pp. 685-710.
  * [12] O. HERNANDEZ-LERMA AND J. LASSERRE,  Further topics on discrete-time Markov control processes, Springer, New York, 1999.
  * [13] M. Y. KITAEV, Semi-Markov and jump Markov controlled models: Average cost criterion, SIAM Theory Probab. Appl., 30 (1995), pp. 272-288.
  * [14] M. Y. KITAEV AND V.V. RYKOV, Controlled Queueing Systems, CRC Press, Boca Raton, 1995.
  * [15] K.S. KUMAR AND C. PAL, Risk-sensitive control of jump process on denumerable state space with near monotone cost, Appl. Math. Optim., 68 (2013), pp. 311-331.
  * [16] K.S. KUMAR AND C. PAL, Risk-sensitive control of continuous-time Markov processes with denumerable state space, Stoch. Anal. Appl., 33 (2015), pp. 863-881.
  * [17] G. B. MASI AND L. STETTNER, Infinite horizon risk-sensitive control of discrete time Markov processes with small risk, Syst. Control Lett., 40 (2000), pp. 15-20.
  * [18] G. B. MASI AND L. STETTNER, Infinite horizon risk-sensitive control of discrete time Markov processes under minorization property, Siam J. Control Optim., 46 (2007), pp. 231-252.
  * [19] C. PAL AND S. PRADHAN, Risk-sensitive control of pure jump processes on a general state space, An International Journal of Probab. and Stoch. Processes, 91 (2)(2019), pp. 155-174.
  * [20] A. PIUNOVSKIY AND Y. ZHANG, Discounted continuous-time Markov decision processes with unbounded rates: The convex analytic approach, SIAM J. Control Optim., 49 (2011), pp. 2032-2061.
  * [21] A. PIUNOVSKIY AND Y. ZHANG, Continuous-Time Markov Decision Processes, Springer, 2020.
  * [22] T. PRIETO-RUMEAU AND O. HERNANDEZ-LERMA, Selected topics in continuous-time controlled Markov chains and Markov games, Imperical College Press, London, 2012.
  * [23] Q. WEI, Continuous-time Markov decision processes with risk-sensitive finite-horizon cost criterion, Math. Methods Oper. Res., 84 (2016), pp. 461-487.
  * [24] P. WHITTLE, Risk-Sensitive Optimal Control, Wiley-Inter science Series in Systems and Optimization, John Wiley & Sons Ltd., Chichester, 1990.
  * [25] Y. ZHANG, Continuous-time Markov decision processes with exponential utility, SIAM J. Control Optim., 55 (2017), pp. 2636-2660.