# Zero-sum risk-sensitive continuous-time stochastic games with unbounded
payoff and transition rates and Borel spaces††thanks: This work was supported
in part by the National Natural Science Foundation of China (Grant No.
61673019, 11931018, 62073346, 61573206) and Guangdong Province Key Laboratory
of Computational Science at the Sun Yat-Sen University (2020B1212060032).
_(Corresponding author: Li Xia.)_

Junyu Zhang, Xianping Guo, Li Xia J. Zhang and X.P. Guo are both with the
School of Mathematics, Sun Yat-Sen University, Guangzhou 510275, China (email:
{mcszhjy;mcsgxp}@mail.sysu.edu.cn). L. Xia is with the Business School, Sun
Yat-Sen University, Guangzhou 510275, China (email: xial@tsinghua.edu.cn). All
the authors are with the Guangdong Province Key Laboratory of Computational
Science at the Sun Yat-Sen University.

Abstract: We study a finite-horizon two-person zero-sum risk-sensitive
stochastic game for continuous-time Markov chains and Borel state and action
spaces, in which payoff rates, transition rates and terminal reward functions
are allowed to be unbounded from below and from above and the policies can be
history-dependent. Under suitable conditions, we establish the existence of a
solution to the corresponding Shapley equation (SE) by an approximation
technique. Then, by the SE and the extension of the Dynkin’s formula, we prove
the existence of a Nash equilibrium and verify that the value of the
stochastic game is the unique solution to the SE. Moreover, we develop a value
iteration-type algorithm for approaching to the value of the stochastic game.
The convergence of the algorithm is proved by a special contraction operator
in our risk-sensitive stochastic game. Finally, we demonstrate our main
results by two examples.

Key Words. Zero-sum risk-sensitive stochastic game, unbounded
transition/payoff rates, Nash equilibrium, iteration algorithm.

##  1 Introduction

Markov chain is a fundamental model to formulate stochastic dynamic systems.
Markov decision process (MDP) is an important methodology to study the
performance optimization of stochastic dynamic systems with a single decision-
maker [8, 21, 41]. The classical MDP theory usually focuses on the performance
criteria of the discounted or average rewards. However, the risk-aware
performance criterion is also important for decision-makers in specific
scenarios, such as financial engineering. The risk-sensitive MDP is motivated
by the seminal work of Howard and Matheson [31] and attracts continuous
attention in the literatures, which involve various risk metrics, such as
exponential utility function [19, 24], variance [29, 47], value at risk (VaR)
[30], conditional VaR [6, 32], percentile [14, 17], and probability metrics
[45].

When we further consider stochastic dynamic systems with multiple decision-
makers, game theory is a typical framework. For stochastic games (also known
as Markov games), Markov model is widely adopted since it has advantages of
memoryless property to capture the system dynamics. The study of stochastic
games is pioneered by the seminal work of Shapely [42] and can be divided into
categories according to two-person or $n$-person, zero-sum or nonzero-sum,
discrete-time or continuous-time, static or dynamic, and other criteria. A
two-person zero-sum stochastic game is the mostly studied model since the
gains of one player is exactly the loss of the other player. The Shapley
equation is the fundamental concept to study the value of a stochastic game
and to prove the existence of Nash equilibria [15, 38].

Most studies on stochastic games focus on the discounted performance criterion
since the associated contract operator is critical to prove the convergence of
games. However, as aforementioned, risk-related performance metrics are also
important for decision-making in game theory, especially considering the fact
that human players are usually risk-averse in terms of gains but risk-seeking
in terms of losses, as indicated by the prospect theory [34]. Although the
risk-sensitive MDPs have been richly investigated, the study of risk-sensitive
stochastic games seems to be relatively limited in the literatures, in which
the risk metric is usually of the form of exponential utility function [31,
33]. That is, let $X$ be the accumulated discounted rewards, which is a random
variable. The targeted risk metric is set as $\mathbb{E}[e^{\theta X}]$,
instead of $\mathbb{E}[X]$ in risk-neutral regime, where $\theta$ is a risk-
sensitive parameter and positive for risk-averse while negative for risk-
seeking. With the Taylor expansion of $\frac{1}{\theta}\ln\mathbb{E}[e^{\theta
X}]$, we can see that the targeted metric takes into account both
$\mathbb{E}[X]$ and $D[X]$ simultaneously and is a proper performance metric
for risk-aware decision-making problems [7, 24].

For risk-sensitive discrete-time stochastic games, Basu and Ghosh study a two-
person zero-sum stochastic game on the infinite horizon with discounted and
ergodic payoff criteria, where the value of games and equilibria are proved by
studying the Hamilton-Jacobi-Bellman-Isaacs (HJBI) equation [3]. They further
extend the results to the cases with countable state space [4] and nonzero-sum
games [5], respectively. Moreover, Ba$\ddot{{\rm u}}$erle and Rieder study a
two-person zero-sum risk-sensitive discrete-time stochastic game with Borel
state and action spaces and bounded rewards, where the existence of equilibria
is proved and the value of the game is obtained by solving the Shapley
equation under continuity and compactness conditions [7].

When the stochastic game is of continuous-time, Başar studies a class of
nonzero-sum risk-sensitive differential games with parameterized nonlinear
dynamics and parameterized cost functions, where the sensitivity of the Nash
equilibrium is also studied from a viewpoint of optimal control [2]. Başar and
his collaborators further extend the risk-sensitive differential game to a
scenario with $n$-person in the model of mean-field, where a Hamilton-Jacobi-
Bellman-Fleming equation is utilized to study the value of the game [43].
There are other excellent works on risk-sensitive differential games, just to
name a few [9, 11, 16], where varied forms of HJBI equations are studied to
prove the equilibrium of games. For the risk-sensitive stochastic games with
continuous-time Markov chains, Ghosh, Kumarb and Pal [18] study infinite-
horizon discounted and ergodic cost risk-sensitive zero-sum stochastic games
for countable Markov chains and bounded transition/cost rates, and prove the
existence of the value and saddle-point equilibrium in the class of Markov
strategies under suitable conditions. For the case of unbounded transition
rates, Wei studies a zero-sum game with continuous-time Markov jump processes
under the risk-sensitive finite-horizon criterion with bounded costs [44].
However, there is no study about risk-sensitive stochastic games with
unbounded payoff rates and general spaces, which is still an open problem as
pointed out by [44].

In this paper, we study a two-person zero-sum continuous-time stochastic game
with exponential utility function, where the state and action spaces are Borel
and the payoff rate, transition rates and terminal reward functions are
unbounded. We study this game in the history-dependent policy space. We first
establish the corresponding Shapley equation for this game and study the
existence of the solution with an approximation technique. Then we further
prove the existence of the Nash equilibrium with Markov policies by utilizing
the Shapley equation and the extension of the Dynkin’s formula. We also verify
that the value of the game is the unique solution to the Shapley equation.
Moreover, we develop a value iteration-type algorithm to approach to the
solution to the Shapley equation, by iteratively solving a series of matrix
games with linear programming techniques. Finally, we give two examples to
demonstrate the main results of our paper. The main contribution of this paper
is that it is the first work to study the risk-sensitive stochastic game with
Borel spaces and unbounded transition and payoff rates, which is the most
general model compared with the literature work like [7, 44] and answers the
open problem raised by [44]. Another noteworthy contribution is that an
iteration algorithm is developed in this paper to compute the Nash equilibrium
of the risk-sensitive stochastic game, which is important while rarely
presented in the literature work. We also guarantee the convergence of the
iteration algorithm by a special contraction operator in the risk-sensitive
stochastic game. Since the contraction operator in our game model is not a
one-step contraction operator widely used in discounted MDPs, the convergence
proof is not trivial and may shed some light on the computation issues of
risk-sensitive stochastic games.

The rest of the paper is organized as follows. In Section 2, we introduce the
model formulation of risk-sensitive continuous-time stochastic games (CTSGs).
After giving technical preliminaries in Section 3, we derive the main results
of this paper in Section 4. In Section 5, we develop a value iteration-type
algorithm for solving the value and the Nash equilibrium of the game and prove
the convergence. In Section 6, we verify the main results by two examples.
Finally, we conclude this paper in Section 7.

##  2 The model of stochastic games

Notation: For any Borel space $X$ endowed with the Borel $\sigma$-algebra
${\cal B}(X)$, we will denote by $I_{E}$ the indicator function on any subset
$E$ of $X$, by $\delta_{z}(dx)$ the Dirac measure at point $z\in X$, by
$\mathbb{B}_{1}(X)$ the set of all bounded Borel measurable functions $u$ on
$X$ with the norm $\|u\|:=\sup_{x\in X}|u(x)|$, and by $P(X)$ the set of all
probability measures on ${\cal B}(X)$.

The model of two-person zero-sum risk-sensitive stochastic games for
continuous-time Markov chains is a five-tuple as below.

$\mathbb{M}:=\\{S,(A,A(x)\in{\cal B}(A),x\in S),(B,B(x)\in{\cal B}(B),x\in
S),r(x,a,b),q(dy|x,a,b),{{g(x)}}\\},$ (2.1)

consisting of the following elements:

  * (a)

a Borel space $S$, called the state space of the games;

  * (b)

Borel spaces $A$ and $B$ for players 1 and 2 respectively, called the action
spaces of the players in the games; $A(x)$ and $B(x)$ denote the sets of
actions available to players 1 and 2 respectively when the system is at state
$x\in S$;

  * (c)

a Borel measurable function $r(x,a,b)$ on $K$, called the payoff rate, where
$K:=\\{(x,a,b)|x\in S,a\in A(x),b\in B(x)\\}$ is assumed in
$\mathcal{B}(S\times A\times B);$ that is, $r(x,a,b)$ is the reward rate for
player 1 and the cost rate for player 2;

  * (d)

transition rates $q(dy|x,a,b)$, a Borel signed measure on ${\cal B}(S)$ given
$K$, satisfying that $q(D|x,a,b)\geq 0$ for all $D\in{\cal B}(S)$ with
$(x,a,b)\in K$ and $x\not\in D$, being conservative in the sense of
$q(S|x,a,b)\equiv 0$, and stable in that of

$\displaystyle q^{*}(x):=\sup_{a\in A(x),b\in B(x)}q(x,a,b)<\infty,\ \ \ \
\forall\ x\in S,$ (2.2)

where $q(x,a,b):=-q(\\{x\\}|x,a,b)\geq 0$ for any $(x,a,b)\in K$;

  * (e)

the real-valued terminal reward function $g(x)$ is measurable on $S$.

Next, we give an informal description of the evolution of CTSGs with model
(2.1).

Roughly speaking, CTSGs evolve as follows: Two players observe states of a
system continuously in time. If the system is at state $x_{t}$ at time $t$,
player 1 chooses an action $a_{t}\in A(x_{t})$ according to a given policy,
player 2 chooses an action $b_{t}\in B(x_{t})$ according to a given policy
simultaneously, as a consequence of which, the following happens:

(i) An payoff for player 1 takes place at the rate $r(x_{t},a_{t},b_{t})$;

(ii) After a random sojourn time (i.e., the holding time at state $x_{t}$),
the system jumps to a set $D$ ($x_{t}\not\in D$) of states with the transition
probability $\frac{q(D|x_{t},a_{t},b_{t})}{q(x_{t},a_{t},b_{t})}$ determined
by the transition rates $q(dy|x_{t},a_{t},b_{t})$. The distribution function
of the sojourn time is $(1-e^{-\int_{t}^{t+\delta}q(x_{s},a_{s},b_{s})ds})$,
where $\delta$ is the sojourn time at state $x_{t}$.

To formalize what is described above, below we describe the construction of
CTSGs under possibly randomized history-dependent policies.

To construct the process of the underlying dynamic game, we introduce some
notations: Let $\Omega_{0}:=(S\times(0,\infty))^{\infty}$,
$\Omega_{k}:=(S\times(0,\infty))^{k}\times
S\times(\\{\infty\\}\times\\{\Delta\\})^{\infty}$ for $k\geq 1$ and some
$\Delta\not\in S$, $\Omega:=\cup_{k=0}^{\infty}\Omega_{k}$, $\mathcal{F}$ the
Borel $\sigma$-algebra on the Borel space $\Omega$. Then, we obtain the
measurable space $(\Omega,\mathcal{F})$. For each $k\geq 1$, and sample
$\omega:=(x_{0},\delta_{1},x_{1},\ldots,\delta_{k},x_{k},\ldots)\in\Omega$,
define

$T_{k}(\omega):=\delta_{1}+\delta_{2}+\ldots+\delta_{k},\
T_{\infty}(\omega):=\lim_{k\rightarrow\infty}T_{k}(\omega),\ {\rm and}\
X_{k}(\omega):=x_{k}.$ (2.3)

In what follows, the argument $\omega$ is always omitted except some special
informational statements. Then, we define the state process $\\{x_{t},t\geq
0\\}$ on $(\Omega,\mathcal{F})$ by

$\displaystyle x_{t}:=\sum_{k\geq 0}I_{\\{T_{k}\leq
t<T_{k+1}\\}}X_{k}+I_{\\{t\geq T_{\infty}\\}}\Delta,\ \ \ \ {\rm for}\ t\geq
0,\ \ ({\rm with}\ T_{0}:=0).$ (2.4)

Obviously, $x_{t}(\omega)$ is right-continuous on $[0,\infty)$. We denote
$x_{t-}(\omega):=\lim_{s\to t-}x_{s}(\omega)$. Here we have used the
convenience that $0\times z=0$ and $0+z=z$ for all $z\in
S_{\Delta}:=S\cup\\{\Delta\\}$.

For each fixed
$\omega=(x_{0},\delta_{1},x_{1},\ldots,\delta_{k},x_{k},\ldots)\in\Omega$,
from (2.3), we see that $T_{k}(\omega)$ ($k\geq 1$) denotes the $k$-th jump
moment of $\\{x_{t},t\geq 0\\}$, $X_{k-1}(\omega)=x_{k-1}$ is the state of the
process on $[T_{k-1}(\omega),T_{k}(\omega))$,
$\delta_{k}=T_{k}(\omega)-T_{k-1}(\omega)$ plays the role of sojourn time at
state $x_{k-1}$, and the sample path $\\{x_{t}(\omega),t\geq 0\\}$ has at most
denumerable states $x_{k}$, $k=0,1,\ldots$. We do not intend to consider the
controlled process $\\{x_{t},t\geq 0\\}$ after moment $T_{\infty}$, and thus
view it to be absorbed in the cemetery state $\Delta$. Hence, we write
$A_{\Delta}:=A\cup\\{a_{\Delta}\\}$, $B_{\Delta}:=B\cup\\{a_{\Delta}\\}$,
$A(\Delta):=\\{a_{\Delta}\\}$, $B(\Delta):=\\{b_{\Delta}\\}$,
$q(\cdot|\Delta,a_{\Delta},b_{\Delta}):\equiv 0$,
$r(\Delta,a_{\Delta},b_{\Delta}):\equiv 0$, where $a_{\Delta}$ and
$b_{\Delta}$ are isolated points.

To precisely define the optimality criterion, we need to introduce the concept
of a policy for each player below, which is an equivalent expression of that
in [23, 25, 40].

######  Definition 2.1.

A _(history-dependent) policy $\pi_{1}$_ for player 1 is determined by a
sequence $\\{\pi_{1}^{k},k\geq 0\\}$ such that, for $t\geq 0$ and
$\omega=(x_{0},\delta_{1},x_{1},\ldots,\delta_{k},x_{k},\ldots)\in\Omega$,

$\displaystyle\pi_{1}(da|\omega,t)$ $\displaystyle=$ $\displaystyle
I_{\\{0\\}}(t)\pi^{0}_{1}(da|x_{0},0)+\sum_{k\geq 0}I_{\\{T_{k}<t\leq
T_{k+1}\\}}\pi_{1}^{k}(da|x_{0},\delta_{1},x_{1},\ldots,\delta_{k},x_{k},t-T_{k})$
$\displaystyle+I_{\\{t\geq T_{\infty}\\}}\delta_{a_{\Delta}}(da),$

where $\pi_{1}^{0}(da|x_{0},0)$ is a stochastic kernel on $A$ given $S$,
$\pi_{1}^{k}(k\geq 1)$ are stochastic kernels on $A$ given
$(S\times(0,\infty))^{k+1}$, such that $\pi_{1}^{k}(A(x_{k})|\cdot)\equiv 1$
for all $k\geq 0$.

A policy $\pi_{1}(da|\omega,t)$ is called Markovian if the corresponding
kernels $\pi_{1}^{k}$ satisfy that
$\pi_{1}^{k}(da|x_{0},\delta_{1},x_{1},\ldots,\delta_{k},x_{k},t-T_{k})=:\pi_{1}^{k}(da|x_{k},t)$
(depending only on the current states $x_{k}$ and time $t$) for all
$k=0,1,\ldots.$ We denote such a Markov policy by $\pi_{1}(da|x,t)$ for
informational implication. Note that $\pi_{1}(da|\omega,t)$ and
$\pi_{1}(da|x,t)$ are not time-homogeneous policies.

We denote by $\Pi_{1}$ and $\Pi_{1}^{m}$ the sets of all history-dependent
policies $\pi_{1}$ and Markov ones respectively for player 1. The
corresponding sets $\Pi_{2}$ and $\Pi_{2}^{m}$ of all history-dependent
policies and all Markov policies for player 2 are respectively defined
similarly, with $B(x)$ in lieu of $A(x)$.

For any initial distribution $\gamma$ on $S$ and pair of policies
$(\pi_{1},\pi_{2})\in\Pi_{1}\times\Pi_{2}$, as shown by the Ionescu Tulcea
theorem (e.g., Proposition 7.45 in [8]), we see that there exists a unique
probability measure $\mathbb{P}_{\gamma}^{\pi_{1},\pi_{2}}$ (depending on
$\gamma$ and $(\pi_{1},\pi_{2})$) on $(\Omega,\mathcal{F})$. Let
$\mathbb{E}_{\gamma}^{\pi_{1},\pi_{2}}$ be the corresponding expectation
operator. In particular, $\mathbb{E}_{\gamma}^{\pi_{1},\pi_{2}}$ and
$\mathbb{P}_{\gamma}^{\pi_{1},\pi_{2}}$ are respectively written as
$\mathbb{E}_{x}^{\pi_{1},\pi_{2}}$ and $\mathbb{P}_{x}^{\pi_{1},\pi_{2}}$ when
$\gamma$ is the Dirac measure at an initial state $x$ in $S$.

Fix any finite horizon $T>0$. For each pair of policies
$(\pi_{1},\pi_{2})\in\Pi_{1}\times\Pi_{2}$ and state $x\in S$, we define the
$T$-horizon _risk-sensitive_ value $J(\pi_{1},\pi_{2},0,x)$ of the continuous-
time dynamic game by

$\displaystyle
J(\pi_{1},\pi_{2},0,x):=\mathbb{E}_{\gamma}^{\pi_{1},\pi_{2}}\left[e^{\theta\int_{0}^{T}\int_{A\times
B}r(x_{t},a,b)\pi_{1}(da|\omega,t)\pi_{2}(db|\omega,t)dt+{\theta
g(x_{T})}}|x_{0}=x\right],$ (2.5)

provided that the integral is well defined, where $\theta$ is a constant
called the risk-sensitive parameter. In the following arguments, we assume
that $\theta>0$ which indicates a risk-averse preference. For the other case
of $\theta<0$ with risk-seeking preference, the corresponding results can be
obtained with $r$ being replaced by $-r$, and thus the similar arguments are
omitted.

Note that the process $\\{x_{t},t\geq 0\\}$ on
$(\Omega,\mathcal{F},\mathbb{P}^{\pi_{1},\pi_{2}}_{\gamma})$ may not be
Markovian since the policies $\pi_{1}$ or $\pi_{2}$ can depend on histories
$(x_{0},\delta_{1},x_{1},\ldots,\delta_{k},x_{k})$. However, for each
$\pi_{1}\in\Pi^{m}_{1}$ and $\pi_{2}\in\Pi^{m}_{2}$, it is well known (e.g.
[13]) that $\\{x_{t},t\geq 0\\}$ is a Markov process on
$(\Omega,\mathcal{F},\mathbb{P}^{\pi_{1},\pi_{2}}_{\gamma})$, and thus for
each $x\in S$ and $t\in[0,T]$, the following expression

$J(\pi_{1},\pi_{2},t,x):=\mathbb{E}_{\gamma}^{\pi_{1},\pi_{2}}\left[e^{\theta\int_{t}^{T}\int_{A\times
B}r(x_{s},a,b)\pi_{1}(da|x_{s},s)\pi_{2}(db|x_{s},s)ds+{\theta
g(x_{T})}}|x_{t}=x\right],$ (2.6)

for $\pi_{1}\in\Pi_{1}^{m},\ \pi_{2}\in\Pi_{2}^{m},$ is well defined (when the
integral exists), and it is called the risk-sensitive value of policy pair
$(\pi_{1},\pi_{2})$ from the horizon $t$ to $T$.

As is well known, the functions on $S$ defined as

$\displaystyle
L(x):=\sup_{\pi_{1}\in\Pi_{1}}\inf_{\pi_{2}\in\Pi_{2}}J(\pi_{1},\pi_{2},0,x),\
\ \ {\rm and}\ \
U(x):=\inf_{\pi_{2}\in\Pi_{2}}\sup_{\pi_{1}\in\Pi_{1}}J(\pi_{1},\pi_{2},0,x)$

are called the lower value and the upper value of the stochastic game,
respectively. It is clear that

$\displaystyle L(x)\leq U(x),\ \quad\forall\ x\in S.$

######  Definition 2.2.

If $L(x)=U(x)$ for all $x\in S$, then the common function is called the value
function of the game model $\mathbb{M}$ and is denoted by $\mathbb{M}(x)$.

######  Definition 2.3.

Suppose that the game $\mathbb{M}$ has a value function $\mathbb{M}(x)$. Then
a policy $\pi_{1}^{*}$ in $\Pi_{1}$ is said to be optimal for player 1 if

$\displaystyle\inf_{\pi_{2}\in\Pi_{2}}J(\pi_{1}^{*},\pi_{2},0,x)=\mathbb{M}(x),\
\ \ \ \forall\ x\in S.$

Similarly, $\pi_{2}^{*}\in\Pi_{2}$ is optimal for player 2 if

$\displaystyle\sup_{\pi_{1}\in\Pi_{1}}J(\pi_{1},\pi_{2}^{*},0,x)=\mathbb{M}(x),\
\ \ \ \forall\ x\in S.$

If $\pi_{k}^{*}\in\Pi_{k}$ is optimal for player $k$, $k=1,2$, then
$(\pi_{1}^{*},\pi_{2}^{*})$ is called a Nash equilibrium of the game.

The aim of this paper is to give conditions for the existence and the
computation of a Nash equilibrium.

##  3 Preliminaries

This section provides some preliminary facts for our arguments below. Since
the transition rate $q(dy|x,a,b)$ and payoff rate $r(x,a,b)$ are allowed to be
unbounded, we next give conditions for the non-explosion of $\\{x_{t},t\geq
0\\}$ and finiteness of $J(\pi_{1},\pi_{2},0,x)$.

######  Assumption 3.1.

There exist a real-valued Borel measurable function $V_{0}(x)\geq 1$ on $S$
and positive constants $\rho_{0},L_{0},M_{0}$, such that

(i)

$\int_{S}V_{0}(y)q(dy|x,a,b)\leq\rho_{0}V_{0}(x)$ _for all_ $(x,a,b)\in K$;

(ii)

$q^{*}(x)\leq L_{0}V_{0}(x)$ for all $x\in S$, where $q^{*}(x)$ is as in
(2.2);

(iii)

$|r(x,a,b)|\leq M_{0}+\frac{\sqrt{2}}{2}\sqrt{\ln V_{0}(x)}$ for all
$(x,a,b)\in K$ and $|g(x)|\leq M_{0}+\frac{\sqrt{2}}{2}\sqrt{\ln V_{0}(x)}$
for all $x\in S$.

######  Lemma 3.1.

Under Assumption 3.1 (i,ii), for each
$(\pi_{1},\pi_{2})\in\Pi_{1}\times\Pi_{2}$, $x\in S,t\geq 0$, the following
assertions hold.

(a)

$\mathbb{P}_{x}^{\pi_{1},\pi_{2}}(x_{t}\in S)=1$,
$\mathbb{P}_{x}^{\pi_{1},\pi_{2}}(T_{\infty}=\infty)=1$, and
$\mathbb{P}_{x}^{\pi_{1},\pi_{2}}(x_{0}=x)=1$.

(b)

$\mathbb{E}_{x}^{\pi_{1},\pi_{2}}[V_{0}(x_{t})]\leq e^{\rho_{0}t}V_{0}(x)$,
and $\mathbb{E}_{\gamma}^{\pi_{1},\pi_{2}}[V_{0}(x_{t})\big{|}x_{s}=x]\leq
e^{\rho_{0}(t-s)}V_{0}(x)$ for $t\geq s\geq 0$ when $(\pi_{1},\pi_{2})$ is in
$\Pi^{m}_{1}\times\Pi_{2}^{m}$.

(c)

If, in addition, Assumption 3.1 (iii) is satisfied, then for each $t\geq 0$

(c1)

$e^{-\theta\left[Te^{\rho_{0}T}+M_{0}T+e^{\rho_{0}T}+M_{0}\right]V_{0}(x)}\leq
J(\pi_{1},\pi_{2},0,x)\leq LV_{0}(x)$,
where $L:=e^{2T\theta(M_{0}+T\theta)+2\theta(M_{0}+\theta)+\rho_{0}T};$

(c2)

$e^{-\theta\left[Te^{\rho_{0}T}+M_{0}T+e^{\rho_{0}T}+M_{0}\right]V_{0}(x)}\leq
J(\pi_{1},\pi_{2},t,x)\leq LV_{0}(x)$ for
$(\pi_{1},\pi_{2})\in\Pi^{m}_{1}\times\Pi_{2}^{m}$.

###### Proof.

Parts (a) and (b) follow from any reference of [20, 23, 25, 27, 40]. We next
prove part (c). Since $|r(x,a,b)|\leq\frac{\sqrt{2}}{2}\sqrt{\ln
V_{0}(x)}+M_{0}\leq T\theta+\frac{\ln\sqrt{V_{0}(x)}}{2T\theta}+M_{0}$ and
$|g(x)|\leq\frac{\sqrt{2}}{2}\sqrt{\ln
V_{0}(x)}+M_{0}\leq\theta+\frac{\ln\sqrt{V_{0}(x)}}{2\theta}+M_{0}$, we have
$e^{T\theta|r(x,a,b)|}\leq e^{2T\theta|r(x,a,b)|}\leq
e^{2T\theta(M_{0}+T\theta)}\sqrt{V_{0}(x)}$ and $e^{\theta|g(x)|}\leq
e^{2\theta|g(x)|}\leq e^{2\theta(M_{0}+\theta)}\sqrt{V_{0}(x)}$. Using the
Jensen inequality with respect to the probability measure $\frac{dt}{T}$ on
${\cal{B}}([0,T])$ and Cauchy-Buniakowsky-Schwarz Inequality, by (2.5) we have

$\displaystyle\mathbb{E}_{x}^{\pi_{1},\pi_{2}}\left[e^{\theta\int_{0}^{T}\int_{A\times
B}r(x_{t},a,b)\pi_{1}(da|\omega,t)\pi_{2}(db|\omega,t)dt+\theta
g(x_{T})}\right]$ (3.1) $\displaystyle\leq$
$\displaystyle\mathbb{E}_{x}^{\pi_{1},\pi_{2}}\left[e^{\theta\int_{0}^{T}\int_{A\times
B}|r(x_{t},a,b)|\pi_{1}(da|\omega,t)\pi_{2}(db|\omega,t)dt+\theta|g(x_{T})|}\right]$
$\displaystyle=$
$\displaystyle\mathbb{E}_{x}^{\pi_{1},\pi_{2}}\left\\{e^{\theta\int_{0}^{T}\left[\int_{A\times
B}|r(x_{t},a,b)|\pi_{1}(da|\omega,t)\pi_{2}(db|\omega,t)+\frac{1}{T}|g(x_{T})|\right]dt}\right\\}$
$\displaystyle=$
$\displaystyle\mathbb{E}_{x}^{\pi_{1},\pi_{2}}\left\\{e^{\theta
T\int_{0}^{T}\frac{1}{T}\left[\int_{A\times
B}|r(x_{t},a,b)|\pi_{1}(da|\omega,t)\pi_{2}(db|\omega,t)+\frac{1}{T}|g(x_{T})|\right]dt}\right\\}$
$\displaystyle\leq$
$\displaystyle\mathbb{E}_{x}^{\pi_{1},\pi_{2}}\left\\{\frac{1}{T}\int_{0}^{T}e^{T\theta\left[\int_{A\times
B}|r(x_{t},a,b)|\pi_{1}(da|\omega,t)\pi_{2}(db|\omega,t)+\frac{1}{T}|g(x_{T})|\right]}dt\right\\}$
$\displaystyle\leq$
$\displaystyle\mathbb{E}_{x}^{\pi_{1},\pi_{2}}\left\\{\frac{1}{T}\int_{0}^{T}e^{2T\theta(M_{0}+T\theta)}\sqrt{V_{0}(x_{t})}e^{2\theta(M_{0}+\theta)}\sqrt{V_{0}(x_{T})}dt\right\\}$
$\displaystyle=$
$\displaystyle\frac{1}{T}e^{2T\theta(M_{0}+T\theta)+2\theta(M_{0}+\theta)}\int_{0}^{T}\mathbb{E}_{x}^{\pi_{1},\pi_{2}}\left[\sqrt{V_{0}(x_{t})}\sqrt{V_{0}(x_{T})}\right]dt$
$\displaystyle\leq$
$\displaystyle\frac{1}{T}e^{2T\theta(M_{0}+T\theta)+2\theta(M_{0}+\theta)}\int_{0}^{T}\sqrt{\mathbb{E}_{x}^{\pi_{1},\pi_{2}}V_{0}(x_{t})\mathbb{E}_{x}^{\pi_{1},\pi_{2}}V_{0}(x_{T})}dt$
$\displaystyle\leq$
$\displaystyle\frac{1}{T}e^{2T\theta(M_{0}+T\theta)+2\theta(M_{0}+\theta)}\int_{0}^{T}\sqrt{e^{\rho_{0}t}V_{0}(x)e^{\rho_{0}T}V_{0}(x)}dt$
$\displaystyle\leq$
$\displaystyle\frac{1}{T}e^{2T\theta(M_{0}+T\theta)+2\theta(M_{0}+\theta)}\int_{0}^{T}e^{\rho_{0}T}V_{0}(x)dt$
$\displaystyle\leq$ $\displaystyle
e^{2T\theta(M_{0}+T\theta)+2\theta(M_{0}+\theta)+\rho_{0}T}V_{0}(x).$

On the other hand, we have

$\displaystyle\mathbb{E}_{x}^{\pi_{1},\pi_{2}}\left[e^{\theta\int_{0}^{T}\int_{A\times
B}r(x_{t},a,b)\pi_{1}(da|\omega,t)\pi_{2}(db|\omega,t)dt+\theta
g(x_{T})}\right]$ $\displaystyle\geq$ $\displaystyle
e^{\mathbb{E}_{x}^{\pi_{1},\pi_{2}}\left[\theta\int_{0}^{T}\int_{A\times
B}r(x_{t},a,b)\pi_{1}(da|\omega,t)\pi_{2}(db|\omega,t)dt+\theta
g(x_{T})\right]}$ $\displaystyle\geq$ $\displaystyle
e^{\mathbb{E}_{x}^{\pi_{1},\pi_{2}}\left[-\theta\int_{0}^{T}\int_{A\times
B}|r(x_{t},a,b)|\pi_{1}(da|\omega,t)\pi_{2}(db|\omega,t)dt-\theta|g(x_{T})|\right]}$
$\displaystyle\geq$ $\displaystyle
e^{\mathbb{E}_{x}^{\pi_{1},\pi_{2}}\left[-\theta\int_{0}^{T}\int_{A\times
B}(\ln\sqrt{V_{0}(x_{t})}+M_{0})\pi_{1}(da|\omega,t)\pi_{2}(db|\omega,t)dt-\theta(\ln\sqrt{V_{0}(x_{T})}+M_{0})\right]}$
$\displaystyle\geq$ $\displaystyle
e^{\mathbb{E}_{x}^{\pi_{1},\pi_{2}}\left[-\theta\int_{0}^{T}(\sqrt{V_{0}(x_{t})}+M_{0})dt-\theta(\sqrt{V_{0}(x_{T})}+M_{0})\right]}$
$\displaystyle\geq$ $\displaystyle
e^{\mathbb{E}_{x}^{\pi_{1},\pi_{2}}\left[-\theta\int_{0}^{T}(V_{0}(x_{t})+M_{0})dt-\theta(V_{0}(x_{T})+M_{0})\right]}$
$\displaystyle=$ $\displaystyle
e^{\left[-\theta\int_{0}^{T}(\mathbb{E}_{x}^{\pi_{1},\pi_{2}}V_{0}(x_{t})+M_{0})dt-\theta(\mathbb{E}_{x}^{\pi_{1},\pi_{2}}V_{0}(x_{T})+M_{0})\right]}$
$\displaystyle\geq$ $\displaystyle
e^{\left[-\theta\int_{0}^{T}(e^{\rho_{0}t}V_{0}(x)+M_{0})dt-\theta(e^{\rho_{0}T}V_{0}(x)+M_{0})\right]}$
$\displaystyle\geq$ $\displaystyle
e^{-\theta\left[Te^{\rho_{0}T}V_{0}(x)+M_{0}T+e^{\rho_{0}T}V_{0}(x)+M_{0}\right]}$
$\displaystyle\geq$ $\displaystyle
e^{-\theta\left[Te^{\rho_{0}T}+M_{0}T+e^{\rho_{0}T}+M_{0}\right]V_{0}(x)},$

which, together with (3.1), implies (c1). Similarly, we see that (c2) is also
true. ∎

######  Remark 3.1.

It can be seen from the proof, when $g(x)=0$ on $S$ and item (iii) in
Assumption 3.1 is weaken to $|r(x,a,b)|\leq M_{0}+\frac{1}{2T\theta}\ln
V_{0}(x)$, item (c) in Lemma 3.1 still holds.

Lemma 3.1 gives conditions for the finiteness of $J(\pi_{1},\pi_{2},t,x)$ as
well as the non-explosion of $\\{x_{t},t\geq 0\\}$. In order to deal with the
game problem for history-dependent policies, we need the analog of the Ito-
Dynkin’s formula in [22] for possible non-Markov processes $\\{x_{t},t\geq
0\\}$ and functions $\varphi(\omega,t,x)$ with an additional element
$\omega\in\Omega$. To do so, we recall some concepts. Take the right-
continuous family of $\sigma$-algebras $\\{{\cal F}_{t}\\}_{t\geq 0}$ with
${\cal F}_{t}:=\sigma(\\{T_{k}\leq s,X_{k}\in D\\}:D\in{\cal{B}}(S),s\leq
t,k\geq 0)$. As in [25, 26, 40], let $\cal P$ be the $\sigma$-algebra of
predictable sets on $\Omega\times[0,\infty)$ related to $\\{{\cal
F}_{t}\\}_{t\geq 0}$, that is,
${\cal{P}}:=\sigma(B\times[0,\infty),C\times(s,\infty):B\in{\cal{F}}_{0},C\in{\cal{F}}_{s-},s>0)$
with ${\cal{F}}_{s-}:=\bigvee_{t<s}{\cal{F}}_{t}:=\sigma({\cal{F}}_{t},t<s)$.
A real-valued function on $\Omega\times[0,\infty)$ is called predictable if it
is measurable with respect to $\cal P$.

Denote by $m_{L}$ the Lebesgue’s measure on $[0,T]$, and by $\mathbb{B}_{\cal
P}(\Omega\times[0,T]\times S)$ the set of real-valued and
${\cal{P}}\times{\cal{B}}(S)$-measurable functions $\varphi$ with the
following features: Given any $x\in
S,(\pi_{1},\pi_{2})\in\Pi_{1}\times\Pi_{2}$, and a.s. $\omega\in\Omega$ with
respect to $\mathbb{P}_{x}^{\pi_{1},\pi_{2}}$, there exists a Borel subset
$E_{(\varphi,\omega,x,\pi_{1},\pi_{2})}$ (depending on the
$\varphi,\omega,x,(\pi_{1},\pi_{2})$) of $[0,T]$ such that the partial
derivative $\frac{\partial\varphi(\omega,t,x)}{\partial t}$ with respect to
$t$ exists for every $t\in E_{(\varphi,\omega,x,\pi_{1},\pi_{2})}$ and
$m_{L}(E_{(\varphi,\omega,x,\pi_{1},\pi_{2})}^{c})=0$. Obviously, if a
function $\varphi(\omega,t,x)$ in $\mathbb{B}_{\cal P}(\Omega\times[0,T]\times
S)$ is independent of $\omega$ (written as $\varphi(t,x)$), then the
corresponding $E_{(\varphi,\omega,x,\pi_{1},\pi_{2})}$ is independent of
$(\omega,\pi_{1},\pi_{2})$, which will be denoted by $E_{(\varphi,x)}$ for
simplicity.

Next we state the extension of the Dynkin’s formula by Lemma 3.2. To do so, we
introduce the following conditions and notations.

######  Assumption 3.2.

There exist a real-valued Borel measurable function $V_{1}(x)\geq 1$ on $S$,
and positive constants $\rho_{1},b_{1}$, and $M_{1}$, such that

(i)

$\int_{S}V_{1}^{2}(y)q(dy|x,a,b)\leq\rho_{1}V_{1}^{2}(x)+b_{1}$ _for all_
$(x,a,b)\in K$;

(ii)

$V_{0}^{2}(x)\leq M_{1}V_{1}(x)$ for all $x\in S$, with $V_{0}(x)$ satisfying
Assumption 3.1.

Assumption 3.2 is used to give a domain for the Dynkin’s formula below, and it
is obviously satisfied when the transition rates are bounded [19, 35, 36, 46].

Given the $V_{k}(k=0,1)$ as in Assumption 3.2 and any Borel set $Z$, a real-
valued function $\varphi$ on $Z\times S$ is called $V_{k}$-bounded if the
$V_{k}$-weighted norm of $\varphi$, $\|\varphi\|_{V_{k}}:=\sup_{(z,x)\in
Z\times S}\frac{|\varphi(z,x)|}{V_{k}(x)}$, is finite. We denote by
$\mathbb{B}_{V_{k}}(Z\times S)$ the Banach space of all $V_{k}$-bounded
functions on $Z\times S$. When $V_{k}(x)\equiv 1$ for all $x\in S$,
$\mathbb{B}_{1}(Z\times S)$ is the space of all bounded functions. In
particular, take $Z=\Omega\times[0,T]$ or $[0,T]$, we define

$\mathbb{B}_{V_{0},V_{1}}^{1}(\Omega\times[0,T]\times
S):=\left\\{\varphi\in\mathbb{B}_{V_{0}}(\Omega\times[0,T]\times
S)\cap\mathbb{B}_{\cal{P}}(\Omega\times[0,T]\times
S)\mid\frac{\partial\varphi}{\partial
t}\in\mathbb{B}_{V_{1}}(\Omega\times[0,T]\times S)\right\\},$

and then

$\mathbb{B}_{V_{0},V_{1}}^{1}([0,T]\times
S):=\left\\{\varphi\in\mathbb{B}_{V_{0}}([0,T]\times
S)\cap\mathbb{B}_{\cal{P}}([0,T]\times S)\mid\frac{\partial\varphi}{\partial
t}\in\mathbb{B}_{V_{1}}([0,T]\times S)\right\\}.$ (3.2)

######  Lemma 3.2.

Suppose Assumptions 3.1 and 3.2 are satisfied. Then, for each
$(s,x)\in[0,T]\times S$, the following assertions hold.

(a)

(The extension of the Dynkin’s formula): For every
$(\pi_{1},\pi_{2})\in\Pi_{1}\times\Pi_{2}$ and
${\varphi}\in\mathbb{B}_{V_{0},V_{1}}^{1}(\Omega\times[0,T]\times S)$,

$\displaystyle\mathbb{E}_{x}^{\pi_{1},\pi_{2}}\left[\int_{0}^{T}\left(\varphi^{\prime}(\omega,t,x_{t})+\int_{S}\int_{A\times
B}{\varphi}(\omega,t,y)q(dy|x_{t},a,b)\pi_{1}(da|\omega,t)\pi_{2}(db|\omega,t)\right)dt\right]$
$\displaystyle\ \
=\mathbb{E}_{x}^{\pi_{1},\pi_{2}}\left[{\varphi}(\omega,T,x_{T})\right]-\mathbb{E}_{x}^{\pi_{1},\pi_{2}}\left[{\varphi}(\omega,0,x)\right],$

where $\\{x_{t},t\geq 0\\}$ may be not Markovian since policies $\pi_{1}$ and
$\pi_{2}$ may depend on histories.

(b)

(The Dynkin’s formula): For each
$(\pi_{1},\pi_{2})\in\Pi_{1}^{m}\times\Pi_{2}^{m}$, and
${\varphi}\in\mathbb{B}_{V_{0},V_{1}}^{1}([0,T]\times S)$,

$\displaystyle\mathbb{E}_{\gamma}^{\pi_{1},\pi_{2}}\left\\{\int_{s}^{T}\left[\left(e^{\int_{s}^{t}\theta
r(x_{v},\pi_{1}^{v},\pi_{2}^{v})dv{}}{\varphi}(t,x_{t})\right)^{\prime}\right.\right.$
$\displaystyle\left.\left.+\int_{S}e^{\int_{s}^{t}\theta
r(x_{v},\pi_{1}^{v},\pi_{2}^{v})dv{}}{\varphi}(t,y)q(dy|x_{t},\pi_{1}^{t},\pi_{2}^{t})\right]dt\Big{|}x_{s}=x\right\\}$
$\displaystyle=$
$\displaystyle\mathbb{E}_{\gamma}^{\pi_{1},\pi_{2}}\left[e^{\int_{s}^{T}\theta
r(x_{t},\pi_{1}^{t},\pi_{2}^{t})dt{}}{\varphi}(T,x_{T})\Big{|}x_{s}=x\right]-{\varphi}(s,x),\
\ \ \forall\ s\in[0,T],$

where, the notation $u(x,\pi_{1}^{t},\pi_{2}^{t})$ means that

$u(x,\pi_{1}^{t},\pi_{2}^{t}):=\int_{A(x)}\int_{B(x)}u(x,a,b)\pi_{1}(da|x,t)\pi_{2}(db|x,t),\
\ \ \forall\ x\in S,t\geq 0.$

###### Proof.

See Theorem 3.1 in [27]. ∎

##  4 The existence of Nash equilibria

In this section, we prove the existence of a Nash equilibrium and a solution
to the following Shapley equation (4.1) for the finite-horizon stochastic game
with the risk-sensitive criterion. The proofs are shown in three steps as
follows: 1) consider the case of bounded transition and payoff rates, 2) deal
with the case of unbounded transition rates but nonnegative payoff, and 3)
study the case of unbounded transition and payoff rates.

######  Assumption 4.1.

(i)

For each $x\in S$, $A(x)$ and $B(x)$ are compact;

(ii)

For each $x\in S$ and $D\in{\cal{B}}(S)$, the function $q(D|x,a,b)$ is
continuous in $(a,b)\in A(x)\times B(x)$;

(iii)

For each $x\in S$, the functions $r(x,a,b)$ and $\int_{S}V_{0}(y)q(dy|x,a,b)$
are continuous in $(a,b)\in A(x)\times B(x)$, with $V_{0}(x)$ as in Assumption
3.1.

The following results are for the case of the bounded transition and bounded
payoff rates.

######  Proposition 4.1.

Under Assumption 4.1 suppose that $\|q\|:=\sup_{x\in S}q^{*}(x)$,
$\|r\|:=\sup_{x\in S,a\in A(x),b\in B(x)}|r(x,a,b)|$ and $\|g\|:=\sup_{x\in
S}|g(x)|$ are finite. Then, the following assertions hold.

(a)

There exists a unique $\varphi$ in $\mathbb{B}^{1}_{1,1}([0,T]\times S)$ (that
is, $V_{0}(x)=V_{1}(x)\equiv 1,x\in S$ in $(\ref{eq:B}$)) satisfying the
following Shapley equation for the risk-sensitive criterion of CTSGs on the
finite horizon:

$\displaystyle\begin{cases}\displaystyle{\varphi}^{\prime}(t,x)+\sup_{\phi\in
P(A(x))}\inf_{\psi\in P(B(x))}\left[\theta
r(x,\phi,\psi){\varphi}(t,x)+\int_{S}{\varphi}(t,y)q(dy|x,\phi,\psi)\right]=0,\\\
{\varphi}(T,x)=e^{\theta g(x)},\end{cases}$ (4.1)

for each $x\in S$ and $t\in E_{({\varphi},x)}$ with
$m_{L}(E_{({\varphi},x)}^{c})=0$.

(b)

There exists a pair of Markov policies
$(\hat{\pi}_{1},\hat{\pi}_{2})\in\Pi_{1}^{m}\times\Pi_{2}^{m}$ such that, for
$x\in S,t\in[0,T]$,

$\displaystyle-{\varphi}^{\prime}(t,x)$ $\displaystyle=$ $\displaystyle\theta
r(x,\hat{\pi}^{t}_{1},\hat{\pi}^{t}_{2}){\varphi}(t,x)+\int_{S}{\varphi}(t,y)q(dy|x,\hat{\pi}^{t}_{1},\hat{\pi}^{t}_{2})$
(4.2) $\displaystyle=$ $\displaystyle\sup_{\phi\in P(A(x))}\left[\theta
r(x,\phi_{,}\hat{\pi}^{t}_{2}){\varphi}(t,x)+\int_{S}{\varphi}(t,y)q(dy|x,\phi,\hat{\pi}^{t}_{2})\right]$
(4.3) $\displaystyle=$ $\displaystyle\inf_{\psi\in P(B(x))}\left[\theta
r(x,\hat{\pi}^{t}_{1},\psi){\varphi}(t,x)+\int_{S}{\varphi}(t,y)q(dy|x,\hat{\pi}^{t}_{1},\psi)\right].$
(4.4)

(c)

$\displaystyle{\varphi}(t,x)=\sup_{\pi_{1}\in\Pi_{1}^{m}}\inf_{\pi_{2}\in\Pi_{2}^{m}}J(\pi_{1},\pi_{2},t,x)=\inf_{\pi_{2}\in\Pi_{2}^{m}}\sup_{\pi_{1}\in\Pi_{1}^{m}}J(\pi_{1},\pi_{2},t,x)=J(\hat{\pi}_{1},\hat{\pi}_{2},t,x)$,
for $(t,x)\in[0,T]\times S$.

(d)

The value function $\mathbb{M}(x)$ exists and is equal to ${\varphi}(0,x)$ for
each $x\in S$, and $(\hat{\pi}_{1},\hat{\pi}_{2})$ is a Nash equilibrium.

(e)

If $r(x,a,b)\geq 0$ for all $(x,a,b)\in K$, then ${\varphi}(t,x)$ is
decreasing in $t\in[0,T]$ for any given $x\in S$, which means that
${\varphi}^{\prime}(t,x)\leq 0,a.e.$.

The proof is rather long and is therefore presented in Appendix A.

######  Remark 4.1.

Even $r(x,a,b)\geq 0$ for all $(x,a,b)\in K$, it is not obvious that
${\varphi}(t,x)$ is decreasing in $t\in[0,T]$ for any given $x\in S$. Although

$J(\pi_{1},\pi_{2},t,x)=\mathbb{E}_{\gamma}^{\pi_{1},\pi_{2}}\left[e^{\theta\int_{t}^{T}\int_{A\times
B}r(x_{v},a,b)\pi_{1}(da|x_{v},v)\pi_{2}(db|x_{v},v)dv+{\theta
g(x_{T})}}|x_{t}=x\right]$

and

$J(\pi_{1},\pi_{2},s,x)=\mathbb{E}_{\gamma}^{\pi_{1},\pi_{2}}\left[e^{\theta\int_{s}^{T}\int_{A\times
B}r(x_{v},a,b)\pi_{1}(da|x_{v},v)\pi_{2}(db|x_{v},v)dv+{\theta
g(x_{T})}}|x_{s}=x\right]$

have the same initial state and policies, their initial times $t$ and $s$ are
different. Therefore, it is not easy to compare $J(\pi_{1},\pi_{2},t,x)$ and
$J(\pi_{1},\pi_{2},s,x)$.

Proposition 4.1 shows the existence of a Nash equilibrium under the bounded
transition and payoff rates. We next extend the results in Proposition 4.1 to
the case of unbounded transition rates and nonnegative payoff rates by
approximations.

######  Proposition 4.2.

Under Assumptions 3.1, 3.2 and 4.1, if in addition $r(x,a,b)\geq 0$ for all
$(x,a,b)\in K$ and $g(x)\geq 0$ for all $x\in S$, then the following
assertions hold.

(a)

There exists a unique ${\varphi}$ in $\mathbb{B}^{1}_{V_{0},V_{1}}([0,T]\times
S)$ satisfying the following Shapley equation for the risk-sensitive criterion
of CTSGs on the finite horizon:

$\displaystyle\begin{cases}\displaystyle{\varphi}^{\prime}(t,x)+\sup_{\phi\in
P(A(x))}\inf_{\psi\in P(B(x))}\left[\theta
r(x,\phi,\psi){\varphi}(t,x)+\int_{S}{\varphi}(t,y)q(dy|x,\phi,\psi)\right]=0,\\\
{\varphi}(T,x)={e^{\theta g(x)}},\end{cases}$ (4.5)

for each $x\in S$ and $t\in E_{({\varphi},x)}$ with
$m_{L}(E_{({\varphi},x)}^{c})=0$.

(b)

$\displaystyle{\varphi}(t,x)=\sup_{\pi_{1}\in\Pi_{1}^{m}}\inf_{\pi_{2}\in\Pi_{2}^{m}}J(\pi_{1},\pi_{2},t,x)=\inf_{\pi_{2}\in\Pi_{2}^{m}}\sup_{\pi_{1}\in\Pi_{1}^{m}}J(\pi_{1},\pi_{2},t,x)$
for $(t,x)\in[0,T]\times S$.

(c)

The value function $\mathbb{M}(x)$ exists and equals to ${\varphi}(0,x)$, for
any $x\in S$.

The proof is rather long and is therefore presented in Appendix B.

Next, we use Proposition 4.2 to prove our main results by approximation from
nonnegative payoff rates to the payoff rates that may be unbounded from above
and from below.

######  Theorem 4.1.

Under Assumptions 3.1, 3.2 and 4.1, the following assertions hold.

(a)

There exists a unique ${\varphi}$ in $\mathbb{B}^{1}_{V_{0},V_{1}}([0,T]\times
S)$ satisfying the following Shapley equation for the risk-sensitive criterion
of CTSGs on the finite horizon:

$\displaystyle\begin{cases}\displaystyle{\varphi}^{\prime}(t,x)+\sup_{\phi\in
P(A(x))}\inf_{\psi\in P(B(x))}\left[\theta
r(x,\phi,\psi){\varphi}(t,x)+\int_{S}{\varphi}(t,y)q(dy|x,\phi,\psi)\right]=0,\\\
{\varphi}(T,x)={e^{\theta g(x)}},\end{cases}$ (4.6)

for each $x\in S$ and $t\in E_{({\varphi},x)}$ with
$m_{L}(E_{({\varphi},x)}^{c})=0$.

(b)

There exists a pair of Markov policies
$(\hat{\pi}_{1},\hat{\pi}_{2})\in\Pi_{1}^{m}\times\Pi_{2}^{m}$ such that, for
$(t,x)\in[0,T]\times S$,

$\displaystyle-{\varphi}^{\prime}(t,x)$ $\displaystyle=$ $\displaystyle\theta
r(x,\hat{\pi}^{t}_{1},\hat{\pi}^{t}_{2}){\varphi}(t,x)+\int_{S}{\varphi}(t,y)q(dy|x,\hat{\pi}^{t}_{1},\hat{\pi}^{t}_{2})$
$\displaystyle=$ $\displaystyle\sup_{\phi\in P(A(x))}\left[\theta
r(x,\phi_{,}\hat{\pi}^{t}_{2}){\varphi}(t,x)+\int_{S}{\varphi}(t,y)q(dy|x,\phi,\hat{\pi}^{t}_{2})\right]$
$\displaystyle=$ $\displaystyle\inf_{\psi\in P(B(x))}\left[\theta
r(x,\hat{\pi}^{t}_{1},\psi){\varphi}(t,x)+\int_{S}{\varphi}(t,y)q(dy|x,\hat{\pi}^{t}_{1},\psi)\right]$

(c)

$\displaystyle{\varphi}(t,x)=\sup_{\pi_{1}\in\Pi_{1}^{m}}\inf_{\pi_{2}\in\Pi_{2}^{m}}J(\pi_{1},\pi_{2},t,x)=\inf_{\pi_{2}\in\Pi_{2}^{m}}\sup_{\pi_{1}\in\Pi_{1}^{m}}J(\pi_{1},\pi_{2},t,x)=J(\hat{\pi}_{1},\hat{\pi}_{2},t,x)$
for $(t,x)\in[0,T]\times S$.

(d)

The value function $\mathbb{M}(x)$ exists and is equal to ${\varphi}(0,x)$ for
each $x\in S$, and $(\hat{\pi}_{1},\hat{\pi}_{2})$ is a Nash equilibrium.

###### Proof.

We only prove (a) and the Nash Equilibrium policies
$(\hat{\pi}_{1},\hat{\pi}_{2})$ since the others can be proved as Proposition
4.1. For each $n\geq 1$, define $r_{n}$ on $K$ as follows: for each
$(x,a,b)\in K,$

$r_{n}(x,a,b):=\max\\{-n,r(x,a,b)\\},\quad{g_{n}(x):=\max\\{-n,g(x)\\}},$

which implies that $\displaystyle\lim_{n\to\infty}r_{n}(x,a,b)=r(x,a,b)$,
$\displaystyle\lim_{n\to\infty}g_{n}(x)=g(x)$ and
$k_{n}(x,a,b):=r_{n}(x,a,b)+n\geq 0$ for each $(x,a,b)\in K$ and
$f_{n}(x):=n+g_{n}(x)\geq 0$ for $(t,x)\in[0,T]\times S,n\geq 1$. Moreover, it
follows from Assumption 3.1(iii) that

$-M_{0}-\frac{\sqrt{2}}{2}\sqrt{\ln
V_{0}(x)}\leq\max\\{-n,-M_{0}-\frac{\sqrt{2}}{2}\sqrt{\ln V_{0}(x)}\\}\leq
r_{n}(x,a,b)\leq M_{0}+\frac{\sqrt{2}}{2}\sqrt{\ln V_{0}(x)}.$

Thus, $|r_{n}(x,a,b)|\leq M_{0}+\frac{\sqrt{2}}{2}\sqrt{\ln V_{0}(x)}$ for all
$(x,a,b)\in K$ for all $n\geq 1$. By the same reasoning, we have
$|g_{n}(x)|\leq M_{0}+\frac{\sqrt{2}}{2}\sqrt{\ln V_{0}(x)}$ for all $x\in S$
for all $n\geq 1$. And so Assumptions 3.1, 3.2 and 4.1 still hold for each
model ${\cal{N}}_{n}$ defined by

${\cal{N}}_{n}:=\left\\{S,(A,A(x),x\in S),(B,B(x),x\in
S),k_{n}(x,a,b),q(dy|x,a,b),{f_{n}(x)}\right\\}.$ (4.7)

For any real-valued Borel measurable function $u$ on $K$ and $g$ on
$[0,T]\times S$ , let

$J_{u,g}(t,x):=\sup_{\pi_{1}\in\Pi_{1}^{m}}\inf_{\pi_{2}\in\Pi_{2}^{m}}\mathbb{E}_{\gamma}^{\pi_{1},\pi_{2}}\left[e^{\theta\int_{t}^{T}\int_{A\times
B}u(x_{s},\pi_{1}(da|x_{s},s)),\pi_{2}(db|x_{s},s))ds+\theta
g(x_{T})}|x_{t}=x\right],$ (4.8)

provided the integral exists. Then, for each $n\geq 1$, since
$k_{n}(x,a,b)\geq 0$ and $f_{n}(x)\geq 0$, by Proposition 4.2(b) we have
$J_{k_{n},f_{n}}$ is in $\mathbb{B}_{V_{0},V_{1}}^{1}([0,T]\times S)$ and
satisfies

$\displaystyle\begin{cases}\displaystyle
J_{k_{n},f_{n}}^{\prime}(t,x)+\sup_{\phi\in P(A(x))}\inf_{\psi\in
P(B(x))}\left[\theta
k_{n}(x,\phi,\psi)J_{k_{n},f_{n}}(t,x)+\int_{S}J_{k_{n},f_{n}}(t,y)q(dy|x,\phi,\psi)\right]=0,\\\
J_{k_{n},f_{n}}(T,x)=e^{\theta f_{n}(x)},\end{cases}$ (4.9)

for all $x\in S$ and $t\in E_{(J_{k_{n},f_{n}},x)}$.

Moreover, since
$J_{k_{n},f_{n}}(t,x)=J_{r_{n}+n,g_{n}+n}(t,x)=J_{r_{n},g_{n}}(t,x)e^{\theta(T-t)n+\theta
n}$, by (4.9) we derive that

$\displaystyle\begin{cases}\displaystyle
J_{r_{n},g_{n}}^{\prime}(t,x)+\sup_{\phi\in P(A(x))}\inf_{\psi\in
P(B(x))}[\theta
r_{n}(x,\phi,\psi)J_{r_{n},g_{n}}(t,x)+\int_{S}J_{r_{n},g_{n}}(t,y)q(dy|x,\phi,\psi)]=0,\\\
J_{r_{n},g_{n}}(T,x)=e^{\theta g_{n}(x)}.\end{cases}$

This is

$\displaystyle J_{r_{n},g_{n}}(t,x)$ $\displaystyle=$ $\displaystyle e^{\theta
g_{n}(x)}+\int_{t}^{T}\sup_{\phi\in P(A(x))}\inf_{\psi\in P(B(x))}\left[\theta
r_{n}(x,\phi,\psi)J_{r_{n},g_{n}}(s,x)\right.$ (4.10)
$\displaystyle\left.+\int_{S}J_{r_{n},g_{n}}(s,y)q(dy|x,\phi,\psi)\right]ds.$

On the other hand, for each $(t,x)\in[0,T]\times S$. it follows from (4.8) and
Lemma 3.1(c) that

$\displaystyle|J_{r_{n},g_{n}}(t,x)|\leq LV_{0}(x),\ \ \ \ n\geq 1.$ (4.11)

Since $r_{n}(x,a,b)$ and $g_{n}(x)$ are decreasing in $n\geq 1$, and so is the
corresponding value functions $J_{r_{n},g_{n}}(t,x)$. Therefore, the limit
${\varphi}(t,x):=\lim_{n\to\infty}J_{r_{n},g_{n}}(t,x)$ exists for each
$(t,x)\in[0,T]\times S$. Then, as the arguments for Proposition 4.2 with
${\varphi}_{n}(t,x)$ replaced with $J_{r_{n},g_{n}}(t,x)$ here, from (4.10)
and (4.11) we can see that (a) is also true.

Moreover, by (4.6) and part (c), we see that $(\hat{\pi}_{1},\hat{\pi}_{2})$
is a Nash equilibrium. ∎

##  5 Algorithm

Until now, we have established the existence of the value function and Nash
equilibria of the risk-sensitive stochastic game. In this section, under the
suitable conditions, we prove that the value function and Nash equilibria of
the game can be approximated by iteratively solving a series of two-person
zero-sum matrix games through a value iteration-type algorithm.

First, we have the following convergence result.

######  Theorem 5.1.

Under Assumption 4.1, suppose that $\|q\|=\sup_{x\in S}q^{*}(x)$ and $\|r\|$
and $\|g\|$ are finite. Let $v_{0}(t,x)$ be an arbitrary function in
$\mathbb{B}^{1}_{1,1}([0,T]\times S)$, for $n\geq 0$,

$\displaystyle v_{n+1}(t,x):=\Gamma v_{n}(t,x)$ $\displaystyle={e^{\theta
g(x)}}+\int_{t}^{T}\sup_{\phi\in P(A(x))}\inf_{\psi\in P(B(x))}\left[\theta
r(x,\phi,\psi)v_{n}(s,x)+\int_{S}v_{n}(s,y)q(dy|x,\phi,\psi)\right]ds,$ (5.1)

where the operator $\Gamma$ is defined in Appendix (A.1). Then, we have

(a)

$\lim_{n\to\infty}v_{n}(t,x)=\varphi(t,x)$, for any $(t,x)\in[0,T]\times S$,
and given $\varepsilon>0$, there exists $N_{1}>0$ such that, for all $n\geq
N_{1}$,

$\displaystyle\|v_{n+1}-v_{n}\|$ $\displaystyle=$
$\displaystyle\sup_{(t,x)\in[0,T]\times
S}|v_{n+1}(t,x)-v_{n}(t,x)|<\frac{\varepsilon}{2e^{\left(\theta\|r\|+2\|q\|\right)T}\left(1+\frac{2\|q\|}{\theta\|r\|}\right)},$
(5.2) $\displaystyle\|\varphi-v_{n+1}\|$ $\displaystyle=$
$\displaystyle\sup_{(t,x)\in[0,T]\times
S}|\varphi(t,x)-v_{n+1}(t,x)|<\frac{\varepsilon}{2},$ (5.3)

where $\varphi(t,x)$ is the value function of the stochastic game, same as in
Proposition 4.1.

(b)

There exist Markov policies $\phi_{n}\in\Pi_{1}^{m},\psi_{n}\in\Pi_{2}^{m}$
such that

$\displaystyle(\phi_{n}(da|x,t),\psi_{n}(db|x,t))\in$
$\displaystyle\arg\sup_{\phi\in P(A(x))}\inf_{\psi\in P(B(x))}\left[\theta
r(x,\phi,\psi)v_{n}(t,x)+\int_{S}v_{n}(t,y)q(dy|x,\phi,\psi)\right],$ (5.4)

and for each $n\geq N_{1}$,

$\displaystyle\sup_{(t,x)\in[0,T]\times
S}|J(\phi_{n},\psi_{n},t,x)-\varphi(t,x)|<\varepsilon,$

where $J(\phi_{n},\psi_{n},t,x)$ is defined in (2.6) and $(\phi_{n},\psi_{n})$
is a strategy pair for $\varepsilon$-Nash equilibrium.

Since the operator $\Gamma$ is not a one-step contraction operator which is
widely used in discounted MDPs, the proof of Theorem 5.1 is not trivial and is
a little complicated. We present it in Appendix C.

From Theorem 5.1, we can see the calculation of function $v_{n+1}(t,x)$ is
critical for solving our risk-sensitive stochastic game. Below, we describe
the algorithm in a recurrent form. For the current iteration $n$, we assume
that we have the function $v_{n}(t,x)$, where $(t,x)\in[0,T]\times S$. The
purpose is to give a computation procedure to iteratively obtain
$v_{n+1}(t,x)$ converging to the value function of the game.

For any given $(t,x)\in[0,T]\times S$, define

$\displaystyle c(t,x,v_{n},a,b):=\theta
r(x,a,b)v_{n}(t,x)+\int_{S}v_{n}(t,y)q(dy|x,a,b),\ \forall a\in A(x),b\in
B(x).$ (5.5)

Then, the stochastic game with the payoff function $c(t,x,v_{n},a,b)$ can be
treated as a _matrix game_ at the current situation. We solve the
corresponding two-person zero-sum matrix game through two linear programs as
below [1].

$\displaystyle\max\ v$ $\displaystyle\mbox{subject to:
}\left\\{\begin{array}[]{l}v\leq\int_{a\in
A(x)}c(t,x,v_{n},a,b)\phi(da),\quad\forall b\in B(x);\\\ \int_{a\in
A(x)}\phi(da)=1;\\\ \phi(da)\geq 0,\quad\forall a\in A(x).\end{array}\right.,$
(5.9)

where $\phi$ is an optimization variable which indicates the mixed policy
(probability distribution) of player 1’s action selection in the action space
$A(x)$. Similarly, we solve the player 2’s action selection probability
distribution $\psi$ through the following linear program.

$\displaystyle\min\ z$ $\displaystyle\mbox{subject to:
}\left\\{\begin{array}[]{l}z\geq\int_{b\in
B(x)}c(t,x,v_{n},a,b){\psi}(db),\quad\forall a\in A(x);\\\ \int_{b\in
B(x)}{\psi}(db)=1;\\\ \tilde{\psi}(db)\geq 0,\quad\forall b\in
B(x).\end{array}\right.$ (5.13)

We can solve the above linear programs by simplex algorithms and obtain the
solutions $(\phi_{n}(da|x,t),a_{n}(t,x))$ and $(\psi_{n}(db|x,t),b_{n}(t,x))$,
respectively. With the classical results of matrix games, we see that
$a_{n}(t,x)=b_{n}(t,x)$, which is the value of the matrix game. By (5.5), for
any $\phi\in P(A(x)),\psi\in P(B(x))$, we denote

$\displaystyle c(t,x,v_{n},\phi,\psi):=\int_{a\in A(x)}\int_{b\in
B(x)}\left[\theta
r(x,a,b)v_{n}(t,x)+\int_{S}v_{n}(t,y)q(dy|x,a,b)\right]\phi(da)\psi(db).$

Then, we have

$\displaystyle v_{n+1}(t,x)$ $\displaystyle=$ $\displaystyle e^{\theta
g(x)}+\int_{t}^{T}\sup_{\phi\in P(A(x))}\inf_{\psi\in P(B(x))}\left[\theta
r(x,\phi,\psi)v_{n}(s,x)+\int_{S}v_{n}(s,y)q(dy|x,\phi,\psi)\right]ds$ (5.14)
$\displaystyle=$ $\displaystyle e^{\theta g(x)}+\int_{t}^{T}\sup_{\phi\in
P(A(x))}\inf_{\psi\in P(B(x))}c(s,x,v_{n},\phi,\psi)ds$ $\displaystyle=$
$\displaystyle e^{\theta g(x)}+\int_{t}^{T}a_{n}(s,x)ds.$

Therefore, the value of $v_{n+1}(t,x)$ can be obtained by solving (5) or (5).
We update the value of $v_{n+1}(t,x)$ for every $(t,x)$ by solving the above
linear programs (we only need to solve either (5) or (5) since their optimal
values are equal). We repeat the same computation procedure based on the
updated values $v_{n+1}(t,x)$’s.

By Theorem 5.1, we have the following value iteration-type Algorithm 1. For
the convenience of computation, we assume that $A,B$ are finite.

1 Algorithm parameter: the payoff rate $r(x,a,b)$, $\|r\|=\sup_{x\in S,a\in
A(x),b\in B(x)}|r(x,a,b)|$; the transition rate $q(dy|x,a,b)$,
$\|q\|=\sup_{x\in S}q^{*}(x)=\sup_{x\in S,a\in A(x),b\in
B(x)}|-q(\\{x\\}|x,a,b)|$; the terminal reward $g(x)$; the risk-sensitive
parameter $\theta>0$; finite horizon $T>0$; Player 1 has $m=|A|$ actions and
Player 2 has $n=|B|$ actions; a small error bound $\varepsilon>0$ determining
the algorithm accuracy

2 Initialize: $v_{0}(t,x)\in\mathbb{B}^{1}_{1,1}([0,T]\times S)$ arbitrarily,
$n=0$

3 repeat

4 $\Delta\leftarrow 0$

5 Loop for  _each $(t,x)\in[0,T]\times S$_ do

6 for  _each $s\in[t,T]$_ do

7 for  _ $i=1;i<m;i++$_ do

8 for  _$j=1;j <n;j++$_ do

9 $c(s,x,v_{n})_{ij}\leftarrow\theta
r(x,a_{i},b_{j})v_{n}(s,x)+\int_{S}v_{n}(s,y)q(dy|x,a_{i},b_{j})$

10

11 Solving the game with matrix $C(s,x,v_{n})$

12 $a_{n}(s,x)\leftarrow\sup_{\phi\in P(A(x))}\inf_{\psi\in
P(B(x))}\phi^{T}C(s,x,v_{n})\psi$

13 $(\phi_{n}(\cdot|x,s),\psi_{n}(\cdot|x,s))\leftarrow\arg\sup_{\phi\in
P(A(x))}\inf_{\psi\in P(B(x))}\phi^{T}C(s,x,v_{n})\psi$

14

15 $v_{n+1}(t,x)\leftarrow e^{\theta g(x)}+\int_{t}^{T}a_{n}(s,x)ds$

16

17 $\Delta\leftarrow\max\\{\Delta,|v_{n+1}(t,x)-v_{n}(t,x)|\\}$

18

19 $n\leftarrow n+1$

20

21until  _
$\Delta<\frac{\varepsilon}{2e^{(\theta\|r\|+2\|q\|)T}\left(1+\frac{2\|q\|}{\theta\|r\|}\right)}$
_;

22Output:

23 $v_{n+1}(0,x)$ and $(\phi_{n}(\cdot|x,t),\psi_{n}(\cdot|x,t)),$
$(t,x)\in[0,T]\times S$

Algorithm 1 A value iteration-type algorithm to solve risk-sensitive
stochastic games

With Theorem 5.1, we can see that the limit of $v_{n+1}(0,x)$ (as
$n\to\infty$) is the value of the game ${\varphi}(0,x)$. When the stopping
condition (5.2) is satisfied and the algorithm stops, the output policy pair
$(\phi_{n},\psi_{n})$ is an $\varepsilon$-Nash equilibrium and the output
value function $v_{n+1}(0,x)$ is within $\varepsilon/2$ error bound from the
optimal value function ${\varphi}(0,x)$. The convergence of the algorithm is
also guaranteed, since it is proved that the algorithm will stop within a
finite number of iterations by Theorem 5.1(a).

If the payoff rates, the transition rates, or the terminal rewards is not
bounded, we can solve a series of the stochastic model ${\mathbb{M}}_{n}^{+}$
(B.4) or ${\cal{N}}_{n}$ (4.7) by finite approximation technique.

##  6 Examples

In this section, we give two examples to illustrate our main results.

######  Example 6.1.

In a system, the state of this system is $x_{t}$ at time $t$ which is
continuous in time. The corresponding state space is $S:=[0,+\infty)$. The
system evolves as follows. If the system is at state $x_{t}$ at time $t$, two
players play the game of scissors, paper, stone. $A=A(x)=B=B(x)=\\{1,2,3\\}$,
$x\in S$, scissors are denoted as 1, paper is denoted as 2, and stone is
denoted as 3. Denote $a_{t}$ the action player 1 taken and $b_{t}$ the action
player 2 taken at time $t$. The winner receives payoffs at rate
$\alpha\sqrt{\ln(1+x_{t})}$ from the loser, $0<\alpha\leq 0.5$. If they are
tied, both of them receive 0. That is,

$r(x,a,b)=\left\\{\begin{array}[]{ll}0,&x\in S,a=b;\\\
\alpha\sqrt{\ln(1+x)},&x\geq 0,(a=1,b=2),(a=2,b=3),(a=3,b=1);\\\
-\alpha\sqrt{\ln(1+x)},&x\geq
0,(a=2,b=1),(a=3,b=2),(a=1,b=3).\end{array}\right.$

Next, state $x_{t}$ is assumed to keep invariable for an exponential-
distributed random time with parameter $\lambda(x_{t},a_{t},b_{t})$
($0<\lambda(x,a,b)\leq L$), and then jump to other states with exponential-
distribution $\rm{exp}(\frac{1}{x_{t}})$. Therefore, the transition rate of
state is represented by, for each $D\in\mathcal{B}(S)$,

$q(D|x,a,b)=\lambda(x,a,b)\left[\int_{y\in
D}\frac{1}{x}e^{-\frac{y}{x}}dy-\delta_{x}(D)\right],\quad x\in S,a\in A,b\in
B.$

The terminal reward function is $g(x)=\frac{\sqrt{\ln(1+x)}}{2}$. For this
zero-sum stochastic game model, player 1 wishes to maximize the risk-sensitive
rewards on a given $T$ horizon over all policies and player 2 wishes to
minimize the risk-sensitive cost on a given $T$ horizon over all policies.

Under the above conditions, we have the following fact.

######  Proposition 6.1.

Example 6.1 satisfies Assumptions 3.1, 3.2 and 4.1, and hence (by Theorem 4.1)
there exists a Nash equilibrium.

###### Proof.

Let us first show that Assumption 3.1 holds.

There exist a real-valued Borel measurable function $V_{0}(x):=1+x\geq 1$ on
$S$ and positive constants $\rho_{0}=1,L_{0}=L,M_{0}=1$, such that

(i)

for any $x\in S,a\in A,b\in B$,

$\displaystyle\int_{S}V_{0}(y)q(dy|x,a,b)$ $\displaystyle=$
$\displaystyle\lambda(x,a,b)\left[\int_{0}^{+\infty}(1+y)\frac{1}{x}e^{-\frac{y}{x}}dy-(1+x)\right]$
$\displaystyle=$ $\displaystyle\lambda(x,a,b)\left[1+x-(1+x)\right]=0\leq
1+x=\rho_{0}V_{0}(x);$

(ii)

$\displaystyle q^{*}(x)=\sup_{a\in A(x),b\in B(x)}q(x,a,b)=\sup_{a\in
A(x),b\in B(x)}-q(\\{x\\}|x,a,b)=\sup_{a\in A(x),b\in B(x)}\lambda(x,a,b)\leq
L\leq L(1+x)$ for all $x\in S$, where $q^{*}(x)$ is as in (2.2);

(iii)

$|r(x,a,b)|\leq\alpha\sqrt{\ln(1+x)}\leq 0.5\sqrt{\ln(1+x)}\leq
1+\frac{\sqrt{2}}{2}\sqrt{\ln V_{0}(x)}$ and
$|g(x)|=\frac{\sqrt{\ln(1+x)}}{2}\leq 1+\frac{\sqrt{2}}{2}\sqrt{\ln V_{0}(x)}$
for any $x\in S,a\in A,b\in B$.

Now we show that Assumption 3.2 holds. A directive calculation gives
$\int_{0}^{\infty}y^{k}\frac{1}{x}e^{-\frac{y}{x}}dy=k!x^{k}$ for all
$k=0,1,\ldots$.

There exist a real-valued Borel measurable function $V_{1}(x):=(1+x)^{2}\geq
1$ on $S$, and positive constants $\rho_{1}=23L,b_{1}=1$, and $M_{1}=1$, such
that

(i)

for any $x\in S,a\in A,b\in B$,

$\displaystyle\int_{S}V_{1}^{2}(y)q(dy|x,a,b)$ $\displaystyle=$
$\displaystyle\lambda(x,a,b)\left[\int_{0}^{+\infty}(1+y)^{4}\frac{1}{x}e^{-\frac{y}{x}}dy-(1+x)^{4}\right]$
$\displaystyle=$
$\displaystyle\lambda(x,a,b)\left[1+4x+12x^{2}+24x^{3}+24x^{4}-1-4x-6x^{2}-4x^{3}-x^{4}\right]$
$\displaystyle=$
$\displaystyle\lambda(x,a,b)\left[6x^{2}+20x^{3}+23x^{4}\right]\leq
23L(1+x)^{4}=\rho_{1}V_{1}^{2}(x)+b_{1};$

(ii)

$V_{0}^{2}(x)=(1+x)^{2}=M_{1}V_{1}(x)=(1+x)^{2}$ for all $x\in S$.

Since $A,B$ are finite, Assumption 4.1 holds.

Thus, Assumptions 3.1, 3.2 and 4.1 hold for Example 6.1, and then Theorem 4.1
guarantees the existence of a Nash equilibrium.

∎

######  Example 6.2.

${\mathbb{M}}_{2}:=\big{\\{}S,(A,A(x),x\in S),(B,B(x),x\in
S),r(x,a,b),q(dy|x,a,b),g(x)\big{\\}},$

where $S=(-\infty,\infty)$, for each $D\in\mathcal{B}(S)$,

$q(D|x,a,b)=\lambda(x,a,b)\left[\int_{y\in
D}\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(y-x)^{2}}{2\sigma^{2}}}dy-\delta_{x}(D)\right],\
\ x\in S,a\in A(x),b\in B(x).$ (6.1)

To ensure the existence of a Nash equilibrium for the model, we consider the
following hypotheses:

  * $(A_{1})$

$0<\lambda(x,a,b)\leq M(x^{2}+1)$ , $|r(x,a,b)|\leq
M_{0}+\frac{\sqrt{2}}{2}\sqrt{\ln(1+x^{2})}$ for all $x\in S,a\in A(x),b\in
B(x)$, $|g(x)|\leq M_{0}+\frac{\sqrt{2}}{2}\sqrt{\ln(1+x^{2})}$ for all $x\in
S$ with some positive constants $M$ and $M_{0}$;

  * $(A_{2})$

$A(x),B(x)$ are assumed to be a compact set of Borel spaces $A,B$ for each
$x\in S$, respectively;

  * $(A_{3})$

$\lambda(x,a,b)$ and $r(x,a,b)$ are Borel measurable on $K$ and continuous in
$a\in A(x),\ b\in B(x)$ for each fixed $x\in S$.

Under the above conditions, we have the following fact.

######  Proposition 6.2.

Example 6.2 satisfies Assumptions 3.1, 3.2 and 4.1, and hence (by Theorem 4.1)
there exists a Nash equilibrium.

###### Proof.

Let us first show that Assumption 3.1 holds.

There exist a real-valued Borel measurable function $V_{0}(x):=1+x^{2}\geq 1$
on $S$ and positive constants $\rho_{0}=M\sigma^{2},L_{0}=M,M_{0}=M_{0}$, such
that

(i)

for any $x\in S,a\in A(x),b\in B(x)$,

$\displaystyle\int_{S}V_{0}(y)q(dy|x,a,b)$ $\displaystyle=$
$\displaystyle\lambda(x,a,b)\left[\frac{1}{\sqrt{2\pi}\sigma}\int_{-\infty}^{+\infty}(y^{2}+1)e^{-\frac{(y-x)^{2}}{2\sigma^{2}}}dy-(x^{2}+1)\right]$
$\displaystyle=$ $\displaystyle\lambda(x,a,b)\sigma^{2}\leq
M\sigma^{2}V_{0}(x);$

(ii)

$\displaystyle q^{*}(x)=\sup_{a\in A(x),b\in B(x)}q(x,a,b)=\sup_{a\in
A(x),b\in B(x)}-q(\\{x\\}|x,a,b)=\sup_{a\in A(x),b\in B(x)}\lambda(x,a,b)\leq
M(x^{2}+1)=MV_{0}(x)$ for all $x\in S$, where $q^{*}(x)$ is as in (2.2);

(iii)

$|r(x,a,b)|\leq
M_{0}+\frac{\sqrt{2}}{2}\sqrt{\ln(1+x^{2})}=M_{0}+\frac{\sqrt{2}}{2}\sqrt{\ln
V_{0}(x)}$, $x\in S,a\in A(x),b\in B(x)$, and $|g(x)|\leq
M_{0}+\frac{\sqrt{2}}{2}\sqrt{\ln(1+x^{2})}=M_{0}+\frac{\sqrt{2}}{2}\sqrt{\ln
V_{0}(x)}$ for any $x\in S$.

Now we show that Assumption 3.2 holds.

A directive calculation gives
$\frac{1}{\sqrt{2\pi}\sigma}\int_{-\infty}^{\infty}(y-x)^{2k+1}e^{-\frac{(y-x)^{2}}{2\sigma^{2}}}dy=0$
and
$\frac{1}{\sqrt{2\pi}\sigma}\int_{-\infty}^{\infty}(y-x)^{2k}e^{-\frac{(y-x)^{2}}{2\sigma^{2}}}dy=1\cdot
3\cdots(2k-1)\sigma^{2k}$ for all $k=0,1,\ldots$.

There exist a real-valued Borel measurable function $V_{1}(x):=1+x^{4}\geq 1$
on $S$, and positive constants
$\rho_{1}=3780M\left(\sigma^{8}+\sigma^{6}+\sigma^{4}+\sigma^{2}\right),b_{1}=1$,
and $M_{1}=2$, such that

(i)

for any $x\in S,a\in A(x),b\in B(x)$,

$\displaystyle\int_{S}V_{1}^{2}(y)q(dy|x,a,b)$ $\displaystyle=$
$\displaystyle\lambda(x,a,b)\left[\frac{1}{\sqrt{2\pi}\sigma}\int_{-\infty}^{+\infty}(y^{4}+1)^{2}e^{-\frac{(y-x)^{2}}{2\sigma^{2}}}dy-(x^{4}+1)^{2}\right]$
$\displaystyle=$
$\displaystyle\lambda(x,a,b)\left(105\sigma^{8}+420x^{2}\sigma^{6}+210x^{4}\sigma^{4}+6\sigma^{4}+12\sigma^{2}x^{2}+28x^{6}\sigma^{2}\right)$
$\displaystyle\leq$ $\displaystyle
420\lambda(x,a,b)\left(\sigma^{8}+\sigma^{6}+\sigma^{4}+\sigma^{2}\right)\left(x^{6}+x^{4}+x^{2}+1\right)$
$\displaystyle\leq$ $\displaystyle
420\lambda(x,a,b)\left(\sigma^{8}+\sigma^{6}+\sigma^{4}+\sigma^{2}\right)\left(3x^{6}+3\right)$
$\displaystyle\leq$ $\displaystyle
1260M\left(\sigma^{8}+\sigma^{6}+\sigma^{4}+\sigma^{2}\right)\left(x^{6}+1\right)(1+x^{2})$
$\displaystyle\leq$ $\displaystyle
3780M\left(\sigma^{8}+\sigma^{6}+\sigma^{4}+\sigma^{2}\right)\left(x^{4}+1\right)^{2}$
$\displaystyle\leq$ $\displaystyle
3780M\left(\sigma^{8}+\sigma^{6}+\sigma^{4}+\sigma^{2}\right)V_{1}^{2}(x)+1.$

(ii)

$V_{0}^{2}(x)=(1+x^{2})^{2}\leq 2(1+x^{4})$ for all $x\in S$.

Thus, Assumptions 3.1, 3.2 and 4.1 (under the hypotheses $A_{1}$–$A_{3}$) hold
for Example 6.2, and then Theorem 4.1 gives the existence of a Nash
equilibrium. ∎

######  Remark 6.1.

In Example 6.1, the payoff rates $r(x,a,b)$ are allowed to be unbounded from
above. In Example 6.2, the payoff rates $r(x,a,b)$, the transition rates
$q(dy|x,a,b)$ and the terminal reward $g(x)$ are all unbounded from below and
from above.

##  7 Conclusion

In this paper we have studied a finite-horizon two-person zero-sum risk-
sensitive stochastic game for continuous-time Markov chains with Borel state
and action spaces, in which the payoff rates, the transition rates and the
terminal rewards are allowed to be unbounded from below and from above and the
policies can be history-dependent. This model is a generalization of that in
the existing literature [7] with bounded payoff rates and Markov policies. To
establish the corresponding Shapley equation and the existence of a Nash
equilibrium for the general model, we develop a finite-approximation
technique. More specifically, for the bounded case (i.e., the payoff rates,
the transition rates and the terminal rewards are bounded), we first prove the
existence of a solution to the Shapley equation by the Banach-fixed-point
theorem with a $k$-step contraction operator, establish the existence of both
the value function and a Nash equilibrium for the stochastic game, and verify
that the value function of the game uniquely solves the Shapley equation by
the extension of the Dynkin’s formula. Then, by developing a finite-
approximation technique, we extend the results for the bounded case to the
general case that the payoff rates, the transition rates and the terminal
rewards are unbounded (Theorem 4.1). As a consequence, our results extend the
findings in [44] and answer an open question posed there.

The computation of Nash equilibria is of significance and desirable for the
practical application of game theory. To the best of our knowledge, our
iteration algorithm developed in this paper for computing the value function
and Nash equilibria of risk-sensitive stochastic games is a first attempt. We
also prove the convergence of the algorithm by a specific contraction
operator. The combination of the iteration algorithm with other approximation
techniques to handle the issue of large scalability, such as reinforcement
learning, deserves further investigation in a regime of so-called multi-agent
reinforcement learning [10, 37].

## Appendix:

##  Appendix A Proof of Proposition 4.1

###### Proof.

(a) Define the following operator $\Gamma$ on
$\mathbb{B}_{1,1}^{1}([0,T]\times S)$ by

$\Gamma{\varphi}(t,x):={e^{\theta g(x)}}+\int_{t}^{T}\sup_{\phi\in
P(A(x))}\inf_{\psi\in P(B(x))}\left[\theta
r(x,\phi,\psi){\varphi}(s,x)+\int_{S}{\varphi}(s,y)q(dy|x,\phi,\psi)\right]ds$
(A.1)

for any $(t,x)\in[0,T]\times S$ and
${\varphi}\in\mathbb{B}_{1,1}^{1}([0,T]\times S).$

Then, for each $(t,x)\in[0,T]\times S$, and any
$\varphi_{1},\varphi_{2}\in\mathbb{B}_{1,1}^{1}([0,T]\times S)$, from (A.1)
and $q(\\{x\\}|x,a,b)+q(S\setminus\\{x\\})|x,a,b)\equiv 0$, we obtain

$\displaystyle|\Gamma\varphi_{1}(t,x)-\Gamma\varphi_{2}(t,x)|$
$\displaystyle\leq$
$\displaystyle(\theta\|r\|+2\|q\|)\int_{t}^{T}\|\varphi_{1}-\varphi_{2}\|ds$
$\displaystyle=$ $\displaystyle\tilde{L}(T-t)\|\varphi_{1}-\varphi_{2}\|,$

where $\tilde{L}:=\theta\|r\|+2\|q\|<\infty$. Furthermore, by induction we can
prove the following fact:

$\displaystyle|\Gamma^{n}\varphi_{1}(t,x)-\Gamma^{n}\varphi_{2}(t,x)|\leq\tilde{L}^{n}\frac{(T-t)^{n}}{n!}\|\varphi_{1}-\varphi_{2}\|,\
\ \ \forall\ (t,x)\in[0,T]\times S,n\geq 1.$ (A.2)

Since
$\sum_{n=0}^{\infty}\tilde{L}^{n}\frac{T^{n}}{n!}\|\varphi_{1}-\varphi_{2}\|=e^{\tilde{L}T}\|\varphi_{1}-\varphi_{2}\|<\infty$,
there exists some integer $k$ such that the constant

$\beta:=\tilde{L}^{k}\frac{T^{k}}{k!}<1.$

Thus, by (A.2) we have
$\|\Gamma^{k}\varphi_{1}-\Gamma^{k}\varphi_{2}\|\leq\beta\|\varphi_{1}-\varphi_{2}\|$.
Therefore, $\Gamma$ is a $k$-step contract operator. Thus, there exists a
function ${\varphi}\in\mathbb{B}_{1,1}^{1}([0,T]\times S)$ such that
$\Gamma{\varphi}={\varphi}$, that is, for any $(t,x)\in[0,T]\times S,$

$\displaystyle{\varphi}(t,x)={e^{\theta g(x)}}+\int_{t}^{T}\sup_{\phi\in
P(A(x))}\inf_{\psi\in P(B(x))}\left[\theta
r(x,\phi,\psi){\varphi}(s,x)+\int_{S}{\varphi}(s,y)q(dy|x,\phi,\psi)\right]ds.$
(A.3)

Since $\|q\|$, $\|r\|$ and $\|g\|$ are finite, by (A.3) we see that
${\varphi}\in\mathbb{B}^{1}_{1,1}([0,T]\times S)$, and thus (a) follows.

(b) By (a) and Fan’s minimax theorem in [12], the minimax measurable selection
theorems Theorem 2.2 in [39] together with Lemma 4.1 in [27], we see that (b)
is true.

(c) Given any $\pi_{1}\in\Pi_{1}$, for each $x\in S$ and $t\in
E_{(\varphi,x)}$, by (4.3) we have

$\displaystyle\begin{cases}-{\varphi}^{\prime}(t,x)\geq\theta{\varphi}(t,x)\int_{A(x)}r(x,a,\hat{\pi}_{2}^{t})\pi_{1}(da|\omega,t)+\int_{S}\int_{A(x)}{\varphi}(t,y)q(dy|x,a,\hat{\pi}_{2}^{t})\pi_{1}(da|\omega,t),\\\
{\varphi}(T,x)=e^{\theta g(x)}.\end{cases}$

which, together with the fact that $x_{t}(\omega)$ is a piece-wise constant
(by (2.4), implies

$\displaystyle-\left(e^{\int_{0}^{t}\int_{A(x_{v})}\theta
r(x_{v},a,\hat{\pi}_{2}^{v})\pi_{1}(da|\omega,v)dv}{\varphi}(t,x_{t})\right)^{\prime}$
$\displaystyle=$ $\displaystyle-e^{\int_{0}^{t}\int_{A(x_{v})}\theta
r(x_{v},a,\hat{\pi}_{2}^{v})\pi_{1}(da|\omega,v)dv}\left\\{\left[\int_{A(x_{t})}\theta
r(x_{t},a,\hat{\pi}_{2}^{t})\pi_{1}(da|\omega,t)\right]{\varphi}(t,x_{t})+{\varphi}^{\prime}(t,x_{t})\right\\}$
$\displaystyle\geq$ $\displaystyle-e^{\int_{0}^{t}\int_{A(x_{v})}\theta
r(x_{v},a,\hat{\pi}_{2}^{v})\pi_{1}(da|\omega,v)dv}\left[\int_{A(x_{t})}\theta
r(x_{t},a,\hat{\pi}_{2}^{t})\pi_{1}(da|\omega,t)\right]{\varphi}(t,x_{t})$
$\displaystyle+e^{\int_{0}^{t}\int_{A(x_{v})}\theta
r(x_{v},a,\hat{\pi}_{2}^{v})\pi_{1}(da|\omega,v)dv}$
$\displaystyle\left[\theta{\varphi}(t,x_{t})\int_{A(x_{t})}r(x_{t},a,\hat{\pi}_{2}^{t})\pi_{1}(da|\omega,t)+\int_{S}\int_{A(x_{t})}{\varphi}(t,y)q(dy|x_{t},a,\hat{\pi}_{2}^{t})\pi_{1}(da|\omega,t)\right]$
$\displaystyle=$
$\displaystyle\int_{S}\int_{A(x_{t})}q(dy|x_{t},a,\hat{\pi}_{2}^{t})\pi_{1}(da|\omega,t)\left(e^{\int_{0}^{t}\int_{A(x_{v})}\theta
r(x_{v},a,\hat{\pi}_{2}^{v})\pi_{1}(da|\omega,v)dv}{\varphi}(t,y)\right),\ \ \
\ \forall\ t\geq 0.$

Thus, by Lemma 3.2(a) we have

$\displaystyle\mathbb{E}_{x}^{\pi_{1},\hat{\pi}_{2}}\left(e^{\int_{0}^{T}\int_{A(x_{t})}\theta
r(x_{t},a,\hat{\pi}_{2}^{t})\pi_{1}(da|\omega,t)dt+\theta
g(x_{T})}\right)-{\varphi}(0,x)$ $\displaystyle=$
$\displaystyle\mathbb{E}_{x}^{\pi_{1},\hat{\pi}_{2}}\left(e^{\int_{0}^{T}\int_{A(x_{t})}\theta
r(x_{t},a,\hat{\pi}_{2}^{t})\pi_{1}(da|\omega,t)dt}{\varphi}(T,x_{T})\right)-{\varphi}(0,x)\leq
0$

and so

$\displaystyle J(\pi_{1},\hat{\pi}_{2},0,x)\leq{\varphi}(0,x)\ \ \ \ {\rm for\
all}\ x\in S,\pi_{1}\in\Pi_{1}.$ (A.4)

Moreover, by (2.6) a similar proof gives

$\displaystyle J(\pi_{1},\hat{\pi}_{2},t,x)\leq{\varphi}(t,x)\ \ \ \ {\rm for\
all}\ t\in[0,T],x\in S,\pi_{1}\in\Pi_{1}^{m}.$ (A.5)

Therefore, since $\pi_{1}$ can be arbitrary, by (A.4)-(A.5) we have

$\displaystyle\inf_{\pi_{2}\in\Pi_{2}}\sup_{\pi_{1}\in\Pi_{1}}J(\pi_{1},\pi_{2},0,x)\leq{\varphi}(0,x),\
{\rm and}\
\inf_{\pi_{2}\in\Pi_{2}^{m}}\sup_{\pi_{1}\in\Pi_{1}^{m}}J(\pi_{1},\pi_{2},t,x)\leq{\varphi}(t,x),$
(A.6)

for all $(t,x)\in[0,T]\times S$.

Furthermore, by (4.4), we have

$\displaystyle\sup_{\pi_{1}\in\Pi_{1}}\inf_{\pi_{2}\in\Pi_{2}}J(\pi_{1},\pi_{2},0,x)\geq{\varphi}(0,x),\
{\rm and}\ \
\sup_{\pi_{1}\in\Pi_{1}^{m}}\inf_{\pi_{2}\in\Pi_{2}^{m}}J(\pi_{1},\pi_{2},t,x)\geq{\varphi}(t,x),\
\ \ \ \ \ \ \ \ $ (A.7) $\displaystyle{\rm and}$ $\displaystyle
J(\hat{\pi}_{1},\hat{\pi}_{2},0,x)={\varphi}(0,x),\quad\forall\ x\in S.$ (A.8)

By (A.6)-(A.8) we have

$\displaystyle\sup_{\pi_{1}\in\Pi_{1}}\inf_{\pi_{2}\in\Pi_{2}}J(\pi_{1},\pi_{2},0,x)=\inf_{\pi_{2}\in\Pi_{2}}\sup_{\pi_{1}\in\Pi_{1}}J(\pi_{1},\pi_{2},0,x)={\varphi}(0,x),$
$\displaystyle\sup_{\pi_{1}\in\Pi_{1}^{m}}\inf_{\pi_{2}\in\Pi_{2}^{m}}J(\pi_{1},\pi_{2},t,x)=\inf_{\pi_{2}\in\Pi_{2}^{m}}\sup_{\pi_{1}\in\Pi_{1}^{m}}J(\pi_{1},\pi_{2},t,x)={\varphi}(t,x),$
$\displaystyle{\rm and}$ $\displaystyle
J(\hat{\pi}_{1},\hat{\pi}_{2},t,x)={\varphi}(t,x),\ \ \ \ \forall\
(t,x)\in[0,T]\times S,$

which gives (c).

(d) Obviously, (d) is from (A).

(e) Fix any $s,t\in[0,T]$ with $s<t$. Then, for any Markov policy
$\pi_{1}\in\Pi_{1}^{M}$, we define the corresponding Markov policy
$\pi_{1,s}^{t}$ as follows: for each $x\in S$,

$\displaystyle\pi_{1,s}^{t}(x,v)=\begin{cases}\pi_{1}(x,v+t-s),&\mbox{$v\geq
s,$}\\\ \pi_{1}(x,v),&\mbox{otherwise,}\end{cases}$ (A.10)

where $\pi_{1}(x,t):=\pi_{1}(da|x,t)$ is a stochastic kernel on $A(x)$. It is
the similar notation for $\pi_{2}\in\Pi_{2}^{M}$. Then, we have, for each
$(x,v)\in S\times[s,s+T-t]$,

$q(dy|x,\pi_{1,s}^{t}(x,v),\pi_{2,s}^{t}(x,v))=q(dy|x,\pi_{1}(x,v+t-s),\pi_{2}(x,v+t-s)),$
$r(x,\pi_{1,s}^{t}(x,v),\pi_{2,s}^{t}(x,v))=r(x,\pi_{1}(x,v+t-s),\pi_{2}(x,v+t-s)).$

Let

$\displaystyle J(\pi_{1},\pi_{2},s\sim
t,x):=\mathbb{E}_{\gamma}^{\pi_{1},\pi_{2}}\left[e^{\int_{s}^{t}\int_{A}\int_{B}\theta
r(x_{v},a,b)\pi_{1}(da|x_{v},v)\pi_{2}(db|x_{v},v)dv+\theta
g(t,x_{t})}|x_{s}=x\right],$ $\displaystyle J_{*}(s\sim
t,x):=\sup_{\pi_{1}\in\Pi_{1}^{m}}\inf_{\pi_{2}\in\Pi_{2}^{m}}J(\pi_{1},\pi_{2},s\sim
t,x).$ (A.11)

By the Markov property of $\\{x_{t},t\geq 0\\}$ (under any Markov policy
$(\pi_{1},\pi_{2})$) and (A.10)-(A.11), we have $X_{u}$ under policies
$\pi_{1},\pi_{2}$ and $X_{t}=x$ has the same distribution with $X_{u+s-t}$
under policies $\pi_{1,s}^{t},\pi_{2,s}^{t}$ and $X_{s}=x$ for any $t\leq
u\leq T$. Therefore, $J(\pi_{1},\pi_{2},t\sim
T,x)=J(\pi_{1,s}^{t},\pi_{2,s}^{t},s\sim T+s-t,x)$. From this, we have

$\displaystyle\inf_{\pi_{2}\in\Pi_{2}^{M}}J(\pi_{1},\pi_{2},t\sim
T,x)=\inf_{\pi_{2}\in\Pi_{2}^{M}}J(\pi_{1,s}^{t},\pi_{2,s}^{t},s\sim T+s-t,x)$
$\displaystyle\geq$
$\displaystyle\inf_{\pi_{2,s}^{t}\in\Pi_{2}^{M}}J(\pi_{1,s}^{t},\pi_{2,s}^{t},s\sim
T+s-t,x),\quad\forall\pi_{1}\in\Pi_{1}^{M}.$

Similarly, we get

$\displaystyle\inf_{\pi_{2,s}^{t}\in\Pi_{2}^{M}}J(\pi_{1,s}^{t},\pi_{2,s}^{t},s\sim
T+s-t,x)=\inf_{\pi_{2,s}^{t}\in\Pi_{2}^{M}}J(\pi_{1},\pi_{2},t\sim T,x)$
$\displaystyle\geq$
$\displaystyle\inf_{\pi_{2}\in\Pi_{2}^{M}}J(\pi_{1},\pi_{2},t\sim
T,x),\quad\forall\pi_{1}\in\Pi_{1}^{M}.$

Thus,

$\inf_{\pi_{2,s}^{t}\in\Pi_{2}^{M}}J(\pi_{1,s}^{t},\pi_{2,s}^{t},s\sim
T+s-t,x)=\inf_{\pi_{2}\in\Pi_{2}^{M}}J(\pi_{1},\pi_{2},t\sim
T,x),\quad\forall\pi_{1}\in\Pi_{1}^{M}.$

Similarly, we have $J_{*}(t\sim T,x)=J_{*}(s\sim T+s-t,x)$. Moreover, since
$r(x,a,b)\geq 0$ on $K$, by (A.11) and $t>s$, we have $J_{*}(t\sim
T,x)=J_{*}(s\sim T+s-t,x)\leq J_{*}(s\sim T,x)$, which, together with
$J_{*}(t\sim T,x)={\varphi}(t,x)$ in part (c), gives (e). ∎

##  Appendix B Proof of Proposition 4.2

###### Proof.

We only prove part (a). This is because parts (b) and (c) can be proved as the
same arguments of (c) and (d) of Proposition 4.1. First, under Assumption 3.1
(iii), we have

$0\leq r(x,a,b)\leq\frac{\sqrt{2}}{2}\sqrt{\ln V_{0}(x)}+M_{0}\ {\rm for\ all\
}(x,a,b)\in K,$

and

$0\leq g(x)\leq\frac{\sqrt{2}}{2}\sqrt{\ln V_{0}(x)}+M_{0}\ {\rm for\ all\
}x\in S.$

For each $n\geq 1$, let $A_{n}(x):=A(x)$ and $B_{n}(x):=B(x)$ for $x\in S$,
$K_{n}:=\\{(x,a,b)|x\in S,a\in A_{n}(x),b\in B_{n}(x)\\}$, and $S_{n}:=\\{x\in
S|V_{0}(x)\leq n\\}$. Moreover, for each $x\in S$, $a\in A_{n}(x)$, $b\in
B_{n}(x)$, let

$q_{n}(dy|x,a,b):=\begin{cases}q(dy|x,a,b),&\text{if }x\in S_{n},\\\
0,&\text{if }x\not\in S_{n};\end{cases}$ (B.1)
$r_{n}^{+}(x,a,b):=\begin{cases}\min\\{n,r(x,a,b)\\},&\text{if }x\in S_{n},\\\
0,&\text{if }x\not\in S_{n};\end{cases}$ (B.2)

and

$g_{n}^{+}(x):=\begin{cases}\min\\{n,g(x)\\},&\text{if }x\in S_{n},\\\
0,&\text{if }x\not\in S_{n}.\end{cases}$ (B.3)

Fix any $n\geq 1$. By (B.1), it is obvious that $q_{n}(dy|x,a,b)$ denotes
indeed transition rates on $S$, which are _conservative_ and _stable_. Then,
we obtain a sequence of models $\\{\mathbb{M}_{n}^{+}\\}$:

${\mathbb{M}}_{n}^{+}:=\big{\\{}S,(A,A(x),x\in S),(B,B(x),x\in
S),r_{n}^{+}(x,a,b),q_{n}(dy|x,a,b),g_{n}^{+}(x)\big{\\}},$ (B.4)

for which the payoff rates $r_{n}^{+}(x,a,b)$, the transition rates
$q_{n}(dy|x,a,b)$ and terminal reward $g_{n}^{+}(x)$ are all bounded by
Assumption 3.1 and (B.1)-(B.3). In the following arguments, any quality with
respect to ${\mathbb{M}}_{n}^{+}$ is labeled by a lower $n$, such the risk-
sensitive value $J_{n}(\pi_{1},\pi_{2},t,x)$ of a pair of Markov policies
$(\pi_{1},\pi_{2})$ and the value function
$J_{n}(t,x):=\sup_{\pi_{1}\in\Pi_{1}^{m}}\inf_{\pi_{2}\in\Pi_{2}^{m}}J_{n}(\pi_{1},\pi_{2},t,x)$.

Obviously, Assumptions 3.1, 3.2 and 4.1 still hold for each model
$\mathbb{M}_{n}^{+}$. Thus, for each $n\geq 1$, it follows from Proposition
4.1 that there exists ${\varphi}_{n}(t,x)\in\mathbb{B}_{1,1}^{1}([0,T]\times
S)$ satisfying (4.5) for the corresponding $\mathbb{M}_{n}^{+}$, that is,

$\displaystyle\begin{cases}\displaystyle{\varphi}_{n}^{\prime}(t,x)+\sup_{\phi\in
P(A(x))}\inf_{\psi\in P(B(x))}\left[\theta
r_{n}^{+}(x,\phi,\psi){\varphi}_{n}(t,x)+\int_{S}{\varphi}_{n}(t,y)q_{n}(dy|x,\phi,\psi)\right]=0,&\\\
{{\varphi}_{n}(T,x)=e^{\theta g_{n}^{+}(x)}},\end{cases}$ (B.5)

for each $x\in S$ and $t\in E_{({\varphi}_{n},x)}$ with
$m_{L}(E_{({\varphi}_{n},x)}^{c})=0$.

From (4.2) in Proposition 4.1(b), (B.5) gives the existence of Markov policies
$\phi_{n}(da|x,t)\in\Pi_{1}^{m},\psi_{n}(db|x,t)\in\Pi_{2}^{m}$ such that,

$\displaystyle\begin{cases}{\varphi}_{n}^{\prime}(t,x)+\theta
r_{n}^{+}(x,\phi_{n}^{t},\psi_{n}^{t}){\varphi}_{n}(t,x)+\int_{S}{\varphi}_{n}(t,y)q_{n}(dy|x,\phi_{n}^{t},\psi_{n}^{t})=0,\\\
{\varphi}_{n}(T,x)=e^{\theta g_{n}^{+}(x)},\end{cases}$ (B.6)

for each $x\in S$ and $t\in E_{({\varphi}_{n},x)}$ with
$m_{L}(E_{({\varphi}_{n},x)}^{c})=0$. From (B.1), (B.2) and (B.3), we obtain

$\displaystyle\begin{cases}{\varphi}_{n}^{\prime}(t,x)+\theta
r_{n}^{+}(x,\phi_{n}^{t},\psi_{n}^{t}){\varphi}_{n}(t,x)+\int_{S}{\varphi}_{n}(t,y)q_{n}(dy|x,\phi_{n}^{t},\psi_{n}^{t})=0,x\in
S_{n},\\\ {\varphi}_{n}^{\prime}(t,x)=0,x\notin S_{n},\\\
{\varphi}_{n}(T,x)=e^{\theta g_{n}^{+}(x)},x\in S_{n},\\\
{\varphi}_{n}(T,x)=1,x\notin S_{n},\end{cases}$

for each $t\in E_{({\varphi}_{n},x)}$ with
$m_{L}(E_{({\varphi}_{n},x)}^{c})=0$.

From Proposition 4.1(b),

$\displaystyle\begin{cases}{\varphi}_{n}^{\prime}(t,x)+\theta
r_{n}^{+}(x,\phi,\psi_{n}^{t}){\varphi}_{n}(t,x)+\int_{S}{\varphi}_{n}(t,y)q_{n}(dy|x,\phi,\psi_{n}^{t})\leq
0,x\in S_{n},\\\ {\varphi}_{n}^{\prime}(t,x)=0,x\notin S_{n},\\\
{\varphi}_{n}(T,x)=e^{\theta g_{n}^{+}(x)},x\in S_{n},\\\
{\varphi}_{n}(T,x)=1,x\notin S_{n},\end{cases}$ (B.7)

for each $\phi\in P(A(x))$ and $t\in E_{({\varphi}_{n},x)}$ with
$m_{L}(E_{({\varphi}_{n},x)}^{c})=0$.

Also, by Assumption 3.1(iii) and (B.2), we have $0\leq r_{n}^{+}(x,a,b)\leq
r(x,a,b)\leq M_{0}+\frac{\sqrt{2}}{2}\sqrt{\ln V_{0}(x)}$ for all $(x,a,b)\in
K$ and $n\geq 1$. Then, using Lemma 3.1 and Proposition 4.1(c) with
$V_{0}=V_{1}\equiv 1$, from (B.6) we have

$\displaystyle
e^{-\theta\left[Te^{\rho_{0}T}+M_{0}T+e^{\rho_{0}T}+M_{0}\right]V_{0}(x)}\leq{\varphi}_{n}(t,x)=J_{n}(\phi_{n},\psi_{n},t,x)\leq
LV_{0}(x),\quad\forall\ n\geq 1.$ (B.8)

Moreover, ${\varphi}_{n}(t,x)\geq 0$. From (B.1), (B.2) and (B.3), we have
$r_{n}^{+}(x,a,b)\geq r_{n-1}^{+}(x,a,b)$ for all $(x,a,b)\in K$,
$q_{n-1}(dy|x,a,b)=q(dy|x,a,b)=q_{n}(dy|x,a,b)$ for $x\in S_{n-1}$,
$q_{n-1}(dy|x,a,b)=0=q_{n}(dy|x,a,b)$ for $x\in S\setminus S_{n},$ and
$0=q_{n-1}(dy|x,a,b)$ for $x\in S_{n}\setminus S_{n-1}$, $g_{n-1}^{+}(x)\leq
g_{n}^{+}(x)$ for $x\in S_{n-1}$, $g_{n-1}^{+}(x)=0=g_{n}^{+}(x)$ for $x\in
S\setminus S_{n},$ and $0=g_{n-1}^{+}(x)\leq g_{n}^{+}(x)$ for $x\in
S_{n}\setminus S_{n-1}$. By (B.7) and Proposition 4.1(b),(e), we have, for all
$t\in E_{({\varphi}_{n},x)}$ and $n\geq 2$,

$\displaystyle\begin{cases}{\varphi}_{n}^{\prime}(t,x)+\theta
r_{n-1}^{+}(x,\phi,\psi_{n}^{t}){\varphi}_{n}(t,x)+\int_{S}{\varphi}_{n}(t,y)q_{n-1}(dy|x,\phi,\psi_{n}^{t})\leq
0,\ x\in S_{n-1},&\\\ {\varphi}_{n}^{\prime}(t,x)+\theta
r_{n-1}^{+}(x,\phi,\psi_{n}^{t}){\varphi}_{n}(t,x)+\int_{S}{\varphi}_{n}(t,y)q_{n-1}(dy|x,\phi,\psi_{n}^{t})={\varphi}_{n}^{\prime}(t,x)\leq
0,\ x\in S_{n}\setminus S_{n-1},&\\\ {\varphi}_{n}^{\prime}(t,x)=0,\ x\in
S\setminus S_{n},&\\\ {\varphi}_{n}(T,x)=e^{\theta g_{n}^{+}(x)},x\in S.\\\
\end{cases}$

Therefore,

$\displaystyle\begin{cases}{\varphi}_{n}^{\prime}(t,x)+\theta
r_{n-1}^{+}(x,\phi,\psi_{n}^{t}){\varphi}_{n}(t,x)+\int_{S}{\varphi}_{n}(t,y)q_{n-1}(dy|x,\phi,\psi_{n}^{t})\leq
0,\ x\in S,&\\\ {\varphi}_{n}(T,x)=e^{\theta g_{n}^{+}(x)},x\in S.\end{cases}$

From the proof of Proposition 4.1(c), we have

$\displaystyle\mathbb{E}_{\gamma}^{\pi_{1},\psi_{n}}\left(e^{\int_{t}^{T}\int_{A(x_{v})}\theta
r_{n-1}^{+}(x_{v},a,\psi_{n}^{v})\pi_{1}(da|x_{v},v)dv+\theta
g_{n}^{+}(x_{T})}|x_{t}=x\right)-{\varphi}_{n}(t,x)$ $\displaystyle=$
$\displaystyle\mathbb{E}_{\gamma}^{\pi_{1},\psi_{n}}\left(e^{\int_{t}^{T}\int_{A(x_{v})}\theta
r_{n-1}^{+}(x_{v},a,\psi_{n}^{v})\pi_{1}(da|x_{v},v)dv}{\varphi}_{n}(T,x_{T})|x_{t}=x\right)-{\varphi}_{n}(t,x)\leq
0.$

Therefore, by (B.3), we have

$\displaystyle J_{n-1}(\pi_{1},\psi_{n},t,x)$ (B.9) $\displaystyle=$
$\displaystyle\mathbb{E}_{\gamma}^{\pi_{1},\psi_{n}}\left(e^{\int_{t}^{T}\int_{A(x_{v})}\theta
r_{n-1}^{+}(x_{v},a,\psi_{n}^{v})\pi_{1}(da|x_{v},v)dv+\theta
g_{n-1}^{+}(x_{T})}|x_{t}=x\right)$ $\displaystyle\leq$
$\displaystyle\mathbb{E}_{\gamma}^{\pi_{1},\psi_{n}}\left(e^{\int_{t}^{T}\int_{A(x_{v})}\theta
r_{n-1}^{+}(x_{v},a,\psi_{n}^{v})\pi_{1}(da|x_{v},v)dv+\theta
g_{n}^{+}(x_{T})}|x_{t}=x\right)$ $\displaystyle\leq$
$\displaystyle{\varphi}_{n}(t,x)\quad{\rm for\ all\ }\pi_{1}\in\Pi_{1}^{m},\
(t\times x)\in[0,T]\times S.$

Further, from (B.9), we obtain

$\displaystyle\sup_{\pi_{1}\in\Pi_{1}^{m}}J_{n-1}(\pi_{1},\psi_{n},t,x)\leq{\varphi}_{n}(t,x)\
\ \ \ {\rm for\ all\ }\ t\in[0,T],x\in S.$

Therefore,

${\varphi}_{n-1}(t,x)=\sup_{\pi_{1}\in\Pi_{1}^{m}}\inf_{\pi_{2}\in\Pi_{2}^{m}}J_{n-1}(\pi_{1},\pi_{2},t,x)\leq\sup_{\pi_{1}\in\Pi_{1}^{m}}J_{n-1}(\pi_{1},\psi_{n},t,x)\leq{\varphi}_{n}(t,x).$

Thus, we have ${\varphi}_{n-1}(t,x)\leq{\varphi}_{n}(t,x)$, that is, the
sequence $\\{{\varphi}_{n},n\geq 1\\}$ is nondecreasing in $n\geq 1$, and thus
the limit

$\displaystyle{\varphi}(t,x):=\lim_{n\to\infty}{\varphi}_{n}(t,x)$ (B.10)

exists for each $(t,x)\in[0,T]\times S.$

For every $n\geq 1$ and $(t,x)\in[0,T]\times S$, we define

$\displaystyle H_{n}(t,x):=\sup_{\phi\in P(A(x))}\inf_{\psi\in
P(B(x))}\left[\theta
r_{n}^{+}(x,\phi,\psi){\varphi}_{n}(t,x)+\int_{S}{\varphi}_{n}(t,y)q_{n}(dy|x,\phi,\psi)\right],$
(B.11) $\displaystyle H(t,x):=\sup_{\phi\in P(A(x))}\inf_{\psi\in
P(B(x))}\left[\theta
r(x,\phi,\psi){\varphi}(t,x)+\int_{S}{\varphi}(t,y)q(dy|x,\phi,\psi)\right].$

We next show that $\lim_{n\to\infty}H_{n}(t,x)=H(t,x)$ for each
$(t,x)\in[0,T]\times S$.

From Assumption 4.1, there exist $\phi_{n}^{t}\in P(A(x))$ and
$\psi_{n}^{t}\in P(B(x))$ such that

$\displaystyle H_{n}(t,x)=\theta
r_{n}^{+}(x,\phi_{n}^{t},\psi_{n}^{t}){\varphi}_{n}(t,x)+\int_{S}{\varphi}_{n}(t,y)q_{n}(dy|x,\phi_{n}^{t},\psi_{n}^{t}).$

By (B.8), we have
$\limsup_{n\to\infty}H_{n}(t,x)=\lim_{m\to\infty}H_{n_{m}}(t,x)$ for some
subsequence $\\{n_{m},m\geq 1\\}$ of $\\{n,n\geq 1\\}$. For each $m\geq 1$,
under Assumption 4.1, the measurable selection theorem (e.g. Proposition 7.50
in [8]) together with Lemma 4.1 in [27] ensures the existence of
$\phi_{n_{m}}\in\Pi_{1}^{M}$ and $\psi_{n_{m}}\in\Pi_{2}^{M}$ such that

$\displaystyle H_{n_{m}}(t,x)=\sup_{\phi\in P(A(x))}\inf_{\psi\in
P(B(x))}\left[\theta
r_{n_{m}}^{+}(x,\phi,\psi){\varphi}_{n_{m}}(t,x)+\int_{S}{\varphi}_{n_{m}}(t,y)q_{n_{m}}(dy|x,\phi,\psi)\right]$
(B.12) $\displaystyle=$ $\displaystyle\theta
r_{n_{m}}^{+}(x,\phi_{n_{m}}^{t},\psi_{n_{m}}^{t}){\varphi}_{n_{m}}(t,x)+\int_{S}{\varphi}_{n_{m}}(t,y)q_{n_{m}}(dy|x,\phi_{n_{m}}^{t},\psi_{n_{m}}^{t}).$

Since $\phi_{n_{m}}^{t}\in P(A(x))$ for all $m\geq 1$ and $P(A(x))$ is
compact, there exists a subsequence $\\{\phi_{n_{m_{k}}}^{t},k\geq 1\\}$ of
$\\{\phi_{n_{m}}^{t},m\geq 1\\}$ and $\bar{\phi}^{t}\in P(A(x))$ (depending on
$(t,x)$) such that $\phi_{n_{m_{k}}}^{t}\to\bar{\phi}^{t}$ as $k\to\infty$. It
is the same for $\psi_{n_{m}}^{t}$. Thus,
$\lim_{m\to\infty}H_{n_{m}}(t,x)=\lim_{k\to\infty}H_{n_{m_{k}}}(t,x)$. Indeed,
for any fixed $(t,x)\in[0,T]\times S$, there exists $n_{0}\geq 1$ such that
$(t,x)\in[0,T]\times S_{n_{0}}$, and then $q_{n}(dy|x,a,b)=q(dy|x,a,b)$ for
all $n\geq n_{0}$ and $\lim_{n\to\infty}r_{n}^{+}(x,a,b)=r(x,a,b)$ for all
$a\in A(x),b\in B(x)$. Thus, by Lemma 8.3.7 in [28] and (B.12), (B.11) and
Assumption 4.1, we have

$\displaystyle\limsup_{n\to\infty}H_{n}(t,x)=\lim_{k\to\infty}H_{n_{m_{k}}}(t,x)$
$\displaystyle=$ $\displaystyle\lim_{k\to\infty}\theta
r_{n_{m_{k}}}^{+}(x,\phi_{n_{m_{k}}}^{t},\psi_{n_{m_{k}}}^{t}){\varphi}_{n_{m_{k}}}(t,x)+\int_{S}{\varphi}_{n_{m_{k}}}(t,y)q_{n_{m_{k}}}(dy|x,\phi_{n_{m_{k}}}^{t},\psi_{n_{m_{k}}}^{t})$
$\displaystyle=$ $\displaystyle\lim_{k\to\infty}\inf_{\psi\in
P(B(x))}\left[\theta
r_{n_{m_{k}}}^{+}(x,\phi_{n_{m_{k}}}^{t},\psi){\varphi}_{n_{m_{k}}}(t,x)+\int_{S}{\varphi}_{n_{m_{k}}}(t,y)q_{n_{m_{k}}}(dy|x,\phi_{n_{m_{k}}}^{t},\psi)\right]$
$\displaystyle\leq$ $\displaystyle\lim_{k\to\infty}\left[\theta
r_{n_{m_{k}}}^{+}(x,\phi_{n_{m_{k}}}^{t},\psi){\varphi}_{n_{m_{k}}}(t,x)+\int_{S}{\varphi}_{n_{m_{k}}}(t,y)q_{n_{m_{k}}}(dy|x,\phi_{n_{m_{k}}}^{t},\psi)\right]$
$\displaystyle=$ $\displaystyle\theta
r(x,\bar{\phi}^{t},\psi){\varphi}(t,x)+\int_{S}{\varphi}(t,y)q(dy|x,\bar{\phi}^{t},\psi),\quad\forall\psi\in
P(B(x)).$

Hence,

$\displaystyle\limsup_{n\to\infty}H_{n}(t,x)\leq\inf_{\psi\in
P(B(x))}\left[\theta
r(x,\bar{\phi}^{t},\psi){\varphi}(t,x)+\int_{S}{\varphi}(t,y)q(dy|x,\bar{\phi}^{t},\psi)\right]$
(B.13) $\displaystyle\leq$ $\displaystyle\sup_{\phi\in P(A(x))}\inf_{\psi\in
P(B(x))}\left[\theta
r(x,\phi,\psi){\varphi}(t,x)+\int_{S}{\varphi}(t,y)q(dy|x,\phi,\psi)\right]=H(t,x).$

By the similar reasoning, we have

$\displaystyle\liminf_{n\to\infty}H_{n}(t,x)\geq H(t,x).$ (B.14)

(B.13) together with (B.14) implies that $\lim_{n\to\infty}H_{n}(t,x)=H(t,x)$.
Thus, by (B.5) we have

$\displaystyle{\varphi}(t,x)={e^{\theta g(x)}}+\int_{t}^{T}\sup_{\phi\in
P(A(x))}\inf_{\psi\in P(B(x))}\left[\theta
r(x,\phi,\psi){\varphi}(s,x)+\int_{S}{\varphi}(s,y)q(dy|x,\phi,\psi)\right]ds.$
(B.15)

Since ${\varphi}(t,x)$ is the integral of a measurable function, it is an
absolutely continuous function, and so ${\varphi}(t,x)$ is differential in
a.e. $t\in[0,T]$ (for each fixed $x\in S$). We can verify that
${\varphi}(t,x)$ satisfies (4.5). To show
${\varphi}(t,x)\in\mathbb{B}_{V_{0},V_{1}}^{1}([0,T]\times S)$, since
${\varphi}(t,x)\in\mathbb{B}_{V_{0}}([0,T]\times S)$ (by (B.8), (B.10)), the
rest verifies that ${\varphi}^{\prime}(t,x)$ is $V_{1}$-bounded. Indeed, since
$\theta T|r(x,a,b)|\leq e^{2T\theta|r(x,a,b)|}\leq
e^{2T\theta(M_{0}+T\theta)}V_{0}(x)$, from (B.15) we have

$\displaystyle|{\varphi}^{\prime}(t,x)|$ $\displaystyle\leq$
$\displaystyle\frac{e^{2T\theta(M_{0}+T\theta)}}{T}\|{\varphi}\|_{V_{0}}V_{0}(x)V_{0}(x)+\|{\varphi}\|_{V_{0}}\left[\rho_{0}V_{0}(x)+2V_{0}(x)q^{*}(x)\right]$
$\displaystyle\leq$
$\displaystyle\|{\varphi}\|_{V_{0}}\left[\frac{e^{2T\theta(M_{0}+T\theta)}}{T}V_{0}^{2}(x)+\rho_{0}V_{0}(x)+2L_{0}V_{0}^{2}(x)\right]$
$\displaystyle\leq$
$\displaystyle\|{\varphi}\|_{V_{0}}M_{1}\left[\frac{e^{2T\theta(M_{0}+T\theta)}}{T}+\rho_{0}+2L_{0}\right]V_{1}(x),$

which implies that ${\varphi}(t,x)$ is in
$\mathbb{B}_{V_{0},V_{1}}^{1}(\times[0,T]\times S)$, and thus (a) is proved. ∎

##  Appendix C Proof of Theorem 5.1

###### Proof.

(a) From (5.1) and (A.2), we have, $\forall n\geq 0,(t,x)\in[0,T]\times S,$

$\displaystyle|v_{n+1}(t,x)-v_{n}(t,x)|$ $\displaystyle=$
$\displaystyle|\Gamma^{n}v_{1}(t,x)-\Gamma^{n}v_{0}(t,x)|$ (C.1)
$\displaystyle\leq$
$\displaystyle\frac{\left(\theta\|r\|+2\|q\|\right)^{n}(T-t)^{n}}{n!}\|v_{1}-v_{0}\|$
$\displaystyle\leq$
$\displaystyle\frac{\left(\theta\|r\|+2\|q\|\right)^{n}T^{n}}{n!}\|v_{1}-v_{0}\|.$

Since
$\sum_{n=0}^{\infty}\frac{\left(\theta\|r\|+2\|q\|\right)^{n}T^{n}}{n!}=e^{\left(\theta\|r\|+2\|q\|\right)T}$,

$\lim_{n\to\infty}\frac{\left(\theta\|r\|+2\|q\|\right)^{n}T^{n}}{n!}=0.$

We also obtain $\frac{\left(\theta\|r\|+2\|q\|\right)^{n}T^{n}}{n!}$
monotonically decreases when $n$ increases and $n>(\theta\|r\|+2\|q\|)T$.
Therefore, given $\varepsilon>0$, there exists $N_{1}>0$ such that, for all
$n\geq N_{1}$,

$\displaystyle\|v_{n+1}-v_{n}\|$ $\displaystyle=$
$\displaystyle\sup_{(t,x)\in[0,T]\times S}\|v_{n+1}(t,x)-v_{n}(t,x)\|$
$\displaystyle\leq$
$\displaystyle\frac{\left(\theta\|r\|+2\|q\|\right)^{n}T^{n}}{n!}\|v_{1}-v_{0}\|<\frac{\varepsilon}{2e^{\left(\theta\|r\|+2\|q\|\right)T}\left(1+\frac{2\|q\|}{\theta\|r\|}\right)}.$

Now, we prove that the sequence $\\{v_{n}(t,x),n=0,1,2,\ldots\\}$ converges.
From (C.1) and (5.2), for all $n\geq N_{1},m\geq 1$, $\forall\
(t,x)\in[0,T]\times S,$

$\displaystyle|v_{n+m}(t,x)-v_{n+1}(t,x)|$ $\displaystyle\leq$
$\displaystyle\sum\limits_{p=1}^{m-1}|v_{n+(p+1)}(t,x)-v_{n+p}(t,x)|$ (C.2)
$\displaystyle\leq$
$\displaystyle\sum\limits_{p=1}^{m-1}\frac{\left(\theta\|r\|+2\|q\|\right)^{n+p}T^{n+p}}{(n+p)!}\|v_{1}-v_{0}\|$
$\displaystyle\leq$
$\displaystyle\frac{\left(\theta\|r\|+2\|q\|\right)^{n}T^{n}}{n!}\sum\limits_{p=1}^{m-1}\frac{\left(\theta\|r\|+2\|q\|\right)^{p}T^{p}}{p!}\|v_{1}-v_{0}\|$
$\displaystyle<$
$\displaystyle\frac{\left(\theta\|r\|+2\|q\|\right)^{n}T^{n}}{n!}e^{\left(\theta\|r\|+2\|q\|\right)T}\|v_{1}-v_{0}\|<\frac{\varepsilon}{2}.$

From the above formula we can see the sequence
$\\{v_{n}(t,x),n=0,1,2,\ldots\\}$ is a Cauchy sequence. Since
$\mathbb{B}^{1}_{1,1}([0,T]\times S)$ is a Banach space, the sequence
$\\{v_{n}(t,x),n=0,1,2,\ldots\\}$ has a limit. From Proposition 4.1, we obtain
$\lim_{n\to\infty}v_{n}(t,x)=\varphi(t,x)$, for any $(t,x)\in[0,T]\times S$.
By (C.2) and let $m\to\infty$, we obtain

$\|\varphi-v_{n+1}\|=\sup_{(t,x)\in[0,T]\times
S}|\varphi(t,x)-v_{n+1}(t,x)|<\frac{\varepsilon}{2},\quad\forall n\geq N_{1}.$

(b) By Assumption 4.1, the compactness of $P(A(x))$ and $P(B(x))$ and the
continuity of $r,q,v$, there exist Markov policies
$\phi_{n}\in\Pi_{1}^{m},\psi_{n}\in\Pi_{2}^{m}$ such that (5.4) holds. That
is, $\forall n=0,1,2,\ldots$, $\forall(t,x)\in[0,T]\times S$, by (5.1),

$\displaystyle v_{n+1}(t,x)$ (C.3) $\displaystyle=$ $\displaystyle e^{\theta
g(x)}+\int_{t}^{T}\sup_{\phi\in P(A(x))}\inf_{\psi\in P(B(x))}\left[\theta
r(x,\phi,\psi)v_{n}(s,x)+\int_{S}v_{n}(s,y)q(dy|x,\phi,\psi)\right]ds$
$\displaystyle=$ $\displaystyle e^{\theta g(x)}+\int_{t}^{T}\sup_{\phi\in
P(A(x))}\left[\theta
r(x,\phi,\psi_{n}^{s})v_{n}(s,x)+\int_{S}v_{n}(s,y)q(dy|x,\phi,\psi_{n}^{s})\right]ds$
$\displaystyle=$ $\displaystyle e^{\theta g(x)}+\int_{t}^{T}\left[\theta
r(x,\phi_{n}^{s},\psi_{n}^{s})v_{n}(s,x)+\int_{S}v_{n}(s,y)q(dy|x,\phi_{n}^{s},\psi_{n}^{s})\right]ds.$
(C.4)

By (C.4) and (C.3), for any $\pi_{1}\in\Pi_{1}^{m}$, for each $x\in S$ and
$t\in E_{(\varphi,x)}$, we have

$\displaystyle-v_{n+1}^{\prime}(t,x)$ (C.5) $\displaystyle=$
$\displaystyle\theta
r(x,\phi_{n}^{t},\psi_{n}^{t})v_{n}(t,x)+\int_{S}v_{n}(t,y)q(dy|x,\phi_{n}^{t},\psi_{n}^{t})$
$\displaystyle\geq$ $\displaystyle\theta
r(x,\pi_{1}^{t},\psi_{n}^{t})v_{n}(t,x)+\int_{S}v_{n}(t,y)q(dy|x,\pi_{1}^{t},\psi_{n}^{t}).$

Together with the fact that $x_{t}(\omega)$ is a piece-wise constant (by
(2.4)), (C.5) implies

$\displaystyle-\left(e^{\int_{0}^{t}\theta
r(x_{v},\pi_{1}^{v},\psi_{n}^{v})dv}v_{n+1}(t,x_{t})\right)^{\prime}$ (C.6)
$\displaystyle=$ $\displaystyle-e^{\int_{0}^{t}\theta
r(x_{v},\pi_{1}^{v},\psi_{n}^{v})dv}\left[\theta
r(x_{t},\pi_{1}^{t},\psi_{n}^{t})v_{n+1}(t,x_{t})+v_{n+1}^{\prime}(t,x_{t})\right]$
$\displaystyle\geq$ $\displaystyle-e^{\int_{0}^{t}\theta
r(x_{v},\pi_{1}^{v},\psi_{n}^{v})dv}\theta
r(x_{t},\pi_{1}^{t},\psi_{n}^{t})v_{n+1}(t,x_{t})+e^{\int_{0}^{t}\theta
r(x_{v},\pi_{1}^{v},\psi_{n}^{v})dv}$ $\displaystyle\left[\theta
v_{n}(t,x_{t})r(x_{t},\pi_{1}^{t},\psi_{n}^{t})+\int_{S}v_{n}(t,y)q(dy|x_{t},\pi_{1}^{t},\psi_{n}^{t})\right]$
$\displaystyle=$ $\displaystyle e^{\int_{0}^{t}\theta
r(x_{v},\pi_{1}^{v},\psi_{n}^{v})dv}\int_{S}v_{n}(t,y)q(dy|x_{t},\pi_{1}^{t},\psi_{n}^{t})$
$\displaystyle+e^{\int_{0}^{t}\theta
r(x_{v},\pi_{1}^{v},\psi_{n}^{v})dv}\theta
r(x_{t},\pi_{1}^{t},\psi_{n}^{t})\left[v_{n}(t,x_{t})-v_{n+1}(t,x_{t})\right]$
$\displaystyle\geq$ $\displaystyle e^{\int_{0}^{t}\theta
r(x_{v},\pi_{1}^{v},\psi_{n}^{v})dv}\int_{S}v_{n}(t,y)q(dy|x_{t},\pi_{1}^{t},\psi_{n}^{t})-e^{t\theta\|r\|}\theta\|r\|\|v_{n+1}-v_{n}\|,\
\ \forall\ t\geq 0.$

Thus, by Lemma 3.2(b) and (C.6), we have

$\displaystyle\mathbb{E}_{\gamma}^{\pi_{1},\psi_{n}}\left[e^{\int_{s}^{T}\theta
r(x_{t},\pi_{1}^{t},\psi_{n}^{t})dt}v_{n+1}(T,x_{T})\Big{|}x_{s}=x\right]-v_{n+1}(s,x)$
$\displaystyle=$
$\displaystyle\mathbb{E}_{\gamma}^{\pi_{1},\psi_{n}}\left\\{\int_{s}^{T}\left[\left(e^{\int_{s}^{t}\theta
r(x_{v},\pi_{1}^{v},\psi_{n}^{v})dv}v_{n+1}(t,x_{t})\right)^{\prime}\right.\right.$
$\displaystyle\left.\left.+\int_{S}e^{\int_{s}^{t}\theta
r(x_{v},\pi_{1}^{v},\psi_{n}^{v})dv}v_{n+1}(t,y)q(dy|x_{t},\pi_{1}^{t},\psi_{n}^{t})\right]dt\Big{|}x_{s}=x\right\\}$
$\displaystyle\leq$
$\displaystyle\mathbb{E}_{\gamma}^{\pi_{1},\psi_{n}}\left\\{\int_{s}^{T}\left[-\int_{S}q(dy|x_{t},\pi_{1}^{t},\psi_{n}^{t})e^{\int_{s}^{t}\theta
r(x_{v},\pi_{1}^{v},\psi_{n}^{v})dv}v_{n}(t,y)\right.\right.$
$\displaystyle+e^{t\theta\|r\|}\theta\|r\|\|v_{n+1}-v_{n}\|\left.\left.+\int_{S}e^{\int_{s}^{t}\theta
r(x_{v},\pi_{1}^{v},\psi_{n}^{v})dv}v_{n+1}(t,y)q(dy|x_{t},\pi_{1}^{t},\psi_{n}^{t})\right]dt\Big{|}x_{s}=x\right\\}$
$\displaystyle=$
$\displaystyle\mathbb{E}_{\gamma}^{\pi_{1},\psi_{n}}\left\\{\int_{s}^{T}\left[\int_{S}q(dy|x_{t},\pi_{1}^{t},\psi_{n}^{t})e^{\int_{s}^{t}\theta
r(x_{v},\pi_{1}^{v},\psi_{n}^{v})dv}\left(v_{n+1}(t,y)-v_{n}(t,y)\right)\right.\right.$
$\displaystyle\left.\left.+e^{t\theta\|r\|}\theta\|r\|\|v_{n+1}-v_{n}\|\right]dt\Big{|}x_{s}=x\right\\}$
$\displaystyle\leq$ $\displaystyle
e^{T\theta\|r\|}\|v_{n+1}-v_{n}\|+2\|q\|\|v_{n+1}-v_{n}\|\int_{0}^{T}e^{t\theta\|r\|}dt$
$\displaystyle\leq$ $\displaystyle
e^{T\theta\|r\|}\|v_{n+1}-v_{n}\|+2\|q\|\|v_{n+1}-v_{n}\|\frac{1}{\theta\|r\|}e^{T\theta\|r\|}$
$\displaystyle\leq$ $\displaystyle
e^{T\theta\|r\|}\|v_{n+1}-v_{n}\|\left(1+\frac{2\|q\|}{\theta\|r\|}\right),\ \
\ \ \forall\ s\in[0,T].$

For all $n\geq N_{1}$, by (5.2),

$\displaystyle
J(\pi_{1},\psi_{n},s,x)-v_{n+1}(s,x)<\frac{\varepsilon}{2}\quad{\rm for\ all}\
(s,x)\in[0,T]\times S,\ {\rm for\ all}\ \pi_{1}\in\Pi_{1}.$

Therefore,

$J(\phi_{n},\psi_{n},t,x)-v_{n+1}(t,x)<\frac{\varepsilon}{2},\quad\forall(t,x)\in[0,T]\times
S.$ (C.7)

A similar proof gives

$J(\phi_{n},\psi_{n},t,x)-v_{n+1}(t,x)>-\frac{\varepsilon}{2},\quad\forall(t,x)\in[0,T]\times
S.$ (C.8)

Thus, by (C.7) and (C.8), we get

$|J(\phi_{n},\psi_{n},t,x)-v_{n+1}(t,x)|<\frac{\varepsilon}{2},\quad\forall(t,x)\in[0,T]\times
S.$ (C.9)

From (5.3) and (C.9),

$\displaystyle\sup_{(t,x)\in[0,T]\times
S}|J(\phi_{n},\psi_{n},t,x)-\varphi(t,x)|$ $\displaystyle=$
$\displaystyle\sup_{(t,x)\in[0,T]\times
S}|J(\phi_{n},\psi_{n},t,x)-v_{n+1}(t,x)+v_{n+1}(t,x)-\varphi(t,x)|$
$\displaystyle\leq$ $\displaystyle\sup_{(t,x)\in[0,T]\times
S}\left(|J(\phi_{n},\psi_{n},t,x)-v_{n+1}(t,x)|+|v_{n+1}(t,x)-\varphi(t,x)|\right)<\varepsilon.$

∎

## References

  * [1] Barron, E.N. (2013). _Game Theory, An Introduction. 2nd Ed._ John Wiley & Sons.
  * [2] Başar, T. (1999). Nash equilibria of risk-sensitive nonlinear stochastic differential games. _J. Optim. Theory Appl._ 100(3), 479-498.
  * [3] Basu, A. and Ghosh, M.K. (2012). Zero-sum risk-sensitive stochastic differential games. _Math. Oper. Res._ 37(3), 437-449.
  * [4] Basu, A. and Ghosh, M.K. (2014). Zero-sum risk-sensitive stochastic games on a countable state space. Stochastic Process. Appl. 124, 961-983.
  * [5] Basu, A. and Ghosh, M.K. (2018). Nonzero-sum risk-sensitive stochastic games on a countable state space. _Mathematics of Operations Research_ 43(2), 516-532.
  * [6] Ba$\ddot{{\rm u}}$erle, N. and Rieder, U. (2014). Markov decision processes with average-value-at-risk criteria, Math. Methods Oper. Res. 74 (2011), 361-379.
  * [7] Ba$\ddot{{\rm u}}$erle, N. and Rieder, U. (2017). Zero-sum risk-sensitive stochastic games. Stochastic Process. Appl. 127, 622-642.
  * [8] Bertsekas, D. and Shreve, S. (1996). Stochastic Optimal Control: The Discrete-Time Case. Academic Press, Inc.
  * [9] Biswas, A., Saha, S. (2020). Zero-Sum Stochastic Differential Games with Risk-Sensitive Cost. _Appl Math Optim_ 81, 113-140.
  * [10] Bowling, M., Veloso, M. (2000). An analysis of stochastic game theory for multi-agent reinforcement learning[R]. Pittsburgh, PA: Technical Report CMU-CS-00-165, CMU.
  * [11] El-Karoui, N., Hamadene, S. (2003). BSDE and risk-sensitive control, zero-sum and nonzero-sum game problems of stochastic functional differential equations, _Stochastic Process. Appl._ 107, 145-169.
  * [12] Fan, K. (1953). Minimax theorems. _Proceedings of the National Academy of Sciences_ 39, 42-47.
  * [13] Feinberg, E.A., Mandava, M. and Shiryaev, A.N. (2014). On solutions of Kolmogorov’s equations for nonhomogeneous jump Markov processes. J. Math. Anal. Appl. 411, 261-270.
  * [14] Filar, J.A., Krass, D., and Ross, K. W. (1995). Percentile performance criteria for limiting average Markov control problems. _IEEE Transactions on Automatic Control_ 40, 2-10.
  * [15] Filar, J.A. and Vrieze, K. (1997). _Competitive Markov Decision Processes_. Springer Verlag, New York.
  * [16] Fleming, W.H., Hernandez-Hernandez, D. (2011). On the value of stochastic differential games. _Commun. Stoch. Anal._ 5, 341-351.
  * [17] Fu, M.C., Hong, L.J., and Hu, J.Q. (2009). Conditional Monte Carlo estimation of quantile sensitivities. _Management Science_ 55(12), 2019-2027.
  * [18] Ghosh, M.K.; Kumar, K.S. and Pal, C. (2016). Zero-sum risk-sensitive stochastic games for continuous time Markov chains. Stoch. Anal. Appl. 34(5), 835-851
  * [19] Ghosh, M.K. and Saha, S. (2014). Risk-sensitive control of continuous time Markov chains. _Stochastics_ 86, 655-675.
  * [20] Guo, X.P. (2007). Continuous–time Markov decision processes with discounted rewards: The case of Polish spaces. Math. Oper. Res. 32, 73-87.
  * [21] Guo, X.P. and Hern$\acute{a}$ndez-Lerma, O. (2009). Continuous-Time Markov Decision Processes: Theory and Applications. Springer-Verlag, Berlin.
  * [22] Guo, X.P., Huang, X.X. and Huang, Y.H. (2015). Finite-horizon optimality for continuous-time Markov decision processes with unbounded transition rates. Adv. in Appl. Probab. 47, 1064-1087.
  * [23] Guo, X.P., Huang, Y.H. and Song, X.Y. (2012). Linear programming and constrained average optimality for general continuous-time Markov decision processes in history-dependent policies. SIAM J. Control Optim. 50, 23-47.
  * [24] Guo, X.P., Liu, Q., Zhang, Y. (2019). Finite horizon risk-sensitive continuous-time Markov decision processes with unbounded transition and cost rates. _4OR_ 17, 427-442.
  * [25] Guo, X.P. and Piunovskiy, A. (2011). Discounted continuous-time Markov decision processes with constraints: unbounded transition and loss rates, Math. Oper. Res. 36, 105-132.
  * [26] Guo, X.P. and Song, X. (2011). Discounted continuous-time constrained Markov decision processes in Polish spaces. Ann. Appl. Probab. 21, 2016-2049.
  * [27] Guo, X.P. and Zhang, J.Y. (2019). Risk-sensitive continuous-time Markov decision processes with unbounded rates and Borel spaces. Discrete Event Dynamic Systems: Theory and Applications. 30, 445-471.
  * [28] Hern$\acute{a}$ndez-Lerma, O. and Lasserre, J.B. (1999). Further Topics on Discrete-Time Markov Control Processes. Springer-Verlag, New York.
  * [29] Hern$\acute{a}$ndez-Lerma, O., Vega-Amaya, O., and Carrasco, G. (1999). Sample-path optimality and variance-minimization of average cost Markov control processes. _ISIAM Journal on Control and Optimization_ 38, 79-93.
  * [30] Hong, L.J., Hu, Z., and Liu, G. (2014). Monte Carlo methods for value-at-risk and conditional value-at-risk: A review. _ACM Transactions on Modeling and Computer Simulation_ 24, 1-37.
  * [31] Howard, R., Matheson, J. (1972). Risk-sensitive Markov decision proceses. _Management Science_ 18, 356-369.
  * [32] Huang, Y.H. and Guo, X.P. (2016). Minimum average value-at-risk for finite horizon semi-Markov decision processes in continuous time. SIAM J. Optim. 26(1), 1-28.
  * [33] Jacobson, D. (1973). Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic differential games. _IEEE Trans. Automat. Contr._ 18(2), 124-131.
  * [34] Kahneman, D. and Tversky, A. (1979). Prospect theory: An analysis of decision under risk. _Econometrica_ 47(2), 263-291.
  * [35] Kumar, K.S. and Chandan, P. (2013). Risk-sensitive control of jump process on denumerable state space with near monotone cost. Appl. Math. Optim. 68, 311-331.
  * [36] Kumar, K.S. and Chandan, P. (2015). Risk-sensitive control of continuous-time Markov processes with denumerable state space. Stochastic Anal. Appl. 33, 863-881.
  * [37] Littman, M.L. (2001). Value-function reinforcement learning in Markov games. _Journal of Cognitive Systems Research_ 2, 55-66.
  * [38] Neyman, A. and Sorin, S. (Eds.) (2003). _Stochastic Games and Applications_. Springer.
  * [39] Nowak, A.S. (1985). Measurable selection theorems for minimax stochastic optimization problems, SIAM J. Control Optim. 23, 466-476.
  * [40] Piunovskiy, A. and Zhang, Y. (2011). Discounted continuous-time Markov decision processes with unbounded rates: the convex analytic approach. SIAM J. Control Optim. 49, 2032-2061.
  * [41] Puterman, M.L. (1994). _Markov Decision Processes: Discrete Stochastic Dynamic Programming_. New York: John Wiley & Sons.
  * [42] Shapley, L.S. (1953). Stochastic games. _Proceedings of the National Academy of Sciences_ 39(10), 1095-1100.
  * [43] Tembine, H., Zhu, Q., Başar, T. (2011). Risk-sensitive mean-field stochastic differential games. _IFAC Proceedings_ 44(1), 3222-3227.
  * [44] Wei, Q.D. (2018). Zero-sum games for continuous-time Markov jump processes with risk-sensitive finite-horizon cost criterion. _Operations Research Letters_ 46, 69-75.
  * [45] White, D.J. (1988). Mean, variance, and probabilistic criteria in finite Markov decision processes: A review. Journal of Optimization: Theory & Applications 56, 1-29.
  * [46] Yushkevich, A.A. (1977). Controlled Markov models with countable state and continuous time. Theory Probab. Appl. 22, 215–235.
  * [47] Xia, L. (2016). Optimization of Markov decision processes under the variance criterion. _Automatica_ 73, 269-278.