# Risk Filtering and Risk-Averse Control of Markovian Systems Subject to Model
Uncertainty

Tomasz R. Bielecki Department of Applied Mathematics, Illinois Institute of
Technology
10 W 32nd Str, Building RE, Room 220, Chicago, IL 60616, USA
Emails<EMAIL_ADDRESS>(T. R. Bielecki), and<EMAIL_ADDRESS>(I. Cialenco)
URLs: http://math.iit.edu/b̃ielecki and http://cialenco.com Igor Cialenco
11footnotemark: 1 Andrzej Ruszczyński Department of Management Science and
Information Systems, Rutgers University,
100 Rockafeller Road, Room 5182, Piscataway, NJ 08854, USA
Email<EMAIL_ADDRESS>URL: http://www.rusz.rutgers.edu/

(First circulated: June 18, 2022)

###### Abstract

We consider a Markov decision process subject to model uncertainty in a
Bayesian framework, where we assume that the state process is observed but its
law is unknown to the observer. In addition, while the state process and the
controls are observed at time $t$, the actual cost that may depend on the
unknown parameter is not known at time $t$. The controller optimizes total
cost by using a family of special risk measures, that we call risk filters and
that are appropriately defined to take into account the model uncertainty of
the controlled system. These key features lead to non-standard and non-trivial
risk-averse control problems, for which we derive the Bellman principle of
optimality. We illustrate the general theory on two practical examples:
optimal investment and clinical trials.

##  1 Introduction

We study a risk-averse Markov decision problem (MDP) subject to uncertainty
about the underlying dynamics as well as uncertainty about the risk-averse
criterion.

Literature concerning risk-averse MDPs is rather abundant, and we refer to
e.g. [FR22, FR18] and references therein. Similarly, there is a vast
literature on MDPs subject to model uncertainty, and we refer to [BCC+19] for
an overview of the classical methodologies on this topic. However, to the best
of our knowledge, the present study is the first systematic study of risk-
averse MDPs subject to model uncertainty. The earlier effort in [LRZ21]
focuses on the CVaR criterion that has an equivalent expected value
formulation. It needs to be stressed that we are not only concerned with
uncertainty regarding the underlying dynamics, but also uncertainty about the
optimization criterion, which is a novel and important practical feature, as
two examples below show. While frequent in machine learning literature, mainly
concerned with the expected value criterion, such as [LS20, SB18], it has not
been addressed in the risk-averse case.

The Knightian uncertainty that we consider is parametric in nature, and our
approach to tackle the respective MDP is rooted in the Bayesian methodology.
It came to us as quite a surprise that accounting for possible uncertainty
about the optimization criterion leads to rather intricate conceptual ideas
and technical manipulations. In order to avoid measurability and integrability
issues that are notorious and intrinsic in MDPs on general state and action
spaces, and that would quite likely burden the main takeaways from this study,
we decided to work with discrete state, action and parameter spaces. However,
morally, the results should hold true in much more generality, that will be
addressed in future works. We chose to use integral notation with respect to
the state variables, which is much more pleasing to the eye and lighter then
the summation notation. We keep the summation with respect to the time
variable though, whenever needed.

The solution to the considered risk-averse MDP hinges on the key and new
concepts of dynamic risk filters and recursive dynamic risk filters. This, in
particular allows to derive a version of the dynamic programming routine
suited to the needs of our uncertain risk-averse MDP.

The paper is organized as follows. In Section 2 we set the stage and define
MDP and the model uncertainty framework. Also here we introduce a series of
probability measures and some of their properties used frequently in the
sequel. Section 3 is devoted to risk filters, starting with the definition and
some fundamental properties of these objects. The key concept of parameter
consistency of risk filters is introduced in Section 3.2, while the time
consistency of risk filters is studied in Section 3.3. In this section we
provide a characterization of parameter consistent and time consistent risk
filters; cf. Theorem 3.14. Also here we discuss two important examples of risk
filters: expectation of an additive functional, Example 3.15, and risk-
sensitive criteria in the context of clinical trials, Example 3.6. The
structure Theorem 3.14 leads to the notation of recursive risk filters,
introduced in Section 4. Section 5.3 is devoted to the risk-averse control
problem. In Section 5.1 we derive the Bayes operator for the posterior of the
parameter of interest. Then, we derive the dynamic programming backward
recursion for the classical additive reward case; Section 5.2. Here, as a
particular case, we briefly discuss the optimal investment and consumption
problem, when the investor faces Knightian uncertainty and unknown risk-
aversion parameter; Example 5.4. We conclude with the solution to the optimal
control problem for a general recursive risk filer: Theorem 5.6.

Finally, we want to mention that while writing this manuscript we strove to
keep a balance between heavy notations and rigor. Nevertheless, some formulas
still may appear overwhelming, which is typically the case for MDPs.

##  2 Markov Decision Processes with Model Uncertainty

We consider an observed, controlled random process
$X=\\{X_{t}\\}_{t=1,\dots,T}$. The corresponding state space is a finite set
$\mathcal{X}$. The underlying probability space that we will work with is
canonical. It includes the space of paths of $X$:
$\varOmega=\underbrace{\mathcal{X}\times\cdots\times\mathcal{X}}_{T\text{
times}}=(\mathcal{X})^{T}$, endowed with the canonical product $\sigma$-field
$\mathcal{F}=\underbrace{2^{\mathcal{X}}\otimes\cdots\otimes
2^{\mathcal{X}}}_{T\text{ times}}$. The elements of $\Omega$ are
$\omega=(\omega_{1},\ldots,\omega_{T})$. We use $x_{t}$ to denote the
canonical projections at time $t$, so that $X_{t}(\omega)=x_{t}=\omega_{t}$.
We let $\\{\mathcal{F}_{t}^{X}\\}_{t=1,\dots,T}$ to denote the canonical
filtration generated by the process $X$, so that
$\mathcal{F}_{t}^{X}=\underbrace{2^{\mathcal{X}}\otimes\cdots\otimes
2^{\mathcal{X}}}_{t\text{
times}}\otimes\underbrace{\\{\Omega,\emptyset\\}\otimes\cdots\otimes\\{\Omega,\emptyset\\}}_{T-t\text{
times}}$. We will make use of the notations $\mathcal{T}=\\{1,\ldots,T\\}$ and
$\mathcal{T}_{t}=\\{t,\ldots,T\\}$.

The control space is given by a finite set $\mathcal{U}$, and the set of
admissible controls at step $t$ is given by a multifunction
$\mathcal{U}_{t}:\mathcal{X}\rightrightarrows\mathcal{U}$ with nonempty
values. We consider a parametric family of transition kernels
$K_{\theta}:\mathcal{X}\times\mathcal{U}\rightarrow\mathcal{P}(\mathcal{X})$,
where $\mathcal{P}(\mathcal{X})$ is the space of probability measures on
$\mathcal{X}$, and $\theta\in\boldsymbol{\Theta}$ represents an unknown
parameter. Here, $\boldsymbol{\Theta}$ is a finite set. The unknown true value
of the parameter $\theta$ is $\theta^{*}$.

We will consider the Bayesian setting, and therefore, we consider the product
space $\widehat{\varOmega}=\varOmega\times\boldsymbol{\Theta}$ endowed with
product $\sigma$-algebra $\widehat{\mathcal{F}}:=\mathcal{F}\otimes
2^{\boldsymbol{\Theta}}$. We denote $\widehat{\omega}=(\omega,\theta)$ and
$\widehat{\omega}_{t}=(\omega_{t},\theta)$. In accordance with the Bayesian
setting we denote by $\Theta$ a random variable on
$(\widehat{\varOmega},\widehat{\mathcal{F}})$ with values in
$\boldsymbol{\Theta}$, and with $\Theta(\widehat{\omega})=\theta$. We also
assume that some _prior distribution_ $\xi_{1}$ of $\Theta$ (supported in
$\boldsymbol{\Theta}$) is available.

The process $\\{X_{t}\\}_{t\in\mathcal{T}}$ considered as a process on
$(\widehat{\varOmega},\widehat{\mathcal{F}})$ is denoted as
$\widehat{X}=\\{\widehat{X}_{t}\\}_{t\in\mathcal{T}}$, and
$\widehat{X}_{t}(\widehat{\omega})=X_{t}(\omega)$. Accordingly, the canonical
filtration generated by the process $\widehat{X}$ is given as
$\\{\mathcal{F}_{t}^{\widehat{X}}=\mathcal{F}_{t}^{X}\otimes
2^{\boldsymbol{\Theta}},\ t\in\mathcal{T}\\}$.

At time $t$, the history of observed states is
$h_{t}=(x_{1},x_{2},\dots,x_{t})$, while all the information available for
making a decision is $g_{t}=(x_{1},u_{1},x_{2},u_{2},\dots,x_{t})$. We use
$\mathcal{H}_{t}:=\mathcal{X}^{t}=\underbrace{\mathcal{X}\times\cdots\times\mathcal{X}}_{t\text{
times}}$ to denote the spaces of possible state histories $h_{t}$. We make
distinction of $g_{t}$ and $h_{t}$ because we should make decision of $u_{t}$
based on $g_{t}$ as the past controls $u_{1},\dots,u_{t-1}$ are also taken
into consideration when estimating the conditional distribution of $\theta$.
We write $H_{t}$ for $(X_{1},\dots,X_{t})$ and $\widehat{H}_{t}$ for
$(\widehat{X}_{1},\dots,\widehat{X}_{t})$.

A _history-dependent admissible policy_ $\pi=(\pi_{1},\dots,\pi_{T})$ is a
sequence of functions $\pi_{t}(g_{t})$ such that
$\pi_{t}(g_{t})\in\mathcal{U}_{t}(x_{t})$ for all possible $g_{t}$. One can
easily prove that for such an admissible policy $\pi$, each $\pi_{t}$ reduces
to a function of $h_{t}=(x_{1},x_{2},\dots,x_{t})$,111We are still using
$\pi_{s}$ to denote the decision rule; it will not lead to any
misunderstanding. as $u_{s}=\pi_{s}(x_{1},\dots,x_{s})$ for all
$s=1,\dots,t-1$. Therefore the set of admissible policies is

$\varPi=\
\big{\\{}\,\pi=(\pi_{1},\dots,\pi_{T})\,:\,\;\pi_{t}(x_{1},\dots,x_{t})\in\mathcal{U}_{t}(x_{t}),\
t\in\mathcal{T}\,\big{\\}}.$

Any policy $\pi\in\varPi$ defines the control process, also denoted by
$\pi=\\{\pi_{t}\\}_{t\in\mathcal{T}}$, with
$\pi_{t}=\pi_{t}(X_{1},\ldots,X_{t})$. We make a distinction between
$u_{t}=\pi_{t}(x_{1},\ldots,x_{t})$ and $\pi_{t}=\pi_{t}(X_{1},\ldots,X_{t})$.

As said in the Introduction, even though we work with discrete spaces
$\mathcal{X}$ and $\boldsymbol{\Theta}$, we are using the more convenient
integral notation, rather than the summation notation.

For a fixed initial state $x_{1}$, every policy $\pi\in\varPi$, and every
$\theta\in\boldsymbol{\Theta}$, a probability measure $P^{\pi}_{\theta}$ on
$(\varOmega,\mathcal{F})$ is uniquely defined by:

$\displaystyle P^{\pi}_{\theta}(A_{1}\times A_{2}\times\ldots$
$\displaystyle\times A_{T-1}\times
A_{T})=\int_{A_{1}}\int_{A_{2}}\cdots\int_{A_{T-1}}K_{\theta}(A_{T}|x_{T-1},\pi_{T-1}(x_{1},\ldots,x_{T-1}))$
$\displaystyle\times
K_{\theta}(dx_{T-1}|x_{T-2},\pi_{T-2}(x_{1},\ldots,x_{T-2}))\times\cdots$
$\displaystyle\quad\cdots\times
K_{\theta}(dx_{2}|x_{1},\pi_{1}(x_{1}))\delta_{x_{1}}(dy),\qquad
A_{t}\subset\mathcal{X},\ t\in\mathcal{T},$ (2.1)

where, as usual, $\delta_{x}$ denotes the Dirac measure concentrated at $x$.
In particular,

$P^{\pi}_{\theta}(A)=P^{\pi}_{\theta}(X\in
A)=P^{\pi}_{\theta}(\\{\omega\in\Omega\,:\,X(\omega)\in A\\}),\quad
A\subset\Omega.$

The true but unknown measure under the policy $\pi$ is $P^{\pi}_{\theta^{*}}$.
This measure gives the true law of the canonical process $X$ subject to
control strategy $\pi$.

Given the prior distribution $\xi_{1}$, a probability measure $P^{\pi}$ on
$(\widehat{\varOmega},{\widehat{\mathcal{F}}})$ is defined as well:

$\displaystyle P^{\pi}(A\times
D)=\int_{D}\,P^{\pi}_{\theta}(A)\;\xi_{1}(d\theta),\quad A\subset\Omega,\
D\subset\boldsymbol{\Theta}.$ (2.2)

In particular,

$P^{\pi}(A\times
D)=P^{\pi}(\\{\widehat{\omega}\in\widehat{\Omega}\,:\,\widehat{X}(\widehat{\omega})\in
A,\,\Theta(\widehat{\omega})\in D\\}).$

Clearly, $\xi_{1}$ is the marginal of $P^{\pi}$, that is
$\xi_{1}(D)=P^{\pi}(\Omega\times D)$. To simplify the ensuing study, we assume
that for any $t\in\mathcal{T}$ and $h_{t}\in\mathcal{H}_{t}$ we have
$P^{\pi}(\widehat{H}_{t}=h_{t})>0.$ This assumption is of course an assumption
about the kernels $K_{\theta}$, $\theta\in\boldsymbol{\Theta}$.

Furthermore, for each $t=1,\dots,T-1$ and for each history
$h_{t}\in\mathcal{H}_{t}$, we define the set of tail control strategies

$\Pi^{t,h_{t}}=\\{\pi^{t,h_{t}}:\pi_{t}^{t,h_{t}}=\pi_{t}(h_{t}),\
\pi_{s}^{t,h_{t}}(x_{t+1},\dots,x_{s})=\pi_{s}(h_{t},x_{t+1},\dots,x_{s}),\
s\in\mathcal{T}_{t+1},\ \pi\in\Pi\\}.$

In addition, for $t=1,\ldots,T-1$, and for each
$\theta\in\boldsymbol{\Theta}$, $h_{t}\in\mathcal{H}_{t}$, and
$\pi^{t,h_{t}}\in\Pi^{t,h_{t}}$ we construct a probability measure
$P^{\pi^{t,h_{t}}}_{\theta,t+1,T}$ on $\mathcal{X}^{T-t}$ in analogy to (2.1).
Specifically, we put

$\displaystyle P^{\pi^{t,h_{t}}}_{\theta,t+1,T}$
$\displaystyle(A_{t+1}\times\cdots\times
A_{T})=\int_{A_{t+1}}\int_{A_{t+2}}\cdots\int_{A_{T-1}}K_{\theta}(A_{T}|x_{T-1},\pi_{T-1}(h_{t},x_{t+1},\ldots,x_{T-1}))$
$\displaystyle\cdot
K_{\theta}(dx_{T-1}|x_{T-2},\pi_{T-2}(h_{t},x_{t+1},\ldots,x_{T-2}))\cdots
K_{\theta}(dx_{t+2}|x_{t+1},\pi_{t+1}(h_{t},x_{t+1}))$
$\displaystyle\qquad\cdot K_{\theta}(dx_{t+1}|x_{t},\pi_{t}(h_{t})),\qquad
A_{s}\subset\mathcal{X},\ s\in\mathcal{T}_{t+1}.$ (2.3)

We proceed with three technical results that are rather straightforward
consequences of the above set-up.

######  Lemma 2.1.

For any $h_{t}\in\mathcal{H}_{t}$, and $A_{s}\subset\mathcal{X},\
s\in\mathcal{T}_{t+1}$, $\pi\in\Pi$, and the corresponding
$\pi^{t,h_{t}}\in\Pi^{t,h_{t}}$ we have that

$P^{\pi^{t,h_{t}}}_{\theta,t+1,T}(A_{t+1}\times\ldots\times
A_{T})=P^{\pi}_{\theta}(X_{t+1}\in A_{t+1},\ldots,X_{T}\in
A_{T}|H_{t}=h_{t}).$ (2.4)

###### Proof.

First, note that222To further simplify the notation we write
$K_{\theta}(x|...)$ in place of $K_{\theta}(\\{x\\}|...)$. In a similar way,
for a probability measure $Q$ on $\mathcal{Q}$, and $y\in\mathcal{Q}$, we may
write $Q(y)$ instead of $Q(\\{y\\})$.

$P_{\theta}^{\pi}(X_{1}=x_{1},\ldots,X_{t}=x_{t})=K_{\theta}(x_{2}|x_{1},\pi_{1}(x_{1}))\,K_{\theta}(x_{3}|x_{2},\pi_{2}(x_{1},x_{2}))\cdots
K_{\theta}(x_{t}|x_{t-1},\pi_{t-1}(x_{1},\ldots x_{t})).$

On the other hand,

$\displaystyle P^{\pi}_{\theta}$ $\displaystyle(X_{t+1}\in
A_{t+1},\ldots,X_{T}\in
A_{T},H_{t}=h_{t})=P^{\pi}_{\theta}(X_{1}=x_{1},\ldots,X_{t}=x_{t},X_{t+1}\in
A_{t+1},\ldots,X_{T}\in A_{T})$
$\displaystyle=\int_{\\{x_{1}\\}}\ldots\int_{\\{x_{t}\\}}P_{\theta,t+1,T}^{\pi^{t,\bar{h}_{t}}}(A_{t+1}\times\cdots\times
A_{T})K_{\theta}(d\bar{x}_{t}|\bar{x}_{t-1},\pi_{t-1}(\bar{h}_{t-1}))\cdots
K_{\theta}(d\bar{x}_{2}|\bar{x}_{1},\pi_{1}(\bar{x}_{1}))\delta_{x_{1}}(y)$
$\displaystyle=P_{\theta,t+1,T}^{\pi^{t,h_{t}}}(A_{t+1}\times\cdots\times
A_{T})K_{\theta}(x_{t}|x_{t-1},\pi_{t-1}(h_{t-1}))\cdots
K_{\theta}(x_{2}|x_{1},\pi_{1}(x_{1})).$

Combining the above we immediately have (2.4). ∎

For future reference we denote by $P^{\pi^{t,h_{t}}}_{\theta,t+1}$ a measure
on $(\mathcal{X},2^{\mathcal{X}})$ defined as

$P^{\pi^{t,h_{t}}}_{\theta,t+1}(B)=P^{\pi^{t,h_{t}}}_{\theta,t+1,T}(B\times\mathcal{X}^{T-t-1})=P^{\pi}_{\theta,t+1,T}(X_{t+1}\in
B).$ (2.5)

Thus, we have that, for $t\leq T-1$,

$P_{\theta,t+1}^{\pi^{t,h_{t}}}(B)=\int_{B}\,K_{\theta}(dx_{t+1}\;|\;x_{t},\pi_{t}(h_{t}))=K_{\theta}(B\;|\;x_{t},\pi_{t}(h_{t}))=P^{\pi}_{\theta}(X_{t+1}\in
B|H_{t}=h_{t}).$ (2.6)

Next, we construct a probability measure $P^{\pi^{t,h_{t}}}_{t+1,T}$ on
$\mathcal{X}^{T-t}\times\boldsymbol{\Theta}$ as

$\displaystyle P^{\pi^{t,h_{t}}}_{t+1,T}(A\times
D)=\int_{D}\,P^{\pi^{t,h_{t}}}_{\theta,t+1,T}(A)\;\xi^{\pi,h_{t}}_{t}(d\theta),\quad
A\in\underbrace{2^{\mathcal{X}}\otimes\cdots\otimes
2^{\mathcal{X}}}_{T-t\text{ times}},\ D\in 2^{\boldsymbol{\Theta}},$ (2.7)

where $\xi^{\pi,h_{t}}_{t}\in\mathcal{P}(\boldsymbol{\Theta})$, is given as

$\xi^{\pi,h_{t}}_{t}(D)=P^{\pi}(\Theta\in D\,|\,\widehat{H}_{t}=h_{t}),\
\textrm{for}\
t=2,\ldots,T,\quad\textrm{and}\quad\xi^{\pi,h_{1}}_{1}(D)=\xi_{1}(D).$ (2.8)

We note that we have the following indenty for the conditional measure

$P_{t+1,T}^{\pi^{t,h_{t}}}(A\;|\;\Theta=\theta)=P_{\theta,t+1,T}^{\pi^{t,h_{t}}}(A),\quad\theta\in\boldsymbol{\Theta},\
A\in 2^{\mathcal{X}}\otimes\cdots\otimes 2^{\mathcal{X}}.$ (2.9)

######  Lemma 2.2.

Let $A\in\underbrace{2^{\mathcal{X}}\otimes\cdots\otimes
2^{\mathcal{X}}}_{T-t\text{ times}}$, $D\in 2^{\boldsymbol{\Theta}}$,
$h_{t}\in\mathcal{H}_{t}$, and $\pi\in\Pi$. Then:
(i) We have

$P^{\pi^{t,h_{t}}}_{t+1,T}(A\times
D)=P^{\pi}((\widehat{X}_{t+1},\widehat{X}_{t+2},\ldots,\widehat{X}_{T-1},\widehat{X}_{T})\in
A,\boldsymbol{\Theta}\in D|\widehat{H}_{t}=h_{t}).$ (2.10)

###### Proof.

First, in view of (2.8) and (2.2) we note that

$\xi_{t}^{\pi,h_{t}}(D)=\frac{\int_{D}P_{\theta}^{\pi}(H_{t}=h_{t})\;\xi_{1}(d\theta)}{P^{\pi}(\widehat{H}_{t}=h_{t})}.$

Thus,

$P^{\pi^{t,h_{t}}}_{t+1,T}(A\times
D)=\frac{\int_{D}P^{\pi^{t,h_{t}}}_{\theta,t+1,T}(A)P_{\theta}^{\pi}(H_{t}=h_{t})\;\xi_{1}(d\theta)}{P^{\pi}(\widehat{H}_{t}=h_{t})}.$
(2.11)

On the other hand, using (2.2) and Lemma 2.1, we have

$\displaystyle
P^{\pi}((\widehat{X}_{t+1},\widehat{X}_{t+2},\ldots,\widehat{X}_{T-1},\widehat{X}_{T})\in
A,\Theta\in D|\widehat{H}_{t}=h_{t})$
$\displaystyle=\frac{\int_{D}P_{\theta}^{\pi}((X_{t+1},\ldots,X_{T})\in
A,H_{t}=h_{t})\;\xi_{1}(d\theta)}{P^{\pi}(\widehat{H}_{t}=h_{t})}$
$\displaystyle=\frac{\int_{D}P_{\theta}^{\pi}((X_{t+1},\ldots,X_{T})\in
A\;|\;H_{t}=h_{t})P^{\pi}_{\theta}(H_{t}=h_{t})\;\xi_{1}(d\theta)}{P^{\pi}(\widehat{H}_{t}=h_{t})}$
$\displaystyle=\frac{\int_{D}P_{\theta,t+1}^{\pi^{t,h_{t}}}(A)P^{\pi}_{\theta}(H_{t}=h_{t})\;\xi_{1}(d\theta)}{P^{\pi}(\widehat{H}_{t}=h_{t})}.$

This, combined with (2.11) concludes the proof of part (i).

∎

######  Remark 2.3.

Formally, taking $T=t+1$ in (2.9) we obtain

$P^{\pi^{t,h_{t}}}_{t+1,t+1|\theta}(A)=P^{\pi^{t,h_{t}}}_{\theta,t+1,t+1}(A)=P^{\pi^{t,h_{t}}}_{\theta,t+1}(A),$
(2.12)

where in the last equality we used (2.5).

######  Lemma 2.4.

Let $t\in\\{1,\ldots,T-1\\}$, and let $F$ be a function on
$\mathcal{X}^{T-t}\times{\boldsymbol{\Theta}}$. Then, for each
$h_{t}\in\mathcal{H}_{t}$ we have

$\displaystyle E^{\pi}[$ $\displaystyle
F(\widehat{X}_{t+1},\widehat{X}_{t+2},\ldots,\widehat{X}_{T-1},\widehat{X}_{T},\Theta)\;|\;\widehat{H}_{t}=h_{t}]=$
$\displaystyle\int_{\boldsymbol{\Theta}}\int_{\mathcal{X}^{T-t}}F(x_{t+1},\ldots,x_{T},\theta)P^{\pi^{T-1,h_{T-1}}}_{\theta,T}(dx_{T})\cdots
P^{\pi^{t,h_{t}}}_{\theta,t+1}(dx_{t+1})\xi^{\pi,h_{t}}_{t}(d\theta),$ (2.13)

where $E^{\pi}$ denotes the expectation with respect to probability $P^{\pi}$.

###### Proof.

In view of Lemma 2.2, we have

$P^{\pi}(dx_{t+1},\ldots,dx_{T};d\theta\;|\;\widehat{H}_{t}=h_{t})=P^{\pi^{t,h_{t}}}_{t+1,T}(dx_{t+1},\ldots,dx_{T};d\theta).$

Consequently, by (2.7), we continue

$P^{\pi^{t,h_{t}}}_{t+1,T}(dx_{t+1},\ldots,dx_{T};d\theta)=P^{\pi^{t,h_{t}}}_{\theta,t+1,T}(dx_{t+1},\ldots,dx_{T})\xi_{t}^{\pi^{t,h_{t}}}(d\theta).$

This combined with (2.3) and (2.6) yields the identity (2.13). ∎

For future reference we denote by $P^{\pi^{t,h_{t}}}_{t+1}$ the measure on
$\mathcal{X}\times\boldsymbol{\Theta}$ defined as

$P^{\pi^{t,h_{t}}}_{t+1}(B\times
D)=P^{\pi^{t,h_{t}}}_{t+1,T}(B\times\mathcal{X}^{T-t-1}\times
D)=\,P^{\pi}(\widehat{X}_{t+1}\in B,\Theta\in D\,|\,\widehat{H}_{t}=h_{t}),$
(2.14)

for $t=1,\ldots,T-1$, with the convention, employed throughout, that
$B\times\mathcal{X}^{0}\times D=B\times D.$

For $t=T$ and $h_{T}\in\mathcal{H}_{T}$ we construct a measure
$P^{\pi^{T,h_{T}}}_{T+1,T}$ on $\boldsymbol{\Theta}$ as

$P^{\pi^{T,h_{T}}}_{T+1,T}(D)=P^{\pi}(\Theta\in
D\,|\,\widehat{H}_{T}=h_{T})=\xi^{\pi,h_{T}}_{T}(D).$

Given that a strategy $\pi$ is used, then at each time $t\in\mathcal{T}$ a
random cost $Z^{\pi}_{\theta^{*},t}$ is incurred, with

$Z^{\pi}_{\theta,t}=c_{t}(X_{t},\pi_{t},\theta),$

where
$c_{t}:\mathcal{X}\times\mathcal{U}\times\boldsymbol{\Theta}\to\mathbb{R}_{+}$.

######  Remark 2.5.

It is important to note that even though $X_{t}$ and $\pi_{t}$ are observed at
time $t$, the actual cost $c_{t}(X_{t},\pi_{t},\theta^{*})$ is not known (or
observed) at time $t$ as $\theta^{*}$ is not known. The dependence of both the
transition kernel and the accrued costs on the unknown parameter is an
important practical situation, leading to non-standard and non-trivial risk-
averse Markov decision problems.

To proceed, for each $t\in\mathcal{T}$ and each history
$h_{t}\in\mathcal{H}_{t}$ we denote

$Z^{\pi,h_{t}}_{\theta,t,t}=c_{t}(x_{t},\pi_{t}(h_{t}),\theta),$ (2.15)

and for each $s=t+1,\dots,T$ we put

$Z^{\pi,h_{t},x_{t+1},\ldots,x_{s}}_{\theta,t,s}=c_{s}(x_{s},\pi^{t,h_{t}}_{s}(x_{t+1},\ldots,x_{s}),\theta).$
(2.16)

Note that, for a fixed strategy $\pi$ and a fixed $h_{t}\in\mathcal{H}_{t}$,
we have that $c_{t}(x_{t},\pi_{t}(h_{t}),\cdot)$ is a function on
$\boldsymbol{\Theta}$, and
$c_{s}(\cdot,\pi^{t,h_{t}}_{s}(\cdot,\ldots,\cdot),\theta)$ is a function on
$\mathcal{X}^{s-t}\times\boldsymbol{\Theta}$.

##  3 Risk Filters for MDPs with Model Uncertainty

###  3.1 Dynamic risk filters

For $t=1,\ldots,T-1$ and $s=t,\ldots,T$, we denote by
$\mathcal{Z}_{t}^{\mathcal{X}}$ and $\mathcal{Z}_{t,s}$ the spaces of real
valued functions on $\mathcal{X}^{t}$ and
$\mathcal{X}^{s-t}\times\boldsymbol{\Theta}$, respectively, where
$\mathcal{X}^{0}\times\boldsymbol{\Theta}:=\boldsymbol{\Theta}$, so that
$\mathcal{Z}_{t,t}$ is the space of real valued functions on
$\boldsymbol{\Theta}$. For $Z_{t,s},W_{t,s}\in\mathcal{Z}_{t,s}$, the
comparison between these functions is understood point-wise; $Z_{t,s}\leq
W_{t,s}$ means that $Z_{t,s}(x_{t+1},\ldots,x_{s},\theta)\leq
W_{t,s}(x_{t+1},\ldots,x_{s},\theta)$ for all
$(x_{t+1},\ldots,x_{s},\theta)\in\mathcal{X}^{s-t}\times\boldsymbol{\Theta}.$

For any policy $\pi\in\varPi$, our objective is to evaluate at each time
$t\in\mathcal{T}$, the riskiness of the sequence of costs
$Z^{\pi,h_{t}}_{\Theta,t},Z^{\pi,h_{t},X_{t+1}}_{\Theta,t,t+1},\dots,Z^{\pi,h_{t},X_{t+1},\ldots,X_{T}}_{\Theta,t,T}$,
given history $h_{t}$, in such a way that the evaluation is
$\mathcal{F}_{t}^{X}$-measurable. We denote by

${\mathcal{Z}}^{t,T}={\mathcal{Z}}_{t,t}\times{\mathcal{Z}}_{t,t+1}\times\dots\times{\mathcal{Z}}_{t,T}$

the space of conditional cost functions333The term conditional refers to the
fact that at any time $t$ we consider cost functions that depend on a history
$h_{t}$. in periods $t,\dots,T$.

For $t=1,\ldots,T$ and $s=t,\ldots,T$ we also use $\mathcal{P}_{t,s}$ to
denote the space of probability measures on the space of paths starting at
time $t$ and ending at time $s$, and on realizations of the parameter
$\theta$, that is on the space $\mathcal{X}^{s-t+1}\times\boldsymbol{\Theta}$.
Additionally, we understand $\mathcal{P}_{T+1,T}$ as
$\mathcal{P}(\boldsymbol{\Theta})$, because no future paths are possible.
Note, in particular, that $P^{\pi^{t,h_{t}}}_{t+1,T}\in\mathcal{P}_{t+1,T}$,
for $t\in\mathcal{T}$, and
$P^{\pi^{t,h_{t}}}_{t+1,t+1}\in\mathcal{P}_{t+1,t+1}$.

Observe that at time $t=1,\ldots,T-1$ we know the history $h_{t}$, and, for
any policy $\pi\in\varPi$ (in principle), we can evaluate the distribution of
$(X_{t+1},\dots,X_{T},\Theta)$ under the measure $P^{\pi^{t,h_{t}}}_{t+1,T}$.

We proceed with stating three key definitions.

######  Definition 3.1.

For a fixed $t\in\mathcal{T}$, a mapping
$\rho_{t}:{\mathcal{Z}}^{t,T}\times\mathcal{P}_{t+1,T}\to\mathbb{R}$, is
called a _conditional risk filter_.

Note that, in particular, for any
$(Z_{t,t},\dots,Z_{t,T})\in{\mathcal{Z}}^{t,T}$ and $\pi\in\Pi$ we have

$\rho_{t}(Z_{t,t},\dots,Z_{t,T};P_{t+1,T})=R(h_{t}),\ \textrm{for all}\
h_{t}\in\mathcal{X}^{t},$

for some function $R:\mathcal{X}^{t}\to\mathbb{R}$.

######  Definition 3.2.

Let $t\in\mathcal{T}$. A conditional risk filter $\rho_{t}$

  * (i)

is _normalized_ if $\rho_{t}(0,0,\dots,0;P_{t+1,T})=0$ for all
$P_{t+1,T}\in\mathcal{P}_{t+1,T}$;

  * (ii)

is _monotonic_ if
$\rho_{t}(Z_{t,t},\dots,Z_{t,T};P_{t+1,T})\leq\rho_{t}(W_{t,t},\dots,W_{t,T};P_{t+1,T})$
for all $P_{t+1,T}\in\mathcal{P}_{t+1,T}$, and all $(Z_{t,t},\dots,Z_{t,T})$
and $(W_{t,t},\dots,W_{t,T})$ in ${\mathcal{Z}}^{t,T}$, such that $Z_{t,s}\leq
W_{t,s}$ for all $s\in\mathcal{T}_{t}$;

  * (iii)

is _translation invariant_ if for all
$(Z_{t,t},\dots,Z_{t,T})\in{\mathcal{Z}}^{t,T}$, all $V\in\mathbb{R}$, and all
$P_{t+1,T}\in\mathcal{P}_{t+1,T}$,

$\rho_{t}(V+Z_{t,t},Z_{t,t+1},\dots,Z_{t,T};P_{t+1,T})=V+\rho_{t}(Z_{t,t},Z_{t,t+1},\dots,Z_{t,T};P_{t+1,T});$

  * (iv)

has the _support property_ , if

$\rho_{t}\big{(}Z_{t,t},\dots,Z_{t,T};P_{t+1,T}\big{)}=\rho_{t}\big{(}Z_{t,t}\mathbbm{1}_{\text{\rm
supp}_{t}(P_{t+1,T})},\dots,Z_{t,T}\mathbbm{1}_{\text{\rm
supp}_{T}(P_{t+1,T})};P_{t+1,T}\big{)},$

for all $(Z_{t,t},\dots,Z_{t,T})\in{\mathcal{Z}}^{t,T}$, and all
$P_{t+1,T}\in\mathcal{P}_{t+1,T}$, and where
$\operatorname{\mathrm{supp}}_{s}(P_{t+1,T})$ denotes the projection of
$\operatorname{\mathrm{supp}}(P_{t+1,T})$ on
$\mathcal{X}^{s-t}\times\boldsymbol{\Theta}$, for $s\geq t$.

######  Remark 3.3.

Let $s=t,\ldots,T$ and let $\\{Z^{y}_{s,T},y\in\mathcal{Y}\\}$ be a family of
functions parameterized by $y$, for some non-empty set $\mathcal{Y}$. Then, by
the normalization property, for any $A\subset\mathcal{Y}$, $y\in\mathcal{Y}$,
and $P\in\mathcal{P}_{t+1,T}$, we have that

$\mathbbm{1}_{A}(y)\rho_{t}(Z^{y}_{t,T},\ldots,Z^{y}_{T,T};P)=\rho_{t}(\mathbbm{1}_{A}(y)Z^{y}_{t,T},\ldots,\mathbbm{1}_{A}(y)Z^{y}_{T,T};P).$

######  Definition 3.4.

A _dynamic risk filter_ $\rho=\big{\\{}\rho_{t}\big{\\}}_{t\in\mathcal{T}}$ is
a sequence of conditional risk filters
$\rho_{t}:{\mathcal{Z}}^{t,T}\times\mathcal{P}_{t+1,T}\to\mathbb{R}$. We say
that it is normalized, monotonic, translation invariant, or has the support
property, if all $\rho_{t}$, $t\in\mathcal{T}$, satisfy the respective
conditions of Definition 3.2.

###  3.2 Parameter Consistency

Let $t=1,\ldots,T-1$ and $s=t,\ldots,T$. For any probability measure
$P_{t,s}\in\mathcal{P}_{t,s}$, we denote by $P_{t,s|\Theta}(\cdot,\cdot)$, the
stochastic kernel from $\boldsymbol{\Theta}$ to $\mathcal{X}^{s-t+1}$ defined
as

$P_{t,s|\Theta}(\theta,A)=\frac{P_{t,s}(A\times\\{\theta\\})}{P_{t,s}(\mathcal{X}^{s-t+1}\times\\{\theta\\})},\quad
A\subset\mathcal{X}^{s-t+1},\ \theta\in\boldsymbol{\Theta}.$ (3.1)

The corresponding marginal on $\boldsymbol{\Theta}$ is denoted by
$P_{t,s,\Theta}$, so that

$P_{t,s,\Theta}(D)=P_{t,s}(\mathcal{X}^{s-t+1}\times D),\quad
D\subset\boldsymbol{\Theta}.$ (3.2)

Clearly, the measure $P_{t,s}$ admits disintegration

$P_{t,s}(A\times
B)=\int_{B}\,P_{t,s|\theta}(A)P_{t,s,\Theta}(d\theta)=:P_{t,s,\Theta}\circledast
P_{t,s|\Theta}(A\times B),$

where we use a simplified notation

$P_{t,s|\theta}(A):=P_{t,s|\Theta}(\theta,A).$ (3.3)

We note that for any stochastic kernel $\kappa_{s,t}(\cdot,\cdot)$ from
$\boldsymbol{\Theta}$ to $\mathcal{X}^{s-t}$ and for any probability measure
$\mu$ on $2^{\boldsymbol{\Theta}}$ one can construct a unique probability
measure on the product space $\mathcal{X}^{s-t+1}\times\boldsymbol{\Theta}$ as

$m_{t,s}(A\times
B)=\int_{B}\,\kappa_{t,s}(\theta,A)\;\mu(d\theta)=:\mu\circledast\kappa_{t,s}(A\times
B).$

In particular, with $\mu=\delta_{\theta}$ and $\kappa_{t,s}=P_{t,s|{\Theta}}$,
with $P_{t,s}\in\mathcal{P}_{t,s}$, we get

$m_{t,s}(A\times B)=\delta_{\theta}\circledast P_{t,s|{\Theta}}(A\times
B)=P_{t,s|\theta}(A)\mathbbm{1}_{B}(\theta)=P_{t,s|\theta}(A)\delta_{\theta}(B).$
(3.4)

######  Remark 3.5.

In our convention, $P_{T+1,T}(\cdot)$ is a measure on ${\boldsymbol{\Theta}}$.
This means that, formally, $P_{T+1,T,\boldsymbol{\Theta}}=P_{T+1,T}$ and
$P_{T+1,T|\boldsymbol{\Theta}}\equiv 1$, in which case (formally)

$\delta_{\theta}\circledast P_{T+1,T|\Theta}=\delta_{\theta}.$

######  Example 3.6.

Fix $t\in\mathcal{T}$, $h_{t}\in\mathcal{H}_{t}$ and $\pi\in\Pi$. Take
$P_{t+1,T}=P^{\pi^{t,h_{t}}}_{t+1,T}\in\mathcal{P}_{t+1,T}$ and
$P_{t+1,t+1}=P^{\pi^{t,h_{t}}}_{t+1,t+1}\in\mathcal{P}_{t+1,t+1}$. Then,

$P_{t+1,T|\theta}=P^{\pi^{t,h_{t}}}_{\theta,t+1,T},\quad
P_{t+1,t+1|\theta}=P_{\theta,t+1}^{\pi^{t,h_{t}}},\quad
P_{t+1,T,\Theta}=\xi_{t}^{\pi^{t,h_{t}}}.$ (3.5)

The first equality above comes from (2.9). The second one comes from (2.12).
The third one is just (2.7) with $A=\mathcal{X}^{T-t}$. Note that (3.4) and
(3.5) imply that

$\delta_{\theta}\circledast P^{\pi^{t,h_{t}}}_{t+1,T|{\Theta}}(A\times
B)=\delta_{\theta}\otimes P^{\pi^{t,h_{t}}}_{\theta,t+1,T}(A\times
B),\quad\delta_{\theta}\circledast
P^{\pi^{t,h_{t}}}_{t+1,t+1|{\Theta}}(A\times B)=\delta_{\theta}\otimes
P^{\pi^{t,h_{t}}}_{\theta,t+1,t+1}(A\times B).$ (3.6)

We introduce the following key concept.

######  Definition 3.7.

A conditional risk filter
$\rho_{t}:{\mathcal{Z}}^{t,T}\times\mathcal{P}_{t+1,T}\to\mathbb{R}$ is
_parameter consistent_ , if for all $(Z_{t,t},\break\dots,Z_{t,T})$,
$(W_{t,t},\dots,W_{t,T})\in\mathcal{Z}^{t,T}$, and all
$P_{t+1,T},Q_{t+1,T}\in\mathcal{P}_{t+1,T}$ the relations

$P_{t+1,T,\Theta}=Q_{t+1,T,\Theta}\\\ $

and

$\rho_{t}\big{(}Z_{t,t},\dots,Z_{t,T};\delta_{\theta}\circledast
P_{t+1,T|\Theta}\big{)}\leq\rho_{t}\big{(}W_{t,t},\dots,W_{t,T};\delta_{\theta}\circledast
Q_{t+1,T|\Theta}\big{)},\quad\textrm{for all}\ \theta\in\boldsymbol{\Theta},$
(3.7)

imply that

$\rho_{t}\big{(}Z_{t,t},\dots,Z_{t,T};P_{t+1,T}\big{)}\leq\rho_{t}\big{(}W_{t,t},\dots,W_{t,T};Q_{t+1,T}\big{)}.$
(3.8)

In words, if the marginal distributions of $P$ and $Q$ on $\Theta$ are the
same, and the conditional risk of $Z_{t:T}:=(Z_{t,t},\dots,Z_{t,T})$ under $P$
is not greater than that of $W_{t:T}:=(W_{t,t},\dots,W_{t,T})$ under $Q$ for
every value of $\theta$, then the risk of $Z_{t:T}$ under $P$ should be not
greater than that of $W_{t:T}$ under $Q$.

######  Remark 3.8.

Note that parameter consistency at $t=T$ follows from the support property,
translation invariance, monotonicity, and normalization of $\rho_{T}$. Indeed,
first observe that according to Remark 3.5 the equality
$P_{T+1,T,\Theta}=Q_{T+1,T,\Theta}=1$ implies that $P_{T+1,T}=Q_{T+1,T}$.
Thus, for any $\theta\in\boldsymbol{\Theta}$

$\displaystyle\rho_{T}(Z_{T,T},\delta_{\theta})\leq\rho_{T}(W_{T,T},\delta_{\theta})\quad$
$\displaystyle\Leftrightarrow\quad\rho_{T}(Z_{T,T}(\theta),\delta_{\theta})\leq\rho_{T}(W_{T,T}(\theta),\delta_{\theta})\quad\Leftrightarrow$
$\displaystyle Z_{T,T}(\theta)+\rho_{T}(0,\delta_{\theta})\leq
W_{T,T}(\theta)+\rho_{T}(0,\delta_{\theta})\quad$
$\displaystyle\Leftrightarrow\quad Z_{T,T}(\theta)\leq W_{T,T}(\theta).$

By monotonicity, we have that

$\rho_{T}(Z_{T,T},P_{T+1,T})\leq\rho_{T}(W_{T,T},P_{T+1,T})=\rho_{T}(W_{T,T},Q_{T+1,T}).$

This remark is used in Proposition 3.10 and also in Theorem 3.14.

We have the following risk decomposition formula.

######  Theorem 3.9.

Take $t=1,\ldots,T$. If a conditional risk filter
$\rho_{t}:{{\mathcal{Z}}^{t,T}}\times\mathcal{P}_{t+1,T}\to{\mathbb{R}}$ is
parameter consistent, then there exists a mapping
$\widehat{\rho}_{t}:\mathcal{Z}_{t,t}\times\mathcal{P}(\boldsymbol{\Theta})\to\mathbb{R}$
such that for all $Z_{t:T}$ and $P_{t+1,T}$,

$\rho_{t}\big{(}Z_{t,t},\dots,Z_{t,T};P_{t+1,T}\big{)}=\widehat{\rho}_{t}\Big{(}\\{\rho_{t}\big{(}Z_{t,t},\dots,Z_{t,T};\delta_{\theta}\circledast
P_{t+1,T|\Theta}\big{)},\,\theta\in\boldsymbol{\Theta}\\};{P_{t+1,T,\Theta}}\Big{)}.$
(3.9)

###### Proof.

Suppose two sequences $Z_{t:T}$ and $W_{t:T}$ in $\mathcal{Z}^{t,T}$, and two
measures $P_{t+1,T}$ and $Q_{t+1,T}$ in $\mathcal{P}_{t+1,T}$ are such that
$P_{t+1,T,\boldsymbol{\Theta}}=Q_{t+1,T,\boldsymbol{\Theta}}$ and

$\rho_{t}\big{(}Z_{t,t},\dots,Z_{t,T};\delta_{\theta}\circledast
P_{t+1,T|\Theta}\big{)}=\rho_{t}\big{(}W_{t,t},\dots,W_{t,T};\delta_{\theta}\circledast
Q_{t+1,T|\Theta}\big{)},\quad\forall\,\theta\in\boldsymbol{\Theta}.$

Then it follows from Definition 3.7 that

$\rho_{t}\big{(}Z_{t,t},\dots,Z_{t,T};P_{t+1,T}\big{)}=\rho_{t}\big{(}W_{t,t},\dots,W_{t,T};Q_{t+1,T}\big{)}.$

This means that formula (3.9) is true. ∎

Thus, parameter consistency allows us to disintegrate the risk filtering task
into two stages. First, we evaluate the risk in a fully observed system, with
the parameter $\theta$ fixed, and then we integrate the results by using the
operator $\widehat{\rho}_{t}$, which we call the _marginal risk filter_.

######  Proposition 3.10.

Take $t=1,\ldots,T$. If the conditional risk filter $\rho_{t}$ is parameter
consistent, normalized, monotonic, and has the translation invariant property,
or the support property, then the mapping $\widehat{\rho}_{t}(\cdot;\cdot)$
has the corresponding properties as well (in the sense indicated in the proof
below).

###### Proof.

Indeed, consider any measure
$\varLambda\in\mathcal{P}({\boldsymbol{\Theta}})$. Then for any
$P_{t+1,T}\in\mathcal{P}_{t+1,T}$ such that $P_{t+1,T|\Theta}=\varLambda$, we
will use the formula (3.9) to analyze the implied properties of
$\widehat{\rho}_{t}$.

1) Suppose $\rho_{t}$ is normalized. Then (the symbol $\mathbf{0}$ below
denotes a function on $\boldsymbol{\Theta}$ that is identically equal to zero)

$\widehat{\rho}_{t}(\mathbf{0};\varLambda)=\widehat{\rho}_{t}\Big{(}\\{\rho_{t}\big{(}0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\boldsymbol{\Theta}}\big{)},\,\theta\in\boldsymbol{\Theta}\\};P_{t+1,T,\Theta}\Big{)}=\rho_{t}\big{(}0,\dots,0;P_{t+1,T}\big{)}=0.$

Thus, $\widehat{\rho}$ is normalized.

2) Suppose $\rho_{t}$ is normalized and translation invariant and has the
support property. Then for any $V\in\mathbb{R}$, we have

$\rho_{t}\big{(}V,0,\dots,0;P_{t+1,T}\big{)}=V+\rho_{t}\big{(}0,0,\dots,0;P_{t+1,T}\big{)}=V.$

Therefore, for any $U\in\mathcal{Z}^{t,t}$ and any $a\in\mathbb{R}$, by the
support, translation invariant, and normalization properties of $\rho_{t}$, we
have for any $\theta\in\boldsymbol{\Theta}$,

$\rho_{t}\big{(}U+a,0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}\big{)}=\rho_{t}\big{(}U(\theta)+a,0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}\big{)}=U(\theta)+a.$ (3.10)

Therefore,

$\displaystyle\widehat{\rho}_{t}(U+a;\varLambda)$
$\displaystyle=\widehat{\rho}_{t}\Big{(}\\{\rho_{t}\big{(}U+a,0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}\big{)},\,\theta\in\boldsymbol{\Theta}\\};P_{t+1,T,\Theta}\Big{)}$
$\displaystyle=\rho_{t}\big{(}U+a,0,\dots,0;P_{t+1,T}\big{)}=a+\rho_{t}\big{(}U,0,\dots,0;P_{t+1,T}\big{)}$
$\displaystyle=a+\widehat{\rho}_{t}\Big{(}\\{\rho_{t}\big{(}U,0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}\big{)},\,\theta\in\boldsymbol{\Theta}\\};P_{t+1,T,\Theta}\Big{)}=a+\widehat{\rho}_{t}(U;\varLambda).$

Hence, $\widehat{\rho}$ is translation invariant.

Similarly, for any $U\in\mathcal{Z}^{t,t}$, noting that
$\operatorname{\mathrm{supp}}_{t,t}(P_{t+1,T})=\operatorname{\mathrm{supp}}(\Lambda)$,
we deduce

$\displaystyle\widehat{\rho}_{t}(U,\varLambda)$
$\displaystyle=\rho_{t}\big{(}U,0,\dots,0;P_{t+1,T}\big{)}=\rho_{t}\big{(}\mathbbm{1}_{\text{\rm
supp}_{t,t}(P_{t+1,T})}U,0,\dots,0;P_{t+1,T}\big{)}$
$\displaystyle=\rho_{t}\big{(}\mathbbm{1}_{\text{\rm
supp}(\varLambda)}U,0,\dots,0;P_{t+1,T}\big{)}=\widehat{\rho}_{t}(\mathbbm{1}_{\text{\rm
supp}(\varLambda)}U,\varLambda).$

Thus, $\widehat{\rho}_{t}$ also has the support property.

3) Suppose $\rho_{t}$ is normalized, translation invariant, and monotonic.
Then, for all $U,W\in\mathcal{Z}^{t,t}$ such that $U\leq W$, employing (3.10)
with $a=0$, we have

$\displaystyle\widehat{\rho}_{t}(U;\varLambda)$
$\displaystyle=\widehat{\rho}_{t}\Big{(}\\{\rho_{t}\big{(}U,0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}\big{)},\,\theta\in\boldsymbol{\Theta}\\};P_{t+1,T,\Theta}\Big{)}=\rho_{t}\big{(}U,0,\dots,0;P_{t+1,T}\big{)}$
$\displaystyle\leq\rho_{t}\big{(}W,0,\dots,0;P_{t+1,T}\big{)}=\widehat{\rho}_{t}\Big{(}\\{\rho_{t}\big{(}W,0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}\big{)},\,\theta\in\boldsymbol{\Theta}\\};P_{t+1,T,\Theta}\Big{)}$
$\displaystyle=\widehat{\rho}_{t}(W;\varLambda),$

and this proves the monotonicity of $\widehat{\rho}_{t}$.

4) If $\rho$ is monotonic, normalized, parameter consistent, translation
invariant, and with support property, then $\widehat{\rho}_{t}$ is also
normalized, monotonic, normalized, translation invariant, and has the support
property, and in view of Remark 3.8, $\widehat{\rho}_{t}$ is also parameter
consistent.

∎

###  3.3 Time Consistency

We now consider the notion of time consistency of risk filters.

######  Definition 3.11.

Let $t\in\\{1,\ldots,T-1\\}$. For any positive measure $\mu_{t+1}$ on
$\mathcal{X}^{T-t}\times\boldsymbol{\Theta}$ and for any $x\in\mathcal{X}$, we
denote by $\mu_{t+1}(\,\cdot\,\|x)$ the measure on
$\mathcal{X}^{T-t-1}\times\boldsymbol{\Theta}$ given as

$\mu_{t+1}(A\times B\|x)=\frac{\mu_{t+1}(\\{x\\}\times A\times
B)}{\mu_{t+1}(\\{x\\}\times\mathcal{X}^{T-t-1}\times\boldsymbol{\Theta})}.$
(3.11)

Clearly, $\mu_{t+1}(\,\cdot\,\|x)$ is a probability measure.

In particular, taking $\mu_{t+1}=\delta_{\theta}\circledast P_{t+1,T|\Theta}$,

$\delta_{\theta}\circledast P_{t+1,T|\Theta}(A\times
B\,\|x)=\frac{\delta_{\theta}\circledast P_{t+1,T|\Theta}(\\{x\\}\times
A\times B)}{\delta_{\theta}\circledast
P_{t+1,T|\Theta}(\\{x\\}\times\mathcal{X}^{T-t-1}\times\boldsymbol{\Theta})}.$
(3.12)

It follows from (3.4) and (3.12) that

$\displaystyle\delta_{\theta}\circledast P_{t+1,T|\Theta}(\,A\times B\,\|\,x)$
$\displaystyle=\frac{P_{t+1,T|\theta}(\\{x\\}\times
A)\delta_{\theta}(B)}{P_{t+1,T|\theta}(\\{x\\}\times\mathcal{X}^{T-t-1})}=\frac{P_{t+1,T|\theta}(\\{x\\}\times
A)}{P_{t+1,t+1|\theta}(\\{x\\})}\delta_{\theta}(B)$ (3.13)
$\displaystyle=:\widetilde{P}_{t+1,T|\theta}(\,A\,\|\,x)\delta_{\theta}(B),$

which, in view of (3.6) gives

$\delta_{\theta}\circledast P^{\pi^{t,h_{t}}}_{t+1,T|\Theta}(A\times
B\|x)=\frac{P^{\pi^{t,h_{t}}}_{\theta,t+1,T}(\\{x\\}\times
A)}{P^{\pi^{t,h_{t}}}_{\theta,t+1,t+1}(\\{x\\})}\delta_{\theta}(B)=:\widetilde{P}^{\pi^{t,h_{t}}}_{\theta,t+1,T}(\,A\,\|x)\delta_{\theta}(B).$
(3.14)

The next definition is a version of the dynamic conditional time consistency
used in [FR18], adapted to the set-up of the present paper.

######  Definition 3.12.

A dynamic risk filter $\rho=\big{\\{}\rho_{t}\big{\\}}_{t=1,\dots,T}$ is _time
consistent_ if for any $t=1,\ldots,T-1$, for any
$P_{t+1,T},Q_{t+1,T}\in\mathcal{P}_{t+1,T}$, such that
$P_{t+1,t+1|\Theta}=Q_{t+1,t+1|\Theta}$, and for any functions444 The notation
$\cdot_{k}$ is a place-holder for $k$ variables.
$Z_{t,s}(\cdot_{s-t},\cdot_{1}),W_{t,s}(\cdot_{s-t},\cdot_{1})\in{\mathcal{Z}_{t,s}}$,
$s=t+1,\ldots,T$, the inequalities

$\displaystyle\rho_{t+1}$
$\displaystyle\big{(}Z_{t,t+1}(x_{t+1},{\cdot_{1}}),\dots,Z_{t,T}(x_{t+1},\cdot_{T-t-1},{\cdot_{1}});\delta_{\theta}\circledast
P_{t+1,T|\Theta}(\,\cdot\,\|x_{t+1})\big{)}$
$\displaystyle\leq\rho_{t+1}\big{(}W_{t,t+1}(x_{t+1},{\cdot_{1}}),\dots,W_{t,T}(x_{t+1},\cdot_{T-t-1},{\cdot_{1}});\delta_{\theta}\circledast
Q_{t+1,T|\Theta}(\,\cdot\,\|x_{t+1})\big{)},$ (3.15)
$\displaystyle\qquad\qquad\qquad\forall\,\theta\in\boldsymbol{\Theta},\quad\forall\,x_{t+1}\in\mathcal{X},$

imply that for any function $f_{t}\in\mathcal{Z}_{t,t}$

$\displaystyle\rho_{t}$
$\displaystyle\big{(}f_{t}({\cdot_{1}}),Z_{t,t+1}(\cdot_{1},{\cdot_{1}}),\dots,Z_{t,T}(\cdot_{T-t},{\cdot_{1}}));\delta_{\theta}\circledast
P_{t+1,T|\Theta})\big{)}$
$\displaystyle\leq\rho_{t}\big{(}f_{t}({\cdot_{1}}),W_{t,t+1}(\cdot_{1},{\cdot_{1}}),\dots,W_{t,T}(\cdot_{T-t},{\cdot_{1}}));\delta_{\theta}\circledast
Q_{t+1,T|\Theta})\big{)},\quad\forall\,\theta\in\boldsymbol{\Theta}.$ (3.16)

######  Lemma 3.13.

Suppose a dynamic risk filter $\big{\\{}\rho_{t}\big{\\}}_{t=1,\dots,T}$ is
normalized, translation invariant, has the support property and is time
consistent. Let $\theta\in{\boldsymbol{\Theta}}$ be fixed. Then the function
on
$\mathcal{Z}_{t,t+1}\times\mathcal{P}(\mathcal{X}^{T-t}\times{\boldsymbol{\Theta}})$
given as

$\rho_{t}\big{(}0,w,0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}\big{)}$ (3.17)

depends only on the probability $P_{t+1,t+1|\theta}$ and on the function
$w(\cdot,\theta)$.

###### Proof.

For any $P_{t+1,T}$ and $x\in\mathcal{X}$, the support property of
$\rho_{t+1}$ implies that

$\rho_{t+1}\big{(}w(x,\cdot),0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}(\cdot\|x)\big{)}=\rho_{t+1}\big{(}w(x,\theta),0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}(\cdot\|x)\big{)}.$

Then, by the translation invariance and the normalization properties of
$\rho_{t+1}$ we obtain

$\rho_{t+1}\big{(}w(x,\cdot),0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}(\cdot\|x)\big{)}=w(x,\theta),$

which does not depend on $P_{t+2,T|\Theta}$. Hence, for any
$Q_{t+1,T}\in\mathcal{P}_{t+1,T}$ we have

$\rho_{t+1}\big{(}w(x,\cdot),0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}(\,\cdot\,\|x)\big{)}=\rho_{t+1}\big{(}w(x,\cdot),0,\dots,0;\delta_{\theta}\circledast
Q_{t+1,T|\Theta}(\,\cdot\,\|x)\big{)}.$

If in addition, $P_{t+1,t+1|\Theta}=Q_{t+1,t+1|\Theta}$, then, by the time
consistency,

$\rho_{t}\big{(}0,w,0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}\big{)}=\rho_{t}\big{(}0,w,0,\dots,0;\delta_{\theta}\circledast
Q_{t+1,T|\Theta}\big{)}.$

which proves that only the conditional measure $P_{t+1,t+1|\Theta}$ matters in
this calculation. The fact that the knowledge of $w(\cdot,\theta)$ is
sufficient, follows from the support property. This concludes the proof. ∎

In accordance with the above lemma we define the functions

$\sigma_{t}:Z^{\mathcal{X}}_{1}\times\mathcal{P}(\mathcal{X})\to{\mathbb{R}},\quad
t=1,\ldots,T-1,$

as

$\sigma_{t}(v;P_{t+1,t+1|\theta})=\rho_{t}\big{(}0,w,0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}\big{)},$ (3.18)

where $w(\cdot,\theta)\equiv v(\cdot)$ and can be arbitrary otherwise. We
refer to these functions as _transition risk mappings_.

Note that if $\rho_{t}$ is normalized, monotonic, translation invariant and
has support property, then so is $\sigma_{t}$.

######  Theorem 3.14.

A dynamic risk filter $\rho=\big{\\{}\rho_{t}\big{\\}}_{t=1,\dots,T}$ is
normalized, monotonic, translation invariant, has the support property, is
parameter consistent, and time consistent, if and only if the following
conditions are satisfied:
1) Marginal risk mappings
$\widehat{\rho}_{t}:\mathcal{Z}_{t,t}\times\mathcal{P}(\boldsymbol{\Theta})\to\mathbb{R}$,
$t\in\mathcal{T}$, exist, which are normalized, monotonic, translation
invariant, and have the support property;
2) Transition risk mappings given in (3.18) are such that

  1. (i)

For all $t=1,\ldots,T-1$, $\sigma_{t}(\cdot;\cdot)$ is normalized, monotonic,
translation invariant, and has the support property;

  2. (ii)

For any $P_{t+1,T}\in\mathcal{P}_{t+1,T}$, $t=1,\dots,T-1$, and for any
functions $Z_{t,s}\in\mathcal{Z}_{t,s}$, $s\in\mathcal{T}_{t}$, we have
that555Recall Definition 3.11. The notation
$\sigma_{t}\big{(}\rho_{t+1}\big{(}Z_{t,t+1}(\diamond,{\theta}),\dots,Z_{t,T}(\diamond,\cdot_{T-t-1},{\theta});\delta_{{\theta}}\circledast
P_{t+1,T|\Theta}(\,\cdot\,\|\diamond)\big{)};P_{t+1,t+1|\theta}\big{)}$, where
we use $\diamond$ as place holder, means that $P_{t+1,t+1|\theta}$ acts on
$w(x,\theta)=\rho_{t+1}\big{(}Z_{t,t+1}(x,{\theta}),\dots,f_{t,T}(x,\cdot_{T-t-1},{\theta});\break\delta_{{\theta}}\circledast
P_{t+1,T|\Theta}(\,\cdot\,\|x)\big{)}$ as a function of $x$.

$\rho_{t}(Z_{t,t},Z_{t,t+1},\dots,Z_{t,T};P_{t+1,T})=\widehat{\rho}_{t}\Big{(}\Big{\\{}Z_{t,t}(\theta)+\sigma_{t}\big{(}\rho_{t+1}\big{(}Z_{t,t+1}(\diamond,\cdot_{1}),\dots,Z_{t,T}(\diamond,\cdot_{T-t-1},\cdot_{1});\\\
\delta_{{\theta}}\circledast
P_{t+1,T|\Theta}(\,\cdot\,\|\diamond)\big{)};P_{t+1,t+1|\theta}\big{)},\theta\in{\boldsymbol{\Theta}}\Big{\\}};P_{t+1,T,\Theta}\Big{)}.$
(3.19)

For any function $Z_{T,T}\in\mathcal{Z}_{T,T}$ and
$P_{T+1,T}\in\mathcal{P}(\boldsymbol{\Theta})$

$\rho_{T}(Z_{T,T};P_{T+1,T})=\widehat{\rho}_{T}\Big{(}Z_{T,T},P_{T+1,T}\Big{)}.$
(3.20)

###### Proof.

We fix $P_{t+1,T}\in\mathcal{P}_{t+1,T}$ and $Z_{t,s}\in\mathcal{Z}_{t,s}$,
$s=t,\ldots,T$. Since $t$ is fixed, to alleviate the notations we simply write
$Z_{s}$, instead of $Z_{t,s}$ in this proof.

Since the risk filter is parameter consistent, Theorem 3.9 yields the
existence of mappings $\widehat{\rho}_{t}$ such that

$\rho_{t}(Z_{t},Z_{t+1},\dots,Z_{T};P_{t+1,T})=\widehat{\rho}_{t}\Big{(}\Big{\\{}\rho_{t}\big{(}Z_{t},Z_{t+1},\dots,Z_{T};\delta_{\theta}\circledast
P_{t+1,T|\Theta}\big{)},\theta\in{\boldsymbol{\Theta}}\Big{\\}};P_{t+1,T,\Theta}\Big{)}.$

It follows from Proposition 3.10 that $\widehat{\rho}$ is normalized,
monotonic, translation invariant, has the support property.

Next, we derive an equivalent expression for the first argument of
$\widehat{\rho}_{t}$ that will prove (3.19). Define the function

$w(x,\theta)=\rho_{t+1}\big{(}Z_{t+1}(x,{\cdot_{1}}),\dots,Z_{T}(x,\cdot_{T-t-1},{\cdot_{1}});\delta_{\theta}\circledast
P_{t+1,T|\boldsymbol{\Theta}}(\,\cdot\,\|x)\big{)},\quad
x\in\mathcal{X},\quad\theta\in\boldsymbol{\Theta}.$

Then, for any fixed $x\in\mathcal{X}$ and $\theta\in\boldsymbol{\Theta}$, we
use the support, translation invariance, and the normalization properties in
the chain of equations below:

$\displaystyle\rho_{t+1}$
$\displaystyle\big{(}w(x,\cdot),0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}(\,\cdot\,\|x)\big{)}$
$\displaystyle=\rho_{t+1}\big{(}w(x,\theta),0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}(\,\cdot\,\|x)\big{)}$
$\displaystyle=w(x,\theta)+\rho_{t+1}\big{(}0,0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}(\,\cdot\,\|x)\big{)}=w(x,\theta)$
$\displaystyle=\rho_{t+1}\big{(}Z_{t+1}(x,{\cdot_{1}}),\dots,Z_{T}(x,\cdot_{T-t-1},{\cdot_{1}});\delta_{\theta}\circledast
P_{t+1,T|\Theta}(\,\cdot\,\|x)\big{)}.$

In view of the assumed time consistency of $\rho$, the above implies that for
every $Z_{t,t}\in\mathcal{Z}_{t,t}$,

$\rho_{t}\big{(}Z_{t},Z_{t+1},\dots,Z_{T};\delta_{\theta}\circledast
P_{t+1,T|\Theta}\big{)}=\rho_{t}\big{(}Z_{t},w,0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}\big{)}=:I_{1}.$

Thus, by using the translation invariance and the support properties again, we
conclude that, for all $\theta\in{\boldsymbol{\Theta}}$,

$I_{1}=\rho_{t}\big{(}Z_{t}(\theta),w,0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}\big{)}=Z_{t}(\theta)+\rho_{t}\big{(}0,w,0,\dots,0;\delta_{\theta}\circledast
P_{t+1,T|\Theta}\big{)}.$

Using (3.18), we get

$I_{1}=Z_{t}(\theta)+\sigma_{t}\big{(}w(\cdot,\theta),P_{t+1,t+1|\theta}\big{)}.$

Finally, from here, noting that by support property

$w(x,\theta)=\rho_{t+1}\big{(}Z_{t+1}(x,\theta),\dots,Z_{T}(x,\cdot_{T-t-1},\theta);\delta_{\theta}\circledast
P_{t+1,T|\Theta}(\,\cdot\,\|x)\big{)},\quad
x\in\mathcal{X},\quad\theta\in\boldsymbol{\Theta},$ (3.21)

we obtain the representation (3.19). The representation (3.20) follows from
the definition of $\rho_{T}$ and the form of $\widehat{\rho}_{T}$.

Next we prove the converse statement by backward induction in time. For $t=T$,
the conditional risk filter (3.20) has all the postulated properties, with the
exception of the time consistency, because $\widehat{\rho}_{T}$ does (see
Proposition 3.10).

Suppose the conditional risk filters $\rho_{s}$, $s=t+1,\dots,T$ are
normalized, monotonic, translation invariant, have the support property, are
parameter consistent, and time consistent. We will verify these properties for
$\rho_{t}$ given by formula (3.19). The translation invariance follows from
the translation invariance of $\widehat{\rho}_{t}$. The normalization and the
monotonicity follow immediately from the normalization and the monotonicity of
$\sigma_{t}$, $\rho_{t+1}$ and $\widehat{\rho}_{t}$.

We now verify the support property. For every $\theta$, and $x$ define
$\mu_{\theta,x}(\cdot)=\delta_{\theta}\circledast P_{t+1,T|\theta}(\cdot\|x)$,
$A(\theta)=\operatorname{\mathrm{supp}}(P_{t+1,t+1|\theta})\subset\mathcal{X}$,
$B=\operatorname{\mathrm{supp}}(P_{t+1,T,\Theta})\subset\boldsymbol{\Theta}$.
Then, by (3.19), (3.21), the support property of $\hat{\rho}_{t}$ and
$\sigma_{t}$, and by Remark 3.3 applied to $\sigma_{t}$, we deduce that

$\displaystyle\rho_{t}(Z_{t},$ $\displaystyle
Z_{t+1},\ldots,Z_{T};P_{t+1,T})=\hat{\rho}_{t}\Big{(}\Big{\\{}Z_{t}(\theta)+\sigma_{t}(w(\diamond,\theta);P_{t+1,t+1,|\theta});\theta\in\boldsymbol{\Theta}\Big{\\}};P_{t+1,T,\Theta}\Big{)}$
$\displaystyle=\hat{\rho}_{t}\Big{(}\Big{\\{}\mathbbm{1}_{B}(\theta)Z_{t}(\theta)+\mathbbm{1}_{B}(\theta)\sigma_{t}(w(\diamond,\theta);P_{t+1,t+1,|\theta});\theta\in\boldsymbol{\Theta}\Big{\\}};P_{t+1,T,\Theta}\Big{)}$
$\displaystyle=\hat{\rho}_{t}\Big{(}\Big{\\{}\mathbbm{1}_{B}(\theta)Z_{t}(\theta)+\sigma_{t}(\mathbbm{1}_{B}(\theta)\mathbbm{1}_{A(\theta)}(\diamond)w(\diamond,\theta);P_{t+1,t+1,|\theta});\theta\in\boldsymbol{\Theta}\Big{\\}};P_{t+1,T,\Theta}\Big{)}.$
(3.22)

By the assumed support property of $\rho_{t+1}$, and in view of Remark 3.3
applied to $\rho_{t+1}$, we obtain, using (3.21) again,

$\mathbbm{1}_{A(\theta)}(x)\mathbbm{1}_{B}(\theta)w(x,\theta)=\rho_{t+1}\big{(}\mathbbm{1}_{A(\theta)}(x)\mathbbm{1}_{B}(\theta)Z_{t+1}(x,\theta),\mathbbm{1}_{A(\theta)}(x)\mathbbm{1}_{B}(\theta)\mathbbm{1}_{\operatorname{\mathrm{supp}}_{t+2}(\mu_{\theta,x})}Z_{t+2}(x,\cdot,\theta),\\\
\dots,\mathbbm{1}_{A(\theta)}(x)\mathbbm{1}_{B}(\theta)\mathbbm{1}_{\operatorname{\mathrm{supp}}_{T}(\mu_{\theta,x})}Z_{T}(x,\cdot_{T-t-1},\theta);\mu_{\theta,x}\big{)},$

for every $x\in\mathcal{X}$ and $\theta\in\boldsymbol{\Theta}$. From here and
(3.22), combined with the normalization property of $\rho_{t+1}$, and the fact
that
$\mathbbm{1}_{A(\theta)}(x)\mathbbm{1}_{B}(\theta)\mathbbm{1}_{\operatorname{\mathrm{supp}}_{s}(\mu_{\theta,x})}\leq
1_{\operatorname{\mathrm{supp}}_{s}(P_{t+1,T})},\ s=t,\ldots,T$, we obtain the
support property of $\rho_{t}$.

Next we prove the parameter consistency. Assume that (3.7) is satisfied for a
fixed $\bar{\theta}\in\boldsymbol{\Theta}$, and denote by
$\bar{P}_{t+1,T}=\delta_{\bar{\theta}}\circledast P_{t+1,T|\Theta}$ and
$\bar{Q}_{t+1,T}=\delta_{\bar{\theta}}\circledast Q_{t+1,T|\Theta}$. We note
that666We use the convention that $\frac{0}{0}=0$ when considering
$\bar{P}_{t+1,t+1|\theta}$ and $\bar{P}^{x}_{t+1,T|\theta}(\cdot\|x)$.

$\bar{P}_{t+1,t+1|\theta}=P_{t+1,t+1|\bar{\theta}}\cdot\mathbbm{1}_{\bar{\theta}}(\theta),\quad\bar{P}_{t+1,T,\Theta}=\delta_{\bar{\theta}},\quad\bar{P}_{t+1,T|\theta}(\cdot\|x)=P_{t+1,T|\bar{\theta}}(\cdot\|x)\mathbbm{1}_{\bar{\theta}}(\theta).$

Using this, and in view of (3.19), we can write (3.7) as follows (with
measures $\bar{P}_{t+1,T}$ and $\bar{Q}_{t+1,T}$ in place of $P_{t+1,T}$ and
$Q_{t+1,T}$):

$\widehat{\rho}_{t}\Big{(}\Big{\\{}Z_{t}(\theta)+\sigma_{t}\big{(}\rho_{t+1}\big{(}Z_{t+1}(\diamond,{\theta}),\dots,Z_{T}(\diamond,\cdot_{T-t-1},{\theta});\\\
\delta_{{\bar{\theta}}}(\cdot)P_{t+1,T|\bar{\theta}}(\,\cdot\,\|\diamond)\mathbbm{1}_{\bar{\theta}}(\theta);P_{t+1,t+1|\bar{\theta}}\mathbbm{1}_{\bar{\theta}}(\theta)\big{)},\theta\in{\boldsymbol{\Theta}}\Big{\\}};\delta_{{\bar{\theta}}}\Big{)}\\\
\leq\widehat{\rho}_{t}\Big{(}\Big{\\{}W_{t}(\theta)+\sigma_{t}\big{(}\rho_{t+1}\big{(}W_{t+1}(\diamond,{\theta}),\dots,W_{T}(\diamond,\cdot_{T-t-1},{\theta});\\\
\delta_{{\bar{\theta}}}(\cdot)Q_{t+1,T|\bar{\theta}}(\,\cdot\,\|\diamond)\mathbbm{1}_{\bar{\theta}}(\theta);Q_{t+1,t+1|\bar{\theta}}\mathbbm{1}_{\bar{\theta}}(\theta)\big{)},\theta\in{\boldsymbol{\Theta}}\Big{\\}};\delta_{{\bar{\theta}}}\Big{)}.$

By the support property of $\hat{\rho}_{t}$ and $\sigma_{t}$, and by the
normalization and monotonicity of $\hat{\rho}_{t}$, we obtain that

$\displaystyle Z_{t}(\bar{\theta})$
$\displaystyle+\sigma_{t}\big{(}\rho_{t+1}\big{(}Z_{t+1}(\diamond,{\bar{\theta}}),\dots,Z_{T}(\diamond,\cdot_{T-t-1},{\bar{\theta}});\delta_{{\bar{\theta}}}(\cdot)P_{t+1,T|\bar{\theta}}(\,\cdot\,\|\diamond);P_{t+1,t+1|\bar{\theta}}\big{)}$
$\displaystyle\leq$ $\displaystyle
W_{t}(\bar{\theta})+\sigma_{t}\big{(}\rho_{t+1}\big{(}W_{t+1}(\diamond,{\bar{\theta}}),\dots,W_{T}(\diamond,\cdot_{T-t-1},{\bar{\theta}});\delta_{{\bar{\theta}}}(\cdot)Q_{t+1,T|\bar{\theta}}(\,\cdot\,\|\diamond);Q_{t+1,t+1|\bar{\theta}}\big{)},$

for any $\bar{\theta}\in\boldsymbol{\Theta}$. From here, applying
$\hat{\rho}_{t}$ to both sides, since we assumed that
$P_{t+1,T,\boldsymbol{\Theta}}=Q_{t+1,T,\boldsymbol{\Theta}}$, employing
monotonicity of $\hat{\rho}_{t}$, we obtain (3.8), and the parameter
consistency of $\rho_{t}$ is proved.

Finally, let us verify the time consistency at time $t$. If the inequalities
(3.15) are satisfied, then it follows from the monotonicity of $\sigma_{t}$
with respect to its first argument that for all
$\theta\in{\boldsymbol{\Theta}}$

$G_{1}(\theta):=\sigma_{t}\big{(}\rho_{t+1}\big{(}Z_{t+1}(\diamond,\theta),\dots,Z_{T}(\diamond,\cdot_{T-t-1},\theta);\delta_{\theta}\circledast
P_{t+1,T|\Theta}(\cdot\|\diamond)\big{)};P_{t+1,t+1|\theta}\big{)}\\\
\leq\sigma_{t}\big{(}\rho_{t+1}\big{(}W_{t+1}(\diamond,\theta),\dots,W_{T}(\diamond,\cdot_{T-t-1},\theta);\delta_{\theta}\circledast
Q_{t+1,T|\Theta}(\cdot\|\diamond)\big{)};P_{t+1,t+1|\theta}\big{)}=:G_{2}(\theta).$

Then, from the monotonicity of $\widehat{\rho}_{t}$ we get, for any function
$f_{t}\in\mathcal{Z}_{t,t}$,

$\hat{\rho}_{t}(f_{t}+G_{1};\delta_{\theta})\leq\hat{\rho}_{t}(f_{t}+G_{2};\delta_{\theta}),\quad\theta\in\boldsymbol{\Theta}.$

From here, using the support property of $\widehat{\rho}_{t},\sigma_{t}$ and
$\rho_{t+1}$, along (3.19), we obtain (3.16), and thus time consistency at $t$
is verified.

By induction, all properties hold true for $t=1,\dots,T$, and the proof is
complete. ∎

######  Example 3.15.

We consider a very special conditional risk filter, given as the expectation
of an additive functional under the measure $P_{t+1,T}$. Specifically, we let

$\displaystyle\rho_{t}(Z_{t,t},\ldots,Z_{t,T};P_{t+1,T})$
$\displaystyle=\int_{\mathcal{X}^{T-t}\times\boldsymbol{\Theta}}\sum_{k=t}^{T}Z_{t,k}(x_{t+1},\ldots,x_{k},\theta)P_{t+1,T}(dx_{t+1},\cdots,dx_{T},d\theta)$
$\displaystyle=$ $\displaystyle E_{P_{t+1,T}}\sum_{k=t}^{T}Z_{t,k}.$

Clearly, this $\rho_{t}$ is normalized, monotonic, translation invariant, and
has the support property (cf. Definition 3.2).

Next, note that for this $\rho_{t}$ the inequality (3.7) becomes (cf. (3.4))

$\displaystyle\int_{\mathcal{X}^{T-t}}\sum_{k=t}^{T}Z_{t,k}(x_{t+1:k},\theta)P_{t+1,T|\theta}(dx_{t+1},\cdots,dx_{T})\leq\int_{\mathcal{X}^{T-t}}\sum_{k=t}^{T}Z_{t,k}(x_{t+1:k},\theta)Q_{t+1,T|\theta}(dx_{t+1},\cdots,dx_{T}),$

for any $\theta\in\boldsymbol{\Theta}$. Assuming that
$P_{t+1,T,\Theta}=Q_{t+1,T,\Theta}$, multiplying the last inequality by
$P_{t+1,T,\Theta}(\theta)$, and summing up with respect to
$\theta\in\boldsymbol{\Theta}$, the inequality (3.8) follows, and hence the
parameter consistency is true.

The time consistency follows by similar arguments. Indeed, (3.15) becomes (cf.
(3.13))

$\displaystyle\int_{\mathcal{X}^{T-t-1}}$
$\displaystyle\sum_{k=t+1}^{T}Z_{t,k}(x_{t+1},x_{t+2:k},\theta)\;\widetilde{P}_{t+1,T|\theta}(dx_{t+2},\cdots,dx_{T}\|x_{t+1})$
$\displaystyle\leq\int_{\mathcal{X}^{T-t-1}}\sum_{k=t+1}^{T}Z_{t,k}(x_{t+1},x_{t+2:k},\theta)\;\widetilde{Q}_{t+1,T|\theta}(dx_{t+2},\cdots,dx_{T}\|x_{t+1}),$

for any $x_{t+1}\in\mathcal{X}$ and $\theta\in\boldsymbol{\Theta}$. Assuming
that $P_{t+1,t+1|\Theta}=Q_{t+1,t+1|\Theta}$, multiplying both parts by
$P_{t+1,t+1|\theta}(x_{t+1})$, and noting that (cf. (3.13))

$\widetilde{P}_{t+1,T|\theta}(\cdot\|x_{t+1})P_{t+1,t+1|\theta}(x_{t+1})=P_{t+1,T|\theta}(x_{t+1},\cdot),$

for any function $f_{t}\in\mathcal{Z}_{t,t}$ we have (cf. (3.13))

$\displaystyle f_{t}(\theta)+\int_{\mathcal{X}^{T-t-1}}$
$\displaystyle\sum_{k=t+1}^{T}Z_{t,k}(x_{t+1},x_{t+2:k},\theta)\;P_{t+1,T|\theta,}(\\{x_{t+1}\\},dx_{t+2},\cdots,dx_{T})$
$\displaystyle\leq
f_{t}(\theta)+\int_{\mathcal{X}^{T-t-1}}\sum_{k=t+1}^{T}Z_{t,k}(x_{t+1},x_{t+2:k},\theta)\;Q_{t+1,T|\theta}(\\{x_{t+1}\\},dx_{t+2},\cdots,dx_{T}),$

After summing up with respect to $x_{t+1}$ we obtain (3.16), and thus the time
consistency is proved.

We complete this example by observing that in the this case we have that
$\widehat{\rho}_{t}(f,{P}^{\prime})=E_{{P}^{\prime}}(f)$, for $f\in Z_{t,t}$
and for ${P}^{\prime}\in\mathcal{P}(\boldsymbol{\Theta})$, and that and
$\sigma_{t}(v;{P}^{\prime\prime})=E_{{P}^{\prime\prime}}(v)$, for
$v\in\mathcal{Z}^{\mathcal{X}}_{1}$ and
${P}^{\prime\prime}\in\mathcal{P}(\mathcal{X})$.

######  Example 3.16.

Let us cast Example 3.15 in the setup of Section 2. For this, we fix a history
$h_{t}=(x_{1},\ldots,x_{t})\in\mathcal{H}_{t}$ and $\pi\in\Pi$, and we take

$Z_{t,t}(\theta):=Z^{\pi,h_{t}}_{\theta,t,t}=c_{t}(x_{t},\pi_{t}(h_{t}),\theta),$
$Z_{t,s}(x_{t+1},\ldots,x_{s},\theta):=Z^{\pi,h_{t},x_{t+1},\ldots,x_{s}}_{\theta,t,s}=c_{s}(x_{s},\pi^{t,h_{t}}_{s}(x_{t+1},\ldots,x_{s}),\theta),s=t+1,\ldots,T,$

and

$P_{t+1,T}=P_{t+1,T}^{\pi^{t,h_{t}}}.$

The conditional risk filter of Example 3.15 becomes a conditional expectation
(cf. Lemma 2.2)

$\displaystyle\rho_{t}\Big{(}c_{t}(x_{t},\pi_{t}(h_{t}),\cdot),c_{t+1}(\cdot,\pi_{t+1}(h_{t},\cdot),\cdot),\cdots,c_{T}(\cdot,\pi_{T}(h_{t},\cdot,\ldots,\cdot),\cdot),P^{\pi^{t,h_{t}}}_{t+1,T}\Big{)}$
$\displaystyle=E^{\pi}\left[c_{t}(x_{t},\pi_{t}(h_{t}),\Theta)+\sum_{s=t+1}^{T}c_{s}(\widehat{X}_{s},\pi_{s}^{t,h_{t}}(\widehat{X}_{t+1},\ldots,\widehat{X}_{s}),\Theta)\;|\;\widehat{H}_{t}=h_{t}\right],$
(3.23)

for $t=1,\ldots,T$, where we use the standard convention that an empty sum is
zero (i.e. $\sum_{s={T+1}}^{T}\cdots=0$ in our case).

In view of (2.7) we also have

$\displaystyle\rho_{t}\Big{(}c_{t}(x_{t},\pi_{t}(h_{t}),\cdot),c_{t+1}(\cdot,\pi_{t+1}(h_{t},\cdot),\cdot),\cdots,c_{T}(\cdot,\pi_{T}(h_{t},\cdot,\ldots,\cdot),\cdot),P^{\pi^{t,h_{t}}}_{t+1,T}\Big{)}$
$\displaystyle=\widehat{\rho}_{t}\Big{(}\Big{\\{}c_{t}(x_{t},\pi_{t}(h_{t}),\theta)+\sigma_{t}\Big{(}\rho_{t+1}\Big{(}c_{t+1}(\diamond,\pi_{t+1}(h_{t},\diamond),\cdot),c_{t+2}(\cdot,\pi_{t+2}(h_{t},\diamond,\cdot),\cdot),$
$\displaystyle\cdots,c_{T}(\cdot,\pi_{T}(h_{t},\diamond,\cdot,\ldots,\cdot),\cdot);\delta_{{\theta}}\circledast
P^{\pi^{t,h_{t}}}_{t+1,T|\Theta}(\,\cdot\,\|\diamond)\Big{)};P^{\pi^{t,h_{t}}}_{t+1,t+1|\theta}\Big{)},\theta\in{\boldsymbol{\Theta}}\Big{\\}};\xi^{\pi,h_{t}}_{t}\Big{)}$
$\displaystyle=\widehat{\rho}_{t}\Big{(}\Big{\\{}c_{t}(x_{t},\pi_{t}(h_{t}),\theta)+\sigma_{t}\Big{(}\rho_{t+1}\Big{(}c_{t+1}(\diamond,\pi_{t+1}(h_{t},\diamond),\cdot),c_{t+2}(\cdot,\pi_{t+2}(h_{t},\diamond,\cdot),\cdot),$
$\displaystyle\cdots,c_{T}(\cdot,\pi_{T}(h_{t},\diamond,\cdot,\ldots,\cdot),\cdot);P^{\pi^{t,h_{t}}}_{\theta,t+1,T}(\\{\diamond\\}\times\cdot)\delta_{\theta}(\cdot)\Big{)};P^{\pi^{t,h_{t}}}_{\theta,t+1}\Big{)},\theta\in{\boldsymbol{\Theta}}\Big{\\}};\xi^{\pi,h_{t}}_{t}\Big{)},$
(3.24)

where in the last equality we used (3.5) and (3.14), and where, for a function
$f$ on ${\boldsymbol{\Theta}}$ and a measure $\xi$ on ${\boldsymbol{\Theta}}$,

$\displaystyle\widehat{\rho}_{t}\Big{(}\big{\\{}f(\theta),\theta\in{\boldsymbol{\Theta}}\big{\\}};\xi^{\pi,h_{t}}_{t}\Big{)}=\widehat{\rho}_{t}\big{(}f;\xi^{\pi,h_{t}}_{t}\big{)}=\int_{\boldsymbol{\Theta}}f(\theta)\;\xi^{\pi,h_{t}}_{t}(d\theta)=E_{\xi^{\pi,h_{t}}_{t}}(f),$
(3.25)

and where, for a function $v$ on $\mathcal{X}$, we have (cf. (3.5), (3.18),
and (3.6))

$\displaystyle\sigma_{t}\left(v,P^{\pi^{t,h_{t}}}_{\theta,t+1}\right)=\int_{\mathcal{X}}v(x)P^{\pi^{t,h_{t}}}_{\theta,t+1}d(x)=E_{P^{\pi^{t,h_{t}}}_{\theta,t+1}}(v).$
(3.26)

######  Example 3.17.

In the previous example we proceeded from $\rho$ to $\sigma$ (via
$\widehat{\rho}$). Here, we will do the opposite.

In clinical trials, the potency of a drug is characterized by an unknown
parameter ${\theta}$. The purpose of the trials is to estimate ${\theta}$ and
to determine the optimal dose. Let us assume for simplicity that ${\theta}$
_is_ the optimal dose. If a dose $u_{1}$ is administered to a patient, a
response $X_{2}$ is observed (the subscript 2 indicates that $X_{2}$ is not
known when $u_{1}$ is determined). $X_{2}$ is a Bernoulli random variable,
with $X_{2}=1$ representing toxic response, and $X_{2}=0$ nontoxic. The
probability of toxic response is a function of $\theta$ and $u_{1}$, that is,
$P[X_{2}=1]=\Psi(\theta,u_{1})$. The “cost” is $c(\theta,u_{1})$; it depends
on both the applied and best doses. The cost is not observed; we only know
whether the patient was toxic or not. In the second stage, the dose $u_{2}$ is
administered to the next patient, the patient’s response $X_{3}$ observed, and
cost $c(\theta,u_{2})$ incurred. The process continues for $T$ stages, with
$u_{T}$ being the final dose recommendation, whose cost is equal to
$c(\theta,u_{T})$. For example, the cost may have the form
$c(\theta,u)=|u-\theta|$ to penalize for the over- and under-dosage. It is
never observed.

The problem can be cast to our setting. The state space $\mathcal{X}$ is
$\\{0,1\\}$, while the unknown parameter space $\boldsymbol{\Theta}$ is an
interval of the real line or a finite subset of the real line. Given the set-
up adopted in this paper, we assume that $\boldsymbol{\Theta}$ is a finite
subset of the real line. The transition kernel does not depend on $X$ at all;
the distribution of the next $X_{t+1}$ depends on $\theta$ and $u$:

$K_{\theta}(0|x,u)=1-\Psi(\theta,u),\quad K_{\theta}(1|x,u)=\Psi(\theta,u).$

Thus, we have

$P^{\pi^{t,h_{t}}}_{\theta,t+1}(y)=K_{\theta}(y|x_{t},\pi_{t}(h_{t}))=\mathbbm{1}_{y=0}(1-\Psi(\theta,\pi_{t}(h_{t})))+\mathbbm{1}_{y=1}\Psi(\theta,\pi_{t}(h_{t})),\quad
y\in\\{0,1\\}.$

There is a considerable leverage in choosing the form of $\sigma_{t}$ in a way
consistent with the above set-up. For example, one can choose $\sigma_{t}$ in
terms of the entropic risk measures, as follows

$\sigma(w,P)=\frac{1}{\kappa}\ln\int_{\mathcal{X}}e^{\kappa w(y)}P(dy),$

for a function $f$ on $\mathcal{X}$, $P\in\mathcal{P}(\mathcal{X})$ and a
constant $\kappa>0$ . Consequently, for $t=1,\ldots,T-1$, using (2.5) and
(2.9) we obtain

$\displaystyle\sigma_{t}(w(\cdot,\theta);P^{\pi^{t,h_{t}}}_{t+1,t+1|\theta})$
$\displaystyle=\frac{1}{\varkappa}\ln\bigg{(}\big{(}1-\Psi(\theta,\pi_{t}(h_{t}))\big{)}e^{\varkappa
w(0,\theta)}+\Psi(\theta,\pi_{t}(h_{t}))e^{\varkappa w(1,\theta)}\bigg{)}$
$\displaystyle=\frac{1}{\varkappa}\ln\int_{\mathcal{X}}e^{\varkappa
w(y,\theta)}P^{\pi^{t,h_{t}}}_{\theta,t+1}(dy),$

with $\sigma_{T}=0$.

Now, for a function $f$ on ${\boldsymbol{\Theta}}$ and a measure
$\xi\in\mathcal{P}({\boldsymbol{\Theta}})$, let

$\widehat{\rho}_{t}\Big{(}\big{\\{}f(\theta),\theta\in{\boldsymbol{\Theta}}\big{\\}};\xi\Big{)}=\widehat{\rho}_{t}(f;\xi)=\frac{1}{\varkappa}\ln\int_{\boldsymbol{\Theta}}e^{\varkappa
f(\theta)}\xi(d\theta),\qquad t\in\mathcal{T}.$

Given the above, we obtain for $t=T$

$\widehat{\rho}_{T}\Big{(}\big{\\{}v(\theta),\theta\in{\boldsymbol{\Theta}}\big{\\}};\xi^{\pi,h_{T}}_{T}\Big{)}=\frac{1}{\varkappa}\ln\int_{\boldsymbol{\Theta}}e^{\varkappa
v(\theta)}\xi^{\pi,h_{T}}_{T}(d\theta),$ (3.27)

and for $t=1,\dots,T-1$,

$\displaystyle\widehat{\rho}_{t}\Big{(}\big{\\{}v(\theta)+\sigma_{t}(w(\cdot,\theta);P^{\pi^{t,h_{t}}}_{t+1,t+1|\theta}),\theta\in{\boldsymbol{\Theta}}\big{\\}};\xi^{\pi,h_{t}}_{t}\Big{)}$
$\displaystyle=\frac{1}{\varkappa}\ln\int_{\boldsymbol{\Theta}}\int_{\mathcal{X}}e^{\varkappa(v(\theta)+w(y,\theta))}P^{\pi^{t,h_{t}}}_{\theta,t+1}(dy)\xi^{\pi,h_{t}}_{t}(d\theta)$
$\displaystyle=\frac{1}{\varkappa}\ln
E^{\pi}[e^{\varkappa(v(\Theta)+w(X_{t+1},\Theta))}|\widehat{H}_{t}=h_{t}],$
(3.28)

where the last equality follows from Lemma 2.4.

We will now derive a generic formula for $\rho_{t}$, generated by (3.19) and
$\sigma_{t}$ and $\widehat{\rho}_{t}$ as above, in case of the generic cost
functions as in (2.15) and (2.16). Let us fix an admissible strategy $\pi$.
For $t=T$ we have

$\displaystyle\rho_{T}(c_{T}(x_{T},\pi_{T}(h_{T}),\cdot),P^{\pi^{T,h_{T}}}_{T+1,T})$
$\displaystyle=\widehat{\rho}_{T}(\\{c_{T}(x_{T},\pi_{T}(h_{T}),\theta),\theta\in{\boldsymbol{\Theta}}\\};P^{\pi^{T,h_{T}}}_{T+1,T,\Theta})$
$\displaystyle=\widehat{\rho}_{T}(c_{T}(x_{T},\pi_{T}(h_{T}),\cdot);P^{\pi^{T,h_{T}}}_{T+1,T,\Theta})=\widehat{\rho}_{T}(c_{T}(x_{T},\pi_{T}(h_{T}),\cdot);\xi^{\pi,h_{T}}_{T})$
$\displaystyle=\frac{1}{\varkappa}\ln\int_{\boldsymbol{\Theta}}e^{\varkappa
c_{T}(x_{T},\pi_{T}(h_{T}),\theta)}\xi^{\pi,h_{T}}_{T}(d\theta)$
$\displaystyle=\frac{1}{\varkappa}\ln E^{\pi}(e^{\varkappa
c_{T}(x_{T},\pi_{T}(h_{T}),\Theta)}|\widehat{H}_{T}=h_{T}).$

Now, note that

$\rho_{T}(c_{T}(x_{T},\pi_{T}(h_{T}),\cdot),\delta_{\theta})=c_{T}(x_{T},\pi_{T}(h_{T}),\theta),$

and thus

$\displaystyle\sigma_{T-1}\big{(}\rho_{T}\big{(}c_{T}(\diamond,\pi_{T}(h_{T-1},\diamond),\theta);\delta_{{\theta}});$
$\displaystyle
P^{\pi^{T-1,h_{T-1}}}_{T,T|\theta}\big{)}=\sigma_{T-1}\big{(}c_{T}(\diamond,\pi_{T}(h_{T-1},\diamond),\theta);P^{\pi^{T-1,h_{T-1}}}_{T,T|\theta}\big{)}$
$\displaystyle=\frac{1}{\varkappa}\ln\int_{\mathcal{X}}e^{\varkappa
c_{T}(x_{T},\pi_{T}(h_{T-1},x_{T}),\theta)}P^{\pi^{T-1,h_{T-1}}}_{\theta,T}(dx_{T}).$

So, for $t=T-1,$ we have

$\displaystyle\rho_{T-1}($ $\displaystyle
c_{T-1}(x_{T-1},\pi_{T-1}(h_{T-1}),\cdot),c_{T}(\cdot,\pi_{T}(h_{T-1},\cdot),\cdot),P^{\pi^{T-1,h_{T-1}}}_{T,T})$
$\displaystyle=\widehat{\rho}_{T-1}\Big{(}\Big{\\{}c_{T-1}(x_{T-1},\pi_{T-1}(h_{T-1}),\theta)+\sigma_{T-1}\big{(}\rho_{T}\big{(}c_{T}(\diamond,\pi_{T}(h_{T-1},\diamond),\cdot);\delta_{{\theta}});P^{\pi^{T-1,h_{T-1}}}_{T,T|\theta}\big{)}\big{)},$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\theta\in{\boldsymbol{\Theta}}\Big{\\}};P^{\pi^{T-1,h_{T-1}}}_{T,T,\Theta}\Big{)}$
$\displaystyle=\widehat{\rho}_{T-1}\Big{(}\Big{\\{}c_{T-1}(x_{T-1},\pi_{T-1}(h_{T-1}),\theta)+\sigma_{T-1}\big{(}\rho_{T}\big{(}c_{T}(\diamond,\pi_{T}(h_{T-1},\diamond),\cdot);\delta_{{\theta}});P^{\pi^{T-1,h_{T-1}}}_{T,T|\theta}\big{)}\big{)},$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\theta\in{\boldsymbol{\Theta}}\Big{\\}};\xi^{\pi,h_{T-1}}_{T-1}\Big{)}$
$\displaystyle=\frac{1}{\varkappa}\ln\int_{\boldsymbol{\Theta}}\int_{\mathcal{X}}e^{\varkappa(c_{T-1}(x_{T-1},\pi_{T-1}(h_{T-1}),\theta)+c_{T}(x_{T},\pi_{T}(h_{T-1},x_{T}),\theta))}P^{\pi^{T-1,h_{T-1}}}_{\theta,T}(dx_{T})\xi^{\pi,h_{T-1}}_{T-1}(d\theta)$
$\displaystyle=\frac{1}{\varkappa}\ln
E^{\pi}(e^{\varkappa(c_{T-1}(x_{T-1},\pi_{T-1}(h_{T-1}),\Theta)+c_{T}(X_{T},\pi_{T}(h_{T-1},X_{T}),\Theta))}|\widehat{H}_{T-1}=h_{T-1}),$

where we used (2.5) and (2.9) for the second to the last equality, and where
the last equality follows from Lemma 2.4.

Next, note that

$\displaystyle\rho_{T-1}\big{(}c_{T-1}($
$\displaystyle\diamond,\pi_{T-1}(h_{T-2},\diamond),\cdot),c_{T}(\cdot,\pi_{T}(h_{T-2},\diamond,\cdot),\cdot);\delta_{\theta}\circledast
P^{\pi^{T-1,(h_{T-2},\diamond)}}_{T,T|\Theta}(\cdot\|\diamond)\big{)}$
$\displaystyle=\frac{1}{\varkappa}\ln\int_{\mathcal{X}}e^{\varkappa(c_{T-1}(\diamond,\pi_{T-1}(h_{T-2},\diamond),\theta)+c_{T}(y,\pi_{T}(h_{T-2},\diamond,y),\theta))}P^{\pi^{T-1,(h_{T-2},\diamond)}}_{\theta,T}(dy),$

and

$\displaystyle\sigma_{T-2}\big{(}\rho_{T-1}$
$\displaystyle\big{(}c_{T-1}(\diamond,\pi_{T-1}(h_{T-2},\diamond),\theta),c_{T}(\cdot,\pi_{T}(h_{T-2},\diamond,\cdot),\theta);$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\delta_{{\theta}}\circledast
P^{\pi^{T-1,(h_{T-2},\diamond)}}_{T,T|\Theta}(\cdot\|\diamond)\big{)};P^{\pi^{T-2,h_{T-2}}}_{T-1,T-1|\theta}\big{)}$
$\displaystyle=\frac{1}{\varkappa}\ln\int_{\mathcal{X}}e^{\varkappa(c_{T-1}(x_{T-1},\pi_{T-1}(h_{T-2},x_{T-1}),\theta)+c_{T}(x_{T},\pi_{T}(h_{T-2},x_{T-1},x_{T}),\theta))}$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad
P^{\pi^{T-1,(h_{T-2},x_{T-1})}}_{\theta,T}(dx_{T})P^{\pi^{T-2,h_{T-2}}}_{\theta,T-1}(dx_{T-1}).$

We take now $t=T-2$. In this case,

$\rho_{T-2,T}(c_{T-2}(x_{T-2},\pi_{T-2}(h_{T-2}),\cdot),c_{T-1}(\diamond,\pi_{T-1}(h_{T-2},\diamond),\cdot),c_{T}(\cdot,\pi_{T}(h_{T-2},\diamond,\cdot),\cdot),P^{\pi^{T-2,h_{T-2}}}_{T-1,T})\\\
=\widehat{\rho}_{T-2}\Big{(}\Big{\\{}c_{T-2}(x_{T-2},\pi_{T-2}(h_{T-2}),\theta)+\sigma_{T-2}\big{(}\rho_{T-1}\big{(}c_{T-1}(\diamond,\pi_{T-1}(h_{T-2},\diamond),\theta),c_{T}(\cdot,\pi_{T}(h_{T-2},\diamond,\cdot),\theta);\\\
\delta_{{\theta}}\circledast
P^{\pi^{T-1,(h_{T-2},\diamond)}}_{T,T|\Theta}(\cdot\|\diamond)\big{)};P^{\pi^{T-2,h_{T-2}}}_{T-1,T-1|\theta}\big{)},\theta\in{\boldsymbol{\Theta}}\Big{\\}};P^{\pi^{T-2,h_{T-2}}}_{T-1,T|\Theta}\Big{)}\\\
=\widehat{\rho}_{T-2}\Big{(}\Big{\\{}c_{T-2}(x_{T-2},\pi_{T-2}(h_{T-2}),\theta)+\sigma_{T-2}\big{(}\rho_{T-1}\big{(}c_{T-1}(\diamond,\pi_{T-1}(h_{T-2},\diamond),\theta),c_{T}(\cdot,\pi_{T}(h_{T-2},\diamond,\cdot),\theta);\\\
\delta_{{\theta}}\circledast
P^{\pi^{T-1,(h_{T-2},\diamond)}}_{T,T|\Theta}(\cdot\|\diamond)\big{)};P^{\pi^{T-2,h_{T-2}}}_{T-1,T-1|\theta}\big{)},\theta\in{\boldsymbol{\Theta}}\Big{\\}};\xi^{\pi,h_{T-2}}_{T-2}\Big{)}\\\
=\frac{1}{\varkappa}\ln\int_{\boldsymbol{\Theta}}\int_{\mathcal{X}}\int_{\mathcal{X}}e^{\varkappa(c_{T-2}(x_{T-2},\pi_{T-2}(h_{T-2}),\theta)+c_{T-1}(x_{T-1},\pi_{T-1}(h_{T-2},x_{T-1}),\theta)+c_{T}(x_{T},\pi_{T}(h_{T-2},x_{T-1},x_{T}),\theta))}\\\
P^{\pi^{T-1,h_{T-2},x_{T-1}}}_{\theta,T}(dx_{T})P^{\pi^{T-2,h_{T-2}}}_{\theta,T-1}(dx_{T-1})\xi^{\pi,h_{T-2}}_{T-2}(d\theta)\\\
=\frac{1}{\varkappa}\ln
E^{\pi}(e^{\varkappa(c_{T-2}(x_{T-2},\pi_{T-2}(h_{T-2}),\Theta)+c_{T-1}(X_{T-1},\pi_{T-1}(h_{T-2},X_{T-1}),\Theta)+c_{T}(X_{T},\pi_{T}(h_{T-2},X_{T-1},X_{T}),\Theta))}\\\
|\widehat{H}_{T-2}=h_{T-2}),$

where the last equality follows from Lemma 2.4.

Proceeding in the analogous way for $t=T-3,\ldots,1$ we finally obtain

$\displaystyle\rho_{1,T}(c_{1}(x_{1},\pi_{1}(h_{1}),$
$\displaystyle\cdot),\ldots,c_{T}(\cdot,\pi_{T}(h_{1},\cdot,\ldots,\cdot),\cdot),P^{\pi^{1,h_{1}}}_{2,T})$
$\displaystyle=\frac{1}{\varkappa}\ln
E^{\pi}\left(e^{\varkappa\sum_{k=1}^{T}\,c_{k}(X_{k},\pi_{k}(h_{k})),\Theta)}|\widehat{H}_{1}=h_{1}\right),$
(3.29)

which gives us the risk-sensitive criterion with entropic utility (cf. [BR14,
DL14]) .

##  4 Recursive Risk Filters

Let us fix $t\in\\{1,\dots,T-1\\}$. To alleviate notation, for all
$\pi\in\varPi$, we write for fixed functions
$Z_{t,s}(\cdot_{s-t},\cdot_{1})\in\mathcal{Z}_{t,s}$, $s=t,\ldots,T$. Since
$t$ is fixed, we will again simply write $Z_{s}$ instead of $Z_{t,s}$, for
$s\in\mathcal{T}_{t}$.

$\displaystyle v_{t}^{\pi}(h_{t})$
$\displaystyle=\rho_{t}(Z_{t},Z_{t+1},\dots,Z_{T};P^{\pi^{t,h_{t}}}_{t+1,T})$
(4.1) $\displaystyle\widetilde{v}_{t+1}^{\pi,\theta}((h_{t},x_{t+1}))$
$\displaystyle:=\rho_{t+1}\big{(}Z_{t+1}(x_{t+1},\cdot_{1}),\dots,Z_{T}(x_{t+1},\cdot_{T-t-1},\cdot_{1});\delta_{{\theta}}\circledast
P^{\pi^{t,h_{t}}}_{t+1,T|\Theta}(\,\cdot\,\|x_{t+1})\big{)}$
$\displaystyle=\rho_{t+1}\big{(}Z_{t+1}(x_{t+1},\cdot_{1}),\dots,Z_{T}(x_{t+1},\cdot_{T-t-1},\cdot_{1});P^{\pi^{t,h_{t}}}_{\theta,t+1,T}(\\{x_{t+1}\\}\times\cdot)\delta_{\theta}(\cdot)\big{)},$
(4.2)

where for the last equality we used (3.14).

The quantity $v_{t}^{\pi}(h_{t})$ evaluates the policy $\pi$ at the time $t$
and with the history $h_{t}$ in the original problem.

Recall that (cf. (3.5))
$P^{\pi^{t,h_{t}}}_{t+1,T,\Theta}=\xi^{\pi,h_{t}}_{t}$. Thus, the key equation
(3.19) can be written more compactly as follows:

$\displaystyle v_{t}^{\pi}(h_{t})$
$\displaystyle=\widehat{\rho}_{t}\Big{(}\Big{\\{}Z_{t}(\theta)+\sigma_{t}\big{(}\rho_{t+1}\big{(}Z_{t+1}(\diamond,\cdot_{1}),\dots,Z_{T}(\diamond,\cdot_{T-t-1},\cdot_{1});$
$\displaystyle\qquad\qquad\qquad\qquad\delta_{{\theta}}\circledast
P^{\pi^{t,h_{t}}}_{t+1,T|\Theta}(\,\cdot\,\|\diamond)\big{)};P^{\pi^{t,h_{t}}}_{t+1,t+1|\theta}\big{)},\theta\in{\boldsymbol{\Theta}}\Big{\\}};P^{\pi^{t,h_{t}}}_{t+1,T,\Theta}\Big{)}$
$\displaystyle=\widehat{\rho}_{t}\Big{(}\Big{\\{}Z_{t}(\theta)+\sigma_{t}\big{(}Z_{t+1}(\diamond,\cdot_{1}),\dots,Z_{T}(\diamond,\cdot_{T-t-1},\cdot_{1});$
$\displaystyle\qquad\qquad\qquad\qquad\delta_{{\theta}}\circledast
P^{\pi^{t,h_{t}}}_{t+1,T|\Theta}(\,\cdot\,\|\diamond)\big{)};P^{\pi^{t,h_{t}}}_{t+1,t+1|\theta}\big{)},\theta\in{\boldsymbol{\Theta}}\Big{\\}};\xi^{\pi,h_{t}}_{t}\Big{)}$
$\displaystyle=\widehat{\rho}_{t}\Big{(}\Big{\\{}Z_{t}(\theta)+\sigma_{t}\big{(}\widetilde{v}_{t+1}^{\pi,\theta}((h_{t},\diamond));P^{\pi^{t,h_{t}}}_{t+1,t+1|\theta}\big{)},\theta\in{\boldsymbol{\Theta}}\Big{\\}};\xi^{\pi,h_{t}}_{t}\Big{)},$
$\displaystyle=\widehat{\rho}_{t}\Big{(}\Big{\\{}Z_{t}(\theta)+\sigma_{t}\big{(}\widetilde{v}_{t+1}^{\pi,\theta}((h_{t},\diamond));P^{\pi^{t,h_{t}}}_{\theta,t+1}\big{)},\theta\in{\boldsymbol{\Theta}}\Big{\\}};\xi^{\pi,h_{t}}_{t}\Big{)},$
(4.3)

with $\sigma_{t}$ given in (3.18), and where we used (2.12) in the last
equality.

Note that in equation (4.3) we have $\widetilde{v}_{t+1}^{\pi,\theta}$ on the
right hand side. Thus, this equation does not provide a convenient recursion
for the quantities $v^{\pi}_{t}$. This leads us to the following concept,

######  Definition 4.1.

A dynamic risk filter $\rho$ is called recursive if it is satisfies the
properties stated in Theorem 3.14 and

$v_{t}^{\pi}(h_{t})=\widehat{\rho}_{t}\Big{(}\Big{\\{}Z_{t,t}(\theta)+\sigma_{t}\big{(}v_{t+1}^{\pi}((h_{t},\diamond));P^{\pi^{t,h_{t}}}_{\theta,t+1}\big{)},\theta\in{\boldsymbol{\Theta}}\Big{\\}};\xi^{\pi,h_{t}}_{t}\Big{)},$

for $t=T-1,\ldots,1$, with

$v_{T}^{\pi}(h_{T})=\widehat{\rho}_{t}\Big{(}\Big{\\{}Z_{T,T}(\theta),\theta\in{\boldsymbol{\Theta}}\Big{\\}};\xi^{\pi,h_{T}}_{T}\Big{)}.$

###  4.1 Examples of recursive dynamic risk-filters

We will show here that risk-filters considered in Example 3.16 and Example
3.17 are recursive.

In the case of the additive risk rewards, that is Example 3.16, using (3.16)
we get

$\displaystyle v^{\pi}_{t}(h_{t})$
$\displaystyle=\rho_{t}\Big{(}c_{t}(x_{t},\pi_{t}(h_{t}),\cdot),c_{t+1}(\cdot,\pi_{t+1}(h_{t},\cdot),\cdot),$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad
c_{t+2}(\cdot,\pi_{t+2}(h_{t},\cdot,\cdot),\cdot),\cdots,c_{T}(\cdot,\pi_{T}(h_{t},\cdot,\ldots,\cdot),\cdot),P^{\pi^{t,h_{t}}}_{t+1,T}\Big{)}$
$\displaystyle=E^{\pi}\left[c_{t}(x_{t},\pi_{t}(h_{t}),\Theta)+\sum_{s=t+1}^{T}c_{s}(\widehat{X}_{s},\pi_{s}(\widehat{X}_{t+1},\ldots,\widehat{X}_{s}),\Theta)\;|\;\widehat{H}_{t}=h_{t}\right],$
(4.4)

for $t\in\mathcal{T}$, where again we use the standard convention that an
empty sum is zero (i.e. $\sum_{s=t+T}^{T}\cdots=0$ in our case).

Given the above, we obtain that the risk filter considered in Example 3.16 is
recursive:

######  Lemma 4.2.

Let $\widehat{\rho}_{t}$ and $\sigma_{t}$ be given as in (3.25) and (3.26),
respectively. We have

$\displaystyle
v^{\pi}_{t}(h_{t})=\widehat{\rho}_{t}\Big{(}\Big{\\{}c_{t}(x_{t},\pi_{t}(h_{t}),\theta)+\sigma_{t}\big{(}v_{t+1}^{\pi}((h_{t},\cdot));P^{\pi^{t,h_{t}}}_{\theta,t+1}\big{)},\theta\in{\boldsymbol{\Theta}}\Big{\\}};\xi^{\pi,h_{t}}_{t}\Big{)}$
(4.5)

for $t=T-1,\ldots,1$, with

$\displaystyle
v^{\pi}_{T}(h_{T})=\int_{\boldsymbol{\Theta}}\,c_{T}(x_{T},\pi_{T}(h_{T}),\theta)\xi^{\pi,h_{T}}_{T}(d\theta).$
(4.6)

###### Proof.

Fix $t\in\\{1,\ldots,T-1\\}.$ First, using (4.1) and the tower property of
conditional expectations we get

$\displaystyle v_{t}^{\pi}(\widehat{H}_{t})$
$\displaystyle=E^{\pi}\left[c_{t}(\widehat{X}_{t},\pi_{t}(\widehat{H}_{t}),\Theta)+\sum_{s=t+1}^{T}c_{s}(\widehat{X}_{s},\pi_{s}^{t,h_{t}}(\widehat{X}_{t+1},\ldots,\widehat{X}_{s}),\Theta)\;|\;\widehat{H}_{t}\right]$
$\displaystyle=E^{\pi}\left[c_{t}(\widehat{X}_{t},\pi_{t}(\widehat{H}_{t}),\Theta)+E^{\pi}\left[\sum_{s=t+1}^{T}c_{s}(\widehat{X}_{s},\pi_{s}(\widehat{X}_{t+1},\ldots,\widehat{X}_{s}),\Theta)\;|\;\widehat{H}_{t+1}\right]\;|\;\widehat{H}_{t}\right],$
$\displaystyle=E^{\pi}[c_{t}(\widehat{X}_{t},\pi_{t}(\widehat{H}_{t}),\Theta)+v_{t+1}^{\pi}(\widehat{H}_{t+1})\;|\;\widehat{H}_{t}],$

and so

$\displaystyle
v_{t}^{\pi}(h_{t})=E^{\pi}[c_{t}(x_{t},\pi_{t}(h_{t}),\Theta)+v_{t+1}^{\pi}(h_{t},\widehat{X}_{t+1})\;|\;\widehat{H}_{t}=h_{t}].$

Next, by Lemma 2.4, we have

$v_{t}^{\pi}(h_{t})=\int_{\boldsymbol{\Theta}}\int_{\mathcal{X}}\left(c_{t}(x_{t},\pi_{t}(h_{t}),\theta)+v_{t+1}^{\pi}(h_{t},x_{t+1})\right)P_{\theta,t+1}^{\pi^{t,h_{t}}}(dx_{t+1})\;\xi_{t}^{\pi^{t,h_{t}}}(d\theta),$
(4.7)

and by taking into account the form of $\widehat{\rho}_{t}$ and $\sigma_{t}$
as in (3.25) and (3.26), respectively, we obtain (4.5). Finally, (4.6) is a
direct consequence of (4.1) and Lemma 2.4.

∎

In the case of the risk-sensitive rewards, that is Example 3.17, the
recursiveness of $\rho$ can be demonstrated in a way analogous to the above.

##  5 Risk-Averse Control Problem

Let $v_{1}^{\pi}$ be as in (4.1). The control problem is to find

$\min_{\pi\in\Pi}v_{1}^{\pi}(h_{1}),$ (5.1)

as well as the optimal policy, say $\pi^{*}$, for which
$v_{1}^{\pi^{*}}(h_{1})=\min_{\pi\in\Pi}v_{1}^{\pi}(h_{1}).$ Note that given
our set-up, an optimal policy does exist because the set $\Pi$ is finite.
However, we are interested in seeking an optimal policy in the class of quasi-
Markov policies.

######  Definition 5.1.

A policy $\pi\in\varPi$ is _quasi-Markov (QMP)_ if

$\pi_{t}(h_{t})=\phi_{t}(x_{t},\xi^{\pi,h_{t}}_{t})$

for some function
$\phi_{t}\,:\,\mathcal{X}\times{\boldsymbol{\Theta}}\rightarrow\mathcal{U}$,
$t=1,\ldots,T$.

###  5.1 The Bayes Operator

At each time $t$ and for every policy $\pi$ and history $h_{t}$, the measure
$P^{\pi^{t,h_{t}}}_{t+1}$ (cf. (2.14)) describes the conditional joint
distribution of the pair $(\widehat{X}_{t+1},\Theta)$ in
$\mathcal{X}\times{\boldsymbol{\Theta}}$.

This measure admits two natural disintegrations. One of them is already
obtained from (2.7), repeated here :

$\displaystyle P^{\pi^{t,h_{t}}}_{t+1}(B\times D)$
$\displaystyle=P^{\pi^{t,h_{t}}}_{t+1,T}(B\times\mathcal{X}^{T-t-1}\times
D)=\int_{D}\,{P^{\pi^{t,h_{t}}}_{\theta,t+1,T}(B\times\mathcal{X}^{T-t-1})}\;\xi^{\pi,h_{t}}_{t}(d\theta)$
$\displaystyle=P^{\pi}[\widehat{X}_{t+1}\in B,\Theta\in
D\,|\,\widehat{H}_{t}=h_{t}],$

where $\xi^{\pi,h_{t}}_{t}\in\mathcal{P}(\boldsymbol{\Theta})$, is given as
(cf. (2.8) and (2.10)) $\xi^{\pi,h_{t}}_{t}(D)=P^{\pi}[\Theta\in
D\,|\,\widehat{H}_{t}=h_{t}]=P^{\pi^{t,h_{t}}}_{t+1,\Theta}(D).$ One can also
disintegrate $P^{\pi^{t,h_{t}}}_{t+1}$ into its marginal on $\mathcal{X}$,
say777For simplicity of notations, we write $P^{\pi^{t,h_{t}}}_{t+1,X}$
instead of more coherent notation $P^{\pi^{t,h_{t}}}_{t+1,X_{t+1}}$. Similar
remark applies to the kernel $P^{\pi^{t,h_{t}}}_{t+1|X}$.
$P^{\pi^{t,h_{t}}}_{t+1,X}$, and the corresponding stochastic kernel, say
$P^{\pi^{t,h_{t}}}_{t+1|X}$ from $\mathcal{X}$ to $\boldsymbol{\Theta}$. That
is, for any $B\times D\subset\mathcal{X}\times\boldsymbol{\Theta}$,

$\displaystyle P^{\pi^{t,h_{t}}}_{t+1}(B\times D)$
$\displaystyle=(P^{\pi^{t,h_{t}}}_{t+1,X}\circledast
P^{\pi^{t,h_{t}}}_{t+1|{X}})(B\times D),$
$\displaystyle=\int_{B}\,P^{\pi^{t,h_{t}}}_{t+1|x}(D)P^{\pi^{t,h_{t}}}_{t+1,X}(dx)$
$\displaystyle=P^{\pi}[\widehat{X}_{t+1}\in B,\Theta\in
D\,|\,\widehat{H}_{t}=h_{t}],$ (5.2)

where we used the simplified notation $P^{\pi^{t,h_{t}}}_{t+1|x}(D)$ for
$P^{\pi^{t,h_{t}}}_{t+1|{X}}(x,D).$

The kernel $P^{\pi^{t,h_{t}}}_{t+1|x}$ is the Bayes operator which describes
the dynamics of the belief states, as documented in the next result.

######  Lemma 5.2.

For $t=1,\ldots,T-1$, $h_{t}\in H_{t}$, $x_{t+1}\in\mathcal{X}$ and
$D\subset{\boldsymbol{\Theta}}$, we have

$\displaystyle\xi^{\pi,(h_{t},x_{t+1})}_{t+1}(D)$
$\displaystyle=P^{\pi^{t,h_{t}}}_{t+1\;|\;x_{t+1}}(D)$
$\displaystyle=\int_{D}\frac{K_{\theta}(x_{t+1}|x_{t},\pi_{t}(h_{t}))}{P^{\pi^{t,h_{t}}}_{t+1}[\\{x_{t+1}\\}\times{\boldsymbol{\Theta}}]}\,\xi^{\pi,h_{t}}_{t}(d\theta),$
(5.3)

where

$\xi^{\pi,x_{1}}_{1}(\theta)=\xi_{1}(\theta).$ (5.4)

###### Proof.

First, note that

$P^{\pi^{t,h_{t}}}_{t+1,X}(B)=P^{\pi}[\widehat{X}_{t+1}\in
B\,|\,\widehat{H}_{t}=h_{t}].$ (5.5)

Take $B=\\{x_{t+1}\\}$. Then, using (5.2) and (5.5), we obtain

$P^{\pi}[\widehat{X}_{t+1}=x_{t+1},\Theta\in
D\,|\,\widehat{H}_{t}=h_{t}]=P^{\pi^{t,h_{t}}}_{t+1|x_{t+1}}(D)P^{\pi}[\widehat{X}_{t+1}=x_{t+1}\,|\,\widehat{H}_{t}=h_{t}],$

and thus

$\displaystyle P^{\pi^{t,h_{t}}}_{t+1|x_{t+1}}(D)$
$\displaystyle=\frac{P^{\pi}[\widehat{X}_{t+1}=x_{t+1},\Theta\in
D\,|\,\widehat{H}_{t}=h_{t}]}{P^{\pi}[\widehat{X}_{t+1}=x_{t+1}\,|\,\widehat{H}_{t}=h_{t}]}$
$\displaystyle=P^{\pi}[\Theta\in
D\,|\,\widehat{H}_{t+1}=(h_{t},x_{t+1})]=\xi^{\pi,(h_{t},x_{t+1})}_{t+1}(D),$

which proves the first equality in (5.2). The second one follows from the
following chain of equalities,

$\displaystyle\xi^{\pi,(h_{t},x_{t+1})}_{t+1}(\theta)$
$\displaystyle=P^{\pi}[\Theta=\theta\,|\,\widehat{H}_{t+1}=(h_{t},x_{t+1})]$
$\displaystyle=P^{\pi}[\Theta=\theta\,|\,\widehat{H}_{t}=h_{t}]\frac{P^{\pi}[\widehat{X}_{t+1}=x_{t+1}\,|\,\boldsymbol{\Theta}=\theta,\widehat{H}_{t}=h_{t}]}{P^{\pi}[\widehat{X}_{t+1}=x_{t+1}\,|\,\widehat{H}_{t}=h_{t}]}$
$\displaystyle=\xi^{\pi,h_{t}}_{t}(\theta)\frac{K_{\theta}(x_{t+1}|x_{t},\pi_{t}(h_{t}))}{P^{\pi^{t,h_{t}}}_{t+1}[\\{x_{t+1}\\}\times{\boldsymbol{\Theta}}]},$

where in last equality we used (2.6) and that
$P^{\pi}_{\theta}(B)=P^{\pi}(B|\Theta=\theta)$. ∎

###  5.2 Optimal control problem corresponding to Example 3.15

In this section we will study the optimal control problem corresponding to the
Example 3.15 classical additive reward case, that will serve as the base for
the general case. In what follows, we denote by $(x,\xi)$ an element of the
set $\mathcal{X}\times\mathcal{P}(\boldsymbol{\Theta}).$

Recall (4.1). Accordingly, we have for $t=T$

$\displaystyle v^{\pi}_{T}(h_{T})$
$\displaystyle=\rho_{T,T}\Big{(}c_{T}(x_{T},\pi_{T}(h_{T}),\cdot),P^{\pi^{T,h_{T}}}_{T+1,T}\Big{)}$
$\displaystyle=\int_{\boldsymbol{\Theta}}c_{T}(x_{T},\pi_{T}(h_{T}),\theta))\;P^{\pi^{T,h_{T}}}_{T+1,T}(d\theta)$
$\displaystyle=\int_{\boldsymbol{\Theta}}c_{T}(x_{T},\pi_{T}(h_{T}),\theta)\;\xi^{\pi,h_{T}}_{T}(d\theta)$
(5.6)
$\displaystyle=E^{\pi}\Big{(}c_{T}(x_{T},\pi_{T}(h_{T}),\Theta)\;|\;\widehat{H}_{T}=h_{T}\Big{)}.$

Thus, observing that $\xi^{\pi,h_{T}}_{T}$, does not depend on $\pi_{T}$,
letting $x_{T}=x$ and $\xi^{\pi,h_{T}}_{T}=\xi$, we compute the candidate-
optimal quasi-Markov control $\phi_{T}$ as

$\phi_{T}(x,\xi)=\operatorname*{arg\,min}_{u\in\mathcal{U}}\,\int_{\boldsymbol{\Theta}}\,c_{T}(x,u,\theta)\;\xi(d\theta).$
(5.7)

We define the Bellman function at time $t=T$:

$V_{T}(x,\xi)=\min_{u\in\mathcal{U}}\,\int_{\boldsymbol{\Theta}}\,c_{T}(x,u,\theta)\,\xi(d\theta)=\int_{\boldsymbol{\Theta}}\,c_{T}(x,\phi_{T}(x,\xi),\theta)\,\xi(d\theta).$
(5.8)

Now, we proceed to time $t=T-1$. Noting that $\xi^{\pi,h_{T-1}}_{T-1}$, does
not depend on $\pi_{T-1}$, letting $x_{T-1}=x$ and
$\xi^{\pi,h_{T-1}}_{T-1}=\xi$, we compute the candidate-optimal quasi-Markov
control $\phi_{T-1}$ as

$\phi_{T-1}(x,\xi)=\operatorname*{arg\,min}_{u\in\mathcal{U}}\,\int_{\boldsymbol{\Theta}}\,\Big{(}c_{T-1}(x,u,\theta)+\int_{\mathcal{X}}V_{T}(x_{T},{\widetilde{\xi}^{u,x_{T},\xi}_{T}})K_{\theta}(dx_{T}|x,u)\Big{)}\,\xi(d\theta),$
(5.9)

where (cf. (5.2))

$\widetilde{\xi}^{u,x_{T},\xi}_{T}(\theta)=\xi(\theta)\frac{K_{\theta}(x_{T}|x,u)}{\int_{{\boldsymbol{\Theta}}}\,K_{\theta}(x_{T}|x,u)\,\xi(d\theta)}.$
(5.10)

The corresponding Bellman function is

$\displaystyle V_{T-1}(x,\xi)$
$\displaystyle=\min_{u\in\mathcal{U}}\,\int_{\boldsymbol{\Theta}}\,\Big{(}c_{T-1}(x,u,\theta)+\int_{\mathcal{X}}V_{T}(x_{T},{\widetilde{\xi}^{u,x_{T},{\xi}}_{T}})K_{\theta}(dx_{T}|x,u)\Big{)}\,\xi(d\theta)$
$\displaystyle=\int_{\boldsymbol{\Theta}}\,\Big{(}c_{T-1}(x,\phi_{T-1}(x,\xi),\theta)+\int_{\mathcal{X}}V_{T}(x_{T},{\widetilde{\xi}^{\phi_{T-1}(x,\xi),x_{T},\xi}_{T}})K_{\theta}(dx_{T}|x,\phi_{T-1}(x,\xi))\Big{)}\,\xi(d\theta).$

Following this pattern, we arrive at the dynamic programming (DP) backward
recursion:

$V_{t}(x,\xi)=\min_{u\in\mathcal{U}}\,\int_{\boldsymbol{\Theta}}\,\Big{(}c_{t}(x,u,\theta)+\int_{\mathcal{X}}V_{t+1}(x_{t+1},{\widetilde{\xi}^{u,x_{t+1},\xi}_{t+1}})K_{\theta}(dx_{t+1}|x,u)\Big{)}\,\xi(d\theta),\quad
t\in\mathcal{T},$ (5.11)

where (cf. (5.2))

$\widetilde{\xi}^{u,x_{t+1},\xi}_{t+1}(\theta)=\xi(\theta)\frac{K_{\theta}(x_{t+1}|x,u)}{\int_{{\boldsymbol{\Theta}}}\,K_{\theta}(x_{t+1}|x,u)\,\xi(d\theta)},$
(5.12)

and

$V_{T+1}\equiv 0.$ (5.13)

Note that (5.11) is a counterpart of (4.5).

Accordingly, for $t=1,\ldots,T$ we define the candidate-optimal quasi-Markov
control $\phi_{t}$ as

$\phi_{t}(x,\xi)=\operatorname*{arg\,min}_{u\in\mathcal{U}}\,\int_{\boldsymbol{\Theta}}\,\Big{(}c_{t}(x,u,\theta)+\int_{\mathcal{X}}V_{t+1}(x_{t+1},{\widetilde{\xi}^{u,x_{t+1},\xi}_{t+1}})K_{\theta}(dx_{t+1}|x,u)\Big{)}\,\xi(d\theta).$
(5.14)

Recall that $\xi_{1}$ is a given prior distribution for $\Theta$. Also, recall
that $h_{1}=x_{1}$.

Next, define a policy $\pi^{*}$ as follows,

$\displaystyle\pi^{*}_{1}(h_{1})$ $\displaystyle=\phi_{1}(x_{1},\xi_{1})$
$\displaystyle\pi^{*}_{t}(h_{t})$
$\displaystyle=\phi_{t}(x_{t},\widehat{\xi}^{\pi^{*},h_{t}}_{t}),\quad
t=2,\ldots,T,$ (5.15)

where

$\displaystyle\widehat{\xi}^{\pi^{*},h_{2}}_{2}=\widetilde{\xi}^{\pi^{*}_{1}(h_{1}),x_{2},\xi_{1}}_{2},\quad\widehat{\xi}^{\pi^{*},h_{3}}_{3}=\widetilde{\xi}^{\pi^{*}_{2}(h_{2}),x_{3},\widehat{\xi}^{\pi^{*},h_{2}}_{2}}_{3},\ldots$
(5.16)

The next result is the optimality verification theorem.

######  Theorem 5.3.

We have,

$\min_{\pi\in\Pi}v^{\pi}_{1}(h_{1})=v^{\pi^{*}}_{1}(h_{1})=V_{1}(x_{1},\xi_{1}).$

###### Proof.

Let $\pi\in\Pi$. For $t=T$ we have

$v^{\pi}_{T}(h_{T})\geq
V_{T}(x_{T},\xi^{\pi,h_{T}}_{T})=\int_{\boldsymbol{\Theta}}\,c_{T}(x_{T},\phi_{T}(x_{T},\xi^{\pi,h_{T}}_{T}),\theta)\,\xi^{\pi,h_{T}}_{T}(d\theta).$

For $t=T-1$, using the above, the recursion (4.7), and (5.2), we have

$\displaystyle v^{\pi}_{T-1}$
$\displaystyle(h_{T-1})=\int_{\boldsymbol{\Theta}}\,\Bigg{(}c_{T-1}(x_{T-1},\pi_{T-1}(h_{T-1}),\theta)+\int_{\mathcal{X}}\,v_{T}^{\pi}(h_{T})K_{\theta}\big{(}dx_{T}|x_{T-1},\pi_{T-1}(h_{T-1})\big{)}\Bigg{)}\xi^{\pi,h_{T-1}}_{T-1}(d\theta)$
$\displaystyle\geq\int_{\boldsymbol{\Theta}}\,\Bigg{(}c_{T-1}(x_{T-1},\pi_{T-1}(h_{T-1}),\theta)+\int_{\mathcal{X}}\,V_{T}(x_{T},\xi^{\pi,h_{T}}_{T})K_{\theta}\big{(}dx_{T}|x_{T-1},\pi_{T-1}(h_{T-1})\big{)}\Bigg{)}\xi^{\pi,h_{T-1}}_{T-1}(d\theta)$
$\displaystyle\geq\int_{\boldsymbol{\Theta}}\,\Bigg{(}c_{T-1}(x_{T-1},\phi_{T-1}(x_{T-1},\xi^{\pi,h_{T-1}}_{T-1}),\theta)$
$\displaystyle\qquad+\int_{\mathcal{X}}\,{V_{T}(x_{T},\widetilde{\xi}^{\phi_{T-1}(x_{T-1},\xi^{\pi,h_{T-1}}_{T-1}),x_{T},\xi^{\pi,h_{T-1}}_{T-1}}_{T})}K_{\theta}\big{(}dx_{T}|x_{T-1},\phi_{T-1}(x_{T-1},\xi^{\pi,h_{T-1}}_{T-1})\big{)}\Bigg{)}\xi^{\pi,h_{T-1}}_{T-1}(d\theta)$
$\displaystyle=V_{T-1}(x_{T-1},\xi^{\pi,h_{T-1}}_{T-1}).$

Likewise, for $t=1,\ldots,T-2$, we have

$v^{\pi}_{t}(h_{t})\geq
V_{t}(x_{t},\xi^{\pi,h_{t}}_{t})=\int_{\boldsymbol{\Theta}}\,\Bigg{(}c_{t}(x_{t},\phi_{t}(x_{t},\xi^{\pi,h_{t}}_{t}),\theta)\\\
+\int_{\mathcal{X}}\,V_{t+1}(x_{t+1},\phi_{t+1}(x_{t+1},\widetilde{\xi}^{\phi_{t}(x_{t},\xi^{\pi,h_{t}}_{t}),x_{{t+1}},\xi^{\pi,h_{t}}_{t}}_{{t+1}}))K_{\theta}\big{(}dx_{{t+1}}|x_{t},\phi_{t}(x_{t},\xi^{\pi,h_{t}}_{t})\big{)}\Bigg{)}\xi^{\pi,h_{t}}_{t}(d\theta).$

Now, if $\pi$ and $\xi^{\pi,h_{t}}_{t}$, $t\in\mathcal{T}$, above are replaced
with $\pi^{*}$ and $\widehat{\xi}^{\,\pi^{*}\\!,h_{t}}_{t}$,
$t\in\mathcal{T}$, respectively, then the inequalities above become
equalities, proving that $\pi^{*}$ is an optimal strategy.

∎

Recalling (3.25) and (3.26), we note that the key DP recursion (5.9) can be
written as

$V_{t}(x,\xi)=\min_{u\in\mathcal{U}}\,\widehat{\rho}_{t}\Big{(}\Big{\\{}c_{t}(x,u,\theta)+{\sigma_{t}(V_{t+1}(\cdot,{\widetilde{\xi}^{u,\cdot,\xi}_{t+1}});K_{\theta}(x,u))},\theta\in{\boldsymbol{\Theta}}\Big{\\}};\xi\Big{)},$

subject to (5.12) and (5.13).

######  Example 5.4.

We remark that the optimal control problem considered in this section can be
cast in the classical optimal investment and consumption problem, now also
subject to model uncertainty. Namely, consider an investor with initial
capital $\bar{x}_{1}$, who can invest in $d$ assets, with $\bar{X}_{t}$
denoting the portfolio value at time $t$, which of course is observed by the
investor. The investor rebalances the portfolio at each time $t$, following a
self-financing trading strategy (policy) $\pi$, that may satisfy additional
trading constrains, such as short selling constrains, turn over constraints,
etc. The investor is also allowed to consume at each time $t$ part of the
wealth, say $z_{t}$, that does not exceed $\bar{X}_{t}$. We postulate that the
investor maximizes the expected utility of consumptions and terminal wealth
using the utility functions $V^{\beta}$ and $U^{\gamma}$, respectively, where
$\beta,\gamma\in\mathbb{R}$ stand for risk aversion-parameters. We refer the
reader to [BR17, Section 4] for detailed formulation of this problem in the
MDP framework.

Additionally, we assume that the investor faces the Knightian uncertainty
about the model of the underlying assets, described in terms of a (finite)
parametric set $\boldsymbol{\Lambda}\subset\mathbb{R}^{m}$; see [BCC+19] for
an overview of MDPs under Knightian uncertainty.

Moreover, we suppose that the investor is also uncertain about her risk
aversion parameters
$(\beta,\gamma)\in\boldsymbol{\Gamma}\subset\mathbb{R}^{2}$. We emphasize that
this additional feature of an unknown risk aversion parameter is practically
important. Generally speaking, it is difficult to determine the investor’s
risk aversion parameter, which is well documented in the behavioral finance
literature. This becomes especially relevant in the context of fast-growing
robo-advising industry that typically deals with unsophisticated investors,
and which establishes investor’s risk preferences without human intervention.
At each time $t$, the investor reports through process $Y_{t}$ her subjective
degree of happiness about the performance of her investment. For example, one
can take $Y_{t}$ to be a Bernoulli random variable with $Y_{t}=1$
corresponding to happy and $Y_{t}=0$ meaning unhappy about her investment, and
then follow a similar setup to the clinical trials Example 3.6 and incorporate
the uncertainty about $(\beta,\gamma)$ into the original MDP formulation.

Now we consider the observed state process $X_{t}=(\bar{X}_{t},Y_{t})$, and we
take
$\theta=(\beta,\gamma,\lambda)\in\boldsymbol{\Theta}=\boldsymbol{\Gamma}\times\boldsymbol{\Lambda}$
representing the model uncertainty in this model. Consequently, we define the
cost functionals

$\displaystyle c_{t}$
$\displaystyle=V^{\beta}(z_{t}(\bar{x}_{t},\pi_{t}))+F(y_{t},\pi_{t},\beta,\gamma),\
\ t=1,\ldots,T-1,$ $\displaystyle c_{T}$
$\displaystyle=V^{\beta}(z_{T}(\bar{x}_{T},\pi_{T}))+U^{\gamma}(\bar{x}_{T}),$

where $F$ is a penalty for ‘deviating’ from the true risk-aversion parameters.
Using the expectation as the risk functional (cf. Example 3.15), the problem
(5.1) becomes the optimal investment and consumption problem. Theorem 5.3
gives the solution to this problem. Detailed model specification and analysis
is beyond the scope of this work and it will be addressed in future works.

###  5.3 Optimal control problem corresponding to Example 3.17

We will present the solution to the optimal control problem for the clinical
trials example with the risk-sensitive criterion. Namely, for
$t\in\mathcal{T}$, we consider

$\displaystyle v^{\pi}_{t}(h_{t})$
$\displaystyle=\rho_{t}\Big{(}c_{t}(x_{t},\pi_{t}(h_{t}),\cdot),c_{t+1}(\cdot,\pi_{t+1}(h_{t},\cdot),\cdot),$
$\displaystyle\qquad\qquad
c_{t+2}(\cdot,\pi_{t+2}(h_{t},\cdot,\cdot),\cdot),\cdots,c_{T}(\cdot,\pi_{T}(h_{t},\cdot,\ldots,\cdot),\cdot),P^{\pi^{t,h_{t}}}_{t+1,T}\Big{)}$
$\displaystyle=\frac{1}{\varkappa}\ln
E^{\pi}\left(\exp\left(\varkappa\Big{(}c_{t}(x_{t},\pi_{t}(h_{t}),\Theta)+\sum_{k=t+1}^{T}c_{k}(\widehat{X}_{k},\pi_{k}(h_{t},\widehat{X}_{t+1},\ldots,\widehat{X}_{k}),\Theta)\Big{)}\right)\bigg{|}\widehat{H}_{t}=h_{t}\right)$
(5.17) $\displaystyle=\frac{1}{\varkappa}\ln w^{\pi}_{t}(h_{t}).$ (5.18)

It is clear that problem (5.1) is equivalent to the following problem

$\min_{\pi\in\Pi}w_{1}^{\pi}(h_{1}),$ (5.19)

For $t=T$ we have

$\displaystyle
w^{\pi}_{T}(h_{T})=E^{\pi}\Big{(}\exp\left(\varkappa\,c_{T}(x_{T},\pi_{T}(h_{T}),\Theta)\right)\big{|}\widehat{H}_{T}=h_{T}\Big{)}=\int_{\boldsymbol{\Theta}}e^{\varkappa\,c_{T}(x_{T},\pi_{T}(h_{T}),\theta)}\xi_{T}^{\pi,h_{T}}(d\theta).$
(5.20)

As above, we denote by $(x,\xi)$ an element of the set
$\mathcal{X}\times\mathcal{P}(\boldsymbol{\Theta}).$ Thus, observing that
$\xi^{\pi,h_{T}}_{T}$ does not depend on $\pi_{T}$, and letting $x_{T}=x$ and
$\xi_{T}=\xi$, we compute the candidate optimal quasi-Markov control
$\varphi_{T}$ as

$\varphi_{T}(x,\xi)=\operatorname*{arg\,min}_{u\in\mathcal{U}}\,\int_{\boldsymbol{\Theta}}\,e^{\varkappa\,c_{T}(x,u,\theta)}\,\xi(d\theta).$
(5.21)

We define the Bellman function at time $t=T$ as

$W_{T}(x,\xi)=\min_{u\in\mathcal{U}}\,\int_{\boldsymbol{\Theta}}\,e^{\varkappa\,c_{T}(x,u,\theta)}\,\xi(d\theta)=\int_{\boldsymbol{\Theta}}\,e^{\varkappa\,c_{T}(x,\varphi_{T}(x,\xi),\theta)}\,\xi(d\theta).$
(5.22)

Now, we proceed to time $t=T-1$. Given $x_{T-1}=x$ and $\xi_{T-1}=\xi$ we
compute the candidate optimal quasi-Markov control $\varphi_{T-1}$ as

$\varphi_{T-1}(x,\xi)=\operatorname*{arg\,min}_{u\in\mathcal{U}}\,\int_{\boldsymbol{\Theta}}\int_{\mathcal{X}}e^{\varkappa\,c_{T-1}(x,u,\theta)}W_{T}(x_{T},{\widetilde{\xi}^{\,u,x_{T},\xi}_{T}})K_{\theta}(dx_{T}|x,u)\,\xi(d\theta),$
(5.23)

where $\widetilde{\xi}^{\,u,x_{T},\xi}_{T}$ is given by (5.10). The
corresponding Bellman function is

$\displaystyle W_{T-1}(x,\xi)$
$\displaystyle=\min_{u}\,\int_{\boldsymbol{\Theta}}\int_{\mathcal{X}}e^{\varkappa\,c_{T-1}(x,u,\theta)}W_{T}(x_{T},\widetilde{\xi}^{\,u,x_{T},\xi}_{T})\;K_{\theta}(dx_{T}|x,u)\;\xi(d\theta)$
$\displaystyle=\int_{\boldsymbol{\Theta}}\int_{\mathcal{X}}e^{\varkappa\,c_{T-1}(x,\varphi_{T-1}(x,\xi),\theta)}W_{T}(x_{T},{\widetilde{\xi}^{\,\varphi_{T-1}(x,\xi),x_{T},\xi}_{T}})\;K_{\theta}(dx_{T}|x,\varphi_{T-1}(x,\xi))\;\xi(d\theta).$
(5.24)

Following this pattern, we arrive at the DP backward recursion:

$W_{t}(x,\xi)=\min_{u}\,\int_{\boldsymbol{\Theta}}\int_{\mathcal{X}}e^{\varkappa\,c_{t}(x,u,\theta)}W_{t+1}(x_{t+1},{\widetilde{\xi}^{\,u,x_{t+1},\xi}_{t+1}})\;K_{\theta}(dx_{t+1}|x,u)\;\xi(d\theta),\quad
t\in\mathcal{T},$ (5.25)

where as in the previous example
$\widetilde{\xi}^{u,x_{t+1},\xi}_{t+1}(\\{\theta\\})$ is given by (5.12). and
$W_{T+1}\equiv 1.$ Note that (5.25) is a counterpart of (4.5).

Accordingly, for $t\in\mathcal{T}$, we define the candidate-optimal quasi-
Markov control $\varphi_{t}$ as

$\varphi_{t}(x,\xi)=\arg\min_{u}\,\int_{\boldsymbol{\Theta}}\int_{\mathcal{X}}e^{\varkappa\,c_{t}(x,u,\theta)}W_{t+1}(x_{t+1},{\widetilde{\xi}^{\,u,x_{t+1},\xi}_{t+1}})\;K_{\theta}(dx_{t+1}|x,u)\;\xi(d\theta),$
(5.26)

with $\xi_{1}$ being the given prior distribution for $\Theta$, and
$h_{1}=x_{1}$.

The policy $\pi^{*}$ is defined by analogy to (5.2). The following
verification theorem can proved in a way analogous to the proof of Theorem
5.3, so we skip its proof.

######  Theorem 5.5.

The following hold true

$\min_{\pi\in\Pi}v^{\pi}_{1}(h_{1})=v^{\pi^{*}}_{1}(h_{1})=\frac{1}{\varkappa}\ln\big{(}W_{1}(x_{1},\xi_{1})\big{)}.$

We emphasise that the key DP recursion (5.23) may be written as

$W_{t}(x,\xi)=\min_{u\in\mathcal{U}}\,\widehat{\rho}_{t}\Big{(}\big{\\{}c_{t}(x,u,\theta)+\sigma_{t}\big{(}W_{t+1}(\cdot,{\widetilde{\xi}^{\,u,\cdot,\xi}_{t+1}});K_{\theta}(x,u)\big{)},\theta\in{\boldsymbol{\Theta}}\big{\\}};\xi\Big{)},$

where for a function $f$ on ${\boldsymbol{\Theta}}$,
$\xi\in\mathcal{P}({\boldsymbol{\Theta}})$, and a function $h$ on
$\mathcal{X}$, we have

$\widehat{\rho}_{t}\big{(}\big{\\{}f(\theta),\theta\in{\boldsymbol{\Theta}}\big{\\}};\xi\big{)}=\int_{\boldsymbol{\Theta}}e^{\varkappa
f(\theta)}\xi(d\theta),$

and where

$\sigma_{t}\big{(}h;K_{\theta}(x,u)\big{)}=\frac{1}{\varkappa}\ln\int_{\mathcal{X}}h(x_{t+1})K_{\theta}(dx_{t+1}|x,u).$

###  5.4 Solution of the optimal control problem for general recursive risk
filters

Let $\rho$ be a recursive risk filter, and let

$\displaystyle
v^{\pi}_{t}(h_{t})=\rho_{t}\Big{(}c_{t}(x_{t},\pi_{t}(h_{t}),\cdot),c_{t+1}(\cdot,\pi_{t+1}(h_{t},\cdot),\cdot),c_{t+2}(\cdot,\pi_{t+2}(h_{t},\cdot,\cdot),\cdot),\cdots$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\cdots,c_{T}(\cdot,\pi_{T}(h_{t},\cdot,\ldots,\cdot),\cdot),P^{\pi^{t,h_{t}}}_{t+1,T}\Big{)},\quad
t\in\mathcal{T}.$ (5.27)

Consider the general problem (5.1) with $v^{\pi}_{t}(h_{t})$ as in (5.4).

Using reasoning analogous to the one employed in Sections 5.2 and 5.3 one can
prove the following result, proof of which we omit here.

######  Theorem 5.6.

There exist operators $\widehat{\rho}_{t}$ and $\sigma_{t}$,
$t\in\mathcal{T}$, and a function $V^{*}$ such that for the functions
$v^{*}_{t}$ defined recursively as

$\displaystyle v^{*}_{T+1}(x)$ $\displaystyle=V^{*}(x),\ x\in\mathcal{X},$
$\displaystyle v^{*}_{t}(x,\xi)$
$\displaystyle=\min_{u\in\mathcal{U}}\,\widehat{\rho}_{t}\Big{(}\Big{\\{}c_{t}(x,u,\theta)+\sigma_{t}(v^{*}_{t+1}(\cdot,{\widetilde{\xi}^{\,u,\cdot,\xi}_{t+1}});K_{\theta}(x,u)),\theta\in{\boldsymbol{\Theta}}\Big{\\}};\xi\Big{)},$
$\displaystyle\qquad\quad\qquad\quad t=T-1,\ldots,1,\quad x\in\mathcal{X},\
\xi\in\mathcal{P}(\boldsymbol{\Theta}),$

subject to

$\widetilde{\xi}^{\,u,x^{\prime},\xi}_{t+1}(\theta)=\xi(\theta)\frac{K_{\theta}(x^{\prime}|x,u)}{\int_{{\boldsymbol{\Theta}}}\,K_{\theta}(x^{\prime}|x,u)\,\xi(d\theta)},\quad
t\in\mathcal{T},\ x,x^{\prime}\in\mathcal{X},\
\xi\in\mathcal{P}(\boldsymbol{\Theta}),$

we have that

$\min_{\pi\in\Pi}v^{\pi}_{1}(h_{1})=v^{*}_{1}(x_{1},\xi_{1}).$

Moreover, the policy $\pi^{*}$ defined as in (5.2) and (5.16), with the
$\phi_{t}$’s given as

$\phi_{t}(x,\xi)=\operatorname*{arg\,min}_{u\in\mathcal{U}}\,\widehat{\rho}_{t}\Big{(}\Big{\\{}c_{t}(x,u,\theta)+\sigma_{t}(v^{*}_{t+1}(\cdot,{\widetilde{\xi}^{u,\cdot,\xi}_{t+1}});K_{\theta}(x,u)),\theta\in{\boldsymbol{\Theta}}\Big{\\}};\xi\Big{)},\\\
t=1,\ldots,T-1,\ x\in\mathcal{X},\ \xi\in\mathcal{P}(\boldsymbol{\Theta}),$

is an optimal policy, that is

$\min_{\pi\in\Pi}v^{\pi}_{1}(h_{1})=v^{\pi^{*}}_{1}(h_{1}).$ (5.28)

The form of the operators $\widehat{\rho}_{t}$ and $\sigma_{t}$,
$t\in\mathcal{T}$, depends on the form of $\rho$, and it can be explicitly
written in terms of $\rho$.

## Acknowledgment

The research of Andrzej Ruszczyński has benefited from partial support from
National Science Foundation Award DMS-1907522 and by the Office of Naval
Research Award N00014-21-1-2161. Tomasz R. Bielecki and Igor Cialenco
acknowledge support from the National Science Foundation grant DMS-1907568.

## References

  * [BCC+19] T. R. Bielecki, T. Chen, I. Cialenco, A. Cousin, and M. Jeanblanc.  Adaptive robust control under model uncertainty.  SIAM Journal on Control and Optimization, 57(2):925–946, 2019.
  * [BR14] N. Bäuerle and U. Rieder.  More Risk-Sensitive Markov Decision Processes.  Mathematics of Operations Research, 39(1):105–120, feb 2014.
  * [BR17] N. Bäuerle and U. Rieder.  Zero-sum risk-sensitive stochastic games.  Stochastic Processes and their Applications, 127(2):622 – 642, 2017\.
  * [DL14] M. Davis and S. Lleo.  Risk-Sensitive Investment Management, volume 19 of Advanced Series on Statistical Science & Applied Probability.  World Sci., 2014.
  * [FR18] J. Fan and A. Ruszczyński.  Risk measurement and risk-averse control of partially observable discrete-time Markov systems.  Math. Methods Oper. Res., 88(2):161–184, 2018.
  * [FR22] J. Fan and A. Ruszczyński.  Process-based risk measures and risk-averse control of discrete-time systems.  Math. Program., 191(1, Ser. B):113–140, 2022.
  * [LRZ21] Y. Lin, Y. Ren, and E. Zhou.  A Bayesian risk approach to MDPs with parameter uncertainty.  Preprint arXiv:2106.02558, 2021.
  * [LS20] T. Lattimore and C. Szepesvári.  Bandit algorithms.  Cambridge University Press, 2020.
  * [SB18] R. S. Sutton and A. G. Barto.  Reinforcement learning: An introduction.  MIT press, 2018.