# Policy Gradient Methods for Risk-Sensitive Distributional Reinforcement
Learning with Provable Convergence

Minheng Xiao Xian Yu and Lei Ying Department of Integrated Systems
Engineering, The Ohio State University, Columbus, OH, USA, Email:
xiao.1120@osu.edu;Corresponding author; Department of Integrated Systems
Engineering, The Ohio State University, Columbus, OH, USA, Email:
yu.3610@osu.edu;Department of Electrical Engineering and Computer Science,
University of Michigan, Ann Arbor, MI, USA, Email<EMAIL_ADDRESS>
###### Abstract

Risk-sensitive reinforcement learning (RL) is crucial for maintaining reliable
performance in many high-stakes applications. While most RL methods aim to
learn a point estimate of the random cumulative cost, distributional RL (DRL)
seeks to estimate the entire distribution of it [7]. The distribution provides
all necessary information about the cost and leads to a unified framework for
handling various risk measures in a risk-sensitive setting. However,
developing policy gradient methods for risk-sensitive DRL is inherently more
complex as it pertains to finding the gradient of a probability measure. This
paper introduces a policy gradient method for risk-sensitive DRL with general
coherent risk measures, where we provide an analytical form of the probability
measure’s gradient. We further prove the local convergence of the proposed
algorithm under mild smoothness assumptions. For practical use, we also design
a categorical distributional policy gradient algorithm (CDPG) based on
categorical distributional policy evaluation [7] and trajectory-based gradient
estimation. Through experiments on a stochastic cliff-walking environment, we
illustrate the benefits of considering a risk-sensitive setting in DRL.

##  1 Introduction

In traditional reinforcement learning (RL), the objective often involves
minimizing the expected cumulative cost (or maximizing the expected cumulative
reward) [46]. This type of problems has been extensively studied using value-
based methods [52, 22, 35, 51] and policy gradient methods [53, 47, 26, 41,
30]. However, for intelligent autonomous systems operated in risky and dynamic
environments, such as autonomous driving, healthcare and finance, it is
equally (or more) important to control the risk under various possible
outcomes. To address this, risk-sensitive RL has been developed to ensure more
reliable performance using different objectives and constraints [23, 16, 12,
13, 48, 49]. Ref. [3] proposed a class of risk measures that satisfy several
natural and desirable properties, called coherent risk measures. In Markov
decision processes (MDP), the risk can be measured on the total cumulative
cost or in a nested way, leading to static or dynamic risk measures. While
[34, 2, 10, 9] have recently shown the global convergence of policy gradient
algorithms in a risk-neutral RL framework, the convergence of policy gradient
algorithms in risk-averse RL has been underexplored. Ref. [24] showed that
Markov coherent risk measures (a class of dynamic risk measures) are not
gradient dominated, and thus the stationary points that policy gradient
methods find are not guaranteed to be globally optimal in general. Recently,
[55] showed the global convergence of risk-averse policy gradient algorithms
for a class of dynamic time-consistent risk measures (i.e., expected
conditional risk measures). In this paper, we mainly focus on static coherent
risk measures and provide local convergence guarantees for risk-averse policy
gradient algorithms using a distributional RL (DRL) framework.

Instead of modeling a point estimate of the random cumulative cost, DRL offers
a more comprehensive framework by modeling the entire distribution of it [7,
8]. Along this line, [7] proposed a C51 algorithm that models the cost
distribution as a categorical distribution with fixed atoms and variable
probabilities, and [18] proposed QR-DQN that transposes the parametrization of
C51 by considering fixed probabilities and variable atom locations using
quantile regression. Besides these value-based methods, various distributional
policy gradient methods have also been proposed, such as D4PG [4], DSAC [33],
and SDPG [43, 42], etc. However, recent attempts to apply policy gradient
methods in risk-sensitive DRL have been primarily based on neural network
architectures, which lack rigorous proof of gradient formulas and convergence
guarantees. Different from these papers, our work aims to fill the gap by
providing analytical gradient forms for general coherent risk measures with
convergence guarantees. Specifically, we first utilize distributional policy
evaluation to obtain the random cumulative cost’s distribution under any given
policy. Then, we compute the gradient of the obtained probability measure,
based on which we calculate the policy gradient for a coherent risk measure.
The policy parameter is then updated in the gradient descent direction. Next,
we review the relevant literature in detail and present our main contributions
and major differences with prior work.

#### Prior Work.

There has been a stream of research papers on risk-sensitive RL with different
objectives and constraints, such as optimizing the worst-case scenario [23,
16, 56, 28], optimizing under safety constraints [12, 13, 1, 45, 14, 19, 29],
optimizing a static risk measure [48, 49, 15, 20], or optimizing a dynamic
risk measure [39, 11, 44, 27, 54, 55, 56]. Among them, [15] studied a static
conditional Value-at-Risk (CVaR) objective and presented an approximate value-
iteration algorithm for CVaR MDPs with convergence rate analysis. Ref. [49]
considered a static CVaR objective over a continuous random variable and
proposed a sampling-based estimator for the gradient of the CVaR, where the
authors proved the convergence of a stochastic gradient descent algorithm to a
local CVaR optimum. Ref. [48] provided policy gradients of both static and
dynamic coherent risk measures for discrete random variables and adopted a
similar sample-based policy gradient method (SPG), where the estimator
asymptotically converges to the true gradient when the sample size goes to
infinity.

Recently, another vein of research has focused on finding risk-sensitive
policies using a distributional perspective (i.e., in DRL). Ref. [36] proposed
a method of approximating the return distribution with particle smoothing and
applied it to a risk-sensitive framework with CVaR as the evaluation
criterion. Building on recent advances in DRL [7], [17] extended QR-DQN
proposed in [18] to implicit quantile networks (IQN) that learn the full
quantile function and allow to optimize any distortion risk measures. Ref.
[31] showed that replacing expectation with CVaR in action-selection strategy
when applying the distributional Bellman optimality operator can result in
convergence to neither the optimal dynamic CVaR nor the optimal static CVaR
policies. The authors then proposed a new distributional Bellman optimality
operator and showed that it expanded the utility of DRL in optimizing CVaR
policies. Ref. [25] designed an optimistic version of the distributional
Bellman operator that moved probability mass from the lower to the upper tail
of the return distribution. The authors demonstrated the efficiency of their
algorithm in finding CVaR-optimal policies. Besides these value-based DRL
methods, D4PG [4] and SDPG [42] are two actor-critic type policy gradient
algorithms based on DRL but are focused on optimizing the mean value of the
return. Ref. [43] then extended SDPG to incorporate CVaR in the action network
and proposed a risk-aware SDPG algorithm. Ref. [50] assumed the cumulative
reward to be Gaussian distributed and focused on optimizing policies for CVaR.
They derived the closed-form expression of CVaR-based objective’s gradient and
designed an actor-critic framework. Ref. [37] introduced a policy gradient
framework that utilized reparameterization of the state distribution for end-
to-end optimization of risk-sensitive utility functions in continuous state-
action MDPs. Ref. [33] proposed a distributional soft actor-critic algorithm
(DSAC) that integrated SAC with DRL and provided a unified framework for
learning risk-related metrics such as percentile, mean-variance, and distorted
expectation.

#### Main Contributions of Our Paper and Comparisons with Prior Work.

The main contributions of this paper are three-fold. First, to the best of our
knowledge, this work presents the first distributional policy gradient theorem
(Theorem 2) that computes the gradient of the cumulative cost’s probability
measure. This gradient is useful for constructing the policy gradient of
coherent risk measures. While prior work such as [48, 49] proposed sample-
based approaches to estimate this gradient, our paper provides an analytical
form based on a DRL perspective. Through numerical experiments conducted in
Section 4, our algorithm converges to a safe policy using substantially fewer
samples and iterations, compared to the sample-based approach in [48]. Second,
we propose a general risk-sensitive distributional policy gradient framework,
which can be applied to a broad class of coherent risk measures and combined
with any policy evaluation methods. Specifically, we propose a categorical
distributional policy gradient algorithm (CDPG) based on categorical policy
evaluation and trajectory-based gradient estimation. The CDPG can be seen as a
complement of the value-based C51 method [7], which works well on a stochastic
cliffwalking environment as we report in Section 4. Third, unlike neural
network (NN)-based distributional policy gradient methods such as D4PG [4],
SDPG [42, 43], with the aid of analytical gradient forms, we derive
assumptions under which the proposed risk-averse distributional policy
gradient algorithms converge to a stationary point within a finite number of
iterations. We review the most relevant work on risk-sensitive RL/DRL and list
the major differences compared to our work in Table 1.

Table 1: Relevant work on risk-sensitive RL/DRL and comparisons with our work.

| Objective | Approach | DRL | Convergence
---|---|---|---|---
Our work (CDPG) |  Coherent risk measure
(Static) |  Policy gradient
\+ Analytical gradient forms | ✓ | ✓
[48, 49] (SPG) |  Coherent risk measure
(Static and dynamic) |  Policy gradient
\+ Analytical gradient forms | ✗ | ✓
[12] |  Expectation
with CVaR-constrained |  Policy Gradient
\+ Analytical Gradient Forms | ✗ | ✓
[4] (D4PG) | Expectation | NN-based policy gradient | ✓ | ✗
[43] (SDPG) | Static CVaR | NN-based policy gradient | ✓ | ✗
[33] (DSAC) |  Percentile, variance
and distorted expectation | NN-based policy gradient | ✓ | ✗
[50] (WCPG) |  Static CVaR
\+ Gaussian Reward |  NN-based policy gradient
\+ Analytical gradient forms | ✓ | ✗
[7] (C51) | Expectation | Categorical Q-learning | ✓ | ✓
[17] (IQN) | Distortion risk measure | NN-based Q-learning | ✓ | ✗

##  2 Preliminaries

Let $Z_{\theta}:\Omega\to[C_{\text{min}},C_{\text{max}}]$ be a bounded random
variable defined on a probability space
$(\Omega,{\mathcal{F}},\eta_{\theta})$, where $\Omega\subset{\mathbb{R}}$ is a
compact set of outcomes, ${\mathcal{F}}$ is a $\sigma$-algebra over $\Omega$
representing the set of events we are interested in. We assume the support of
$Z_{\theta}$ to be fixed and its probability measure $\eta_{\theta}$ to be
parameterized by $\theta$, i.e.,
$\eta_{\theta}(A)={\mathbb{P}}\\{\omega\in\Omega:Z(\omega)\in A\\}$ for all
$A\subset{\mathbb{R}}$. Denote $\mathcal{Z}$ as the space of all such random
variables, ${\mathcal{P}}({\mathbb{R}})$ as the space of all probability
measures over ${\mathbb{R}}$, and ${\mathcal{M}}({\mathbb{R}})$ as the space
of all signed measures over ${\mathbb{R}}$. In this paper, we interpret
$Z_{\theta}$ as a random cost, and the smaller the realizations of
$Z_{\theta}$, the better. For $Z,Z^{\prime}\in\mathcal{Z}$, we denote by
$Z\preceq Z^{\prime}$ the pointwise partial order meaning $Z(\omega)\leq
Z^{\prime}(\omega)$ for $\eta$-almost $\omega\in\Omega$. Different from [48],
we consider a nonatomic probability space
$(\Omega,{\mathcal{F}},\eta_{\theta})$ with continuous random variables
$Z_{\theta}$, i.e., $d\eta_{\theta}=f_{Z_{\theta}}(\omega)d\omega$, where
$f_{Z_{\theta}}(\omega)=f_{Z}(\omega,\theta)$ is the (continuous) probability
density function of random variable $Z_{\theta}$. We denote
$F_{Z}(\omega,\theta)$ as the cumulative density function of $Z_{\theta}$. We
will derive the distributional policy gradient results with convergence
guarantees under this general setting and propose a categorical distributional
policy gradient algorithm (CDPG) in Section 3.3 that uses a class of discrete
distributions to approximate any continuous distribution.

Consider a discounted, infinite-horizon MDP, denoted by
$\mathcal{M}=(\mathcal{S},\mathcal{A},P,C,\gamma)$, where $\mathcal{S}$ is a
finite space of states, $\mathcal{A}$ is a finite space of actions,
$P:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S})$ is the
transition kernel, $C(s,a)$ is a deterministic immediate cost given state $s$
and action $a$ 111Our results can be easily extended to random immediate
costs., and $\gamma\in[0,1)$ is the discount factor. We aim to find a
stationary Markov policy
$\pi_{\theta}:\mathcal{S}\rightarrow\Delta(\mathcal{A})$ which is
parameterized by $\theta\in\Theta$222We note that an optimal policy for a
static risk measure is in general history-dependent (neither stationary nor
Markov). However, as shown in [5], by augmenting the state space with the
accumulated cost, a stationary Markov policy is optimal in the augmented state
space for static CVaR objectives.. For a fixed policy $\pi_{\theta}$, with
some abuse of notation, define $Z_{\theta}^{s}$ and $Z_{\theta}^{(s,a)}$ as
the random variables representing the sum of discounted costs along the
trajectory following policy $\pi_{\theta}$ starting from state $s$ and
starting from $(s,a)$, respectively, i.e.,
$Z_{\theta}^{s}=\sum_{t=0}^{\infty}\gamma^{t}C(s_{t},a_{t}),\ s_{t}\sim
P(\cdot|s_{t-1},a_{t-1}),\ a_{t}\sim\pi_{\theta}(\cdot|s_{t}),\ s_{0}=s$ and
$Z_{\theta}^{(s,a)}=\sum_{t=0}^{\infty}\gamma^{t}C(s_{t},a_{t}),\ s_{t}\sim
P(\cdot|s_{t-1},a_{t-1}),\ a_{t}\sim\pi_{\theta}(\cdot|s_{t}),\ s_{0}=s,\
a_{0}=a$. Let $\eta_{\theta}^{s}$ and $\eta_{\theta}^{(s,a)}$ denote their
probability measures, respectively. We make the following assumption
throughout the paper, which is commonly seen in the literature, e.g., in [48,
49].

######  Assumption 1.

For $\eta_{\theta}$-almost all $\omega\in\Omega$, the gradient
$\frac{\partial}{\partial\theta}f_{Z}(\omega,\theta)$ exists and is bounded.

###  2.1 Policy Gradient Methods

In traditional RL, the objective is to minimize the value function defined as
$V_{\theta}(s)=\mathbb{E}_{\pi}[Z_{\theta}^{s}]=\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}C(s_{t},a_{t})|s_{0}=s\right]$.
Similarly, the state-action value function (Q-function) is defined as
$Q_{\theta}(s,a)=\mathbb{E}_{\pi}[Z_{\theta}^{(s,a)}]=\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}C(s_{t},a_{t})\mid
s_{0}=s,a_{0}=a\right]$. The agent’s goal is to find $\theta^{*}$ such that
$\theta^{*}=\arg\min_{\theta\in\Theta}V_{\theta}(s)$. Following first-order
methods in nonlinear optimization, policy gradient algorithms consist of
updating the policy parameter $\theta$ in the gradient descent direction,
i.e., $\theta\leftarrow\theta-\delta\nabla_{\theta}V_{\theta}(s)$, where
$\delta$ is the stepsize (learning rate). The fundamental result underlying
policy gradient algorithms is the policy gradient theorem [47], which provides
the analytical form of $\nabla_{\theta}V_{\theta}(s)$ as follows:

$\displaystyle\nabla_{\theta}V_{\theta}(s)=\sum_{x}d_{\pi}^{s}(x)\sum_{a}\nabla_{\theta}\pi(a|x)\cdot
Q_{\theta}(x,a),$ (1)

where
$d^{s}_{\pi}(x)=\sum_{t=0}^{\infty}\gamma^{t}\text{Pr}(s_{t}=x|s_{0}=s,\pi)$
is the state-visitation distribution.

###  2.2 Coherent Risk Measures

In many situations, an agent would like to manage not only the expected
outcome but also the risk associated with extreme cases or the variability of
costs. A risk measure $\rho:\mathcal{Z}\rightarrow\mathbb{R}$ is a function
that maps a random variable $Z$ to the real line. A risk measure is called
coherent if it satisfies several desirable properties, which we present in
Definition 2, Appendix A.1. The following theorem shows that a coherent risk
measure can be expressed uniquely by a risk envelope.

######  Theorem 1 ([40], Theorem 6.7).

A risk measure $\rho:{\mathcal{Z}}\rightarrow{\mathbb{R}}$ is coherent if and
only if there exists a convex bounded and closed set
${\mathcal{U}}\subset{\mathcal{B}},$ called risk envelope, such that

$\displaystyle\rho(Z_{\theta})=\max_{\xi\in{\mathcal{U}}}{\mathbb{E}}_{\xi}[Z_{\theta}]$
(2)

where
${\mathcal{B}}=\\{\xi:\int_{\Omega}\xi(\omega)f_{Z}(\omega,\theta)d\omega=1,\
\xi\succeq 0\\}$ and
$\mathbb{E}_{\xi}[Z_{\theta}]=\int_{\omega\in\Omega}\xi(\omega)f_{Z}(\omega,\theta)Z(\omega)d\omega$
is the expectation of $Z$ reweighted by $\xi$.

Ref. [48] gives a general form of risk envelope ${\mathcal{U}}$ as follows:

$\displaystyle{\mathcal{U}}=\bigg{\\{}\xi\succeq 0:\
g_{e}(\xi,f_{Z_{\theta}})=0,\ \forall e\in{\mathcal{E}},\
h_{i}(\xi,f_{Z_{\theta}})\leq 0,\ \forall i\in{\mathcal{I}},\
\int_{\omega\in\Omega}\xi(\omega)f_{Z}(\omega,\theta)d\omega=1\bigg{\\}}$

with a set of necessary conditions, which we present in Assumption 2, Appendix
C.4. Under this setting, the Lagrangian function of problem (2) can be written
as
${\mathcal{L}}_{\theta}(\xi,\lambda^{{\mathcal{P}}},\lambda^{\mathcal{E}},\lambda^{\mathcal{I}})=$

$\displaystyle\int_{\Omega}\xi(\omega)f_{Z}(\omega,\theta)Z(\omega)d\omega-\lambda^{{\mathcal{P}}}\left(\int_{\Omega}\xi(\omega)f_{Z}(\omega,\theta)d\omega-1\right)-\sum_{e\in{\mathcal{E}}}\lambda^{\mathcal{E}}(e)g_{e}(\xi,f_{Z_{\theta}})-\sum_{i\in{\mathcal{I}}}\lambda^{\mathcal{I}}(i)h_{i}(\xi,f_{Z_{\theta}}).$
(3)

In this paper, we consider coherent risk measures $\rho$ that have an explicit
form of risk envelope ${\mathcal{U}}$ and saddle points of the Lagrangian
function (3), such as CVaR, mean-semideviation [48], and we aim to solve the
following optimization problem

$\displaystyle\min_{\theta}\rho(Z_{\theta}^{s})$ (4)

where $Z_{\theta}^{s}$ is the random variable representing the sum of
discounted costs along the trajectory following policy $\pi_{\theta}$ starting
from state $s$.

###  2.3 Distributional Reinforcement Learning

Instead of learning the value function, DRL focuses on learning the full
distribution of the random variables $Z_{\theta}^{s}$ and $Z_{\theta}^{(s,a)}$
directly. To facilitate the analysis, we first define a pushforward operator
on the measure space ${\mathcal{M}}({\mathbb{R}})$ as follows:

######  Definition 1.

Given a measure $\nu\in{\mathcal{M}}({\mathbb{R}})$ and a measurable function
$f:{\mathbb{R}}\to{\mathbb{R}}$, the pushforward measure
$f_{\\#}\nu\in{\mathcal{M}}({\mathbb{R}})$ is defined by
$f_{\\#}\nu(A)=\nu(f^{-1}(A))$ for all Borel sets $A\subset{\mathbb{R}}$.

This pushforward operator shifts the support of measure $\nu$ according to the
map $f$. Specifically, we focus on the bootstrap function
$b_{c,\gamma}:{\mathbb{R}}\to{\mathbb{R}}$ defined as
$b_{c,\gamma}(z)=c+\gamma z$.

Given a policy $\pi_{\theta}$, the distributional Bellman operator is defined
as [38]:

$\displaystyle({\mathcal{T}}^{\pi}\eta)^{(s,a)}=\sum_{s^{\prime}\in{\mathcal{S}}}P(s^{\prime}|s,a)\sum_{a^{\prime}\in{\mathcal{A}}}\pi_{\theta}(a^{\prime}|s^{\prime})(b_{C(s,a),\gamma})_{\\#}\eta^{(s^{\prime},a^{\prime})},\quad\forall\eta\in{\mathcal{P}}({\mathbb{R}})^{{\mathcal{S}}\times{\mathcal{A}}}$
(5)

Intriguingly, analogous to the Bellman equation in classical RL, a
distributional Bellman equation also exists in terms of the probability
measure $\eta_{\theta}$. The next lemma is adapted from [38] and we present a
proof in Appendix C.1 for completeness.

######  Lemma 1 ([38], The Distributional Bellman Equation).

For each state $s\in{\mathcal{S}}$ and action $a\in{\mathcal{A}}$, let
$\eta_{\theta}^{s}$ and $\eta_{\theta}^{(s,a)}$ be the probability measure of
random variables $Z_{\theta}^{s}$ and $Z_{\theta}^{(s,a)}$. Then we have

$\displaystyle\eta_{\theta}^{(s,a)}=\sum_{s^{\prime}\in{\mathcal{S}}}P(s^{\prime}|s,a)\sum_{a^{\prime}\in{\mathcal{A}}}\pi_{\theta}(a^{\prime}|s^{\prime})(b_{C(s,a),\gamma})_{\\#}\eta_{\theta}^{(s^{\prime},a^{\prime})}=\sum_{s^{\prime}\in{\mathcal{S}}}P(s^{\prime}|s,a)(b_{C(s,a),\gamma})_{\\#}\eta_{\theta}^{s^{\prime}}$

######  Proposition 1 ([7], Lemma 3, Contraction Property of
${\mathcal{T}}^{\pi}$).

The distributional Bellman operator ${\mathcal{T}}^{\pi}$ defined in Eq. (5)
is a $\gamma$-contraction mapping in the maximal form of the Wasserstein
metric $\bar{d}_{p}$ (see Definition 3 in Appendix A.2) for all $p\geq 1$.
Furthermore, for any initial
$\eta\in{\mathcal{P}}({\mathbb{R}})^{{\mathcal{S}}\times{\mathcal{A}}}$, we
have $({\mathcal{T}}^{\pi})^{m}\eta\to\eta_{\theta}$ in $\bar{d}_{p}$, as
$m\to+\infty$.

##  3 Policy Gradient Methods for Risk-Sensitive Distributional Reinforcement
Learning

In this section, we present a general risk-sensitive distributional policy
gradient framework in Algorithm 1 to solve the optimization problem (4):
$\min_{\theta}\rho(Z_{\theta}^{s})$ for coherent risk measure $\rho$.
Algorithm 1 consists of two steps: (i) Distributional policy evaluation, where
we evaluate the probability measure
$\eta_{\theta}\in{\mathcal{P}}({\mathbb{R}})^{{\mathcal{S}}\times{\mathcal{A}}}$
of $Z_{\theta}$ based on Proposition 1 given any policy $\pi_{\theta}$ and
(ii) Distributional policy improvement, where compute the policy gradient
$\nabla_{\theta}\rho(Z_{\theta}^{s})$ based on $\eta_{\theta}^{s}$ and update
the policy parameter using
$\theta_{t+1}:=\theta_{t}-\delta\cdot\nabla_{\theta}\rho(Z_{\theta_{t}}^{s})$
with $\delta$ being the stepsize (learning rate).

Algorithm 1 Distributional Policy Gradient Algorithm

Initial Parameter $\theta_{0}$, Stepsize $\delta$

for $t=0,1,\dots,T-1$ do

if $\|\nabla_{\theta}\rho(Z_{\theta_{t}}^{s})\|<\epsilon$ then

return $\theta_{t}$

end if

while not converge do

$\eta\leftarrow{\mathcal{T}}^{\theta_{t}}\eta$ # Distributional policy
evaluation

end while

$\theta_{t+1}\leftarrow\theta_{t}-\delta\cdot\nabla_{\theta}\rho(Z_{\theta_{t}}^{s})$
# Distributional policy improvement

end for

return $\theta_{T}$

To this end, we first present a distributional policy gradient theorem
(Theorem 2) that provides the analytical form of
$\nabla_{\theta}\eta_{\theta}^{s}$ in Section 3.1, based on which we compute
$\nabla_{\theta}\rho(Z_{\theta}^{s})$. Then we prove the local convergence of
the distributional policy gradient algorithm (Algorithm 1) in Section 3.2
under mild assumptions. In Section 3.3, we propose a categorical
distributional policy gradient algorithm (CDPG, Algorithm 2) that uses a class
of categorical distributions to approximate any distributions [7]. All the
proofs are presented in Appendix C.

###  3.1 Distributional Policy Gradient Theorem

We first use the distributional Bellman equation in Lemma 1 to derive
$\nabla_{\theta}\eta_{\theta}^{s}$ in the following theorem.

######  Theorem 2 (Distributional Policy Gradient Theorem).

Let
$\eta_{\theta}\in{\mathcal{P}}({\mathbb{R}})^{{\mathcal{S}}\times{\mathcal{A}}}$
be the limiting distribution of the distributional Bellman operator
${\mathcal{T}}^{\pi}$ in Proposition 1. Let $\tau_{\theta}$ be a trajectory
starting from state $s_{0}=s$ and following policy $\pi_{\theta}$ and
$\tau_{\theta}(s_{0},s_{t})$ be the t-step sub-trajectory of $\tau_{\theta}$
from $s_{0}$ to $s_{t}$ for any $t\geq 1$, respectively. Then we have

$\displaystyle\nabla_{\theta}\eta_{\theta}^{s}={\mathbb{E}}_{\tau_{\theta}}\bigg{[}\sum_{a\in{\mathcal{A}}}\nabla_{\theta}\pi_{\theta}(a|s)\cdot\eta_{\theta}^{(s,a)}+\sum_{t=1}^{\infty}{\mathcal{B}}^{\tau_{\theta}(s_{0},s_{t})}\bigg{(}\sum_{a\in{\mathcal{A}}}\nabla_{\theta}\pi_{\theta}(a|s_{t})\cdot\eta_{\theta}^{(s_{t},a)}\bigg{)}\bigg{]}$
(6)

where ${\mathcal{B}}^{\tau_{\theta}(s_{0},s_{t})}$ is the $t$-step pushforward
operator defined by
${\mathcal{B}}^{\tau_{\theta}(s_{0},s_{t})}=(b_{{c_{0}},\gamma})_{\\#}\dots(b_{c_{t-1},\gamma})_{\\#}=(b_{c_{t-1}+\gamma
c_{t-2}+\dots+\gamma^{t-1}c_{0},\gamma^{t}})_{\\#}$ with
$\tau_{\theta}(s_{0},s_{t})=(s_{0}=s,a_{0},c_{0},\dots,s_{t-1},a_{t-1},c_{t-1},s_{t})$.

Once we obtain $\eta_{\theta}$ from the distributional policy evaluation step,
$\nabla_{\theta}\eta_{\theta}^{s}$ can be derived by applying the multi-step
pushforward operators ${\mathcal{B}}^{\tau_{\theta}(s_{0},s_{t})}$ on
$\eta_{\theta}^{(s_{t},a)}$ following Theorem 2. Based on
$\nabla_{\theta}\eta_{\theta}^{s}$, we calculate the gradient of the
probability density function
$\frac{\partial}{\partial\theta}f_{Z^{s}}(x,\theta)$, as shown in the next
corollary.

######  Corollary 1.

If $\nabla_{\theta}\eta_{\theta}^{s}$ is well-defined, then
$\nabla_{\theta}\eta_{\theta}^{s}\in{\mathcal{M}}({\mathbb{R}})$ is a signed
measure with a total mass of 0, i.e.,
$\nabla_{\theta}\eta_{\theta}^{s}(\Omega)=0$. Furthermore, if
$\frac{\partial}{\partial
x}\frac{\partial}{\partial\theta}F_{Z^{s}}(x,\theta)$ and
$\frac{\partial}{\partial\theta}f_{Z^{s}}(x,\theta)$ are continuous, then we
have
$\frac{\partial}{\partial\theta}f_{Z^{s}}(x,\theta)=\frac{\partial}{\partial
x}\nabla_{\theta}\eta_{\theta}^{s}((-\infty,x])$.

Theorem 2 and Corollary 1 are useful for computing
$\nabla_{\theta}\rho(Z_{\theta}^{s})$ as we show in the next theorem.

######  Theorem 3 (Gradient Formula for Static Coherent Risk Measure).

Under Assumptions 1 and 2, for any saddle point
$(\xi_{\theta}^{*},\lambda_{\theta}^{*,{\mathcal{P}}},\lambda_{\theta}^{*,{\mathcal{E}}},\lambda_{\theta}^{*,{\mathcal{I}}})$
of the Lagrangian function (3), we have

$\displaystyle\nabla_{\theta}\rho(Z_{\theta}^{s})=\int_{\Omega}\xi_{\theta}^{*}(\omega)\frac{\partial}{\partial\theta}f_{Z^{s}}(\omega,\theta)(Z^{s}(\omega)-\lambda_{\theta}^{*,\mathcal{P}})d\omega-\sum_{e\in{\mathcal{E}}}\lambda_{\theta}^{*,{\mathcal{E}}}(e)\nabla_{\theta}g_{e}(\xi_{\theta}^{*};f_{Z^{s}_{\theta}})-\sum_{i\in{\mathcal{I}}}\lambda_{\theta}^{*,{\mathcal{I}}}(i)\nabla_{\theta}h_{i}(\xi_{\theta}^{*};f_{Z^{s}_{\theta}}).$
(7)

Note that Theorem 3 also holds for discrete random variables as shown in
Theorem 4.2 in [48], where they use $\nabla_{\theta}p^{\theta}_{i}$ (gradient
of the point mass) instead of
$\frac{\partial}{\partial\theta}f_{Z^{s}}(\omega,\theta)$. We extend their
proof to the continuous random variable case, which is presented in Appendix
C.4. The following gives an example of computing the gradient of a static
coherent risk measure using Theorems 2-3. We also present two other examples
in Appendix A.1.

######  Example 1 (CVaR).

Given a risk level $\alpha\in[0,1]$, the CVaR of a random variable $Z$ is
defined as the $\alpha$-tail expectation, i.e.,
$\rho_{\textit{CVaR}}(Z;\alpha)=\inf_{t\in{\mathbb{R}}}\left\\{t+\frac{1}{\alpha}{\mathbb{E}}[(Z-t)_{+}]\right\\}$.
The risk envelope for CVaR is known to be
${\mathcal{U}}=\\{\xi:\xi(\omega)\in[0,\alpha^{-1}],\int_{\Omega}\xi(\omega)f_{Z}(\omega)d\omega=1\\}$
[40]. Furthermore, [40] showed that the saddle points of Lagrangian function
(3) for CVaR satisfy $\xi_{\theta}^{*}(\omega)=\alpha^{-1}$ when
$Z_{\theta}^{s}(\omega)>\lambda_{\theta}^{*,{\mathcal{P}}}$ and
$\xi_{\theta}^{*}(\omega)=0$ when
$Z_{\theta}^{s}(\omega)<\lambda_{\theta}^{*,{\mathcal{P}}}$, where
$\lambda_{\theta}^{*,{\mathcal{P}}}=q_{\alpha}$ is the $(1-\alpha)$-quantile
of $Z_{\theta}^{s}$. As a result, following Eq. (7), the gradient of CVaR can
be written as

$\displaystyle\nabla_{\theta}\rho_{\textit{CVaR}}(Z_{\theta}^{s};\alpha)$
$\displaystyle=\frac{1}{\alpha}\int_{\Omega}\frac{\partial}{\partial\theta}f_{Z^{s}}(\omega,\theta)\big{(}Z^{s}(\omega)-q_{\alpha}\big{)}\cdot\mathbf{1}_{\\{Z^{s}(\omega)>q_{\alpha}\\}}d\omega$
(8)

Using Algorithm 1, we first evaluate $Z_{\theta}^{s}$ under any given policy
$\pi_{\theta}$. Based on Theorem 2 and Corollary 1, we compute
$\frac{\partial}{\partial\theta}f_{Z^{s}}(\omega,\theta)$ and plug it into Eq.
(8) to get a closed form of
$\nabla_{\theta}\rho_{\textit{CVaR}}(Z_{\theta}^{s};\alpha)$.

######  Remark 1 (Comparison with Classical Policy Gradient Theorem).

Compared to the classical policy gradient theorem (Eq. (1)) with both sides
being real numbers, our Theorem 2 extends it to a measure version, i.e., both
sides of Eq. (6) are signed measures, which provide a richer set of
information on the gradient. Once we get
$\frac{\partial}{\partial\theta}f_{Z^{s}}(x,\theta)$ from Corollary 1, it can
be plugged into Eq. (7) to get the objective’s gradient if we have an explicit
form of the risk envelope and Lagrangian saddle points.

######  Remark 2 (Comparison with Sample-based Policy Gradient Algorithms).

In [48, 49], the authors proposed a sample-average estimator of
$\frac{\partial}{\partial\theta}f_{Z^{s}}(x,\theta)$ by assuming i.i.d.
samples of $Z^{s}_{\theta}$. On the contrary, we obtain a closed form of
$Z^{s}_{\theta}$ and $\eta^{s}_{\theta}$ by Proposition 1 and provide an
analytical form of $\frac{\partial}{\partial\theta}f_{Z^{s}}(x,\theta)$ based
on Theorem 2 and Corollary 1. While we still need to use trajectories to
estimate the expectation in Eq. (6), as demonstrated in Section 4, our method
converges to a safe policy using much fewer samples and iterations, compared
to the sample-based method in [48].

###  3.2 Convergence Guarantees of Distributional Policy Gradient Algorithms

Since Model (4) is a non-convex optimization problem, we aim to search for a
locally optimal $\theta$. In the next theorem, we show that Algorithm 1
converges to a stationary point at rate $O(\frac{1}{\sqrt{T}})$, provided that
the distributional policy evaluation step is exact. We list a set of necessary
conditions in Assumption 3, together with the proof of Theorem 4 in Appendix
C.5.

######  Theorem 4 (Local Convergence Guarantee).

Under Assumptions 1-3, the objective function (4) is $\beta$-smooth.
Furthermore, with exact distributional policy evaluation, if
$\\{\theta_{k}\\}_{k\geq 0}$ is the sequence generated by Algorithm 1 with
stepsize $\delta=\frac{1}{\beta}$, then we have
$\min_{t=0,\dots,T-1}\|\nabla_{\theta}\rho(Z_{\theta_{t}}^{s})\|\leq\frac{\sqrt{2\beta(\rho(Z_{\theta_{0}}^{s})-\rho(Z_{\theta^{*}}^{s}))}}{\sqrt{T}}$.

######  Remark 3.

Note that the iteration complexity provided in Theorem 4 only works when we
have exact distributional policy evaluation. However, if the policy evaluation
step is subject to some error term, we may still be able to derive an
iteration bound for Algorithm 1 based on the error bound of inexact policy
evaluation, which we leave as future work.

###  3.3 Categorical Distributional Policy Gradient Algorithm (CDPG)

From a computational viewpoint, representing a continuous probability
distribution requires an infinite number of parameters, which is intractable.
In this section, we propose an approximate algorithm based on a categorical
representation of distributions [7, 38], named CDPG. Specifically, we
approximate any probability distribution using the parametric family
${\mathcal{P}}_{N}=\left\\{\sum_{i=1}^{N}p_{i}^{\theta}\delta_{z_{i}}\mid
p_{1}^{\theta},\dots,p_{N}^{\theta}\geq
0,\sum_{i=1}^{N}p_{i}^{\theta}=1\right\\}$ with a set of equally-spaced
supports $C_{\min}=z_{1}<\dots<z_{N}=C_{\max}$ over the interval
$[C_{\min},C_{\max}]$. Note that $\\{z_{i}\\}_{i=1}^{N}$ are predefined and
fixed and $\\{p_{i}^{\theta}\\}_{i=1}^{N}$ are parametrized by the policy
parameter $\theta$. Since ${\mathcal{P}}_{N}$ is not closed under the
distributional Bellman operator ${\mathcal{T}}^{\pi}$, i.e.,
${\mathcal{T}}^{\pi}\eta$ may not belong to ${\mathcal{P}}_{N}$ for
$\eta\in{\mathcal{P}}_{N}$, we introduce a projection operator
$\Pi_{\mathcal{C}}:{\mathcal{P}}({\mathbb{R}})\to{\mathcal{P}}_{N}$ defined as
follows

$\displaystyle\Pi_{\mathcal{C}}(\delta_{y})=\begin{cases}\delta_{z_{1}}&\text{if
}y\leq z_{1}\\\
\frac{z_{i+1}-y}{z_{i+1}-z_{i}}\delta_{z_{i}}+\frac{y-z_{i}}{z_{i+1}-z_{i}}\delta_{z_{i+1}}&\text{if
}z_{i}<y\leq z_{i+1}\\\ \delta_{z_{N}}&\text{if }y>z_{N}\end{cases}$ (9)

This projection operator can be extended to a mixture of Diracs, i.e.,
$\Pi_{\mathcal{C}}(\sum_{i=1}^{K}p_{i}\delta_{y_{i}})=\sum_{i=1}^{K}p_{i}\Pi_{\mathcal{C}}(\delta_{y_{i}})$,
as well as to the more general measure space
$\Pi_{\mathcal{C}}:{\mathcal{M}}({\mathbb{R}})\to{\mathcal{M}}_{N}$, where
${\mathcal{M}}_{N}=\\{\sum_{i=1}^{N}p_{i}^{\theta}\delta_{z_{i}}\\}$ is the
$N$-categorical measure space without the simplex requirements. The projected
distributional Bellman operator $\Pi_{\mathcal{C}}{\mathcal{T}}^{\pi}$ is then
defined as

$\displaystyle(\Pi_{\mathcal{C}}{\mathcal{T}}^{\pi}\eta)^{(s,a)}=\Pi_{\mathcal{C}}\left(\sum_{s^{\prime}}P(s^{\prime}|s,a)\sum_{a^{\prime}}\pi(a^{\prime}|s^{\prime})(b_{C(s,a),\gamma})_{\\#}\eta^{(s^{\prime},a^{\prime})}\right),\quad\forall\eta\in{\mathcal{P}}({\mathbb{R}})^{{\mathcal{S}}\times{\mathcal{A}}}$
(10)

The categorical distributional policy evaluation step updates the probability
measure $\eta$ by applying the projected distributional Bellman operator
iteratively: $\eta\leftarrow\Pi_{\mathcal{C}}{\mathcal{T}}^{\pi}\eta$, which
will eventually converge to a limiting distribution (i.e., the fixed point of
$\Pi_{\mathcal{C}}{\mathcal{T}}^{\pi}$) because of the following propositions.

######  Proposition 2 ([38], Proposition 2).

The projected distributional Bellman operator
$\Pi_{\mathcal{C}}{\mathcal{T}}^{\pi}$ defined in Eq. (10) is a contraction
mapping in $\bar{l}_{2}$, where $\bar{l}_{2}$ is the supremum-Cramér distance
(see Definition 4 in Appendix A.3). Furthermore, there exists a unique
distribution
$\eta_{\theta,N}\in{\mathcal{P}}_{N}^{{\mathcal{S}}\times{\mathcal{A}}}$ such
that for any initial distribution
$\eta_{0}\in{\mathcal{P}}({\mathbb{R}})^{{\mathcal{S}}\times{\mathcal{A}}}$,
we have $(\Pi_{\mathcal{C}}{\mathcal{T}}^{\pi})^{m}\eta_{0}\to\eta_{\theta,N}$
in $\bar{l}_{2}$ as $m\to\infty$.

######  Proposition 3 (The Projected Distributional Bellman Equation).

For each state $s\in{\mathcal{S}}$ and action $a\in{\mathcal{A}}$, let
$\eta_{\theta,N}$ be the limiting distribution in Proposition 2, then we have

$\displaystyle\eta_{\theta,N}^{(s,a)}{=}\Pi_{\mathcal{C}}\left(\sum_{s^{\prime}\in{\mathcal{S}}}P(s^{\prime}|s,a)\sum_{a^{\prime}\in{\mathcal{A}}}\pi_{\theta}(a^{\prime}|s^{\prime})(b_{C(s,a),\gamma})_{\\#}\eta_{\theta,N}^{(s^{\prime},a^{\prime})}\right){=}\sum_{s^{\prime}\in{\mathcal{S}}}P(s^{\prime}|s,a)\Pi_{\mathcal{C}}(b_{C(s,a),\gamma})_{\\#}\eta_{\theta,N}^{s^{\prime}}$

We present the detailed steps of the categorical distributional policy
evaluation in Algorithm 3, Appendix 2. Having evaluated the probability
measure $\eta_{\theta,N}$ for any fixed policy $\pi_{\theta}$, the following
theorem shows how to compute the gradient of $\eta_{\theta,N}$ in a
categorical setting:

######  Theorem 5 (Projected Distributional Policy Gradient).

Let $\eta_{\theta,N}\in{\mathcal{P}}_{N}^{{\mathcal{S}}\times{\mathcal{A}}}$
be the limiting distribution of the projected distributional Bellman operator
$\Pi_{\mathcal{C}}{\mathcal{T}}^{\pi}$ in Proposition 2. The gradient of such
a $N$-categorical distribution can be computed by

$\displaystyle\nabla_{\theta}\eta_{\theta,N}^{s}={\mathbb{E}}_{\tau_{\theta}}\bigg{[}\sum_{a}\nabla_{\theta}\pi_{\theta}(a|s)\cdot\eta_{\theta,N}^{(s,a)}+\sum_{t=1}^{\infty}\tilde{{\mathcal{B}}}^{\tau_{\theta}(s_{0},s_{t})}\bigg{(}\sum_{a}\nabla_{\theta}\pi_{\theta}(a|s_{t})\cdot\eta_{\theta,N}^{(s_{t},a)}\bigg{)}\bigg{]}$
(11)

where $\tilde{{\mathcal{B}}}^{\tau_{\theta}(s_{0},s_{t})}$ is the $t$-step
projected pushforward operator defined by
$\tilde{{\mathcal{B}}}^{\tau_{\theta}(s_{0},s_{t})}=\Pi_{\mathcal{C}}(b_{c_{0},\gamma})_{\\#}\dots\Pi_{\mathcal{C}}(b_{c_{t-1},\gamma})_{\\#}$
with
$\tau_{\theta}(s_{0},s_{t})=(s_{0}=s,a_{0},c_{0},\dots,s_{t-1},a_{t-1},c_{t-1},s_{t})$.

######  Remark 4.

In the categorical setting,
$\eta_{\theta,N}^{s}=\sum_{i=1}^{N}p_{i}^{\theta}\delta_{z_{i}}\in{\mathcal{P}}_{N}$
and thus
$\nabla_{\theta}\eta_{\theta,N}^{s}=\sum_{i=1}^{N}\nabla_{\theta}p_{i}^{\theta}\delta_{z_{i}}\in{\mathcal{M}}_{N}$.
To obtain $\nabla_{\theta}p_{i}^{\theta}$, we first compute the right-hand
side of Eq. (11) by applying the projected pushforward operator
$\tilde{{\mathcal{B}}}^{\tau_{\theta}(s_{0},s_{t})}$ on $\eta_{\theta,N}$ and
denote the resulting $N$-categorical measure as
$\sum_{i=1}^{N}q_{i}^{\theta}\delta_{z_{i}}\in{\mathcal{M}}_{N}$. Then we
match the point mass by setting $\nabla_{\theta}p_{i}^{\theta}=q_{i}^{\theta}$
for each $i=1,\ldots,N$. These gradients
$\\{\nabla_{\theta}p_{i}^{\theta}\\}_{i=1}^{N}$ can be further plugged into
Eq. (7) replacing $\frac{\partial}{\partial_{\theta}}f_{Z^{s}}(\omega,\theta)$
to compute the gradient $\nabla_{\theta}\rho(Z_{\theta}^{s})$. The detailed
steps of this CDPG are presented in Algorithm 2.

It is then natural to ask how this limiting distribution $\eta_{\theta,N}$
differs from the true probability measure $\eta_{\theta}$ if we do not use
categorical representation. The next proposition measures the error bound
between $\eta_{\theta,N}$ and $\eta_{\theta}$, which is adapted from [38].

######  Proposition 4 ([38], Proposition 3).

Let $\eta_{\theta,N}$ and $\eta_{\theta}$ be the limiting distribution of
$\Pi_{\mathcal{C}}{\mathcal{T}}^{\pi}$ (in Proposition 2) and
${\mathcal{T}}^{\pi}$ (in Proposition 1), respectively. If
$\eta_{\theta}^{(s,a)}$ is supported on $[z_{1},z_{N}]=[C_{\min},C_{\max}]$
for all $(s,a)\in{\mathcal{S}}\times{\mathcal{A}}$, then we have
$\bar{l}_{2}^{2}(\eta_{\theta,N},\eta_{\theta})\leq\frac{1}{1-\gamma}\max_{1\leq
i\leq N}(z_{i+1}-z_{i})$. Furthermore, when $N\to\infty$, we have
$\eta_{\theta,N}\to\eta_{\theta}$ in $\bar{l}_{2}$.

Proposition 4 implies that as the fineness of the grid ${z_{1},\ldots,z_{N}}$
increases, we gradually recover the true cost distribution and obtain an exact
distributional policy evaluation, which can again guarantee the convergence of
CDPG (Algorithm 2) to locally optimal policies by Theorem 4.

##  4 Numerical Experiment

In this section, we test our CDPG (Algorithm 2) on a stochastic Cliffwalk
environment under risk-averse/risk-neutral settings and compare CDPG with SPG
(non-DRL and sample-based policy gradient) proposed in [48]. The experiments
are conducted on a Macbook Air using Apple M2 chip.

#### Experiment Setup.

Consider a stochastic $3\times 3$ Cliffwalk environment [46] in Figure 1(a),
where an agent starts from the bottom left and aims to reach the bottom right.
Each step incurs a cost of 10, while falling off the cliff results in a cost
of 30 and forces a restart. Assume the state above the cliff is slippery, with
a probability $p=0.2$ of falling off the cliff when entered. The discount
factor is set to $\gamma=0.95$. The shortest path is the red path in Figure
1(a), which takes a cost of 40 if not entering the cliff and incurs a higher
cost otherwise; a safer path is the blue path, which induces a deterministic
cost of 60. We present the detailed steps for computing the discounted average
and CVaR costs in Appendix D. We adopt a softmax parameterization for the
policy, i.e.,
$\pi_{\theta}(a|s)=\frac{\exp(\theta_{a,s})}{\sum_{a^{\prime}\in\mathcal{A}(s)}\exp(\theta_{a^{\prime},s})}$,
and focus on the CVaR objective, where a smaller $\alpha$ represents a more
risk-averse attitude.

(a) (b) (c)

Figure 1: Comparison between risk-averse and risk-neutral policies. Figure (a)
describes environment settings, Figure (b) displays cost distribution, and
Figures (c) and (d) show the average test cost and number of fails with
respect to the number of iterations.

#### Result Comparison between Risk-Averse and Risk-Neutral Settings.

We first compare the performance under risk-averse ($\alpha=0.1$) and risk-
neutral ($\alpha=1$) settings. Since each problem is a non-convex optimization
model, we start from the same set of initial policy parameters $\theta$ with
the same learning rate $\delta=0.3$ and evaluate the test cost on the best
initial $\theta$ for each model. We perform the task over 10 independent
simulation runs and report the cost distribution, average episodic test costs,
and numbers of fails (falling off the cliff) for each setting in Figures
1(b),(c),(d), respectively, where in these figures, colored curves represent
the mean and colored shades represent the standard deviation over the 10
independent runs. As shown in Figure 1(b), the cost distribution for the risk-
neutral policy exhibits a long right tail (with maximum cost being over 100),
showing a high risk and variance. From Figures 1(c) and (d), the risk-averse
model converges to the safe path with a steady cost of 60 and 0 fail within 20
iterations, while the risk-neutral model converges to the shortest path with a
lower cost mean but higher variance as well as a higher number of fails. These
results illustrate the importance of considering a risk-averse setting to
reduce variance and improve safety.

(a) (b) (c) (d)

Figure 2: Comparison between CDPG and SPG [48] algorithms. Figure (a) shows
the divergence to safe path with different sample sizes after 100 iterations.
Figure (b) shows the divergence to safe path w.r.t. iterations. Figures (c)
and (d) show the average test cost and number of fails w.r.t. iterations.

#### Result Comparison between CDPG (Algorithm 2) and SPG.

We fix $\alpha=0.1$ and compare our CDPG (Algorithm 2) with SPG proposed in
[48] while using the same initial $\theta$ and learning rate. All gradients
are normalized to ensure the same length (the only difference lies in the
direction). In Figure 2(a), we vary the sample size from 100 to 5000 and
report the divergence to the safe path after 100 training iterations. From the
figure, our CDPG converges to a close-to-optimal policy even with a small
sample size, whereas the SPG requires a significantly larger sample size for
similar performance. Then we fix the sample size at 100 and report the
divergence to safe path, average test cost, and number of fails with respect
to the iteration number in Figures 2(b), (c), (d), respectively. From these
figures, using the same sample size, our CDPG also converges to a safe policy
faster than SPG and results in fewer failures during learning.

##  5 Conclusion

In this paper, we developed a new policy gradient method for risk-sensitive
MDPs using a distributional perspective, which can be applied to a broad class
of coherent risk measures. Specifically, we first used distributional policy
evaluation to estimate the full distribution of the cumulative cost function.
With the aid of an analytical form of the probability measure’s gradient, we
computed the policy gradients and proposed a distributional policy gradient
algorithm with local convergence guarantees. For practical use, we also used a
categorical distribution to approximate any continuous distribution and
proposed a categorical distributional policy gradient algorithm (CDPG). Our
numerical experiments on a stochastic Cliffwalk environment illustrated the
importance of considering risk aversion and the effectiveness of our approach
compared to a non-DRL sample-based counterpart.

#### Limitations and Future Work.

There are several limitations in the current work, which lead to possible
future directions. First, the current framework only applies to coherent risk
measures with explicit forms of risk envelope and Lagrangian saddle points.
One possible direction is to consider more general risk measures that do not
satisfy these nice properties. Second, we used a categorical representation of
distributions, while several other parametric families (e.g., quantile,
Gaussian representations) are also worth investigating.

## References

  * [1] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel.  Constrained policy optimization.  In International Conference on Machine Learning, pages 22–31. PMLR, 2017.
  * [2] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan.  On the theory of policy gradient methods: Optimality, approximation, and distribution shift.  Journal of Machine Learning Research, 22(98):1–76, 2021.
  * [3] Philippe Artzner, Freddy Delbaen, Jean-Marc Eber, and David Heath.  Coherent measures of risk.  Mathematical Finance, 9(3):203–228, 1999.
  * [4] Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap.  Distributed distributional deterministic policy gradients.  arXiv preprint arXiv:1804.08617, 2018.
  * [5] Nicole Bäuerle and Jonathan Ott.  Markov decision processes with average-Value-at-Risk criteria.  Mathematical Methods of Operations Research, 74:361–379, 2011.
  * [6] Amir Beck.  First-order methods in optimization.  SIAM, 2017.
  * [7] Marc G Bellemare, Will Dabney, and Rémi Munos.  A distributional perspective on reinforcement learning.  In International Conference on Machine Learning, pages 449–458. PMLR, 2017.
  * [8] Marc G Bellemare, Will Dabney, and Mark Rowland.  Distributional reinforcement learning.  MIT Press, 2023.
  * [9] Jalaj Bhandari and Daniel Russo.  Global optimality guarantees for policy gradient methods.  Operations Research, 2024.
  * [10] Shicong Cen, Yuejie Chi, S Du, and Lin Xiao.  Faster last-iterate convergence of policy optimization in zero-sum markov games.  In International Conference on Learning Representations (ICLR), 2023.
  * [11] Yin-Lam Chow and Marco Pavone.  Stochastic optimal control with dynamic, time-consistent risk constraints.  In 2013 American Control Conference, pages 390–395. IEEE, 2013.
  * [12] Yinlam Chow and Mohammad Ghavamzadeh.  Algorithms for CVaR optimization in MDPs.  Advances in Neural Information Processing Systems, 27, 2014.
  * [13] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone.  Risk-constrained reinforcement learning with percentile risk criteria.  Journal of Machine Learning Research, 18(167):1–51, 2018.
  * [14] Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh.  A lyapunov-based approach to safe reinforcement learning.  Advances in Neural Information Processing Systems, 31, 2018.
  * [15] Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone.  Risk-sensitive and robust decision-making: a CVaR optimization approach.  Advances in Neural Information Processing Systems, 28, 2015.
  * [16] Stefano P Coraluppi and Steven I Marcus.  Mixed risk-neutral/minimax control of discrete-time, finite-state Markov decision processes.  IEEE Transactions on Automatic Control, 45(3):528–532, 2000.
  * [17] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos.  Implicit quantile networks for distributional reinforcement learning.  In International Conference on Machine Learning, pages 1096–1105. PMLR, 2018.
  * [18] Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos.  Distributional reinforcement learning with quantile regression.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [19] Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic.  Natural policy gradient primal-dual method for constrained Markov decision processes.  Advances in Neural Information Processing Systems, 33:8378–8390, 2020.
  * [20] Yingjie Fei, Zhuoran Yang, Yudong Chen, Zhaoran Wang, and Qiaomin Xie.  Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret.  Advances in Neural Information Processing Systems, 33:22384–22395, 2020.
  * [21] Gerald B Folland.  Real analysis: modern techniques and their applications, volume 40.  John Wiley & Sons, 1999.
  * [22] Hado Hasselt.  Double Q-learning.  Advances in Neural Information Processing Systems, 23, 2010.
  * [23] Matthias Heger.  Consideration of risk in reinforcement learning.  In Machine Learning Proceedings 1994, pages 105–111. Elsevier, 1994.
  * [24] Audrey Huang, Liu Leqi, Zachary C Lipton, and Kamyar Azizzadenesheli.  On the convergence and optimality of policy gradient for Markov coherent risk.  arXiv preprint arXiv:2103.02827, 2021.
  * [25] Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill.  Being optimistic to be conservative: Quickly learning a CVaR policy.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4436–4443, 2020.
  * [26] Vijay Konda and John Tsitsiklis.  Actor-critic algorithms.  Advances in Neural Information Processing Systems, 12, 1999.
  * [27] Umit Köse and Andrzej Ruszczyński.  Risk-averse learning by temporal difference methods with Markov risk measures.  Journal of Machine Learning Research, 22(38):1–34, 2021.
  * [28] Navdeep Kumar, Esther Derman, Matthieu Geist, Kfir Y Levy, and Shie Mannor.  Policy gradient for rectangular robust Markov decision processes.  Advances in Neural Information Processing Systems, 36, 2024.
  * [29] Prashanth La and Mohammad Ghavamzadeh.  Actor-critic algorithms for risk-sensitive MDPs.  Advances in Neural Information Processing Systems, 26, 2013.
  * [30] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  arXiv preprint arXiv:1509.02971, 2015.
  * [31] Shiau Hong Lim and Ilyas Malik.  Distributional reinforcement learning for risk-sensitive policies.  Advances in Neural Information Processing Systems, 35:30977–30989, 2022.
  * [32] David G Luenberger.  Optimization by Vector Space Methods.  John Wiley & Sons, 1997.
  * [33] Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao.  Dsac: Distributional soft actor critic for risk-sensitive reinforcement learning.  arXiv preprint arXiv:2004.14547, 2020.
  * [34] Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans.  On the global convergence rates of softmax policy gradient methods.  In International Conference on Machine Learning, pages 6820–6829. PMLR, 2020.
  * [35] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  Nature, 518(7540):529–533, 2015.
  * [36] Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka.  Nonparametric return distribution approximation for reinforcement learning.  In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 799–806, 2010.
  * [37] Noah Patton, Jihwan Jeong, Mike Gimelfarb, and Scott Sanner.  A distributional framework for risk-sensitive end-to-end planning in continuous MDPs.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 9894–9901, 2022.
  * [38] Mark Rowland, Marc Bellemare, Will Dabney, Rémi Munos, and Yee Whye Teh.  An analysis of categorical distributional reinforcement learning.  In International Conference on Artificial Intelligence and Statistics, pages 29–37. PMLR, 2018.
  * [39] Andrzej Ruszczyński.  Risk-averse dynamic programming for Markov decision processes.  Mathematical Programming, 125:235–261, 2010.
  * [40] Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszczyński.  Lectures on Stochastic Programming: Modeling and Theory.  SIAM, 2009.
  * [41] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller.  Deterministic policy gradient algorithms.  In International Conference on Machine Learning, pages 387–395. PMLR, 2014.
  * [42] Rahul Singh, Keuntaek Lee, and Yongxin Chen.  Sample-based distributional policy gradient.  In Learning for Dynamics and Control Conference, pages 676–688. PMLR, 2022.
  * [43] Rahul Singh, Qinsheng Zhang, and Yongxin Chen.  Improving robustness via risk averse distributional reinforcement learning.  In Learning for Dynamics and Control, pages 958–968. PMLR, 2020.
  * [44] Sumeet Singh, Yinlam Chow, Anirudha Majumdar, and Marco Pavone.  A framework for time-consistent, risk-sensitive model predictive control: Theory and algorithms.  IEEE Transactions on Automatic Control, 64(7):2905–2912, 2018.
  * [45] Adam Stooke, Joshua Achiam, and Pieter Abbeel.  Responsive safety in reinforcement learning by pid lagrangian methods.  In International Conference on Machine Learning, pages 9133–9143. PMLR, 2020.
  * [46] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [47] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour.  Policy gradient methods for reinforcement learning with function approximation.  Advances in Neural Information Processing Systems, 12, 1999.
  * [48] Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor.  Policy gradient for coherent risk measures.  Advances in Neural Information Processing Systems, 28, 2015.
  * [49] Aviv Tamar, Yonatan Glassner, and Shie Mannor.  Optimizing the CVaR via sampling.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
  * [50] Yichuan Charlie Tang, Jian Zhang, and Ruslan Salakhutdinov.  Worst cases policy gradients.  arXiv preprint arXiv:1911.03618, 2019.
  * [51] Hado Van Hasselt, Arthur Guez, and David Silver.  Deep reinforcement learning with double Q-learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
  * [52] Christopher JCH Watkins and Peter Dayan.  Q-learning.  Machine Learning, 8:279–292, 1992.
  * [53] Ronald J Williams.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  Machine Learning, 8:229–256, 1992.
  * [54] Xian Yu and Siqian Shen.  Risk-averse reinforcement learning via dynamic time-consistent risk measures.  In 2022 IEEE 61st Conference on Decision and Control (CDC), pages 2307–2312. IEEE, 2022.
  * [55] Xian Yu and Lei Ying.  On the global convergence of risk-averse policy gradient methods with expected conditional risk measures.  In International Conference on Machine Learning, pages 40425–40451. PMLR, 2023.
  * [56] Runyu Zhang, Yang Hu, and Na Li.  Regularized robust MDPs and risk-sensitive MDPs: Equivalence, policy gradient, and sample complexity.  arXiv preprint arXiv:2306.11626, 2023.

Appendix

The appendix is organized as follows.

  * •

Appendix A: Omitted Definitions.

  * •

Appendix B: Useful Properties of the Operators.

  * •

Appendix C: Omitted Proofs.

  * •

Appendix D: Numerical Experiment Details.

  * •

Appendix E: Categorical Distributional Policy Gradient Algorithm.

##  Appendix A Omitted Definitions

In this appendix, we provide detailed information on omitted definitions used
in this paper. In Sections A.1-A.3, we provide definitions of coherent risk
measures, Wasserstein and Cramer Distance, respectively. In Sections A.4, we
explain the divergence used in our numerical experiment (Section 4).

###  A.1 Coherent Risk Measures

######  Definition 2.

A risk measure is called coherent if it satisfies the following properties for
all $X,Y\in\mathcal{Z}$ [3]:

  * •

Convexity: $\forall\lambda\in[0,1],\rho(\lambda
X+(1-\lambda)Y)\leq\lambda\rho(X)+(1-\lambda)\rho(Y)$;

  * •

Monotonicity: if $X\preceq Y$, then $\rho(X)\leq\rho(Y)$;

  * •

Translation invariance: $\forall a\in{\mathbb{R}},\rho(X+a)=\rho(X)+a$;

  * •

Positive homogeneity: if $\lambda\geq 0$, then $\rho(\lambda
X)=\lambda\rho(X)$.

######  Example 2 ([48], Expectation).

The gradient of the expectation of random variable $Z_{\theta}$ under policy
$\pi$ with the probability measure $\eta_{\theta}$ is given by

$\displaystyle\nabla_{\theta}{\mathbb{E}}[Z_{\theta}]={\mathbb{E}}\big{[}\nabla_{\theta}\log
f_{Z}(\omega,\theta)Z\big{]}$

Next, we introduce two coherent risk measures with explicit gradient formulas.

######  Example 3 ([48], Mean-Semideviation).

The mean-semideviation of the cost random variable $Z_{\theta}$ with
probability measure $\eta_{\theta}$ at risk level $\alpha\in[0,1]$ is defined
by

$\displaystyle\rho_{\textit{MSD}}(Z_{\theta};\alpha)={\mathbb{E}}[Z_{\theta}]+\alpha\bigg{(}{\mathbb{E}}\big{[}(Z_{\theta}-{\mathbb{E}}[Z_{\theta}])_{+}^{2}\big{]}\bigg{)}^{1/2},$

Then the gradient $\nabla_{\theta}\rho_{\textit{MSD}}(Z_{\theta};\alpha)$ is
given by

$\displaystyle\nabla_{\theta}\rho_{\textit{MSD}}(Z_{\theta};\alpha)=\nabla_{\theta}{\mathbb{E}}[Z_{\theta}]+\frac{\alpha{\mathbb{E}}[(Z-{\mathbb{E}}[Z])_{+}(\nabla_{\theta}\log
f_{Z}(\omega,\theta)(Z-{\mathbb{E}}[Z])-\nabla_{\theta}{\mathbb{E}}[Z])]}{\mathbb{SD}(Z)}$

###  A.2 Wasserstein Metric

######  Definition 3.

The p-Wasserstein distance $d_{p}$ is defined as

$\displaystyle
d_{p}(\nu_{1},\nu_{2})=\bigg{(}\inf_{\lambda\in\Lambda(\nu_{1},\nu_{2})}\int_{{\mathbb{R}}^{2}}|x-y|^{p}\lambda(dx,dy)\bigg{)}^{1/p}$

for all $\nu_{1},\nu_{2}\in{\mathcal{P}}({\mathbb{R}})$, where
$\Lambda(\nu_{1},\nu_{2})$ is the set of probability distributions on
${\mathbb{R}}^{2}$ with marginals $\nu_{1}$ and $\nu_{2}$. The supremum-p-
Wasserstein metric $\bar{d}_{p}$ is defined on
${\mathcal{P}}({\mathbb{R}})^{{\mathcal{S}}\times{\mathcal{A}}}$ by

$\displaystyle\bar{d}_{p}(\eta,\nu)=\sup_{(s,a)\in{\mathcal{S}}\times{\mathcal{A}}}d_{p}\bigg{(}\eta^{(s,a)},\nu^{(s,a)}\bigg{)},$

for all
$\eta,\nu\in{\mathcal{P}}({\mathbb{R}})^{{\mathcal{S}}\times{\mathcal{A}}}$.

###  A.3 Cramér Distance

######  Definition 4.

The Cramér distance $l_{2}$ between two distributions
$\nu_{1},\nu_{2}\in\mathcal{P}(\mathbb{R})$, with cumulative distribution
functions $F_{\nu_{1}}$ and $F_{\nu_{2}}$ respectively, is defined by:

$\displaystyle
l_{2}(\nu_{1},\nu_{2})=\left(\int_{\mathbb{R}}(F_{\nu_{1}}(x)-F_{\nu_{2}}(x))^{2}\,dx\right)^{1/2}.$

Furthermore, the supremum-Cramér metric $\bar{l}_{2}$ is defined between two
distribution functions
$\eta,\mu\in\mathcal{P}(\mathbb{R})^{{\mathcal{S}}\times{\mathcal{A}}}$ by

$\displaystyle\bar{l}_{2}(\eta,\mu)=\sup_{(s,a)\in{\mathcal{S}}\times{\mathcal{A}}}l_{2}(\eta(s,a),\mu(s,a)).$

###  A.4 Divergence in Numerical Experiments (Section 4)

Given a target state trajectory $s=(s_{0},\dots,s_{T})$, the divergence
between two policies $\pi_{1}$ and $\pi_{2}$ is defined as

$\displaystyle{\mathcal{D}}(\pi_{1},\pi_{2})=\sqrt{\sum_{t=0}^{T}\sum_{a\in{\mathcal{A}}}\bigg{|}\pi_{1}(a|s_{t})-\pi_{2}(a|s_{t})\bigg{|}^{2}}$

For instance, $\pi^{*}$ is a specific target policy (e.g., safe path in Figure
1(a)), then ${\mathcal{D}}(\pi^{*},\pi)$ measures the distance from policy
$\pi$ to the target policy $\pi^{*}$.

##  Appendix B Useful Properties of the Operators

In this appendix, we present some useful properties of the pushforward and
projection operators. We first provide the following properties of the
pushforward operator $(b_{c,\gamma})_{\\#}$:

######  Proposition 5.

The pushforward operator $(b_{c,\gamma})_{\\#}$ has the following properties:

  * •

$\nabla_{\theta}(b_{c,\gamma})_{\\#}\eta_{\theta}^{s}=(b_{c,\gamma})_{\\#}\nabla_{\theta}\eta_{\theta}^{s}$
for all $\eta_{\theta}^{s}\in{\mathcal{P}}({\mathbb{R}})$;

  * •

$(b_{c,\gamma})_{\\#}(\sum_{s}p_{s}\eta^{s})=\sum_{s}p_{s}(b_{c,\gamma})_{\\#}\eta^{s}$
for all $\eta^{s}\in{\mathcal{P}}({\mathbb{R}})$ and $p_{s}\in{\mathbb{R}}$.

###### Proof.

Given any set $A\subset{\mathbb{R}}$, by Definition 1, we have

$\displaystyle(b_{c,\gamma})_{\\#}\nabla_{\theta}\eta_{\theta}^{s}(A)=\nabla_{\theta}\eta_{\theta}^{s}[(b_{c,\gamma})^{-1}(A)].$

Similarly, we have

$\displaystyle\nabla_{\theta}(b_{c,\gamma})_{\\#}\eta_{\theta}^{s}(A)=\nabla_{\theta}\left(\eta_{\theta}^{s}[(b_{c,\gamma})^{-1}(A)]\right).$

Hence, we have
$\nabla_{\theta}(b_{c,\gamma})_{\\#}\eta_{\theta}^{s}=(b_{c,\gamma})_{\\#}\nabla_{\theta}\eta_{\theta}^{s}$.
Also, we have

$\displaystyle(b_{c,\gamma})_{\\#}(\sum_{s}p_{s}\eta^{s})(A)$
$\displaystyle=\bigg{(}\sum_{s}p_{s}\eta^{s}\bigg{)}[(b_{c,\gamma})^{-1}(A)]$
$\displaystyle=\sum_{s}p_{s}\eta^{s}[(b_{c,\gamma})^{-1}(A)]=\sum_{s}p_{s}(b_{c,\gamma})_{\\#}\eta^{s}(A)$

This completes the proof. ∎

We then provide the following properties of the projection operator
$\Pi_{\mathcal{C}}$:

######  Proposition 6.

The projected operator $\Pi_{\mathcal{C}}$ has the following properties:

  * •

$\nabla_{\theta}\Pi_{\mathcal{C}}\eta_{\theta}^{s}=\Pi_{\mathcal{C}}\nabla_{\theta}\eta_{\theta}^{s}$
for all $\eta_{\theta}^{s}\in{\mathcal{P}}_{N}$;

  * •

$\Pi_{\mathcal{C}}(\sum_{s}p_{s}\eta^{s})=\sum_{s}p_{s}\Pi_{\mathcal{C}}\eta^{s}$
for all $\eta^{s}\in{\mathcal{P}}_{N}$.

###### Proof.

Assume $\eta_{\theta}^{s}=\sum_{i=1}^{N}P^{\theta}_{i}\delta_{y_{i}}$. Since
$\Pi_{\mathcal{C}}(\sum_{i=1}^{N}P^{\theta}_{i}\delta_{y_{i}})=\sum_{i=1}^{N}P^{\theta}_{i}\Pi_{\mathcal{C}}(\delta_{y_{i}})$,
we have

$\displaystyle\Pi_{\mathcal{C}}\nabla_{\theta}\eta_{\theta}^{s}=\Pi_{\mathcal{C}}\bigg{\\{}\nabla_{\theta}\bigg{(}\sum_{i=1}^{N}P^{\theta}_{i}\delta_{y_{i}}\bigg{)}\bigg{\\}}=\Pi_{\mathcal{C}}\bigg{\\{}\sum_{i=1}^{N}\nabla_{\theta}P^{\theta}_{i}\delta_{y_{i}}\bigg{\\}}=\sum_{i=1}^{N}\nabla_{\theta}P^{\theta}_{i}\Pi_{\mathcal{C}}(\delta_{y_{i}})$

and

$\displaystyle\nabla_{\theta}\Pi_{\mathcal{C}}\eta_{\theta}^{s}=\nabla_{\theta}\Pi_{\mathcal{C}}\bigg{\\{}\sum_{i=1}^{N}P^{\theta}_{i}\delta_{y_{i}}\bigg{\\}}=\nabla_{\theta}\bigg{\\{}\sum_{i=1}^{N}P^{\theta}_{i}\Pi_{\mathcal{C}}(\delta_{y_{i}})\bigg{\\}}=\sum_{i=1}^{N}\nabla_{\theta}P^{\theta}_{i}\Pi_{\mathcal{C}}(\delta_{y_{i}})$

Similarly, let $\eta^{s}=\sum_{i=1}^{N}P_{i}^{s}\delta_{y_{i}}$, then we have

$\displaystyle\Pi_{\mathcal{C}}(\sum_{s}p_{s}\eta^{s})$
$\displaystyle=\Pi_{\mathcal{C}}\bigg{(}\sum_{s}p_{s}\sum_{i=1}^{N}P_{i}^{s}\delta_{y_{i}}\bigg{)}=\sum_{s}\sum_{i=1}^{N}p_{s}P_{i}^{s}\Pi_{\mathcal{C}}(\delta_{y_{i}})$
$\displaystyle=\sum_{s}p_{s}\sum_{i=1}^{N}P^{s}_{i}\Pi_{\mathcal{C}}(\delta_{y_{i}})=\sum_{s}p_{s}\Pi_{\mathcal{C}}\eta^{s}$

∎

Combining Propositions 5 and 6, we get the following properties of projected
pushforward operator $\Pi_{\mathcal{C}}(b_{c,\gamma})_{\\#}$:

######  Proposition 7.

The projected pushforward operator $\Pi_{\mathcal{C}}(b_{c,\gamma})_{\\#}$ has
the following properties:

  * •

$\nabla_{\theta}\Pi_{\mathcal{C}}(b_{c,\gamma})_{\\#}\eta_{\theta}^{s}=\Pi_{\mathcal{C}}(b_{c,\gamma})_{\\#}\nabla_{\theta}\eta_{\theta}^{s}$
for all $\eta_{\theta}^{s}\in{\mathcal{P}}_{N}$;

  * •

$\Pi_{\mathcal{C}}(b_{c,\gamma})_{\\#}(\sum_{s}p_{s}\eta^{s})=\sum_{s}p_{s}\Pi_{\mathcal{C}}(b_{c,\gamma})_{\\#}\eta^{s}$
for all $\eta^{s}\in{\mathcal{P}}_{N}$.

##  Appendix C Omitted Proofs

In this appendix, we present all the omitted proofs.

###  C.1 Proof of Lemma 1

###### Proof of Lemma 1.

Given a deterministic cost function $C(s,a)$, we have

$\displaystyle\eta_{\theta}^{(s,a)}$
$\displaystyle\overset{(i)}{=}({\mathcal{T}}^{\pi}\eta_{\theta})^{(s,a)}$
$\displaystyle\overset{(ii)}{=}\sum_{s^{\prime}\in{\mathcal{S}}}P(s^{\prime}|s,a)\sum_{a^{\prime}\in{\mathcal{A}}}\pi_{\theta}(a^{\prime}|s^{\prime})(b_{C(s,a),\gamma})_{\\#}\eta^{(s^{\prime},a^{\prime})}_{\theta}$
$\displaystyle\overset{(iii)}{=}\sum_{s^{\prime}\in{\mathcal{S}}}P(s^{\prime}|s,a)(b_{C(s,a),\gamma})_{\\#}\eta^{s^{\prime}}_{\theta}$

where $(i)$ is the distributional Bellman equation from [38], $(ii)$ is based
on Eq. (5) and $(iii)$ uses
$\eta_{\theta}^{s}=\sum_{a\in\mathcal{A}}\pi_{\theta}(a|s)\eta_{\theta}^{(s,a)}$
and Proposition 5. ∎

###  C.2 Proof of Theorem 2

###### Proof of Theorem 2.

Denote $g(s)=\sum_{a}\nabla_{\theta}\pi(a|s)\cdot\eta_{\theta}^{(s,a)}$ for
notation simplicity and set $s_{0}=s$, then we have

$\displaystyle\nabla_{\theta}\eta_{\theta}^{s_{0}}$
$\displaystyle\overset{(i)}{=}\nabla_{\theta}\bigg{[}\sum_{a_{0}}\pi(a_{0}|s_{0})\cdot\eta_{\pi}^{(s_{0},a_{0})}\bigg{]}=\sum_{a_{0}}\bigg{[}\nabla_{\theta}\pi(a_{0}|s_{0})\cdot\eta_{\theta}^{(s_{0},a_{0})}+\pi(a_{0}|s_{0})\cdot\nabla_{\theta}\eta_{\theta}^{(s_{0},a_{0})}\bigg{]}$
$\displaystyle\overset{(ii)}{=}\sum_{a_{0}}\bigg{[}\nabla_{\theta}\pi(a_{0}|s_{0})\cdot\eta_{\theta}^{(s_{0},a_{0})}+\pi(a_{0}|s_{0})\cdot\nabla_{\theta}\bigg{(}\sum_{s_{1}}P(s_{1}|s_{0},a_{0})(b_{C(s_{0},a_{0}),\gamma})_{\\#}\eta_{\theta}^{s_{1}}\bigg{)}\bigg{]}$
$\displaystyle\overset{(iii)}{=}g(s_{0})+\sum_{a_{0}}\pi(a_{0}|s_{0})\sum_{s_{1}}P(s_{1}|s_{0},a_{0})(b_{C(s_{0},a_{0}),\gamma})_{\\#}\nabla_{\theta}\eta_{\theta}^{s_{1}}$
$\displaystyle\overset{(iv)}{=}g(s_{0})+\sum_{a_{0}}\pi(a_{0}|s_{0})\sum_{s_{1}}P(s_{1}|s_{0},a_{0})(b_{C(s_{0},a_{0}),\gamma})_{\\#}g(s_{1})$
$\displaystyle\quad+\sum_{a_{0}}\pi(a_{0}|s_{0})\sum_{s_{1}}P(s_{1}|s_{0},a_{0})\sum_{a_{1}}\pi(a_{1}|s_{1})\sum_{s_{2}}P(s_{2}|s_{1},a_{1})\bigg{[}(b_{C(s_{0},a_{0}),\gamma})_{\\#}(b_{C(s_{1},a_{1}),\gamma})_{\\#}\bigg{]}g(s_{2})$
$\displaystyle\quad+\dots\dots$
$\displaystyle\overset{(v)}{=}{\mathbb{E}}_{\tau_{\theta}}\bigg{[}\sum_{a}\nabla_{\theta}\pi(a|s_{0})\cdot\eta_{\theta}^{(s_{0},a)}+\sum_{t=1}^{\infty}{\mathcal{B}}^{\tau_{\theta}(s_{0},s_{t})}\bigg{(}\sum_{a}\nabla\pi(a|s_{t})\cdot\eta_{\theta}^{(s_{t},a)}\bigg{)}\bigg{]},$

where $(i)$ follows because $\eta_{\theta}^{s_{0}}$ is a mixture of
probabilities, $(ii)$ utilizes the distributional Bellman equation (Lemma 1),
$(iii)$ holds because of Proposition 5, and $(iv)$ results from an iterative
expansion of $\nabla_{\theta}\eta_{\theta}^{s}$ with Proposition 5 and $(v)$
holds because each trajectory
$\tau_{\theta}=(s_{0},a_{0},c_{0},s_{1},a_{1},c_{1},\ldots)$ has a probability
of
$\pi(a_{0}|s_{0})P(s_{1}|s_{0},a_{0})\pi(a_{1}|s_{1})P(s_{2}|s_{1},a_{1})\cdots$.
Furthermore, for any two pushforward operators and any measure
$\nu\in{\mathcal{M}}({\mathbb{R}})$, we have

$\displaystyle(b_{c_{0},\gamma})_{\\#}(b_{c_{1},\gamma})_{\\#}\nu(A)$
$\displaystyle=(b_{c_{0},\gamma})_{\\#}\nu(b_{c_{1},\gamma}^{-1}(A))=\nu(b_{c_{0},\gamma}^{-1}(b_{c_{1},\gamma}^{-1}(A)))$
$\displaystyle=$
$\displaystyle\nu((b_{c_{1},\gamma}b_{c_{0},\gamma})^{-1}(A))=(b_{c_{1},\gamma}b_{c_{0},\gamma})_{\\#}\nu(A)=(b_{c_{1}+\gamma
c_{0},\gamma^{2}})_{\\#}\nu(A),\ \forall A\subset{\mathbb{R}}$

Thus, $(b_{c_{0},\gamma})_{\\#}(b_{c_{1},\gamma})_{\\#}=(b_{c_{1}+\gamma
c_{0},\gamma^{2}})_{\\#}$, and the multi-step pushforward operator can be
combined as
${\mathcal{B}}^{\tau_{\theta}(s_{0},s_{t})}=(b_{{c_{0}},\gamma})_{\\#}\dots(b_{c_{t-1},\gamma})_{\\#}=(b_{c_{t-1}+\gamma
c_{t-2}+\dots+\gamma^{t-1}c_{0},\gamma^{t}})_{\\#}$. ∎

###  C.3 Proof of Corollary 1

###### Proof of Corollary 1.

We first show that
$\nabla_{\theta}\eta_{\theta}^{s}=\lim_{\theta_{1}\to\theta_{2}}\frac{\eta_{\theta_{1}}^{s}-\eta_{\theta_{2}}^{s}}{\theta_{1}-\theta_{2}}$
is a signed measure, if it exists. First of all,

$\displaystyle\nabla_{\theta}\eta_{\theta}^{s}(\emptyset)=\lim_{\theta_{1}\to\theta_{2}}\frac{\eta_{\theta_{1}}^{s}-\eta_{\theta_{2}}^{s}}{\theta_{1}-\theta_{2}}(\emptyset)=\lim_{\theta_{1}\to\theta_{2}}\frac{\eta_{\theta_{1}}^{s}(\emptyset)-\eta_{\theta_{2}}^{s}(\emptyset)}{\theta_{1}-\theta_{2}}=0$

Next, we show that it is $\sigma$-additive:

$\displaystyle\nabla_{\theta}\eta_{\theta}^{s}(\cup_{n=1}^{\infty}A_{n})=$
$\displaystyle\lim_{\theta_{1}\to\theta_{2}}\frac{\eta_{\theta_{1}}^{s}-\eta_{\theta_{2}}^{s}}{\theta_{1}-\theta_{2}}(\cup_{n=1}^{\infty}A_{n})=\lim_{\theta_{1}\to\theta_{2}}\frac{\eta_{\theta_{1}}^{s}(\cup_{n=1}^{\infty}A_{n})-\eta_{\theta_{2}}^{s}(\cup_{n=1}^{\infty}A_{n})}{\theta_{1}-\theta_{2}}$
$\displaystyle\overset{(i)}{=}$
$\displaystyle\lim_{\theta_{1}\to\theta_{2}}\sum_{n=1}^{\infty}\frac{\eta_{\theta_{1}}^{s}(A_{n})-\eta_{\theta_{2}}^{s}(A_{n})}{\theta_{1}-\theta_{2}}\overset{(ii)}{=}\sum_{n=1}^{\infty}\lim_{\theta_{1}\to\theta_{2}}\frac{\eta_{\theta_{1}}^{s}(A_{n})-\eta_{\theta_{2}}^{s}(A_{n})}{\theta_{1}-\theta_{2}}$
$\displaystyle=$
$\displaystyle\sum_{n=1}^{\infty}\nabla_{\theta}\eta_{\theta}^{s}(A_{n})$

where $(i)$ is due to the $\sigma$-additivity of $\eta_{\theta_{1}}^{s}$ and
$\eta_{\theta_{2}}^{s}$ and $(ii)$ is because
$\frac{\eta_{\theta_{1}}^{s}(A_{n})-\eta_{\theta_{2}}^{s}(A_{n})}{\theta_{1}-\theta_{2}}$
is bounded. As a result, $\nabla_{\theta}\eta_{\theta}^{s}$ is a measure
(because it satisfies two measure properties) and a signed measure (it can
take values from the real line instead of $[0,1]$). Furthermore, since
$\eta_{\theta}^{s}(\Omega)=1$, we have
$\nabla_{\theta}\eta_{\theta}^{s}(\Omega)=0$, i.e.,
$\nabla_{\theta}\eta_{\theta}^{s}$ has a total mass of 0.

From the definition of probability measure $\eta_{\theta}^{s}$, we have
$\eta_{\theta}^{s}((-\infty,x])={\mathbb{P}}\\{\omega\in\Omega:Z_{\theta}^{s}(\omega)\in(-\infty,x]\\}=F_{Z^{s}}(x,\theta)$.
Taking derivative with respect to $\theta$ on both sides, we have

$\displaystyle\nabla_{\theta}\eta_{\theta}^{s}((-\infty,x])=\frac{\partial}{\partial\theta}F_{Z^{s}}(x,\theta)$
(12)

Now taking the derivative with respect to $x$ again, we have

$\displaystyle\frac{\partial}{\partial
x}\nabla_{\theta}\eta_{\theta}^{s}((-\infty,x])=\frac{\partial}{\partial
x}\frac{\partial}{\partial\theta}F_{Z^{s}}(x,\theta)$

Since $\frac{\partial}{\partial
x}\frac{\partial}{\partial\theta}F_{Z^{s}}(x,\theta)$ and
$\frac{\partial}{\partial\theta}f_{Z^{s}}(x,\theta)$ are continuous, we can
switch the order of partial derivatives and get

$\displaystyle\frac{\partial}{\partial
x}\nabla_{\theta}\eta_{\theta}^{s}((-\infty,x])=\frac{\partial}{\partial
x}\frac{\partial}{\partial\theta}F_{Z^{s}}(x,\theta)=\frac{\partial}{\partial\theta}\frac{\partial}{\partial
x}F_{Z^{s}}(x,\theta)=\frac{\partial}{\partial\theta}f_{Z^{s}}(x,\theta).$

This completes the proof. ∎

###  C.4 Proof of Theorem 3

######  Assumption 2 (The General Form of Risk Envelopes).

For any given policy parameter $\theta\in\Theta$, the risk envelope
${\mathcal{U}}$ of a coherent risk measure can be written as

$\displaystyle{\mathcal{U}}=\bigg{\\{}\xi\succeq 0:\
g_{e}(\xi,f_{Z_{\theta}})=0,\ \forall e\in{\mathcal{E}},\
h_{i}(\xi,f_{Z_{\theta}})\leq 0,\ \forall i\in{\mathcal{I}},\
\int_{\omega\in\Omega}\xi(\omega)f_{Z}(\omega,\theta)d\omega=1\bigg{\\}}$

where each constraint $g_{e}(\xi,f_{Z_{\theta}})$ is an affine function in
$\xi$, each constraint $h_{i}(\xi,f_{Z_{\theta}})$ is a convex function in
$\xi$, and there exists a strictly feasible point $\bar{\xi}$. ${\mathcal{E}}$
and ${\mathcal{I}}$ here denote the sets of equalify and inequality
constraints, respectively. Furthermore, for any given $\xi\in{\mathcal{B}}$,
$h_{i}(\xi,f_{Z_{\theta}})$ and $g_{e}(\xi,f_{Z_{\theta}})$ are twice
differentiable in $f_{Z_{\theta}}$, and there exists a $M>0$ such that for all
$\omega\in\Omega$, we have

$\max\bigg{\\{}\max_{i\in{\mathcal{I}}}\bigg{|}\frac{\partial
h_{i}(\xi,f_{Z_{\theta}})}{\partial
f_{Z}(\omega,\theta)}\bigg{|},\max_{e\in{\mathcal{E}}}\bigg{|}\frac{\partial
g_{e}(\xi,f_{Z_{\theta}})}{\partial
f_{Z}(\omega,\theta)}\bigg{|}\bigg{\\}}\leq M.$

######  Theorem 6 (Differentiation in Measure Theory [21]).

Let $\Theta$ be an open subset of ${\mathbb{R}}$, and $\Omega$ be a measure
space. Suppose $f:\Theta\times\Omega\rightarrow{\mathbb{R}}$ satisfies the
following conditions:

  * (i)

$f(\theta,\omega)$ is a Lebesgue-integrable function of $\omega$ for each
$\theta\in\Theta$.

  * (ii)

For almost all $\omega\in\Omega$, the derivative
$\frac{\partial}{\partial\theta}f(\theta,\omega)$ exists for all
$\theta\in\Theta$.

  * (iii)

There is an integrable function $\Gamma:\Omega\rightarrow{\mathbb{R}}$ such
that $|\frac{\partial}{\partial\theta}f(\theta,\omega)|\leq\Gamma(\omega)$ for
all $\theta\in\Theta$.

Then for all $\theta\in\Theta$,
$\frac{d}{d\theta}\int_{\Omega}f(\theta,\omega)d\omega=\int_{\Omega}\frac{\partial}{\partial\theta}f(\theta,\omega)d\omega$.

###### Proof of Theorem 3.

For continuous random variable $Z_{\theta}$, the Lagrangian function of
problem (2) can be written as

$\displaystyle{\mathcal{L}}_{\theta}(\xi,\lambda^{{\mathcal{P}}},\lambda^{\mathcal{E}},\lambda^{\mathcal{I}})$
$\displaystyle=\int_{\Omega}\xi(\omega)f_{Z}(\omega,\theta)Z(\omega)d\omega-\lambda^{{\mathcal{P}}}\left(\int_{\Omega}\xi(\omega)f_{Z}(\omega,\theta)d\omega-1\right)$
$\displaystyle\quad-\sum_{e\in{\mathcal{E}}}\lambda^{\mathcal{E}}(e)g_{e}(\xi,f_{Z_{\theta}})-\sum_{i\in{\mathcal{I}}}\lambda^{\mathcal{I}}(i)h_{i}(\xi,f_{Z_{\theta}}),$
(13)

which is concave in $\xi$ and convex in
$(\lambda^{{\mathcal{P}}},\lambda^{\mathcal{E}},\lambda^{\mathcal{I}})$. By
Assumption 2 and Theorem 1 in Section 8.6, Page 224 in [32], strong duality
holds, i.e., $\rho(Z_{\theta})=\max_{\xi\geq
0}\min_{\lambda^{{\mathcal{P}}},\lambda^{\mathcal{E}},\lambda^{\mathcal{I}}\geq
0}{\mathcal{L}}_{\theta}(\xi,\lambda^{{\mathcal{P}}},\lambda^{\mathcal{E}},\lambda^{\mathcal{I}})=\min_{\lambda^{{\mathcal{P}}},\lambda^{\mathcal{E}},\lambda^{\mathcal{I}}\geq
0}\max_{\xi\geq
0}{\mathcal{L}}_{\theta}(\xi,\lambda^{{\mathcal{P}}},\lambda^{\mathcal{E}},\lambda^{\mathcal{I}})$.
By Assumption 1, for almost all $\omega\in\Omega$, the gradient of the
probability density function
$\frac{\partial}{\partial\theta}f_{Z}(\omega,\theta)$ exists and is bounded by
a constant for all $\theta\in\Theta$. Since $\Omega$ is a compact set with
finite Lebesgue measure, $\frac{\partial}{\partial\theta}f_{Z}(\omega,\theta)$
is also bounded by an integrable function. Then by Theorem 6, it is guaranteed
that
$\nabla_{\theta}\int_{\Omega}f_{Z}(\omega,\theta)d\omega=\int_{\Omega}\frac{\partial}{\partial\theta}f_{Z}(\omega,\theta)d\omega$.
Hence, by taking derivative with respect to $\theta$ on the both sides of Eq.
(13) at any saddle point
$(\xi^{*}_{\theta},\lambda^{*,{\mathcal{P}}}_{\theta},\lambda^{*,{\mathcal{E}}}_{\theta},\lambda^{*,{\mathcal{I}}}_{\theta})$,
we have

$\displaystyle\nabla_{\theta}{\mathcal{L}}_{\theta}(\xi,\lambda^{{\mathcal{P}}},\lambda^{\mathcal{E}},\lambda^{\mathcal{I}})\bigg{|}_{(\xi^{*}_{\theta},\lambda^{*,{\mathcal{P}}}_{\theta},\lambda^{*,{\mathcal{E}}}_{\theta},\lambda^{*,{\mathcal{I}}}_{\theta})}=$
$\displaystyle\int_{\Omega}\xi_{\theta}^{*}(\omega)\frac{\partial}{\partial\theta}f_{Z}(\omega,\theta)\big{(}Z(\omega)-\lambda_{\theta}^{*,{\mathcal{P}}}\big{)}d\omega$
$\displaystyle-\sum_{e\in{\mathcal{E}}}\lambda^{*,{\mathcal{E}}}_{\theta}(e)\nabla_{\theta}g_{e}(\xi_{\theta}^{*},f_{Z_{\theta}})-\sum_{i\in{\mathcal{I}}}\lambda^{*,{\mathcal{I}}}_{\theta}(i)\nabla_{\theta}h_{i}(\xi_{\theta}^{*},f_{Z_{\theta}})$

The rest follows the same procedure in the proof of Theorem 4.2 in [48]. ∎

###  C.5 Proof of Theorem 4

######  Assumption 3.

Given any static coherent risk measure that satisfies Assumption 2, assume for
almost all $\omega\in\Omega$, the second-order partial derivative of
$f_{Z}(\omega,\theta)$ exists and is bounded, i.e.,
$\|\frac{\partial^{2}}{\partial\theta^{2}}f_{Z}(\omega,\theta)\|_{\infty}\leq
C_{2}$. Additionally, assume the first-order derivatives of Lagrangian
multipliers exist and are bounded for almost all $\omega\in\Omega$ and the
first- and second-order derivatives of the constraint functions exist and are
bounded, i.e.,:

$\displaystyle\|\xi_{\theta}^{*}(\omega)\|_{\infty}\leq C_{3},\
\|\nabla_{\theta}\xi_{\theta}^{*}(\omega)\|_{\infty}\leq C_{4},\ \text{for
almost all $\omega\in\Omega$},$
$\displaystyle\|\lambda_{\theta}^{*,i}\|_{\infty}\leq C_{5},\
\|\nabla_{\theta}\lambda_{\theta}^{*,i}\|_{\infty}\leq C_{6},\ \forall
i\in{\mathcal{I}}\cup{\mathcal{E}}\cup{\mathcal{P}},$
$\displaystyle\|\nabla_{\theta}g_{e}(\xi;f_{Z})\|_{\infty}\leq C_{7},\
\|\nabla_{\theta}^{2}g_{e}(\xi;f_{Z})\|_{\infty}\leq C_{8},\ \forall
e\in{\mathcal{E}},$
$\displaystyle\|\nabla_{\theta}h_{i}(\xi;f_{Z})\|_{\infty}\leq C_{7},\
\|\nabla_{\theta}^{2}h_{i}(\xi;f_{Z})\|_{\infty}\leq C_{8},\ \forall
i\in{\mathcal{I}}$

######  Remark 5.

Assumption 3 is commonly seen in the literature to provide smoothness
guarantees, see, e.g., in [24, 47]. We show that it can be satisfied for a
broad class of coherent risk measures, such as Conditional Value-at-Risk
(CVaR). As have been seen in Example 1, for CVaR,
$\xi_{\theta}^{*}(\omega)=\alpha^{-1}$ when
$Z_{\theta}(\omega)>\lambda_{\theta}^{*,{\mathcal{P}}}$ and
$\xi_{\theta}^{*}(\omega)=0$ when
$Z_{\theta}(\omega)<\lambda_{\theta}^{*,{\mathcal{P}}}$, and
$\lambda_{\theta}^{*,{\mathcal{P}}}=q_{\alpha}$. The risk envelope of CVaR
does not involve $g_{e}$ and $f_{i}$. It is obvious that $\xi_{\theta}^{*}$ is
bounded. If
$\\{\omega\in\Omega:Z_{\theta}(\omega)=\lambda_{\theta}^{*,{\mathcal{P}}}\\}$
has a measure of 0, then $\xi_{\theta}^{*}$ is differentiable and the gradient
is bounded for almost all $\omega\in\Omega$. Since we already assume that
$Z_{\theta}$ is a continuous random variable, then
$\lambda_{\theta}^{*,{\mathcal{P}}}=q_{\alpha}$ is also bounded and
differentiable with bounded gradient by [49].

######  Theorem 7 ([6], Theorem 10.15).

Suppose that $f:{\mathbb{R}}^{d}\rightarrow(-\infty,\infty)$ is proper and
closed, dom($f$) is convex and $f$ is $\beta$-smooth over $int(dom(f))$. Let
$\\{x_{t}\\}_{t\geq 0}$ be the sequence generated by the gradient descent
algorithm with stepsize $\eta=1/\beta$ such that $x_{t+1}=x_{t}-\eta\nabla
f(x_{t})$, then we have

  * •

The sequence $\\{f(x_{t})\\}_{t\geq 0}$ is non-increasing;

  * •

$\nabla f(x_{t})\rightarrow 0$ as $t\rightarrow\infty$;

  * •

$\min_{t=0,1,\dots,T-1}\|\nabla
f(x_{t})\|\leq\frac{\sqrt{2\beta(f(x_{0})-f(x^{*}))}}{\sqrt{T}}$.

###### Proof of Theorem 4.

By Theorem 3, for any saddle point
$(\xi_{\theta}^{*},\lambda_{\theta}^{*,{\mathcal{P}}},\lambda_{\theta}^{*,{\mathcal{E}}},\lambda_{\theta}^{*,{\mathcal{I}}})$
of the Lagrangian function (3), the gradient of the coherent risk measure
$\rho$ is

$\displaystyle\nabla_{\theta}\rho(Z_{\theta}^{s})=\int_{\Omega}\xi_{\theta}^{*}(\omega)\frac{\partial}{\partial\theta}f_{Z^{s}}(\omega,\theta)(Z^{s}(\omega)-\lambda_{\theta}^{*,\mathcal{P}})d\omega-\sum_{e\in{\mathcal{E}}}\lambda_{\theta}^{*,{\mathcal{E}}}(e)\nabla_{\theta}g_{e}(\xi_{\theta}^{*};f_{Z^{s}_{\theta}})-\sum_{i\in{\mathcal{I}}}\lambda_{\theta}^{*,{\mathcal{I}}}(i)\nabla_{\theta}h_{i}(\xi_{\theta}^{*};f_{Z^{s}_{\theta}}).$

Hence, we have

$\displaystyle\nabla_{\theta}\int_{\Omega}\xi_{\theta}^{*}(\omega)\frac{\partial}{\partial\theta}f_{Z^{s}}(\omega,\theta)(Z^{s}(\omega)-\lambda_{\theta}^{*,\mathcal{P}})d\omega$
$\displaystyle\overset{(i)}{=}\int_{\Omega}\frac{\partial}{\partial\theta}\bigg{[}\xi_{\theta}^{*}(\omega)\frac{\partial}{\partial\theta}f_{Z^{s}}(\omega,\theta)\big{(}Z^{s}(\omega)-\lambda_{\theta}^{*,{\mathcal{P}}}\big{)}\bigg{]}d\omega$

where $(i)$ is due to Theorem 6 and Assumption 3. Specifically, we have

$\displaystyle\frac{\partial}{\partial\theta}\bigg{[}\xi_{\theta}^{*}(\omega)\frac{\partial}{\partial\theta}f_{Z^{s}}(\omega,\theta)\big{(}Z^{s}(\omega)-\lambda_{\theta}^{*,{\mathcal{P}}}\big{)}\bigg{]}$
$\displaystyle=\underbrace{\frac{\partial}{\partial\theta}\xi_{\theta}^{*}(\omega)\frac{\partial}{\partial\theta}f_{Z^{s}}(\omega,\theta)\big{(}Z^{s}(\omega)-\lambda_{\theta}^{*,{\mathcal{P}}}\big{)}}_{(a)}+\underbrace{\xi_{\theta}^{*}(\omega)\frac{\partial^{2}}{\partial\theta^{2}}f_{Z^{s}}(\omega,\theta)\big{(}Z^{s}(\omega)-\lambda_{\theta}^{*,{\mathcal{P}}}\big{)}}_{(b)}-\underbrace{\xi_{\theta}^{*}(\omega)\frac{\partial}{\partial\theta}f_{Z^{s}}(\omega,\theta)\frac{\partial}{\partial\theta}\lambda_{\theta}^{*,{\mathcal{P}}}}_{(c)}$

For $(a)$, we have for almost all $\omega\in\Omega$,

$\displaystyle\bigg{\|}\frac{\partial}{\partial\theta}\xi_{\theta}^{*}(\omega)\frac{\partial}{\partial\theta}f_{Z^{s}}(\omega,\theta)\big{(}Z^{s}(\omega)-\lambda_{\theta}^{*,{\mathcal{P}}}\big{)}\bigg{\|}_{\infty}\leq
C_{1}C_{4}|C_{\text{max}}-\lambda_{\theta}^{*,{\mathcal{P}}}|$

where
$\bigg{\|}\frac{\partial}{\partial\theta}f_{Z^{s}}(\omega,\theta)\bigg{\|}_{\infty}\leq
C_{1}$ is due to Assumption 1. For (b), we have for almost all
$\omega\in\Omega$,

$\displaystyle\bigg{\|}\xi_{\theta}^{*}(\omega)\frac{\partial^{2}}{\partial\theta^{2}}f_{Z^{s}}(\omega,\theta)\big{(}Z^{s}(\omega)-\lambda_{\theta}^{*,{\mathcal{P}}}\big{)}\bigg{\|}_{\infty}\leq
C_{2}C_{3}|C_{\text{max}}-\lambda_{\theta}^{*,{\mathcal{P}}}|$

and for (c), we have for almost all $\omega\in\Omega$,

$\displaystyle\bigg{\|}\xi_{\theta}^{*}(\omega)\frac{\partial}{\partial\theta}f_{Z^{s}}(\omega,\theta)\frac{\partial}{\partial\theta}\lambda_{\theta}^{*,{\mathcal{P}}}\bigg{\|}_{\infty}\leq
C_{1}C_{3}C_{6}$

Since
$\frac{\partial}{\partial\theta}\bigg{[}\xi_{\theta}^{*}(\omega)\frac{\partial}{\partial\theta}f_{Z^{s}}(\omega,\theta)\big{(}Z^{s}(\omega)-\lambda_{\theta}^{*,{\mathcal{P}}}\big{)}\bigg{]}$
is bounded by a constant for almost all $\omega\in\Omega$ and $\Omega$ is a
compact set with finite Lebesgue measure ($m(\Omega)<+\infty$), we can switch
the derivative and integral in $(i)$ and furthermore,
$\nabla_{\theta}\int_{\Omega}\xi_{\theta}^{*}(\omega)\frac{\partial}{\partial\theta}f_{Z^{s}}(\omega,\theta)(Z^{s}(\omega)-\lambda_{\theta}^{*,\mathcal{P}})d\omega$
is bounded by
$(C_{1}C_{4}|C_{\text{max}}-\lambda_{\theta}^{*,{\mathcal{P}}}|+C_{2}C_{3}|C_{\text{max}}-\lambda_{\theta}^{*,{\mathcal{P}}}|+C_{1}C_{3}C_{6})m(\Omega)$.
For constraint functions, we have

$\displaystyle\bigg{\|}\nabla_{\theta}\bigg{(}\sum_{e\in{\mathcal{E}}}\lambda_{\theta}^{*,{\mathcal{E}}}(e)\nabla_{\theta}g_{e}(\xi_{\theta}^{*};f_{Z^{s}_{\theta}})\bigg{)}\bigg{\|}_{\infty}=\bigg{\|}\sum_{e\in{\mathcal{E}}}\bigg{(}\nabla_{\theta}\lambda_{\theta}^{*,{\mathcal{E}}}(e)\nabla_{\theta}g_{e}(\xi_{\theta}^{*};f_{Z^{s}_{\theta}})+\lambda_{\theta}^{*,{\mathcal{E}}}(e)\nabla^{2}_{\theta}g_{e}(\xi_{\theta}^{*};f_{Z^{s}_{\theta}})\bigg{)}\bigg{\|}_{\infty}$
$\displaystyle\quad\leq|{\mathcal{E}}|\cdot\|\nabla_{\theta}\lambda_{\theta}^{*,{\mathcal{E}}}(e)\|_{\infty}\|\nabla_{\theta}g_{e}(\xi_{\theta}^{*};f_{Z^{s}_{\theta}})\|_{\infty}+|{\mathcal{E}}|\cdot\
\|\lambda_{\theta}^{*,{\mathcal{E}}}(e)\|_{\infty}\|\nabla_{\theta}^{2}g_{e}(\xi_{\theta}^{*};f_{Z^{s}_{\theta}})\|_{\infty}\leq(C_{6}C_{7}+C_{5}C_{8})|{\mathcal{E}}|$

and similarly,

$\displaystyle\bigg{\|}\nabla_{\theta}\big{(}\sum_{i\in{\mathcal{I}}}\lambda_{\theta}^{*,{\mathcal{I}}}(i)\nabla_{\theta}h_{i}(\xi_{\theta}^{*};f_{Z^{s}_{\theta}})\big{)}\bigg{\|}_{\infty}\leq(C_{6}C_{7}+C_{5}C_{8})|{\mathcal{I}}|$

Overall, we have

$\displaystyle\|\nabla_{\theta}^{2}\rho(Z^{s}_{\theta})\|\leq((C_{1}C_{4}+C_{2}C_{3})|C_{\text{max}}-\lambda_{\theta}^{*,{\mathcal{P}}}|+C_{1}C_{3}C_{6})m(\Omega)+(|{\mathcal{E}}|+|{\mathcal{I}}|)(C_{6}C_{7}+C_{5}C_{8})=\beta$

and hence, $\rho(Z^{s}_{\theta})$ is a $\beta$-smooth function. The remaining
proof follows from the standard gradient descent theorem (Theorem 7). ∎

###  C.6 Proof of Proposition 3

###### Proof of Proposition 3.

We have

$\displaystyle\eta_{\theta,N}^{(s,a)}$
$\displaystyle\overset{(i)}{=}\Pi_{\mathcal{C}}(\sum_{s^{\prime}\in{\mathcal{S}}}P(s^{\prime}|s,a)\sum_{a^{\prime}\in{\mathcal{A}}}\pi_{\theta}(a^{\prime}|s^{\prime})(b_{C(s,a),\gamma})_{\\#}\eta_{\theta,N}^{(s^{\prime},a^{\prime})})$
$\displaystyle\overset{(ii)}{=}\sum_{s^{\prime}\in{\mathcal{S}}}P(s^{\prime}|s,a)\Pi_{\mathcal{C}}(\sum_{a^{\prime}\in{\mathcal{A}}}\pi_{\theta}(a^{\prime}|s^{\prime})(b_{C(s,a),\gamma})_{\\#}\eta_{\theta,N}^{(s^{\prime},a^{\prime})})$
$\displaystyle\overset{(iii)}{=}\sum_{s^{\prime}\in{\mathcal{S}}}P(s^{\prime}|s,a)\Pi_{\mathcal{C}}((b_{C(s,a),\gamma})_{\\#}\eta_{\theta,N}^{s^{\prime}})$

where $(i)$ is because $\eta_{\theta,N}$ is the fixed point of
$\Pi_{\mathcal{C}}{\mathcal{T}}^{\pi}$ [38]; $(ii)$ holds due to Proposition
6; and $(iii)$ follows from Proposition 5 and
$\sum_{a^{\prime}\in{\mathcal{A}}}\pi_{\theta}(a^{\prime}|s^{\prime})\eta_{\theta,N}^{(s^{\prime},a^{\prime})}=\eta_{\theta,N}^{s^{\prime}}$.
∎

###  C.7 Proof of Theorem 5

###### Proof of Theorem 5.

Denote
$\tilde{g}(s)=\sum_{a}\nabla_{\theta}\pi_{\theta}(a|s)\cdot\eta_{\theta,N}^{(s,a)}$
for notation simplicity and set $s_{0}=s$, then we have

$\displaystyle\nabla_{\theta}\eta_{\theta,N}^{s_{0}}=\nabla_{\theta}\bigg{[}\sum_{a_{0}}\pi_{\theta}(a_{0}|s_{0})\cdot\eta_{\theta,N}^{(s_{0},a_{0})}\bigg{]}=\sum_{a_{0}}\bigg{[}\nabla_{\theta}\pi_{\theta}(a_{0}|s_{0})\cdot\eta_{\theta,N}^{(s_{0},a_{0})}+\pi_{\theta}(a_{0}|s_{0})\cdot\nabla_{\theta}\eta_{\theta,N}^{(s_{0},a_{0})}\bigg{]}$
$\displaystyle\overset{(i)}{=}\sum_{a_{0}}\bigg{[}\nabla_{\theta}\pi_{\theta}(a_{0}|s_{0})\cdot\eta_{\theta,N}^{(s_{0},a_{0})}+\pi_{\theta}(a_{0}|s_{0})\cdot\nabla_{\theta}\bigg{(}\sum_{s_{1}}P(s_{1}|s_{0},a_{0})\Pi_{\mathcal{C}}(b_{C(s_{0},a_{0}),\gamma})_{\\#}\eta_{\theta,N}^{s_{1}}\bigg{)}\bigg{]}$
$\displaystyle\overset{(ii)}{=}\tilde{g}(s_{0})+\sum_{a_{0}}\pi_{\theta}(a_{0}|s_{0})\sum_{s_{1}}P(s_{1}|s_{0},a_{0})\Pi_{\mathcal{C}}(b_{C(s_{0},a_{0}),\gamma})_{\\#}\nabla_{\theta}\eta_{\theta,N}^{s_{1}}$
$\displaystyle\overset{(iii)}{=}\tilde{g}(s_{0})+\sum_{a_{0}}\pi_{\theta}(a_{0}|s_{0})\sum_{s_{1}}P(s_{1}|s_{0},a_{0})\Pi_{\mathcal{C}}(b_{C(s_{0},a_{0}),\gamma})_{\\#}\tilde{g}(s_{1})$
$\displaystyle\quad+\sum_{a_{0}}\pi_{\theta}(a_{0}|s_{0})\sum_{s_{1}}P(s_{1}|s_{0},a_{0})\sum_{a_{1}}\pi_{\theta}(a_{1}|s_{1})\sum_{s_{2}}P(s_{2}|s_{1},a_{1})\Pi_{\mathcal{C}}(b_{C(s_{0},a_{0}),\gamma})_{\\#}\Pi_{\mathcal{C}}(b_{C(s_{1},a_{1}),\gamma})_{\\#}\tilde{g}(s_{2})$
$\displaystyle\quad+\dots\dots$
$\displaystyle\overset{(iv)}{=}{\mathbb{E}}_{\tau_{\theta}}\bigg{[}\sum_{a}\nabla_{\theta}\pi_{\theta}(a|s_{0})\cdot\eta_{\theta,N}^{(s_{0},a)}+\sum_{t=1}^{\infty}\tilde{{\mathcal{B}}}^{\tau_{\theta}(s_{0},s_{t})}\bigg{(}\sum_{a}\nabla\pi_{\theta}(a|s_{t})\cdot\eta_{\theta,N}^{(s_{t},a)}\bigg{)}\bigg{]}.$

where $(i)$ is due to the projected distributional Bellman equation
(Proposition 3); $(ii)$ is due to Proposition 7; $(iii)$ results from an
iterative expansion of $\nabla_{\theta}\eta_{\theta,N}^{s_{1}}$ with
Proposition 7 and $(iv)$ holds because each trajectory
$\tau_{\theta}=(s_{0},a_{0},c_{0},s_{1},a_{1},c_{1},\ldots)$ has a probability
of
$\pi(a_{0}|s_{0})P(s_{1}|s_{0},a_{0})\pi(a_{1}|s_{1})P(s_{2}|s_{1},a_{1})\cdots$.
∎

##  Appendix D Numerical Experiment Details

In this appendix, we compute the expectation and CVaR cost of the shortest and
safe paths. In our environment, the discounted factor $\gamma=0.95$, the
probability of dropping off the cliff is $p=0.2$, the cost of dropping off the
cliff is $x=30$, and the cost of each step is $c=10$. The expected cost of the
shortest path from the starting state $6$ can be computed by solving the
following Bellman equation:

$\displaystyle v_{6}$ $\displaystyle=c+\gamma v_{3}$ $\displaystyle v_{3}$
$\displaystyle=p(x+\gamma v_{6})+(1-p)(c+\gamma v_{4})$ $\displaystyle v_{4}$
$\displaystyle=c+\gamma v_{5}$ $\displaystyle v_{5}$ $\displaystyle=c+\gamma
v_{8}$ $\displaystyle v_{8}$ $\displaystyle=0$

Thus, the expected cost of the shortest path is $v_{6}=45.61$, and the
expected cost of the safe path is
$60(\gamma^{0}+\gamma^{1}+\dots+\gamma^{5})=52.98$. As for the CVaR, for the
shortest path, the CVaR is greater than 74.14 and for the safe path, the CVaR
is exactly 52.98. Thus, our risk-averse policy should choose the safe path
(with a lower CVaR) and the risk-neutral policy should choose the shortest
path (with a lower expectation), which agrees with our numerical results in
Section 4.

Figure 3: Cost distribution of the shortest (red) and safe (blue) path.

##  Appendix E Categorical Distributional Policy Gradient Algorithm

In this appendix, we present the details of the categorical distributional
policy evaluation in Algorithm 3 and the categorical distributional policy
gradient algorithm (CDPG) in Algorithm 2.

Algorithm 2 Categorical Distributional Policy Gradient (CDPG)

initial policy parameter $\theta_{0}$, learning rate $\delta$, number of
training iterations $T$, discount factor $\gamma$, cost bound
$C_{\text{min}},C_{\text{max}}$, number of atoms $N$, number of trajectories
$M$

for $t=0,\dots,T-1$ do

### Categorical distributional policy evaluation and collect trajectories

For $\theta_{t}$, get $\eta_{\theta_{t},N}\text{ and
}\\{\tau_{m}\\}_{m=1}^{M}$ following Algorithm 3

### Policy gradient estimation

$g\leftarrow 0$ # initialize the gradient

for $m=1,\dots,M$ do

$g_{m}\leftarrow 0$ # initial the gradient from one trajectory

for $k=0,1,\dots,K={\\#}_{s}(\tau_{m})$ do # $\\#_{s}(\tau_{m})$ is the number
of state visited in trajectory $\tau_{m}$

Compute
$g(s_{k})=\sum_{a}\nabla_{\theta}\pi_{\theta_{t}}(a|s_{k})\cdot\eta_{\theta_{t},N}^{(s_{k},a)}$

$g_{m}\leftarrow
g_{m}+\tilde{{\mathcal{B}}}^{\tau_{\theta_{t}}(s_{0},s_{k})}(g(s_{k}))$ #
$\tilde{{\mathcal{B}}}^{\tau_{\theta_{t}}}$ is the projected pushforward
operator in Theorem 5

end for

$g\leftarrow g+g_{m}/M$

$\nabla_{\theta}\eta_{\theta_{t}}\leftarrow g$

end for

Compute $\nabla_{\theta}\rho(Z_{\theta_{t}})$ following Eq. (7) and Remark 4.

if $\|\nabla_{\theta}\rho(Z_{\theta_{t}})\|<\epsilon$ then

return $\theta_{t}$

else

### Policy improvement

$\theta_{t+1}\leftarrow\theta_{t}-\delta\nabla_{\theta}\rho(Z_{\theta_{t}})$

end if

end for

return $\theta_{T}$

Algorithm 3 Categorical Distributional Policy Evaluation

Policy parameter $\theta$, discount factor $\gamma$, cost bound
$C_{\text{min}},C_{\text{max}}$, number of atoms $N$, number of trajectories
$M$

$\Delta z\leftarrow(C_{\text{max}}-C_{\text{min}})/(N-1)$

$z_{j}\leftarrow C_{\text{min}}+j*\Delta z,\ j=0,\dots,N-1$ # atom locations

Initialize
$\eta_{\theta,N}^{(s,a)}=\sum_{j=0}^{N-1}m_{j}^{(s,a)}\delta_{z_{j}},\
\forall(s,a)\in{\mathcal{S}}\times{\mathcal{A}}$ # categorical distribution on
N atoms

for $m=1,\dots,M$ do

Start an episode, observe the initial state $s$

$\tau_{m}\leftarrow[\ ]$ # record the trajectory

while episode not terminated do

Execute an action $a\sim\pi(\cdot|s)$, observe $c,s^{\prime}$

$\tau\leftarrow\tau+[s,a,c,s^{\prime}]$

Sample an action $a^{\prime}\sim\pi(\cdot|s^{\prime})$

Initialize $m_{j}^{(s,a)}\leftarrow 0\quad\forall j=0,\dots,N-1$ # probability
vector for N atoms

### pushforward and projection

for $j=0,\dots,N-1$ do

${\mathcal{T}}^{\pi}z_{j}\leftarrow c+\gamma z_{j}$

if ${\mathcal{T}}^{\pi}z_{j}<C_{\text{min}}$ then

$m_{0}^{(s,a)}\leftarrow
m_{0}^{(s,a)}+\eta_{\theta,j}^{(s^{\prime},a^{\prime})}$

else if ${\mathcal{T}}^{\pi}z_{j}>C_{\text{max}}$ then

$m_{N-1}^{(s,a)}\leftarrow
m_{N-1}^{(s,a)}+\eta_{\theta,j}^{(s^{\prime},a^{\prime})}$

else

$b_{j}\leftarrow({\mathcal{T}}^{\pi}z_{j}-C_{\text{min}})/\Delta z$

$l\leftarrow\lfloor b_{j}\rfloor,u\leftarrow\lceil b_{j}\rceil$

$m_{l}^{(s,a)}\leftarrow
m_{l}^{(s,a)}+\eta_{\theta,j}^{(s^{\prime},a^{\prime})}(u-b_{j})$

$m_{u}^{(s,a)}\leftarrow
m_{u}^{(s,a)}+\eta_{\theta,j}^{(s^{\prime},a^{\prime})}(b_{j}-l)$

end if

end for

$\eta_{\theta,N}^{(s,a)}\leftarrow\sum_{j=0}^{N-1}m_{j}^{(s,a)}\delta_{z_{j}},\
\forall(s,a)\in{\mathcal{S}}\times{\mathcal{A}}$ # update distribution

$s\leftarrow s^{\prime}$ # update current state

end while

end for

return $\eta_{\theta,N},\\{\tau_{m}\\}_{m=1}^{M}$ # return the distribution
and a set of trajectories