# Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning
under Distribution Shifts

Tobias Enders Technical University of Munich, Munich, Germany,
<EMAIL_ADDRESS>James Harrison  Google DeepMind, San Francisco, CA, USA,
<EMAIL_ADDRESS>Maximilian Schiffer  Technical University of Munich,
Munich, Germany<EMAIL_ADDRESS>
###### Abstract

We study the robustness of deep reinforcement learning algorithms against
distribution shifts within contextual multi-stage stochastic combinatorial
optimization problems from the operations research domain. In this context,
risk-sensitive algorithms promise to learn robust policies. While this field
is of general interest to the reinforcement learning community, most studies
up-to-date focus on theoretical results rather than real-world performance.
With this work, we aim to bridge this gap by formally deriving a novel risk-
sensitive deep reinforcement learning algorithm while providing numerical
evidence for its efficacy. Specifically, we introduce discrete Soft Actor-
Critic for the entropic risk measure by deriving a version of the Bellman
equation for the respective $Q$-values. We establish a corresponding policy
improvement result and infer a practical algorithm. We introduce an
environment that represents typical contextual multi-stage stochastic
combinatorial optimization problems and perform numerical experiments to
empirically validate our algorithm’s robustness against realistic distribution
shifts, without compromising performance on the training distribution. We show
that our algorithm is superior to risk-neutral Soft Actor-Critic as well as to
two benchmark approaches for robust deep reinforcement learning. Thereby, we
provide the first structured analysis on the robustness of reinforcement
learning under distribution shifts in the realm of contextual multi-stage
stochastic combinatorial optimization problems.

Keywords: deep reinforcement learning, robustness, risk-sensitivity,
distribution shifts, combinatorial optimization problems

##  1 Introduction

Model-free deep reinforcement learning (DRL) has recently been used
increasingly to solve contextual multi-stage stochastic combinatorial
optimization (CO) problems from the operations research domain. Such problems
arise, among others, in inventory control [45, 13, 6] or control of mobility
on demand systems [42, 12, 36, 7], where an agent must solve a CO problem at
each time step to decide its policy. In this context, DRL benefits from using
neural networks (NNs), which are well suited to represent and learn complex
policies in such contextual environments. However, a policy trained by DRL to
solve a respective Markov decision process (MDP) is sensitive to changes of
environment parameters [22]. Yet, the resulting question of how to improve
DRL’s robustness against such disturbances has received limited attention in
the literature on DRL for multi-stage stochastic CO so far. With this work, we
take a first step to close this research gap.

While works on robust reinforcement learning (RL) exist, they are often
tailored to robotics applications and typically focus on topics such as safety
during exploration, robustness against action perturbations, or robustness
against adversarial attacks. In contrast, robustness against distribution
shifts is particularly relevant for contextual multi-stage stochastic CO
problems. Such distribution shifts can occur because of unforeseeable changes
in the environment, e.g., altered transportation patterns after the surge of
Covid or disruptions of supply chains triggered by geopolitical events.
Besides, imperfect simulators can lead to distribution shifts when training in
simulation before deployment in the real world. Additionally, continuous
action spaces are prevalent in robotics, while we often encounter discrete
action spaces in CO problems. Furthermore, existing works on robust RL often
focus on finding the optimal policy for a worst-case scenario. Instead, in a
CO context, one focuses on the tradeoff between learning a policy that
consistently achieves high expected returns on the training distribution and a
policy that is robust against distribution shifts at the price of lower yet
good performance across distributions. In the following, we develop a
methodology that provides a principled approach to explicitly control this
consistency-robustness tradeoff.

###  1.1 Related work

Multiple approaches to improve the robustness of RL exist, see the extensive
review in [26]. However, most of them have limited applicability in practice,
since they are not scalable with NNs as function approximators or require lots
of additional machinery, e.g., solving a bi-level optimization problem or
introducing adversaries. Alternative approaches are model-based or do not give
control over the consistency-robustness tradeoff. In contrast, we focus on
efficient and tractable approaches that can build on state-of-the-art model-
free DRL algorithms to achieve robustness under realistic distribution shifts.
Some approaches can fulfill these requirements: manipulation of training data,
entropy regularization, and risk-sensitive RL. We use the former two
approaches as benchmarks and focus our study on risk-sensitive RL. Thus, we
defer the discussion of further details on the benchmarks to the experiments
and results sections and focus the remaining discussion on risk-sensitive RL.

In risk-sensitive RL, one usually optimizes a risk measure, e.g., mean-
variance or conditional value at risk (CVaR), of the return instead of the
expected return. Theoretical connections exist between risk-sensitive and
robust RL when optimizing the CVaR [4] as well as for coherent risk measures
and risk-sensitivity under expected exponential utility functions [31]. Other
works report promising empirical results for risk-sensitive RL algorithms,
see, e.g., [50, 30]. Moreover, a risk-sensitive RL approach can lead to
algorithms similar to risk-neutral RL, thus requiring little extra
implementation effort. Depending on the risk measure, risk-sensitive RL
algorithms can have a hyperparameter controlling the tradeoff between expected
return and risk-sensitivity.

Some works consider a constrained optimization problem, where the expected
return is maximized while constraining a risk measure of the returns to set a
threshold for unwanted outcomes, see, e.g., [34, 33, 35, 49]. This approach
has a strong focus on safety and/or worst-case outcomes and thus fits our
purposes less than directly optimizing a risk-sensitive objective function. An
approach to optimize risk-sensitive objective functions is distributional RL,
see, e.g., [25, 37, 44]. It has the advantage that the learned distributional
information can be used to consider any risk measure, i.e., the proposed
algorithms are agnostic to the choice of risk measure. However, this comes at
the cost of additional algorithmic complexity.

Thus, we focus on approaches that directly optimize a risk-sensitive
objective. To do so, [19, 24, 32] use dynamic programming, assuming knowledge
of the underlying MDP. The authors of [41, 3, 4] propose policy gradient
and/or actor-critic and/or approximate value iteration algorithms for
objective functions based on variance or CVaR. In [40], a policy gradient and
actor-critic algorithm is derived for the whole class of coherent risk
measures [2]. All of these works date back to the year 2015 or earlier and
develop basic RL algorithms, rather than building on today’s state-of-the-art.
In particular, they do not include NNs as function approximators to deal with
large state spaces. The reported experiments focus on robustness under action
perturbations or risk-sensitivity in finance applications without a relation
to robustness or a CO context.

Recently, exponential criteria and the entropic risk measure have been
increasingly used as a risk-sensitive objective, with [10, 9, 11] focusing on
theoretical regret analysis rather than practical state-of-the-art algorithms
or experimental results. The authors of [27] derive a policy gradient
algorithm for the entropic risk measure and apply it in a robotics context. In
[29], a risk-sensitive variant of the REINFORCE algorithm for exponential
criteria is introduced and tested on the Cart Pole and Acrobot environments.
The risk-sensitive algorithm outperforms its risk-neutral counterpart, even
though the environment at test time is the same as during training (no
disturbance). This work is extended to an actor-critic algorithm, which
outperforms its risk-neutral counterpart on Cart Pole and Acrobot environments
when they are disturbed by varying the pole lengths during testing [30].
Although these works use NNs for function approximation, they are still the
risk-sensitive counterparts to basic instead of state-of-the-art risk-neutral
RL algorithms. This is not the case for [50], which develops a risk-sensitive
version of TD3 for a mean-variance objective and evaluates its performance on
MuJoCo environments with disturbed actions. However, this algorithm is not
compatible with a discrete action space, which is the focus of our study in a
CO context.

Concluding, to the best of our knowledge, no model-free risk-sensitive DRL
algorithm for discrete actions that is based on the state-of-the-art in risk-
neutral DRL exists. Moreover, none of the existing works investigates
robustness against distribution shifts, and most works on risk-sensitive RL
compare the performance of risk-sensitive RL algorithms only to their risk-
neutral counterparts. Finally, there exists no published work with a
structured analysis of the robustness of RL in CO problems.

###  1.2 Contributions

We aim to close the research gap outlined above by introducing a novel risk-
sensitive DRL algorithm: discrete Soft Actor-Critic (SAC) for the entropic
risk measure, which effectively learns policies that exhibit robustness
against distribution shifts. Specifically, we derive a version of the Bellman
equation for $Q$-values for the entropic risk measure. We establish a
corresponding policy improvement result and infer a practical model-free, off-
policy algorithm that learns from single trajectories. From an implementation
perspective, our algorithm requires only a small modification relative to
risk-neutral SAC and is therefore easily applicable in practice. Furthermore,
our algorithm allows to control the consistency-robustness tradeoff through a
hyperparameter. For empirical evaluation, we propose a grid world environment
that abstracts multiple relevant contextual multi-stage stochastic CO
problems.

We show that our algorithm improves robustness against distribution shifts
without performance loss on the training distribution compared to risk-neutral
SAC. Moreover, we evaluate our algorithm in comparison to two other
practically viable approaches to achieve robustness: manipulating the training
data and entropy regularization. The performance analysis of these approaches
within our environment under distribution shifts is of independent interest.
While manipulating the training data leads to good empirical results, it is
less generally applicable than our risk-sensitive algorithm and entropy
regularization. Entropy regularization achieves better robustness but worse
performance on the training distribution compared to our risk-sensitive
algorithm. To facilitate a direct comparison of the two approaches, we study
the weighted average of the performance on the training distribution and the
performance under distribution shifts: our risk-sensitive algorithm
outperforms entropy regularization if we assign at least 37% weight to the
performance on the training distribution. Overall, we provide the first
structured analysis of the robustness of RL under distribution shifts in a CO
context. To foster future research and ensure reproducibility, our code is
publicly available at
https://github.com/tumBAIS/RiskSensitiveSACforRobustDRLunderDistShifts.

##  2 Risk-sensitive Soft Actor-Critic for discrete actions

We base our novel risk-sensitive DRL algorithm on the variant of SAC [17] for
discrete actions [5]. We chose SAC for three reasons: firstly, it is a state-
of-the-art algorithm with very good performance across a wide range of
environments [18, 5, 21, 47, 39, 7, 48, 20]; secondly, it is an off-policy
algorithm and consequently more sample-efficient than on-policy algorithms;
thirdly, it already incorporates entropy regularization. The last reason makes
SAC particularly well suited for our robustness analysis: entropy
regularization can increase robustness, as shown theoretically in [8] and
empirically in [15, 16, 8]. Consequently, we can benchmark the robustness of
our risk-sensitive algorithm with turned off entropy regularization against
the robustness of risk-neutral SAC with different intensities of entropy
regularization.

In the remainder, we use the following basic notation: $t$ denotes a time
step, $s$ a state, $a$ an action, $r$ a reward, $d$ a done signal, $\gamma$
the discount factor, and $s^{\prime}$ the next state if there is no time index
when considering a single transition. We denote a (stochastic) policy by $\pi$
and its entropy given state $s$ by $\mathcal{H}\left(\pi\left(\cdot\lvert
s\right)\right)$. Moreover, $\alpha\geq 0$ is the entropy coefficient
hyperparameter, $\Pi$ denotes the space of tractable policies and $Q^{\pi}$
the $Q$-values under policy $\pi$. The notation
$\left(s_{t+1},...\right)\sim\rho_{\pi}$ refers to sampling a state-action
trajectory, i.e., $s_{t+1}\sim p(s_{t+1}\lvert
s_{t},a_{t}),\,a_{t+1}\sim\pi(a_{t+1}\lvert s_{t+1}),\,s_{t+2}\sim
p(s_{t+2}\lvert s_{t+1},a_{t+1}),\,...\,,\,s_{T}\sim p(s_{T}\lvert
s_{T-1},a_{T-1})$, where $p$ denotes the state transition probability
distribution and $T\in\mathbb{N}\cup\\{\infty\\}$ the terminal time step.
Furthermore, $D_{\text{KL}}(p\lvert\lvert q)$ is the Kullback-Leibler
divergence (KL-divergence) of probability distribution $q$ from probability
distribution $p$, while $D$ is the replay buffer. When considering a
parameterized policy $\pi_{\phi}$ or parameterized $Q$-values $Q_{\theta}$, we
denote the actor network parameters by $\phi$ and critic network parameters by
$\theta$, as well as target critic network parameters by $\overline{\theta}$.
When we write $\pi(s)$ instead of $\pi(a\lvert s)$ or $Q(s)$ instead of
$Q(s,a)$, we refer to the vector of all action probabilities or the vector of
$Q$-values for all actions, respectively, given state $s$ (as opposed to the
single entry of this vector for the specific action $a$).

###  2.1 Risk-neutral Soft Actor-Critic

We provide a short summary of the risk-neutral SAC algorithm for discrete
actions, before deriving our risk-sensitive version. SAC is an off-policy RL
algorithm which concurrently trains an actor network that parameterizes a
stochastic policy, i.e., a probability distribution over all possible actions,
and a critic network that outputs the $Q$-values for all possible actions,
given an input state. It regularizes rewards with an additional entropy term
to explicitly incentivize exploration, such that the optimization objective
reads

$\max_{\pi}\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}\left(r\left(s_{t},a_{t}\right)+\alpha\mathcal{H}\left(\pi\left(\cdot\lvert
s_{t}\right)\right)\right)\right].$

The entropy coefficient $\alpha$ controls the trade-off between rewards and
the entropy. It can be set as a hyperparameter or learned such that the
resulting policy has a certain target entropy chosen based on some heuristic.
In the remainder, we use the former option.

In the discrete actions setting, the loss functions for the actor and the
critic read

$\displaystyle J_{\pi}(\phi)$ $\displaystyle=\mathbb{E}_{s\sim
D}\left[\pi_{\phi}(s)^{T}\cdot\left(\alpha\log\pi_{\phi}(s)-Q_{\theta}(s)\right)\right],$
(1) $\displaystyle J_{Q}(\theta)$
$\displaystyle=\mathbb{E}_{(s,a,r,d,s^{\prime})\sim
D}\left[\frac{1}{2}\left(Q_{\theta}(s,a)-\hat{Q}\right)^{2}\right],$
$\displaystyle\hat{Q}$
$\displaystyle=r+(1-d)\gamma\cdot\pi_{\phi}(s^{\prime})^{T}\cdot\left(Q_{\overline{\theta}}\left(s^{\prime}\right)-\alpha\log\pi_{\phi}(s^{\prime})\right).$

In practice, we train two (target) critic networks and use the minimum of the
two $Q$-values in the policy loss and $\hat{Q}$ calculation, to mitigate the
overestimation bias. We use this minimum of two critics for our risk-sensitive
algorithm analogously but do not explicitly write the minimum of two
$Q$-values for conciseness.

In the following, we firstly introduce our risk-sensitive objective function.
Secondly, we derive a Bellman equation for the $Q$-values under this
objective. Thirdly, we show how to achieve policy improvement. Finally, we
derive a practical algorithm with function approximation based on these
theoretical results.

###  2.2 Risk-sensitive objective

Instead of the risk-neutral objective function, i.e., the expected value of
the sum of discounted rewards, we use the entropic risk measure [19, 24, 23,
46] as our risk-sensitive objective:

$\max_{\pi}\frac{1}{\beta}\log\mathbb{E}_{\pi}\left[e^{\beta\cdot\sum_{t=0}^{\infty}\gamma^{t}\left(r\left(s_{t},a_{t}\right)+\alpha\mathcal{H}\left(\pi\left(\cdot\lvert
s_{t}\right)\right)\right)}\right].$ (2)

Here, we adjust the standard entropic risk measure by introducing entropy
regularization of rewards to obtain a risk-sensitive variant of SAC.
Analogously to risk-neutral SAC, entropy regularization allows us to
explicitly control the exploration-exploitation tradeoff in our risk-sensitive
algorithm via the entropy coefficient. The hyperparameter $\beta\in\mathbb{R}$
controls the risk-sensitivity of the objective:

$\frac{1}{\beta}\log\mathbb{E}\left[e^{\beta
R}\right]=\mathbb{E}[R]+\frac{\beta}{2}\mathrm{Var}[R]+\mathcal{O}\left(\beta^{2}\right),$
(3)

where we set $R=\sum_{t=0}^{\infty}\gamma^{t}\cdot r\left(s_{t},a_{t}\right)$.
Since the variance measures uncertainty, we obtain a risk-averse objective
function for $\beta<0$, for $\beta>0$ it is risk-seeking, and
$\beta\rightarrow 0$ recovers the common risk-neutral objective.

In principle, we could use any risk measure. However, the entropic risk
measure is a natural choice, particularly for operations research
applications, as it is the certainty-equivalent expectation of the exponential
utility function, see [19, 24]. Risk-sensitive MDP s with the expected
exponential utility function as optimization objective are closely connected
to robust MDP s [31]. Also, the entropic risk measure is a convex risk
measure, thus fulfilling multiple desirable properties from a mathematical
risk management perspective, even though it is not coherent [2]. Furthermore,
alternative measures such as CVaR are computed solely based on the tail of the
return distribution, using only a small portion of the data. Contrarily, the
entropic risk measure bases on all data, which is important in an RL context,
where sample efficiency is a major concern. Besides, CVaR does not give
explicit control over the tradeoff between expected return and risk, as
opposed to the entropic risk measure: Equation (3) reveals that the entropic
risk measure allows to explicitly control this tradeoff by setting $\beta$
accordingly.

Moreover, we will show below that the entropic risk measure is well suited to
derive a practical RL algorithm without adding a lot of machinery. The
resulting algorithm can still leverage a lot of valuable techniques developed
for risk-neutral RL, like off-policy learning with experience replay and
entropy regularization for effective exploration.

Since we theoretically expect risk-averse rather than risk-seeking behavior to
improve robustness, we use negative values for $\beta$ in the empirical
evaluation. Nevertheless, the theoretical results which we derive in the
following as well as the practical algorithm also apply to the risk-seeking
case with $\beta>0$.

In the following, we use two approximations (see Appendix A.1 for details):
(i) for $\beta$ close to zero and a real-valued random variable $X$, we get
$\mathbb{E}\left[e^{\beta X}\right]\approx
e^{\beta\,\mathbb{E}\left[X\right]}$; (ii) for $\gamma$ close to one, it holds
that
$\mathbb{E}\left[X^{\gamma}\right]\approx\left(\mathbb{E}[X]\right)^{\gamma}$.

###  2.3 Bellman equation

To facilitate value iteration, we derive a Bellman equation for the $Q$-values
under our risk-sensitive objective. We want to use the Bellman equation as the
basis for a model-free RL algorithm that can learn from single trajectories.
Thus, the Bellman equation should take the form
$Q^{\pi}\\!\\!\left(s_{t},a_{t}\right)=\mathbb{E}_{\left(s_{t+1},a_{t+1}\right)\sim\rho_{\pi}}\left[\cdot\right]$,
which allows to obtain an unbiased sampling-based estimate of the right-hand
side (RHS). With the logarithm in Equation (2), we cannot obtain a Bellman
equation of that form. Thus, we define

$\overline{Q}^{\pi}\\!\\!(s_{t},a_{t}):=e^{\beta\cdot
Q^{\pi}\\!\\!(s_{t},a_{t})}\iff
Q^{\pi}\\!\\!\left(s_{t},a_{t}\right)=\frac{1}{\beta}\log\overline{Q}^{\pi}\\!\\!\left(s_{t},a_{t}\right)$
(4)

and derive a Bellman equation for $\overline{Q}$. While we use the same
rationale as [30] here, we cannot use this work’s result, as it considers
$V$-values and does not incorporate entropy regularization. In the following,
we use the notation
$\overline{Q}^{\pi}_{t}:=\overline{Q}^{\pi}\\!\\!\left(s_{t},a_{t}\right)$,
$r_{t}:=r\left(s_{t},a_{t}\right)$, and
$\mathcal{H}^{\pi}_{t}:=\mathcal{H}\left(\pi\left(\cdot\lvert
s_{t}\right)\right)$ to save space.

######  Proposition 1 (Bellman equation).

For the risk-sensitive objective in Equation (2), $\gamma$ close to one, and
with $\overline{Q}$ as defined in Equation (4), it holds that

$\overline{Q}^{\pi}_{t}=\mathbb{E}_{\left(s_{t+1},a_{t+1}\right)\sim\rho_{\pi}}\left[\exp\left(\beta
r_{t}+\beta\gamma\alpha\mathcal{H}^{\pi}_{t+1}+\gamma\log\overline{Q}^{\pi}_{t+1}\right)\right].$
(5)

###### Proof.

With the definition of $Q$-values, we obtain

$\displaystyle\overline{Q}^{\pi}_{t}$ $\displaystyle=e^{\beta
r_{t}}\cdot\mathbb{E}_{\left(s_{t+1},...\right)\sim\rho_{\pi}}\left[\exp\left(\beta\cdot\sum_{l=1}^{\infty}\gamma^{l}\left(r_{t+l}+\alpha\mathcal{H}^{\pi}_{t+l}\right)\right)\right]$
$\displaystyle\stackrel{{\scriptstyle\mathmakebox[\widthof{=}]{\text{(a)}}}}{{=}}e^{\beta
r_{t}}\cdot\mathbb{E}_{\left(s_{t+1},a_{t+1}\right)\sim\rho_{\pi}}\Biggl{[}\exp\left(\beta\gamma\left(r_{t+1}+\alpha\mathcal{H}^{\pi}_{t+1}\right)\right)\Biggr{.}$
$\displaystyle\hskip
54.06006pt\Biggl{.}\cdot\mathbb{E}_{\left(s_{t+2},...\right)\sim\rho_{\pi}}\left[\exp\left(\beta\cdot\sum_{l=2}^{\infty}\gamma^{l}\left(r_{t+l}+\alpha\mathcal{H}^{\pi}_{t+l}\right)\right)\right]\Biggr{]}$
$\displaystyle\stackrel{{\scriptstyle\mathmakebox[\widthof{=}]{\text{(b)}}}}{{=}}e^{\beta
r_{t}}\cdot\mathbb{E}_{\left(s_{t+1},a_{t+1}\right)\sim\rho_{\pi}}\Biggl{[}\exp\left(\beta\gamma\alpha\mathcal{H}^{\pi}_{t+1}\right)\Biggr{.}$
$\displaystyle\Biggl{.}\cdot\left(e^{\beta
r_{t+1}}\cdot\mathbb{E}_{\left(s_{t+2},...\right)\sim\rho_{\pi}}\left[\exp\left(\beta\cdot\sum_{l=2}^{\infty}\gamma^{l-1}\left(r_{t+l}+\alpha\mathcal{H}^{\pi}_{t+l}\right)\right)\right]\right)^{\gamma}\Biggr{]}$
$\displaystyle\stackrel{{\scriptstyle\mathmakebox[\widthof{=}]{\text{(c)}}}}{{=}}e^{\beta
r_{t}}\cdot\mathbb{E}_{\left(s_{t+1},a_{t+1}\right)\sim\rho_{\pi}}\left[\exp\left(\beta\gamma\alpha\mathcal{H}^{\pi}_{t+1}\right)\cdot\left(\overline{Q}^{\pi}_{t+1}\right)^{\gamma}\right]$
$\displaystyle=\mathbb{E}_{\left(s_{t+1},a_{t+1}\right)\sim\rho_{\pi}}\left[\exp\left(\beta
r_{t}+\beta\gamma\alpha\mathcal{H}^{\pi}_{t+1}+\gamma\log\overline{Q}^{\pi}_{t+1}\right)\right].$

Equality (a) is based on the observation that the first term in the sum does
not depend on the transitions after $t+1$. Equality (b) follows from
Approximation (ii). Equality (c) uses the definition of $Q$-values for time
step $t+1$. ∎

###  2.4 Policy improvement

Given the $Q$-values obtained under an old policy $\pi_{\text{old}}$, we
obtain a new policy as

$\displaystyle\pi_{\text{new}}$
$\displaystyle=\arg\min_{\pi^{\prime}\in\Pi}D_{\text{KL}}\left(\pi^{\prime}\left(\cdot\lvert
s_{t}\right)\Bigg{\lvert}\Bigg{\lvert}\frac{e^{\frac{1}{\alpha}Q^{\pi_{\text{old}}}\\!\left(s_{t},\cdot\right)}}{Z^{\pi_{\text{old}}}\left(s_{t}\right)}\right)$
$\displaystyle=\arg\min_{\pi^{\prime}\in\Pi}J_{\pi_{\text{old}}}\left(\pi^{\prime}\left(\cdot\lvert
s_{t}\right)\right),$ (6)

exactly as in the original SAC paper [17]. Here, the partition function
$Z^{\pi_{\text{old}}}\left(s_{t}\right)$ normalizes the distribution. We show
that despite our changed objective function and thus different Bellman
equation, this definition of a new policy still implies policy improvement.

######  Proposition 2 (Policy improvement).

For an old policy $\pi_{\text{old}}$ and the new policy $\pi_{\text{new}}$ as
defined in Equation (6), it holds that $Q^{\pi_{\text{new}}}(s_{t},a_{t})\geq
Q^{\pi_{\text{old}}}(s_{t},a_{t})$ for any state $s_{t}$ and action $a_{t}$,
assuming that $\gamma$ is close to one and $\beta$ is close to zero.

###### Proof.

We sketch the proof here and provide details in Appendix A.2.

We can always choose $\pi_{\text{new}}=\pi_{\text{old}}$, such that
$J_{\pi_{\text{old}}}\left(\pi_{\text{new}}\left(\cdot\lvert
s_{t}\right)\right)\leq
J_{\pi_{\text{old}}}\left(\pi_{\text{old}}\left(\cdot\lvert
s_{t}\right)\right)$, which yields

$\displaystyle\mathbb{E}_{a_{t}\sim\pi_{\text{new}}}\left[Q^{\pi_{\text{old}}}_{t}-\alpha\log\pi_{\text{new}}\left(a_{t}\lvert
s_{t}\right)\right]$
$\displaystyle\geq\mathbb{E}_{a_{t}\sim\pi_{\text{old}}}\left[Q^{\pi_{\text{old}}}_{t}-\alpha\log\pi_{\text{old}}\left(a_{t}\lvert
s_{t}\right)\right].$

For $\beta<0$, this is equivalent to

$e^{\beta\alpha\mathcal{H}^{\pi_{\text{new}}}_{t}}\cdot\mathbb{E}_{a_{t}\sim\pi_{\text{new}}}\left[\overline{Q}^{\pi_{\text{old}}}_{t}\right]\leq
e^{\beta\alpha\mathcal{H}^{\pi_{\text{old}}}_{t}}\cdot\mathbb{E}_{a_{t}\sim\pi_{\text{old}}}\left[\overline{Q}^{\pi_{\text{old}}}_{t}\right].$

Repeated application of the Bellman equation derived before and this
inequality gives
$\overline{Q}^{\pi_{\text{old}}}_{t}\geq\overline{Q}^{\pi_{\text{new}}}_{t}$,
such that we obtain $Q^{\pi_{\text{old}}}_{t}\leq Q^{\pi_{\text{new}}}_{t}$.
The proof for $\beta>0$ works analogously. ∎

###  2.5 Practical algorithm

We can use the Bellman equation and the policy improvement result in a
straightforward manner to obtain a practical risk-sensitive off-policy DRL
algorithm. It is similar to risk-neutral SAC, with the following adjustments:

We can learn $\overline{Q}$ with the corresponding critic loss function

$\displaystyle J_{\overline{Q}}(\theta)$
$\displaystyle=\mathbb{E}_{(s,a,r,d,s^{\prime})\sim
D}\left[\frac{1}{2}\left(\overline{Q}_{\theta}(s,a)-\hat{Q}\right)^{2}\right],$
$\displaystyle\hat{Q}$ $\displaystyle=\exp\left(\beta
r-\beta\gamma\alpha\>\pi_{\phi}(s^{\prime})^{T}\log\pi_{\phi}(s^{\prime})\right)\cdot\pi_{\phi}(s^{\prime})^{T}\left(\overline{Q}_{\overline{\theta}}\left(s^{\prime}\right)\right)^{\gamma},$

where the target $\hat{Q}$ is an unbiased, sampling-based estimate of Equation
(5). We computed the entropy and the expectation over next actions directly,
which is possible because of the discrete action space, such that we only
sample the next state to estimate Equation (5). If the done signal $d$ is
$\mathrm{TRUE}$, we set the terms after $e^{\beta r}$ to one. To ensure that
$\overline{Q}>0$, we use a softplus instead of a linear activation on the
output layer of the critic networks.

Based on Equation (6), the policy loss function is the same as in risk-neutral
SAC, shown in Equation (1), but we calculate the needed critic values as
$Q_{\theta}(s)=\frac{1}{\beta}\log\overline{Q}_{\theta}(s)$ from the output
$\overline{Q}_{\theta}(s)$ of the critic network(s).

While the resulting algorithm is simple, its usability is limited since it is
numerically unstable in practice as we empirically show in Appendix B. We
hypothesize that the numerical instability is caused by the fact that we learn
$\overline{Q}$ instead of $Q$. To mitigate this numerical instability, we note
that Equation (5) is equivalent to

$\displaystyle Q^{\pi}_{t}$
$\displaystyle=\frac{1}{\beta}\log\left(\overline{Q}^{\pi}_{t}\right)$
$\displaystyle=\frac{1}{\beta}\log\left(\mathbb{E}_{\left(s_{t+1},a_{t+1}\right)\sim\rho_{\pi}}\left[e^{\beta\left(r_{t}+\gamma\left(\alpha\mathcal{H}^{\pi}_{t+1}+Q^{\pi}_{t+1}\right)\right)}\right]\right).$
(7)

When learning $Q$, the RHS of Equation (7) is the target in the mean squared
error loss function. Due to the discrete action space, we can compute the
expectation over next actions directly, given a next state.

Still, we need a sampling-based estimate of the RHS: since we want to obtain a
model-free algorithm that learns from single trajectories, we cannot compute
the expectation over next states directly. As the expectation appears inside
the logarithm, replacing the expectation by a mean over samples does not give
an unbiased estimate of the RHS. We do so nevertheless and observe that this
leads to a well-performing algorithm in practice. We have only one next state
$s^{\prime}$ corresponding to each state $s$ that we sample from the replay
buffer to compute the loss. Consequently, we remove the expectation over next
states in Equation (7) and replace $s_{t+1}$ by the next state $s^{\prime}$
from the replay buffer. Then, the critic loss function becomes

$\displaystyle J_{Q}(\theta)$
$\displaystyle=\mathbb{E}_{(s,a,r,d,s^{\prime})\sim
D}\left[\frac{1}{2}\left(Q_{\theta}(s,a)-\hat{Q}\right)^{2}\right],$
$\displaystyle\hat{Q}$
$\displaystyle=\frac{1}{\beta}\log\Bigl{(}\pi_{\phi}(s^{\prime})^{T}\Bigr{.}$
$\displaystyle\hskip
31.2982pt\Bigl{.}\cdot\exp\left(\beta\left(r+\gamma\left(Q_{\overline{\theta}}\left(s^{\prime}\right)-\alpha\cdot\pi_{\phi}(s^{\prime})^{T}\log\pi_{\phi}(s^{\prime})\right)\right)\right)\Bigr{)}.$

The computation of this target $\hat{Q}$ has a log-sum-exp structure, which is
known to be numerically unstable when implemented naively. Thus, we rewrite it
as follows (see Appendix C for details):

$\displaystyle\hat{Q}$
$\displaystyle=r-\gamma\alpha\cdot\pi_{\phi}(s^{\prime})^{T}\log\pi_{\phi}(s^{\prime})+\gamma\max_{a^{\prime}}Q_{\overline{\theta}}(s^{\prime},a^{\prime})$
$\displaystyle\quad+\frac{1}{\beta}\log\left(\sum_{a^{\prime}}\pi_{\phi}(a^{\prime}\lvert
s^{\prime})\cdot\exp\left(\beta\gamma\left(Q_{\overline{\theta}}(s^{\prime},a^{\prime})-\max_{a^{\prime}}Q_{\overline{\theta}}(s^{\prime},a^{\prime})\right)\right)\right).$

With this new critic loss function, we can learn $Q$ directly, without the
need to learn $\overline{Q}$. The resulting algorithm is identical to risk-
neutral SAC for discrete actions, except for the adapted critic loss, which
gives the intended risk-sensitivity. We provide a pseudocode in Appendix D.

##  3 Experiments

We test the proposed risk-sensitive SAC algorithm for discrete actions on an
environment that abstracts several multi-stage stochastic CO problems. In the
following, we introduce the environment, discuss our benchmark algorithms, and
explain how we evaluate the algorithms’ performance and robustness. For
details on our state encoding, NN architectures, and hyperparameters, see
Appendix E.

###  3.1 Environment

We consider a discrete time horizon comprising 200 time steps per episode. Our
environment is a 2D grid containing 5x5 cells, in which an agent can move
around freely in the four cardinal directions, i.e., the action space at each
time step is {no move, move up, move right, move down, move left}. If an
attempted move causes the agent to exit the grid, it remains in its current
cell. Each move incurs a negative reward (cost) of -1.

Within this grid, items appear stochastically based on a spatial probability
distribution at each time step. We assume that the distribution remains
unknown to the agent, which can only observe the resulting data upon
interacting with the environment. The items disappear after a maximum response
time of ten time steps. When the agent reaches a cell containing an item
before it disappears, it collects the item and should then transport it to a
fixed target location (see Figure 1). Delivering an item to the target
location leads to a positive reward (revenue) of +15. The agent cannot carry
more than one item at a time. This problem setting satisfies the Markov
property, since the probability that an item appears in a specific cell during
the current time step does neither depend on the agent’s past actions nor on
which items appeared in prior time steps.

Our environment is an abstraction of various classical CO problems requiring
sequential online decision-making under uncertainty, e.g., various routing and
dispatching problems in the context of warehouses, mobility on demand systems,
(crowd-sourced) delivery, or ambulance operations. While our environment is
simple, it is also general and thus well suited to study robustness of DRL in
multi-stage stochastic CO problems. Moreover, our environment is well suited
to investigate distribution shifts, which can occur in all aforementioned
applications: Figure 2 depicts the item-generating probability distributions
considered here. We train on data sampled from the gradient 1 item
distribution, while testing evaluates the trained model on data sampled from
all twelve distributions, simulating a wide range of distribution shifts.

One comment on the action space is in order: our agent takes the micro-
decision to which neighboring cell to move next, rather than taking the macro-
decision to which cell (even if it is not a neighboring one) to move next. The
latter option might appear more natural, e.g., for the decision which item to
collect. However, we intentionally chose the former option, as it is more
general and any macro-decision can be reconstructed through a series of micro-
decisions.

Figure 1: Illustration of the environment. The gray cell is the target
location. Figure 2: Per-time-step probability that an item appears in the
respective cell, for twelve different item distributions.

###  3.2 Benchmarks

We compare our risk-sensitive SAC algorithm against two benchmarks:
manipulating the training data and entropy regularization. Moreover, we
compare all three approaches to improve robustness against risk-neutral SAC
for discrete actions, which is our non-robust baseline algorithm. We further
hypothesized that L2 regularization of the NNs’ parameters could improve
robustness, as it is frequently used in (supervised) machine learning to use a
complex model but reduce overfitting, and learning a policy that is less
tailored to the training data might help robustness against distribution
shifts. However, we did not find evidence for this hypothesis. We omit details
on this important negative result in the main body of the paper, but provide
them in Appendix F.

#### Manipulating the training data.

In supervised learning, manipulating the training data has been used
successfully to achieve robustness, see [14, 43, 38]. We transfer this
approach to RL by inserting noise into the training process. We expect the
policy to learn to perform well under this noise, generalize better, and thus
be more robust against disturbances during testing.

For distribution shifts in our environment, this means to replace a part of
the item locations in the training data by item locations sampled from a
different spatial distribution. Since we assume the distribution shift to be
unknown a priori, we use the uniformly random distribution to sample the
manipulated item locations. Specifically, we replace each item in the training
data with a certain probability, for which we test a large range of different
values.

For a fair comparison, we train risk-neutral SAC on the manipulated training
data and test the resulting policy on the not-manipulated test data for all
distributions in Figure 2. This includes gradient 1, to assess how
manipulating the training data changes the trained policy’s test performance
without distribution shifts.

#### Entropy regularization.

The use of entropy regularization can increase the robustness of RL against
disturbances in the environment, as shown theoretically in [8] and empirically
in [15, 16, 8]. Risk-neutral SAC already incorporates entropy regularization.
Its intensity can be controlled via the entropy coefficient hyperparameter
$\alpha$. For our non-robust baseline algorithm, for which we focus on
maximizing performance on the gradient 1 distribution, we found a scheduled
$\alpha$ to be effective: we use a tuned $\alpha>0$ at the beginning of
training for effective exploration, then set it to zero after a tuned number
of iterations, and continue training with $\alpha=0$ until convergence,
obtaining the final policy without entropy regularization. We use the same
schedule for $\alpha$ in our risk-sensitive SAC algorithm. When we use entropy
regularization to improve the robustness of the risk-neutral algorithm, we
keep the tuned $\alpha$-value at the beginning of training, but then reset it
to a positive value for the remainder of training until convergence. We
conduct experiments for a large range of different values for this final
$\alpha$.

###  3.3 Performance evaluation

We measure an algorithm’s performance in our environment in terms of its
percentage improvement over a greedy baseline algorithm, as greedy algorithms
typically perform well in related CO problems [7]. Advanced algorithms
typically outperform greedy algorithms by only a few percentage points. Still,
the effort to develop such algorithms is meaningful, since they are typically
deployed at a large scale in practice, such that the small percentage
improvements translate into large absolute gains [36].

We implement the following greedy algorithm: if the agent has collected but
not yet delivered an item, it moves to the target location on the shortest
route. If the agent has no item on board, but there is an item available that
can be reached before it disappears and will lead to a positive profit
(accounting for revenue and cost for moving to the item and then to the target
location), the agent moves towards the item’s location on the shortest route.
If there are multiple such items available, the agent moves towards the item
that will lead to the highest profit. If there is no such item available, the
agent does not move, i.e., it stays at the target location. When the agent has
started to move towards an item and then a more profitable item appears, the
agent changes direction and moves towards the more profitable item.

This greedy algorithm is not tailored to a specific item distribution, i.e.,
it is robust against distribution shifts. Thus, when we report performance
improvement over the greedy algorithm, we report performance improvement over
a robust baseline, which makes the greedy algorithm particularly well suited
for our purposes.

When we report the performance of a DRL algorithm under a distribution shift,
i.e., the performance on test data from a distribution different than the
training distribution, we report its performance gain over greedy relative to
the performance gain of the policy obtained by training a risk-neutral SAC
agent on data from the respective shifted distribution. Thereby, we view the
performance of a DRL agent trained on the “true” distribution as an upper
bound on the performance of a robust or non-robust DRL agent trained on the
“wrong” distribution.

We use 1,000 episodes of sampled data for each of the item distributions,
which we split into 800 training, 100 validation, and 100 testing episodes.
The test performance is the non-discounted cumulative reward per episode,
averaged over the 100 testing episodes. We repeat every training run with
three different random seeds and select the model with the highest validation
reward across seeds for testing.

To reduce the impact of the random sampling process on the reported results
for our benchmark based on manipulated training data, we repeat the training
data manipulation process three times with three different random seeds. Then,
we repeat every experiment for each of the three manipulated data sets.
Finally, we report the test performance averaged over the three policies
trained on the three different data sets.

##  4 Results and discussion

In the following, we briefly discuss the convergence behavior of our risk-
sensitive algorithm. Then, we analyze the performance of our algorithm and the
two benchmarks in detail.

###  4.1 Convergence analysis

Figure 3 shows the validation reward over the course of training for our risk-
sensitive algorithm with $\beta=-1$ in comparison to risk-neutral SAC. The
training curves illustrate that for small absolute $\beta$-values, our risk-
sensitive algorithm shows stable convergence behavior. For larger absolute
values of $\beta$, our algorithm still exhibits stable convergence to a good
policy for specific random seeds, but not across all tested random seeds.
Moreover, for all tested values of $\beta$, our risk-sensitive algorithm
requires a similar number of samples and a similar computational time as risk-
neutral SAC.

Figure 3: Convergence behavior of our risk-sensitive algorithm with $\beta=-1$
compared to risk-neutral SAC. For each algorithm, we show the training curves
for three different random seeds. The non-transparent lines correspond to the
best-performing seed, the transparent ones to the other seeds.

###  4.2 Performance analysis

Figure 4: Performance of the three approaches to improve robustness. The top
row shows their performance on the test data for the training distribution.
The bottom row shows the average over all other distributions, i.e., the
performance under distribution shifts. The left-most data point in each plot
shows the results for the non-robust discrete SAC algorithm. We report all
results relative to SAC’s performance when trained on the shifted, i.e., the
true distribution as explained in Section 3.3.

Figure 4 shows the performance of the three approaches to improve robustness,
both on the test data for the training distribution and under distribution
shifts. As theoretically expected, the non-robust SAC algorithm performs worse
under distribution shifts than when it is trained on data from the true
distribution. Specifically, the performance gain over greedy under
distribution shifts is only 20% of the achievable performance gain. This
illustrates that our environment and the considered item distributions are
suitable to experimentally investigate the robustness of DRL algorithms under
distribution shifts.

Our risk-sensitive algorithm improves the performance under distribution
shifts compared to risk-neutral SAC, as it reaches up to 53% of the upper
bound’s performance improvement over greedy. Up to $\beta=-2$, an increasing
absolute value of $\beta$ and therefore increasing risk-aversion improves the
performance under distribution shifts. As the absolute value of $\beta$
becomes even larger, the risk-sensitive algorithm still outperforms the risk-
neutral algorithm under distribution shifts, but performance decreases again
and the performance pattern becomes slightly unstable. This instability is in
line with the previously described worsening convergence behavior as the
absolute value of $\beta$ increases. These empirical observations align with
our theoretical results in Section 2, which assume that $\beta$ is close to
zero. Finally, as the absolute value of $\beta$ becomes too large, performance
collapses, as the agent becomes too risk-averse and the learning process too
unstable to learn a good policy.

Except for too large absolute $\beta$-values, the risk-averse policy
consistently outperforms the risk-neutral policy on the training distribution,
improving the risk-neutral policy’s performance gain over greedy up to 114
percentage points. This result might be surprising at first sight, as one
might expect a consistency-robustness tradeoff. However, it is in line with
experimental results in the literature [25, 29], where risk-averse DRL
algorithms outperform their risk-neutral counterparts on the training
environments. The authors of [29] explain this observation by a variance
reduction due to the risk-averse objective, which in turn helps to converge to
a better policy for the training environment. Based on Equation (3), our risk-
sensitive objective can be interpreted as the variance-regularized expected
return, penalizing high variance for negative $\beta$-values. This explains
our risk-averse algorithm’s performance improvement on the gradient 1
distribution.

The manipulation of the training data also improves the performance under
distribution shifts, to about 60-80% of the upper bound’s performance
improvement over greedy when we sample at least 40% of item locations from the
uniform distribution. Simultaneously, the performance on the gradient 1
distribution increases for small proportions of introduced noise, but
decreases as this proportion becomes larger. Consequently, manipulating the
training data leads to better robustness results than our risk-sensitive
algorithm. Which approach is favorable depends on how one weighs consistency
versus robustness, as our risk-sensitive algorithm achieves better results on
the training distribution. Besides, the manipulation of the training data
requires the ability to change the training environment deliberately,
typically in a simulator, and the domain expertise how to manipulate the
training data effectively for the respective problem setting. We can use our
risk-sensitive algorithm as well as entropy regularization even when these
requirements are not fulfilled, such that these are more generally applicable.

With entropy regularization, the performance under distribution shifts
improves to 60% of the upper bound’s performance improvement over greedy as
the entropy regularization coefficient $\alpha$ increases. For further
increasing $\alpha$, the performance under distribution shifts decreases again
because the over-regularization hurts performance. On the training
distribution, the performance decreases as $\alpha$ increases. Consequently,
entropy regularization leads to slightly better robustness than our risk-
sensitive algorithm, at the price of lower performance on the training
distribution. Note that we also combined our risk-sensitive algorithm with
entropy regularization, but found that this does not further improve
performance, see Appendix G.

Figure 5 compares the consistency-robustness tradeoff for our risk-sensitive
algorithm and entropy regularization. Here, we consider the weighted average
of the performance on gradient 1 and the performance under distribution shifts
to evaluate these two approaches to improve the robustness of DRL based on a
single metric. When we assign at least 37% weight to the performance on the
training distribution, our risk-sensitive algorithm outperforms entropy
regularization. Thus, assuming that good performance on the training
distribution is not only a secondary objective with less than 37% weight, our
algorithm is superior to entropy regularization in our experiments.
Nevertheless, the final decision which approach is best suited to ensure
robustness depends on the needs of the considered problem setting.

Figure 5: Consistency-robustness tradeoff for entropy regularization with
$\alpha=0.05$ and our risk-sensitive algorithm with $\beta=-2$. Using the
performance under distribution shifts vs. non-robust SAC trained on the
shifted distributions as the performance metric, we show the weighted average
of the performance on the training distribution and the performance under
distribution shifts.

##  5 Conclusion

We present discrete SAC for the entropic risk measure, which is the first
model-free risk-sensitive DRL algorithm for discrete actions that builds on
the state-of-the-art in risk-neutral DRL. Specifically, we derive a version of
the Bellman equation for $Q$-values for the entropic risk measure. We
establish a corresponding policy improvement result and infer a practical off-
policy algorithm that learns from single trajectories. Our algorithm allows to
control its risk-sensitivity via a hyperparameter, and implementing our
algorithm requires only a small modification relative to risk-neutral SAC,
such that it is easily applicable in practice. We conduct experiments within
an environment representing typical contextual multi-stage stochastic CO
problems from the operations research domain. Thereby, we demonstrate that our
risk-sensitive algorithm significantly improves robustness against
distribution shifts compared to risk-neutral SAC. Simultaneously, it improves
the performance on the training distribution. We compare our risk-sensitive
algorithm to (i) manipulation of the training data and (ii) entropy
regularization, showing that our algorithm is superior to these benchmarks due
to its more general applicability and a better consistency-robustness
tradeoff. Our study is the first structured analysis of the robustness of RL
under distribution shifts in the realm of contextual multi-stage stochastic CO
problems. Moreover, we note that the presented algorithm is relevant beyond
this application domain and may be used for any MDP with discrete actions
where risk-sensitivity or robustness against environment perturbations is
relevant.

In future work, we will extend this research in the following directions: we
will investigate the performance of our algorithm in a more complex, large-
scale version of our environment with multiple agents. Furthermore, we will
develop a version of our risk-sensitive SAC algorithm for continuous actions.
Finally, we will implement and test a risk-sensitive Deep Q-Network (DQN)
algorithm for the entropic risk measure based on the derived Bellman equation.

## References

  * [1]
  * Artzner et al. [1999] Philippe Artzner, Freddy Delbaen, Jean-Marc Eber, and David Heath. 1999.  Coherent Measures of Risk.  _Mathematical Finance_ 9, 3 (1999), 203–228.
  * Chow and Ghavamzadeh [2014] Yinlam Chow and Mohammad Ghavamzadeh. 2014.  Algorithms for CVaR Optimization in MDPs. In _Advances in Neural Information Processing Systems_.
  * Chow et al. [2015] Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. 2015\.  Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach. In _Advances in Neural Information Processing Systems_.
  * Christodoulou [2019] Petros Christodoulou. 2019\.  Soft Actor-Critic for Discrete Action Settings.  _arXiv preprint arXiv:1910.07207_ (2019).
  * De Moor et al. [2022] Bram J. De Moor, Joren Gijsbrechts, and Robert N. Boute. 2022\.  Reward shaping to improve the performance of deep reinforcement learning in perishable inventory management.  _European Journal of Operational Research_ 301, 2 (2022), 535–545.
  * Enders et al. [2023] Tobias Enders, James Harrison, Marco Pavone, and Maximilian Schiffer. 2023.  Hybrid Multi-agent Deep Reinforcement Learning for Autonomous Mobility on Demand Systems. In _Proceedings of the 5th Conference on Learning for Dynamics and Control (L4DC)_.
  * Eysenbach and Levine [2022] Benjamin Eysenbach and Sergey Levine. 2022.  Maximum Entropy RL (Provably) Solves Some Robust RL Problems.  _International Conference on Learning Representations (ICLR)_ (2022).
  * Fei et al. [2021b] Yingjie Fei, Zhuoran Yang, Yudong Chen, and Zhaoran Wang. 2021b.  Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning. In _Advances in Neural Information Processing Systems_.
  * Fei et al. [2020] Yingjie Fei, Zhuoran Yang, Yudong Chen, Zhaoran Wang, and Qiaomin Xie. 2020.  Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret. In _Advances in Neural Information Processing Systems_.
  * Fei et al. [2021a] Yingjie Fei, Zhuoran Yang, and Zhaoran Wang. 2021a.  Risk-Sensitive Reinforcement Learning with Function Approximation: A Debiasing Approach. In _Proceedings of the 38th International Conference on Machine Learning_.
  * Gammelli et al. [2021] Daniele Gammelli, Kaidi Yang, James Harrison, Filipe Rodrigues, Francisco C. Pereira, and Marco Pavone. 2021.  Graph Neural Network Reinforcement Learning for Autonomous Mobility-on-Demand Systems. In _Proceedings of the 60th IEEE Conference on Decision and Control (CDC)_.
  * Gijsbrechts et al. [2022] Joren Gijsbrechts, Robert N. Boute, Jan A. Van Mieghem, and Dennis J. Zhang. 2022.  Can Deep Reinforcement Learning Improve Inventory Management? Performance on Dual Sourcing, Lost Sales and Multi-Echelon Problems.  _Manufacturing & Service Operations Management_ 24, 3 (2022), 1261–1885.
  * Goodfellow et al. [2015] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015.  Explaining and Harnessing Adversarial Examples. In _International Conference on Learning Representations (ICLR)_.
  * Haarnoja et al. [2018a] Tuomas Haarnoja, Sehoon Ha, Aurick Zhou, Jie Tan, George Tucker, and Sergey Levine. 2018a.  Learning to Walk via Deep Reinforcement Learning.  _arXiv preprint arXiv:1812.11103_ (2018).
  * Haarnoja et al. [2018b] Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine. 2018b.  Composable Deep Reinforcement Learning for Robotic Manipulation. In _2018 International Conference on Robotics and Automation (ICRA)_.
  * Haarnoja et al. [2018c] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018c.  Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In _Proceedings of the 35th International Conference on Machine Learning_.
  * Haarnoja et al. [2018d] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. 2018d.  Soft Actor-Critic Algorithms and Applications.  _arXiv preprint arXiv:1812.05905_ (2018).
  * Howard and Matheson [1972] Ronald A. Howard and James E. Matheson. 1972.  Risk-Sensitive Markov Decision Processes.  _Management Science_ 18, 7 (1972), 349–463.
  * Hu et al. [2023] Yifan Hu, Junjie Fu, and Guanghui Wen. 2023.  Graph Soft Actor–Critic Reinforcement Learning for Large-Scale Distributed Multirobot Coordination.  _IEEE Transactions on Neural Networks and Learning Systems_ (2023).
  * Iqbal and Sha [2019] Shariq Iqbal and Fei Sha. 2019.  Actor-Attention-Critic for Multi-Agent Reinforcement Learning. In _Proceedings of the 36th International Conference on Machine Learning_.
  * Iyengar [2005] Garud N. Iyengar. 2005\.  Robust Dynamic Programming.  _Mathematics of Operations Research_ 30, 2 (2005), 257–544.
  * Jacobson [1973] David H. Jacobson. 1973\.  Optimal Stochastic Linear Systems with Exponential Performance Criteria and Their Relation to Deterministic Differential Games.  _IEEE Trans. Automat. Control_ 18, 2 (1973), 124–131.
  * Jacquette [1976] Stratton C. Jacquette. 1976\.  A Utility Criterion for Markov Decision Processes.  _Management Science_ 23, 1 (1976), 43–49.
  * Ma et al. [2020] Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. 2020.  DSAC: Distributional Soft Actor Critic for Risk-Sensitive Reinforcement Learning.  _arXiv preprint arXiv:2004.14547_ (2020).
  * Moos et al. [2022] Janosch Moos, Kay Hansel, Hany Abdulsamad, Svenja Stark, Debora Clever, and Jan Peters. 2022\.  Robust Reinforcement Learning: A Review of Foundations and Recent Advances.  _Machine Learning and Knowledge Extraction_ 4 (2022), 276–315.
  * Nass et al. [2019] David Nass, Boris Belousov, and Jan Peters. 2019\.  Entropic Risk Measure in Policy Search. In _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_.
  * Nesterov [2005] Yu Nesterov. 2005\.  Smooth minimization of non-smooth functions.  _Mathematical Programming_ 103 (2005), 127–152.
  * Noorani and Baras [2021] Erfaun Noorani and John S. Baras. 2021.  Risk-sensitive REINFORCE: A Monte Carlo Policy Gradient Algorithm for Exponential Performance Criteria. In _Proceedings of the 60th IEEE Conference on Decision and Control (CDC)_.
  * Noorani et al. [2022] Erfaun Noorani, Christos Mavridis, and John Baras. 2022\.  Risk-Sensitive Reinforcement Learning with Exponential Criteria.  _arXiv preprint arXiv:2212.09010_ (2022).
  * Osogami [2012] Takayuki Osogami. 2012\.  Robustness and risk-sensitivity in Markov decision processes. In _Advances in Neural Information Processing Systems_.
  * Patek [2001] Stephen D. Patek. 2001\.  On terminating Markov decision processes with a risk-averse objective function.  _Automatica_ 37, 9 (2001), 1379–1386.
  * Prashanth [2014] L. A. Prashanth. 2014\.  Policy Gradients for CVaR-Constrained MDPs. In _International Conference on Algorithmic Learning Theory_.
  * Prashanth and Ghavamzadeh [2013] L. A. Prashanth and Mohammad Ghavamzadeh. 2013.  Actor-Critic Algorithms for Risk-Sensitive MDPs. In _Advances in Neural Information Processing Systems_.
  * Prashanth and Ghavamzadeh [2016] L. A. Prashanth and Mohammad Ghavamzadeh. 2016.  Variance-constrained actor-critic algorithms for discounted and average reward MDPs.  _Machine Learning_ (2016).
  * Sadeghi Eshkevari et al. [2022] Soheil Sadeghi Eshkevari, Xiaocheng Tang, Zhiwei Qin, Jinhan Mei, Cheng Zhang, Qianying Meng, and Jia Xu. 2022\.  Reinforcement Learning in the Wild: Scalable RL Dispatching Algorithm Deployed in Ridehailing Marketplace. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_.
  * Singh et al. [2020] Rahul Singh, Qinsheng Zhang, and Yongxin Chen. 2020\.  Improving Robustness via Risk Averse Distributional Reinforcement Learning. In _Proceedings of the 2nd Conference on Learning for Dynamics and Control (L4DC)_.
  * Sinha et al. [2018] Aman Sinha, Hongseok Namkoong, Riccardo Volpi, and John Duchi. 2018.  Certifying Some Distributional Robustness with Principled Adversarial Training. In _International Conference on Learning Representations (ICLR)_.
  * Sun et al. [2022] Wenjing Sun, Yuan Zou, Xudong Zhang, Ningyuan Guo, Bin Zhang, and Guodong Du. 2022\.  High robustness energy management strategy of hybrid electric vehicle based on improved soft actor-critic deep reinforcement learning.  _Energy_ 258 (2022), 124806.
  * Tamar et al. [2015] Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. 2015\.  Policy Gradient for Coherent Risk Measures. In _Advances in Neural Information Processing Systems_.
  * Tamar et al. [2012] Aviv Tamar, Dotan Di Castro, and Shie Mannor. 2012\.  Policy Gradients with Variance Related Risk Criteria. In _Proceedings of the 29th International Conference on Machine Learning_.
  * Tang et al. [2019] Xiaocheng Tang, Zhiwei Qin, Fan Zhang, Zhaodong Wang, Zhe Xu, Yintai Ma, Hongtu Zhu, and Jieping Ye. 2019\.  A Deep Value-Network Based Approach for Multi-Driver Order Dispatching. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_.
  * Tramèr et al. [2018] Florian Tramèr, Kurakin Alexey, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. 2018.  Ensemble Aversarial Training: Attacks and Defenses. In _International Conference on Learning Representations (ICLR)_.
  * Urpí et al. [2021] Núria Armengol Urpí, Sebastian Curi, and Andreas Krause. 2021\.  Risk-Averse Offline Reinforcement Learning. In _International Conference on Learning Representations (ICLR)_.
  * Vanvuchelen et al. [2020] Nathalie Vanvuchelen, Joren Gijsbrechts, and Robert Boute. 2020\.  Use of Proximal Policy Optimization for the Joint Replenishment Problem.  _Computers in Industry_ 119 (2020), 103239.
  * Whittle [1981] Peter Whittle. 1981\.  Risk-Sensitive Linear/Quadratic/Gaussian Control.  _Advances in Applied Probability_ 13, 4 (1981), 764–777.
  * Wong et al. [2021] Ching-Chang Wong, Shao-Yu Chien, Hsuan-Ming Feng, and Hisasuki Aoyama. 2021.  Motion Planning for Dual-Arm Robot Based on Soft Actor-Critic.  _IEEE Access_ 9 (2021), 26871–26885.
  * Yang et al. [2023] Jiachen Yang, Jipeng Zhang, Meng Xi, Yutian Lei, and Yiwen Sun. 2023.  A Deep Reinforcement Learning Algorithm Suitable for Autonomous Vehicles: Double Bootstrapped Soft-Actor-Critic-Discrete.  _IEEE Transactions on Cognitive and Developmental Systems_ 15, 4 (2023), 2041–2052.
  * Yang et al. [2021] Qisong Yang, Thiago D. Simão, Simon H. Tinemans, and Matthijs T. J. Spaan. 2021.  WCSAC: Worst-Case Soft Actor Critic for Safety-Constrained Reinforcement Learning. In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Zhang et al. [2021] Shangtong Zhang, Bo Liu, and Shimon Whiteson. 2021.  Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning. In _Proceedings of the AAAI Conference on Artificial Intelligence_.

##  Appendix A Details on derivations in Section 2

###  A.1 Approximations

Approximation (i): for $\beta$ close to zero and a real-valued random variable
$X$, we get

$\mathbb{E}\left[e^{\beta
X}\right]=\exp\left(\beta\,\mathbb{E}\left[X\right]+\frac{\beta^{2}}{2}\mathrm{Var}\left[X\right]+\mathcal{O}\left(\beta^{3}\right)\right)\approx
e^{\beta\,\mathbb{E}\left[X\right]}.$

Approximation (ii): for a function $f$, it holds that

$\mathbb{E}\left[f(X)\right]\approx
f\left(\mathbb{E}[X]\right)+\frac{1}{2}f^{\prime\prime}\left(\mathbb{E}[X]\right)\cdot\mathrm{Var}[X].$

With $f(x)=x^{\gamma}$, $f^{\prime}(x)=\gamma x^{\gamma-1}$,
$f^{\prime\prime}(x)=\gamma(\gamma-1)x^{\gamma-2}$, we get

$\mathbb{E}\left[X^{\gamma}\right]\approx\left(\mathbb{E}[X]\right)^{\gamma}+\frac{1}{2}\gamma(\underbrace{\gamma-1}_{\approx
0})\left(\mathbb{E}[X]\right)^{\gamma-2}\mathrm{Var}[X]\approx\left(\mathbb{E}[X]\right)^{\gamma}$

for $\gamma$ close to one.

###  A.2 Policy improvement proof

The proof of Proposition 2 follows the same arguments as the policy
improvement proof in [17]. Since we can always choose
$\pi_{\text{new}}=\pi_{\text{old}}$, it holds that

$J_{\pi_{\text{old}}}\left(\pi_{\text{new}}\left(\cdot\lvert
s_{t}\right)\right)\leq
J_{\pi_{\text{old}}}\left(\pi_{\text{old}}\left(\cdot\lvert
s_{t}\right)\right).$

Consequently,

$\displaystyle\mathbb{E}_{a_{t}\sim\pi_{\text{new}}}\left[\log\pi_{\text{new}}\left(a_{t}\lvert
s_{t}\right)-\frac{1}{\alpha}Q^{\pi_{\text{old}}}_{t}+\log
Z^{\pi_{\text{old}}}\left(s_{t}\right)\right]$
$\displaystyle\leq\mathbb{E}_{a_{t}\sim\pi_{\text{old}}}\left[\log\pi_{\text{old}}\left(a_{t}\lvert
s_{t}\right)-\frac{1}{\alpha}Q^{\pi_{\text{old}}}_{t}+\log
Z^{\pi_{\text{old}}}\left(s_{t}\right)\right].$

Since $Z^{\pi_{\text{old}}}$ depends only on the state, not the action, this
yields

$\displaystyle\mathbb{E}_{a_{t}\sim\pi_{\text{new}}}\left[Q^{\pi_{\text{old}}}_{t}-\alpha\log\pi_{\text{new}}\left(a_{t}\lvert
s_{t}\right)\right]$
$\displaystyle\geq\mathbb{E}_{a_{t}\sim\pi_{\text{old}}}\left[Q^{\pi_{\text{old}}}_{t}-\alpha\log\pi_{\text{old}}\left(a_{t}\lvert
s_{t}\right)\right].$

We assume $\beta<0$ in the following and note later that the case $\beta>0$
works similarly. For $\beta<0$, we get

$\displaystyle\beta\cdot\mathbb{E}_{a_{t}\sim\pi_{\text{new}}}\left[Q^{\pi_{\text{old}}}_{t}-\alpha\log\pi_{\text{new}}\left(a_{t}\lvert
s_{t}\right)\right]$
$\displaystyle\leq\beta\cdot\mathbb{E}_{a_{t}\sim\pi_{\text{old}}}\left[Q^{\pi_{\text{old}}}_{t}-\alpha\log\pi_{\text{old}}\left(a_{t}\lvert
s_{t}\right)\right]$ $\displaystyle\iff$
$\displaystyle\beta\cdot\left(\alpha\mathcal{H}^{\pi_{\text{new}}}_{t}+\mathbb{E}_{a_{t}\sim\pi_{\text{new}}}\left[Q^{\pi_{\text{old}}}_{t}\right]\right)$
$\displaystyle\leq\beta\cdot\left(\alpha\mathcal{H}^{\pi_{\text{old}}}_{t}+\mathbb{E}_{a_{t}\sim\pi_{\text{old}}}\left[Q^{\pi_{\text{old}}}_{t}\right]\right)$
$\displaystyle\iff$
$\displaystyle\exp\left(\beta\alpha\mathcal{H}^{\pi_{\text{new}}}_{t}+\beta\cdot\mathbb{E}_{a_{t}\sim\pi_{\text{new}}}\left[Q^{\pi_{\text{old}}}_{t}\right]\right)$
$\displaystyle\leq\exp\left(\beta\alpha\mathcal{H}^{\pi_{\text{old}}}_{t}+\beta\cdot\mathbb{E}_{a_{t}\sim\pi_{\text{old}}}\left[Q^{\pi_{\text{old}}}_{t}\right]\right)$
$\displaystyle\iff$ $\displaystyle
e^{\beta\alpha\mathcal{H}^{\pi_{\text{new}}}_{t}}\cdot\mathbb{E}_{a_{t}\sim\pi_{\text{new}}}\left[e^{\beta
Q^{\pi_{\text{old}}}_{t}}\right]$ $\displaystyle\leq
e^{\beta\alpha\mathcal{H}^{\pi_{\text{old}}}_{t}}\cdot\mathbb{E}_{a_{t}\sim\pi_{\text{old}}}\left[e^{\beta
Q^{\pi_{\text{old}}}_{t}}\right]$ $\displaystyle\iff$ $\displaystyle
e^{\beta\alpha\mathcal{H}^{\pi_{\text{new}}}_{t}}\cdot\mathbb{E}_{a_{t}\sim\pi_{\text{new}}}\left[\overline{Q}^{\pi_{\text{old}}}_{t}\right]$
$\displaystyle\leq
e^{\beta\alpha\mathcal{H}^{\pi_{\text{old}}}_{t}}\cdot\mathbb{E}_{a_{t}\sim\pi_{\text{old}}}\left[\overline{Q}^{\pi_{\text{old}}}_{t}\right],$

where we used Approximation (i). Repeated application of the Bellman equation
derived before and this inequality yields

$\displaystyle\overline{Q}^{\pi_{\text{old}}}_{t}$ $\displaystyle=e^{\beta
r_{t}}\cdot\mathbb{E}_{s_{t+1}\sim
p}\left[\mathbb{E}_{a_{t+1}\sim\pi_{\text{old}}}\left[\overline{Q}^{\pi_{\text{old}}}_{t+1}\right]\cdot
e^{\beta\alpha\mathcal{H}^{\pi_{\text{old}}}_{t+1}}\right]^{\gamma}$
$\displaystyle\geq e^{\beta r_{t}}\cdot\mathbb{E}_{s_{t+1}\sim
p}\left[\mathbb{E}_{a_{t+1}\sim\pi_{\text{new}}}\left[\overline{Q}^{\pi_{\text{old}}}_{t+1}\right]\cdot
e^{\beta\alpha\mathcal{H}^{\pi_{\text{new}}}_{t+1}}\right]^{\gamma}$
$\displaystyle\;\;\vdots$ $\displaystyle=e^{\beta
r_{t}}\cdot\mathbb{E}_{\left(s_{t+1},...\right)\sim\rho_{\pi_{\text{new}}}}\left[\exp\left(\beta\cdot\sum_{l=1}^{\infty}\gamma^{l}\left(r_{t+l}+\alpha\mathcal{H}^{\pi_{\text{new}}}_{t+l}\right)\right)\right]$
$\displaystyle=\overline{Q}^{\pi_{\text{new}}}_{t}.$

Thus, we obtain

$Q^{\pi_{\text{old}}}_{t}=\frac{1}{\beta}\log\overline{Q}^{\pi_{\text{old}}}_{t}\leq\frac{1}{\beta}\log\overline{Q}^{\pi_{\text{new}}}_{t}=Q^{\pi_{\text{new}}}_{t},$

which proves the policy improvement for $\beta<0$. The proof for $\beta>0$
works analogously (twice, the $\leq$ or $\geq$-sign does not turn).

##  Appendix B Results when learning $\overline{Q}$

Figure 6 depicts the convergence behavior of the risk-sensitive algorithm that
is based on learning $\overline{Q}$ as opposed to $Q$. For negative values of
$\beta$, the plot looks similar to the one for $\beta=0.01$, i.e., the
algorithm converges to zero rewards (we do not include the results for
$\beta<0$ since they overlap too much with the results for $\beta=0.01$).
While the training curve is reasonable for $\beta=0.1$, we only obtain a “do
not move” policy with zero rewards as $\beta\rightarrow 0$ and for $\beta<0$.

Figure 6: Convergence behavior of the risk-sensitive algorithm based on
learning $\overline{Q}$ compared to risk-neutral SAC.

##  Appendix C Reformulating target in critic loss for numerically stable log-
sum-exp computation

The critic loss function reads

$\displaystyle J_{Q}(\theta)$
$\displaystyle=\mathbb{E}_{(s,a,r,d,s^{\prime})\sim
D}\left[\frac{1}{2}\left(Q_{\theta}(s,a)-\hat{Q}\right)^{2}\right],$
$\displaystyle\hat{Q}$
$\displaystyle=\frac{1}{\beta}\log\Bigl{(}\pi_{\phi}(s^{\prime})^{T}\Bigr{.}$
$\displaystyle\hskip
31.2982pt\Bigl{.}\cdot\exp\left(\beta\left(r+\gamma\left(Q_{\overline{\theta}}\left(s^{\prime}\right)-\alpha\cdot\pi_{\phi}(s^{\prime})^{T}\log\pi_{\phi}(s^{\prime})\right)\right)\right)\Bigr{)}.$

The computation of this target $\hat{Q}$ has a log-sum-exp structure, which is
known to be numerically unstable when implemented naively. Thus, we rewrite it
using the following trick, based on Section 5.2 of [28]: for some variable
$x_{a}$ that depends on the action $a$, we define $y_{a}=x_{a}-\overline{x}$,
with $\overline{x}$ independent of $a$, and get

$\frac{1}{\beta}\log\left(\sum_{a}\pi(a\lvert s)\cdot e^{\beta
x_{a}}\right)=\overline{x}+\frac{1}{\beta}\log\left(\sum_{a}\pi(a\lvert
s)\cdot e^{\beta y_{a}}\right).$

We choose $\overline{x}=\max_{a}x_{a}$ and use this identity to rewrite
$\hat{Q}$ as follows:

$\displaystyle\hat{Q}$
$\displaystyle=r-\gamma\alpha\cdot\pi_{\phi}(s^{\prime})^{T}\log\pi_{\phi}(s^{\prime})+\gamma\max_{a^{\prime}}Q_{\overline{\theta}}(s^{\prime},a^{\prime})$
$\displaystyle\quad+\frac{1}{\beta}\log\left(\sum_{a^{\prime}}\pi_{\phi}(a^{\prime}\lvert
s^{\prime})\cdot\exp\left(\beta\gamma\left(Q_{\overline{\theta}}(s^{\prime},a^{\prime})-\max_{a^{\prime}}Q_{\overline{\theta}}(s^{\prime},a^{\prime})\right)\right)\right).$

##  Appendix D Pseudocode

Algorithm 1 Risk-sensitive discrete Soft Actor-Critic

1:Initialize NN parameters $\phi,\theta_{i},\overline{\theta}_{i}$ for
$i\in\\{1,2\\}$

2:Initialize an empty replay buffer $D$

3:Initialize the environment, observe $s$

4:for each iteration do

5: $a\sim\pi_{\phi}(a\lvert s)$

6: Execute $a$ in the environment and observe $r$, $s^{\prime}$

7: $D\leftarrow D\cup\\{(s,a,r,s^{\prime})\\}$

8: $s\leftarrow s^{\prime}$

9: Sample a batch of transitions from $D$

10:
$\theta_{i}\leftarrow\theta_{i}-\lambda\cdot\nabla_{\theta_{i}}J_{Q}(\theta_{i})$
for $i\in\\{1,2\\}$

11:
$\overline{\theta}_{i}\leftarrow(1-\tau)\cdot\overline{\theta}_{i}+\tau\cdot\theta_{i}$
for $i\in\\{1,2\\}$

12: $\phi\leftarrow\phi-\lambda\cdot\nabla_{\phi}J_{\pi}(\phi)$

13:end for

We provide the pseudocode for our risk-sensitive discrete SAC algorithm in
Algorithm 1. Here, $\theta_{i}$ for $i\in\\{1,2\\}$ refers to the parameters
of the two critics which we train concurrently as explained in Section 2.1.
Besides, $\lambda$ is the learning rate and $\tau$ is the smoothing factor for
the exponential moving average used to update the target critic parameters.

##  Appendix E Details on experiments

In the following, we firstly provide details on our state encoding and NN
architectures. Secondly, we report the hyperparameters used in our
experiments.

###  E.1 State encoding and neural networks

The state of our environment can be naturally encoded as a 5x5 image with one
channel per type of element contained in the system: the target location, the
agent, and the items. The target location channel has only zero entries except
for the target location, which we represent by a one. The agent channel has
only zero entries except for the agent’s current location. At this location,
the entry is 0.5 if the agent has picked up but not yet delivered an item.
Otherwise, this entry is one. The items channel has only zero entries except
for locations with items that have not been picked up and did not disappear
because the maximum response time elapsed. Such an entry is equal to the
number of time steps remaining until the respective item will disappear,
normalized by the maximum response time.

We use a combination of convolutional and fully connected layers in our NNs.
Except for the output activation function, all NNs have the same architecture.
We use the following sequence of layers:

  * •

Convolutional layer 1: 2D convolutional layer with 32 filters of size 3x3 with
stride one, same padding and ReLU activation

  * •

Convolutional layer 2: 2D convolutional layer with 64 filters of size 2x2 with
stride one, same padding and ReLU activation

  * •

Convolutional layer 3: 2D convolutional layer with 64 filters of size 2x2 with
stride one, same padding and ReLU activation

  * •

Flatten layer

  * •

Fully connected layer 1: fully connected layer with 256 units and ReLU
activation

  * •

Fully connected layer 2: fully connected layer with 256 units and ReLU
activation

  * •

Output layer: fully connected layer with 5 units (for the 5 possible actions)
and softmax activation for the actor network and linear activation for the
critic networks, respectively

We use L2 regularization for the NN parameters with a regularization
coefficient of $10^{-4}$.

###  E.2 Hyperparameters

We train for 2 million steps, update the network parameters every 20 steps,
and test the performance of the current policy on the validation data every
5,000 steps. During the first 20,000 steps, we collect experience with a
random policy and do not update the network parameters.

We set the discount factor to 0.99. We sample batches of size 512 from a
replay buffer with maximum size 200,000. When we sample transitions from the
replay buffer, we normalize the sampled rewards by dividing them by the
standard deviation of all rewards currently stored in the replay buffer. For
the critic loss, we use the Huber loss with a delta value of 2 instead of the
squared error. Moreover, we use gradient clipping with a clipping ratio of 10
for actor and critic gradients. To update the NN parameters, we use the Adam
optimizer with a learning rate of $3\cdot 10^{-4}$. For the update of the
target critic parameters, we use an exponential moving average with smoothing
factor $5\cdot 10^{-3}$.

We tune the entropy coefficient individually per experiment and use values
between 0.1 and 0.3 across our experiments. After 800,000 training steps, we
set it to zero as explained in Section 3.2.

##  Appendix F L2 regularization

The non-robust discrete SAC algorithm already uses L2 regularization with an
L2 regularization coefficient of 0.0001, since we find through hyperparameter
tuning that this maximizes performance on the validation data from the
gradient 1 distribution. To investigate if L2 regularization improves the
algorithm’s robustness against distribution shifts, we increase the L2
regularization coefficient. Figure 7 shows the experimental results. We also
tried even larger values for the L2 regularization coefficient than the ones
depicted here, but find that they lead to learning a “do not move” policy with
zero rewards due to over-regularization. Based on the results in Figure 7, we
conclude that L2 regularization does not improve the robustness of SAC against
distribution shifts within our environment.

Figure 7: Performance of L2 regularization to improve robustness. The left
plot shows the performance on the test data for the training distribution. The
right plot shows the average over all other distributions, i.e., the
performance under distribution shifts. The data points for an L2
regularization coefficient of 0.0001 show the results for the non-robust
discrete SAC algorithm. We report all results relative to SAC’s performance
when trained on the shifted, i.e., the true distribution as explained in
Section 3.3.

##  Appendix G Combination of risk-sensitivity and entropy regularization

We combine our risk-sensitive algorithm with entropy regularization to
evaluate if this combination improves upon the two approaches’ individual
performance. Figure 8 shows the experimental results. None of the tested
configurations for $\beta$ and $\alpha$ improves upon the best results for
pure risk-sensitivity or pure entropy regularization in Figure 4.

Figure 8: Performance of combining our risk-sensitive algorithm with entropy
regularization to further improve robustness. The top row shows the
performance on the test data for the training distribution. The bottom row
shows the average over all other distributions, i.e., the performance under
distribution shifts. The left-most data point in each plot shows the results
for our risk-sensitive algorithm without entropy regularization. There are no
data points for $\alpha\in\\{0.15,0.2\\}$ in the right-most plots, because the
corresponding training runs converge to a validation reward that is
substantially worse than the greedy performance on the gradient 1
distribution. We report all results relative to SAC’s performance when trained
on the shifted, i.e., the true distribution as explained in Section 3.3.