none [ALA ’23]Proc. of the Adaptive and Learning Agents Workshop (ALA 2023)
May 29-30, 2023London, UK, https://alaworkshop2023.github.io/Cruz, Hayes,
Wang, Yates (eds.) 2023 2023 printacmref=false 60 Imperial College London
London United Kingdom Imperial College London London United Kingdom

# Approximate Shielding of Atari Agents for Safe Exploration

Alexander W. Goodall<EMAIL_ADDRESS>and  Francesco Belardinelli
<EMAIL_ADDRESS>
###### Abstract.

Balancing exploration and conservatism in the constrained setting is an
important problem if we are to use reinforcement learning for meaningful tasks
in the real world. In this paper, we propose a principled algorithm for _safe
exploration_ based on the concept of _shielding_. Previous approaches to
shielding assume access to a safety-relevant abstraction of the environment or
a high-fidelity simulator. Instead, our work is based on _latent shielding_ \-
another approach that leverages world models to verify policy roll-outs in the
latent space of a learned dynamics model. Our novel algorithm builds on this
previous work, using safety critics and other additional features to improve
the stability and farsightedness of the algorithm. We demonstrate the
effectiveness of our approach by running experiments on a small set of Atari
games with state dependent safety labels. We present preliminary results that
show our approximate shielding algorithm effectively reduces the rate of
safety violations, and in some cases improves the speed of convergence and
quality of the final agent.

###### Key words and phrases:

Safe Reinforcement Learning, Formal Verification, World Models

##  1\. Introduction

Reinforcement learning (RL) Sutton and Barto (2018) has become a principled
and powerful tool for training agents to complete tasks in complex and dynamic
environments. While RL promises a lot in theory, it unfortunately comes with
no guarantees on worst-case performance. In safety-critical applications such
as healthcare, robotics, autonomous driving and industrial control systems, it
is imperative that decision making algorithms avoid unsafe or harmful
situations Amodei et al. (2016). Formal verification Baier and Katoen (2008)
poses as a mathematically precise technique for verifying system performance
and can be used to verify that learned policies respect safety-constraints
during training and deployment.

Recently there has been increasing interest in applying model-based RL (MBRL)
algorithms in the constrained setting. This increase in interest can be
attributed in part to exciting developments in MBRL Hafner et al. (2021, 2023)
and the superior sample complexity of model-based approaches Hafner et al.
(2020); Janner et al. (2019). With better sample-complexity, MBRL algorithms
should in theory commit far fewer safety violations during training than their
model-free counterparts. This is important in the problem of _safe
exploration_ Amodei et al. (2016) where collecting experience is costly and
unsafe behaviour can lead to catastrophic consequences in the real world.

In this work we focus on a method for safe exploration called _shielding_
Alshiekh et al. (2018); Jansen et al. (2018). In its original form, shielding
forces hard constraints on the actions performed by the agent to ensure that
the agent stays within a verified boundary on the state space. To compute this
boundary we typically require a safety-relevant abstraction of the environment
that is compact enough to efficiently perform exact verification techniques.
Instead we opt to be less restrictive and make minimal assumptions about what
we have access to a priori. As in previous work He et al. (2022), we only
assume that there exists some expert labelling of the states and we do not
have access to a compact model or a safety-relevant abstraction of the
environment. The key motivation for making these minimal assumptions is to
obtain a more general algorithm that can be applied in many real-life
applications where an abstraction is typically not available, as the system
might be too complex or unknown in advance.

Figure 1. Simple grid-world with goal (Eve) in the top right corner. With a
Manhattan distance look-ahead of 2 (blue squares) Wall-E is forever doomed to
fall into the acid during exploration, as he is unable to determine that the
conveyor belt leads to an unavoidable unsafe state. With safety critics Wall-E
can learn the cost value of the conveyor belt squares and avoid them without a
further look-ahead horizon. *

* This image was created with the assistance of DALL·E 2 A simple grid world environment were Wall-E the agent is trying to get to Eva the goal. Wall-E only has a Manhattan look-ahead horizon of 2. There is a conveyor belt with length 5 that ends in acid, without a large look-ahead distance wall-E is doomed to fall into the acid if he steps on to the conveyor belt during exploration.

Bounded prescience shielding (BPS) Giacobbe et al. (2021) is an approach to
shielding that removes the requirement of access to a compact representation
or abstraction of the environment. Instead, BPS assumes access to a black-box
simulator of the environment which can be queried for look-ahead shielding of
the learned policy. Giacobbe et al. demonstrated that pre-trained state-of-
the-art model-free Atari agents consistently violate safety-constraints
provided by domain experts. And that BPS with a look-ahead horizon of $H=5$
reduced the rate of several shallow safety properties. In this paper we use
the same state-dependent safety-labels for Atari games provided by Giacobbe et
al., although we note that our approach has several distinct advantages over
BPS: (1) we do not assume access to a black-box model to assist decision
making, (2) we are able to apply our shielding algorithm during training
without substantial computational overhead, (3) we are able to look-ahead
further into the future ($>5$) with deeper model roll-outs and safety critics.

In a similar fashion to _latent shielding_ He et al. (2022), we use world
models Ha and Schmidhuber (2018); Hafner et al. (2020, 2021) to learn a
dynamics model of the environment for policy optimisation and approximate
shielding. The key differences between _latent shielding_ and our approach are
outlined in Section 3. Most notably we utilise safety critics, which are
crucial for obtaining further look-ahead ability without explicitly increasing
the shielding horizon, see Fig. 1.

#### Contributions

Our main contributions are summarised as follows: (1) we augment _latent
shielding_ He et al. (2022) with safety critics used to bootstrap the end of
imagined trajectories for look-ahead shielding further into the future, (2) we
use twin delayed target critics to reduce the overestimation of expected costs
and reduce overly conservative behaviour, (3) we ground our approach in a
logical formalism, namely probabilistic computation tree logic (PCTL) Baier
and Katoen (2008), (4) we derive PAC-style bounds on the probability of
accurately estimating a constraint violation under the assumption of perfect
transition dynamics, (5) we empirically show that our approach dramatically
reduces the rate of safety violations on a small set of Atari games with
state-dependent safety labels and in some cases our algorithm greatly improves
the speed of convergence and quality of the learned policy with respect to
accumulated reward.

##  2\. Preliminaries

In this section we describe the relevant background material and notation
required to understand the main results of this paper. We start by introducing
the problem setup and the specification language used to formalise the notion
of safety used throughout this paper. We then continue with an outline of the
world model components and the policy optimisation scheme.

###  2.1. Problem Setup

Atari games in the Arcade Learning Environment (ALE) Machado et al. (2018) are
built on top of the Atari 2600 Stella emulator. The emulator manipulates
128-bytes of RAM which represent the underlying state of the game. However,
Agents typically only observe $3\times 210\times 160$ dimensional tensors
representing each of the pixel values of the screen. Therefore, we model the
system as a partially observed Markov decision processes (POMDP) Puterman
(1990), which in this case is more appropriate than the traditional MDP
formulation.

For our purposes we also extend the POMDP tuple with an additional labelling
function Baier and Katoen (2008). Formally, we define a POMDP as a tuple
$\mathcal{M}=(S,A,p,\iota_{init},R,\Omega,O,AP,L)$ where, $S$ is a finite set
of states, $A$ is a finite set of actions, $p:S\times A\times S\to[0,1]$ is
the probabilistic state-action transition function, $\iota_{init}:S\to[0,1]$
is the initial state distribution such that $\sum_{s\in S}\iota_{init}(s)=1$,
$R:S\times A\to\mathbb{R}$ is the reward function, $\Omega$ is a finite set of
observations, $O:S\times A\times\Omega\to[0,1]$ is the observation
probabilistic function, which defines the probability of an observation
conditional on the previous state-action pair, AP is a set of atomic
propositions which maps to the set of states by an ‘expert’ labelling function
$L:S\to 2^{\textit{AP}}$.

In particular, at each timestep $t$ the agent receives an observation
$o_{t}\in\Omega$, a reward $r_{t}$ and a set of labels $L(s_{t})\in
2^{\textit{AP}}$. Given some state formula $\Phi$, the agent can determine if
the underlying state $s_{t}$ satisfies $\Phi$ with the following relation,

$\begin{array}[]{@{}r@{{}\mathrel{}}c@{\mathrel{}{}}l@{}}s\models\textrm{true}&\text{for
all}&s\in S\\\ s\models a&\text{iff}&a\in L(s)\\\
s\models\neg\Phi&\text{iff}&s\not\models\Phi\\\
s\models\Phi_{1}\land\Phi_{2}&\text{iff}&s\models\Phi_{1}\land
s\models\Phi_{2}\end{array}$

The goal is to find a policy $\pi$ that maximises expected reward, i.e.
$\pi^{*}=\arg\max_{\pi}\mathbb{E}[\sum_{t=1}^{\infty}\gamma^{t-1}R(s_{t},\pi(s_{t}))]$,
while minimising violations of the state formula $\Phi$ (that encodes the
safety-constraints) during training. Here $\gamma$ is the discount factor
Sutton and Barto (2018).

###  2.2. Probabilistic Computation Tree Logic

Probabilistic computation tree logic (PCTL) is a branching time temporal logic
that extends CTL with probabilistic quantifiers Baier and Katoen (2008). PCTL
is particularly useful for specifying reachability and safety properties for
discrete stochastic systems which makes it useful for our purposes. A valid
PCTL formula can be constructed as follows,

$\displaystyle\Phi::=$ $\displaystyle\textrm{true}\mid
a\mid\neg\Phi\mid\Phi\land\Phi\mid\mathbb{P}_{J}(\phi)$ $\displaystyle\phi::=$
$\displaystyle X\Phi\mid\Phi U\Phi\mid\Phi U^{\leq n}\Phi$

where $a\in AP$ is an atomic proposition, negation ($\neg$) and conjunction
($\land$) are the familiar logical operators, $J\subset[0,1]$,
$J\neq\varnothing$ is a non-empty subset of the unit interval, and next ($X$),
until ($U$) and bounded until ($U^{\leq n}$) are temporal operators. We
distinguish here between state formula $\Phi$ and path formula $\phi$ which
are interpreted over states and paths respectively.

We write $s\models\Phi$ to indicate that a state $s$ satisfies a state formula
$\Phi$, where the satisfaction relation is defined as before, see Baier and
Katoen (2008) for details. Similarly, we can define the satisfaction relation
for path formula $\phi$, this is given in the next section for the specific
fragment of PCTL that we require. Also note that the common operators
eventually ($\lozenge$) and always ($\square$) and their bounded counter parts
($\lozenge^{\leq n}$ and $\square^{\leq n}$) can be defined in a familiar way,
see Baier and Katoen (2008).

The reason behind using PCTL as our safety specification language is because
it allows us to meaningfully trade-off safety and progress by specifying the
probability with which we force the agent to satisfy to a given temporal logic
formula.

###  2.3. Bounded Safety

The notion of bounded safety for Atari agents introduced by Giacobbe et al.
can be straightforwardly grounded in PCTL. Consider some fixed (stochastic)
policy $\pi:O\times A\to[0,1]$ and POMDP
$\mathcal{M}=(S,A,p,\iota_{init},R,\Omega,O,AP,L)$. Together $\pi$ and
$\mathcal{M}$ define a transition system $\mathcal{T}:S\times S\to[0,1]$,
where $\sum_{s^{\prime}\in S}\mathcal{T}(s,s^{\prime})=1$. A finite trace with
length $n$ of the transition system $\mathcal{T}$, is a sequence of states
$s_{0}\to s_{1}\to...\to s_{n}$ denoted $\tau$, the $i^{\textrm{th}}$ state of
$\tau$ is given by $\tau[i]$. A trace $\tau$ satisfies bounded safety if and
only if all of its states satisfy the state formula $\Phi$ that encodes the
safety constraints. Formally,

$\displaystyle\tau\models\square^{\leq n}\Phi$ iff $\displaystyle\text{for all
}0\leq i\leq n,\tau[i]\models\Phi$ (1)

for some bounded look-ahead $n$. Now in PCTL we can say that a state $s\in S$
satisfies $\varepsilon$-bounded safety as follows,

$s\models\mathbb{P}_{1-\varepsilon}(\square^{\leq n}\Phi)\;\;\text{iff}\\\
\mu_{s}(\\{\tau\mid\tau[0]=s,\text{ for all }0\leq i\leq
n,\tau[i]\models\Phi\\})\in[1-\varepsilon,1]$ (2)

where $\mu_{s}$ is a well-defined probability measure induced by the
transition system $\mathcal{T}$, over the set of traces staring from $s$ and
with finite length $n$, see Baier and Katoen (2008) for details. We denote
$\mu_{s\models\phi}$ as shorthand for the measure
$\mu_{s}(\\{\tau\mid\tau[0]=s,\text{ for all }0\leq i\leq
n,\tau[i]\models\Phi\\})$, where $\phi::=\square^{\leq n}\Phi$ is the path
formula we care about. By framing bounded safety in this way, we obtain a
meaningful way to trade off safety and progress with the $\varepsilon$
parameter.

###  2.4. World Models

To learn a world model for behaviour learning and look-ahead shielding we
leverage DreamerV2 Hafner et al. (2021), which was used to master Atari games
in the ALE Machado et al. (2018). DreamerV2 is composed of the following
components: an image encoder $z_{t}\sim q_{\theta}(z_{t}\>|\>o_{t},h_{t})$
that learns a posterior latent representation conditional on the current
observation $o_{t}$ and recurrent state $h_{t}$, the recurrent state space
model (RSSM) Hafner et al. (2019) which is a mixture of deterministic and
stochastic categorical latents, and the image, reward and discount predictors.

The RSSM consists of two main components: the recurrent model
$h_{t}=f_{\theta}(h_{t-1},z_{t-1},a_{t-1})$, which computes the next
deterministic latents given the past state $s_{t-1}=(h_{t-1},z_{t-1})$ and
action $a_{t-1}$, and the transition predictor $\hat{z}_{t}\sim
p_{\theta}(\hat{z}_{t}\>|\>h_{t})$, which is used as the prior distribution
over the stochastic latents conditional on the deterministic latents.

The image predictor or decoder $\hat{o}_{t}\sim
p_{\theta}(\hat{o}_{t}\>|\>h_{t},z_{t})$ is trained to predict the current
observation $o_{t}$ with a reconstruction loss. The image predictor provides
useful self-supervised gradients that help the world model learn a structured
latent space for effective policy optimisation Hafner et al. (2021). The
reward predictor $\hat{r}_{t}\sim p_{\theta}(\hat{r}_{t}\>|\>h_{t},z_{t})$ and
discount predictor $\hat{\gamma}_{t}\sim
p_{\theta}(\hat{\gamma}_{t}\>|\>h_{t},z_{t})$, also provide useful self-
supervised gradients. However, they are primarily used to construct targets
for policy optimisation.

All components of the world model are implemented as neural networks and
jointly trained with backpropagation and straight through gradients Bengio et
al. (2013). In addition, KL-balancing Hafner et al. (2021) is used to stop the
prior and posterior being regularised at the same rate to prevent instability
during training.

###  2.5. Behaviour Learning

In DreamerV2 Hafner et al. (2021), policy optimisation is performed entirely
on experience ‘imagined’ by rolling out the world model with a fixed
(stochastic) policy. A replay buffer $\mathcal{D}$ is used to retain
experience from the real environment. At each training step a batch $B$ is
sampled from the replay buffer $\mathcal{D}$ and the RSSM is used to sample
sequences of compact latent states $\hat{s}_{1:H}$, using each of the
observations in $B$ as a starting point. Here $H$ refers to the ‘imagination’
horizon, which is typically set to a relatively small number ($H=15$) to avoid
compounding model errors that are likely to harm the learned policy.

The task policy $\pi^{\textrm{task}}$ parameterised by $\psi^{\textrm{task}}$
is trained to maximise accumulated reward. In addition, a task critic
$v^{\textrm{task}}$ parameterised by $\xi^{\textrm{task}}$ is used to guide
the learning of the policy. TD-$\lambda$ targets Sutton and Barto (2018) are
constructed by rolling out the world model with the task policy
$\pi^{\textrm{task}}$,

$V_{t}^{\text{task},\lambda}=\hat{r}_{t}+\hat{\gamma}_{t}\begin{cases}(1-\lambda)v^{\text{task}}(\hat{s}_{t+1})+\lambda
V_{t+1}^{\text{task},\lambda}&\text{if $t<H$,}\\\
v^{\text{task}}(\hat{s}_{H})&\text{if $t=H$}\end{cases}$ (3)

The $\lambda$ parameter trades of the bias and variance of the estimate, with
$\lambda=0.0$ giving the high variance n-step Monte-Carlo return and
$\lambda=1.0$ giving the high bias one-step return. The task critic
$v^{\textrm{task}}$ is regressed towards the value estimates with the
following loss function,

$\mathcal{L}(\xi^{\text{task}})=\mathbb{E}_{\pi^{\text{task}},p_{\theta}}\left[\sum^{H-1}_{t=1}\frac{1}{2}(v^{\text{task}}(\hat{s}_{t})-sg(V_{t}^{\text{task},\lambda}))^{2}\right]$
(4)

where the $sg(\cdot)$ operator stops the flow of gradients to the input
argument. The task policy $\pi^{\text{task}}$ is trained with reinforce
gradients Sutton and Barto (2018) and an entropy regulariser to encourage
exploration. In addition, the difference of the TD-$\lambda$ targets
$V^{\text{task},\lambda}$ and the critic estimates $v^{\text{task}}$ are used
as a baseline to reduce the variance of the reinforce gradients. This gives
the following loss function for the task policy $\pi^{\text{task}}$,

$\mathcal{L}(\psi^{\text{task}})=\\\
\mathbb{E}_{\pi^{\text{task}},p_{\theta}}\bigg{[}\sum^{H-1}_{t=1}\underbrace{-\log\pi^{\text{task}}(a_{t}\>|\>\hat{s}_{t})sg(V_{t}^{\text{task},\lambda}-v^{\text{task}}(\hat{s}_{t}))}_{\text{reinforce}}\\\
\underbrace{-\eta
H(\pi^{\text{task}}(\cdot\>|\>\hat{s}_{t}))}_{\text{entropy}}\bigg{]}$ (5)

##  3\. Approximate Shielding

In this section we introduce our approximate shielding algorithm for Atari
agents. The general idea is to learn a world model for task policy
optimisation, safe policy synthesis and bounded look-ahead shielding. The
world model of choice is DreamerV2 Hafner et al. (2021) which has demonstrated
state-of-the-art performance on the Atari benchmark. While our approach is
similar to _latent shielding_ He et al. (2022) we note that it has following
key differences:

  * •

We learn a cost predictor to estimate state dependent costs, rather than a
labelling function $L_{\vartheta}:S\to\\{\textrm{safe},\textrm{unsafe}\\}$.

  * •

We train a safe policy to minimise expected costs, which is used as the backup
policy if a safety-violation is detected.

  * •

We use safety critics to obtain further look-ahead capabilities without having
to roll-out the world model further into the future.

  * •

We don’t need to use intrinsic punishment Alshiekh et al. (2018) or any sort
of shield introduction schedule He et al. (2022).

  * •

We test our approach in a much more sophisticated domain, specifically the ALE
Machado et al. (2018).

In what follows, we describe the notable components used in our approach,
followed by a precise description of the shielding procedure and an outline of
the full learning algorithm. However, we will first present some PAC-style
bounds on the probability of accurately predicting a constraint violation
using our the shielding procedure. It should then become clear in what sense
our algorithm approximate. Specifically, ‘approximate’ comes from the fact
that we use a learned approximation of the true environment dynamics and we
use Monte-Carlo estimation to predict constraint violations.

###  3.1. Probabilistic Guarantees

Recall that to ensure $\varepsilon$-bounded safety we are interested in
verifying PCTL formula of the form $\mathbb{P}_{1-\varepsilon}(\square^{\leq
n}\Phi)$, where $\Phi$ is the state formula that encodes the safety-
constraints. To do so we fix the task policy $\pi^{\textrm{task}}$ and the
learned world model $p_{\theta}$ to obtain an approximate transition system
$\widehat{\mathcal{T}}:S\times S\to[0,1]$. Even with the ‘true’ transition
system $\mathcal{T}:S\times S\to[0,1]$, exact PCTL verification is
$O(\textrm{poly}(size(\mathcal{T}))\cdot n\cdot|\Phi|)$, which is much too big
for Atari games. Instead we rely on Monte-Carlo estimation of the measure
$\mu_{s\models\phi}$ by sampling traces $\tau$ from the approximate transition
system $\widehat{\mathcal{T}}$.

######  Proposition 3.1.

Given access to the ‘true’ transition system $\mathcal{T}$, with probability
$1-\delta$ we can estimate the measure $\mu_{s\models\phi}$ up to some
approximation error $\epsilon$, by sampling $m$ traces $\tau\sim\mathcal{T}$,
provided,

$m\geq\frac{1}{2\epsilon^{2}}\log\left(\frac{2}{\delta}\right)$ (6)

###### Proof.

The proof is a straightforward application of Hoeffding’s inequality. We can
estimate $\mu_{s\models\phi}$ by sampling $m$ traces
$\langle\tau_{j}\rangle^{m}_{j=1}$ from $\mathcal{T}$. Let $X_{1},...,X_{m}$
be indicator r.v.s such that,

$X_{j}=\begin{cases}1&\text{if $\tau_{j}\models\square^{\leq n}\Phi$,}\\\
0&\text{otherwise}\end{cases}$ (7)

Let,

$\hat{\mu}_{s\models\phi}=\frac{1}{m}\sum^{m}_{j=1}X_{j},\;\text{where}\;\>\mathbb{E}_{\mathcal{T}}[\hat{\mu}_{s\models\phi}]=\mu_{s\models\phi}$
(8)

Then by Hoeffding’s inequality,

$\mathbb{P}\left[|\hat{\mu}_{s\models\phi}-\mu_{s\models\phi}|\geq\epsilon\right]\leq
2\exp\left(-2m\epsilon^{2}\right)$

Bounding the RHS from above with $\delta$ and rearranging completes the proof.
∎

With these probabilistic guarantees we can set $m$ appropriately for some
domain specific requirements. In the following proposition, we demonstrate how
we may be sure that a given state $s$ satisfies $\varepsilon$-bounded safety
given our estimate $\hat{\mu}_{s\models\phi}$.

######  Proposition 3.2.

Suppose we have an estimate
$\hat{\mu}_{s\models\phi}\in[\mu_{s\models\phi}-\epsilon,\mu_{s\models\phi}+\epsilon]$,
if $\hat{\mu}_{s\models\phi}\in[1-\varepsilon+\epsilon,1]$ then it must be the
case that $\mu_{s\models\phi}\in[1-\varepsilon,1]$ and that
$s\models\mathbb{P}_{1-\varepsilon}(\square^{\leq n}\Phi)$.

Note that $\varepsilon$ is the bounded safety parameter used to trade-off
exploration and progress and $\epsilon$ is the approximation error from
Proposition 6.

###### Proof.

Suppose
$\hat{\mu}_{s\models\phi}\in[\mu_{s\models\phi}-\epsilon,\mu_{s\models\phi}+\epsilon]$,
$\hat{\mu}_{s\models\phi}\in[1-\varepsilon+\epsilon,1]$ and
$\mu_{s\models\phi}\not\in[1-\varepsilon,1]$. Then
$\hat{\mu}_{s\models\phi}-\mu_{s\models\phi}>\epsilon$ which contradicts
$\hat{\mu}_{s\models\phi}\in[\mu_{s\models\phi}-\epsilon,\mu_{s\models\phi}+\epsilon]$.
This implies that indeed $\mu_{s\models\phi}\in[1-\varepsilon,1]$ and that
$s\models\mathbb{P}_{1-\varepsilon}(\square^{\leq n}\Phi)$ by Eq. 2. ∎

It is important to note that checking the condition
$\hat{\mu}_{s\models\phi}\in[1-\varepsilon+\epsilon,1]$ could lead to overly
conservative behaviour, if $\epsilon$ is not very small. This is because for
$\mu_{s\models\phi}\in[1-\varepsilon,1-\varepsilon+\epsilon]$ we may falsely
predict that $s\not\models\mathbb{P}_{1-\varepsilon}(\square^{\leq n}\Phi)$
with some probability up to $1-\delta$. Instead we could check that
$\hat{\mu}_{s\models\phi}\in[1-\varepsilon-\epsilon,1]$, although this may
lead to overly permissive behaviour. In words, the former configuration admits
no false positives and the latter admits no false negatives (with probability
$1-\delta$). Either configuration can be used, although we opt for the former.

To get similar bounds for the approximate transition system
$\widehat{\mathcal{T}}$ we can try to get a bound on the total variation (TV)
distance between $\widehat{\mathcal{T}}$ and $\mathcal{T}$. However, this is
left for future work.

###  3.2. RSSM with Costs

We augment the RSSM of DreamerV2 Hafner et al. (2021) with a cost predictor
$\hat{c}_{t}\sim p_{\theta}(\hat{c}_{t}\>|\>h_{t},z_{t})$ used to predict
state dependent costs and a safety-discount predictor
$\hat{\gamma}^{\text{safe}}_{t}\sim
p_{\theta}(\hat{\gamma}^{\text{safe}}_{t}\>|\>h_{t},z_{t})$ which is used to
help improve the stability of the safety critics.

In the same fashion as the reward predictor $\hat{r}_{t}\sim
p_{\theta}(\hat{r}_{t}\>|\>h_{t},z_{t})$ the cost predictor $\hat{c}_{t}\sim
p_{\theta}(\hat{c}_{t}\>|\>h_{t},z_{t})$ parameterises a Gaussian
distribution. We construct targets for the cost predictor as follows,

$c_{t}=\begin{cases}0,&\text{if $s_{t}\models\Phi$}\\\
C,&\text{otherwise}\end{cases}$ (9)

where $s_{t}$ refers to the true underlying state of the environment, $\Phi$
is the state formula that encodes the safety-constraints, and $C>0$ is an
arbitrary hyperparameter that determines the cost incurred at a violating
state. Using a cost predictor in this way allows the agent to distribute its
uncertainty about a constraint violation over several consecutive states.

The safety-discount predictor $\hat{\gamma}^{\text{safe}}_{t}\sim
p_{\theta}(\hat{\gamma}^{\text{safe}}_{t}\>|\>h_{t},z_{t})$ is a binary
classifier trained, in a similar way, to predict if a state is violating or
not. We construct targets for the safety-discount predictor as follows,

$\gamma^{\text{safe}}_{t}=\begin{cases}\gamma,&\text{if $s_{t}\models\Phi$}\\\
0,&\text{otherwise}\end{cases}$ (10)

The key purpose of the safety-discount factor is to reduce the overestimation
of the expected costs by the safety critics. Using the safety-discount
predictions $\hat{\gamma}^{\textrm{safe}}_{t}$ to construct targets for the
safety critics, instead of the usual discount predictions $\hat{\gamma}_{t}$,
effectively transforms the MDP into one where violating states are terminal
states. This means the safety critics should always be upper bounded by $C$.
The full RSSM loss function can now be written as follows,

$\mathcal{L}(\theta)=\mathcal{L}_{\text{image}}+\mathcal{L}_{\text{reward}}+\mathcal{L}_{\text{discount}}+\mathcal{L}_{\text{cost}}\\\
+\mathcal{L}_{\text{safe-discount}}+\mathcal{L}_{\text{KL-B}}$ (11)

###  3.3. Safe Policy

The safe policy $\pi^{\text{safe}}$ is used as the backup policy if we detect
that the task policy $\pi^{\text{task}}$ is likely to commit a safety
violation in the next $T$ steps. Since we have no access to an abstraction of
the environment we cannot synthesise a shield before training and so the safe
policy must be learned.

The safe policy $\pi^{\text{safe}}$ is only concerned with minimising expected
costs and so we use the cost predictor $\hat{c}_{t}\sim
p_{\theta}(\hat{c}_{t}\>|\>h_{t},z_{t})$ to construct TD-$\lambda$ targets as
follows,

$V_{t}^{\text{safe},\lambda}=\hat{c}_{t}+\hat{\gamma}_{t}\begin{cases}(1-\lambda)v^{\text{safe}}(\hat{s}_{t+1})+\lambda
V_{t+1}^{\text{safe},\lambda}&\text{if $t<H$,}\\\
v^{\text{safe}}(\hat{s}_{H})&\text{if $t=H$}\end{cases}$ (12)

The safe critic $v^{\text{safe}}$ parameterised by $\xi^{\text{safe}}$ is
regressed towards the TD-$\lambda$ targets with a similar loss function as
before,

$\mathcal{L}(\xi^{\text{safe}})=\mathbb{E}_{\pi^{\text{safe}},p_{\theta}}\left[\sum^{H-1}_{t=1}\frac{1}{2}(v^{\text{safe}}(\hat{s}_{t})-sg(V_{t}^{\text{safe},\lambda}))^{2}\right]$
(13)

The safe policy $\pi^{\text{safe}}$ parameterised by $\psi^{\text{safe}}$ is
also trained with biased reinforce gradients and an entropy regulariser as
before,

$\mathcal{L}(\psi^{\text{safe}})=\mathbb{E}_{\pi^{\text{safe}},p_{\theta}}\bigg{[}\sum^{H-1}_{t=1}\underbrace{\log\pi^{\text{safe}}(a_{t}\>|\>\hat{s}_{t})sg(V_{t}^{\text{safe},\lambda}-v^{\text{safe}}(\hat{s}_{t}))}_{\text{reinforce}}\\\
\underbrace{-\eta
H(\pi^{\text{safe}}(\cdot\>|\>\hat{s}_{t}))}_{\text{entropy}}\bigg{]}$ (14)

Note that the sign is flipped here, so that the safe policy
$\pi^{\text{safe}}$ minimises expected costs rather than maximises them.

###  3.4. Safety Critics

Safety critics estimate the expected costs under the task policy
$\pi^{\text{task}}$. They give us an idea of how safe specific states are
under the task policy state distribution. Additionally, we can use them to
bootstrap the end of ‘imagined’ trajectories for further look-ahead
capabilities.

To estimate the expected costs under the task policy $\pi^{\text{task}}$ we
use two safety critics $v^{C}_{1}$ and $v^{C}_{2}$ parameterised by
$\xi^{C}_{1}$ and $\xi^{C}_{2}$ respectively. To prevent overestimation, the
safety critics are jointly trained with a TD3-style algorithm Fujimoto et al.
(2018) to estimate the following quantity,

$\mathbb{E}_{\pi^{\text{task}},p_{\theta}}\left[\sum^{\infty}_{t=1}(\hat{\gamma}^{\text{safe}}_{t})^{t-1}\cdot\hat{c}_{t}\right]$
(15)

Each of the two safety critics $v^{C}_{1}$ and $v^{C}_{2}$, has its own target
critic ${v^{C}_{1}}^{\prime}$ and ${v^{C}_{2}}^{\prime}$, that are updated
periodically with slow updates. The TD-$\lambda$ targets are constructed by
taking a minimum of the two target critics ${v^{C}_{1}}^{\prime}$ and
${v^{C}_{2}}^{\prime}$ as follows,

$V_{t}^{C,\lambda}=\hat{c}_{t}+\hat{\gamma}^{\text{safe}}_{t}\begin{cases}(1-\lambda)\min\\{{v^{C}_{1}}^{\prime}(\hat{s}_{t}),{v^{C}_{2}}^{\prime}(\hat{s}_{t})\\}+\lambda
V_{t+1}^{C,\lambda}&\text{if $t<H$,}\\\
\min\\{{v^{C}_{1}}^{\prime}(\hat{s}_{H}),{v^{C}_{2}}^{\prime}(\hat{s}_{H})\\}&\text{if
$t=H$}\end{cases}$ (16)

Both safety critics $v^{C}_{1}$ and $v^{C}_{2}$ are regressed towards the
TD-$\lambda$ targets with the following loss function,

$\mathcal{L}(\xi^{C},v^{C})=\mathbb{E}_{\pi^{\text{task}},p_{\theta}}\left[\sum^{H-1}_{t=1}\frac{1}{2}(v^{C}(\hat{s}_{t})-sg(V_{t}^{C,\lambda}))^{2}\right]$
(17)

where
$(\xi^{C},v^{C})\in\\{(\xi^{C}_{1},v^{C}_{1}),(\xi^{C}_{2},v^{C}_{2})\\}$.

###  3.5. Algorithm

The full learning algorithm is split into two distinct phases: world model
learning and policy optimisation (including safe policy synthesis and safety
critic learning). To generate experience for world model learning we need to
interact with the real environment and to mitigate safety violations in the
real environment we pick actions with the shielded policy,

$\pi^{\text{shield}}(\cdot\>|\>s)=\begin{cases}\pi^{\text{task}}(\cdot\>|\>s)&\text{if
$\hat{\mu}_{s\models\phi}\in[1-\varepsilon+\epsilon,1]$}\\\
\pi^{\text{safe}}(\cdot\>|\>s)&\text{otherwise}\end{cases}$ (18)

To estimate $\mu_{s\models\phi}$ we roll-out the world model $p_{\theta}$ with
the task policy $\pi^{\text{task}}$ to generate a batch of $m$ sequences of
compact latent states $\langle\hat{s}^{(i)}_{1:H}\rangle^{m}_{i=1}$. For each
trace $\tau^{(i)}=\hat{s}^{(i)}_{1},...,\hat{s}^{(i)}_{H}$ we compute the
discounted cost as follows,

$\textrm{cost}(\tau^{(i)})=\sum^{H}_{t=1}(\hat{\gamma}^{(i)}_{t})^{t-1}\cdot\hat{c}^{(i)}_{t}$
(19)

######  Proposition 3.3.

Under the ‘true’ transition system $\mathcal{T}$ if
$\textrm{cost}(\tau)<\gamma^{H-1}\cdot C$ then necessarily
$\tau\models\square^{\leq H}\Phi$

###### Proof.

The proof is a straightforward argument. By construction $c_{t}=C$ if and only
if $\tau[t]\not\models\Phi$, therefore $\textrm{cost}(\tau)<\gamma^{H-1}\cdot
C$ implies that $\forall\>1\leq t\leq H\;c_{t}=0$ which implies that
$\forall\>1\leq t\leq H\;\tau[t]\models\Phi$. ∎

Using this idea, our estimate
$\hat{\mu}_{s\models\phi}\approx\mu_{s\models\phi}$ is then computed as
follows,

$\hat{\mu}_{s\models\phi}=\frac{1}{m}\sum^{m}_{i=1}\mathbbm{1}\left(\textrm{cost}(\tau^{(i)})<\gamma^{H-1}\cdot
C\right)$ (20)

If we train safety critics then we can use the bootstrapped costs instead,

$\textrm{b-cost}(\tau^{(i)})=\left(\sum^{H-1}_{t=1}(\hat{\gamma}^{(i)}_{t})^{t-1}\cdot\hat{c}^{(i)}_{t}\right)+\min\left\\{v_{1}^{C}(\hat{s}^{(i)}_{H}),v_{2}^{C}(\hat{s}^{(i)}_{H})\right\\}$
(21)

And we can estimate $\mu_{s\models\phi}$ with a larger horizon $T>H$, since
the safety critics should capture the expected costs from $\hat{s}_{H}^{(i)}$
and beyond,

$\hat{\mu}_{s\models\phi}=\frac{1}{m}\sum^{m}_{i=1}\mathbbm{1}\left(\textrm{b-cost}(\tau^{(i)})<\gamma^{T-1}\cdot
C\right)$ (22)

After several environment interactions with the shielding policy
$\pi^{\text{shield}}$, a batch of data $B$ is sampled from the replay buffer
$\mathcal{D}$, for world model learning, task policy optimisation, safe policy
optimisation and safety critic learning. The full algorithm is presented on
the following page.

Algorithm 1 DreamerV2 Hafner et al. (2021) with Shielding

Initialise: replay buffer $\mathcal{D}$ with $S$ random epsiodes.
Initialise: $\theta$, $\psi^{\text{task}}$, $\psi^{\text{safe}}$,
$\xi^{\text{task}}$, $\xi^{\text{safe}}$, $\xi^{C}_{1}$, $\xi^{C}_{2}$,
${\xi^{C}_{1}}^{\prime}$, ${\xi^{C}_{2}}^{\prime}$ randomly.

while not converged do

// World model learning

Sample $B\sim\mathcal{D}$.

For every $o_{t}\in B$ compute sequences $\hat{s}_{t:t+H}$ with RSSM.

Update RSSM parameters $\theta$ with Eq. 11

// Task policy optimisation

From every $o_{t}\in B$ imagine sequences $\hat{s}_{t:t+H}$ with
$\pi^{\text{task}}$.

Compute TD-$\lambda$ targets with Eq. 3.

Update task critic parameters $\xi^{\text{task}}$ with Eq. 4.

Update task policy parameters $\psi^{\text{task}}$ with Eq. 5.

// Safety critic optimisation

Compute safety critic targets with Eq. 16.

Update safety critic parameters $\xi^{C}_{1}$ and $\xi^{C}_{1}$ with Eq. 17.

For $i\in[1,2]$
${\xi^{C}_{i}}^{\prime}\leftarrow\nu\xi^{C}_{i}+(1-\nu){\xi^{C}_{i}}^{\prime}$
(soft update Fujimoto et al. (2018)).

// Safe policy optimisation

From every $o_{t}\in B$ imagine sequences $\hat{s}_{t:t+H}$ with
$\pi^{\text{safe}}$.

Compute TD-$\lambda$ targets with Eq. 12.

Update safe critic parameters $\xi^{\text{safe}}$ with Eq. 13.

Update safe policy parameters $\psi^{\text{safe}}$ with Eq. 14.

// Environment interaction

for $k=1,...,K$ do

Observe $o_{t}$ from environment and compute $\hat{s}_{t}=(z_{t},h_{t})$.

From $\hat{s}_{t}$ sample $m$ sequences
$\langle\hat{s}^{(i)}_{1:H}\rangle^{m}_{i=1}$ with $\pi^{\text{task}}$.

Estim. $\hat{\mu}_{s\models\phi}\approx\mu_{s\models\phi}$ with safety
critics, Eq. 21 and Eq. 22.

Play $a\sim\pi^{\text{shield}}(a\>|\>\hat{s}_{t})$ and observe $r_{t},o_{t+1}$
and $L(s_{t})$.

Construct $c_{t}$ with Eq. 9 and $\gamma^{\text{safe}}_{t}$ with Eq. 10.

Append $\langle
o_{t},a_{t},r_{t},c_{t},\gamma^{\text{safe}}_{t},o_{t+1}\rangle$ to
$\mathcal{D}$.

end for

end while

##  4\. Experiments

In this section we conduct a simple analysis and compare our algorithm,
DreamerV2 with shielding, to DreamerV2 without shielding. We present results
for two Atari games with state dependent labels: Assault and Seaquest (see
Fig. 2). We start by giving a summary of the environments, followed by the
experimental results and an accompanying discussion.

###  4.1. Assault

_Assault_ is a fixed shooter game similar to _Space Invaders_. The goal is to
shoot and destroy alien ships continuously deployed by a mothership. The
smaller ships shoot lasers at the player which the player must avoid,
otherwise they loose a life. In addition, the player’s weapon can overheat if
they fire too often, which also results in them loosing a life. The state
dependent formula $\Phi$ that the agent aims to satisfy at each timestep is
given as follows,

$\Phi=\neg\textbf{hit}\land\neg\textbf{overheat}$ (23)

where $\textbf{hit}=\textrm{true}$ iff the player is hit by a laser and
$\textbf{overheat}=\textrm{true}$ iff the player’s weapon overheats. We chose
this environment because Giacobbe et al. demonstrated state-of-the-art agents
only concerned with reward overheat the weapon frequently and that BPS
Giacobbe et al. (2021) alleviated the issue to some degree. The idea is that
when the task policy $\pi^{\text{task}}$ is about to overheat the weapon the
shield kicks in and the safe policy $\pi^{\text{safe}}$ prevents the agent
from firing the weapon while avoiding any incoming lasers.

###  4.2. Seaquest

_Seaquest_ is an underwater shooter in which the player controls a submarine
equipped with an infinite supply of missiles. The goal is to rescue divers,
shoot enemy sharks and submarines, while managing a limited supply of oxygen
and resurfacing when necessary. The player receives points and a full supply
of oxygen when they surface with a diver on board and if they surface with six
divers they are awarded additional points based on the amount of oxygen they
have left. However, surfacing without any divers is not permitted and results
in the player loosing a life. The state formula $\Phi$ for Seaquest is a
little more involved and is defined as follows,

$\Phi=(\textbf{surface}\Rightarrow((\textbf{diver}\land\textbf{low-
oxygen})\lor\textbf{very-low-oxygen}\lor\\\ \textbf{six-
divers}))\land\neg\textbf{out-of-oxygen}\land\neg\textbf{hit}$ (24)

where $\textbf{surface}=\textrm{true}$ iff the submarine surfaces,
$\textbf{diver}=\textrm{true}$ iff the submarine has at least one diver on
board, $\textbf{low-oxygen}=\textrm{true}$ iff the players oxygen supply ¡ 16,
$\textbf{very-low-oxygen}=\textrm{true}$ iff the players oxygen supply ¡ 4,
$\textbf{six- divers}=\textrm{true}$ iff the submarine has six divers on
board, $\textbf{out-of-oxygen}=\textrm{true}$ iff the player runs out of
oxygen and hit is defined similarly as before.

In words, it is only permissible to surface if the agent has a diver and is
low on oxygen, has six divers or has very low on oxygen (with or without a
diver). Surfacing with a diver when oxygen supplies are plentiful is deemed
unsafe since it makes the game unnecessarily harder.

With Seaquest the agent needs to balance multiple objectives at once which is
why it is a useful environment to test our approach. In our experiments we
demonstrate that the learned safe policy $\pi^{\text{safe}}$ is able to deal
with a slightly more complex set of constraints and prevent the agent from
making costly mistakes during training.

(a) Assault

(b) Seaquest

Figure 2. Screenshots from the two Atari environments.

Screenshots from Atari Assault and Atari Seaquest.

###  4.3. Training Details

The agents are trained on a single Nvidia Tesla A30 (24GB RAM) GPU and a
24-core/48 thread Intel Xeon CPU with 256GB RAM. Due to time constraints and
limited compute resources all agents are trained on one seed and for precisely
40M frames on Atari environments provided by the ALE Bellemare et al. (2013);
Machado et al. (2018).

All the hyperparameters for DreamerV2 are set as their default values for
Atari games, which are given in Hafner et al. (2021). Notably, for all
experiments we set the imagination horizon $H=15$, TD-$\lambda$ discount
$\lambda=0.95$ and discount factor $\gamma=0.999$.

The cost and safety-discount predictors are implemented as neural networks
with identical architectures to the reward and discount predictors used in
DreamerV2. The safe policy, critic and safety critics are also implemented as
neural networks in the same way that the task policy and critic are
implemented in DreamerV2. See Hafner et al. (2021) for all details. The
shielding hyperparameters are also fixed in all experiments as follows,
specifically we set the bounded safety parameter $\varepsilon=0.1$, number of
samples $m=512$, approximation error $\epsilon=0.09$111The gives us roughly
$\delta=0.1$, using a tighter bound than Eq. 6 which bounds the probability of
overestimating $\mu_{s\models\phi}$., shield horizon $T=30$ (2 seconds in real
time) and safety critic smooth parameter updates $\nu=0.005$.

###  4.4. Results

We evaluate our algorithm by comparing the performance of DreamerV2 Hafner et
al. (2021) with and without shielding. Specifically, we compare the reward
curves during training, the best episode return and the cumulative violations
during training. Table 1 presents the best episode scores and total violations
during training for DreamerV2 and DreamerV2 with shielding. In addition, Fig.
3 displays the learning curves for both algorithms.

(a) Training reward curve for Assault

(b) Training reward curve for Seaquest

(c) Cumulative violations for Assault

(d) Cumulative violations for Seaquest

Figure 3. Training curves for DreamerV2 Hafner et al. (2021) and DreamerV2
with approximate shielding (ours).*

* The reward curves are smoothed with simple exponential smoothing with $w=0.6$. Training reward curves and violations during training plots for Atari Assault and Atari Seaquest. In Atari Assault DeamerV2 with shielding obtains significantly better reward during training and fewer violations than DreamerV2 without shielding. In Atari Seaquest DeamerV2 with shielding makes fewer violations than DreamerV2 without shielding, although it obtains slightly worse performance.

Table 1. Best episode scores and cumulative violations for for DreamerV2 Hafner et al. (2021) and DreamerV2 with approximate shielding. Env | _DreamerV2_ | _DreamerV2 w/ Shielding_
---|---|---
Best Score | # Violations | Best Score | # Violations
Assault | 34753 | 11726 | 57504 | 8579
Seaquest | 11400 | 15697 | 7040 | 4889

#### Discussion

As seen in Table 1 and Fig. 3 our approximate shielding algorithm reduces the
rate of safety violations for both _Assault_ and _Seaquest_. In terms of
reward, our shielding procedure has dramatically improved the speed of
convergence for _Assault_ and maintained comparable performance for
_Seaquest_. We must note that these results are far from complete as we have
not these run experiments over multiple random seeds or for the typical 200M
frames, which is used as a common benchmark Hafner et al. (2021).
Nevertheless, we claim that our results provide compelling evidence that
something is going on, which should motivate further investigation.

##  5\. Related Work

In this section we provide a discussion on the three main areas of research
that our contribution is based on: world models, safe RL, and shielding.

World Models were first introduced by Ha and Schmidhuber Ha and Schmidhuber
(2018) in a paper of the same name, although their inspiration is much more
deeply rooted in psychology Tolman (1948) and Bayesian theories of the brain
Friston (2003). Dyna – “an integrated architecture for learning, planning and
reacting” – proposed in Sutton (1991), introduced the idea of not only
utilising reward signals to learn good policies, but also learning a dynamics
model through observed transitions Sutton (1991). In theory, planning with the
learned dynamics model could speed up convergence of the policy, but many
early approaches suffered from model bias Atkeson and Santamaria (1997).
_Gaussian processes_ (GPs) Williams and Rasmussen (2006) were quickly used as
the stand-in dynamics model for the Dyna architecture as they reduced model
bias by quantifying their own uncertainty Deisenroth and Rasmussen (2011).
However, GPs struggle in high-dimensional settings and so the use neural
architectures has been increasingly explored instead.

More recently, with the neural architecture Dreamer Hafner et al. (2020),
Hafner et al. demonstrated that policies can be learnt purely from imagined
experience and transfer well to the original environment. Additionally,
DreamerV2 Hafner et al. (2021) and DreamerV3 Hafner et al. (2023) demonstrated
state-of-the-art performance in a variety of domains including the Atari
benchmark Bellemare et al. (2013); Machado et al. (2018) and MineRL Guss et
al. (2019) both of which have been notorious challenges for MBRL.

Once a world model is learned it can be used in a flexible manner for policy
optimisation Hafner et al. (2020), online planning schemes Hafner et al.
(2019); Hung et al. (2022); Wu et al. (2022b), risk measures Zhang et al.
(2020), and defining intrinsic rewards for improved exploration Sekar et al.
(2020); Latyshev and Panov (2023). As a result world model have been applied
in a variety of domains, such as robotics Wu et al. (2022a), imitation
learning DeMoss et al. (2023), continual learning Kessler et al. (2022) and
safe RL As et al. (2022).

Safe RL is typically categorised as the problem of maximising reward, while
maintaining some reasonable system performance during learning and deployment
of the agent Garcıa and Fernández (2015). This definition has been interpreted
in many different ways, stemming from different objectives in different
domains. For example, reward hacking Amodei et al. (2016); Skalse et al.
(2022) refers to an agent ‘gaming’ or exploiting a misspecified reward
function, which can lead to undesired outcomes. Robustifying policies to
distributional shift Andrychowicz et al. (2020); Peng et al. (2018); Urpí et
al. (2021) and the alignment problem Russell and Norvig (2021); Di Langosco et
al. (2022) are also important areas of research in safe RL. However, we tackle
the problem of _safe exploration_ Pecka and Svoboda (2014); Garcıa and
Fernández (2015) which can be described as the problem of minimising the
violation of safety-constraints during the exploratory phase of training and
beyond.

The constrained Markov decision process (CMDP) Altman (1999) is a widely used
framework for modeling decision-making problems with safety constraints. In
addition to maximising expected reward, agents must satisfy a set of
constraints encoded as a cost function that penalises unsafe state-action
pairs. In the tabular case, linear programs can be used to solve CMDPs Altman
(1999). In the non-tabular case, a variety of model-free algorithms with
function approximation have been proposed Chow et al. (2017); Achiam et al.
(2017); Bohez et al. (2019); Liu et al. (2020a); Yang et al. ([n.d.]).

Model-based approaches for safe RL utilise a variety of different techniques
for dynamics modelling and policy optimisation. Berkenkamp et al. use GPs to
quantify model uncertainty in a principled way to safely learn neural network
policies. Other approaches use ensembles of neural networks (NNs) to quantify
uncertainty and either deploy MPC Liu et al. (2020b); Thomas et al. (2021),
perform policy optimisation within a certified region of the state space Luo
and Ma (2021), or use constrained policy optimisation with Lagrangian
relaxation Zanger et al. (2021) to learn safety-aware policies. Notable work
by As et al. leverages Dreamer Hafner et al. (2021) and stochastic weight
averaging Gaussian (SWAG) Maddox et al. (2019) to obtain a Bayesian predictive
distribution over possible world models that explain the dynamics of the
environment. As et al. also stress the importance of policy optimisation with
safety critics over shortsighted MPC schemes.

Shielding for RL has been introduced as a correct by construction reactive
(shield), which prevents the learned policy from entering unsafe states
defined by some temporal logic formula Alshiekh et al. (2018). The shield
itself can be applied before the agent picks an action (preemptive), modifying
the action space of the agent. Alternatively, the shield can be applied after
the agent picks an action (post-posed), overriding actions proposed by the
agent if they lead to a violation. Both types of shield require the ability to
construct and solve a safety game Bloem et al. (2015) on a relatively compact
representation of the MDP. Similar to control barrier functions (CBFs) Ames et
al. (2016) from optimal control theory Kirk (2004), the shield projects the
learned policy back into a verified safe set on the state space.

Recent work on shielding includes generalising it to partially observed Carr
et al. (2022) and multi-agent ElSayed-Aly et al. (2021) settings, as well as
resource constrained partially observed MDPs Ajdarów et al. (2022). Many of
these methods still require a suitable abstraction of the environment or
sufficient domain knowledge for synthesising a shield. However, these strong
assumptions come hand in hand with strong guarantees on safety of the learned
policy, specifically Alshiekh et al. (2018) show that by construction their
shield synthesis procedure guarantees safety with minimal interference.

Learning a shield online is an alternative approach to shielding RL policies
without requiring significant prior knowledge. For example, Shperberg et al.
propose tabular and parametric shields which are learning online to prevent
agents from repeating catastrophic mistakes in the partially observed setting
Shperberg et al. (2022). Other online shielding approaches include _latent
shielding_ He et al. (2022) and BPS, both of which have substantially
influenced our work. For a more complete review of reactive methods based on
shielding we refer the interested reader to Odriozola-Olalde et al. (2023).

##  6\. Conclusions

In this paper we presented an approximate shielding algorithm for safe
exploration of Atari agents and more general RL policies. Building on
DreamerV2 Hafner et al. (2021) and previous work, such as, _latent shielding_
He et al. (2022) and BPS Giacobbe et al. (2021), we propose a more general
algorithm that uses safety critics and policy roll-outs to perform look-ahead
shielding in the latent space of a learned world model.

In contrast to previous work, we are able to successfully apply our
approximate shielding algorithm with minimal hyperparameter tuning and no
shielding introduction schedules. While we loose the benefit of strict and
formal guarantees obtained by earlier shielding approaches Alshiekh et al.
(2018), we are able to derive some probabilistic guarantees, although this is
incomplete and further work should be done to derive bounds for the
approximate transition system.

Nevertheless, our empirical results are promising and provide some good
evidence that general RL agents can benefit from shielding in certain
settings, not only in terms of complying with safety specifications, but also
in terms of improved performance. The aim of this research is to shed light on
the promise of this approach and we hope this opens the door to further
investigation.

This work was supported by UK Research and Innovation [grant number
EP/S023356/1], in the UKRI Centre for Doctoral Training in Safe and Trusted
Artificial Intelligence (www.safeandtrustedai.org).

## References

  * (1)
  * Achiam et al. (2017) Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. 2017\.  Constrained policy optimization. In _International conference on machine learning_. PMLR, 22–31.
  * Ajdarów et al. (2022) Michal Ajdarów, Šimon Brlej, and Petr Novotnỳ. 2022.  Shielding in Resource-Constrained Goal POMDPs.  _arXiv preprint arXiv:2211.15349_ (2022).
  * Alshiekh et al. (2018) Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. 2018.  Safe reinforcement learning via shielding. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 32.
  * Altman (1999) Eitan Altman. 1999\.  _Constrained Markov decision processes: stochastic modeling_.  Routledge.
  * Ames et al. (2016) Aaron D Ames, Xiangru Xu, Jessy W Grizzle, and Paulo Tabuada. 2016\.  Control barrier function based quadratic programs for safety critical systems.  _IEEE Trans. Automat. Control_ 62, 8 (2016), 3861–3876.
  * Amodei et al. (2016) Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016\.  Concrete problems in AI safety.  _arXiv preprint arXiv:1606.06565_ (2016).
  * Andrychowicz et al. (2020) OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. 2020\.  Learning dexterous in-hand manipulation.  _The International Journal of Robotics Research_ 39, 1 (2020), 3–20.
  * As et al. (2022) Yarden As, Ilnura Usmanova, Sebastian Curi, and Andreas Krause. 2022.  Constrained Policy Optimization via Bayesian World Models. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=PRZoSmCinhf
  * Atkeson and Santamaria (1997) Christopher G Atkeson and Juan Carlos Santamaria. 1997.  A comparison of direct and model-based reinforcement learning. In _Proceedings of international conference on robotics and automation_ , Vol. 4. IEEE, 3557–3564.
  * Baier and Katoen (2008) Christel Baier and Joost-Pieter Katoen. 2008.  _Principles of model checking_.  MIT press.
  * Bellemare et al. (2013) M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. 2013.  The Arcade Learning Environment: An Evaluation Platform for General Agents.  _Journal of Artificial Intelligence Research_ 47 (jun 2013), 253–279.
  * Bengio et al. (2013) Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013\.  Estimating or propagating gradients through stochastic neurons for conditional computation.  _arXiv preprint arXiv:1308.3432_ (2013).
  * Berkenkamp et al. (2017) Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. 2017.  Safe model-based reinforcement learning with stability guarantees.  _Advances in neural information processing systems_ 30 (2017).
  * Bloem et al. (2015) Roderick Bloem, Bettina Könighofer, Robert Könighofer, and Chao Wang. 2015.  Shield synthesis: Runtime enforcement for reactive systems. In _Tools and Algorithms for the Construction and Analysis of Systems: 21st International Conference, TACAS 2015, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2015, London, UK, April 11-18, 2015, Proceedings 21_. Springer, 533–548.
  * Bohez et al. (2019) Steven Bohez, Abbas Abdolmaleki, Michael Neunert, Jonas Buchli, Nicolas Heess, and Raia Hadsell. 2019.  Value constrained model-free continuous control.  _arXiv preprint arXiv:1902.04623_ (2019).
  * Carr et al. (2022) Steven Carr, Nils Jansen, Sebastian Junges, and Ufuk Topcu. 2022\.  Safe Reinforcement Learning via Shielding under Partial Observability.  _arXiv preprint arXiv:2204.00755_ (2022).
  * Chow et al. (2017) Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. 2017.  Risk-constrained reinforcement learning with percentile risk criteria.  _The Journal of Machine Learning Research_ 18, 1 (2017), 6070–6120.
  * Deisenroth and Rasmussen (2011) Marc Deisenroth and Carl E Rasmussen. 2011.  PILCO: A model-based and data-efficient approach to policy search. In _Proceedings of the 28th International Conference on machine learning (ICML-11)_. 465–472.
  * DeMoss et al. (2023) Branton DeMoss, Paul Duckworth, Nick Hawes, and Ingmar Posner. 2023.  DITTO: Offline Imitation Learning with World Models.  _arXiv preprint arXiv:2302.03086_ (2023).
  * Di Langosco et al. (2022) Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. 2022\.  Goal misgeneralization in deep reinforcement learning. In _International Conference on Machine Learning_. PMLR, 12004–12019.
  * ElSayed-Aly et al. (2021) Ingy ElSayed-Aly, Suda Bharadwaj, Christopher Amato, Rüdiger Ehlers, Ufuk Topcu, and Lu Feng. 2021\.  Safe Multi-Agent Reinforcement Learning via Shielding. In _Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems_. 483–491.
  * Friston (2003) Karl Friston. 2003\.  Learning and inference in the brain.  _Neural Networks_ 16, 9 (2003), 1325–1352.
  * Fujimoto et al. (2018) Scott Fujimoto, Herke Hoof, and David Meger. 2018.  Addressing function approximation error in actor-critic methods. In _International conference on machine learning_. PMLR, 1587–1596.
  * Garcıa and Fernández (2015) Javier Garcıa and Fernando Fernández. 2015.  A comprehensive survey on safe reinforcement learning.  _Journal of Machine Learning Research_ 16, 1 (2015), 1437–1480.
  * Giacobbe et al. (2021) Mirco Giacobbe, Mohammadhosein Hasanbeig, Daniel Kroening, and Hjalmar Wijk. 2021\.  Shielding Atari Games with Bounded Prescience. In _AAMAS ’21: 20th International Conference on Autonomous Agents and Multiagent Systems, Virtual Event, United Kingdom, May 3-7, 2021_ , Frank Dignum, Alessio Lomuscio, Ulle Endriss, and Ann Nowé (Eds.). ACM, 1507–1509.  https://doi.org/10.5555/3463952.3464141
  * Guss et al. (2019) William H Guss, Cayden Codel, Katja Hofmann, Brandon Houghton, Noboru Kuno, Stephanie Milani, Sharada Mohanty, Diego Perez Liebana, Ruslan Salakhutdinov, Nicholay Topin, et al. 2019\.  NeurIPS 2019 competition: the MineRL competition on sample efficient reinforcement learning using human priors.  _arXiv preprint arXiv:1904.10079_ (2019).
  * Ha and Schmidhuber (2018) David Ha and Jürgen Schmidhuber. 2018.  Recurrent World Models Facilitate Policy Evolution. In _Advances in Neural Information Processing Systems_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc.  https://proceedings.neurips.cc/paper_files/paper/2018/file/2de5d16682c3c35007e4e92982f1a2ba-Paper.pdf
  * Hafner et al. (2020) Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. 2020.  Dream to Control: Learning Behaviors by Latent Imagination. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=S1lOTC4tDS
  * Hafner et al. (2019) Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. 2019.  Learning latent dynamics for planning from pixels. In _International conference on machine learning_. PMLR, 2555–2565.
  * Hafner et al. (2021) Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. 2021.  Mastering Atari with Discrete World Models. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=0oabwyZbOu
  * Hafner et al. (2023) Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. 2023.  Mastering Diverse Domains through World Models.  _arXiv preprint arXiv:2301.04104_ (2023).
  * He et al. (2022) Peter He, Borja G León, and Francesco Belardinelli. 2022\.  Do Androids Dream of Electric Fences? Safety-Aware Reinforcement Learning with Latent Shielding. In _Proceedings of the Workshop on Artificial Intelligence Safety 2022 (SafeAI 2022)_ _(CEUR Workshop Proceedings, 3087)_ , Gabriel Pedroza, José Hernández-Orallo, Xin Cynthia Chen, Xiaowei Huang, Huáscar Espinoza, Mauricio Castillo-Effen, John McDermid, Richard Mallah, and Seán Ó hÉigeartaigh (Eds.). Aachen.  https://ceur-ws.org/Vol-3087/paper_50.pdf
  * Hung et al. (2022) Chia-Man Hung, Shaohong Zhong, Walter Goodwin, Oiwi Parker Jones, Martin Engelcke, Ioannis Havoutis, and Ingmar Posner. 2022\.  Reaching through latent space: From joint statistics to path planning in manipulation.  _IEEE Robotics and Automation Letters_ 7, 2 (2022), 5334–5341.
  * Janner et al. (2019) Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. 2019\.  When to trust your model: Model-based policy optimization.  _Advances in neural information processing systems_ 32 (2019).
  * Jansen et al. (2018) Nils Jansen, Bettina Könighofer, Sebastian Junges, and Roderick Bloem. 2018.  _Shielded Decision-Making in MDPs_.  WorkingPaper. Cornell University Library.
  * Kessler et al. (2022) Samuel Kessler, Piotr Miłoś, Jack Parker-Holder, and Stephen J. Roberts. 2022.  The Surprising Effectiveness of Latent World Models for Continual Reinforcement Learning. In _Deep Reinforcement Learning Workshop NeurIPS 2022_.  https://openreview.net/forum?id=-lHOOgHuWwu
  * Kirk (2004) Donald E Kirk. 2004\.  _Optimal control theory: an introduction_.  Courier Corporation.
  * Latyshev and Panov (2023) Artem Latyshev and Aleksandr I Panov. 2023.  Intrinsic Motivation in Model-based Reinforcement Learning: A Brief Review.  _arXiv preprint arXiv:2301.10067_ (2023).
  * Liu et al. (2020a) Yongshuai Liu, Jiaxin Ding, and Xin Liu. 2020a.  Ipo: Interior-point policy optimization under constraints. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 34. 4940–4947.
  * Liu et al. (2020b) Zuxin Liu, Hongyi Zhou, Baiming Chen, Sicheng Zhong, Martial Hebert, and Ding Zhao. 2020b.  Constrained model-based reinforcement learning with robust cross-entropy method.  _arXiv preprint arXiv:2010.07968_ (2020).
  * Luo and Ma (2021) Yuping Luo and Tengyu Ma. 2021.  Learning barrier certificates: Towards safe reinforcement learning with zero training-time violations.  _Advances in Neural Information Processing Systems_ 34 (2021), 25621–25632.
  * Machado et al. (2018) Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, and Michael Bowling. 2018.  Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents.  _Journal of Artificial Intelligence Research_ 61 (2018), 523–562.
  * Maddox et al. (2019) Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. 2019\.  A simple baseline for bayesian uncertainty in deep learning.  _Advances in neural information processing systems_ 32 (2019).
  * Odriozola-Olalde et al. (2023) Haritz Odriozola-Olalde, Maider Zamalloa, and Nestor Arana-Arexolaleiba. 2023.  Shielded Reinforcement Learning: A review of reactive methods for safe learning. In _2023 IEEE/SICE International Symposium on System Integration (SII)_. IEEE, 1–8.
  * Pecka and Svoboda (2014) Martin Pecka and Tomas Svoboda. 2014.  Safe exploration techniques for reinforcement learning–an overview. In _Modelling and Simulation for Autonomous Systems: First International Workshop, MESAS 2014, Rome, Italy, May 5-6, 2014, Revised Selected Papers 1_. Springer, 357–375.
  * Peng et al. (2018) Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. 2018.  Sim-to-real transfer of robotic control with dynamics randomization. In _2018 IEEE international conference on robotics and automation (ICRA)_. IEEE, 3803–3810.
  * Puterman (1990) Martin L Puterman. 1990\.  Markov decision processes.  _Handbooks in operations research and management science_ 2 (1990), 331–434.
  * Russell and Norvig (2021) Stuart Russell and Peter Norvig. 2021.  (4 ed.).  Pearson Education Limited, 49–52.
  * Sekar et al. (2020) Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. 2020.  Planning to explore via self-supervised world models. In _International Conference on Machine Learning_. PMLR, 8583–8592.
  * Shperberg et al. (2022) Shahaf S Shperberg, Bo Liu, and Peter Stone. 2022.  Learning a Shield from Catastrophic Action Effects: Never Repeat the Same Mistake.  _arXiv preprint arXiv:2202.09516_ (2022).
  * Skalse et al. (2022) Joar Max Viktor Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. 2022\.  Defining and Characterizing Reward Gaming. In _Advances in Neural Information Processing Systems_ , Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (Eds.).  https://openreview.net/forum?id=yb3HOXO3lX2
  * Sutton (1991) Richard S Sutton. 1991\.  Dyna, an integrated architecture for learning, planning, and reacting.  _ACM Sigart Bulletin_ 2, 4 (1991), 160–163.
  * Sutton and Barto (2018) Richard S Sutton and Andrew G Barto. 2018.  _Reinforcement learning: An introduction_.  MIT press.
  * Thomas et al. (2021) Garrett Thomas, Yuping Luo, and Tengyu Ma. 2021.  Safe reinforcement learning by imagining the near future.  _Advances in Neural Information Processing Systems_ 34 (2021), 13859–13869.
  * Tolman (1948) Edward C Tolman. 1948\.  Cognitive maps in rats and men.  _Psychological review_ 55, 4 (1948), 189.
  * Urpí et al. (2021) Núria Armengol Urpí, Sebastian Curi, and Andreas Krause. 2021\.  Risk-Averse Offline Reinforcement Learning. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=TBIzh9b5eaz
  * Williams and Rasmussen (2006) Christopher KI Williams and Carl Edward Rasmussen. 2006.  _Gaussian processes for machine learning_. Vol. 2.  MIT press Cambridge, MA.
  * Wu et al. (2022a) Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. 2022a.  DayDreamer: World Models for Physical Robot Learning. In _6th Annual Conference on Robot Learning_.  https://openreview.net/forum?id=3RBY8fKjHeu
  * Wu et al. (2022b) Zifan Wu, Chao Yu, Chen Chen, Jianye HAO, and Hankz Hankui Zhuo. 2022b.  Plan To Predict: Learning an Uncertainty-Foreseeing Model For Model-Based Reinforcement Learning. In _Advances in Neural Information Processing Systems_ , Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (Eds.).  https://openreview.net/forum?id=L9YayWPcHA_
  * Yang et al. ([n.d.]) Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. [n.d.].  Projection-Based Constrained Policy Optimization. In _International Conference on Learning Representations_.
  * Zanger et al. (2021) Moritz A Zanger, Karam Daaboul, and J Marius Zöllner. 2021\.  Safe continuous control with constrained model-based policy optimization. In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 3512–3519.
  * Zhang et al. (2020) Jesse Zhang, Brian Cheung, Chelsea Finn, Sergey Levine, and Dinesh Jayaraman. 2020.  Cautious adaptation for reinforcement learning in safety-critical settings. In _International Conference on Machine Learning_. PMLR, 11055–11065.


# FairRank: Fairness-aware Single-tower Ranking Framework
for News Recommendation

Chuhan Wu1, Fangzhao Wu2, Tao Qi1, Yongfeng Huang1 1Department of Electronic
Engineering & BNRist, Tsinghua University, Beijing 100084
2Microsoft Research Asia, Beijing 100080, China
wuchuhan15,wufangzhao,taoqi.qt<EMAIL_ADDRESS>
(2022)

###### Abstract.

Single-tower models are widely used in the ranking stage of news
recommendation to accurately rank candidate news according to their fine-
grained relatedness with user interest indicated by user behaviors. However,
these models can easily inherit the biases related to users’ sensitive
attributes (e.g., gender) encoded in user behavior data for model training,
and result in recommendation results unfair for users with certain attributes.
In this paper, we propose FairRank, a fairness-aware single-tower ranking
framework for news recommendation. Since candidate news selection in existing
recommender systems can be biased, we propose to use a shared candidate-aware
user model to match user interest with a real displayed candidate news and a
random news. It learns a candidate-aware user embedding that reflects user
interest in candidate news, and a candidate-invariant user embedding that
indicates intrinsic user interest, respectively. We apply adversarial learning
to both of them to reduce the biases brought by sensitive user attributes. In
addition, we use a KL loss to regularize the attribute labels inferred from
the two user embeddings to be similar, which can make the model capture less
candidate-aware bias information. Extensive experiments on two datasets show
that FairRank can improve the fairness of various single-tower news ranking
models with minor performance loss.

News recommendation, Fairness, Single-tower

††journalyear: 2022††copyright: acmcopyright††price: 15.00

##  1\. Introduction

Recall and ranking are two essential steps in personalized news recommendation
(Wu et al., 2020; Wu et al., 2021c). The recall step aims to select a small
subset of candidate news from the large news pool to comprehensively cover
different user interests (Qi et al., 2021b). In this step, two-tower models,
which learn user interest and news embeddings independently, are common
choices due to efficiency considerations (Liu et al., 2020; Ge et al., 2020;
Wu et al., 2021e; Wu et al., 2021a). In contrast, the ranking step needs to
accurately rank candidate news according to their relevance with user interest
(Wang et al., 2020). Thus, single tower models that can capture the fine-
grained relatedness between candidate news and user interest indicated by
different user behaviors, are widely used as the ranking algorithms in news
recommendation (Qi et al., 2021a; Jia et al., 2021; Zhang et al., 2021a, b).

News ranking models are usually learned on users’ click behavior data (Lee et
al., 2020). In fact, click behavior data inherently encodes biases related to
users’ sensitive attributes such as demographics (Hu et al., 2007; Beutel et
al., 2019). The model learned on such data can inherit and even amplify these
biases, which may generate biased recommendation results that are unfair to
users with certain sensitive attributes (Wu et al., 2021d).111We find single-
tower models may even have more severe unfairness than two-tower models, see
experiments. Adversarial learning is a widely used technique to remove the
biases related to sensitive attributes from deep models to help make fair
predictions (Elazar and Goldberg, 2018; Zhang et al., 2018; Xu et al., 2019),
which also has the potential to empower fairness-aware news recommendation.
For example, Wu et al. (2021d) proposed a decomposed adversarial learning
framework that learns a bias-aware user model and a bias-free user model to
capture bias information and bias-independent user interest information for
news recommendation, respectively. However, existing methods are based on two-
tower architectures that model user interest and candidate news independently,
and the fairness issue of single-tower news ranking models is rarely studied.

In this paper, we propose a fairness-aware single-tower ranking framework
named FairRank for news recommendation, which can effectively improve the
fairness of various single-tower news ranking models. Since the selection of
candidate news in existing recommender systems can be biased, in FairRank we
propose to use a shared candidate-aware user model to match user interest with
a real displayed candidate news and a randomly selected news, respectively, to
learn a candidate-aware user embedding that indicates user interest on a
specific candidate news and a candidate-invariant user embedding that encodes
intrinsic user interest in terms of expectation. To reduce the bias on
sensitive user attributes encoded by the model, we apply adversarial learning
to both user embeddings to remove bias information. In addition, to further
eliminate bias information encoded by the selection of candidate news, we
propose to use a Kullback–Leibler (KL) divergence loss to regularize the
attribute labels inferred from the two user embeddings to be similar, which
can encourage the model to be less sensitive to the inherent bias introduced
by candidate news selection. Extensive experiments on two real-world news
recommendation datasets show that FairRank can improve the fairness of various
single-tower ranking models with minor performance sacrifice.

##  2\. FairRank

Next, we introduce the FairRank approach for fairness-aware news ranking, as
shown in Fig. 1. It incorporates a candidate-aware user model to capture the
interactions between clicked news and a candidate news or a random news, to
model user’s interest in a specific candidate news and candidate-invariant
user interest (in the sense of expectation). We then introduce its details and
how to learn this model to generate fair news rankings.

###  2.1. Candidate-aware Ranking Model

Since candidate news usually can only match parts of user interest indicated
by clicked news, modeling the fine-grained interactions between clicked news
and candidate news may achieve better news ranking accuracy than directly
matching candidate news with overall user interest. Thus, we use a candidate-
aware user model to capture users’ candidate-aware interest. It can be
implemented by many architectures, such as (1) using candidate-aware attention
to select clicked news according to their relevance to candidate news (Wang et
al., 2018); (2) using 3-D convolutional neural network to match candidate news
and clicked news based on their texts (Wang et al., 2020); and (3)
concatenating clicked news and candidate news into a long document as the
input of a Transformer (Vaswani et al., 2017). It takes a user’s clicked news
(denoted as $[D_{1},D_{2},...,D_{N}]$, where $N$ is the history length) and a
candidate news $D_{c}$ as input, and outputs a candidate-aware user embedding
$\mathbf{u}_{c}$. This embedding is further used to compute a click score
$\hat{y}$ for ranking based on the relevance between $\mathbf{u}_{c}$ and the
hidden embedding of candidate news $D_{c}$ (denoted as $\mathbf{h}_{c}$)
learned in the user model, which can be formulated as
$\hat{y}=f(\mathbf{u}_{c},\mathbf{h}_{c})$, where $f(\cdot)$ is a relevance
function that is often implemented by inner product or feedforward neural
networks.

Figure 1. The framework of FairRank.

###  2.2. Fairness-aware Single-tower Ranking

We then introduce how to learn a fairness-aware single tower ranking model. A
straightforward way is applying adversarial learning to the candidate-aware
user embedding to reduce biases on sensitive attributes. Unfortunately, the
candidate news list to be ranked may also be biased, and it is difficult to
purify the candidate-aware user embedding. To address this challenge, we
propose to use a candidate-aware user model to jointly match clicked news with
a candidate news $D_{c}$ and a randomly selected news $D_{r}$, to learn a
candidate-aware user embedding $\mathbf{u}_{c}$ and a candidate-invariant user
embedding $\mathbf{u}_{r}$, respectively. Since the random news is uniformly
selected from the entire news set, the candidate-invariant user embedding can
reflect the intrinsic user interest in the sense of expectation. We further
apply adversarial learning to both kinds of user embeddings to learn debiased
user interest. We use two independent dense layers to learn hidden
representations of them (denoted as $\mathbf{d}_{c}$ and $\mathbf{d}_{r}$) for
inferring the sensitive user attribute, and a shared attribute predictor is
used as the discriminator to predict sensitive attribute labels (denoted as
$\mathbf{\hat{z}}$ and $\mathbf{\tilde{z}}$) as follows:

(1) $\mathbf{\hat{z}}=\rm{softmax}(\mathbf{W}\mathbf{d}_{c}+\mathbf{b}),$ (2)
$\mathbf{\tilde{z}}=\rm{softmax}(\mathbf{W}\mathbf{d}_{r}+\mathbf{b}),$

where $\mathbf{W}$ and $\mathbf{b}$ are parameters in the discriminator. We
use an adversarial loss $\mathcal{L}_{A}$ to train the discriminator by using
the crossentropy comparing $\mathbf{\hat{z}}$ or $\mathbf{\tilde{z}}$ against
the attribute label. The adversarial gradients of $\mathcal{L}_{A}$ will be
propagated to the user model to remove its encoded biases on sensitive
attributes.

However, existing studies show that adversarial learning may not fully remove
sensitive attribute biases especially when the selection of candidate news has
already been biased (Wu et al., 2021d). We notice that if bias information
does not exist in the user model, the class distribution of the predicted
attribute labels $\mathbf{\hat{z}}$ and $\mathbf{\tilde{z}}$ should be both
uniform (we assume classes are balanced), which means that $\mathbf{\hat{z}}$
and $\mathbf{\tilde{z}}$ should have the same distributions. Thus, we apply an
additional KL divergence loss $\mathcal{L}_{D}$ to $\mathbf{\hat{z}}$ and
$\mathbf{\tilde{z}}$ to regularize them to be similar, which is formulated as
follows:

(3) $\mathcal{L}_{D}=KL(\mathbf{\hat{z}},\mathbf{\tilde{z}}).$

By optimizing this loss function, the bias distribution encoded in candidate
selection is encouraged to be similar to uniform distribution, which means
that less bias information can be encoded by the candidate-aware user model.

###  2.3. Model Training

We finally introduce the learning of FairRank. For the main news ranking task,
following (Wu et al., 2019b) we use negative sampling strategies to build
training samples and use the InfoNCE (Oord et al., 2018) loss for
optimization. The unified loss $\mathcal{L}$ for model training is as follows:

(4) $\mathcal{L}=\mathcal{L}_{R}-\lambda\mathcal{L}_{A}+\mathcal{L}_{D},$

where $\lambda$ is a coefficient to control the relative importance of
adversarial gradients.

##  3\. Experiments

###  3.1. Datasets and Experimental Settings

We use two real-world datasets for experiments. The first one is the dataset
used in (Wu et al., 2021d) (denoted as FairNews). There are 2,484 male users
and 1,744 female users with observed gender labels. The second dataset is
constructed by ourselves by collecting users’ click logs between 06/23/2019
and 07/20/2019 on a commercial news feeds platform (denoted as FairFeeds).
There are 32,124 male users and 17,876 female users with gender labels. The
logs in the first 3 weeks are for training and validation (9:1 split), and the
rest are for test. The statistics of both datasets are listed in Table 1.

Table 1. Statistics of two news datasets.

| #News | #Users | #Impressions | #Clicks
---|---|---|---|---
FairNews | 42,255 | 10,000 | 360,428 | 503,698
FairFeeds | 112,052 | 337,286 | 500,000 | 853,291
|  |  |  |

In our experiments we use Adam (Kingma and Ba, 2015) as the optimizer, and the
learning rate is 1e-4. The adversarial loss weight $\lambda$ is 0.5. The batch
size is 32. We search hyperparameters according to validation results.
Motivated by (Wu et al., 2021d), we choose AUC and nDCG@10 to evaluate ranking
accuracy on impression data, and use the accuracy of gender prediction from
the top 10 and 20 ranked news within 100 randomly selected candidate news as
fairness metrics. These metrics are indications of casual fairness (Zhang and
Bareinboim, 2018). We report the average results of 5 independent experiments.

###  3.2. Fairness Comparison between Single-tower and Two-tower Models

Table 2. Fairness comparison between different single-tower and two-tower
models.

Methods | FairNews | FairFeeds
---|---|---
Acc@10 | Acc@20 | Acc@10 | Acc@20
DKN | 56.72 | 57.87 | 57.69 | 58.53
DAN | 57.50 | 58.38 | 58.20 | 59.11
FIM | 58.32 | 59.64 | 59.33 | 60.17
UNBERT | 58.98 | 60.16 | 59.92 | 60.84
EBNR | 54.85 | 56.10 | 55.24 | 56.33
NAML | 56.22 | 56.83 | 56.84 | 57.40
NPA | 56.42 | 57.01 | 57.11 | 57.82
NRMS | 56.68 | 57.54 | 57.22 | 57.96
PLM-NR | 56.98 | 57.84 | 57.59 | 58.20
|  |  |  |

We first compare the recommendation fairness of different single-tower and
two-tower methods. The compared single-tower models include: (1) DKN (Wang et
al., 2018), a news recommendation method based on candidate-aware attention;
(2) DAN (Zhu et al., 2019), user modeling candidate-aware attention and
attentional LSTM; (3) FIM (Wang et al., 2020), a fine-grained interest
matching approach for news recommendation; (4) UNBERT (Zhang et al., 2021b), a
BERT-based news recommendation method that concatenates candidate news and
clicked news into a single document. The compared two-tower models include:
(1) EBNR (Okura et al., 2017), embedding-based news recommendation with GRU
networks; (2)NAML (Wu et al., 2019a), attentive multi-view learning for news
recommendation; (3) NPA (Wu et al., 2019b), news recommendation with
personalized attention; (4) NRMS (Wu et al., 2019c), multi-head self-attention
networks for news recommendation; (5) PLM-NR (Wu et al., 2021b), pretrained
language model empowered news modeling for news recommendation. The results
are shown in Table 2. We find that most single-tower ranking models generate
unfairer recommendation results than two-tower models. This may be because
single-tower models conduct finer-grained matching between user behaviors and
the candidate (Yu et al., 2021), and thereby has a higher risks of finding
shortcuts and encoding bias information. It shows that single-tower ranking
models are suffered more from the biases encoded in user behavior data, and
learning fair single-tower ranking models is more challenging.

Table 3. Ranking fairness of different methods. Lower scores indicate better
fairness.

Methods | FairNews | FairFeeds
---|---|---
Acc@10 | Acc@20 | Acc@10 | Acc@20
DKN | 56.72 | 57.87 | 57.69 | 58.53
DKN+AL | 55.46 | 56.50 | 56.45 | 57.20
DKN+FairRec | 54.95 | 56.12 | 56.13 | 56.88
DKN+FairRank | 52.42 | 53.50 | 52.92 | 53.70
DAN | 57.50 | 58.38 | 58.20 | 59.11
DAN+AL | 55.92 | 57.09 | 56.46 | 57.32
DAN+FairRec | 55.00 | 55.39 | 55.81 | 56.43
DAN+FairRank | 52.62 | 53.66 | 52.95 | 53.68
FIM | 58.32 | 59.64 | 59.33 | 60.17
FIM+AL | 56.50 | 57.72 | 57.25 | 57.98
FIM+FairRec | 55.65 | 56.33 | 55.94 | 56.69
FIM+FairRank | 53.06 | 53.78 | 53.29 | 53.88
UNBERT | 58.98 | 60.16 | 59.92 | 60.84
UNBERT+AL | 57.47 | 58.81 | 57.92 | 58.80
UNBERT+FairRec | 56.21 | 56.64 | 56.87 | 57.44
UNBERT+FairRank | 53.70 | 54.24 | 53.96 | 54.55
|  |  |  |

Table 4. Ranking accuracy of different methods. Higher scores indicate better
accuracy.

Methods | FairNews | FairFeeds
---|---|---
AUC | nDCG@10 | AUC | nDCG@10
DKN | 60.36 | 38.83 | 65.24 | 32.10
DKN+AL | 59.97 | 38.48 | 64.76 | 31.88
DKN+FairRec | 59.83 | 38.39 | 64.65 | 31.82
DKN+FairRank | 59.91 | 38.44 | 64.69 | 31.86
DAN | 61.40 | 39.78 | 66.12 | 32.56
DAN+AL | 60.82 | 39.25 | 65.59 | 32.29
DAN+FairRec | 60.84 | 39.26 | 65.48 | 32.23
DAN+FairRank | 60.92 | 39.31 | 65.66 | 32.34
FIM | 63.23 | 41.48 | 67.15 | 33.18
FIM+AL | 62.57 | 40.93 | 66.52 | 32.84
FIM+FairRec | 62.64 | 40.98 | 66.60 | 32.88
FIM+FairRank | 62.73 | 41.06 | 66.70 | 32.94
UNBERT | 64.40 | 42.45 | 68.33 | 33.96
UNBERT+AL | 63.87 | 41.90 | 67.67 | 33.60
UNBERT+FairRec | 63.81 | 41.85 | 67.70 | 33.64
UNBERT+FairRank | 63.96 | 41.98 | 67.82 | 33.72
|  |  |  |

(a) FairNews. (b) FairFeeds.

Figure 2. Effectiveness of the candidate-invariant user embedding and KL loss.

(a) FairNews. (b) FairFeeds.

Figure 3. Influence of the coefficient $\lambda$.

###  3.3. Performance Comparison

Next, we verify whether FairRank can effectively mitigate the heavy unfairness
issue of single-tower news ranking framework. We compare FairRank with two
fairness-aware methods, i.e., (1) AL (Beutel et al., 2017), applying
adversarial learning to the candidate-aware user embedding; (2) FairRec (Wu et
al., 2021d), applying the decomposed adversarial learning method to the
candidate-aware user model. The ranking fairness and accuracy of different
methods based on different basic models are shown in Tables 3 and 4,
respectively. From the results, we find all basic single-tower models are
heavily affected by the biases related to sensitive attributes, and their
ranking results are unfair. In addition, both AL and FairRec cannot
effectively reduce the biases encoded in ranking results. This is because the
selection of candidate news within an impression can be biased, and it is
difficult for the single-tower models to remove their biases. By contrast, our
FairRank approach can achieve much better fairness and even slightly better
accuracy than AL and FairRec, showing that FairRank is more appropriate for
learning fair news ranking models.

###  3.4. Ablation Study

We then study the influence of applying adversarial learning to the candidate-
invariant user embedding and the additional KL regularization loss by removing
them from FairRank to compare the accuracy and fairness changes. We use FIM as
the basic model, and the results on FairNews are shown in Fig. 2. We find that
removing bias information from the candidate-invariant user embedding is
important for improving fairness, since it is free from the interference of
biases encoded by impression data and can better remove the intrinsic biases
captured by the user model. In addition, the KL loss can also improve ranking
fairness. This may be because it can help the model to encode less bias
information to make the attribute labels inferred from both candidate-aware
and candidate-invariant user embeddings to be uniform.

###  3.5. Hyperparameter Analysis

Finally, we discuss the impact of adversarial loss weight $\lambda$ on
accuracy and fairness. The results on FairNews are shown in Fig. 3. When
$\lambda$ is larger, the accuracy usually decreases because the adversarial
loss will influence the main recommendation task, while the fairness score
does not have notable improvements when $\lambda$ is larger than 0.5. This is
because the main recommendation task cannot gain sufficient supervision
signals when the adversarial loss is too strong. Thus, we choose $\lambda=0.5$
to achieve good fairness without hurting too much ranking accuracy.

##  4\. Conclusion

In this paper, we propose a fairness-aware single-tower ranking framework for
news recommendation, named FairRank. We propose to use a candidate-aware user
model to match clicked news with both candidate news and random news to learn
a candidate-aware user embedding and a candidate-invariant user embedding,
respectively, to help reduce the bias introduced by candidate news selection.
We apply adversarial learning to them to remove bias information, and further
add a KL loss that regularizes the attribute labels inferred from the two user
embeddings to be similar to make the user model less bias-sensitive.
Experiments on two datasets show that FairRank can effectively improve the
fairness of many single-tower news ranking models with minor accuracy losses.

## References

  * (1)
  * Beutel et al. (2019) Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Li Wei, Yi Wu, Lukasz Heldt, Zhe Zhao, Lichan Hong, Ed H Chi, et al. 2019\.  Fairness in recommendation ranking through pairwise comparisons. In _KDD_. 2212–2220.
  * Beutel et al. (2017) Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H Chi. 2017\.  Data decisions and theoretical implications when adversarially learning fair representations.  _arXiv preprint arXiv:1707.00075_ (2017).
  * Elazar and Goldberg (2018) Yanai Elazar and Yoav Goldberg. 2018.  Adversarial Removal of Demographic Attributes from Text Data. In _EMNLP_. 11–21.
  * Ge et al. (2020) Suyu Ge, Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2020.  Graph Enhanced Representation Learning for News Recommendation. In _WWW_. 2863–2869.
  * Hu et al. (2007) Jian Hu, Hua-Jun Zeng, Hua Li, Cheng Niu, and Zheng Chen. 2007.  Demographic prediction based on user’s browsing behavior. In _WWW_. 151–160.
  * Jia et al. (2021) Qinglin Jia, Jingjie Li, Qi Zhang, Xiuqiang He, and Jieming Zhu. 2021.  RMBERT: News Recommendation via Recurrent Reasoning Memory Network over BERT. In _SIGIR_. 1773–1777.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _ICLR_.
  * Lee et al. (2020) Dongho Lee, Byungkook Oh, Seungmin Seo, and Kyong-Ho Lee. 2020\.  News recommendation with topic-enriched knowledge graphs. In _CIKM_. 695–704.
  * Liu et al. (2020) Danyang Liu, Jianxun Lian, Shiyin Wang, Ying Qiao, Jiun-Hung Chen, Guangzhong Sun, and Xing Xie. 2020.  KRED: Knowledge-aware document representation for news recommendations. In _Recsys_. 200–209.
  * Okura et al. (2017) Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. 2017\.  Embedding-based news recommendation for millions of users. In _KDD_. 1933–1942.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ (2018).
  * Qi et al. (2021a) Tao Qi, Fangzhao Wu, Chuhan Wu, and Yongfeng Huang. 2021a.  Personalized News Recommendation with Knowledge-aware Interactive Matching. In _SIGIR_. 61–70.
  * Qi et al. (2021b) Tao Qi, Fangzhao Wu, Chuhan Wu, Peiru Yang, Yang Yu, Xing Xie, and Yongfeng Huang. 2021b.  HieRec: Hierarchical User Interest Modeling for Personalized News Recommendation. In _ACL_. 5446–5456.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _NIPS_. 5998–6008.
  * Wang et al. (2020) Heyuan Wang, Fangzhao Wu, Zheng Liu, and Xing Xie. 2020\.  Fine-grained Interest Matching for Neural News Recommendation. In _ACL_. 836–845.
  * Wang et al. (2018) Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. 2018\.  DKN: Deep Knowledge-Aware Network for News Recommendation. In _WWW_. 1835–1844.
  * Wu et al. (2019a) Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019a.  Neural News Recommendation with Attentive Multi-View Learning. In _IJCAI_. 3863–3869.
  * Wu et al. (2019b) Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019b.  Npa: Neural news recommendation with personalized attention. In _KDD_. 2576–2584.
  * Wu et al. (2019c) Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. 2019c.  Neural News Recommendation with Multi-Head Self-Attention. In _EMNLP-IJCNLP_. 6390–6395.
  * Wu et al. (2021a) Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. 2021a.  Personalized News Recommendation: A Survey.  _arXiv preprint arXiv:2106.08934_ (2021).
  * Wu et al. (2021b) Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2021b.  Empowering News Recommendation with Pre-trained Language Models. In _SIGIR_. 1652–1656.
  * Wu et al. (2021c) Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2021c.  Two Birds with One Stone: Unified Model Learning for Both Recall and Ranking in News Recommendation.  _arXiv preprint arXiv:2104.07404_ (2021).
  * Wu et al. (2021d) Chuhan Wu, Fangzhao Wu, Xiting Wang, Yongfeng Huang, and Xing Xie. 2021d.  FairRec:Fairness-aware News Recommendation with Decomposed Adversarial Learning. In _AAAI_. 4462–4469.
  * Wu et al. (2021e) Chuhan Wu, Fangzhao Wu, Yang Yu, Tao Qi, Yongfeng Huang, and Qi Liu. 2021e.  NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application. In _EMNLP Findings_. 3285–3295.
  * Wu et al. (2020) Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, et al. 2020\.  MIND: A Large-scale Dataset for News Recommendation. In _ACL_. 3597–3606.
  * Xu et al. (2019) Depeng Xu, Yongkai Wu, Shuhan Yuan, Lu Zhang, and Xintao Wu. 2019.  Achieving causal fairness through generative adversarial networks. In _IJCAI_.
  * Yu et al. (2021) Puxuan Yu, Hongliang Fei, and Ping Li. 2021.  Cross-lingual Language Model Pretraining for Retrieval. In _WWW_. 1029–1039.
  * Zhang et al. (2018) Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018\.  Mitigating unwanted biases with adversarial learning. In _AIES_. 335–340.
  * Zhang and Bareinboim (2018) Junzhe Zhang and Elias Bareinboim. 2018.  Fairness in decision-making—the causal explanation formula. In _AAAI_ , Vol. 32.
  * Zhang et al. (2021a) Qi Zhang, Qinglin Jia, Chuyuan Wang, Jingjie Li, Zhaowei Wang, and Xiuqiang He. 2021a.  AMM: Attentive Multi-field Matching for News Recommendation. In _SIGIR_. 1588–1592.
  * Zhang et al. (2021b) Qi Zhang, Jingjie Li, Qinglin Jia, Chuyuan Wang, Jieming Zhu, Zhaowei Wang, and Xiuqiang He. 2021b.  UNBERT: User-News Matching BERT for News Recommendation. In _IJCAI_. 3356–3362.
  * Zhu et al. (2019) Qiannan Zhu, Xiaofei Zhou, Zeliang Song, Jianlong Tan, and Li Guo. 2019.  Dan: Deep attention neural network for news recommendation. In _AAAI_ , Vol. 33. 5973–5980.