# Reinforcement Learning by Guided Safe Exploration

Qisong Yang Equal contribution. Thiago D. Simão  Nils Jansen  Simon H.
Tindemans  Matthijs T. J. Spaan  Delft University of Technology – The
Netherlands  Radboud University, Nijmegen – The Netherlands

###### Abstract

Safety is critical to broadening the application of reinforcement learning
(RL). Often, we train RL agents in a controlled environment, such as a
laboratory, before deploying them in the real world. However, the real-world
target task might be unknown prior to deployment. Reward-free RL trains an
agent without the reward to adapt quickly once the reward is revealed. We
consider the _constrained_ reward-free setting, where an agent (the guide)
learns to explore safely without the reward signal. This agent is trained in a
controlled environment, which allows unsafe interactions and still provides
the safety signal. After the target task is revealed, safety violations are
not allowed anymore. Thus, the guide is leveraged to compose a safe behaviour
policy. Drawing from transfer learning, we also regularize a target policy
(the student) towards the guide while the student is unreliable and gradually
eliminate the influence of the guide as training progresses. The empirical
analysis shows that this method can achieve safe transfer learning and helps
the student solve the target task faster.

##  1 Introduction

Despite the numerous achievements of reinforcement learning (RL) [45, 35],
safety concerns still prevent the wide adoption of RL [11]. The lack of
knowledge about the environment forces standard agents to rely on trial-and-
error strategies. However, this approach is incompatible with safety-critical
scenarios [15]. For instance, recommender systems should not suggest extremist
content [10]. Constrained Markov decision processes (CMDP) [4] express such
safety constraints with a cost signal indicating unsafe interactions. Such
costs are decoupled from the rewards to facilitate the learning of safe
behaviours.

Developments in safe RL have allowed us to learn safe policies in CMDPs. For
instance, SAC-Lagrangian [19] combines the Soft Actor-Critic (SAC) [21, 22]
algorithm with Lagrangian methods to learn a safe policy in an off-policy way.
This algorithm solves high-dimensional problems with a sample complexity lower
than on-policy algorithms. Unfortunately, it only finds a safe policy at the
end of the training process and may be unsafe while learning. In terms of
safety, we consider episode-wise constraints instead of step-wise constraints,
so a few unsafe actions are allowed in an episode.

Some knowledge about the safety dynamics can ensure safety during learning.
One can pre-compute unsafe behaviour and mask unsafe actions using a so-called
shield [3, 26, 8], or start from an initially safe baseline policy and
gradually improve its performance while remaining safe [2, 49, 56]. However,
these approaches may necessitate numerous interactions with the environment
before they find an adequate policy [59]. Moreover, reusing a pre-trained
policy can have a detrimental effect, since the agent encounters a new
trajectory distribution as the policy changes [24]. Therefore, we investigate
_how to efficiently solve a task without violating the safety constraints_.

We make two key observations. First, RL agents often learn in a controlled
environment, such as a laboratory or a simulator, before being deployed in the
real world [15]. Second, an agent can often benefit from expert guidance
instead of solely relying on trial and error [36]. For instance, in autonomous
driving, the driver agent can quickly learn by mimicking an expert’s behaviour
to handle dangerous situations. Such a process is referred to as policy
distillation. Furthermore, under expert guidance, the agent can safely explore
before taking dangerous actions.

Transfer learning [48] investigates how to improve the learning of a target
task with some knowledge from a source task. In these settings, the source
task may provide only partial knowledge of the target task. We adopt a
transfer learning framework and refer to (i) the controlled environment as the
source task (${\diamond}$) and (ii) the real world as the target task
(${\odot}$). In our setting, the controlled environment provides only the cost
signals related to safety but not the reward signals of the target task in the
real world. The central problem is then to avoid safety violations after the
target task has been revealed.

Our approach. We show how to transfer knowledge encoded by a policy to enhance
safety. Here, we refer to the policy that has been learned in the source task
as the safe guide (SaGui, Figure 1). The intuition is that, in the real world,
the agent is guided to accomplish the target task in a safe manner. We propose
to transfer SaGui from the source task to the target task. Our approach has
three central steps:  1. train the SaGui policy and _transfer_ it to the
target task;  2. _distill_ the guide’s policy into a _student policy_ which is
dedicated to the target task, and  3. _compose_ a behaviour policy that
balances safe exploration (using the guide) and exploitation (using the
student).

$s,c$
---
$a$
---
$s,r,c$
---
${\pi_{b}}$
---
$a$
---
$\pi^{\diamond}$
---
$\pi^{\diamond}$
---
source (controlled environment)
---
target (real world)
---
$\pi^{\diamond}$
---
$\pi^{\odot}$
---
$\mathcal{M}^{{\diamond}}$
---
$\mathcal{M}^{{\odot}}$
---
transfer
---
distillation
---
composition
---

Figure 1: Transferring the Safe Guide (SaGui) policy $\pi^{\diamond}$ from the
source task (${\diamond}$) to the target task (${\odot}$) with three steps.

As we train the guide in a reward-free constrained RL setting [34], the agent
only observes the costs related to safety, and it does not access reward
signals. This task-agnostic approach allows us to train a guide independently
of the reward of the target task, so this guide can be useful for different
reward functions. Furthermore, we assume the source task preserves the
dynamics related to safety, which allows us to train a guide that can act
safely when transferred to the target task. Inspired by advances in robotics
where an agent is trained under strict supervision, we assume the source task
is a simulated/controlled environment [40, 53]. Therefore, safety is not
required while training the SaGui policy. Once the target task is revealed,
SaGui safely collects the initial trajectories in the target environment and
the student starts learning based on these trajectories. To ensure that the
new policy quickly learns how to act safely, we also employ a policy
distillation method, encouraging the student to imitate the guide.

Contributions. Our main contributions are: we  1. formalize transfer learning
for RL from a safety perspective;  2. propose to guide learning using a task-
agnostic agent with exploration benefits;  3. show how to adaptively
regularize the student policy to the guide policy based on the student’s
safety;  4. investigate when to sample from the student or from the guide to
ensure safe behaviour in the target environment and fast convergence of the
student policy; and  5. demonstrate empirically that, compared to learning
from scratch and adapting a pre-trained policy, our method can solve the
target task faster without violating the safety constraints in the target
task.

##  2 Related Work

Safe RL has multiple facets [15], ranging from alternative optimization
criteria [54, 9] to safe exploration based on some prior knowledge [2, 3, 26,
44, 57, 42]. We review methods to train the guide and solve new tasks using a
pre-trained policy.

Multiple algorithms have been proposed for generalizing policies from reward-
free RL for better performance in target tasks [60, 17, 43]. However, only a
few works have considered reward-free RL with constraints [34, 39]. They focus
on tabular and linear settings while we consider general function
approximation algorithms.

Work in transfer learning has leveraged meta-RL [14] for safe adaptation [18,
32, 30]. Our work is also related to curriculum learning [5, 51, 33]. We first
train an agent to be safe and later solve a target task. However, our approach
focuses on safe exploration and is able to transfer to tasks with different
reward functions, so the guide’s training is ignored.

Our work resembles certain safe transfer-RL frameworks [27, 57], which also
leverage prior knowledge to aid learning in a target task. However, the SaGui
framework differs from them in terms of safety definition, knowledge
acquisition in the source task, or knowledge usage in the target task. Our
prior knowledge is more effective for various downstream tasks, and SaGui is
the only framework that is safe while learning in the target task.

##  3 Background

We formalize the safe RL problem and describe typical approaches.

###  3.1 Constrained Markov Decision Processes

We consider tasks formulated by constrained Markov decision processes (CMDPs)
[4, 7]. A CMDP is defined as a tuple
$\mathcal{M}=\langle\mathcal{S},\mathcal{A},\mathcal{P},r,c,d,\gamma\rangle$:
a state space $\mathcal{S}$, an action space $\mathcal{A}$, a probabilistic
transition function
$\mathcal{P}\colon\mathcal{S}\times\mathcal{A}\mapsto\mathit{Dist}(\mathcal{S})$,
a reward function
$r\colon\mathcal{S}\times\mathcal{A}\mapsto[r_{min},r_{max}]$, a cost function
$c\colon\mathcal{S}\times\mathcal{A}\mapsto[c_{min},c_{max}]$, a safety
threshold $d\in\mathbb{R}^{+}$, and a discount factor $\gamma\in[0,1)$. We
also consider an initial state distribution
$\iota\colon\mathcal{S}\mapsto[0,1]$. In a _constrained RL_ problem, an agent
interacts with a CMDP without knowledge about the transition, reward, and cost
functions, generating a trajectory
$\tau=\langle(s_{0},a_{0},r_{0},c_{0},s_{0}^{\prime}),(s_{1},a_{1},r_{1},c_{1},s_{1}^{\prime}),\cdots\rangle$.
A trajectory starts from $s_{0}\sim\iota(\cdot)$. Then, at each timestep $t$
the agent is in a state $s_{t}\in\mathcal{S}$, and takes an action
$a_{t}\in\mathcal{A}$. It subsequently gets a reward $r_{t}=r(s_{t},a_{t})$, a
cost $c_{t}=c(s_{t},a_{t})$, and steps into a new state
$s_{t}^{\prime}\sim\mathcal{P}(\cdot\mid s_{t},a_{t})$. This process repeats
starting from $s_{t+1}=s_{t}^{\prime}$ until a terminal condition is met and a
new trajectory starts. The goal is to learn a policy $\pi$ that maximizes the
expected discounted return such that the expected discounted cost-return
remains below $d$:

$\displaystyle\max_{\pi}$
$\displaystyle\operatorname{\mathbb{E}}_{\rho_{\pi}}\left[\sum_{t=0}^{\infty}\gamma^{t}r_{t}\right]\quad{\rm
s.t.}\quad\operatorname{\mathbb{E}}_{\rho_{\pi}}\left[\sum_{t=0}^{\infty}\gamma^{t}c_{t}\right]\leq
d,$ (1)

where $\rho_{\pi}$ indicates the trajectory distribution induced by
$s_{0}\sim\iota(\cdot)$, $a_{t}\sim\pi(\cdot\mid s_{t})$, and
$s_{t+1}\sim\mathcal{P}(\cdot\mid s_{t},a_{t})$. We define the discounted
return starting from $s,a$ and following $\pi$ as
$Q_{\pi}^{r}(s,a)=\operatorname{\mathbb{E}}_{\rho_{\pi}}\left[\sum^{\infty}_{t=0}\gamma^{t}r_{t}\middle|s_{0}=s,a_{0}=a\right],$
and, similarly, the discounted cost-return $Q_{\pi}^{c}(s,a)$.

From the safe RL perspective, if a policy has an expected cost-return lower
than the safety-threshold $d$, then this policy is considered safe. Therefore,
the objective of a safe RL agent is to find a policy, among the safe policies,
that has the highest expected return.

###  3.2 Maximum Entropy Reinforcement Learning

A common strategy to improve the exploration and robustness of RL agents is to
favour policies that induce diverse behaviours [62, 13]. We can incorporate it
in the safe RL main objective by augmenting the problem with a term that aims
to maximize the policy entropy [20]:

$\max_{\pi}\operatorname{\mathbb{E}}_{\rho_{\pi}}\left[\sum_{t=0}^{\infty}\gamma^{t}\left(r_{t}{+}\alpha\mathcal{H}(\pi(s_{t}))\right)\right]\textrm{
s.t.}\operatorname{\mathbb{E}}_{\rho_{\pi}}\left[\sum_{t=0}^{\infty}\gamma^{t}c_{t}\right]{\leq}d,$
(2)

where $\mathcal{H}(\cdot)$ is the entropy of a distribution over a random
variable, and $\alpha$ is the entropy weight. In general, this objective
encourages the agent to use maximally stochastic policies. Alternatively, we
can encourage the policy to have at least a minimum entropy
${\overline{\mathcal{H}}}$ [22] by adding the following constraint to (1):

$\operatorname{\mathbb{E}}_{\rho_{\pi}}\left[-\log(\pi(a_{t}\mid
s_{t}))\right]\geq{\overline{\mathcal{H}}},\quad\forall t,$ (3)

where ${\overline{\mathcal{H}}}$ is the given entropy threshold to ensure a
minimum degree of randomness. This approach allows the policy to converge to a
more deterministic behaviour than (2). Besides, it only requires the system’s
designer to define ${\overline{\mathcal{H}}}$ and it lets the RL agent
automatically find a trade-off between the policy’s entropy and rewards.
Therefore, $\alpha$ becomes an intrinsic parameter of the RL algorithm.

cost-return
---
episode
---
$\Delta$ time to safety
---
safety
---
jump-start
learning from
---
scratch
transfer
---
safety
---
threshold

(a) Unsafe transfer

cost-return
---
episode
---
safety
---
threshold
$\Delta$ time to safety
---
transfer
---
safety
---
jump-start
learning from
---
scratch

(b) Fully safe transfer.

guided
---
exploration
episode
---
$\Delta$ time to optimum
---
return
---
return
---
jump-start
conservative
---
exploration
optimal
---
performance

(c) Return transfer.

Figure 2: Transfer metrics for safe reinforcement learning. Usually, we
consider safety jump-start and $\Delta$ time to safety. If we can develop
agents that learn without violating the safety requirements, we can also
consider _return jump-start_ and $\Delta$ time to optimum.

The maximum entropy RL with safety constraint (2) can be solved by the SAC-
Lagrangian (SAC-$\lambda$) [19] method. SAC-$\lambda$ is a SAC-based method
that has two critics and uses an adaptive entropy weight $\alpha$
(parameterized by $\theta_{\alpha}$) and an adaptive safety weight $\beta$
(parameterized by $\theta_{\beta}$) to manage a trade-off among exploration,
reward, and safety. The reward critic estimates the expected return $Q^{r}$
(parameterized by $\theta_{R}$), possibly with an entropy bonus to promote
exploration, while the safety critic estimates the cost-return $Q^{c}$
(parameterized by $\theta_{C}$) to encourage safety. The policy $\pi$ is
parameterized by $\theta_{\pi}$. Appendix A provides a detailed description of
how to learn each component, including the losses. Throughout the paper, we
represent learning rates with $\eta$, replay buffers with $\mathcal{D}$, and
losses with $J$. We only update $\alpha$ when a desirable
${\overline{\mathcal{H}}}$ is given, so $\alpha$ is fixed whenever we use the
formulation (2).

##  4 Safe and Efficient Exploration

Naturally, to train RL agents without violating the safety constraints, some
prior knowledge is required [44]. Often, a safe initial policy collects the
initial trajectories [2, 49, 56]. However, these approaches largely neglect
how this policy is computed or what makes it effective. Therefore, we consider
the problem of how to obtain an initial policy that can safely expedite
learning in the target task. Next, we formalize the problem and provide an
overview of our approach.

###  4.1 Problem Setting

We formalize our problem setting using the transfer learning (TL) framework.
In general, TL allows RL agents to use expertise from source tasks to speed up
the learning process on a target task [48, 61]. The source tasks
$\\{\mathcal{M}^{\diamond}\\}$ should provide some knowledge
$\mathcal{K}^{\diamond}$ to an agent learning in the target task
$\mathcal{M}^{\odot}$, such that, by leveraging $\mathcal{K}^{\diamond}$, the
agent learns the target task $\mathcal{M}^{\odot}$ faster.

As we are particularly interested in the safety properties of the transfer, we
consider a reward-free source task, which only provides information about the
safety dynamics. Moreover, we use a policy to encode the knowledge
transferred. Formally, given a source task
$\mathcal{M}^{\diamond}=\langle\mathcal{S}^{\diamond},\mathcal{A}^{\diamond},\mathcal{P}^{\diamond},\emptyset,c^{\diamond},d^{\diamond},\iota^{\diamond},\gamma\rangle$,
we compute the policy $\pi^{\diamond}$ in the absence of a reward signal. This
provides knowledge $\mathcal{K}^{\diamond}=\\{\pi^{\diamond}\\}$ to help
solving the target task
$\mathcal{M}^{\odot}=\langle\mathcal{S}^{\odot},\mathcal{A}^{\odot},\mathcal{P}^{\odot},r^{\odot},c^{\odot},d^{\odot},\iota^{\odot},\gamma\rangle$.

To apply the source policy $\pi^{\diamond}$ in the target task
$\mathcal{S}^{\odot}$, we have a mapping from the source state space to the
target state space $\Xi:\mathcal{S}^{\odot}\rightarrow\mathcal{S}^{\diamond}$.
Then, we can define a target policy $\pi^{{\diamond}\rightarrow{\odot}}$ as
follows: $\pi^{{\diamond}\rightarrow{\odot}}(s)=\pi^{\diamond}(\Xi(s)).$
Furthermore, we assume the source task $\mathcal{M}^{\diamond}$ and target
task $\mathcal{M}^{\odot}$ share the same action space. Section B.1 describes
how to obtain the source task based on $\Xi$ and the target task.

######  Assumption 1.

$\mathcal{A}^{\diamond}=\mathcal{A}^{\odot}=\mathcal{A}$.

To enable the knowledge transferable between tasks, having the same action
spaces ensures that the policy learned in the source task is directly
applicable to the target task.

###  4.2 Transfer Metrics

To evaluate a safe transfer RL algorithm, Figure 2(a) presents a schematic of
metrics related to safety (inspired by transfer in RL [48]): safety jump-start
indicates how much closer to the safety threshold the expected cost-return of
an agent learning with the source knowledge is compared to the expected cost-
return of an agent learning from scratch in the first episodes, and $\Delta$
time to safety is the difference in the number of interactions required to
become safe.

Notice that a trained agent might start with an expected cost-return lower
than the safety threshold, for instance, when the safety threshold in the
source task is lower than in the target task (Figure 2(b)). In this case,
safety jump-start would be the difference between the safety threshold and the
cost-return of an agent learning from scratch. Similarly, the $\Delta$ time to
safety would be the number of interactions an agent learning from scratch
needs to become safe.

In the case of two methods that can solve the target task without violating
the safety constraints, we can also consider the usual metrics of transfer
learning with respect to the reward [48]. For instance, Figure 2(c) shows the
initial improvement in terms of performance which we call _return jump-start_
, and the time necessary to reach an optimum performance, which we call the
$\Delta$ time to optimum.

#### Problem statement.

We aim to maximize the safety jump-start (potentially preventing safety
violations in the target task) and to reduce the time to optimum (improving
exploration) when transferring a policy $\pi^{\diamond}$ from a source task
$\mathcal{M}^{\diamond}$ to a target task $\mathcal{M}^{\odot}$.

###  4.3 Method Overview

Recall that for our transfer setting, we consider a single source task that
only provides the safety signals, which we use to train the guide. Without the
reward signal, the guide aims to explore the world safely and efficiently. We
are interested in using the guide’s safe exploration capabilities to train the
student on the target task without violating the safety constraints. Notably,
i) the guide and the student are trained separately; ii) the guide is only
trained once and can support the training of different students; and iii) the
guide only has access to safety information and no knowledge about the
student’s task.

To ensure the source policy is safe when deployed in the target task, we
assume that the source task has a safety threshold lower than or equal to the
target task, and $\Xi$ is a state abstraction that preserves the safety
dynamics, as formalized next.

######  Assumption 2.

The safety threshold of the target task upper bounds the safety threshold of
the source task: $d^{\diamond}\leq d^{\odot}$.

######  Assumption 3.

$\Xi$ is a $Q_{\pi}^{c}$-irrelevance abstraction [31], therefore

$\Xi(s){=}\Xi(s^{\prime})\Rightarrow
Q_{\pi^{\odot}}^{c}(s,a)=Q_{\pi^{\odot}}^{c}(s^{\prime},a),\forall
s,s^{\prime}\in\mathcal{S}^{\odot},a\in\mathcal{A},\pi^{\odot}.$

Now, we can connect the expected cost-return of a policy on the source task to
the expected cost-return on the target task.

######  Lemma 1.

Given 1 and 3, we have

$Q_{\pi^{\diamond}}^{c,{\diamond}}(\Xi(s),a)=Q_{\pi^{{\diamond}\rightarrow{\odot}}}^{c,{\odot}}(s,a)\quad\forall
s\in\mathcal{S}^{\odot},a\in\mathcal{A},\pi^{\diamond}.$

That is, the expected cost of a source policy is the same in the source task
and in the target task.

###### Proof.

Section B.2 provides the proof. ∎

######  Theorem 4.1.

If $~{}\Xi$ is a $Q_{\pi}^{c}$-irrelevant state abstraction, then any policy
that is safe on the source task $\mathcal{M}^{\diamond}$ is also safe when
deployed on the target task $\mathcal{M}^{\odot}$.

######  Proof 4.2.

$Q_{\pi^{{\diamond}\rightarrow{\odot}}}^{c,{\odot}}(s,a)\overset{\text{\lx@cref{creftype~refnum}{lem:equal_cost}}}{=}Q_{\pi^{{\diamond}}}^{c,{\diamond}}(\Xi(s),a)\overset{\text{Premise}}{\leq}d^{\diamond}\overset{\text{\lx@cref{creftype~refnum}{a:cost}}}{\leq}d^{\odot}.$

It is important to note, however, that the reward function $r^{\odot}$ in the
target task may be unrelated to the state space of the source task
$\mathcal{S}^{\diamond}$. Therefore, although a policy that is safe on the
source task is also safe on the target task, the behaviour required to
accomplish the target task may not be defined on the source task. Consider,
for instance, an agent with access to its position and the position of a
threat. In each target task, the agent might need to visit a different goal
position, which is not defined in the source task. Then, a safe policy may be
conditioned only on the positions of the agent and the threat, but to achieve
the target, the agent must consider the goal position. This highlights the
need to compute a policy dedicated to the target task.

##  5 Guided Safe Exploration

In this section, we consider how to train the safe guide (SaGui) policy. Then,
we describe how the student learns to imitate the SaGui policy after the task
is revealed while learning to complete the target task. Finally, we
investigate how to prevent safety violations while the student has not yet
learned how to act safely.

###  5.1 Training the Safe Guide

Since the source task does not provide information regarding the reward of the
target task, we adopt a reward-free exploration approach to train the guide.
To efficiently explore the world, we first consider maximizing the policy
entropy under safety constraints. Then, we can solve the problem defined in
Equation 2 with $r(s,a)=0:\forall s\in\mathcal{S},a\in\mathcal{A}$ to get a
guide MaxEnt. However, although MaxEnt tends to have diverse behaviours, that
does not imply efficient exploration of the environment. Especially for
continuous state and action spaces, it is possible that a policy provides
limited exploration even if it has high entropy.

To enhance the exploration of the guide, we adopt an auxiliary reward that
motivates the agent to visit novel states. To measure the novelty, we first
define the metric space $(\mathcal{S}^{\ddagger},\delta)$, where
$\mathcal{S}^{\ddagger}$ is an abstracted state space and
$\delta:\mathcal{S}^{\ddagger}\times\mathcal{S}^{\ddagger}\rightarrow[0,\infty)$
is a distance function:

$\displaystyle\delta(s,s^{\prime})=0$ $\displaystyle\Leftrightarrow
s=s^{\prime},$ $\displaystyle\delta(s,s^{\prime})$
$\displaystyle=\delta(s^{\prime},s),\text{and}$
$\displaystyle\delta(s^{\prime},s^{\prime\prime})$
$\displaystyle\leq\delta(s,s^{\prime})+\delta(s,s^{\prime\prime}),$
$\displaystyle\forall s,s^{\prime},s^{\prime\prime}\in\mathcal{S}.$

Note that $\mathcal{S}^{\ddagger}$ may not be the original state space
$\mathcal{S}$. Especially when $\mathcal{S}$ is high-dimensional,
$\mathcal{S}^{\ddagger}$ can be some selected dimensions from $\mathcal{S}$,
or a latent space from representation learning. Next, we define the auxiliary
rewards as the expected distance between the current state and the successor
state:

$\displaystyle
r^{\delta}_{t}(s_{t},a_{t})=\operatorname{\mathbb{E}}_{s_{t+1}\sim\mathcal{P}(\cdot\mid
s_{t},a_{t})}\left[\delta(f^{\ddagger}(s_{t}),f^{\ddagger}(s_{t+1}))\right],$
(4)

where we may apply a potential abstraction
$f^{\ddagger}:\mathcal{S}\rightarrow\mathcal{S}^{\ddagger}$. So, we train the
guide agent by solving the constraint optimization problem (2) based on the
auxiliary reward $r^{\delta}$. Then, we can use SAC-$\lambda$ directly
employed to solve (2), as Algorithm 2 shows (Appendix A). In future research,
we will also investigate different distance functions to understand their
effects on exploration.

This auxiliary reward does not explicitly promote exploration, but we find
that increasing the step size and policy entropy significantly improves
exploration in practice. Overall, our experiment with the auxiliary reward
aimed to evaluate the impact of the exploration of the guide on how safely and
quickly the student learns.

We could also consider more sophisticated reward-free exploration strategies
such as maximizing the entropy of the state occupancy distribution [41, 47,
23]. We leave this as future work and focus on using the guide to improve how
the student learns.

Algorithm 1 Guided Safe Exploration

Input: $\mathcal{M}^{\odot}$, $\pi^{\diamond}$, ${\overline{\mathcal{H}}}$,
$d$
Initialize: $\mathcal{D}\leftarrow\emptyset$, $\theta^{\odot}_{\chi}\text{ for
}\chi\in\\{\pi,R,C,\alpha,\beta\\}$
Output: Optimized parameters $\theta^{\odot}_{\pi}$ for $\pi^{\odot}$

1: for each iteration do

2: for each environment step do

3: if  _linear-decay_ then

4: $b\leftarrow f_{\text{ld}}({\diamond},{\odot})$ $\triangleright$ linearly
eliminate the effect of $\pi^{\diamond}$

5: else if  _control-switch_ then

6: $b\leftarrow f_{\text{cs}}({\diamond},{\odot})$ $\triangleright$
$\pi^{\diamond}$ takes control if unsafe

7: end if

8: $a_{t}\sim{\pi_{b}}(\cdot\mid s_{t})$ $\triangleright$ Composite sampling
(6)

9: $\mathcal{I}_{t}\leftarrow\mathcal{I}(s_{t},a_{t})$ $\triangleright$ IS
ratio (7)

10: $r^{\odot}_{t}\leftarrow r^{\odot}(s_{t},a_{t})$

11: $r^{\diamond}_{t}\leftarrow\log\pi^{\diamond}(a_{t}\mid\Xi(s_{t}))$

12: $c^{\odot}_{t}\leftarrow c^{\odot}(s_{t},a_{t})$

13: $s_{t+1}\sim\mathcal{P}^{\odot}(\cdot\mid s_{t},a_{t})$

14:
$\mathcal{D}\leftarrow\mathcal{D}\cup\\{(s_{t},a_{t},r^{\odot}_{t},r^{\diamond}_{t},c^{\odot}_{t},\mathcal{I}_{t},s_{t+1})\\}$

15: end for

16: for each gradient step do

17: Sample experience from $\mathcal{D}$

18: for $\chi\in\\{\pi,R,C,\alpha,\beta\\}$ do

19:
$\theta^{\odot}_{\chi}\leftarrow\theta^{\odot}_{\chi}-\eta_{\chi}\hat{\nabla}_{\theta^{\odot}_{\chi}}\mathcal{I}J_{\chi}(\theta^{\odot}_{\chi})$
$\triangleright$ Updating $\theta^{\odot}_{\chi}$

20: end for

21: end for

22: end for

###  5.2 Policy Distillation From the Safe Guide

When the agent is trained for a certain task, it is difficult to generalize
when faced with a new task [24]. Similarly, it is not trivial to adjust the
guide’s policy that was trained to explore the environment to perform the
target task. Therefore, we train a new policy, referred as the student,
dedicated to the target task.

We can leverage the guide to quickly learn how to act safely. Through the
mapping function $\Xi$, the transferred policy can be used by most constrained
RL algorithms to regularize the student policy $\pi^{\odot}$ towards the guide
policy $\pi^{\diamond}$ using KL divergences, as shown in Figure 3. So, with
$\pi^{\diamond}$ fixed, we have an augmented reward function
$r^{\prime}_{t}=r^{\odot}_{t}+\omega r^{\textrm{KL}}_{t}+\alpha
r^{\mathcal{H}}_{t},$ where
$r^{\textrm{KL}}_{t}=\log\frac{\pi^{\diamond}(a_{t}\mid\Xi(s_{t}))}{\pi^{\odot}(a_{t}\mid
s_{t})}$ and $r^{\mathcal{H}}_{t}=-\log\pi^{\odot}(a_{t}\mid s_{t})$. The
weights $\omega$ and $\alpha$ indicate the strengths of the KL and entropy
regularization (respectively). Appendix C shows that setting
$r^{\diamond}_{t}=\log\pi^{\diamond}(a_{t}\mid\Xi(s_{t}))$ we obtain $\omega
r^{\textrm{KL}}{+}\alpha r^{\mathcal{H}}=\omega
r^{\diamond}+(\omega+\alpha)r^{\mathcal{H}}$. Therefore, we can define the
student’s objective:

$\displaystyle\mathop{\max}\limits_{\pi^{\odot}}\operatorname{\mathbb{E}}_{\tau\sim\rho_{\pi^{\odot}}}\sum^{\infty}_{t=0}\gamma^{t}\Big{[}r^{\odot}_{t}+\omega
r^{\diamond}_{t}+(\alpha+\omega)r^{\mathcal{H}}_{t}\Big{]}.$ (5)

Student
---
Guide
---
$s^{\diamond}$
---
$s^{\odot}$
---
$\Xi(s^{\odot})$
---
Observation
---
Reward
---
$\qquad r^{\odot}$
---
$r^{\odot}+\omega r^{\textrm{KL}}$
---
$D_{\text{KL}}(\pi^{\diamond}(\cdot|s^{\diamond})\\!\parallel\\!\pi^{\odot}(\cdot|s^{\odot}))$
---
Distillation Bonus
---
safety-related
---
safety-related
---
reward-related
---
$\pi^{\odot}(\cdot\mid s^{\odot})$
---
$\pi^{\diamond}(\cdot\mid s^{\diamond})$
---
$r^{\textrm{KL}}$
---

Figure 3: Overview of the policy distillation. Through the mapping function
$\Xi$, the transferred policy can be used to regularize the student policy
$\pi^{\odot}$ towards the guide policy $\pi^{\diamond}$.

To find an appropriate $\omega$, our goal is to follow the guide more for
safer exploration if the student’s policy is unsafe, but eliminate the
influence from the guide and focus more on the performance if the student’s
policy is safe. Therefore, we propose to set $\omega=\beta$ to determine the
strength of the KL regularization since the adaptive safety weight $\beta$
reflects the safety of the current policy.

In summary, we have an entropy regularized expected return with redefined
(regularized) reward $r^{\prime\prime}_{t}=r^{\odot}_{t}+\beta
r^{\diamond}_{t}.$ This augmented reward encourages the student to yield
actions that are more likely to be generated by the guide. Then, SAC-$\lambda$
can be directly used to solve (5) with the additional entropy constraint
(Algorithm 1, lines 16-19).

###  5.3 Composite Sampling

To enhance safety and improve the student during training (Algorithm 1, lines
2-14), we leverage a composite sampling strategy, which means our behaviour
policy (${\pi_{b}}$) is a mixture of the guide’s policy ($\pi^{\diamond}$) and
the student’s policy ($\pi^{\odot}$). So, at each environment step,
$a_{t}\sim{\pi_{b}}(\cdot\mid s_{t}),s_{t}\in\mathcal{S}^{\odot}$ where

${\pi_{b}}(\cdot\mid
s_{t})=\begin{cases}\pi^{\diamond}(\cdot\mid\Xi(s_{t})),&\quad\text{if
}b={\diamond},\\\ \pi^{\odot}(\cdot\mid
s_{t}),&\quad\text{otherwise}.\end{cases}$ (6)

We investigate two strategies to define $b$.

Linear-decay (Algorithm 3 in Appendix D). This strategy, denoted as
$b=f_{\text{ld}}({\diamond},{\odot})$, linearly decreases the probability of
using $\pi^{\diamond}$ with a constant decay rate after each iteration of the
algorithm, conversely increasing the probability of using $\pi^{\odot}$. We
have two modes with linear-decay: _step-wise_ , where in each time step we may
change ${\pi_{b}}$; and _trajectory-wise_ , where ${\pi_{b}}$ only changes at
the start of a trajectory. The mode is decided before executing an episode,
and smoothly switches from the complete _step-wise_ to the complete
_trajectory-wise_ over the training process.

Control-switch (Algorithm 4 in Appendix D). To balance between the safe
exploration and the sample efficiency (the samples from the target policy is
relatively more valuable), the student policy keeps sampling, i.e.,
${\pi_{b}}=\pi^{\odot}$ at the start of a trajectory; after we meet the first
$c_{t-1}>0$, we have ${\pi_{b}}=\pi^{\diamond}$ until the end of the
trajectory. Therefore, the guide policy serves as a rescue policy to improve
safety during sampling. We denote this strategy as
$b=f_{\text{cs}}({\diamond},{\odot})$.

With the composite sampling strategy, the function approximation may diverge,
because $\pi^{\odot}$ and ${\pi_{b}}$ are too different, especially when we
collect most data following $\pi^{\diamond}$. This phenomenon is related to
the deadly triad [46]. To eliminate its negative effect, we endow each sample
with an importance sampling (IS) ratio:

$\mathcal{I}(s,a)=\min\left(\max\left(\frac{\pi^{\odot}(a\mid
s)}{{\pi_{b}}(a\mid s)},\mathcal{I}_{l}\right),\mathcal{I}_{u}\right).$ (7)

The clipping hyper-parameters $\mathcal{I}_{u}$ and $\mathcal{I}_{l}$ are
introduced to reduce the variance of the off-policy TD target. Notice that if
${\pi_{b}}$ is using the student $\pi^{\odot}$ then $\mathcal{I}(s,a)=1$.
Here, in addition to use the IS ratio $\mathcal{I}$ for learning values (the
critics), we also use it in the policy update, as shown in line 19 of
Algorithm 1.

##  6 Empirical Analysis

We evaluate how well our method transfers from the reward-free setting using
the SafetyGym engine [38], where a random-initialized robot navigates in a 2D
map to reach target positions while trying to avoid dangerous areas and
obstacles (Figure 4). These tasks are particularly complex due to the
observation space; instead of observing its location, the agent observes the
relative location of other objects with a lidar sensor. We considered three
environments with different complexity levels. A static environment with a
point robot and a hazard. The locations of the hazard and goal are fixed in
all episodes. A semi-dynamic environment with a car robot, four hazards, and
four vases. The locations of the hazards and vases are the same in all
episodes. The location of the goal is randomly-initialized in each episode. A
dynamic environment with a point robot, eight hazards, and a vase. The
locations of the goal, vase, and hazards are randomly-initialized in each
episode.

Figure 4: Navigation tasks with different complexity levels: static where all
objects are fixed (left), semi-dynamic where the goal is randomly initialized
before each episode (center), and dynamic where all objects are randomly
initialized locations before each episode (right).

The guide agent is trained without the goals, and its auxiliary reward is the
magnitude of displacement at each time step. We provide a detailed description
of the safety-mapping function in Appendix G. Since our focus is on the target
task and the guide is trained in a controlled environment, we do not consider
the guide’s training in the evaluation. In the target tasks, we use the
original reward signal from Safety Gym, i.e., the distance towards the goal
plus a constant for finishing the task [38]. In all environments: $c=1$, if an
unsafe interaction happens, and $c=0$, otherwise. We repeat each experiment 10
times with different random seeds and the plots show the mean and standard
deviation of all runs.

To evaluate the performance during training, we use the following metrics:
safety of the behaviour policy (Cost-Return ${\pi_{b}}$), performance of the
behaviour policy (Return ${\pi_{b}}$), safety of the target policy (Cost-
Return $\pi^{\odot}$), and performance of the target policy (Return
$\pi^{\odot}$). To check the convergence of the target policy, we have a test
process with 100 episodes after each epoch (in parallel to the training) to
evaluate Return $\pi^{\odot}$ and Cost-Return $\pi^{\odot}$. Appendix F
reports the evaluation of $\pi^{\odot}$ and Appendix G the hyperparameters
used. The supplemental material provides the code of the experiments.

###  6.1 Ablation Study

We investigate each component of the proposed SaGui algorithm individually to
answer the following questions: i) Does the auxiliary reward enlarge the
exploration range? ii) Does a better guide agent result in a better student in
the target task? iii) How does the adaptive strength of the KL regularization
affect the performance? iv) How does the composite sampling benefit the safe
transfer learning?

i) Auxiliary reward leads to more diverse trajectories. We performed an
ablation of our approach where no auxiliary reward is added while training the
guide agent, called MaxEnt. We refer to the agent with the auxiliary reward as
SaGui. This teases apart the role the designed auxiliary task plays in the
exploration. In Figure 5, we can see that SaGui can explore larger areas in
Static and Semi-Dynamic, which have the same layout in each episode. We notice
that MaxEnt is safe, but the explored space is limited. That is also the case
in Dynamic, as shown in the attached videos.

ii) An effective guide can speed up the student’s training. We compare how
these guides (MaxEnt and SaGui) affect the learning in the target task. In
Figure 7 (Appendix E), we notice that both methods can collect samples safely,
but the agent using the auxiliary reward needs fewer interactions to find
high-performing policies.

iii) Safety-adaptive regularization improves the student’s convergence rate.
To combine the original reward with the bonus to follow the guide ($\omega$),
we have the following choices: fix the weights of the bonus and make it to be
a hyperparameter to tune (FixReg); apply a decay rate to linearly decrease the
weights during training (DecReg); and, adapt the weights of the bonus based on
the safety performance (SaGui). In Figure 7(a) (Appendix E) we observe that
this weight does not affect the safety of the agent, but both FixReg and
DecReg cause the student to converge slower in terms of performance (Figure
7(b) in Appendix E).

iv) Composite sampling enhances safety and final performance. We modify the
composite sampling approach, sampling only from the guide (GuiSam) or the
student (StuSam) instead. From the results in Figure 7(a) (Appendix E), we can
see that GuiSam can ensure safety, but the student does not learn a safe
optimal policy (Figure 7(b) in Appendix E). Compared to our method, StuSam
performs similarly converging to a safe target policy, but fails to satisfy
the constraint at the early stage of training. So, composite sampling is
necessary to avoid the dangerous actions from a naive policy and to ensure the
target task is solved.

Static | Semi-Dynamic
---|---
|  |  |
MaxEnt | SaGui | MaxEnt | SaGui
Figure 5: Exploration analysis with trajectories collected by the different
guide agents in Static and Semi-Dynamic.

(a) Static

(b) Semi-Dynamic

(c) Dynamic

Figure 6: Evaluation of ${\pi_{b}}$ for CPO, CPO-pre, SAC-$\lambda$,
SAC-$\lambda$-pre, EGPO, and SaGui over 10 seeds. The solid lines are the
average of all runs, and the shaded area is the standard deviation. The black
dashed lines indicate the safety thresholds.

###  6.2 Comparison with Baselines

Finally, we compare SaGui (control-switch) and SaGui (linear-decay) with five
baselines, divided into three groups.

Learning from scratch.

(1) SAC-$\lambda$ [19] shows the performance when starting to learn from
scratch, representing an off-policy algorithm. Similarly, (2) CPO [2] is an
on-policy algorithm that maximizes the reward in a small neighbourhood to
enforce the safety constraints.

Pre-training.

(3) CPO-pre and (4) SAC-$\lambda$-pre demonstrate how CPO and SAC-$\lambda$
perform after being pre-trained in a task that replaces the target reward by
the auxiliary reward. So, we also encourage exploration in the task for pre-
training, which shares the same observation space with the target task.

Expert-in-the-loop.

(5) As an upper bound, we also consider the Expert Guided Policy Optimization
( EGPO) [36] algorithm, which uses knowledge from the target task in the form
of an expert to train a student policy. EGPO proposes a guardian mechanism
that replaces the actions of the student by the expert when the student takes
actions too different from the expert. In summary, EGPO uses an expert policy
as a demonstrator as well as a safety guardian (see Appendix H for more
details).

Notice, for CPO-pre, SAC-$\lambda$-pre and EGPO we adapt the source task to
have the same observation space as the target task, which gives them an
advantage compared to SaGui. Furthermore, EGPO has access to a policy trained
on the target task, while SaGui only has access to the source task without the
goal observations.

Safety during training. In Figure 6, we observe that SaGui (control-switch)
and EGPO are the only methods that exhibit safe behaviour during the full
training process.

Learning from scratch is unsafe and may converge to sub-optimal and even
unsafe policies. SAC-$\lambda$ and CPO can learn safe policies in relatively
simpler environments (Static and Semi-Dynamic) but they violate the safety
constraints at the beginning of training, which is expected. In Dynamic,
SAC-$\lambda$ and CPO fail to attain safe performance. However, with benefits
from the guide, SaGui (control-switch), on the basis of SAC-$\lambda$, attains
a better balance between safety and performance.

Pre-training is insufficient. With pre-training, a safe initialization cannot
benefit CPO-pre and SAC-$\lambda$-pre in safety, and may have negative
effects. We infer that it is difficult to generalize a task when faced with a
new reward signal [24]. Especially for SAC-$\lambda$-pre with an initialized
$Q^{r}$, the difficulty to adapt is evident.

Fast convergence rates. Benefiting from the targeted expert policy, the
behaviour policy of EGPO has a high return throughout the training in the
target environment. But SaGui (control-switch) quickly finds policies with
similar performance despite lack of knowledge of the target task (Figure 6).

The distillation mechanism ensures the safety of the target policy. Figure 8
(Appendix F) shows that SaGui (control-switch) can learn a well-performing
target policy in a safe way. Without the policy distillation mechanism like
SaGui, EGPO (learning only from the expert demonstrations) fails to find a
safe target policy. This indicates that the target policy computed with SaGui
may eventually take full control of the target task, while the policy computed
by EGPO may still require interventions from the expert.

Control-switch can be more effective than linear-decay. SaGui (linear-decay),
which lacks samples from $\pi^{\odot}$ at the early stage of training, does
not achieve similar performance as SaGui (control-switch). Figures 6(b) and
6(c) show that linear-decay fails to compose the behaviour policy ${\pi_{b}}$
safely.

Summary. Overall, SaGui does not violate the safety constraints on the target
environment, quickly finds high-performing policies, and can train a student
able to act independently from the guide.

##  7 Conclusion

This work handles multiple challenges of reinforcement learning with safety
constraints. It shows how we can use a safe exploration policy (the guide)
during data collection and gradually switch to a policy that is dedicated to
the target task (the student). It tackles the off-policy issue that arises
from collecting data with a policy different from the target policy. It shows
how the student can make the best use of the guide’s policy using an incentive
to imitate the guide, which makes the student learn faster how to behave
safely. It demonstrates that simply initializing an agent with a safe policy
may not be as effective as learning a new policy dedicated to the target task
through policy distillation. Finally, it proposes a method that can collect
diverse trajectories, which reduces the sample complexity of the student on
the target task. In summary, the framework proposed is a safe and sample-
efficient way of training the agent on a target task.

Limitations. Our framework assumes that the source task provides information
on the cost function, allowing the guide policy to accumulate the same cost in
the target task as in the source task (Section 4.3). This assumption enables
safe learning in the target task. However, if the cost function or trajectory
distribution changes, the source task may not provide useful safety
information for the target task. In such cases, alternative methods should be
considered to ensure safe exploration. We focus on downstream tasks where pre-
trained agents are utilized for safe exploration knowledge, disregarding the
interactions used to train the SaGui policy. Sample efficiency in the target
task is emphasized, not including samples used for source task learning.
Nevertheless, the pre-trained policy can be reused for multiple target tasks,
enabling us to amortize the guide’s training across them, making the number of
samples required to train the guide negligible as the number of downstream
tasks increases. While efficient learning of a SaGui policy is a significant
challenge, we view it as a separate research direction [23].

Future work. While we consider a relatively simple strategy to achieve rich
exploration, our framework allows the translation of any progress in reward-
free RL into training the guide agent. For instance, we could adopt works with
the entropy of the state density [23, 29, 41, 25, 60, 47, 52, 37, 55], or with
the adaptive reward functions to explore various skills [12]. Another option
to improve exploration is to find a set of diverse policies to the same
problem [16, 28, 58]. Our framework could easily combine multiple guides. As
to composite sampling strategies, recovery and shielding mechanisms [3, 50]
could be further explored to combine with a safe guide, in particular using
the control-switch mechanism that we evaluated. Nevertheless, we highlight
that while a student using a recovery policy must explore alone, the safe
guide can enhance the student’s exploration, accelerating the learning of the
target task.

We thank the reviewers for their insightful comments. This work has been
partially funded by the ERC Starting Grant 101077178 (DEUCE) and the NWO grant
NWA.1160.18.238 (PrimaVera). Qisong Yang is supported by Xidian University.

## References

  * [1] David Abel, D. Ellis Hershkowitz, and Michael L. Littman, ‘Near optimal behavior via approximate state abstraction’, in ICML, pp. 2915–2923, (2016).
  * [2] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel, ‘Constrained Policy Optimization’, in ICML, pp. 22–31, (2017).
  * [3] Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu, ‘Safe Reinforcement Learning via Shielding’, in AAAI, pp. 2669–2678, (2018).
  * [4] Eitan Altman, Constrained Markov decision processes, volume 7, CRC Press, 1999.
  * [5] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston, ‘Curriculum learning’, in ICML, pp. 41–48, (2009).
  * [6] Dimitri P Bertsekas, Constrained Optimization and Lagrange Multiplier Methods, volume 1, Academic press, 1982.
  * [7] Vivek S Borkar, ‘An actor-critic algorithm for constrained Markov decision processes’, Systems & control letters, 54(3), 207–213, (2005).
  * [8] Steven Carr, Nils Jansen, Sebastian Junges, and Ufuk Topcu, ‘Safe reinforcement learning via shielding under partial observability’, in AAAI, pp. 14748–14756, (2023).
  * [9] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone, ‘Risk-constrained reinforcement learning with percentile risk criteria’, JMLR, 18(1), 6070–6120, (2017).
  * [10] Tommaso Di Noia, Nava Tintarev, Panagiota Fatourou, and Markus Schedl, ‘Recommender systems under european ai regulations’, Communications of the ACM, 65(4), 69–73, (2022).
  * [11] Gabriel Dulac-Arnold, Nir Levine, Daniel J. Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester, ‘Challenges of real-world reinforcement learning: definitions, benchmarks and analysis’, Mach. Learn., 110(9), 2419–2468, (2021).
  * [12] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine, ‘Diversity is all you need: Learning skills without a reward function’, in ICLR, (2019).
  * [13] Benjamin Eysenbach and Sergey Levine, ‘Maximum entropy RL (provably) solves some robust RL problems’, in ICLR, (2022).
  * [14] Chelsea Finn, Pieter Abbeel, and Sergey Levine, ‘Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks’, in ICML, pp. 1126–1135, (2017).
  * [15] Javier García and Fernando Fernández, ‘A Comprehensive Survey on Safe Reinforcement Learning’, JMLR, 16(1), 1437–1480, (2015).
  * [16] Mahsa Ghasemi, Evan Scope Crafts, Bo Zhao, and Ufuk Topcu, ‘Multiple Plans are Better than One: Diverse Stochastic Planning’, in ICAPS, pp. 140–148, (2021).
  * [17] Michael Gimelfarb, Andre Barreto, Scott Sanner, and Chi-Guhn Lee, ‘Risk-aware transfer in reinforcement learning using successor features’, in NeurIPS, pp. 17298–17310, (2021).
  * [18] Djordje Grbic and Sebastian Risi, ‘Safe Reinforcement Learning through Meta-learned Instincts’, in ALIFE, pp. 183–291, (2020).
  * [19] Sehoon Ha, Peng Xu, Zhenyu Tan, Sergey Levine, and Jie Tan, ‘Learning to Walk in the Real World with Minimal Human Effort’, in CoRL, pp. 1110–1120, (2020).
  * [20] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine, ‘Reinforcement Learning with Deep Energy-Based Policies’, in ICML, pp. 1352–1361, (2017).
  * [21] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine, ‘Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor’, in ICML, pp. 1861–1870, (2018).
  * [22] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine.  Soft Actor-Critic Algorithms and Applications, 2018.  _arXiv:1812.05905_.
  * [23] Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest, ‘Provably Efficient Maximum Entropy Exploration’, in ICML, pp. 2681–2691, (2019).
  * [24] Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson, ‘Transient Non-stationarity and Generalisation in Deep Reinforcement Learning’, in ICLR, (2021).
  * [25] Riashat Islam, Zafarali Ahmed, and Doina Precup.  Marginalized State Distribution Entropy Regularization in Policy Optimization, 2019.  _arXiv:1912.05128_.
  * [26] Nils Jansen, Bettina Könighofer, Sebastian Junges, Alex Serban, and Roderick Bloem, ‘Safe Reinforcement Learning Using Probabilistic Shields (Invited Paper)’, in CONCUR, pp. 1–16, (2020).
  * [27] Thommen George Karimpanal, Santu Rana, Sunil Gupta, Truyen Tran, and Svetha Venkatesh, ‘Learning transferable domain priors for safe exploration in reinforcement learning’, in IJCNN, pp. 1–10. IEEE, (2020).
  * [28] Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn, ‘One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL’, in NeurIPS, p. 8198–8210, (2020).
  * [29] Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov.  Efficient Exploration via State Marginal Matching, 2019.  _arXiv:1906.05274_.
  * [30] Thomas Lew, Apoorva Sharma, James Harrison, Andrew Bylard, and Marco Pavone, ‘Safe Active Dynamics Learning and Control: A Sequential Exploration–Exploitation Framework’, IEEE Transactions on Robotics, 38(5), 2888–2907, (2022).
  * [31] Lihong Li, Thomas J Walsh, and Michael L Littman, ‘Towards a Unified Theory of State Abstraction for MDPs’, in AI&M, pp. 1–10, (2006).
  * [32] Michael Luo, Ashwin Balakrishna, Brijen Thananjeyan, Suraj Nair, Julian Ibarz, Jie Tan, Chelsea Finn, Ion Stoica, and Ken Goldberg.  MESA: Offline Meta-RL for Safe Adaptation and Fault Tolerance, 2021\.  _arXiv:2112.03575_.
  * [33] Luca Marzari, Davide Corsi, Enrico Marchesini, and Alessandro Farinelli, ‘Curriculum learning for safe mapless navigation’, in SAC, pp. 766–769, (2022).
  * [34] Sobhan Miryoosefi and Chi Jin, ‘A simple reward-free approach to constrained reinforcement learning’, in ICML, pp. 15666–15698, (2022).
  * [35] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis, ‘Human-level control through deep reinforcement learning’, Nature, 518(7540), 529–533, (2015).
  * [36] Zhenghao Peng, Quanyi Li, Chunxiao Liu, and Bolei Zhou, ‘Safe driving via expert guided policy optimization’, in CoRL, pp. 1554–1563, (2022).
  * [37] Zengyi Qin, Yuxiao Chen, and Chuchu Fan, ‘Density Constrained Reinforcement Learning’, in ICML, pp. 8682–8692, (2021).
  * [38] Alex Ray, Joshua Achiam, and Dario Amodei.  Benchmarking Safe Exploration in Deep Reinforcement Learning, 2019.  https://cdn.openai.com/safexp-short.pdf.
  * [39] Yagiz Savas, Melkior Ornik, Murat Cubuktepe, and Ufuk Topcu, ‘Entropy Maximization for Constrained Markov Decision Processes’, in 56th Annual Allerton Conference on Communication, Control, and Computing, pp. 911–918, (2018).
  * [40] Erik Schuitema, Martijn Wisse, Thijs Ramakers, and Pieter Jonker, ‘The design of LEO: A 2D bipedal walking robot for online autonomous Reinforcement Learning’, in IROS, pp. 3238–3243, (2010).
  * [41] Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee, ‘State Entropy Maximization with Random Encoders for Efficient Exploration’, in ICML, pp. 9443–9454, (2021).
  * [42] Thiago D. Simão, Nils Jansen, and Matthijs T. J. Spaan, ‘AlwaysSafe: Reinforcement Learning Without Safety Constraint Violations During Training’, in AAMAS, p. 1226–1235, (2021).
  * [43] Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn.  Learning to be Safe: Deep RL with a Safety Critic, 2020.  _arXiv:2010.14603_.
  * [44] Yanan Sui, Alkis Gotovos, Joel Burdick, and Andreas Krause, ‘Safe Exploration for Optimization with Gaussian Processes’, in ICML, pp. 997–1005, (2015).
  * [45] Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, volume 2, MIT press, 2018.
  * [46] Richard S. Sutton, A. Rupam Mahmood, and Martha White, ‘An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning’, JMLR, 17(1), 2603–2631, (2016).
  * [47] Oleg Svidchenko and Aleksei Shpilman.  Maximum Entropy Model-based Reinforcement Learning, 2021.  _arXiv:2112.01195_.
  * [48] Matthew E. Taylor and Peter Stone, ‘Transfer Learning for Reinforcement Learning Domains: A Survey’, JMLR, 10(56), 1633–1685, (2009).
  * [49] Chen Tessler, Daniel J. Mankowitz, and Shie Mannor, ‘Reward Constrained Policy Optimization’, in ICLR, (2019).
  * [50] Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg, ‘Recovery RL: Safe Reinforcement Learning With Learned Recovery Zones’, IEEE Robotics and Automation Letters, 6(3), 4915–4922, (2021).
  * [51] Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal, ‘Safe Reinforcement Learning via Curriculum Induction’, in NeurIPS, pp. 12151–12162, (2020).
  * [52] Giulia Vezzani, Abhishek Gupta, Lorenzo Natale, and Pieter Abbeel.  Learning latent state representation for speeding up exploration, 2019\.  _arXiv:1905.12621_.
  * [53] Zhaoming Xie, Patrick Clary, Jeremy Dao, Pedro Morais, Jonathan W. Hurst, and Michiel van de Panne, ‘Learning Locomotion Skills for Cassie: Iterative Design and Sim-to-Real’, in CoRL, pp. 317–329, (2019).
  * [54] Qisong Yang, Thiago D. Simão, Simon H. Tindemans, and Matthijs T. J. Spaan, ‘WCSAC: Worst-Case Soft Actor Critic for Safety-Constrained Reinforcement Learning’, in AAAI, pp. 10639–10646, (2021).
  * [55] Qisong Yang and Matthijs T. J. Spaan, ‘CEM: Constrained Entropy Maximization for Task-Agnostic Safe Exploration’, in AAAI, pp. 10798–10806, (2023).
  * [56] Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge, ‘Projection-Based Constrained Policy Optimization’, in ICLR, (2020).
  * [57] Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge, ‘Accelerating safe reinforcement learning with constraint-mismatched baseline policies’, in ICML, pp. 11795–11807, (2021).
  * [58] Tom Zahavy, Brendan O’Donoghue, André Barreto, Sebastian Flennerhag, Volodymyr Mnih, and Satinder Singh.  Discovering Diverse Nearly Optimal Policies with Successor Features, 2021.  _arXiv:2106.00669_.
  * [59] Moritz A. Zanger, Karam Daaboul, and J. Marius Zöllner, ‘Safe Continuous Control with Constrained Model-Based Policy Optimization’, in IROS, pp. 3512–3519, (2021).
  * [60] Jesse Zhang, Brian Cheung, Chelsea Finn, Sergey Levine, and Dinesh Jayaraman, ‘Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings’, in ICML, p. 11055–11065, (2020).
  * [61] Zhuangdi Zhu, Kaixiang Lin, Anil K. Jain, and Jiayu Zhou, ‘Transfer learning in deep reinforcement learning: A survey’, IEEE Transactions on Pattern Analysis and Machine Intelligence, 1–20, (2023).
  * [62] Brian D Ziebart, Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy, Ph.D. dissertation, Carnegie Mellon University, 2010\.

##  Appendix A SAC-Lagrangian

In this section, we present how we learn the parameters in SAC-$\lambda$. In
SAC-$\lambda$, the constrained optimization problem is solved by Lagrangian
methods [6], where an entropy weight $\alpha$ and a safety weight $\beta$
(Lagrange-multipliers) are introduced to the constrained optimization:

$\max_{\pi}\min_{\alpha\geq 0}\min_{\beta\geq 0}f(\pi)-\alpha e(\pi)-\beta
g(\pi),$ (8)

where
$f(\pi)=\operatorname{\mathbb{E}}_{s_{0}\sim\iota(\cdot),a_{0}\sim\pi(\cdot\mid
s_{0})}\left[Q_{\pi}^{r}(s_{0},a_{0})\right]$,
$e(\pi)=\operatorname{\mathbb{E}}_{s_{t}\sim\rho_{\pi}}\left[\log(\pi(\cdot\mid
s_{t}))+{\overline{\mathcal{H}}}\right]$ , and
$g(\pi)=\operatorname{\mathbb{E}}_{s_{0}\sim\iota(\cdot),a_{0}\sim\pi(\cdot\mid
s_{0})}\left[Q_{\pi}^{c}(s_{0},a_{0})-d\right]$. In (8), the max-min
optimization problem can be solved by gradient ascent on $\pi$, and descent on
$\alpha$ and $\beta$.

Initially, SAC-$\lambda$ was developed for local constraints, which means that
the safety cost is constrained at each timestep [19]. However, it can be
easily generalized to constrain the expected cost-return111A similar approach
can be found at https://github.com/openai/safety-starter-agents..

Using a similar formulation [22], we can get the actor loss:

$J_{\pi}(\theta_{\pi})=-\operatorname{\mathbb{E}}_{\begin{subarray}{c}s_{t}\sim\mathcal{D}\\\
a_{t}\sim\pi(\cdot\mid
s_{t})\end{subarray}}\left[Q_{\pi}^{r}(s_{t},a_{t})-\alpha\log\pi(a_{t}\mid
s_{t})-\beta Q_{\pi}^{c}(s_{t},a_{t})\right],$ (9)

where $\mathcal{D}$ is the replay buffer and $\theta_{\pi}$ indicates the
parameters of the policy $\pi$.

The safety and reward critics (including a bonus for the policy entropy) are,
respectively, trained to minimize

$J_{C}(\theta_{C})\\!=\\!\\!\operatorname{\mathbb{E}}_{(s_{t},a_{t})\sim\mathcal{D}}\left[\frac{1}{2}\left(Q^{c}_{\theta_{C}}(s_{t},a_{t})-(c_{t}\\!+\\!\gamma
Q^{c}_{\theta_{C}}(s_{t\\!+\\!1},a_{t\\!+\\!1}))\right)^{2}\right]$ (10)

and

$\displaystyle
J_{R}(\theta_{R})=\operatorname{\mathbb{E}}_{(s_{t},a_{t})\sim\mathcal{D}}\Big{[}\frac{1}{2}(Q^{r}_{\theta_{R}}(s_{t},a_{t})-(r_{t}+\gamma(Q^{r}_{\theta_{R}}(s_{t+1},a_{t+1})-\alpha\log(\pi(a_{t+1}\mid
s_{t+1})))))^{2}\Big{]},$ (11)

where $a_{t+1}\sim\pi(\cdot\mid s_{t+1})$, $Q^{c}$ and $Q^{r}$ are
parameterized by $\theta_{C}$ and $\theta_{R}$, respectively.

Algorithm 2 Maximum exploration RL for safe guide

Input: $\mathcal{M}^{\diamond}$, $\alpha$, $d$
Initialize: $\mathcal{D}\leftarrow\emptyset$, $\theta^{\diamond}_{\chi}\text{
for }\chi\in\\{\pi,R,C,\beta\\}$
Output: Optimized parameters $\theta^{\diamond}_{\pi}$ for $\pi^{\diamond}$

1: for each iteration do

2: for each environment step do

3: $a_{t}\sim\pi^{\diamond}(\cdot\mid s_{t})$

4: $s_{t+1}\sim\mathcal{P}(\cdot\mid s_{t},a_{t})$

5: $r^{\delta}_{t}\leftarrow\delta(f^{\ddagger}(s_{t}),f^{\ddagger}(s_{t+1}))$
$\triangleright$ Auxiliary task (4)

6: $c^{\diamond}_{t}\leftarrow c^{\diamond}(s_{t},a_{t})$

7:
$\mathcal{D}\leftarrow\mathcal{D}\cup\\{(s_{t},a_{t},r^{\delta}_{t},c^{\diamond}_{t},s_{t+1})\\}$
$\triangleright$ Replay buffer

8: end for

9: for each gradient step do

10: Sample experience from replay buffer $\mathcal{D}$

11: for $\chi\in\\{\pi,R,C,\beta\\}$ do

12:
$\theta^{\diamond}_{\chi}\leftarrow\theta^{\diamond}_{\chi}-\eta_{\chi}\hat{\nabla}_{\theta^{\diamond}_{\chi}}J_{\chi}(\theta^{\diamond}_{\chi})$
$\triangleright$ Parameter updating

13: end for

14: end for

15: end for

Finally, let $\theta_{\alpha}$ and $\theta_{\beta}$ be the parameters learned
for the exploration and safety weight such that $\alpha={\rm
softplus}(\theta_{\alpha})$ and $\beta={\rm softplus}(\theta_{\beta})$, where

$\operatorname*{softplus}(x)=\log(\exp(x)+1).$

We can learn $\alpha$ and $\beta$ by minimizing the loss functions:

$\displaystyle
J_{\alpha}(\theta_{\alpha})=\operatorname{\mathbb{E}}_{\begin{subarray}{c}s_{t}\sim\mathcal{D}\\\
a_{t}\sim\pi(\cdot\mid s_{t})\end{subarray}}\left[-\alpha(\log(\pi(a_{t}\mid
s_{t}))+{\overline{\mathcal{H}}})\right],$ (12)

and

$\displaystyle
J_{\beta}(\theta_{\beta})=\operatorname{\mathbb{E}}_{\begin{subarray}{c}s_{t}\sim\mathcal{D}\\\
a_{t}\sim\pi(\cdot\mid
s_{t})\end{subarray}}\left[\beta(d-Q_{\pi}^{c}(s_{t},a_{t}))\right].$ (13)

So the corresponding weight will be adjusted if the constraints are violated,
that is, if we estimate that the current policy is unsafe or if it does not
have enough entropy.

In this paper, we train the guide agent by solving the constraint optimization
problem (2) based on the auxiliary reward $r^{\delta}$, defined by (4). Then,
we can use SAC-$\lambda$ directly employed to solve (2), as Algorithm 2 shows.

##  Appendix B Relation between source and target tasks

In this section, we describe the source task given a target task and the
mapping from the target task to the source task.

###  B.1 State Abstraction

To build the source task based on a target task and a mapping $\Xi$ from the
target state space to the source state space, we assume $\Xi$ is a state
abstraction function [31].

Let
$\mathcal{M}^{\odot}=\langle\mathcal{S}^{\odot},\mathcal{A}^{\odot},\mathcal{P}^{\odot},r^{\odot},c^{\odot},d^{\odot},\iota^{\odot},\gamma\rangle$
be the target task,
$\mathcal{M}^{\diamond}=\langle\mathcal{S}^{\diamond},\mathcal{A}^{\diamond},\mathcal{P}^{\diamond},\emptyset,c^{\diamond},d^{\diamond},\iota^{\diamond},\gamma\rangle$
be the source task, and
$\Xi:\mathcal{S}^{\odot}\rightarrow\mathcal{S}^{\diamond}$ the state
abstraction function. We define $\Xi^{-1}$ as the inverse of the abstraction
function such that
$\Xi^{-1}(s^{\odot})=\\{s^{\diamond}\in\mathcal{S}^{\diamond}|\Xi(s^{\diamond})=s^{\odot}\\}$.
We assume a weighting function $w\colon\mathcal{S}\mapsto[0,1]$, where

$\sum_{s^{\odot}\in\Xi^{-1}(s^{\diamond})}w(s^{\odot})=1,\forall
s^{\diamond}\in\mathcal{S}^{\diamond}.$ (14)

Now we can define the transition and cost function of the target task:

$\displaystyle\mathcal{P}^{{\diamond}}(s^{{\diamond}^{\prime}}\mid
s^{\diamond},a)~{}$
$\displaystyle=\sum_{s^{\odot}\in\Xi^{-1}(s^{\diamond})}\quad\sum_{s^{{\odot}^{\prime}}\in\Xi^{-1}(s^{{\diamond}^{\prime}})}w(s^{\odot})\mathcal{P}^{{\odot}}(s^{{\odot}^{\prime}}\mid
s^{\odot},a)$ (15) $\displaystyle c^{\diamond}(s^{\diamond},a)~{}$
$\displaystyle=\sum_{s^{\odot}\in\Xi^{-1}(s^{\diamond})}w(s^{\odot})c^{\odot}(s^{\odot},a)$
(16) $\displaystyle\iota^{\diamond}(s^{\diamond})~{}$
$\displaystyle=\sum_{s^{\odot}\in\Xi^{-1}(s^{\diamond})}w(s^{\odot})\iota^{\odot}(s^{\odot}).$
(17)

###  B.2 Proof of Lemma 1

In this section, we show that if $\Xi$ is a $Q_{\pi}^{c}$-irrelevance state
abstraction, then the expected cost of any source policy is the same in the
source task and in the target task. For the convenience of the reader, we
restate our assumption and lemma.

#### Assumption 3.

$\Xi$ is a $Q_{\pi}^{c}$-irrelevance abstraction [31], therefore

$\Xi(s)=\Xi(s^{\prime})\Rightarrow
Q_{\pi^{\odot}}^{c}(s,a)=Q_{\pi^{\odot}}^{c}(s^{\prime},a),\forall
s,s^{\prime}\in\mathcal{S}^{\odot},a\in\mathcal{A},\pi^{\odot}.$

#### Lemma 1.

Given 1 and 3, we have

$Q_{\pi^{\diamond}}^{c,{\diamond}}(\Xi(s),a)=Q_{\pi^{{\diamond}\rightarrow{\odot}}}^{c,{\odot}}(s,a)\quad\forall
s\in\mathcal{S}^{\odot},a\in\mathcal{A},\pi^{\diamond}.$

That is, the expected cost of a source policy is the same in the source task
and in the target task.  Our proof follows an induction strategy inspired by
previous work [1, Claim 1].

######  Proof B.1.

Let us consider a non-Markovian constrained decision process
$\mathcal{M}_{T}=\langle\mathcal{S}_{T},\mathcal{A},\mathcal{P}_{T},\emptyset,c^{T},d^{\diamond},\iota_{T},\gamma\rangle$
which is parameterized by an integer $T$. In this process, the agent takes $T$
steps on the source task and then switches to the target task. Thus,

$\displaystyle\mathcal{S}_{T}~{}$
$\displaystyle=\begin{cases}\mathcal{S}^{\odot}&\text{ if }T=0\\\
\mathcal{S}^{\diamond}&\text{ otherwise.}\end{cases}$ (18) $\displaystyle
c_{T}(s,a)~{}$ $\displaystyle=\begin{cases}c^{\odot}(s,a)&\text{ if }T=0\\\
c^{\diamond}(s,a)&\text{ otherwise.}\end{cases}$ (19)
$\displaystyle\mathcal{P}_{T}(s^{\prime}\mid s,a)~{}$
$\displaystyle=\begin{cases}\mathcal{P}^{{\odot}}(s^{\prime}\mid s,a)&\text{
if }T=0\\\
\sum_{s^{\odot}\in\Xi^{-1}(s)}w(s^{\odot})\mathcal{P}^{{\odot}}(s^{\prime}\mid
s^{\odot},a)&\text{ if }T=1\\\ \mathcal{P}^{{\diamond}}(s^{\prime}\mid
s,a)&\text{ otherwise. }\end{cases}$ (20) $\displaystyle\iota_{T}(s)~{}$
$\displaystyle=\begin{cases}\iota^{\odot}(s)&\text{ if }T=0\\\
\iota^{\diamond}(s)&\text{ otherwise.}\end{cases}$ (21)

The $Q_{\pi}^{c,{\odot}}(s,a)$-value for taking action $a\in\mathcal{A}$ in
state $s\in\mathcal{S}_{T}$ and follow the policy $\pi$ is:

$\displaystyle Q_{T,\pi}^{c}(s,a)=\begin{cases}Q_{\pi}^{c,{\odot}}(s,a)&\text{
if }T=0\\\
\sum_{s^{\odot}\in\Xi^{-1}(s)}w(s^{\odot})Q_{\pi}^{c,{\odot}}(s^{\odot},a)&\text{
if }T=1\\\
c^{\diamond}(s,a)+\gamma\sum_{s^{\prime}\in\mathcal{S}^{\diamond}}\mathcal{P}^{\diamond}(s^{\prime}\mid
s,a)\sum_{a^{\prime}\in\mathcal{A}}\pi(a^{\prime}\mid
s^{\prime})Q_{T-1,\pi}^{c}(s^{\prime},a^{\prime})&\text{
otherwise.}\end{cases}$ (22)

We proceed by induction on $T$ to show that

$\forall
T,s^{\odot},a,\pi:Q_{\pi}^{c,{\odot}}(s_{T},a)=Q_{\pi}^{c,{\odot}}(s^{\odot},a),$

where $s_{T}=s^{\odot}$ if $T=0$ and $s_{T}=\Xi(s^{\odot})$ otherwise.

#### Base case: $T=0$.

As $Q_{0}^{c}=Q^{c,{\odot}}$ this case follows trivially.

#### Base case: $T=1$.

From the definition of $Q_{1,\pi}^{c}$, we have:

$\displaystyle
Q_{1,\pi}^{c}(s_{T},a)=\sum_{s^{{\odot}^{\prime}}\in\Xi^{-1}(s_{T})}w(s^{{\odot}^{\prime}})Q_{\pi}^{c,{\odot}}(s^{{\odot}^{\prime}},a)$
(23)
$\displaystyle=\sum_{s^{{\odot}^{\prime}}\in\Xi^{-1}(s_{T})}w(s^{{\odot}^{\prime}})Q_{\pi}^{c,{\odot}}(s^{\odot},a)$
(24)
$\displaystyle=Q_{\pi}^{c,{\odot}}(s^{\odot},a)\sum_{s^{{\odot}^{\prime}}\in\Xi^{-1}(s)}w(s^{{\odot}^{\prime}})$
(25) $\displaystyle=Q_{\pi}^{c,{\odot}}(s^{\odot},a).$ (26)

In Equation 24, we replace every $s^{{\odot}^{\prime}}$ by the state
$s^{\odot}$ applying Assumption 3. As $s^{\odot}$ is independent of
$s^{{\odot}^{\prime}}$, in Equation 25, we can move the Q-values out of the
summation. Finally, in Equation 26, we can use Equation 14 to replace the
summation by 1, which concludes this case.

#### Incuctive case: $T>1$.

We assume as our inductive hypothesis that:

$\forall
s^{\odot},a,\pi:Q_{T-1,\pi}^{c}(s_{T},a)=Q_{\pi}^{c,{\odot}}(s^{\odot},a).$

We start applying the definition of $Q_{T}$ for $T>1$:

$\displaystyle Q_{T,\pi}^{c}$
$\displaystyle(s_{T},a)=c^{\diamond}(s_{T},a)+\gamma\sum_{s^{\prime}\in\mathcal{S}^{\diamond}}\mathcal{P}^{\diamond}(s^{\prime}\mid
s_{T},a)\sum_{a^{\prime}\in\mathcal{A}}\pi(a^{\prime}\mid
s^{\prime})Q_{T-1,\pi}^{c}(s^{\prime},a^{\prime})$ (27)
$\displaystyle=\sum_{s^{\odot}\in\Xi^{-1}(s_{T})}w(s^{\odot})c^{\odot}(s^{\odot},a)+\gamma\sum_{s^{\prime}\in\mathcal{S}^{\diamond}}\sum_{s^{\odot}\in\Xi^{-1}(s_{T})}\sum_{s^{{\odot}^{\prime}}\in\Xi^{-1}(s^{\prime})}w(s^{\odot})\mathcal{P}^{{\odot}}(s^{{\odot}^{\prime}}\mid
s^{\odot},a)\sum_{a^{\prime}\in\mathcal{A}}\pi(a^{\prime}\mid
s^{\prime})Q_{T-1,\pi}^{c}(s^{\prime},a^{\prime})$ (28)
$\displaystyle=\sum_{s^{\odot}\in\Xi^{-1}(s_{T})}w(s^{\odot})c^{\odot}(s^{\odot},a)+\sum_{s^{\odot}\in\Xi^{-1}(s_{T})}w(s^{\odot})\gamma\sum_{s^{\prime}\in\mathcal{S}^{\diamond}}\sum_{s^{{\odot}^{\prime}}\in\Xi^{-1}(s^{\prime})}\mathcal{P}^{{\odot}}(s^{{\odot}^{\prime}}\mid
s^{\odot},a)\sum_{a^{\prime}\in\mathcal{A}}\pi(a^{\prime}\mid
s^{\prime})Q_{T-1,\pi}^{c}(s^{\prime},a^{\prime})$ (29)
$\displaystyle=\sum_{s^{\odot}\in\Xi^{-1}(s_{T})}w(s^{\odot})\left[c^{\odot}(s^{\odot},a)+\gamma\sum_{s^{\prime}\in\mathcal{S}^{\diamond}}\sum_{s^{{\odot}^{\prime}}\in\Xi^{-1}(s^{\prime})}\mathcal{P}^{{\odot}}(s^{{\odot}^{\prime}}\mid
s^{\odot},a)\sum_{a^{\prime}\in\mathcal{A}}\pi(a^{\prime}\mid
s^{\prime})Q_{T-1,\pi}^{c}(s^{\prime},a^{\prime})\right]$ (30)
$\displaystyle=\sum_{s^{\odot}\in\Xi^{-1}(s_{T})}w(s^{\odot})\left[c^{\odot}(s^{\odot},a)+\gamma\sum_{s^{\prime}\in\mathcal{S}^{\diamond}}\sum_{s^{{\odot}^{\prime}}\in\Xi^{-1}(s^{\prime})}\mathcal{P}^{{\odot}}(s^{{\odot}^{\prime}}\mid
s^{\odot},a)\sum_{a^{\prime}\in\mathcal{A}}\pi(a^{\prime}\mid
s^{\prime})Q_{\pi}^{c,{\odot}}(s^{{\odot}^{\prime}},a^{\prime})\right]$ (31)
$\displaystyle=\sum_{s^{\odot}\in\Xi^{-1}(s_{T})}w(s^{\odot})\left[c^{\odot}(s^{\odot},a)+\gamma\sum_{s^{{\odot}^{\prime}}\in\mathcal{S}^{\odot}}\mathcal{P}^{{\odot}}(s^{{\odot}^{\prime}}\mid
s^{\odot},a)\sum_{a^{\prime}\in\mathcal{A}}\pi(a^{\prime}\mid
s^{\prime})Q_{\pi}^{c,{\odot}}(s^{{\odot}^{\prime}},a^{\prime})\right]$ (32)
$\displaystyle=\sum_{s^{\odot}\in\Xi^{-1}(s_{T})}w(s^{\odot})Q_{\pi}^{c,{\odot}}(s^{\odot},a)$
(33) $\displaystyle=Q_{\pi}^{c,{\odot}}(s^{\odot},a).$ (34)

In this derivation, Equation 28 applies the definitions of $c^{\diamond}$ and
$\mathcal{P}^{\diamond}$, Equations 29 and 30 rearrange our terms, Equation 31
applies our inductive hypothesis, Equation 32 join the two summations as we
are considering all possible states in $\mathcal{S}^{\odot}$, and Equation 33
we apply the Q-value definition. Finally, in Equation 34 we can choose any
arbitrary state $s^{\odot}\in\Xi^{-1}(s_{T})$, which concludes our proof.

##  Appendix C Regularized Reward

$\displaystyle\omega r^{\textrm{KL}}{+}\alpha r^{\mathcal{H}}=$
$\displaystyle\omega\log\frac{\pi^{\diamond}(a_{t}\mid\Xi(s_{t}))}{\pi^{\odot}(a_{t}\mid
s_{t})}+\omega r^{\mathcal{H}}$ (35) $\displaystyle=$
$\displaystyle\omega(\log(\pi^{\diamond}(a|\Xi(s)))-\log(\pi^{\odot}(a|s)))+\alpha
r^{\mathcal{H}}$ $\displaystyle=$
$\displaystyle\omega\log(\pi^{\diamond}(a|\Xi(s)))+\omega(-\log(\pi^{\odot}(a|s)))+\alpha
r^{\mathcal{H}}$ $\displaystyle=$
$\displaystyle\omega\log(\pi^{\diamond}(a|\Xi(s)))+\omega
r^{\mathcal{H}}+\alpha r^{\mathcal{H}}$ $\displaystyle=$ $\displaystyle\omega
r^{\diamond}+(\omega+\alpha)r^{\mathcal{H}}.$

##  Appendix D Two strategies in composite sampling

#### Linear-decay (Algorithm 3).

This strategy linearly decreases the probability of using $\pi^{\diamond}$
with a constant decay rate after each iteration of the algorithm, conversely
increasing the probability of using $\pi^{\odot}$. We have two modes with
linear-decay: _step-wise_ , where in each time step we may change ${\pi_{b}}$;
and _trajectory-wise_ , where ${\pi_{b}}$ only changes at the start of a
trajectory. The mode is decided before executing an episode and smoothly
switches from the complete _step-wise_ to the complete _trajectory-wise_ over
the training process. We linearly decrease the probability of executing the
_step-wise_ and use the guide with a constant decay rate after each iteration
of the algorithm, conversely increasing the probability of executing the
_trajectory-wise_ and using the student policy. So, we initialize the
probabilities $P_{\pi}=1$ to determine ${\pi_{b}}$, and $P_{wise}=1$ to
determine the mode at the beginning (line 3). We linearly decrease them with a
constant decay rate $\upsilon$ (lines 11 and 17), determined by the training
length. At the beginning of each episode, we sample $\kappa_{wise}\sim
U(0,1)$, so if $\kappa_{wise}<P_{wise}$, we will execute _step-wise_ , or we
are in _trajectory-wise_ (lines 4-10). Under _step-wise_ , at each time step,
we sample from the guide $\pi^{\diamond}$ with probability $P_{\pi}$, and
sample from the student $\pi^{\odot}$ with probability $1-P_{\pi}$ (lines
13-14). Under _trajectory-wise_ , we only make a decision once at the
beginning of the trajectory (line 9).

#### Control-switch (Algorithm 4).

To balance between the safe exploration and the sample efficiency (the samples
from the target policy are relatively more valuable), the student policy keeps
sampling, i.e., ${\pi_{b}}=\pi^{\odot}$ at the start of a trajectory (line 2);
after we meet the first $c_{t-1}>0$, we have ${\pi_{b}}=\pi^{\diamond}$ until
the end of the trajectory (lines 12-14). Therefore, the guide policy serves as
a rescue policy to improve safety during sampling. In addition, we leverage
two replay buffers $\mathcal{D}^{\diamond}$ and $\mathcal{D}^{{\odot}}$ to
save the guide and student samples separately (lines 7-11), so as to control
the probability $P_{\mathcal{D}^{\odot}}$ to use the more on-policy samples in
$\mathcal{D}^{{\odot}}$. Thus, we have the probability
$P_{\mathcal{D}^{\diamond}}=1-P_{\mathcal{D}^{\odot}}$ to sample from
$\mathcal{D}^{\diamond}$. In practice, we train the safe guide to achieve
$Q_{\pi^{\diamond}}^{c}(s,a)\leq
d,s\sim\mathcal{D},a\sim\pi^{\diamond}(\cdot\mid s)$. From the definition of
$Q_{\pi^{\diamond}}^{c}(s,a)$, we can basically ensure
$\operatorname{\mathbb{E}}_{\tau\sim\rho_{\pi^{\diamond}}}\left[\sum^{\infty}_{t=0}\gamma^{t}c_{t}\middle|s_{0}=s,a_{0}=a\right]\leq
d$ even starting with $c_{0}>0$.

####  Main difference.

The key distinction between linear-decay and control-switch approaches lies in
the number of off-policy interactions from the student’s perspective. Linear-
decay entails the collection of more samples from the guide during early
episodes, whereas control-switch enables the agent to collect more on-policy
samples and only occasionally relies on off-policy samples from the guide
following unsafe interactions. Additionally, linear-decay necessitates
predefined schedules for the behaviour policy, while control-switch is
adaptive. The pursuit of novel adaptive schedules presents a promising avenue
for future research.

Algorithm 3 Composite sampling (linear-decay)

Input: $\pi^{\diamond}$, $\pi^{\odot}$, $\upsilon$
Initialize: $P_{\pi}\leftarrow 1$, $P_{\textrm{wise}}\leftarrow 1$
Output: ${\pi_{b}}$

1: for each iteration do

2: $P_{b}({\diamond})=P_{\pi}$ $\triangleright$ The probability of using
$\pi^{\diamond}$

3: $P_{b}({\odot})=1-P_{\pi}$ $\triangleright$ The probability of using
$\pi^{\odot}$

4: Sample $\kappa_{\textrm{wise}}\sim U(0,1)$

5: if $\kappa_{\textrm{wise}}<P_{\textrm{wise}}$ then

6:  _step-wise_ $\leftarrow\mathit{true}$

7: else

8:  _step-wise_ $\leftarrow\mathit{false}$

9: $b\sim P_{b}$ $\triangleright$ Choose behaviour policy

10: end if

11: $P_{\textrm{wise}}=P_{\textrm{wise}}-\upsilon$ $\triangleright$ Decrease
the probability of step-wise

12: for each environment step do

13: if  _step-wise_ then

14: $b\sim P_{b}$ $\triangleright$ Choose behaviour policy

15: end if

16: end for

17: $P_{\pi}=P_{\pi}-\upsilon$ $\triangleright$ Decrease the probability of
using $\pi^{\diamond}$

18: end for

Algorithm 4 Composite sampling (control-switch)

Input: $\pi^{\diamond}$, $\pi^{\odot}$
Initialize: $\mathcal{D}^{\diamond}\leftarrow\emptyset$,
$\mathcal{D}^{\odot}\leftarrow\emptyset$
Output: ${\pi_{b}}$

1: for each iteration do

2: $b\leftarrow{\odot}$ $\triangleright$ Start sampling from the student

3:  _control-switch_ $(t)$ $\leftarrow\mathit{false}$

4: for each environment step do

5: $a_{t}\sim{\pi_{b}}(\cdot\mid s_{t})$

6:
$E\leftarrow(s_{t},a_{t},r^{\odot}_{t},r^{\diamond}_{t},c_{t},\mathcal{I}_{t},s_{t+1})$
$\triangleright$ Generate experience

7: if $b={\diamond}$ then

8: $\mathcal{D}^{\diamond}\leftarrow\mathcal{D}^{\diamond}\cup\\{E\\}$
$\triangleright$ Save the guide samples

9: else

10: $\mathcal{D}^{\odot}\leftarrow\mathcal{D}^{\odot}\cup\\{E\\}$
$\triangleright$ Save the student samples

11: end if

12: if $\neg$ _control-switch_ $(t)$ $\wedge$ $c_{t}>0$ then

13: $b\leftarrow{\diamond}$ $\triangleright$ Switch behaviour policy

14:  _control-switch_ $(t)$ $\leftarrow\mathit{true}$

15: end if

16: end for

17: end for

##  Appendix E Ablation Study

(a) Behaviour policy

(b) Target policy

Figure 7: Ablation study in Static showing the safety and performance of the
behaviour policy (a) and target policy (b). The black dashed line indicates
the safety threshold.

##  Appendix F Evaluation of the target policy

#### Comparison with baselines

In Figure 6, we evaluate the behaviour policy ${\pi_{b}}$ for all algorithms:
CPO, SAC-$\lambda$, CPO-pre, SAC-$\lambda$-pre, EGPO, and SaGui. So, in Figure
8, we show how their resulted target policy will perform during training. In
all these algorithms, SaGui (control-switch) is the only one that can find a
safe optimal target policy in all environments. However, SaGui (linear-decay)
cannot achieve similar performance, especially in Semi-dynamic and Dynamic. We
infer that SaGui (linear-decay) lack samples from the target policy,
especially at the early stage of training. The behaviour policy of EGPO (with
benefits from the targeted expert policy) has outstanding performance during
training (Figure 6), but EGPO fails to find a safe target policy finally. As
to the pre-training baselines, CPO-pre and SAC-$\lambda$-pre do not attain
obvious improvement compared to CPO and SAC-$\lambda$ that are trained from
scratch. Instead, pre-training may have some negative impacts on getting a
good target policy. The only exception is that CPO-pre is largely improved in
the relatively simple environment Static.

(a) Static

(b) Semi-Dynamic

(c) Dynamic

Figure 8: Evaluation of $\pi^{\odot}$ for CPO, CPO-pre, SAC-$\lambda$,
SAC-$\lambda$-pre, EGPO, SaGui (linear-decay), and SaGui (control-switch) over
ten seeds. The solid lines are the average of all runs, and the shaded area is
the standard deviation. The black dashed lines indicate the safety thresholds.

##  Appendix G Hyperparameters

We list the hyperparameters used in SaGui, which are summarized in Table 1. As
to the baselines, we use the default hyperparameters in
https://github.com/openai/safety-starter-agents. All runs in the experiment
use separate feedforward Multilayer Perceptron (MLP) actor and critic
networks. The size of the neural network (all actors and critics of the
algorithms) depend on the complexity of the tasks. We use a replay buffer of
size $10^{6}$ for each off-policy algorithm to store the experience. The
discount factor is set to be $\gamma=0.99$, the target smoothing coefficient
is set to be $0.005$ to update the target networks, and the learning rate to
$0.001$. The clipping intervale hyper-parameters
$[\mathcal{I}_{l},\mathcal{I}_{u}]$ is set to $[0.1,2.0]$, while the sampling
probabilities $P_{\mathcal{D}^{\diamond}}$ and $P_{\mathcal{D}^{\odot}}$ are
set to $0.25$ and $0.75$, respectively. The maximum episode length is 1000
steps in all experiments. We set the safety constraint $d$ based on the
problem. The rest of the hyperparameters are explained in the Empirical
Analysis part of the paper. All experiments are performed on an Intel(R)
Xeon(R<EMAIL_ADDRESS>with 16 GB of RAM.

Parameter | Static | Semi-Dynamic | Dynamic | Note
---|---|---|---|---
Size of networks | $(32,32)$ | $(64,64)$ | $(256,256)$ |
Size of replay buffer | $10^{6}$ | $10^{6}$ | $10^{6}$ | $|\mathcal{D}|$
Batch size | 32 | 64 | 256 |
Number of epochs | $50$ | $100$ | $150$ |
Safety constraint | $5$ | $8$ | $25$ | $d$
Table 1: Summary of hyperparameters in SaGui.

#### Safety-mapping function.

The state spaces of the source and target task differ by the presence of the
LiDAR observation of the target location. While the source task only has a
safety-related signal $x_{c}$, the target task has an additional goal-related
signal $x_{r}$. Thus, following the definition in Section 4.2, we can map the
target state $[x_{c},x_{r}]$ to the source state ignoring the target-related
signal: $\Xi([x_{c},x_{r}])=[x_{c}]$.

##  Appendix H Expert Guided Policy Optimization

We also compare our algorithms to an Expert-in-the-loop RL method called
Expert Guided Policy Optimization (EGPO) that incorporates a well-performing
expert policy as a demonstrator as well as a safety guardian [36]. However,
EGPO constrains safety behaviours at each timestep, which is different from
our safety defined on long-term cost-return. In terms of the safe guide, EGPO
assumes the access to the well-performing expert policy, but our safe guide is
task-agnostic. Thus, the expert in EGPO depends on the target task and does
not undertake the task of exploration, while our safe guide can be useful for
different reward functions and enhance the exploration capabilities of the
student. Even though, EGPO can be easily adapted to our setting. The
constraint of EGPO on the guardian intervention frequency can be directly
transferred to be our safety constraint. Also, we do not minimize intervention
anymore. Once the EGPO agent starts to take unsafe actions, the expert policy
can take over the control until the end.