# FOSP: Fine-tuning Offline Safe Policy through World Models

Chenyang Cao, Yucheng Xin, Silang Wu, Longxiang He, Zichen Yan, Junbo Tan,
Xueqian Wang

Shenzhen International Graduate School
Tsinghua University

###### Abstract

Model-based Reinforcement Learning (RL) has shown its high training efficiency
and capability of handling high-dimensional tasks. Regarding safety issues,
safe model-based RL can achieve nearly zero-cost performance and effectively
manage the trade-off between performance and safety. Nevertheless, prior works
still pose safety challenges due to the online exploration in real-world
deployment. To address this, some offline RL methods have emerged as
solutions, which learn from a static dataset in a safe way by avoiding
interactions with the environment. In this paper, we aim to further enhance
safety during the deployment stage for vision-based robotic tasks by fine-
tuning an offline-trained policy. We incorporate in-sample optimization,
model-based policy expansion, and reachability guidance to construct a safe
offline-to-online framework. Moreover, our method proves to improve the
generalization of offline policy in unseen safety-constrained scenarios.
Finally, the efficiency of our method is validated on simulation benchmarks
with five vision-only tasks and a real robot by solving some deployment
problems using limited data.

> Keywords: Safe Model-based Reinforcement Learning, Offline-to-Online

##  1 Introduction

Model-based Reinforcement Learning (MBRL) has proven to be a powerful tool to
enhance sample efficiency by constructing a world model of the environment[1,
2, 3, 4, 5]. The world model reduces random explorations by predicting the
outcomes of different actions, leading to a fast convergence speed during
online fine-tuning after minimal interactions [6]. Moreover, it significantly
improves action safety in high-dimensional tasks such as vision-based
operations due to its prediction ability[7, 8]. Consequently, We consider
utilizing model-based methods to tackle safety-critical problems in real-world
deployment.

Offline RL has been widely studied within the academic community, using pre-
collected datasets to learn policies without exploration[9]. At the offline
training stage, safety constraints will be incorporated into policy learning,
which means learning to be safe other than learning safely. In this way,
offline trained policy is promising to avoid constraint violations during the
online deployment. However, offline dataset cannot fully simulate complicated
real-world environments and will cause out-of-distribution (OOD) issues in the
case of unseen data. Such generalization tasks are significant challenges for
offline learning.

In this work, we propose developing an offline-to-online approach to address
the generalization issue with safety guarantees. We consider the problem of
pretraining an offline policy and a world model, and then fine-tuning them
safely during online interactions. However, such a strategy usually causes
degradation of policy performance and a lack of safety after fine-tuning[10].
The main reason lies in too many mixed constraints (both soft and hard), such
as safety constraints and behavior regularization, making it difficult to
ensure both optimality and safety. Meanwhile, the world model will introduce
prediction errors due to distribution shift during the fine-tuning phase,
leading to increasing safety risks. Therefore, we seek to unify and handle
mixed constraints, and use the offline safe policy as a guidance for online
correction. Based on this key insight, we proposed FOSP (Fine-tuning Offline
Safe Policy through World Models), a method that enhances offline safe
training and bridges model-based offline training with online fine-tuning.

To evaluate our method, we design different experiments in the simulation
including offline training and online fine-tuning. It shows robust performance
in both offline and online fine-tuning tasks in Safety-Gymnasium[11] and
achieves nearly zero-cost. We also deploy FOSP in a real-world Franka
manipulator for trajectory planning tasks. It turns out that our method
performs well across tasks with different safety regions and can be
transferred to unseen scenarios through few-shot fine-tuning.

We summarize our contributions as follows: (1) To the best of our knowledge,
FOSP is the first approach that extends model-based RL to safe offline-to-
online areas. (2) It handles the trade-off between the performance and
constraints satisfaction across offline and online training phases on various
vision-only tasks. (3) In real-world deployment, it does not need sim-to-real
transfer and can be safely fine-tuned in unseen safety-constrained scenarios.
(4) To inspire further research in this area, we will publicly release our
code and dataset.

Figure 1: Fine-tuning offline safe policy through world models. We propose an
approach to offline pretrain and online fine-tuning the world model. We also
consider the safe constraints and generalize it to unseen safe regions.

##  2 Related Works

#### Offline-to-online RL

Offline RL, which trains a policy by leveraging a large amount of existing
data, is valuable to real-world scenarios[12, 13]. Prior offline RL works
focus on addressing the distribution shift and value overestimation
problem[14, 15, 16, 17, 18, 19]. Recently, in-sample methods have demonstrated
their robust performance by avoiding out-of-distribution (OOD) state-action
pairs[20, 21, 22, 23, 10, 24]. Unlike training solely on offline data,
offline-to-online RL involves further fine-tuning the offline policy online.
And many offline RL algorithms have extended to online fine-tuning[20, 10].
But these methods have shown that the more effective the offline RL method is,
the worse online fine-tuning performance is[25]. Some previous works have
explored how to effectively fine-tune from offline pretrained policy,
including balanced sampling in replay buffer[26], policy calibrating[27, 28],
and parameter transferring[29, 30]. However, existing solutions are restricted
by policy transitions or the need to estimate the density of the online policy
distribution[31]. PEX[32] is an approach that bridges offline and online
algorithms and is adaptive for fine-tuning different online methods with
offline policy guidance. As it is shown as an effective approach in offline-
to-online training, we extend it to model-based algorithms that not only
require fine-tuning the policy but also the model.

#### Safe RL

Safe RL focuses on optimizing objectives with different constraints[33]. Many
methods are based on policy search algorithms that ensure adherence to near-
constraints iteratively[34] and Lagrangian methods that convert it to an
unconstrained problem to solve[35, 36, 37]. Model-based RL methods utilize
world models for action selection so that they ensure safety by background
planning. Some recent works are trying to address challenges remaining in
vision-input tasks[38, 39, 3]. LAMDBA[7] is a method extending DreamerV1[1] to
safe RL by using augmented Lagrangian. Safe-SLAC[40] incorporating the
Lagrangian method into SLAC[41] improves the computation complexity of LAMBDA.
SafeDreamer[8] is the latest work in safe model-based reinforcement learning.
It enhances safety in online or background planning with DreamerV3[5],
achieving state-of-the-art (SOTA) performance. However, prior works only aim
to address issues in the online setting, which cannot avoid costs incurring
during exploration in the real world. By contrast, some safe offline RL can
learn policies from offline datasets by using OOD action detection[42, 43,
44], DICE-based theory[45], and sequence modeling[46, 47]. Nonetheless, they
only utilize soft constraints which sometimes can not effectively decrease
constraint violations. Some works consider the hard constraints by using
control theory methods such as control barrier function (CBF)[48] and HJ
reachability[49, 50, 51, 52]. Our work draws from these works but unifies all
the constraints with model-based background planning.

##  3 Preliminaries

#### World Model

Our work extends DreamerV3[5], a powerful and robust baseline model-based RL
using world model and actor-critic. It has the ability to model high-
dimensional observations of the environment and increase the data efficiency
of RL. The world model learns compressed image input representations and
predicts future states and rewards sequences for decision-making. It is
implemented as a Recurrent State-Space Model (RSSM)[3]. First, the model maps
the high-dimensional observations $x_{t}$ to stochastic states $z_{t}$. And
deterministic latent states $h_{t}$ are predicted by $z_{t}$ and actions
$a_{t}$. By concatenating both of them, we get the model state
$s_{t}=\\{h_{t},z_{t}\\}$. Finally, we facilitate $s_{t}$ to predict the next
observations and rewards. Due to the safety considered in the environment, we
add a cost decoder to the original model. All components in the world model
are as follows:

$\begin{split}\begin{array}[]{ll}\text{latent encoder:}&{\bm{z}}_{t}\sim
q_{\theta}({\bm{z}}_{t}\mid{\bm{h}}_{t},{\bm{x}}_{t})\\\ \text{deterministic
state:}&{\bm{h}}_{t}=f_{\theta}({\bm{z}}_{t-1},{\bm{h}}_{t-1},{\bm{a}}_{t-1})\\\
\text{stochastic state:}&\hat{{\bm{z}}}_{t}\sim
p_{\theta}({\bm{z}}_{t}\mid{\bm{h}}_{t})\\\
\end{array}\end{split}\;\begin{split}\begin{array}[]{ll}\text{observation
decoder:}&\hat{{\bm{x}}}_{t}\sim
p_{\theta}({\bm{x}}_{t}\mid{\bm{z}}_{t},{\bm{h}}_{t})\\\ \text{reward
decoder:}&\hat{{\bm{r}}}_{t}\sim
p_{\theta}({\bm{r}}_{t}\mid{\bm{z}}_{t},{\bm{h}}_{t})\\\ \text{cost
decoder:}&\hat{{\bm{c}}}_{t}\sim
p_{\theta}({\bm{c}}_{t}\mid{\bm{z}}_{t},{\bm{h}}_{t})\\\
\end{array}\end{split}$

The model is trained end-to-end by the ELBO or variational free energy of a
hidden Markov model loss function with KL balancing:

$\displaystyle\mathcal{L}^{\text{model}}(\theta)=\operatorname*{\mathbb{E}}_{\tau\sim\mathcal{D}}\Big{[}\sum_{t=1}^{T}-\ln
p_{\theta}({\bm{x}}_{t}\mid{\bm{s}}_{t})-\ln
p_{\theta}({\bm{r}}_{t}\mid{\bm{s}}_{t})-\ln
p_{\theta}({\bm{c}}_{t}\mid{\bm{s}}_{t})+$
$\displaystyle\mathbb{D}_{KL}[sg(q_{\theta}({\bm{z}}_{t}|{\bm{h}}_{t},{\bm{x}}_{t}))||p_{\theta}({\bm{z}}_{t}|{\bm{h}}_{t})]+\beta\mathbb{D}_{KL}[q_{\theta}({\bm{z}}_{t}|{\bm{h}}_{t},{\bm{x}}_{t})||sg(p_{\theta}({\bm{z}}_{t}|{\bm{h}}_{t}))]\Big{]}.$
(1)

The stop-gradient operator makes the representations more predictable in the
imagination training.

#### Safe Model-based RL

Safe RL is frequently formulated as a Constrained Markov Decision Process
(CMDP)
$\mathcal{M}=(\mathcal{S},\mathcal{A},\mathbb{P},\mathcal{R},\mathcal{C},\mu,\gamma)$[33].
$\mathcal{S}$ and $\mathcal{A}$ denote the state and action space.
$\mathbb{P}(s^{\prime}|s,a)$ represents the transition probability from
$s^{\prime}$ to $s$ under action $a$. $\mathcal{R}$ is reward space that maps
$\mathcal{S}\times\mathcal{A}$ to $\mathbb{R}$. And $\mathcal{C}$ is a cost
function set containing cost functions
$c:\mathcal{S}\times\mathcal{A}\rightarrow[0,C_{max}]$.
$\mu(\cdot):\mathcal{S}\rightarrow[0,1]$ is the initial state distribution and
$\gamma\in(0,1)$ is the discount factor. Given a policy distribution
$\pi_{\psi}$, we define the performance and constraints as follows:

$J^{\mathcal{R}}(\pi_{\psi})=\operatorname*{\mathbb{E}}_{{\bm{a}}_{t}\sim\pi,{\bm{s}}_{t+1}\sim
p_{\theta},{\bm{s}}_{0}\sim\rho}\left[\sum_{t=0}^{T}r_{t}\big{\rvert}{\bm{s}}_{0}\right],$
(2)

$J^{\mathcal{C}}_{i}(\pi_{\psi})=\operatorname*{\mathbb{E}}_{{\bm{a}}_{t}\sim\pi,{\bm{s}}_{t+1}\sim
p_{\theta},{\bm{s}}_{0}\sim\rho}\left[\sum_{t=0}^{T}c_{t}^{i}\big{\rvert}{\bm{s}}_{0}\right]\leq
d^{i},\;\forall i\in\\{1,\dots,C\\},$ (3)

where $i$ denotes different safety constraints we want to avoid and $d^{i}$
are cost thresholds. Therefore, our safe model-based RL problem can be
formulated as follows:

$\max_{\psi}J^{\mathcal{R}}(\pi_{\psi})\;\;\;\text{s.t.
}\;J^{\mathcal{C}}_{i}(\pi_{\psi})\leq d^{i},\;\forall i\in\\{1,\dots,C\\}.$
(4)

##  4 Methods

In this section, we introduce our framework (Figure 1) for fine-tuning safe
actions with offline pretrained world models. It alleviates the distribution
shift error during offline training and balances the trade-off between
performance and violations via the reachability estimation function. In the
online fine-tuning phase, using offline-trained policy as guidance helps
mitigate the performance drop and further improves the performance of the
algorithm. In section 4.1, to address the distribution shift in offline
training, we use the in-sample optimization approach to train critics. Next,
we introduce an offline-to-online approach called model-based policy expansion
to further improve online performance in section 4.2. The reachability
guidance for safety consideration is provided in section 4.3.

###  4.1 In-sample Optimization for Offline Training

Value overestimation is an extensive challenge in offline training due to the
distribution shift between the state-action pair from the dataset and
generated by the learned policy. Moreover, model-based offline RL algorithms
not only learn a policy but also a dynamic model $p_{\theta}$ by the offline
dataset. It makes this challenge more severe that the latent dynamic
estimation and return prediction are also affected. Therefore, we introduce
the in-sample optimization to address this issue.

First, we consider a sequence of data
$\mathcal{B}=({\bm{x}}_{1:T},{\bm{a}}_{1:T},{\bm{r}}_{1:T},{\bm{c}}_{1:T})$
sampled from the offline dataset. Then, the model generates latent states
${\bm{s}}_{1:T}\sim q_{\theta}({\bm{s}}_{1:T}|{\bm{h}}_{1:T},{\bm{x}}_{1:T})$
and uses them as initial states to generate rollouts:

$\hat{{\bm{a}}}_{1:T}^{h}\sim\pi_{\psi}({\bm{a}}|\hat{{\bm{s}}}_{1:T}^{h-1}),\
\hat{{\bm{s}}}_{1:T}^{h+1}\sim
p_{\theta}({\bm{s}}|\hat{{\bm{a}}}_{1:T}^{h},\hat{{\bm{s}}}_{1:T}^{h}),\
\hat{{\bm{r}}}_{1:T}^{h}\sim p_{\theta}({\bm{r}}|\hat{{\bm{s}}}_{1:T}^{h}),\
\hat{{\bm{c}}}_{1:T}^{h}\sim p_{\theta}({\bm{c}}|\hat{{\bm{s}}}_{1:T}^{h}).$
(5)

We use $Q_{\phi}({\bm{s}},{\bm{a}})$ as the critic network and
$\hat{Q}_{\phi}(s,a)$ as the target network. Similar to Implicit Q-learning
(IQL)[24], we only use in-sample actions as calculated components of
$\lambda$-return. Specifically, a large target value estimated by $Q_{\phi}$
might not represent an existing action leading to a good state due to the out-
of-sample actions $\pi({\bm{s}})$. Thus, a separate value network
$V_{\varphi}({\bm{s}})=\operatorname*{\mathbb{E}}_{{\bm{a}}\sim\pi(\cdot|{\bm{s}})}[Q_{\phi}({\bm{s}},{\bm{a}})]$
only conditioned on states is proposed to mitigate the out-of-distribution
issue. It approaches the action distribution expectation of Q-value and can be
optimized by expectile regression:

$\mathcal{L}_{V}(\varphi)=|\tau-\mathds{1}_{\\{Q_{\phi}({\bm{s}}_{t}^{j},{\bm{a}}_{t}^{j})-V_{\varphi}({\bm{s}}_{t}^{j})<0\\}}|(Q_{\phi}({\bm{s}}_{t}^{j},{\bm{a}}_{t}^{j})-V_{\varphi}({\bm{s}}_{t}^{j}))^{2},$
(6)

where $\tau\in(0,1)$ is a constant. Note that the action ${\bm{a}}$ will be
sampled from the dataset according to the loss. Hence, we use rollouts in (5)
to estimate returns by TD($\lambda$) (see Appendix.B). The objective can
replace the TD-target that was originally used to optimize the Q-value.
However, only updating the critic depending on model rollout is not enough,
while the real state-action pairs from the offline dataset can supervise the
critic to additionally learn in-distribution information. So the critic
distribution is optimized on both model data and offline data:

$\mathcal{L}_{Q}^{1}(\phi)=-\frac{1}{HT}\sum_{t=1}^{T}\sum_{h=1}^{H}\log{\text{Pr}_{Q_{\phi}}(R_{t}^{h}|{\bm{s}}_{t}^{h})},\;\;\;\mathcal{L}_{Q}^{2}(\phi)=-\frac{1}{T}\sum_{t=1}^{T}\log{\text{Pr}_{Q_{\phi}}({\bm{r}}_{t}+\gamma
V_{\varphi}({\bm{s}}_{t+1})|{\bm{s}}_{t})}$ (7)

where ${\bm{r}}_{t}$ is the reward from the offline dataset. Finally, the
critic loss is the combination of the above.

$\mathcal{L}_{Q}(\phi)=\mathcal{L}_{Q}^{1}(\phi)+\mathcal{L}_{Q}^{2}(\phi)$
(8)

###  4.2 Model-based Policy Expansion for World Models Fine-tuning

After offline pretrain, we fine-tune the world model with online exploration
to enhance performance and improve its generalization to similar tasks.
However, directly using online algorithms suffers from non-recoverable
performance drops potentially caused by the distribution shift and the change
of the dynamic model. On the other hand, standard offline RL algorithms
typically exhibit worse online fine-tuning performance due to their
conservative design[20, 17]. Therefore, we try to bridge offline-online RL by
fully utilizing offline pretraining policies[32].

Suppose that $\pi_{\beta}$ is the offline policy. Instead of directly fine-
tuning it in the online stage, we optimize a new policy $\pi_{\psi}$ by
gradually learning from prior policy $\pi_{\beta}$ and online exploring
simultaneously. Concretely, given a current state ${\bm{s}}$, an action set is
composed of actions generated by two policies:
$\mathcal{A}=\\{{\bm{a}}_{\beta}\sim\pi_{\beta}({\bm{s}}),{\bm{a}}_{\psi}\sim\pi_{\psi}({\bm{s}})\\}.$
The actions are selected probabilistically based on their potential
utilization value:

$P(\omega)[k]=\frac{\exp(Q_{\phi}({\bm{s}},{\bm{a}}_{k})/\alpha)}{\exp(Q_{\phi}({\bm{s}},{\bm{a}}_{\beta})/\alpha)+\exp(Q_{\phi}({\bm{s}},{\bm{a}}_{\psi})/\alpha)},\;\forall
k\in\\{\beta,\psi\\},$ (9)

where $\alpha$ is temperature. According to this probability function, the two
policies can be trained in a joint adaptive manner that has the advantage of
adaptiveness in the sense of allowing generalizing of new abilities. In this
way, the offline replay-buffer can be directly transferred to the online fine-
tuning stage simplifying the complexity of balanced sampling[26].

In the model-based setting, the dynamic model also needs to be fine-tuned
since the model primarily generates the policy update data. The world model
benefits from its robust design in capturing temporal dependencies and
modeling latent dynamics, which contributes to its stability during fine-
tuning. However, the dynamic model might be influenced by the incorrect
estimates learned from conservative offline training, making online fine-
tuning more challenging. Thus, we develop an additional dynamic model for
fine-tuning, expanding the policy to adapt to new tasks.

###  4.3 Reachability Estimation Function as Safety Guarantee

To ensure safety throughout the entire training process, prior works consider
adopting the relaxation of the Lagrangian multiplier method on (4). The
optimization problem is transformed into an unconstrained safe model-based
problem with the Lagrangian multiplier $\lambda_{i}$:

$\max_{\pi_{\psi}}\min_{\lambda_{i}\geq
0}J^{\mathcal{R}}(\pi_{\psi})-\lambda_{i}(J^{\mathcal{C}}_{i}(\pi_{\psi})-d^{i}),\;\forall
i\in\\{1,\dots,C\\}.$ (10)

However, the soft constraints will still lead to a certain chance of
violation. It becomes more severe in the offline setting because of the
difficulty in estimating the cost values. Hence, we replace soft constraints
with hard constraints by the feasibility guidance approach. We first divide
our optimization problem into two parts: the feasible part and the infeasible
part.

For the feasible part, we want the agent to maintain safety within the
feasible set and maximize the reward return. For the infeasible part, we want
the policy to avoid the violations as little as possible and return to the
feasible set. As the safe offline RL problem has another objective to
optimize: $D(\pi_{\psi}||\pi_{b})$ ($\pi_{b}$ is the behavior policy), we can
reformulate our offline safe model-based problem as follows:

$\textbf{Feasible\
part:}\;\;\max_{\pi_{\psi}}\operatorname*{\mathbb{E}}_{{\bm{s}}}\Big{[}V_{\varphi}^{r}({\bm{s}})\cdot\mathds{1}_{s\in\mathcal{S}_{f}}\Big{]},\;\;\text{s.t.}\;\;V_{\varphi}^{c}({\bm{s}})=0,\;\;D(\pi_{\psi}||\pi_{b})\leq\epsilon,$
(11) $\textbf{Infeasible\
part:}\;\;\max_{\pi_{\psi}}\operatorname*{\mathbb{E}}_{{\bm{s}}}\Big{[}-V_{\varphi}^{c}({\bm{s}})\cdot\mathds{1}_{s\notin\mathcal{S}_{f}}\Big{]},\;\;\text{s.t.}\;\;D(\pi_{\psi}||\pi_{b})\leq\epsilon,$
(12)

where
$V_{\varphi}^{r}({\bm{s}})=\operatorname*{\mathbb{E}}_{{\bm{a}}\sim\pi_{\psi}}[\sum_{t=0}^{\infty}\gamma^{t}r({\bm{s}}_{t},{\bm{a}}_{t})]$
and
$V_{\varphi}^{c}({\bm{s}})=\operatorname*{\mathbb{E}}_{{\bm{a}}\sim\pi_{\psi}}[\sum_{t=0}^{\infty}\gamma^{t}c({\bm{s}}_{t},{\bm{a}}_{t})]$
represent the expected cumulative rewards and costs, $\mathcal{S}_{f}$ is the
feasible set containing all feasible states. Note that the policy constraint
is a soft constraint so the feasible part becomes a problem incorporating
coupled hard and soft constraints. Thus, we use the reachability estimation
function
$u^{\pi}({\bm{s}})=\operatorname*{\mathbb{E}}_{\tau\sim\pi}[\max_{{\bm{s}}_{t}\in\tau}\mathds{1}_{({\bm{s}}_{t}|{\bm{s}}_{0},\pi)\in\mathcal{S}_{f}}]$
to handle the mixed constraints while prioritizing the hard constraints from
RESPO[51]. It also assists the agent in reentering the feasible region even
when it is in the infeasible region.

Therefore, we couple the out-of-distribution actions avoidance into the
problem by using advantage functions. Also, we take advantage of the Augmented
Lagrangian with a proximal relaxation method that is quite effective in safe
model-based RL algorithms. The final problem is as follows:

$\max_{\pi_{\psi}}\;\operatorname*{\mathbb{E}}_{{\bm{s}}}\Big{[}(A^{r}({\bm{s}},{\bm{a}})-\Phi(V_{\varphi}^{c}({\bm{s}}),\lambda_{p}^{k},\mu^{k}))\cdot
u^{\pi}_{\psi}-A^{c}({\bm{s}},{\bm{a}})\cdot(1-u^{\pi}_{\psi})\Big{]},\;\;\text{s.t.}\;\;D(\pi_{\psi}||\pi_{b})\leq\epsilon.$
(13)

where
$A^{r}({\bm{s}},{\bm{a}})=Q_{\phi}^{r}({\bm{s}},{\bm{a}})-V_{\varphi}^{r}({\bm{s}})$
and
$A^{c}({\bm{s}},{\bm{a}})=Q_{\phi}^{c}({\bm{s}},{\bm{a}})-V_{\varphi}^{c}({\bm{s}})$.
We can finally obtain a closed-form solution of $\pi_{\psi}$, which we can use
to extract the optimal policy. (For more details see Appendix.A.1)

$\pi^{*}({\bm{a}}|{\bm{s}})=\frac{1}{Z}w\cdot\pi_{b}({\bm{a}}|{\bm{s}}),$ (14)
$\mathcal{L}_{\pi}(\psi)=-\operatorname*{\mathbb{E}}_{\tau\sim\mathcal{D}}\Big{[}\sum_{t=1}^{T}(w\cdot\log\pi_{\psi}({\bm{a}}_{t}|{\bm{s}}_{t})+\eta
H[\pi_{\psi}({\bm{a}}_{t}|{\bm{s}}_{t})])-\Phi(V_{\varphi}^{c}({\bm{s}}),\lambda_{p}^{k},\mu^{k})\cdot
u^{\pi}_{\psi}\Big{]},$ (15)
$\text{where}\;\;\;w=u^{\pi}_{\psi}\cdot\exp(\beta_{1}A^{r}({\bm{s}},{\bm{a}}))+(1-u^{\pi}_{\psi})\cdot\exp(-\beta_{2}A^{c}({\bm{s}},{\bm{a}})).$
(16)

In the online fine-tuning stage, we apply policy expansion as follows: use
$\pi_{b}$ as the frozen offline policy trained by (15) and learn a new policy
$\pi_{\psi}$ without policy behavior constraint in (13). Thus, the online
policy loss can change the advantage weight as:

$w=u^{\pi}_{\psi}\beta_{1}A^{r}({\bm{s}},{\bm{a}})-(1-u^{\pi}_{\psi})\beta_{2}A^{c}({\bm{s}},{\bm{a}}).$
(17)

For cost critic $Q^{c}(s,a)$ and value $V^{c}(s)$ training, we use the same
way as Section 4.1. The whole algorithm training outline is shown in
Appendix.B.

Figure 2: Offline experimental results. Comparing FOSP to baselines across
five image-based safety tasks. The results for all three algorithms are
obtained after training for 1 million steps. Reward: averaged episode reward
return. Cost: averaged episode cost return. Cost Regret: averaged cost value
throughout the training phase.

##  5 Experimental Results

In this section, we evaluate our method on different agents and tasks from the
Safety-Gymnasium in simulation. We deploy it in motion planning control tasks
on a real Franka robot as well. We aim to address the following issues: (1)
How does FOSP perform in offline and online fine-tuning? (2) How does each
component in FOSP contribute to its overall functionality? (3) Does FOSP
successfully handle unseen safety regions in the real world?

###  5.1 Experimental Setup

#### Simulation Tasks

We consider five tasks on Safety-Gymnasium benchmark[11] environments. The
agents need to navigate to predetermined goals without collision with hazards
and vases. We measure the performance with three metrics: average undiscounted
episode reward return (Reward), average undiscounted episode cost return
(Cost) and average cost value throughout the training phase (Cost Regret). To
evaluate the advantage of world models in handling high-dimensional features,
we use 64$\times$64 pixels RGB image as inputs obtained from the first-person
perspective of the agent, which is more representative of real-world
environments. The offline datasets are sampled by three different behaviors:
unsafe policy, safe policy and random policy to evaluate the performance of
FOSP on diverse behavior samples. They are mixed in a 1:1:1 ratio, with each
part containing 200 trajectories. For more information see Appendix.C.

#### Baselines

Because our work is the first approach in the safe model-based offline-to-
online area to the best of our knowledge, there are not many baselines for us
to compare. Prior works in offline safe RL use the model-free method which
typically performs poorly in vision-only tasks. Therefore, we compare FOSP
with the following two model-based baselines: SafeDreamer[8], an online
powerful algorithm in safe model-based reinforcement learning without online
planning and DreamerV3[5], a strong baseline in visual tasks but overlooks
constraint violations.

Figure 3: Ablation studies. We evaluate ablations in SafetyPointGoal2 with
means of three seeds. The vertical line divides the offline and online phases.
(1M steps for offline and 0.75M steps for online)

#### Real Robot

The real robot environment includes a 7-DOF Franka Emika Panda robot as the
controlled agent and a static third-person Intel RealSense camera to obtain
images as inputs. Also, the agent has additional inputs: the robot end-
effector pose information. We design the task SafeReach that controls the
robot to reach a desired goal while avoiding collisions with predefined
obstacles in its field of view. The rewards and costs are sparse and detected
manually determining whether the robot reaches the goal or violates the
constraints. The dataset consists of demonstration data, safety violation
data, and failure data. Furthermore, we design some unseen safe regions and
different targets that are not contained in the offline dataset. These unseen
tasks are related to some undefined obstacles that need to be identified
through observation which can not be solved by motion planning methods. The
task further illustrates the practicality of our approach.

###  5.2 Main Results

#### Offline Pretrain

We show our offline training result in Figure 2. FOSP outperformed the similar
safe algorithm SafeDreamer, excelling in decreasing costs and better balancing
task performance with constraint avoidance. Furthermore, It has comparable or
slightly lower performance than DreamerV3 while achieving nearly zero
violations. Note that all methods struggle to get higher rewards in
SafetyPointPush because the line-of-sight obstructions necessitate online
exploration.

#### Online Fine-tuning

The results of online fine-tuning are shown on Table 1. It proves that our
algorithm not only performs well offline but can also be fine-tuned to achieve
better performance. Meanwhile, FOSP further reduces the costs during the
online fine-tuning. We also compare our method with the OSRP version of
SafeDreamer which can only be trained online. We notice that the SafeDreamer
training exclusively online outperforms the offline-to-online fine-tuning
approach. This may be because direct fine-tuning is affected by distribution
shifts, making it difficult to improve performance in limited steps. So FOSP
is necessary for us to bridge the safety of offline and online while
maintaining its performance. (For more detail see Appendix.F)

#### Ablation Studies

In this section, we compare the effects of different components of the model
on its performance in Figure 3. Specifically, we provide the following
ablations: (i) learning value function without in-sample optimization (Eq.6)
in Section 4.1; (ii) directly fine-tuning the model without model-based policy
expansion (Eq.9) in Section 4.2; (iii) update safe actor without reachability
estimation function in Section 4.3; (iv) design the penalty without the
Augmented Lagrangian and use the feasible value function[52]. The results show
that each component of our method individually contributes to ensuring offline
training performance, further improvement during online fine-tuning, and
maintaining near-zero cost constraint violations.

Table 1: Online fine-tuning results. Reward return and cost return of methods
in offline-to-online fine-tuning (1M steps for offline and 0.5M steps for
online fine-tuning). And results of online algorithms only trained online
(1.5M steps). Means of three seeds.

| Offline-to-Online | Online-only
---|---|---
| DreamerV3 | SafeDreamer | FOSP(Ours) | SafeDreamer(+planning)
Tasks | Reward | Cost | Reward | Cost | Reward | Cost | Reward | Cost
PointGoal1 | 17.4$\rightarrow$25.8 | 82.3$\rightarrow$85.1 | 12.7$\rightarrow$19.2 | 1.5$\rightarrow$0.02 |  18.7$\rightarrow$21.5 |  1.4$\rightarrow$0.2 | 20.1 | 0.6
PointButton1 | 15.1$\rightarrow$20.8 | 167.7$\rightarrow$157.9 | 8.9$\rightarrow$10.7 | 10.2$\rightarrow$4.5 | 14.7$\rightarrow$18.1 | 9.5$\rightarrow$2.1 | 12.4 | 5.5
PointPush1 | 2.3$\rightarrow$14.6 | 30.9$\rightarrow$26.3 | 1.3$\rightarrow$10.3 |  17.8$\rightarrow$0.12 |  4.0$\rightarrow$13.2 | 18.1$\rightarrow$0.1 | 8.1 | 0.7
PointGoal2 | 9.5$\rightarrow$18.9 | 367.1$\rightarrow$290.2 | 3.6$\rightarrow$12.7 | 8.6$\rightarrow$0.3 | 8.1$\rightarrow$13.5 | 7.5$\rightarrow$0.23 | 13.0 | 1.7
CarGoal2 | 16.7$\rightarrow$24.1 | 231.0$\rightarrow$287.9 | 6.9$\rightarrow$9.8 |  0.7$\rightarrow$0.09 | 10.1$\rightarrow$14.5 | 1.6$\rightarrow$0.07 | 11.2 | 0.34
Average | 20.8 | 171.1 | 12.5 | 1.01 | 16.6 | 0.54 | 12.9 | 1.77

###  5.3 Unseen Safe Tasks

In the real-world environment, robots will incur many safety constraints while
performing various tasks. Our key challenge is how to safely train the robots
to adapt to unseen constraints by safe learning-based methods. Following this
insight, we introduce obstacles and targets of different shapes and sizes in
Reach tasks, where the robot has only encountered a limited number of these
during offline training. Our goal is to make the robot use a few trials to
learn planning strategies in environments with new obstacles as safely as
possible. Thus, we design three real-world transfer tasks (see details in
Appendix.C) and show their results in Table 2.

Table 2: Real-world unseen tasks. We record the success rate (SR, %) and the
constraint violation rate (CV, %) over 20 tests in three tasks while it will
be labeled as a violation if it collides with an obstacle. The robot has fine-
tuned 40 steps. See Appendix.C for more details.

| Before FT | During FT | After FT
---|---|---|---
Tasks | SR | CV | SR | CV | SR | CV
1 | 40 | 50 | 45 | 40 | 50 | 30
2 | 30 | 50 | 35 | 45 | 35 | 40
3 | 20 | 60 | 40 | 40 | 50 | 30

In brief, FOSP can successfully finish these tasks safely, demonstrating its
strong offline learning and adjustment capabilities. In contrast, only
learning from the offline dataset makes it difficult to generalize to new
scenarios. Its strong adaptability in fine-tuning showcases the robustness and
great potential of FOSP.

##  6 Conclusion

We introduce FOSP, a novel method that integrates model-based reinforcement
learning with offline-to-online fine-tuning to enhance safety in robotic
tasks. FOSP addresses the challenges of balancing performance and safety
during both offline and online phases using the reachability estimation
function to unify different constraints and offline safe policies to guide
online exploration. Our experimental results demonstrate that FOSP achieves
robust performance in various safe planning tasks, especially those with high-
dimensional vision inputs. Furthermore, we evaluate FOSP on unseen safety-
critical tasks in dynamic real-world environments highlighting its practical
value in safe few-shot fine-tuning. FOSP is the first approach to extend MBRL
into safe offline-to-online training, showing promising results in the Safety-
Gymnasium benchmark. Overall, FOSP shows a significant step forward in
ensuring safe and efficient decision-making in embodied intelligence.

## References

  * Hafner et al. [2019] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi.  Dream to control: Learning behaviors by latent imagination.  _arXiv preprint arXiv:1912.01603_ , 2019.
  * Cang et al. [2021] C. Cang, A. Rajeswaran, P. Abbeel, and M. Laskin.  Behavioral priors and dynamics models: Improving performance and domain transfer in offline rl.  _arXiv preprint arXiv:2106.09119_ , 2021.
  * Hafner et al. [2019] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson.  Learning latent dynamics for planning from pixels.  In _International conference on machine learning_ , pages 2555–2565. PMLR, 2019.
  * Hafner et al. [2020] D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba.  Mastering atari with discrete world models.  _arXiv preprint arXiv:2010.02193_ , 2020.
  * Hafner et al. [2023] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap.  Mastering diverse domains through world models.  _arXiv preprint arXiv:2301.04104_ , 2023.
  * Feng et al. [2023] Y. Feng, N. Hansen, Z. Xiong, C. Rajagopalan, and X. Wang.  Finetuning offline world models in the real world.  _arXiv preprint arXiv:2310.16029_ , 2023.
  * As et al. [2022] Y. As, I. Usmanova, S. Curi, and A. Krause.  Constrained policy optimization via bayesian world models.  _arXiv preprint arXiv:2201.09802_ , 2022.
  * Huang et al. [2023] W. Huang, J. Ji, B. Zhang, C. Xia, and Y. Yang.  Safe dreamerv3: Safe reinforcement learning with world models.  _arXiv preprint arXiv:2307.07176_ , 2023.
  * Levine et al. [2020] S. Levine, A. Kumar, G. Tucker, and J. Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ , 2020.
  * Nair et al. [2020] A. Nair, A. Gupta, M. Dalal, and S. Levine.  Awac: Accelerating online reinforcement learning with offline datasets.  _arXiv preprint arXiv:2006.09359_ , 2020.
  * Ji et al. [2023] J. Ji, B. Zhang, J. Zhou, X. Pan, W. Huang, R. Sun, Y. Geng, Y. Zhong, J. Dai, and Y. Yang.  Safety gymnasium: A unified safe reinforcement learning benchmark.  _Advances in Neural Information Processing Systems_ , 36, 2023.
  * Wu et al. [2019] Y. Wu, G. Tucker, and O. Nachum.  Behavior regularized offline reinforcement learning.  _arXiv preprint arXiv:1911.11361_ , 2019.
  * Fujimoto et al. [2019] S. Fujimoto, D. Meger, and D. Precup.  Off-policy deep reinforcement learning without exploration.  In _International conference on machine learning_ , pages 2052–2062. PMLR, 2019.
  * Yu et al. [2020] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma.  Mopo: Model-based offline policy optimization.  _Advances in Neural Information Processing Systems_ , 33:14129–14142, 2020.
  * Yu et al. [2021] T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn.  Combo: Conservative offline model-based policy optimization.  _Advances in neural information processing systems_ , 34:28954–28967, 2021.
  * Rafailov et al. [2021] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn.  Offline reinforcement learning from images with latent space models.  In _Learning for dynamics and control_ , pages 1154–1168. PMLR, 2021\.
  * Kumar et al. [2020] A. Kumar, A. Zhou, G. Tucker, and S. Levine.  Conservative q-learning for offline reinforcement learning.  _Advances in Neural Information Processing Systems_ , 33:1179–1191, 2020.
  * Kidambi et al. [2020] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims.  Morel: Model-based offline reinforcement learning.  _Advances in neural information processing systems_ , 33:21810–21823, 2020.
  * Fujimoto and Gu [2021] S. Fujimoto and S. S. Gu.  A minimalist approach to offline reinforcement learning.  _Advances in neural information processing systems_ , 34:20132–20145, 2021.
  * Kostrikov et al. [2021] I. Kostrikov, A. Nair, and S. Levine.  Offline reinforcement learning with implicit q-learning.  _arXiv preprint arXiv:2110.06169_ , 2021.
  * Xu et al. [2023] H. Xu, L. Jiang, J. Li, Z. Yang, Z. Wang, V. W. K. Chan, and X. Zhan.  Offline rl with no ood actions: In-sample learning via implicit value regularization.  _arXiv preprint arXiv:2303.15810_ , 2023.
  * Garg et al. [2023] D. Garg, J. Hejna, M. Geist, and S. Ermon.  Extreme q-learning: Maxent rl without entropy.  _arXiv preprint arXiv:2301.02328_ , 2023.
  * Peng et al. [2019] X. B. Peng, A. Kumar, G. Zhang, and S. Levine.  Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.  _arXiv preprint arXiv:1910.00177_ , 2019.
  * He et al. [2024] L. He, L. Shen, J. Tan, and X. Wang.  Aligniql: Policy alignment in implicit q-learning through constrained optimization, 2024.
  * Xiao et al. [2023] C. Xiao, H. Wang, Y. Pan, A. White, and M. White.  The in-sample softmax for offline reinforcement learning.  _arXiv preprint arXiv:2302.14372_ , 2023.
  * Lee et al. [2022] S. Lee, Y. Seo, K. Lee, P. Abbeel, and J. Shin.  Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble.  In _Conference on Robot Learning_ , pages 1702–1712. PMLR, 2022.
  * Nakamoto et al. [2024] M. Nakamoto, S. Zhai, A. Singh, M. Sobol Mark, Y. Ma, C. Finn, A. Kumar, and S. Levine.  Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Rudner et al. [2021] T. G. Rudner, C. Lu, M. A. Osborne, Y. Gal, and Y. Teh.  On pathologies in kl-regularized reinforcement learning from expert demonstrations.  _Advances in Neural Information Processing Systems_ , 34:28376–28389, 2021.
  * Xie et al. [2021] T. Xie, N. Jiang, H. Wang, C. Xiong, and Y. Bai.  Policy finetuning: Bridging sample-efficient offline and online reinforcement learning.  _Advances in neural information processing systems_ , 34:27395–27407, 2021.
  * Rajeswaran et al. [2017] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine.  Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.  _arXiv preprint arXiv:1709.10087_ , 2017.
  * Zhao et al. [2022] Y. Zhao, R. Boney, A. Ilin, J. Kannala, and J. Pajarinen.  Adaptive behavior cloning regularization for stable offline-to-online reinforcement learning.  _arXiv preprint arXiv:2210.13846_ , 2022.
  * Zhang et al. [2023] H. Zhang, W. Xu, and H. Yu.  Policy expansion for bridging offline-to-online reinforcement learning.  _arXiv preprint arXiv:2302.00935_ , 2023.
  * Altman [2021] E. Altman.  _Constrained Markov decision processes_.  Routledge, 2021.
  * Achiam et al. [2017] J. Achiam, D. Held, A. Tamar, and P. Abbeel.  Constrained policy optimization.  In _International conference on machine learning_ , pages 22–31. PMLR, 2017.
  * Chow et al. [2018] Y. Chow, M. Ghavamzadeh, L. Janson, and M. Pavone.  Risk-constrained reinforcement learning with percentile risk criteria.  _Journal of Machine Learning Research_ , 18(167):1–51, 2018.
  * Ding et al. [2020] D. Ding, K. Zhang, T. Basar, and M. Jovanovic.  Natural policy gradient primal-dual method for constrained markov decision processes.  _Advances in Neural Information Processing Systems_ , 33:8378–8390, 2020.
  * Tessler et al. [2018] C. Tessler, D. J. Mankowitz, and S. Mannor.  Reward constrained policy optimization.  _arXiv preprint arXiv:1805.11074_ , 2018.
  * Sikchi et al. [2022] H. Sikchi, W. Zhou, and D. Held.  Learning off-policy with online planning.  In _Conference on Robot Learning_ , pages 1622–1633. PMLR, 2022.
  * Hansen et al. [2022] N. Hansen, X. Wang, and H. Su.  Temporal difference learning for model predictive control.  _arXiv preprint arXiv:2203.04955_ , 2022.
  * Hogewind et al. [2022] Y. Hogewind, T. D. Simao, T. Kachman, and N. Jansen.  Safe reinforcement learning from pixels using a stochastic latent representation.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Lee et al. [2020] A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine.  Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model.  _Advances in Neural Information Processing Systems_ , 33:741–752, 2020.
  * Cao et al. [2024] C. Cao, Z. Yan, R. Lu, J. Tan, and X. Wang.  Offline goal-conditioned reinforcement learning for safety-critical tasks with recovery policy.  _arXiv preprint arXiv:2403.01734_ , 2024.
  * Xu et al. [2022] H. Xu, X. Zhan, and X. Zhu.  Constraints penalized q-learning for safe offline reinforcement learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 8753–8760, 2022.
  * Li et al. [2022] J. Li, X. Zhan, H. Xu, X. Zhu, J. Liu, and Y.-Q. Zhang.  When data geometry meets deep function: Generalizing offline reinforcement learning.  _arXiv preprint arXiv:2205.11027_ , 2022.
  * Lee et al. [2022] J. Lee, C. Paduraru, D. J. Mankowitz, N. Heess, D. Precup, K.-E. Kim, and A. Guez.  Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation.  _arXiv preprint arXiv:2204.08957_ , 2022.
  * Lin et al. [2023] Q. Lin, B. Tang, Z. Wu, C. Yu, S. Mao, Q. Xie, X. Wang, and D. Wang.  Safe offline reinforcement learning with real-time budget constraints.  In _International Conference on Machine Learning_ , pages 21127–21152. PMLR, 2023.
  * Liu et al. [2023] Z. Liu, Z. Guo, Y. Yao, Z. Cen, W. Yu, T. Zhang, and D. Zhao.  Constrained decision transformer for offline safe reinforcement learning.  In _International Conference on Machine Learning_ , pages 21611–21630. PMLR, 2023.
  * Choi et al. [2020] J. Choi, F. Castaneda, C. J. Tomlin, and K. Sreenath.  Reinforcement learning for safety-critical control under model uncertainty, using control lyapunov functions and control barrier functions.  _arXiv preprint arXiv:2004.07584_ , 2020.
  * Zheng et al. [2023] Y. Zheng, J. Li, D. Yu, Y. Yang, S. E. Li, X. Zhan, and J. Liu.  Feasibility-guided safe offline reinforcement learning.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Yu et al. [2023] D. Yu, W. Zou, Y. Yang, H. Ma, S. E. Li, Y. Yin, J. Chen, and J. Duan.  Safe model-based reinforcement learning with an uncertainty-aware reachability certificate.  _IEEE Transactions on Automation Science and Engineering_ , 2023.
  * Ganai et al. [2024] M. Ganai, Z. Gong, C. Yu, S. Herbert, and S. Gao.  Iterative reachability estimation for safe reinforcement learning.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Fisac et al. [2019] J. F. Fisac, N. F. Lugovoy, V. Rubies-Royo, S. Ghosh, and C. J. Tomlin.  Bridging hamilton-jacobi safety analysis and reinforcement learning.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 8550–8556. IEEE, 2019.
  * Kakade and Langford [2002] S. Kakade and J. Langford.  Approximately optimal approximate reinforcement learning.  In _Proceedings of the Nineteenth International Conference on Machine Learning_ , pages 267–274, 2002.

##  Appendix A Theoretical Interpretations

###  A.1 Derivation of the Final Optimization Problem

This section analyzes the derivation from Eq.(21) to Eq.(13) by using the
Augmented Lagrangian and advantage function. First, we introduce Proposition
1.

######  Proposition 1.

The optimization objective of Eq.(11) is the necessary condition of
$\max_{\pi}\operatorname*{\mathbb{E}}_{{\bm{s}}}[A^{r}({\bm{s}},{\bm{a}})\cdot\mathds{1}_{s\in\mathcal{S}_{f}}]$.

Proof. We start with the advantage function estimation in [53] and [23].
Suppose that
$d_{\pi}({\bm{s}})=\sum_{t=0}^{\infty}\gamma^{t}p({\bm{s}}_{t}={\bm{s}}|\pi)$
is discounted state distribution with $\pi$, and
$p({\bm{s}}_{t}={\bm{s}}|\pi)$ is the probability of the state ${\bm{s}}$
guided by $\pi$ for $t$ steps. The discounted sum of advantage expectation
under $d_{\pi}({\bm{s}})$ can be represented as follows:

$\displaystyle\operatorname*{\mathbb{E}}_{{\bm{s}}_{1:\infty}\sim
d_{\pi}({\bm{s}})}\left[\sum_{t=0}^{\infty}\gamma^{t}A_{\mu}({\bm{s}}_{t},{\bm{a}}_{t})\right]$
$\displaystyle=\operatorname*{\mathbb{E}}_{{\bm{s}}_{1:\infty}\sim
d_{\pi}({\bm{s}})}\left[\sum_{t=0}^{\infty}\gamma^{t}\left(r({\bm{s}}_{t},{\bm{a}}_{t})+\gamma
V_{\mu}({\bm{s}}_{t+1})-V_{\mu}({\bm{s}}_{t})\right)\right]$
$\displaystyle=\operatorname*{\mathbb{E}}_{{\bm{s}}_{1:\infty}\sim
d_{\pi}({\bm{s}})}\left[-V_{\mu}({\bm{s}}_{0})+\sum_{t=0}^{\infty}\gamma^{t}r({\bm{s}}_{t},{\bm{a}}_{t})\right]$
$\displaystyle=-\operatorname*{\mathbb{E}}_{{\bm{s}}_{0}\sim
d_{\pi}({\bm{s}}_{0})}\left[V_{\mu}({\bm{s}}_{0})\right]+\operatorname*{\mathbb{E}}_{{\bm{s}}_{1:\infty}\sim
d_{\pi}({\bm{s}})}\left[\sum_{t=0}^{\infty}\gamma^{t}r({\bm{s}}_{t},{\bm{a}}_{t})\right]$
$\displaystyle=-J(\mu)+J(\pi),$

where $\mu$ is the behavior policy used to sample data. So we get the
objective representation $J(\pi)=\operatorname*{\mathbb{E}}_{{\bm{s}}\sim
d_{\pi}({\bm{s}})}\Big{[}\operatorname*{\mathbb{E}}_{{\bm{a}}\sim\pi({\bm{a}}|{\bm{s}})}[A_{\mu}({\bm{s}}_{t},{\bm{a}}_{t})]\Big{]}+J(\mu)$.
Then, we have the following derivation:

$\displaystyle\max_{\pi}\operatorname*{\mathbb{E}}_{{\bm{s}}\sim
d_{\pi}({\bm{s}})}[A^{r}_{\pi}({\bm{s}},{\bm{a}})]$
$\displaystyle\Rightarrow\max_{\pi}\operatorname*{\mathbb{E}}_{{\bm{s}}\sim
d_{\pi}({\bm{s}})}\Big{[}\operatorname*{\mathbb{E}}_{{\bm{a}}\sim\pi({\bm{a}}|{\bm{s}})}[A_{\mu}^{r}({\bm{s}}_{t},{\bm{a}}_{t})\Big{]}$
$\displaystyle=\max_{\pi}\operatorname*{\mathbb{E}}_{{\bm{s}}\sim
d_{\pi}({\bm{s}})}\Big{[}\operatorname*{\mathbb{E}}_{{\bm{a}}\sim\pi({\bm{a}}|{\bm{s}})}[A_{\mu}^{r}({\bm{s}}_{t},{\bm{a}}_{t})\Big{]}+J(\mu)$
$\displaystyle=\max_{\pi}\;J(\pi)$
$\displaystyle=\max_{\pi}\;\operatorname*{\mathbb{E}}[V_{\pi}^{r}({\bm{s}})].$

The last equation holds by the definition of $V_{\pi}^{r}({\bm{s}})$.
$\square$

Due to the symmetry of
$\max_{\pi}\operatorname*{\mathbb{E}}[V_{\pi}^{r}({\bm{s}})]$ and
$\max_{\pi}\operatorname*{\mathbb{E}}[V_{\pi}^{c}({\bm{s}})]$, the same proof
also applies to $\max_{\pi}\operatorname*{\mathbb{E}}[V_{\pi}^{c}({\bm{s}})]$.
So we have proposition 2.

######  Proposition 2.

The optimization objective of Eq.(12) is the necessary condition of
$\max_{\pi}\operatorname*{\mathbb{E}}_{{\bm{s}}}[-A^{c}({\bm{s}},{\bm{a}})\cdot\mathds{1}_{s\notin\mathcal{S}_{f}}]$.

The problem converts to:

$\textbf{Feasible\
part:}\;\;\max_{\pi_{\psi}}\operatorname*{\mathbb{E}}_{{\bm{s}}}\Big{[}A^{r}({\bm{s}},{\bm{a}})\cdot\mathds{1}_{s\in\mathcal{S}_{f}}\Big{]},\;\;\text{s.t.}\;\;V_{\varphi}^{c}({\bm{s}})=0,\;\;D(\pi_{\psi}||\pi_{b})\leq\epsilon,$
(18) $\textbf{Infeasible\
part:}\;\;\max_{\pi_{\psi}}\operatorname*{\mathbb{E}}_{{\bm{s}}}\Big{[}-A^{c}({\bm{s}},{\bm{a}})\cdot\mathds{1}_{s\in\mathcal{S}_{f}}\Big{]},\;\;\text{s.t.}\;\;D(\pi_{\psi}||\pi_{b})\leq\epsilon.$
(19)

However, it is hard for us to get a close form of $\pi_{\psi}$ in this
problem. Therefore, we will first disregard the constraints of the behavior
policy and address the hard safety constraints. The Augmented Lagrangian is
used to deal with the feasible part. Thus, we use the following relaxation:

$\max_{\pi_{\psi}}\min_{\lambda\geq
0}\left[A^{r}({\bm{s}},{\bm{a}})\mathds{1}_{s\in\mathcal{S}_{f}}-\lambda
V_{\varphi}^{c}({\bm{s}})+\frac{1}{\mu_{k}}(\lambda-\lambda_{k})^{2}\right],$
(20)

where $\mu_{k}$ is a non-decreasing penalty term corresponding to gradient
step $k$. By smoothing the left-hand side term in Eq.(18), the last term
ensures that $\lambda$ stays near its previous estimate. Differentiating
Eq.(18) with respect to $\lambda_{p}^{k}$ leads to the following update rule
for the Lagrange multiplier:

$\lambda_{p}^{k+1}=\begin{cases}\lambda_{p}^{k}+\mu^{k}V_{\varphi}^{c}&\text{
if }\lambda_{p}^{k}+\mu^{k}V_{\varphi}^{c}\geq 0,\\\ 0&\text{ otherwise
}.\end{cases}$

The objective becomes the following form:

$J(\pi,\lambda_{k},\mu_{k})=A^{r}({\bm{s}},{\bm{a}})\mathds{1}_{s\in\mathcal{S}_{f}}-\Psi\left(V_{\varphi}^{c},\lambda_{p}^{k},\mu^{k}\right),$

$\Psi\left(V_{\varphi}^{c},\lambda_{p}^{k},\mu^{k}\right)=\begin{cases}\lambda_{p}^{k}V_{\varphi}^{c}+\frac{\mu^{k}}{2}(V_{\varphi}^{c})^{2}&\text{
if }\lambda_{p}^{k}+\mu^{k}V_{\varphi}^{c}\geq 0,\\\
-\frac{\left(\lambda_{p}^{k}\right)^{2}}{2\mu^{k}}&\text{ otherwise
}.\end{cases}$

Then, we utilize the REF function to connect the two parts of the problem. The
REF function is defined as follows:

######  Definition 1.

The reachability estimation function (REF) $u:\mathcal{S}\rightarrow[0,1]$ is:

$u^{\pi}({\bm{s}}):=\operatorname*{\mathbb{E}}_{\tau\sim\pi}[\max_{{\bm{s}}_{t}\in\tau}\mathds{1}_{({\bm{s}}_{t}|{\bm{s}}_{0},\pi)\in\mathcal{S}_{f}}]$

And the problem becomes:

$\max_{\pi_{\psi}}\;\operatorname*{\mathbb{E}}_{{\bm{s}}}\Big{[}V_{\varphi}^{r}({\bm{s}})\cdot
u^{\pi}_{\psi}-V_{\varphi}^{c}({\bm{s}})\cdot(1-u^{\pi}_{\psi})\Big{]},\;\;\text{s.t.}\;\;V_{\varphi}^{c}({\bm{s}})=0,\;\;D(\pi_{\psi}||\pi_{b})\leq\epsilon.$
(21)

We introduce a proposition from [10] to solve the soft constraints from
offline training to provide a solution of Eq.(13).

######  Proposition 3.

Suppose that the optimizing problem has the following form:

$\max_{\pi}\operatorname*{\mathbb{E}}_{{\bm{a}}\sim\pi}[A(s,a)]\;\;\text{s.t.}\;D(\pi_{\psi}||\pi_{b})\leq\epsilon.$

The optimal solution will satisfy:

$\pi^{*}({\bm{a}}|{\bm{s}})\;\propto\;\exp(\alpha
A(s,a))\pi_{b}({\bm{a}}|{\bm{s}})$

Proof. The Lagrange function can be formulated as follows:

$L(\pi,\lambda,\mu)=\operatorname*{\mathbb{E}}_{{\bm{a}}\sim\pi}[A({\bm{s}},{\bm{a}})]-\lambda(D(\pi_{\psi}||\pi_{b})-\epsilon),$

Then we take the partial derivative with respect to $\pi$ and set it to 0:

$\frac{\partial
L}{\partial\pi}=A(s,a)-\lambda\log\pi_{b}({\bm{a}}|{\bm{s}})+\lambda\log\pi({\bm{a}}|{\bm{s}})+\lambda=0$

So we have:

$\pi^{*}({\bm{a}}|{\bm{s}})=\frac{1}{Z}\exp(\alpha
A(s,a))\pi_{b}({\bm{a}}|{\bm{s}}),$

where $Z$ is a normalizing constant. $\square$

Therefore, we have a solution to our optimization objective (13). The solved
$\pi_{\psi}$ can be explicitly expressed by Eq.(15).

##  Appendix B Implementation Details

The TD($\lambda$) return can be estimated as follows:

$R_{t}^{H}=V_{\varphi}({\bm{s}}_{t}^{H}),\;R_{t}^{h}={\bm{r}}_{t}^{h}+\gamma((1-\lambda)V_{\varphi}({\bm{s}}_{t}^{h})+\lambda
R_{t}^{h+1})$ (22)

where $H$ is the horizon of the model and $h\leq H$.

To implement the reachability estimation function $u_{\xi}$, we use the
theorem introduced by [51] (Theorem 1). The REF function can be trained as
follows:

$u({\bm{s}}_{t})=\max\\{\mathds{1}_{s\in\mathcal{S}_{f}},\gamma_{u}u({\bm{s}}_{t+1})\\},$
(23)

where $\gamma_{u}$ is a discount parameter $0<<\gamma_{u}<1$ to ensure
convergence of $u({\bm{s}})$. Meanwhile, we ensure that its learning rate is
greater than that of the critic and policy, but less than that of the
Lagrangian multiplier updates.

Algorithm 1 FOSP: Fine-tuning Offline Safe Policy through World Models

0: Offline dataset $\mathcal{D}$, policy ${\pi}_{\psi}$, critics
$Q_{\phi_{1}}$ and cost critics $Q^{c}_{\phi_{2}}$, value network
$V_{\varphi_{1}},V^{c}_{\varphi_{2}}$, reachability function network
$u_{\xi}$, world model $p_{\theta}$, policy rollout length $H$, number of
offline training steps $N_{\text{offline}}$, number of online fine-tuning
steps $N_{\text{online}}$.

1: Initialize neural network parameters
$\theta,\psi,\phi_{i},\varphi_{i},\xi$.$(i=1,2)$

2: for $i=1,2,3,\cdots,N_{\text{offline}}$ do

3: Sample a batch of trajectories $B\sim\mathcal{D}$.

4: Compute model state ${\bm{s}}_{t}\sim
p_{\theta}({\bm{s}}_{t}|{\bm{s}}_{t-1})$

5: Update $\theta$ using Eq. (D).

6: Generate $H$-step latent policy rollouts using world model $p_{\theta}$.

7: Compute target estimation $R_{t}$ by TD($\lambda$).

8: Update $V_{\varphi_{1}},V_{\varphi_{2}}$ using Eq. (6).

9: Update $Q_{\phi_{1}},Q_{\phi_{2}}$ using Eq. (8).

10: Update ${\pi}_{\psi}$ using Eq. (15).

11: Update $u_{\xi}$ using Eq. (23).

12: Initialize a new policy $\pi_{\psi^{\prime}}$ and freeze offline policy
$\pi_{\psi}$ , transfer critics and values.

13: for $i=1,2,3,\cdots,N_{\text{online}}$ do

14: Switch a policy $\pi$ according to Eq.(9) and rollout the policy in the
environment for an episode to collect a new trajectory $\tau$.

15: $\mathcal{D}=\mathcal{D}\cup\tau$.

16: for each training step do

17: Sample a batch of trajectories from $\mathcal{D}$.

18: Update $\theta$ using Eq. (D).

19: Generate $H$-step latent policy rollouts using world model $p_{\theta}$.

20: Compute target estimation $R_{t}$ by TD($\lambda$).

21: Update $V_{\varphi_{1}},V_{\varphi_{2}}$ using Eq. (6).

22: Update $Q_{\phi_{1}},Q_{\phi_{2}}$ using Eq. (8).

23: Update ${\pi}_{\psi^{\prime}}$ using Eq. (17).

24: Update $u_{\xi}$ using Eq. (23).

##  Appendix C Experimental Details

###  C.1 Real World Experiments

The experiments verify the effectiveness of the algorithm by controlling a
robotic arm to avoid obstacles and complete a reach task to the target
objection, called SafeReach. As shown in Figure 4(c), the main hardware setup
consists of three parts: a Franka Panda robotic arm, an experimental platform
for placing obstacles and target objections, and an Intel RealSense D435
camera for perception. On the experimental platform, we used white lines to
designate a 25cm$\times$25cm operational area as the task space for the
robotic arm.

#### SafeReach

For the task of reaching, the robot needs to consider the image information
acquired by the camera along with the current joint posture information as
planning conditions to determine the movement decision at the next step. As
shown in Figure 4, the robotic arm needs to navigate within the designated
movement space, bypassing the obstacles which are represented by the green
rectangular blocks, to touch the predefined target which is represented by the
red square block.

Figure 4: Simulation and real-world environment. Different tasks and agents in
the simulation and real-world. (a) Simulation tasks: we consider four
different tasks in the simulation: Push1, Goal1, Button1, Goal2. (from upper-
left to lower-right) The preceding words indicate the task type and the
following number represents the difficulty level. (b) Agent type: Car (upper)
and Point (lower). (c) Real-robot setup: we use raw images as inputs and
enable the robotic arm to complete obstacle avoidance tasks safely. Figure 5:
Unseen Tasks. We choose three unseen tasks to show the generalization of our
algorithm with different obstacles and goals.

#### Dataset Collection

To train the model for the SafeReach task, 200 motion trajectories were
collected as a dataset by teleoperating the robotic arm using a 3dconnexion
SpaceMouce. The dataset consists of the following: 120 trajectories where the
robotic arm did not touch any obstacles and successfully reached the target
object; 30 trajectories where it did not touch any obstacles but also did not
successfully reach the target; 40 trajectories where it touched obstacles but
successfully reached the target; 10 trajectories where it touched obstacles
and did not successfully reach the target. Each trajectory contains 80 sets of
data points at different time steps. Each data point includes: state: a
64$\times$64 RGB image captured by the camera, a 6-dimensional pose list of
the robotic arm; action: a 6-dimensional list recording the changes in the
robotic arm’s pose; cost: 1-dimensional cost index indicating whether the
robotic arm touched an obstacle; reward 1-dimensional reward index indicating
whether the object reached the target object. The labeling methods for the
cost and reward index are shown in Figure 6. The cost is set to 1 when the
robotic arm collides with an obstacle and it is set to 0 otherwise. The reward
is set to 1 when the object touches the target and it is set to 0 otherwise.
We defined 3 placement points for target objects and 6 placement points for
obstacles in the scenario. By restricting the placement positions of the
target objects and obstacles, we ensure that the model can converge more
efficiently on a limited dataset.

Figure 6: Dataset collection. Each row in the figure represents five frames of
the robotic arm’s motion captured from the same single trajectory. From the
moment the robotic arm starts touching an obstacle to the moment the
obstacle’s posture stops changing, the cost label in the dataset is marked as
1, while in other cases it is marked as 0, as shown in the red-framed part in
the image. When the robotic arm touches the target object, the reward label in
the dataset is marked as 1, as shown in the yellow-framed part in the image.

#### Transfer Tasks

To validate the generalization of the algorithm, we design a series of
transfer experiments. As illustrated in the left side of Figure 5, the
original task involves 2 green rectangular obstacles and 1 red cylindrical
target objection. We design three different transfer tasks, depicted on the
right side of Figure 5. The tasks include altering the shape of the obstacles,
changing the shape of the target object, and increasing the number of
obstacles. These experiments aim to verify the algorithm’s generalization in
avoiding obstacles of various shapes and quantities, as well as in reaching
target objects of different shapes.

###  C.2 Simulation Environment

#### SafetyGoal

The agent’s objective is to reach a goal while avoiding obstacles in its
environment. Each time the agent successfully reaches the designated goal, the
environment randomly generates a new one. The agent earns rewards for
approaching or reaching the goal but incurs penalties when it encounters
obstacles. These obstacles include immovable hazards and movable vases. The
agent’s observation state is represented by the images from its front and
back. Two specific tasks are selected: the Point-Goal task and the Car-Goal
task, which are performed by the Point and Car agents, respectively. The Point
agent is a robot that operates on a 2D plane and is capable of rotating and
moving both forward and backward. On the other hand, the Car agent, which is
slightly more complex, features two independent parallel wheels and a freely
rolling rear wheel.

#### SafetyButton

The agent’s goal is to navigate around both stationary and moving obstacles in
the environment to press one of several target buttons. Similar to the goal
task, the agent is rewarded for approaching or pressing the target button. The
Button task introduces dynamic obstacles that move quickly along set paths,
making it more challenging than the Goal task due to the costs incurred from
collisions with these moving obstacles.

#### SafetyPush

The agent’s goal is to push an object to reach a desired goal while avoiding
hazards and vases in the environment. It will get rewards when the object
successfully pushes the yellow object to the goal. The agent does not fully
control the object’s movement, making the task more difficult to handle. And
there are also some line-of-sight obstruction problems in vision-only
situations.

###  C.3 Simulation Dataset

We collect simulation data from three policies: random policy, safe policy and
unsafe policy. Each policy collected 200 trajectories for each task, which
were then combined into the final dataset. This approach ensures that our
dataset uniformly includes all types of data: low cost with low return, low
cost with moderate return, high cost with low return, and high cost with high
return. Thus, it can make a better trade-off in training different offline
policies. The safe policy is trained by SafeDreamer[8] and the unsafe policy
is trained by DreamerV3[5] for 1.5M steps on each task. It shows that unsafe
policies tend to collect data with higher returns but significant cost
uncertainty in these scenarios.

Figure 7: The dataset distribution on five simulation tasks. We plot the
reward return and the cost return of every data trajectory.

##  Appendix D Baselines

Figure 8: Online experimental results. Comparing FOSP to baselines across five
image-based safety tasks in online fine-tuning. The results for all three
algorithms are obtained after fine-tuning for 750,000 steps. As we can see,
FOSP can maintain safety and achieve better performance during fine-tuning.

#### DreamerV3

DreamerV3[5] is a model-based reinforcement learning method that outperforms
specialized methods across over 150 diverse tasks. It has a stable performance
without adjusting hyperparameters in exploring farsighted strategies from
pixels and sparse rewards in an open world. However, it overlooks the safety
considerations in the environment, which brings high costs in safety-critical
tasks.

It uses RSSM as the world model, and the loss function is similar to Eq.(D)
but lacks the cost head.

$\displaystyle\mathcal{L}^{\text{model}}(\theta)=$
$\displaystyle\operatorname*{\mathbb{E}}_{\tau\sim\mathcal{D}}\Big{[}\sum_{t=1}^{T}-\ln
p_{\theta}({\bm{x}}_{t}\mid{\bm{s}}_{t})-\ln
p_{\theta}({\bm{r}}_{t}\mid{\bm{s}}_{t})+\mathbb{D}_{KL}[sg(q_{\theta}({\bm{z}}_{t}|{\bm{h}}_{t},{\bm{x}}_{t}))||p_{\theta}^{i_{t}}({\bm{z}}_{t}|{\bm{h}}_{t})]+$
$\displaystyle\beta\mathbb{D}_{KL}[q_{\theta}({\bm{z}}_{t}|{\bm{h}}_{t},{\bm{x}}_{t})||sg(p_{\theta}^{i_{t}}({\bm{z}}_{t}|{\bm{h}}_{t}))]\Big{]}.$

And a simple actor-critic framework is used in its decision-making part.

$\displaystyle\text{Actor:
}{\bm{a}}_{t}\sim\pi_{\psi}({\bm{a}}_{t}|{\bm{s}}_{t})\;\;\;\text{Critic:
}V_{\phi}(R_{t}|{\bm{s}}_{t})=\operatorname*{\mathbb{E}}\Big{[}\sum_{t}\gamma^{t}{\bm{r}}_{t}\Big{]}$

$\displaystyle\text{Actor loss:
}L(\psi)=-\sum_{t}((R_{t}-V_{\phi}({\bm{s}}_{t}))/\max(1,S))\log\pi_{\psi}({\bm{a}}_{t}|{\bm{s}}_{t})+\eta
H\Big{[}\pi_{\psi}({\bm{a}}_{t}|{\bm{s}}_{t})\Big{]}$

$\displaystyle\text{Critic loss: }L(\phi)=-\sum_{t}\log
p_{\phi}(R_{t}|{\bm{s}}_{t})$

In the offline pretraining phase, we removed the online planning component and
used it only for online fine-tuning. To achieve robust performance in the
offline phase, we train a latent dynamics model ensemble and use the
uncertainty estimation approach[14]:

$\displaystyle\hat{{\bm{r}}}_{\theta}({\bm{s}}_{t},{\bm{a}}_{t})={\bm{r}}_{\theta}({\bm{s}})-\alpha\cdot\text{std}(\\{\log(p_{\theta}^{i}({\bm{z}}|{\bm{h}}))\\}_{i=1}^{N}),$

where ${\bm{r}}_{\theta}({\bm{s}})$ is the reward predicted by the reward
encoder.

#### SafeDreamer

SafeDreamer[8] incorporates Lagrangian-based methods into world model planning
processes and achieves nearly zero-cost performance on various tasks. It
performs well in high-dimensional vision-only input safety-critical tasks,
surpassing the prior works[7, 40] and balancing performance and safety. The
framework uses the same world models based on DreamerV3[5]. We use the BSRP-
Lag version of the model, which utilizes the Lagrangian method in background
safety-reward planning that avoids online planning. And this version of
SafeDreamer also achieves the best performance of the three versions.

It adds a cost head into world models like our method and uses the same loss
Eq.(D) in model training. It constructs a cost critic
$V_{\phi}^{c}(R_{t}|{\bm{s}}_{t})$ like $V_{\phi}(R_{t}|{\bm{s}}_{t})$ and
utilize Augmented Lagrangian method to update the actor:

$\displaystyle\mathcal{L}(\theta)=-\sum_{t=1}^{T}R^{\lambda}(s_{t})+\eta\mathrm{H}\left[\pi_{\theta}\left(a_{t}\mid
s_{t}\right)\right]-\Psi\left(C^{\lambda}(s_{t}),\lambda_{p}^{k},\mu^{k}\right)$

$\displaystyle\Psi\left(C^{\lambda}(s_{t}),\lambda_{p}^{k},\mu^{k}\right),\lambda_{p}^{k+1}=\begin{cases}\lambda_{p}^{k}\Delta+\frac{\mu^{k}}{2}\Delta^{2},\lambda_{p}^{k}+\mu^{k}\Delta&\text{
if }\lambda_{p}^{k}+\mu^{k}\Delta\geq 0\\\
-\frac{\left(\lambda_{p}^{k}\right)^{2}}{2\mu^{k}},0&\text{ otherwise
}\end{cases}$

where $\delta=C^{\lambda}(s_{t})-b$.

##  Appendix E Hyperparameters

The experiments for FOSP were conducted in a Python 3.10 environment with JAX
0.4.26. Our setup included CUDA version 12.1, running on Ubuntu 20.04. The
hardware used comprised of four GeForce RTX 4090 GPUs and an Intel(R) Xeon(R)
Platinum 8358P CPU @ 2.60GHz CPU.

Table 3: Hyperparameters for FOSP

Module | Name | Symbol | Value
---|---|---|---
World Model | Number of latent | $N_{l}$ | 48
Classes per latent | $C_{l}$ | 48
Batch size | $B$ | 64
Batch length | $T$ | 16
Learning rate | $l_{wm}$ | $10^{-4}$
Coefficient of KL-divergence | $\beta$ | 0.1
Generation horizon | $H$ | 15
Augmented Lagrangian | Penalty term | $\nu$ | $5^{-9}$
Initial Penalty multiplier | $\mu^{0}$ | $1^{-6}$
Initial Lagrangian multiplier | $\lambda_{p}^{0}$ | 0.01
Actor Critic | Discount horizon | $\gamma$ | 0.997
Reward lambda | $\lambda_{r}$ | 0.95
Cost lambda | $\lambda_{c}$ | 0.95
Expectile | $\tau$ | $0.8$
AWR temperature |  $\beta_{1}$, $\beta_{2}$ | 10
REF discount | $\gamma_{u}$ | 0.99
PEX temperature | $\alpha$ | 10
Actor entropy regularize | $\eta$ | $3\cdot 10^{-4}$
Learning rate | $l_{ac}$ | $3\cdot 10^{-5}$
General | Number of MLP layers |  | 5
Number of MLP layer units |  | 512
Action repeat |  | 4

##  Appendix F Additional Experimental Results

We present the training curves of FOSP during online fine-tuning in simulation
experiments in Figure 8. The results show that with minimal fine-tuning, FOSP
achieves better performance, outperforming SafeDreamer. Although it does not
match the performance of DreamerV3, FOSP achieves nearly zero-cost of the
agent throughout the entire process.

##  Appendix G Limitations and Future Work

#### Limitations

Despite achieving good results in many simulated environment experiments, the
proposed framework still has some limitations. During the fine-tuning process
with new trajectory data, the model struggles to update and converge
efficiently with zero constraint violations. Also, we find that the model
performs poorly in tasks with occluded vision. In experiments with real
robots, our test results remain unstable and are sensitive to hyperparameter
adjustments during training. This indicates the need for significant effort in
tuning the model’s hyperparameters to achieve better performance. In transfer
experiments, we find that when the positions of obstacles and target objects
are random, the model has difficulty in converging with limited data despite
fine-tuning. This is why we constrain the placement range of obstacles and
target objects during our experiments, which also highlights the model’s high
dependency on the dataset distribution.

#### Future Work

The possible future work will concentrate on three main fronts: improving
safety considerations, assessing the algorithm across diverse tasks, and
fortifying its robustness in real-world unseen tasks. To further decrease the
violations, the safety component of our method can be improved during the
offline-online stage. The SafeReach task is a preliminary experiment in the
real world and it is possible to expand our method to general robot tasks such
as grasping, pulling, and pushing. Moreover, the real-world task performance
of our method can be further improved. To address the issue of inaccurate
position determination from a single perspective and line of sight occlusion,
the multi-view RL shows its efficiency in catching better features from
images. What’s more, increasing the dataset capacity and uniforming the
dataset distribution can probably enhance the robustness of the performance
and expand it to more complex scenarios.