# Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement
Learning

Tianle Zhang JD Explore Academy, China  Jiayi Guan JD Explore Academy, China
Tongji University, China  Lin Zhao JD Explore Academy, China  Beijing
Institute of Technology, China  Yihang Li JD Explore Academy, China  Dongjiang
Li JD Explore Academy, China  Zecui Zeng JD Explore Academy, China  Lei Sun JD
Explore Academy, China  Yue Chen JD Explore Academy, China  Xuelong Wei JD
Explore Academy, China  Lusong Li JD Explore Academy, China  Xiaodong He JD
Explore Academy, China

###### Abstract

Offline reinforcement learning (RL) aims to learn optimal policies from
previously collected datasets. Recently, due to their powerful
representational capabilities, diffusion models have shown significant
potential as policy models for offline RL issues. However, previous offline RL
algorithms based on diffusion policies generally adopt weighted regression to
improve the policy. This approach optimizes the policy only using the
collected actions and is sensitive to Q-values, which limits the potential for
further performance enhancement. To this end, we propose a novel preferred-
action-optimized diffusion policy for offline RL. In particular, an expressive
conditional diffusion model is utilized to represent the diverse distribution
of a behavior policy. Meanwhile, based on the diffusion model, preferred
actions within the same behavior distribution are automatically generated
through the critic function. Moreover, an anti-noise preference optimization
is designed to achieve policy improvement by using the preferred actions,
which can adapt to noise-preferred actions for stable training. Extensive
experiments demonstrate that the proposed method provides competitive or
superior performance compared to previous state-of-the-art offline RL methods,
particularly in sparse reward tasks such as Kitchen and AntMaze. Additionally,
we empirically prove the effectiveness of anti-noise preference optimization.

##  1 Introduction

Offline reinforcement learning (RL) seeks to learn effective policies from
previous experiences without interacting with the environment[1]. Avoiding
costly or risky online interactions makes offline RL appealing for numerous
real-world applications. Extending online RL methods to the offline setting
primarily faces the issue of distribution shift [2]. Existing offline RL
methods predominantly concentrate on addressing this issue by employing
conservative updates for Q-functions [3, 4, 5], constraining a policy to stay
close to the behavior policy via weighted regression (WR) [6, 7], or
integrating the two approaches [8, 9]. However, most offline RL methods
usually parameterize a policy as an unimodal Gaussian model with the learned
mean and variance. This scheme may not be feasible when the collected data
distribution is complex and diverse. Moreover, the collected behaviors can be
highly diverse in the real world and present strong multi-modalities [10].
Therefore, more expressive policy models are urgently needed.

Fortunately, diffusion models [11, 12] are usually used to model diverse or
multimodal distributions due to their strong expressiveness, and have achieved
the latest benchmark in image generation tasks [13, 14]. Recently, the offline
RL works using the diffusion model as a policy model have shown great
potential [15] and made good progress [10]. Despite the impressive improvement
that these works have achieved, they still adopt the WR method [6] for policy
improvement. However, the WR method has two critical drawbacks preventing the
performance improvement of diffusion policies. _First_ , WR aims to help
prevent querying out-of-sample actions, which can result in policy performance
being greatly limited by the collected data. In most offline tasks, the
collected data is only a small part of the whole state-action space. Thus, WR
has a performance upper constraint and cannot greatly improve the performance
of policies. _Second_ , WR typically uses Q-value to calculate the updated
weights of policies and is sensitive to Q-value. In offline RL, the estimation
of the Q-value generally exits error due to the out-of-distribution (OOD)
problem [16]. Consequently, WR sensitive to Q-value has unstable training,
which can degrade performance.

Recently, RL with preference-based reward learning has shown great potential
for further improving the performance of large language models [17].
Meanwhile, direct preference optimization (DPO) [18] deduces that a simple
classification loss using preference models can be directly used to solve the
RL problem for policy improvement. Inspired by this, we propose a novel
Preferred-Action-Optimized Diffusion Policy (PAO-DP) for offline RL. PAO-DP
focuses on utilizing preferred actions to optimize the diffusion policy via a
preference model (e.g., Bradley-Terry [19]), instead of WR. To our knowledge,
PAO-DP is the first attempt to integrate preference optimization with
diffusion models for offline RL. Specifically, PAO-DP mainly has the following
characteristics: 1) The diverse distribution of the behavior policy is
represented using a conditional diffusion model by the behavior cloning
approach. 2) Preferred actions are automatically generated by the critic
function based on the diffusion model, rather than manually annotating. All
preferred actions are sampled from the same distribution of the behavior
policy for accuracy evaluation. 3) An anti-noise preference optimization is
designed to improve the diffusion policy further by using preferred actions,
while also handling noise-preferred actions for stable training. We test PAO-
DP on the popular D4RL benchmarking [20]. The experiment results demonstrate
the proposed method achieves competitive or superior performance compared with
prior offline RL methods, especially in sparse-reward tasks such as Kitchen
and AntMaze.

##  2 Preliminaries

###  2.1 Offline Reinforcement Learning

Offline RL Settings. A decision-making problem is usually modeled as a Markov
decision process (MDP). The MDP is defined as a tuple [21]: ($\mathcal{S}$,
$\mathcal{A}$, $\mathcal{P}$, $r$, $\gamma$), where $\mathcal{S}$ and
$\mathcal{A}$ denote the state and action spaces respectively,
$\mathcal{P}(\bm{s}^{\prime}|\bm{s},\bm{a}):\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$
denotes the transition probability from state $\bm{s}$ to the next state
$\bm{s}^{\prime}$ after taking the action $\bm{a}$,
$r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ represents the reward
function, $\gamma\in[0,1)$ is the discount factor. The goal is to learn a
policy $\pi(\bm{a}|\bm{s})$ that maximizes the cumulative discounted reward
$\sum_{t=0}^{\infty}\gamma^{t}r(\bm{s}_{t},\bm{a}_{t})$. In an offline
setting, the objective is to learn a policy that maximizes reward from a
static sample dataset $\mathcal{D}$ collected by an unknown behavioral policy
$\pi^{b}$, without further online interaction with the environment [22, 23].

Weighted Regression. In offline RL algorithms, due to the importance of
adhering to the behavior policy $\pi^{b}$, previous works [24, 6] explicitly
constraint the learned policy $\pi$ to be closed to $\pi^{b}$, while
maximizing the expected value of the Q-function [25]:

$J(\pi)=\arg\max_{\pi}\mathbb{E}_{\bm{s}}\left[\sum_{\bm{a}}\pi(\bm{a}|\bm{s})Q_{\phi}(\bm{s},\bm{a})\right]-{\eta}\mathbb{E}_{\bm{s}}\left[D_{KL}(\pi(\cdot|\bm{s})||\pi^{b}(\cdot|\bm{s}))\right].$
(1)

The first term corresponds to the optimization objective of the policy $\pi$,
in which $Q_{\phi}$ is a learned Q-function of $\pi$. The second term acts as
a regularization term to constrain the learned policy within the bounds of the
dataset $\mathcal{D}$ with $\eta$ being the coefficient. By solving the Eq.
(1) for the extreme points of the policy $\pi$, the closed-form solution for
the optimal policy $\pi^{*}$ can be obtained [6]:

$\pi^{*}=\frac{1}{Z(\bm{s})}\pi^{b}(\bm{a}|\bm{s})\exp(\frac{1}{\eta}Q_{\phi}(\bm{s},\bm{a})),\\\
$ (2)

where $Z(\bm{s})$ is the partition function. Eq. (2) can be regarded as a
policy improvement step. Due to the difficulty in modeling the behavior policy
$\pi^{b}$ and the problem of inaccurate estimation of the Q-function, it is
challenging to sample and obtain the optimal action directly. The existing
methods [24, 8, 22, 25] typically project the optimal policy $\pi^{*}$ onto a
parameterized policy $\pi_{\theta}$ to bypass this issue:

$\begin{split}\mathop{\arg\min}\limits_{\bm{\theta}}\;\mathbb{E}_{\bm{s}\sim\mathcal{D}}[D_{KL}(\pi^{*}(\cdot|\bm{s})||\pi_{\theta}(\cdot|\bm{s}))]=\mathop{\arg\max}\limits_{\bm{\theta}}\;\mathbb{E}_{(\bm{s},\bm{a})\sim\mathcal{D}}\left[\frac{1}{Z(\bm{s})}\log\pi_{\theta}(\bm{a}|\bm{s})\exp(\frac{1}{\eta}Q_{\phi}(\bm{s},\bm{a}))\right].\end{split}$
(3)

Such a method is commonly known as weighted regression, where
$\exp(\frac{1}{\eta}Q_{\phi}(\bm{s},\bm{a}))$ serves as the regression
weights.

###  2.2 Diffusion Probabilistic Model

Diffusion-based generative models [26, 11] mainly include forward and reverse
diffusion processes. In particular, suppose there is a real data distribution
$q(x)$, from which a sample $x^{0}\sim q(x)$ is drawn. The forward diffusion
process followed a Markov chain gradually adds Gaussian noise to the sample
through $K$ incremental steps, yielding a sequence of noisy samples
$x^{1},...,x^{K}$ controlled by a pre-defined variance schedule
$\beta^{1},...,\beta^{K}$. The forward process is expressed as:

$q(\bm{x}^{1:K}|\bm{x}^{0})=\prod_{k=1}^{K}q(\bm{x}^{k}|\bm{x}^{k-1}),\quad
q(\bm{x}^{k}|\bm{x}^{k-1})=\mathcal{N}(\bm{x}^{k};\sqrt{1-\beta^{k}}\bm{x}^{k-1},\beta^{k}\bm{I}).$
(4)

In the reverse diffusion processes, diffusion models mainly learn a
conditional distribution $p_{\theta}(\bm{x}^{k-1}|x^{k})$ for generating new
samples. The reverse process is represented as:

$p_{\theta}(\bm{x}^{0:K})=p(\bm{x}^{K})\prod_{k=1}^{K}p_{\theta}(\bm{x}^{k-1}|\bm{x}^{k}),\quad
p_{\theta}(\bm{x}^{k-1}|\bm{x}^{k})=\mathcal{N}(\bm{x}^{k-1};\bm{\mu}_{\theta}(\bm{x}^{k},k),\bm{\Sigma}_{\theta}(\bm{x}^{k},k)),$
(5)

where $p(\bm{x}^{K})=\mathcal{N}(\bm{0},\bm{I})$. The diffusion training is
executed by maximizing the evidence lower bound (ELBO):
$\mathbb{E}_{q}[\ln{p_{\theta}(\bm{x}^{0:K})}-\ln q(\bm{x}^{1:K}|\bm{x}^{0})]$
[27, 10].

##  3 Method

In this section, we detail the overall design of the proposed preferred-action
optimized diffusion policy (PAO-DP). Firstly, we formulate an RL behavior
policy using a conditional diffusion model. Then, the preferred-action
automatic generation module is presented. The module aims to utilize the
behavior policy and critic function to generate preferred actions. Finally,
the policy anti-noise preference optimization is given to achieve policy
improvement by using the preferred actions.

###  3.1 Conditional Diffusion Policy

Generally, the reverse process of a conditional diffusion model is used as an
RL parametric policy [15, 10]:

$\pi^{b}_{\theta}(\bm{a}_{t}|\bm{s}_{t})=p_{\theta}(\bm{a}^{0:K}_{t}|\bm{s}_{t})=p(\bm{a}^{k}_{t})\prod_{k=1}^{K}p_{\theta}(\bm{a}_{t}^{k-1}|\bm{a}_{t}^{k},\bm{s}_{t}),\\\
$ (6)

where $k\in\\{1,...,K\\}$ denotes the diffusion timestep, $t\in\\{1,...,T\\}$
represents the trajectory timestep,
$\bm{a}^{K}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$. Herein, the end sample
$\bm{a}^{0}$ of the reverse chain is the action for the RL policy.
$p_{\theta}(\bm{a}_{t}^{k-1}|\bm{a}_{t}^{k},\bm{s}_{t})$ can usually be
modeled as a Gaussian distribution. Following the denoising diffusion
probabilistic model (DDPM) [11],
$p_{\theta}(\bm{a}_{t}^{k-1}|\bm{a}_{t}^{k},\bm{s}_{t})$ is parameterized as a
noise prediction model with a fixed covariance matrix:
$\mathbf{\Sigma}_{\theta}(\bm{a}^{k}_{t},k;\bm{s}_{t})=\eta^{k}\mathbf{I}$ and
mean built as:

$\mathbf{\mu}_{\theta}(\bm{a}_{t}^{k},k;\bm{s}_{t})=\frac{1}{\sqrt{\alpha_{k}}}\left(\bm{a}^{k}_{t}-\frac{\beta_{k}}{\sqrt{1-\bar{\alpha}_{k}}}\bm{\epsilon}_{\theta}(\bm{a}^{k}_{t},k;\bm{s}_{t})\right),\\\
$ (7)

where $\alpha^{k}=1-\beta^{k}$, $\bar{\alpha}^{k}=\prod_{i=1}^{k}\alpha^{i}$,
$\bm{\epsilon}_{\theta}$ is a parametric noise model. In DDPM, we can obtain
an action through the following sampling process:

$\bm{a}^{k-1}_{t}=\frac{\bm{a}^{k}_{t}}{\sqrt{\alpha^{k}}}-\frac{\beta^{k}}{\sqrt{\alpha(1-\bar{\alpha}^{k})}}\bm{\epsilon}(\bm{a}^{k}_{t},k;\bm{s}_{t})+\sqrt{\beta_{i}}\bm{\epsilon},\quad\bm{\epsilon}\sim\mathcal{N}(0,I),\quad
for\,\>k=K,...,1.$ (8)

Similar to DDPM, the objective proposed by [11] can be simplified to train the
$\bm{\epsilon}$ model by:

$\mathcal{L}_{d}(\theta)=\mathbb{E}_{k\sim\mathcal{U},\bm{\epsilon}\sim\mathcal{N}(0,I),(\bm{s}_{t},\bm{a}_{t}^{0})\sim\mathcal{D}}\left[||\bm{\epsilon}-\bm{\epsilon}_{\theta}\left(\sqrt{\bar{\alpha}_{k}}\bm{a}_{t}^{0}+\sqrt{1-\bar{\alpha}_{k}}\bm{\epsilon},k;\bm{s}_{t}\right)||^{2}\right],$
(9)

where $\mathcal{U}$ is a uniform distribution over $\\{1,...,K\\}$ and
$\mathcal{D}$ denotes the offline dataset collected by a behavior policy
$\pi^{b}$. The diffusion loss $\mathcal{L}_{d}(\theta)$ aims to learn the
behavior policy through the conditional diffusion model with
$\bm{\epsilon}_{\theta}$, and it is just a loss of behavior cloning. Herein,
$\pi^{b}_{\bm{\epsilon}_{\theta}}(\bm{a}_{t}|\bm{s}_{t})$ is used to represent
the learned diffusion behavior policy.

###  3.2 Preferred-Action Automatic Generation

Based on the learned behavior policy, we further use preference optimization
to achieve policy improvement, instead of weighted regression. One of the keys
to preference optimization is to obtain preferred actions. In the field of
preference learning [28], preference data is generally provided in advance.
However, this requires a lot of manpower and material resources. To avoid this
issue, we design an automatic acquisition approach that relies on the
Q-function. Specifically, we need to automatically generate preferred-action
samples, i.e.,
$\\{\bm{s}_{t},\bm{a}_{t},\hat{\bm{a}}_{t},\Gamma_{Q(\bm{s}_{t},\bm{a}_{t})>Q(\bm{s}_{t},\hat{\bm{a}}_{t})}\\}\in\mathcal{D}_{p}$,
where $\bm{a}_{t}$ denotes the behavior action corresponding to the state
$\bm{s}_{t}$ in the offline data $\mathcal{D}$, $\hat{\bm{a}}_{t}$ is another
action obtained through the following sampling method,
$\Gamma_{Q(\bm{s}_{t},\bm{a}_{t})>Q(\bm{s}_{t},\hat{\bm{a}}_{t})}$ is a
symbolic function used to determine which action has a larger Q-value:

$\displaystyle\begin{split}\Gamma_{Q(\bm{s}_{t},\bm{a}_{t})>Q(\bm{s}_{t},\hat{\bm{a}}_{t})}=\left\\{\begin{array}[]{rr}1,&Q(\bm{s}_{t},\bm{a}_{t})>Q(\bm{s}_{t},\hat{\bm{a}}_{t})\\\
-1,&otherwise\end{array}\right.\end{split}.$ (10)

Meanwhile, the Q-function $Q_{\phi}(\bm{s}_{t},\bm{a}_{t})$ parameterized by
$\phi$ can be learned by utilizing the existing Q-learning framework, e.g.,
TD3 [29] and implicit Q-learning (IQL) [5]

Preferred-Action Sampling. Considering the OOD problem in Q-value estimation,
we hope another action $\hat{\bm{a}}_{t}$ can be obtained from the
distribution of the behavior policy. Due to the learned policy
$\pi^{b}_{\bm{\epsilon}_{\theta}}$ modeling the distribution of the behavior
policy, $\hat{\bm{a}}_{t}$ can be sampled from
$\pi^{b}_{\bm{\epsilon}_{\theta}}(\hat{\bm{a}}_{t}|\bm{s}_{t})$ directly.
Generally, higher-quality actions are beneficial for the improvement of
policies, especially in preference learning. This further motivates us to
sample the action $\hat{\bm{a}}_{t}$ from the optimal policy $\pi^{*}$.

Since the exact and derivable density function of $\pi^{*}$ cannot be
determined, it is difficult to directly draw samples from $\pi^{*}$. Drawing
on insights from [25] and according to Eq. (2), the optimal policy $\pi^{*}$
can be further expressed as:

$\pi^{*}(\bm{a}_{t}|\bm{s}_{t})\propto\pi^{b}_{\bm{\epsilon}_{\theta}}(\bm{a}_{t}|\bm{s}_{t})\exp(\eta
Q_{\phi}(\bm{s}_{t},\bm{a}_{t})),\\\ $ (11)

Based on this formula, we use an importance sampling technique to sample
actions. In particular, for each state $\bm{s}_{t}$, we firstly draw $N$
actions from the learned behavior policy $\pi^{b}_{\bm{\epsilon}_{\theta}}$.
Then, these actions are evaluated with the learned Q-function $Q_{\phi}$.
Finally, an action is drawn from $N$ actions with $\exp(\eta
Q_{\phi}(\hat{\bm{a}}_{t},\bm{s}_{t})$ being the sampling weights [25].
Additionally, we can use greedy sampling to obtain $\hat{\bm{a}}_{t}$ from $N$
actions.

###  3.3 Policy Anti-Noise Preference Optimization

After obtaining the preferred action samples, the behavior policy
$\pi^{b}_{\bm{\epsilon}_{\theta}}$ can be further improved stably by anti-
noise preference optimization. Specifically, firstly, according to Eq. (2), we
can obtain an expression for Q-function:

$Q_{\phi}(\bm{s}_{t},\bm{a}_{t}))=\eta\log\frac{\pi^{*}(\bm{a}_{t}|\bm{s}_{t})}{\pi^{b}_{\bm{\epsilon}_{\theta}}(\bm{a}_{t}|\bm{s}_{t})}+\eta\log
Z(s).\\\ $ (12)

Then, based on
$\\{\bm{s}_{t},\bm{a}_{t},\hat{\bm{a}}_{t},\Gamma_{Q(\bm{s}_{t},\bm{a}_{t})>Q(\bm{s}_{t},\hat{\bm{a}}_{t})}\\}$,
the popular Bradley-Terry (BT) [19, 18] model is used to model the action
preference distribution $p^{*}$:

$p^{*}(\hat{\bm{a}}_{t}\succ\bm{a}_{t}|\bm{s}_{t})=\frac{\exp(Q_{\phi}(\bm{s}_{t},\hat{\bm{a}}_{t}))}{\exp(Q_{\phi}(\bm{s}_{t},\hat{\bm{a}}_{t}))+\exp(Q_{\phi}(\bm{s}_{t},\bm{a}_{t}))}.\\\
$ (13)

Putting the expression of Q in Eq. (12) into Eq. (13), we can obtain the
action preference probability in terms of only the optimal policy $\pi^{*}$
and the behavior policy $\pi^{b}_{\bm{\epsilon}_{\theta}}$, expressed as [18]:

$p^{*}(\hat{\bm{a}}_{t}\succ\bm{a}_{t}|\bm{s}_{t})=\frac{1}{1+\exp\left(\eta\log\frac{\pi^{*}(\bm{a}_{t}|\bm{s}_{t})}{\pi^{b}_{\bm{\epsilon}_{\theta}}(\bm{a}_{t}|\bm{s}_{t})}-\eta\log\frac{\pi^{*}(\hat{\bm{a}}_{t}|\bm{s}_{t})}{\pi^{b}_{\bm{\epsilon}_{\theta}}(\hat{\bm{a}}_{t}|\bm{s}_{t})}\right)}.\\\
$ (14)

The derivation is in the Appendix B.2. Herein, we parameterize the optimal
policy by $\psi$, represented as the surrogate optimal policy $\pi_{\psi}$,
where the structure of $\pi_{\psi}$ is the same as
$\pi^{b}_{\bm{\epsilon}_{\theta}}$, and $\pi_{\psi}$ is further optimized
based on $\pi^{b}_{\bm{\epsilon}_{\theta}}$ for the policy improvement.
Subsequently, we can formulate a maximum likelihood objective for
$\pi_{\psi}$. The policy improvement objective can be expressed as:

$\\!\\!\mathcal{L}_{imp}(\psi;\theta)\\!\\!=\\!\\!-\mathbb{E}_{(\bm{s}_{t},\bm{a}_{t},\hat{\bm{a}}_{t},\Gamma)\in\mathcal{D}_{p}}\left[\log\sigma\left((\eta*\Gamma)\log\frac{\pi_{\psi}(\hat{\bm{a}}_{t}|\bm{s}_{t})}{\pi^{b}_{\bm{\epsilon}_{\theta}}(\hat{\bm{a}}_{t}|\bm{s}_{t})}-(\eta*\Gamma)\log\frac{\pi_{\psi}(\bm{a}_{t}|\bm{s}_{t})}{\pi^{b}_{\bm{\epsilon}_{\theta}}(\bm{a}_{t}|\bm{s}_{t})}\right)\right],\\\
$ (15)

where $\sigma$ is the logistic function. Furthermore, considering the
imprecision possibility of estimating Q-values in offline RL, such
inaccuracies may introduce noises in the preference action samples. This may
lead to instability in training during the policy improvement process. To
alleviate this issue, we further establish an anti-noise optimization
objective. Specifically, assume that the preferred action label may be flipped
with a small probability $\lambda\in(0,0.5)$ and $\hat{\bm{a}}_{t}$ is
preferred $\bm{a}_{t}$, we can obtain a conservative preferred-action
probability $p^{*}(\hat{\bm{a}}_{t}\succ\bm{a}_{t}|\bm{s}_{t})=1-\lambda$
[30]. Hence, we can calculate a cross-entropy loss as the anti-noise
optimization objective [31]:

$\mathcal{L}_{anti}(\psi;\theta)=(1-\lambda)\mathcal{L}_{imp}(\psi;\theta)+\lambda\mathcal{L}_{imp}^{-\Gamma}(\psi;\theta),$
(16)

where $\mathcal{L}_{imp}^{-\Gamma}(\psi;\theta)$ represents the objective
whose preference actions have been flipped.

In our paper, to train more efficiently, we adopt a strategy of simultaneously
training behavior policy $\pi^{b}_{\bm{\epsilon}_{\theta}}$ and surrogate
optimal policy $\pi_{\psi}$, rather than dividing the training into two
stages. Hence, the loss function of the surrogate optimal policy is a linear
combination of policy cloning and policy improvement, represented as
$\mathcal{L}(\psi)=\mathcal{L}_{d}(\psi)+\xi\mathcal{L}_{anti}(\psi;\theta)$,
where $\xi$ is a hyperparameter to provide a weight for the policy improvement
part. This parameter can also be used to measure the effect of the policy
improvement. The overall algorithm is given in Algorithm 1

Input: Dataset
$\mathcal{D}=\\{(\bm{s}_{t},\bm{a}_{t},r_{t},\bm{s}_{t+1})_{i=0}^{n}\\}$

Output: Surrogate optimal policy $\pi_{\psi}$, Behavior policy
$\pi^{b}_{\bm{\epsilon}_{\theta}}$

1 for  _each batch_ do

2 Sample a batch of transitions $(\bm{s}_{t},\bm{a}_{t},r_{t},\bm{s}_{t+1})$
from the buffer $\mathcal{D}$

3 # Behavior policy learning

4 Update $\bm{\epsilon}_{\theta}$ via the Eq. (9)

5 # Critic learning

6 Update $Q_{\phi}$ via the existing Q-learning method, e.g., IQL

7 # Preferred-action automatic generation

8 Sample $N$ actions in $\bm{s}_{t}$ by the behavior policy
$\pi^{b}_{\bm{\epsilon}_{\theta}}$ using the Eq. (8)

9 Select an action from $N$ actions via the importance sampling using Eq. (11)

10 Generate preferred-action sample
$\\{\bm{s}_{t},\bm{a}_{t},\hat{\bm{a}}_{t},\Gamma_{Q(\bm{s}_{t},\bm{a}_{t})>Q(\bm{s}_{t},\hat{\bm{a}}_{t})}\\}$
via Eq. (10)

11 # Policy improvement

12 Calculate the anti-noise preference loss $\mathcal{L}_{anti}(\psi;\theta)$
via Eq. (16)

13 Calculate the diffusion loss $\mathcal{L}_{d}(\psi)$ via Eq. (9)

14 Update the surrogate optimal policy $\pi_{\psi}$ via the total loss
function:
$\mathcal{L}(\psi)=\mathcal{L}_{d}(\psi)+\xi\mathcal{L}_{anti}(\psi;\theta)$

15

Algorithm 1 PAO-DP Algorithm

##  4 Experiments

We conduct extensive experiments on the popular D4RL benchmark [20] to
validate the effectiveness of the proposed method, especially in sparse-reward
tasks. Additionally, we performed ablation studies to dissect the
contributions of the main components of PAO-DP. Furthermore, we carried out a
parameter sensitivity analysis across different methods to assess their
robustness and performance.

###  4.1 Experimental Settings

Baselines. The proposed method is evaluated in four different domains in D4RL,
including Kitchen, AntMaze, Adroit, and Gym-locomation. In our experiments,
Kitchen and AntMaze are all composed of sparse-reward tasks, while Adroit and
Gym-locomation are not. For each domain, we consider different classes of
baselines for providing comprehensive evaluation. For policy regularization-
based methods, the simple method is the canonical behavior cloning (BC)
baseline. AWAC [6] and CRR [8] achieve policy improvement by adding advantage-
based weights in the policy loss function. For Q-value constraint methods, CQL
[3] and IQL [5] adopt conservative Q-function updates or expectile regression
for constraining the policy evaluation process. For sequence modeling
approaches, we consider the Decision Transformer (DT) [32] baseline. For
diffusion policies, the diffusion Q-learning (DIFF-QL) [15] baseline first
represents the policy as a diffusion model, and couples the behavior cloning
and policy improvement using WR. Based on DIFF-QL, the efficient diffusion
policy (EDP) [10] method approximately constructs actions from corrupted ones
for realizing efficient training. CRR+EDP and IQL+EDP respectively adopt CRR
and IQL based on EDP for policy improvement. CRR+EDP focuses on advantage-
based weight regression, while IQL+EDP uses Q-based weight regression. The
SfBC [25] baseline decouples the learned policy into a diffusion behavior
model and an action evaluation model, without a policy improvement process.

Experimental Details. In our experiments, consistent with the EDP framework,
we utilize a critic function network architecture comprising a three-layer
MLP, employing the Mish activation function [33]. This architecture is
uniformly applied across all tasks. To obtain a more accurate estimate of the
Q-value from offline data, the critic function is updated through the IQL
approach using expectile regression. The noise prediction network
$\bm{\epsilon}_{\theta}(\bm{a}^{k}_{t},k;\bm{s}_{t})$ of the diffusion policy
initially employs sinusoidal embedding [34] to encode timestep $k$,
subsequently concatenating it with the noisy action $\bm{a}^{k}_{t}$ and the
conditional state $\bm{s}_{t}$. Similar to EDP, the policy is trained for 2000
epochs on the Gym-locomation domain and 1000 epochs on the other domains.
Every epoch is made up of 1000 policy updates with a batch size of 256. The
third-order version of DPM-Solver [35] is used to perform the denoising
process of the diffusion policy with 15 steps. For preferred-action automatic
generation, we found that greedy strategy sampling performed outstandingly in
most tasks, so we used it. The number of sampled actions is 10, i.e., $N=10$.
Throughout our paper, the results are reported by averaging 3 random seeds. In
addition, all the hyperparameters are listed in the Appendix E.

Evaluation Metrics. We evaluate the proposed PAO-DP method using two metrics:
Online Model Selection (OMS) and Running Average at Training (RAT) [10]. OMS,
introduced by Diffusion-QL [15], selects the best-performing model during
training, indicating the algorithm’s potential. In contrast, RAT [10]
calculates the running average of evaluation performance over ten consecutive
checkpoints and reports the final score, providing a more stable and realistic
assessment of an algorithm’s performance throughout training. Besides, our
paper uses the average normalized score of D4RL to evaluate different methods.

###  4.2 Evaluation Results

We compare our proposed PAO-DP method with the baseline methods across four
task domains, with the results detailed in Table 1. The domain-specific
analyses are as follows:

Results for Kitchen Domain: The Kitchen domain presents a challenging long-
term task requiring the agent to accomplish a series of sequential subtasks to
reach a goal state configuration. This domain is characterized by sparse
rewards and the complexity of achieving the final goal through intermediate
steps, making long-term value optimization crucial. The proposed PAO-DP method
significantly outperforms the baselines in the Kitchen domain across all three
datasets: Complete, Partial, and Mixed. In the Complete task, our method
achieves an impressive score of 87.2, which is notably higher than the next-
best performance of 75.5 by IQL+EDP using WR. The average performance in the
Kitchen domain shows PAO-DP at 67.5, demonstrating a substantial improvement
over all other methods. These results underscore the effectiveness of PAO-DP
in optimizing long-term value in environments with sparse rewards and complex
sequential tasks. By integrating policy improvement using preference
optimization with diffusion policies, PAO-DP significantly enhances the
agent’s performance in this challenging domain, outperforming diffusion
policies that incorporate weighted regression.

Results for AntMaze Domain: PAO-DP demonstrates superior performance in the
AntMaze domain, which is characterized by sparse rewards and challenging sub-
optimal trajectory tasks. The average performance across the AntMaze domain
for PAO-DP is 79.7, significantly higher than all other baselines. Although
CQL marginally surpasses PAO-DP in the Diverse task for AntMaze-umaze, scoring
84.0 compared to PAO-DP’s 83.7, PAO-DP consistently excels in other settings.
The robustness of PAO-DP is particularly evident in the AntMaze-medium and
AntMaze-large tasks, where other methods, such as BC, AWAC, and DT methods,
struggle to perform effectively.

Table 1: Average normalized score of the baselines and PAO-OP evaluated using the RAT metric. Dataset | Environment | BC | AWAC | CQL | IQL | DT | CRR+EDP | IQL+EDP | PAO-DP (Ours)
---|---|---|---|---|---|---|---|---|---
Complete | Kitchen | 33.8 | 39.3 | 43.8 | 62.5 | – | 73.9 | 75.5 |  $\bm{87.2}$ $\pm$ 1.05
Partial | Kitchen | 33.8 | 36.6 | 49.8 | 46.3 | – | 40.0 | 46.3 |  $\bm{53.3}$ $\pm$ 0.48
Mixed | Kitchen | 47.5 | 22.0 | 51.0 | 51.0 | – | 46.1 | 56.5 |  $\bm{62.2}$ $\pm$ 2.83
Average (FrankaKitchen) | 38.4 | 32.6 | 48.2 | 53.3 | – | 53.3 | 59.4 |  $\bm{67.5}$ $\pm$ 14.45
Dataset | Environment | BC | AWAC | CQL | IQL | DT | CRR+EDP | IQL+EDP | PAO-DP (Ours)
Default | AntMaze-umaze | 54.6 | 56.7 | 74.0 | 87.5 | 59.2 | 95.9 | 94.2 |  $\bm{97.3}$ $\pm$ 1.69
Diverse | AntMaze-umaze | 45.6 | 49.3 | $\bm{84.0}$ | 62.2 | 53.0 | 15.9 | 79.0 |  83.7 $\pm$ 1.69
Play | AntMaze-medium | 0.0 | 0.0 | 61.2 | 71.2 | 0.0 | 33.5 | 81.8 |  $\bm{85.3}$ $\pm$ 0.47
Diverse | AntMaze-medium | 0.0 | 0.7 | 53.7 | 70.0 | 0.0 | 32.7 | 82.3 |  $\bm{83.0}$ $\pm$ 2.16
Play | AntMaze-large | 0.0 | 0.0 | 15.8 | 39.6 | 0.0 | 26.0 | 42.3 |  $\bm{58.7}$ $\pm$ 1.25
Diverse | AntMaze-large | 0.0 | 1.0 | 14.9 | 47.5 | 0.0 | 58.5 | 60.6 |  $\bm{68.7}$ $\pm$ 0.94
Average (AntMaze) | 16.7 | 18.0 | 50.6 | 63.0 | 18.7 | 43.8 | 73.4 |  $\bm{79.7}$ $\pm$ 12.10
Dataset | Environment | BC | AWAC | CQL | IQL | DT | CRR+EDP | IQL+EDP | PAO-DP (Ours)
human | pen | 25.8 | 15.6 | 37.5 | 71.5 | – | 70.2 | 72.7 |  $\bm{77.6}$ $\pm$ 0.92
cloned | pen | 38.3 | 24.7 | 39.2 | 37.3 | – | 54.2 | 70.0 |  $\bm{83.6}$ $\pm$ 1.75
Average (Adroit) | 32.1 | 20.2 | 38.4 | 54.4 | – | 62.1 | 71.3 |  $\bm{80.6}$ $\pm$ 3.31
Dataset | Environment | BC | AWAC | CQL | IQL | DT | CRR+EDP | IQL+EDP | PAO-DP (Ours)
Med-Expert | HalfCheetah | 55.2 | 42.8 | $\bm{91.6}$ | 86.7 | 86.8 | 85.6 | 86.7 |  91.0 $\pm$ 0.48
Med-Expert | Hopper | 52.5 | 55.8 | 105.4 | 91.5 | $\bm{107.6}$ | 92.9 | 99.6 |  101.1 $\pm$ 1.01
Med-Expert | Walker | 107.5 | 74.5 | 108.8 | 109.6 | 108.1 | $\bm{110.1}$ | 109.0 |  109.3 $\pm$ 0.05
Medium | HalfCheetah | 42.6 | 43.5 | 44.0 | 47.4 | 42.6 | $\bm{49.2}$ | 48.1 |  47.0 $\pm$ 0.28
Medium | Hopper | 52.9 | 57.0 | 58.5 | 66.3 | 67.6 | $\bm{78.7}$ | 63.1 |  57.4 $\pm$ 0.96
Medium | Walker | 75.3 | 72.4 | 72.5 | 78.3 | 74.0 | 82.5 | $\bm{85.4}$ |  82.3 $\pm$ 1.33
Med-Replay | HalfCheetah | 36.6 | 40.5 | $\bm{45.5}$ | 44.2 | 36.6 | 43.5 | 43.8 |  43.8 $\pm$ 0.04
Med-Replay | Hopper | 18.1 | 37.2 | 95.0 | 94.7 | 82.7 | 99.0 | $\bm{99.1}$ |  90.6 $\pm$ 0.69
Med-Replay | Walker | 26.0 | 27.0 | 77.2 | 73.9 | 66.6 | 63.3 | $\bm{84.0}$ |  83.9 $\pm$ 0.39
Average (Locomotion) | 51.9 | 50.1 | 77.6 | 77.0 | 74.7 | 78.3 | $\bm{79.9}$ |  78.5 $\pm$ 22.23

Results for Adroit Domain: The Adroit domain presents unique challenges due to
its datasets being primarily collected through human behavior, resulting in a
narrow state-action region in the offline data. In the Adroit domain, PAO-DP
once again demonstrates its effectiveness. For the human dataset in the
Adroit-pen environment, PAO-DP achieves a score of 77.6, surpassing IQL+EDP’s
72.7 and CRR+EDP’s 70.2. The average performance across the Adroit domain for
PAO-DP is 80.6, highlighting its superior capability compared to other
methods. These results demonstrate that PAO-DP can maintain high performance
within the narrow operational bounds defined by the offline data. The ability
of PAO-DP to maintain expected behaviors while improving performance through
preference optimization underscores its advantage in constrained environments
over the use of weighted regression for policy improvement.

Results for Locomotion Domain: The Locomotion domain contains many standard
tasks with relatively smooth reward functions. Although PAO-DP does not always
achieve the highest scores in this domain, its performance remains competitive
and reliable. For the Med-Expert dataset in HalfCheetah, PAO-DP scores 91.0,
just shy of CQL’s 91.6. In the Med-Replay dataset, PAO-DP also performs well,
particularly in Walker with a score of 83.9, second only to IQL+EDP. The
average performance in the Locomotion domain for PAO-DP is 78.5, showcasing
its consistent effectiveness across various standard tasks. While the
Locomotion domain’s smooth reward functions contrast with the sparse rewards
found in domains like Kitchen, PAO-DP’s ability to perform well across these
diverse environments underscores its robustness and adaptability. This
versatility highlights PAO-DP’s effectiveness in handling a wide range of task
complexities.

Results for Peak Performance Evaluation:  To evaluate the peak performance
potential of PAO-DP across different tasks, we conduct experiments using the
OMS metric. As illustrated in Table 2, our method demonstrates competitive
performance compared to other methods. In the FrankaKitchen and AntMaze
domains, PAO-DP surpasses all baselines, achieving average scores of 80.6 and
96.7, respectively. This highlights its robustness in handling sparse rewards
and complex trajectory tasks, showcasing its ability to optimize long-term
value in challenging environments. For the Adroit domain, PAO-DP scores 121.6,
performing competitively against IQL+EDP (134.3) and CRR+EDP (116.9). The
relatively smooth reward functions in the Locomotion domain contrast with the
sparse rewards found in other domains, potentially diminishing PAO-DP’s
relative advantage. Despite this, PAO-DP’s performance remains competitive,
underscoring its versatility and robustness across diverse environments.

Table 2: Performance of the baselines and our method evaluated using the OMS metric. Dataset | DIFF-QL | SfBC | CRR+EDP | IQL+EDP | PAO-DP (Ours)
---|---|---|---|---|---
Average (FrankaKitchen) | 69.0 | 57.1 | 70.6 | 79.2 |  80.6 $\pm$ 13.79
Average (AntMaze) | 69.6 | 74.2 | 76.7 | 89.2 |  96.7 $\pm$ 4.71
Average (Adroit) | 65.1 | – | 116.9 | 134.3 |  121.6 $\pm$ 3.06
Average (Locomotion) | 88.0 | 75.6 | 87.3 | 84.7 |  84.1 $\pm$ 23.40

###  4.3 Ablation Studies

Comparative Analysis of Diffusion Policy Methods:  As illustrated in Figure
1-(a), the ablation study on the Kitchen-complete task underscores the
effectiveness of the main component (i.e., preference optimization for policy
improvement) of PAO-DP in comparison to other diffusion policy methods. PAO-DP
significantly outperforms the DP method, which relies solely on policy cloning
using a diffusion model, without any policy improvement. This highlights the
benefits of policy improvement using preference optimization. Furthermore,
PAO-DP demonstrates superior performance over IQL+EDP and CRR+EDP, both of
which employ weighted regression for policy improvement. The sensitivity of WR
to Q-value inaccuracies, coupled with constrained optimization using only
sample actions within the fixed dataset, limits its efficacy. In contrast,
PAO-DP’s strategy of leveraging high-quality preferred actions and
incorporating anti-noise preference optimization ensures more stable and
robust training, leading to consistently better results in the later training.

Analysis of Preferred-Action Sampling Strategies:  Figure 1-(b) shows the
ablation study on the Kitchen-complete task, comparing PAO-DP with different
sampling strategies in preferred-action generation. PAO-DP-Max, PAO-DP-Min,
and PAO-DP-mean respectively use the action with the highest, lowest, and
middle weight $\exp(\eta Q_{\phi}(s,a))$ as the preferred action. PAO-DP-Max
consistently achieves the best performance, compared with PAO-DP-Min and PAO-
DP-Mean, which highlights the advantage of prioritizing actions with the
highest weights. This study demonstrates that leveraging the important
sampling technique for action selection leads to superior and more stable
performance throughout training. More analysis results are shown in the
Appendix D.

Effect of Anti-Noise Optimization Figure 1-(c) illustrates the ablation study
on the AntMaze-large-play task, examining the effectiveness of different
values of $\lambda$ in our policy anti-noise preference optimization. As
$\lambda$ increases from 0.0 to 0.4, we observe that the performance improves
consistently throughout the training steps. The results demonstrate that
incorporating a moderate level of noise helps stabilize the training process
and enhances policy improvement, especially in low-quality datasets. More
analysis results are shown in the Appendix 4.

Figure 1:  RAT curves of the ablation methods on representative tasks. The
ordinate is the average normalized score.

###  4.4 Sensitivity Analysis to $\xi$

Figure 2 illustrates the sensitivity analysis to the hyperparameter $\xi$ on
the Kitchen-complete task for PAO-DP, IQL+EDP, and CRR+EDP. The hyperparameter
$\xi$ can reflect the effect of the policy improvement part in the entire
training. In Figure 2-(a), PAO-DP demonstrates robust performance across
different $\xi$ values, showing minimal sensitivity to changes in $\xi$.
Conversely, Figures 2-(b) and 2-(c) illustrate that both IQL+EDP and CRR+EDP
exhibit significant sensitivity to $\xi$, which indicates that the weighted
regression for policy improvement has training instability. The analysis
further verifies the robustness and effectiveness of the policy improvement
part using the preference optimization of our method.

(a) PAO-DP

(b) IQL+EDP

(c) CRR+EDP

Figure 2: RAT curves for different methods with different $\xi$ on the
Kitchen-complete task. The ordinate is the average normalized score.

##  5 Related Work

Policy Regularization and Conservative Estimation. Policy regularization and
pessimistic conservative estimation are important approaches to addressing
offline RL issues [36, 37, 38, 39]. Policy regularization methods introduce
regularization terms during the policy improvement process. For instance, the
earliest proposed BCQ [2] directly improves a policy perturbation model based
on the behavior model of CVAE [40]. Based on this, some works such as BEAR
[41] and TD3+BC [7] incorporate the loss of behavior cloning into the policy
improvement process by minimizing Maximum Mean Discrepancy (MMD) or maximizing
likelihood estimation. Additionally, BRAC [42] has explored regularization
methods in KL divergence, MMD, and Wasserstein duality. On the other hand,
some works [43, 44] introduce pessimistic conservative estimation during
policy evaluation to mitigate extrapolation errors caused by out-of-
distribution actions. For instance, CQL [3] supplements an additional penalty
term in Q-values estimation to achieve a non-pointwise lower bound estimation
of Q-values. Additionally, some works such as CRR [8] and IQL [5] have
achieved promising performance by leveraging advantage actions or within-
distribution sampling.

Diffusion and Transformer in RL. The powerful sequence modeling capability and
multi-modal representation ability of diffusion models and transformers enable
them to be widely explored in RL [45, 46, 47, 48]. Early proposals such as DT
[32] and TT [49] are based on the transformer framework to generate state
sequences or action-state sequences maximizing returns from expected
trajectory rewards. On the other hand, the multi-modal representation
capability of diffusion models enables them to exhibit significant advantages
in offline RL tasks with datasets featuring multi-modal distributions [50,
51].Diffusion-RL [15] is the first exploration of diffusion policies in
reinforcement learning policy modeling under multi-modal datasets. To guide
policy sampling towards actions with high Q-values, some works [52, 10]
construct policies as Q-value-guided conditional generative policies through
weighted regression. In addition, some approaches [25, 50] decouple policy
learning into behavior learning and action evaluation and introduce in-
distribution sampling to avoid extrapolation errors. To incorporate explicit
Q-value guidance, some work [15, 53] further subtracts the weighted
expectation of Q-values during policy training to guide diffusion policy
updates.

##  6 Conclusion

This paper proposes a novel preferred-action-optimized diffusion policy (PAO-
DP) for offline RL. PAO-DP utilizes a preference model with preferred actions
to improve the diffusion policy, instead of weighted regression. In
particular, a conditional diffusion model is used to represent the
distribution of behavior policy. Preferred actions are automatically generated
through the critic function based on the diffusion model. Moreover, an anti-
noise preference optimization using the preferred actions is designed to
improve the policy. To the best of our knowledge, this paper is the first
attempt to combine preference optimization and diffusion models for offline
RL. Experimental results on the D4RL benchmark demonstrate that PAO-DP has
competitive or superior performance compared to other RL methods. Ablation
studies and parameter sensitivity analysis further validate the effectiveness
of the key components of PAO-DP.

## References

  * Levine et al. [2020] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ , 2020.
  * Fujimoto et al. [2019] Scott Fujimoto, David Meger, and Doina Precup.  Off-policy deep reinforcement learning without exploration.  In _International conference on machine learning_ , pages 2052–2062. PMLR, 2019.
  * Kumar et al. [2020] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.  Conservative q-learning for offline reinforcement learning.  _Advances in Neural Information Processing Systems_ , 33:1179–1191, 2020.
  * Yu et al. [2021] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn.  Combo: Conservative offline model-based policy optimization.  _Advances in neural information processing systems_ , 34:28954–28967, 2021.
  * Kostrikov et al. [2021] Ilya Kostrikov, Ashvin Nair, and Sergey Levine.  Offline reinforcement learning with implicit q-learning.  In _International Conference on Learning Representations_ , 2021.
  * Nair et al. [2020] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine.  Awac: Accelerating online reinforcement learning with offline datasets.  _arXiv preprint arXiv:2006.09359_ , 2020.
  * Fujimoto and Gu [2021] Scott Fujimoto and Shixiang Shane Gu.  A minimalist approach to offline reinforcement learning.  _Advances in neural information processing systems_ , 34:20132–20145, 2021.
  * Wang et al. [2020] Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al.  Critic regularized regression.  _Advances in Neural Information Processing Systems_ , 33:7768–7778, 2020.
  * Ma et al. [2024] Xiao Ma, Bingyi Kang, Zhongwen Xu, Min Lin, and Shuicheng Yan.  Mutual information regularized offline reinforcement learning.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Kang et al. [2024] Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan.  Efficient diffusion policies for offline reinforcement learning.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Song et al. [2020] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Wang et al. [2022] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou.  Diffusion policies as an expressive policy class for offline reinforcement learning.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Ma et al. [2021] Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani.  Conservative offline distributional reinforcement learning.  _Advances in neural information processing systems_ , 34:19235–19247, 2021.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in neural information processing systems_ , 35:27730–27744, 2022.
  * Rafailov et al. [2024] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Bradley and Terry [1952] Ralph Allan Bradley and Milton E Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345, 1952.
  * Fu et al. [2020] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine.  D4rl: Datasets for deep data-driven reinforcement learning.  _arXiv preprint arXiv:2004.07219_ , 2020.
  * Puterman [2014] Martin L Puterman.  _Markov decision processes: discrete stochastic dynamic programming_.  John Wiley & Sons, 2014.
  * Chen et al. [2020] Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross.  Bail: Best-action imitation learning for batch deep reinforcement learning.  _Advances in Neural Information Processing Systems_ , 33:18353–18363, 2020.
  * Agarwal et al. [2020] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi.  An optimistic perspective on offline reinforcement learning.  In _International Conference on Machine Learning_ , pages 104–114. PMLR, 2020.
  * Peng et al. [2019] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine.  Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.  _arXiv preprint arXiv:1910.00177_ , 2019.
  * Chen et al. [2022] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu.  Offline reinforcement learning via high-fidelity generative behavior modeling.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Song and Ermon [2019] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  _Advances in neural information processing systems_ , 32, 2019.
  * Blei et al. [2017] David M Blei, Alp Kucukelbir, and Jon D McAuliffe.  Variational inference: A review for statisticians.  _Journal of the American statistical Association_ , 112(518):859–877, 2017.
  * Hejna and Sadigh [2024] Joey Hejna and Dorsa Sadigh.  Inverse preference learning: Preference-based rl without a reward function.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Fujimoto et al. [2018] Scott Fujimoto, Herke Hoof, and David Meger.  Addressing function approximation error in actor-critic methods.  In _International conference on machine learning_ , pages 1587–1596. PMLR, 2018.
  * Mitchell [2023] Eric Mitchell.  A note on dpo with noisy preferences and relationship to ipo.  _arxiv_ , 2023.  URL https://ericmitchell.ai/cdpo.pdf.
  * Chowdhury et al. [2024] Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan.  Provably robust dpo: Aligning language models with noisy feedback.  _arXiv preprint arXiv:2403.00409_ , 2024.
  * Chen et al. [2021] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  _Advances in neural information processing systems_ , 34:15084–15097, 2021.
  * Misra [2019] Diganta Misra.  Mish: A self regularized non-monotonic activation function.  _arXiv preprint arXiv:1908.08681_ , 2019.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Lu et al. [2022] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  _Advances in Neural Information Processing Systems_ , 35:5775–5787, 2022.
  * An et al. [2021] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song.  Uncertainty-based offline reinforcement learning with diversified q-ensemble.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Zhou et al. [2020] Wenxuan Zhou, Sujay Bajracharya, and David Held.  Plas: Latent action space for offline reinforcement learning.  In _Conference on Robot Learning_ , 2020.
  * Wei et al. [2021] Hua Wei, Deheng Ye, Zhao Liu, Hao Wu, Bo Yuan, Qiang Fu, Wei Yang, and Zhenhui Li.  Boosting offline reinforcement learning with residual generative modeling.  In _International Joint Conference on Artificial Intelligence_ , 2021\.
  * Guan et al. [2023a] Jiayi Guan, Shangding Gu, Zhijun Li, Jing Hou, Yiqin Yang, Guang Chen, and Changjun Jiang.  Uac: Offline reinforcement learning with uncertain action constraint.  _IEEE Transactions on Cognitive and Developmental Systems_ , 2023a.
  * Sohn et al. [2015] Kihyuk Sohn, Honglak Lee, and Xinchen Yan.  Learning structured output representation using deep conditional generative models.  _Advances in neural information processing systems_ , 28, 2015.
  * Kumar et al. [2019] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine.  Stabilizing off-policy q-learning via bootstrapping error reduction.  _Advances in neural information processing systems_ , 32, 2019.
  * Wu et al. [2019] Yifan Wu, George Tucker, and Ofir Nachum.  Behavior regularized offline reinforcement learning.  _arXiv preprint arXiv:1911.11361_ , 2019.
  * Guan et al. [2023b] Jiayi Guan, Guang Chen, Jiaming Ji, Long Yang, Ao Zhou, Zhijun Li, and changjun jiang.  VOCE: Variational optimization with conservative estimation for offline safe reinforcement learning.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023b.
  * Guan et al. [2024] Jiayi Guan, Li Shen, Ao Zhou, Lusong Li, Han Hu, Xiaodong He, Guang Chen, and Changjun Jiang.  POCE: Primal policy optimization with conservative estimation for multi-constraint offline reinforcement learning.  In _Conference on Computer Vision and Pattern Recognition_ , 2024\.
  * Li et al. [2023] Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha.  Hierarchical diffusion for offline decision making.  In _International Conference on Machine Learning_ , pages 20035–20064. PMLR, 2023.
  * Ajay et al. [2022] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B Tenenbaum, Tommi S Jaakkola, and Pulkit Agrawal.  Is conditional generative modeling all you need for decision making?  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Berrueta et al. [2024] Thomas A Berrueta, Allison Pinosky, and Todd D Murphey.  Maximum diffusion reinforcement learning.  _Nature Machine Intelligence_ , pages 1–11, 2024.
  * Xu et al. [2022] Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan.  Prompting decision transformer for few-shot policy generalization.  In _international conference on machine learning_ , pages 24631–24645. PMLR, 2022.
  * Janner et al. [2021] Michael Janner, Qiyang Li, and Sergey Levine.  Offline reinforcement learning as one big sequence modeling problem.  _Advances in neural information processing systems_ , 34:1273–1286, 2021.
  * Lu et al. [2023] Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu.  Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning.  In _International Conference on Machine Learning_ , pages 22825–22855. PMLR, 2023.
  * Hansen-Estruch et al. [2023] Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine.  Idql: Implicit q-learning as an actor-critic method with diffusion policies.  _arXiv preprint arXiv:2304.10573_ , 2023.
  * Chen et al. [2023] Kehua Chen, Xianda Chen, Zihan Yu, Meixin Zhu, and Hai Yang.  Equidiff: A conditional equivariant diffusion model for trajectory prediction.  In _2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC)_ , pages 746–751. IEEE, 2023.
  * Ada et al. [2024] Suzan Ece Ada, Erhan Oztop, and Emre Ugur.  Diffusion policies for out-of-distribution generalization in offline reinforcement learning.  _IEEE Robotics and Automation Letters_ , 2024.

##  Appendix A The overall design of PAO-DP

To visually present the proposed method, the overall design of PAO-DP is
further shown in Fig. 3.

Figure 3: Overall design of the proposed method

##  Appendix B Mathematical Derivations

###  B.1 The Optimal Closed-form Solution of the Regularized Objective in
Offline RL

In offline RL, we follow previous works [24, 6, 25] by maximizing the expected
Q-value under policy $\pi(\bm{a}|\bm{s})$ while constraining the distance
between policy $\pi(\bm{a}|\bm{s})$ and the behavior policy
$\pi^{b}(\bm{a}|\bm{s})$. Therefore, the objective can be represented as
$J(\pi)=\arg\max_{\pi}\mathbb{E}_{\bm{s}}\left[\sum_{\bm{a}}\pi(\bm{a}|\bm{s})Q_{\phi}(\bm{s},\bm{a})\right]-{\eta}\mathbb{E}_{\bm{s}}\left[D_{KL}(\pi(\cdot|\bm{s})||\pi^{b}(\cdot|\bm{s}))\right]$.
Consequently, the optimal closed-form solution for this objective can be
expressed as:

$\displaystyle\pi^{*}(\cdot|\bm{s})=\frac{1}{Z(\bm{s})}\pi^{b}(\cdot|\bm{s})\exp\left[\frac{1}{\eta}Q(\bm{s},\bm{a})\right].$
(17)

Proof. We review the objective of offline RL with a regularization term, and
expanding the KL divergence term of the regularization yields the following
expression:

$\displaystyle
J(\pi)=\arg\max_{\pi}\mathbb{E}_{\bm{s}}\left[\sum_{\bm{a}}\pi(\bm{a}|\bm{s})Q_{\phi}(\bm{s},\bm{a})\right]-{\eta}\mathbb{E}_{\bm{s}}\left[D_{KL}(\pi(\cdot|\bm{s})||\pi^{b}(\cdot|\bm{s}))\right]$
(18)
$\displaystyle\quad\quad~{}=\arg\max_{\pi}\mathbb{E}_{\bm{s}}\left[\sum_{\bm{a}}\pi(\bm{a}|\bm{s})Q_{\phi}(\bm{s},\bm{a})\right]-{\eta}\mathbb{E}_{\bm{s}}\left[\sum_{\bm{a}}\pi(\cdot|\bm{s})\log\frac{\pi(\cdot|\bm{s})}{\pi^{b}(\cdot|\bm{s})}\right]$
$\displaystyle\quad\quad~{}=\arg\max_{\pi}\mathbb{E}_{\bm{s}}\left[\sum_{\bm{a}}\pi(\bm{a}|\bm{s})Q_{\phi}(\bm{s},\bm{a})\right]-{\eta}\mathbb{E}_{\bm{s}}\left[\sum_{\bm{a}}\pi(\cdot|\bm{s})\left[\log\pi(\cdot|\bm{s})-\log\pi^{b}(\cdot|\bm{s})\right]\right].$

To derive the closed-form solution for Eq. (LABEL:appendix_b1_1), we
differentiate Eq. (LABEL:appendix_b1_1) concerning $\pi(\bm{a}|\bm{s})$ ,
resulting in the following expression:

$\displaystyle\mathbb{E}_{\bm{s}}\left[\sum_{\bm{a}}\pi(\bm{a}|\bm{s})Q_{\phi}(\bm{s},\bm{a})\right]-{\eta}\mathbb{E}_{\bm{s}}\left[\sum_{\bm{a}}\pi(\cdot|\bm{s})\left[\log\pi(\cdot|\bm{s})-\log\pi^{b}(\cdot|\bm{s})\right]\right]^{\prime}_{{\pi}(\cdot|\bm{s})}$
(19)
$\displaystyle\overset{\\{i\\}}{=}\mathbb{E}_{\bm{s}}Q_{\phi}(\bm{s},\bm{a})-{\eta}\mathbb{E}_{\bm{s}}\left[\log\pi(\cdot|\bm{s})-\log\pi^{b}(\cdot|\bm{s})+\sum_{\bm{a}}\pi(\cdot|\bm{s})\frac{1}{\pi(\cdot|\bm{s})}\right]$
$\displaystyle=\mathbb{E}_{\bm{s}}\left[Q_{\phi}(\bm{s},\bm{a})-{\eta}\log\frac{\pi(\cdot|\bm{s})}{\pi^{b}(\cdot|\bm{s})}-{\eta}\right].$

In practical computations, the KL divergence typically employs the natural
logarithm base $e$. Consequently, in step $\\{i\\}$,
$[\log\pi(\cdot|\bm{s})]^{\prime}_{\pi(\cdot|\bm{s})}=\pi(\cdot|\bm{s})^{-1}$
holds true. To obtain the explicit expression of the closed-form solution
$\pi^{*}(\cdot|s)$ for Eq. (LABEL:appendix_b1_1), we set Eq
(LABEL:appendix_b1_2) to zero
$\mathbb{E}_{\bm{s}}\left[Q_{\phi}(\bm{s},\bm{a})-{\eta}\log\frac{\pi(\cdot|\bm{s})}{\pi^{b}(\cdot|\bm{s})}-{\eta}\right]=0$,
yielding the following expression:

$\displaystyle
Q_{\phi}(\bm{s},\bm{a})-{\eta}\log\frac{\pi^{*}(\cdot|\bm{s})}{\pi^{b}(\cdot|\bm{s})}-{\eta}=0\iff\log\frac{\pi^{*}(\cdot|\bm{s})}{\pi^{b}(\cdot|\bm{s})}=\eta
Q(\bm{s},\bm{a})-1$ (20)
$\displaystyle\iff\pi^{*}(\cdot|\bm{s})=\pi^{b}(\cdot|\bm{s})\exp[\frac{1}{\eta}Q(\bm{s},\bm{a})-1]$
$\displaystyle\iff\pi^{*}(\cdot|\bm{s})=\frac{1}{Z(\bm{s})}\pi^{b}(\cdot|\bm{s})\exp[\frac{1}{\eta}Q(\bm{s},\bm{a})]$

where $\pi^{*}$ is the optimal closed-form solution of Eq.
LABEL:appendix_b1_1, and ${Z}(\bm{s})=\exp(1)$ is the partition function.

From Eq. (LABEL:appendix_b1_3), it is evident that there is a direct
relationship between the policy $\pi^{*}(\bm{a}|\bm{s})$ and the behavior
policy $\pi^{b}(\bm{a}|\bm{s})$. However, accurately estimating the behavior
policy $\pi^{b}(\bm{a}|\bm{s})$ is challenging. Therefore, following previous
work[24, 8, 22, 25], we project the policy $\pi^{*}(\bm{a}|\bm{s})$ onto the
parameterized policy $\pi_{\theta}(\bm{a}|\bm{s})$. Consequently, the final
optimization objective can be expressed as:

$\begin{split}&J(\pi_{\theta})=\mathop{\arg\max}\limits_{\bm{\theta}}\;\mathbb{E}_{(\bm{s},\bm{a})\sim\mathcal{D}}\left[\frac{1}{Z(\bm{s})}\log\pi_{\theta}(\bm{a}|\bm{s})\exp(\frac{1}{\eta}Q_{\phi}(\bm{s},\bm{a}))\right].\end{split}$
(21)

Proof. We utilize the KL divergence to constrain the distance between
$\pi^{*}$ and $\pi_{\theta}$, achieving the projection of $\pi^{*}$ onto the
parameterized $\pi_{\theta}$. The The derivation process can be expressed as
follows:

$\begin{split}&\mathop{\arg\min}\limits_{\theta}\;\mathbb{E}_{\bm{s}\sim\mathcal{D}}[D_{KL}(\pi^{*}(\cdot|\bm{s})||\pi_{\theta}(\cdot|\bm{s}))]\\\
&=\mathop{\arg\min}\limits_{\theta}\mathbb{E}_{\bm{s}\sim\mathcal{D}}\left[\sum_{\bm{a}}\pi^{*}(\cdot|\bm{s})\left[\log\pi^{*}(\cdot|\bm{s})-\log\pi_{\theta}(\cdot|\bm{s})\right]\right]\\\
&=\mathop{\arg\min}\limits_{\theta}\mathbb{E}_{\bm{s}\sim\mathcal{D}}\mathbb{E}_{\bm{a}\sim{\pi^{*}(\cdot|\bm{s})}}\left[\log\pi^{*}(\cdot|\bm{s})-\log\pi_{\theta}(\cdot|\bm{s})\right]\\\
&=\mathop{\arg\min}\limits_{\theta}\underbrace{\mathbb{E}_{\bm{s}\sim\mathcal{D}}\mathbb{E}_{\bm{a}\sim\pi^{*}(\cdot|\bm{s})}\log\pi^{*}(\cdot|\bm{s})}_{\mathcal{C}}-\mathbb{E}_{\bm{s}\sim\mathcal{D}}\mathbb{E}_{\bm{a}\sim\pi^{*}(\cdot|\bm{s})}\log\pi_{\theta}(\cdot|\bm{s})\\\
&\overset{\\{i\\}}{\iff}\mathop{\arg\min}\limits_{\theta}-\mathbb{E}_{\bm{s}\sim\mathcal{D}}\mathbb{E}_{\bm{a}\sim\pi^{*}(\cdot|\bm{s})}\log\pi_{\theta}(\cdot|\bm{s})\\\
&\iff\mathop{\arg\max}\limits_{\theta}\mathbb{E}_{\bm{s}\sim\mathcal{D}}\mathbb{E}_{\bm{a}\sim\pi^{*}(\cdot|\bm{s})}\log\pi_{\theta}(\cdot|\bm{s})\\\
&=\mathop{\arg\max}\limits_{\theta}\mathbb{E}_{\bm{s}\sim\mathcal{D}}\left[\sum_{\bm{a}}\pi^{*}(\cdot|\bm{s})\log\pi_{\theta}(\cdot|\bm{s})\right]\\\
&=\mathop{\arg\max}\limits_{\theta}\mathbb{E}_{\bm{s}\sim\mathcal{D}}\left[\sum_{\bm{a}}\pi^{b}(\cdot|\bm{s})\frac{\pi^{*}(\cdot|\bm{s})}{\pi^{b}(\cdot|\bm{s})}\log\pi_{\theta}(\cdot|\bm{s})\right]\\\
&=\mathop{\arg\max}\limits_{\theta}\mathbb{E}_{\bm{s}\sim\mathcal{D}}\mathbb{E}_{\bm{a}\sim\pi^{b}(\cdot|\bm{s})}\left[\frac{\pi^{*}(\cdot|\bm{s})}{\pi^{b}(\cdot|\bm{s})}\log\pi_{\theta}(\cdot|\bm{s})\right]\\\
&\overset{\\{ii\\}}{=}\mathop{\arg\max}\limits_{\theta}\mathbb{E}_{\bm{s}\sim\mathcal{D}}\mathbb{E}_{\bm{a}\sim\pi^{b}(\cdot|\bm{s})}\left[\frac{\frac{1}{Z(\bm{s})}\pi^{b}(\cdot|\bm{s})\exp[\frac{1}{\eta}Q(\bm{s},\bm{a})]}{\pi^{b}(\cdot|\bm{s})}\log\pi_{\theta}(\cdot|\bm{s})\right]\\\
&=\mathop{\arg\max}\limits_{\theta}\mathbb{E}_{\bm{s}\sim\mathcal{D}}\mathbb{E}_{\bm{a}\sim\pi^{b}(\cdot|\bm{s})}\left[\frac{1}{Z(\bm{s})}\log\pi_{\theta}(\cdot|\bm{s})\exp\left[\frac{1}{\eta}Q(\bm{s},\bm{a})\right]\right]\\\
&\overset{\\{iii\\}}{=}\mathop{\arg\max}\limits_{\theta}\mathbb{E}_{\bm{s}\sim\mathcal{D}}\mathbb{E}_{\bm{a}\sim\mathcal{D}}\left[\frac{1}{Z(\bm{s})}\log\pi_{\theta}(\cdot|\bm{s})\exp\left[\frac{1}{\eta}Q(\bm{s},\bm{a})\right]\right]\\\
&=\mathop{\arg\max}\limits_{\theta}\mathbb{E}_{(\bm{s},\bm{a})\sim\mathcal{D}}\left[\frac{1}{Z(\bm{s})}\log\pi_{\theta}(\cdot|\bm{s})\exp\left[\frac{1}{\eta}Q(\bm{s},\bm{a})\right]\right]\end{split}$
(22)

where ${Z}(\bm{s})$ is the same partition function as in Eq.
(LABEL:appendix_b1_3). In step $\\{i\\}$, since
$\mathbb{E}_{\bm{s}\sim\mathcal{D}}\mathbb{E}_{\bm{a}\sim{\pi^{*}(\cdot|\bm{s})}}\left[\log\pi^{*}(\cdot|\bm{s})\right]$
is a constant concerning the parameter $\theta$ during the optimization of Eq.
(22), it does not affect the maximization objective and can thus be directly
ignored. In step $\\{ii\\}$, Eq. (LABEL:appendix_b1_3) is substituted into the
current equation. In step $\\{iii\\}$, $\pi^{b}(\bm{a}|\bm{s})$ is the
behavior policy of the dataset $\mathcal{D}$, thus
$\pi^{b}(\bm{a}|\bm{s})\sim\mathcal{D}$.

###  B.2 Deriving the Preference Optimization Objective of PAO-DP Under the
Bradley-Terry Model

In Section 3.3, the popular Bradley-Terry (BT) model [19, 18] is used to model
the preference distribution $p^{*}$ for actions:

$p^{*}(\hat{\bm{a}}_{t}\succ\bm{a}_{t}|\bm{s}_{t})=\frac{\exp(Q(\bm{s}_{t},\hat{\bm{a}}_{t}))}{\exp(Q(\bm{s}_{t},\hat{\bm{a}}_{t}))+\exp(Q(\bm{s}_{t},\bm{a}_{t}))}.\\\
$ (23)

Given Eq. (23), we can directly derive the preference optimization objective
of PAO-DP under the BT preference model. Meanwhile, in Section 3.3, based on
Eq. (2), we can express the ground-truth Q-value through its corresponding
optimal policy:

$Q(\bm{s}_{t},\bm{a}_{t}))=\eta\log\frac{\pi^{*}(\bm{a}_{t}|\bm{s}_{t})}{\pi^{b}(\bm{a}_{t}|\bm{s}_{t})}+\eta\log
Z(\bm{s}),\\\ $ (24)

Substituting Eq. (24) into Eq. (23) we obtain:

$\begin{split}p^{*}(\hat{\bm{a}}_{t}\succ\bm{a}_{t}|\bm{s}_{t})&=\frac{\exp\left(\eta\log\frac{\pi^{*}(\hat{\bm{a}}_{t}|\bm{s}_{t})}{\pi^{b}(\hat{\bm{a}}_{t}|\bm{s}_{t})}+\eta\log
Z(s)\right)}{\exp\left(\eta\log\frac{\pi^{*}(\hat{\bm{a}}_{t}|\bm{s}_{t})}{\pi^{b}(\hat{\bm{a}}_{t}|\bm{s}_{t})}+\eta\log
Z(s)\right)+\exp\left(\eta\log\frac{\pi^{*}(\bm{a}_{t}|\bm{s}_{t})}{\pi^{b}(\bm{a}_{t}|\bm{s}_{t})}+\eta\log
Z(s)\right)}\\\ &=\frac{\exp(\eta\log
Z(\bm{s}))\exp\left(\eta\log\frac{\pi^{*}(\hat{\bm{a}}_{t}|\bm{s}_{t})}{\pi^{b}(\hat{\bm{a}}_{t}|\bm{s}_{t})}\right)}{\exp(\eta\log
Z(\bm{s}))\left[\exp\left(\eta\log\frac{\pi^{*}(\hat{\bm{a}}_{t}|\bm{s}_{t})}{\pi^{b}(\hat{\bm{a}}_{t}|\bm{s}_{t})}\right)+\exp\left(\eta\log\frac{\pi^{*}(\bm{a}_{t}|\bm{s}_{t})}{\pi^{b}(\bm{a}_{t}|\bm{s}_{t})}\right)\right]}\\\
&=\frac{\exp\left(\eta\log\frac{\pi^{*}(\hat{\bm{a}}_{t}|\bm{s}_{t})}{\pi^{b}(\hat{\bm{a}}_{t}|\bm{s}_{t})}\right)}{\exp\left(\eta\log\frac{\pi^{*}(\hat{\bm{a}}_{t}|\bm{s}_{t})}{\pi^{b}(\hat{\bm{a}}_{t}|\bm{s}_{t})}\right)+\exp\left(\eta\log\frac{\pi^{*}(\bm{a}_{t}|\bm{s}_{t})}{\pi^{b}(\bm{a}_{t}|\bm{s}_{t})}\right)}\\\
&=\frac{1}{1+\exp\left(\eta\log\frac{\pi^{*}(\bm{a}_{t}|\bm{s}_{t})}{\pi^{b}(\bm{a}_{t}|\bm{s}_{t})}-\eta\log\frac{\pi^{*}(\hat{\bm{a}}_{t}|\bm{s}_{t})}{\pi^{b}(\hat{\bm{a}}_{t}|\bm{s}_{t})}\right)}\\\
&=\sigma\left(\eta\log\frac{\pi^{*}(\hat{\bm{a}}_{t}|\bm{s}_{t})}{\pi^{b}(\hat{\bm{a}}_{t}|\bm{s}_{t})}-\eta\log\frac{\pi^{*}(\bm{a}_{t}|\bm{s}_{t})}{\pi^{b}(\bm{a}_{t}|\bm{s}_{t})}\right),\end{split}$
(25)

$\sigma$ is the logistic function. The final line represents the per-instance
loss in Eq. (15).

##  Appendix C More Results

Table 3: Performance of the baselines and PAO-OP evaluated using the OMS metric. Dataset | Environment | DIFF-QL | CRR+EDP | IQL+EDP | PAO-DP (Ours)
---|---|---|---|---|---
Complete | Kitchen | 84.0 | 95.8 | 95.0 |  100.0 $\pm$ 0.00
Partial | Kitchen | 60.5 | 56.7 | 72.5 |  70.0 $\pm$ 2.04
Mixed | Kitchen | 62.5 | 59.2 | 70.0 |  71.8 $\pm$ 0.94
Average (FrankaKitchen) | 69.0 | 70.6 | 79.2 |  80.6 $\pm$ 13.79
Default | AntMaze-umaze | 93.6 | 98.0 | 98.0 |  100.0 $\pm$ 0.0
Diverse | AntMaze-umaze | 66.2 | 80.0 | 90.0 |  100.0 $\pm$ 0.0
Play | AntMaze-medium | 76.6 | 82.0 | 89.0 |  100.0 $\pm$ 0.0
Diverse | AntMaze-medium | 78.6 | 72.0 | 88.0 |  100.0 $\pm$ 0.0
Play | AntMaze-large | 46.4 | 57.0 | 52.0 |  90.0 $\pm$ 0.0
Diverse | AntMaze-large | 56.6 | 71.0 | 68.0 |  90.0 $\pm$ 0.0
Average (AntMaze) | 69.6 | 76.7 | 89.2 |  96.7 $\pm$ 4.71
human | pen | 72.8 | 127.8 | 130.3 |  124.1 $\pm$ 1.59
cloned | pen | 57.3 | 106.0 | 138.2 |  119.2 $\pm$ 2.05
Average (Adroit) | 65.1 | 116.9 | 134.3 |  121.6 $\pm$ 3.06
Medium-Expert | HalfCheetah | 96.8 | 93.5 | 80.9 |  92.4 $\pm$ 0.27
Medium-Expert | Hopper | 111.1 | 109.4 | 95.7 |  111.1 $\pm$ 0.97
Medium-Expert | Walker | 110.1 | 112.3 | 111.5 |  110.6 $\pm$ 0.49
Medium | HalfCheetah | 51.1 | 50.2 | 48.7 |  47.4 $\pm$ 0.25
Medium | Hopper | 90.5 | 95.0 | 97.3 |  71.4 $\pm$ 1.27
Medium | Walker | 87.0 | 85.8 | 88.7 |  88.0 $\pm$ 0.31
Medium-Replay | HalfCheetah | 47.8 | 47.8 | 45.5 |  44.5 $\pm$ 0.10
Medium-Replay | Hopper | 101.3 | 101.7 | 100.9 |  100.8 $\pm$ 0.49
Medium-Replay | Walker | 95.5 | 89.8 | 93.4 |  90.9 $\pm$ 0.67
Average (Locomotion) | 88.0 | 87.3 | 84.7 |  84.1 $\pm$ 23.40

Table 3 presents the detailed performance of various baseline methods and our
proposed PAO-DP evaluated using the OMS metric across multiple environments.
For the FrankaKitchen domain, PAO-DP consistently outperforms other methods,
particularly excelling in the Complete dataset. This indicates PAO-DP’s
ability to handle tasks requiring sequential subtask completion. In the
AntMaze domain, PAO-DP again demonstrates superior performance, achieving
perfect scores in different datasets. This highlights the method’s
effectiveness in navigating complex mazes and dealing with sparse rewards. For
the Adroit domain, although PAO-DP’s average score of 121.6 is slightly lower
than IQL+EDP, it remains competitive, showing strong performance in human and
cloned tasks. This reflects its capability in high-dimensional robotic
manipulation tasks. In the Locomotion domain, the performance is more varied.
PAO-DP performs well in some tasks but shows a wider range in scores. This
variability suggests that while PAO-DP is versatile, its relative advantage
may be reduced in environments with smoother reward functions compared to
sparse-reward settings. This may be because in sparse-reward environments,
preferred actions with significant quality differences can be generated,
efficiently promoting preference optimization for policy improvement, but less
in smooth-reward environments. Overall, PAO-DP demonstrates robust and
competitive performance across diverse and challenging environments,
particularly excelling in scenarios with complex trajectories and sparse
rewards.

##  Appendix D More Ablation Analysis

Analysis of Preferred-Action Sampling Strategies.  Figures 4-(a) and 4-(b)
explore scenarios where $\lambda=0.0$, indicating the absence of the anti-
noise strategy. In the relatively noise-free Kitchen-complete dataset, PAO-DP-
Max shows superior performance, suggesting that in datasets with minimal
noise, the highest sampling weight is reliable and effective because high-
quality actions may be close to the distribution of optimal actions.
Conversely, in the more complex and randomization AntMaze-large-play dataset,
PAO-DP-Max performs less reliably, highlighting that in challenging settings,
the highest sampling weight may lead to sub-optimal policy updates. Figures
4-(b) and 4-(c) further illustrate the impact of introducing the anti-noise
mechanism by varying $\lambda$. With $\lambda=0.0$, PAO-DP-Max’s performance
is significantly lower compared to PAO-DP-Mean and PAO-DP-Min, indicating
unreliability in relatively noisy datasets. However, increasing $\lambda$ to
0.4 shows a recovery in PAO-DP-Max’s performance, aligning it closely with
optimal results observed in PAO-DP-Mean and PAO-DP-Min. This recovery
underscores the effectiveness of the anti-noise strategy in mitigating the
adverse effects of noise, enhancing robustness, and improving policy
reliability. This analysis highlights the critical role of preferred-action
sampling strategies for policy improvement and the need for rational selection
of the sampling strategies based on the anti-noise parameter $\lambda$.

(a) $\lambda=0.0$

(b) $\lambda=0.0$

(c) $\lambda=0.4$

Figure 4: Performance of PAO-DP variants in Kitchen-complete and AntMaze-
large-play environments, comparing different sampling strategies with
$\lambda=0.0$ and $\lambda=0.4$.

The Impact of $\lambda$ on Different Tasks. Figure 5-(a) illustrates that in
the relatively noise-free Kitchen-complete dataset, PAO-DP-Max with different
$\lambda$ values yields similar performance outcomes, indicating the minimal
impact of the anti-noise strategy. This suggests that in low-noise datasets,
policy performance remains robust across various $\lambda$ settings, with the
highest sampling weight (PAO-DP-Max) consistently achieving excellent results.
In contrast, Figure 5-(b) reveals significant performance variations in the
AntMaze-large-play environment as $\lambda$ increases. With $\lambda=0.0$,
performance is notably lower, especially for PAO-DP-Max (See Figure 4-(b)),
highlighting the challenges posed by the complex and noisy nature of this
environment. As $\lambda$ increases, performance improves, particularly for
$\lambda=0.4$, where PAO-DP-Max aligns closely with PAO-DP-Mean and PAO-DP-Min
(See Figure 4-(c)). This indicates that the proposed anti-noise mechanism
effectively mitigates noise, enhancing policy reliability and robustness in
challenging noisy, complex settings.

(a)

(b)

Figure 5: Impact of varying $\lambda$ values on PAO-DP performance in Kitchen-
complete and AntMaze-large-play environments.

##  Appendix E Hyperparameters

The detailed hyperparameters of the proposed method are presented in Table 4.

Table 4: Hyperparameters of PAO-DP. Dataset | Environment | Learning rate | Epochs | Batch size | $\eta$ | $\lambda$ | $\xi$
---|---|---|---|---|---|---|---
Complete | Kitchen | 3e-4 | 1000 | 256 | 0.1 | 0.0 | 1.0
Partial | Kitchen | 3e-4 | 1000 | 256 | 0.1 | 0.2 | 1.0
Mixed | Kitchen | 3e-4 | 1000 | 256 | 0.1 | 0.2 | 1.0
Default | AntMaze-umaze | 3e-4 | 1000 | 256 | 0.1 | 0.2 | 1.0
Diverse | AntMaze-umaze | 3e-4 | 1000 | 256 | 0.1 | 0.2 | 1.0
Play | AntMaze-medium | 3e-4 | 1000 | 256 | 0.1 | 0.1 | 1.0
Diverse | AntMaze-medium | 2e-4 | 1000 | 256 | 0.1 | 0.5 | 1.0
Play | AntMaze-large | 1e-4 | 1000 | 256 | 0.1 | 0.4 | 1.0
Diverse | AntMaze-large | 1e-4 | 1000 | 256 | 0.1 | 0.4 | 1.0
human | pen | 1e-4 | 1000 | 256 | 0.1 | 0.4 | 1.0
cloned | pen | 5e-5 | 1000 | 256 | 0.1 | 0.2 | 1.0
Medium-Expert | HalfCheetah | 3e-4 | 2000 | 256 | 0.1 | 0.2 | 1.0
Medium-Expert | Hopper | 3e-4 | 2000 | 256 | 0.1 | 0.2 | 1.0
Medium-Expert | Walker | 3e-4 | 2000 | 256 | 0.1 | 0.2 | 1.0
Medium | HalfCheetah | 3e-4 | 2000 | 256 | 0.1 | 0.2 | 1.0
Medium | Hopper | 3e-4 | 2000 | 256 | 0.1 | 0.2 | 1.0
Medium | Walker | 3e-4 | 2000 | 256 | 0.1 | 0.1 | 1.0
Medium-Replay | HalfCheetah | 3e-4 | 2000 | 256 | 0.1 | 0.1 | 1.0
Medium-Replay | Hopper | 3e-4 | 2000 | 256 | 0.1 | 0.1 | 1.0
Medium-Replay | Walker | 3e-4 | 2000 | 256 | 0.1 | 0.4 | 1.0

##  Appendix F Limitations and Future Work

The effectiveness of the proposed PAO-DP hinges on accurate Q-value estimation
to select preferred actions. However, offline RL continues to grapple with the
out-of-distribution problem, where discrepancies between evaluation and
training data may lead to Q-value estimation errors. These inaccuracies
propagate throughout the learning process, resulting in suboptimal policy
performance. Additionally, the accuracy of Q-values is impacted by the quality
and complexity of datasets, further challenging the robustness of the policy.

Future research could enhance PAO-DP by incorporating trajectory-based
preference optimization. We directly generate preferred trajectory data for
preference optimization, instead of preferred action data using Q-values. This
avoids the OOD problem using Q-values. We can compare two trajectories based
on success or time to generate preferred trajectory data. This approach is
expected to improve policy robustness and generalization, especially in
complex environments. In addition, developing methods to encode and utilize
trajectory information, possibly using advanced sequence modeling techniques,
holds promise for more stable and strong offline RL.