# RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization
Method for Alignment of Large Language Models

Saeed Khaki
Amazon
<EMAIL_ADDRESS>
&JinJin Li
Amazon
<EMAIL_ADDRESS>
Lan Ma
Amazon
<EMAIL_ADDRESS>
&Liu Yang
Amazon
<EMAIL_ADDRESS>
&Prathap Ramachandra
Amazon
<EMAIL_ADDRESS>

###### Abstract

Reinforcement learning from human feedback (RLHF) has been extensively
employed to align large language models with user intent. However, proximal
policy optimization (PPO) based RLHF is occasionally unstable requiring
significant hyperparameter finetuning, and computationally expensive to
maximize the estimated reward during alignment. Recently, direct preference
optimization (DPO) is proposed to address those challenges. However, DPO
relies on contrastive responses generated from human annotator and alternative
LLM, instead of the policy model, limiting the effectiveness of the RLHF. In
this paper, we addresses both challenges by systematically combining rejection
sampling (RS) and DPO. Our proposed method, RS-DPO, initiates with the
development of a supervised fine-tuned policy model (SFT). A varied set of k
responses per prompt are sampled directly from the SFT model. RS-DPO
identifies pairs of contrastive samples based on their reward distribution.
Finally, we apply DPO with the contrastive samples to align the model to human
preference. Our experiments indicate that our proposed method effectively
fine-tunes LLMs with limited resource environments, leading to improved
alignment with user intent. Furthermore, it outperforms existing methods,
including RS, PPO, and DPO.

RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method
for Alignment of Large Language Models

Saeed Khaki Amazon<EMAIL_ADDRESS>JinJin Li Amazon<EMAIL_ADDRESS>

Lan Ma Amazon<EMAIL_ADDRESS>Liu Yang Amazon<EMAIL_ADDRESS>Prathap
Ramachandra Amazon<EMAIL_ADDRESS>

##  1 Introduction

State-of-the-art (SOTA) LLMs such as GPT-4 (OpenAI, 2023), LLaMa (Touvron et
al., 2023) etc., are trained with several stages. With pre-training and
supervised instruction tuning, LLMs learns to follow specific instructions to
complete various tasks with zero-shot or few-shot prompts (Chowdhery et al.,
2022). To further improve the LLMs’ intelligence as close as to human and
ensure a more helpful and harmless model, alignment is important as the last-
mile LLM training procedure (Ziegler et al., 2019; Stiennon et al., 2020b; Bai
et al., 2022; Ouyang et al., 2022). Reinforcement learning with human feedback
(RLHF) (Christiano et al., 2017) is the most adopted approach for alignment
training and it usually involves training a reward model with human preference
datasets which optimizes a reward function based on the human-annotated
preference. Then LLMs are fine-tuned to learn to maximize the reward of their
responses using reinforcement learning algorithms, including proximal policy
optimization (PPO) (Schulman et al., 2017), REINFORCE (Williams, 2004), and
similar variants. While PPO is used by SOTA LLMs due to its ease of use and
good performance, training with PPO has few limitations, including complexity
of training multiple LLMs, and sampling from policy model in training loop,
high GPU memory requirement with hosting multiple LLMs during training, and
sensitivity to training data and reward models.

To make RLHF training more efficient, there are methods proposed from
different perspective. In order to reduce the preference data effort by human
annotation, (Lee et al., 2023) and (Tunstall et al., 2023) proposed to train
the LLM to align to the LLM’s preference rating in order to save human effort.
(Santacroce et al., 2023) proposed a combined strategy to merge SFT and reward
models as well as in PPO with LoRA selection in order to reduce latency and
memory footprint. (Dong et al., 2023; Gulcehre et al., 2023) used reward model
to select ranked high-reward good samples to supervise fine-tune the models
and iteratively repeating this process yield good results. To reduce the
memory and save training resources, (Rafailov et al., 2023) proposed the
direct preference optimization (DPO) to remove the need of training reward
model, and directly optimize the policy model using a simple classification to
maximize the difference between likelihood of human preference pairs. This
method proves equivalent performance by implicitly maximize the reward.
However, it is mainly trained on human preference data to learn the alignment,
instead of sampling the policy model’s response for optimization. LLaMa2
(Touvron et al., 2023) adopts several rounds of rejection sampling to select
the best samples from $k$ model-generated samples for fine-tuning before PPO
in order to boost the model performance. But rejection sampling only selects
the best samples instead of preference pairs, with low data usage efficiency.
RSO (Liu et al., 2023) proposes to generate preference data from the target
optimal policy using rejection sampling, enabling a more accurate estimation
of the optimal policy. Compared to RSO, our proposed method (RS-DPO) directly
employs a point-wise reward model for response ranking and optimization,
utilizing logistic loss exclusively during policy optimization. Unlike RSO’s
approach of statistical rejection sampling and tournament ranking for response
generation and selection, RS-DPO generates a fixed number of responses per
prompt and relies on computing reward gaps between responses for preference
data generation, resulting in reduced computational expense. Additionally,
while RSO lacks evaluation on standard alignment benchmarks and comparison
against PPO, RS-DPO demonstrates its effectiveness against other RLHF methods
on such benchmarks.

In this work, we propose RS-DPO method for RLHF training that combines the
advantages of existing efficient methods, including offline preference data
generation using rejection sampling, and using DPO in order to reduce the
training GPU memory consumption. Specifically, RS-DPO generates responses from
the large language model directly, and leverages rejection sampling (RS) to
sample synthetic preference pairs based on the reward distribution of LLMs
responses. Then, it uses the generated preference pairs for alignment with
DPO. The main contributions of our proposed RLHF training method can be
summarized as follows: (1) RS-DPO demonstrates stability and robustness
against variations in the reward model quality, consistently outperforming
existing methods like DPO, PPO and RS. (2) In contrast to the rejection
sampling approach that focuses solely on the best response among $k$ generated
responses for alignment, RS-DPO selects pairs of contrastive samples based the
reward distribution, thereby enhancing overall performance. (3) RS-DPO samples
contrastive data directly from the SFT model, distinguishing itself from DPO
which relies on responses from alternative language models or human
annotations. This approach contributes to the superior performance of RS-DPO.
(4) Our proposed method is efficient, being less resource-intensive compared
to PPO, making it practical for applications in limited resource environments.

##  2 Method

The aim of this study is to utilize reinforcement learning from human feedback
(RLHF) to train a policy model with the purpose of aligning a large language
model to user intent. As the pipeline shown in 1, our proposed method, RS-DPO,
systematically combines RS and DPO. It starts by generating a diverse set of
$k$ distinct responses for each prompt, selecting a pair of contrasting
samples based on their reward distribution. Subsequently, the method employs
DPO to enhance the performance of the language model (LLM), thereby achieving
improved alignment. Our proposed method consists the following steps:

Figure 1: The pipeline of RS-DPO. Our proposed method systematically combines
rejection sampling (RS) and direct preference optimization (DPO). We start by
creating a SFT model and use it to generate a diverse set of $k$ distinct
responses for each prompt. Then, it selects a pair of contrastive samples
based on their reward distribution. Subsequently, the method employs DPO to
enhance the performance of the language model (LLM), thereby achieving
improved alignment.

###  2.1 Supervised Fine-Tuning (SFT)

As a prerequisite to RLHF, this step involves fine-tuning a pre-trained LLM,
$\pi$, using a dataset consisting of high-quality instruction and response
pairs or chat data, denoted as
$\mathcal{D_{\text{sft}}}=\\{(x_{1},y_{1}),\ldots,(x_{m},y_{m})\\}$ (Ouyang et
al., 2022; Wang et al., 2023a; Chung et al., 2022; Wang et al., 2022).
Starting from a base LLM $\pi$, SFT maximizes the likelihood of response $y$
given prompt $x$ as defined in the Equation 1.

$\mathcal{L^{\text{SFT}}}={\text{argmax}}\sum_{(x,y)\in\mathcal{D_{\text{sft}}}}\log\pi(y|x)$
(1)

###  2.2 Reward Model Training (RM)

This step involves training a reward model to assess the quality of a response
in accordance with human preferences, with a focus on desired downstream
attributes like helpfulness and harmlessness (Wang et al., 2023a; Ouyang et
al., 2022). The reward model, denoted as $R(x,y)$, takes a prompt $x$ and a
response $y$, and maps them to a scalar value $r$. Let’s assume that we have a
preference dataset, denoted as
$\mathcal{D_{\text{RM}}}=\\{(x_{1},y_{1l},y_{1w}),\ldots,(x_{n},y_{nl},y_{nw})\\}$,
where $x$ represents the input prompt, and $y_{l}$ and $y_{w}$ are considered
the worse and the better responses, respectively, as determined by human
assessment. Reward model training uses ranked answers from
$\mathcal{D_{\text{RM}}}$ to estimate the preference distribution $p$ as
written in Equation 2 (Bradley and Terry, 1952).

$p(y_{w}\succ
y_{l}|x)=\frac{\exp{(r(x,y_{w}))}}{\exp{(r(x,y_{w}))}+\exp{(r(x,y_{l}))}}$ (2)

Subsequently, we can estimate the parameters of the reward model through
maximum likelihood by maximizing the reward gap between $y_{w}$ and $y_{l}$,
as illustrated in Equation 3 (Wang et al., 2023b; Rafailov et al., 2023).

$\begin{split}R(x,y)={\text{argmin}}\sum_{(x,y_{l},y_{w})\in\mathcal{D}_{\text{RM}}}\\\
-\log\sigma(\exp{(r(x,y_{w}))}-\exp{(r(x,y_{l}))})\end{split}$ (3)

###  2.3 Preference Data Generation via Rejection Sampling (PDGRS)

The goal of this step is to create a synthetic preference pair dataset for our
alignment task using the trained SFT and RM. Let’s denote
$\\{x_{1},\ldots,x_{n}\\}$ as a set of sampled prompts from
$\mathcal{D_{\text{RM}}}$ dataset. While it is possible to sample from other
prompt datasets, it is crucial to ensure that our prompt sampling remains
within the reward model’s prompt distribution for optimal performance. We
first generate $k$ distinct responses from $\mathcal{L^{\text{SFT}}}$ model
for each prompt $x$. Then, we evaluate the quality of each response using our
trained reward model $R(x,y)$. Finally, we compute the reward gap for all
possible pairwise combinations of responses per prompt, ${k}\choose{2}$. If
the reward gap surpasses a predefined threshold, we include the pair of
responses in our synthetic preference dataset. The process of preference data
generation is illustrated in Algorithm 1.

Since the preference data generation process generates responses from
$\mathcal{L_{\text{SFT}}}$ model, it ensures that our RLHF is focused on
aligning the $\mathcal{L_{\text{SFT}}}$ behaviour to the human preference
rather than distilling knowledge from a larger model or human annotations. We
term this process preference data generation via rejection sampling (PDGRS),
as it involves evaluating each possible preference data triplet combination
(superior and inferior responses), and discarding those with reward gaps below
predefined threshold. In addition, our proposed preference data generation
process bootstraps and substantially augments the quantity of preference data,
compared to the initial static preference dataset $\mathcal{D_{\text{RM}}}$
used in the reward model training.

Algorithm 1 Preference Data Generation via Rejection Sampling

Result:

$\mathcal{D_{\text{P}}}=\\{(x,y_{l},y_{w})\\}_{3m}$ : Preference dataset

Input:

$\\{x_{1},\ldots,x_{n}\\}$ : Sample prompts from $\mathcal{D_{\text{RM}}}$

$\mathcal{L^{\text{SFT}}}$: SFT model

$R(x,y)$: Reward model

$\tau$: Temperature

$\eta$: Threshold for preference data selection

for $i=1:n$ do

$\,(y_{i1},\ldots,y_{ik})\,|\,y_{ik}\sim\mathcal{L^{\text{SFT}}}(\raisebox{0.21529pt}{$\cdot$}|x_{i})$
$\triangleright$ generate $k$ responses from $\mathcal{L^{\text{SFT}}}$ model
for prompt $x_{i}$

$\,(r_{i1},\ldots,r_{ik})\,|\,r_{ij}=R(x_{i},y_{ij})$ $\triangleright$ compute
the reward for each of generated responses

for $j=1:k$ do

for $l=j+1:k$ do

$r_{\text{gap}}=\,\sigma(\frac{r_{ij}-r_{il}}{\tau})$ $\triangleright$ compute
the reward gap between the pair of responses $y_{il}$ and $y_{ij}$

if $r_{\text{gap}}>\eta$ then

$\mathcal{D_{\text{P}}}=\\{\mathcal{D_{\text{P}}};(x_{i},y_{il},y_{ij})\\}$
$\triangleright$ append the accepted sample

end if

end for

end for

end for

###  2.4 Direct Preference Optimization (DPO)

DPO fine-tunes $\mathcal{L^{\text{SFT}}}$ by directly optimizing the policy
model on static preference data $(x,y_{l},y_{w})$, maximizing the likelihood
of the preferred $y_{w}$ over $y_{l}$. This approach eliminates the necessity
of fitting an explicit reward model by using the ratio of likelihood between
the policy $\mathcal{L^{\text{RL}}}$ model and the original
$\mathcal{L^{\text{SFT}}}$ model as an implicit reward signal (Rafailov et
al., 2023). During training process, DPO optimizes the objective function as
written in Equation 4.

$\begin{split}\mathcal{L}^{\text{RL}}={\text{argmax}}&\sum_{(x,y_{l},y_{w})\in\mathcal{D}_{\text{P}}}\\\
&\log\sigma(\beta\log\frac{\mathcal{L}^{\text{RL}}(y_{w}|x)}{\mathcal{L}^{\text{SFT}}(y_{w}|x)}\\\
&-\beta\log\frac{\mathcal{L}^{\text{RL}}(y_{l}|x)}{\mathcal{L}^{\text{SFT}}(y_{l}|x)})\end{split}$
(4)

Our proposed method employs DPO on the synthetic preference dataset
$\mathcal{D_{\text{P}}}$ generated in step 2.3 using PDGRS, in order to align
the policy model with human preferences.

##  3 Experiments Details

This section outlines our experiments to demonstrate the effectiveness of our
proposed RS-DPO method for the alignment task. We conduct all of our
experiments on the Llama-2-7B LLM (Touvron et al., 2023) which is one of the
state-of-the-art LLMs at 7B parameter scale. We perform supervised fine-tuning
(SFT), reward modeling (RM), DPO training, and PPO training based on
Huggingface Transformer Reinforcement Learning (TRL) library (von Werra et
al., 2020). We utilize DeepSpeed ZeRO-3 (Rajbhandari et al., 2020) for
optimizing GPU memory and training speed. All experiments are conducted on 8
A100s GPUs with 40G memory per GPU.

###  3.1 Datasets

We use the following datasets in our experiments:

Open Assistant: Open Assistant (OASST1) (Köpf et al., 2023) is a multilingual
human-generated conversation dataset ranked for quality. In our experiment, we
utilize the highest quality partition based on quality ranking, comprising of
9k samples.

Anthropic/HH-RLHF: Anthropic released this dataset that includes 169.55k
conversation pairs between humans and an AI assistant to train a helpful and
safe AI assistant. This preference dataset has two subsets namely helpfulness
and harmlessness (Bai et al., 2022; Ganguli et al., 2022). In our experiments,
we only use a random sample of the helpfulness subset of the data with the
size of roughly 10,300 samples.

WebGPT: WebGPT (Nakano et al., 2021) dataset includes long-form question
answering preference dataset annotated by humans for reward modeling. After
cleaning this dataset, we get 17,814 samples from this dataset.

###  3.2 Experimental Setup

We start our experiments by training a Llama-2-7B SFT model using the Open
Assistant conversation dataset. We specifically choose this SFT dataset for
two primary reasons: (1) the same SFT model is used across different
preference datasets in RLHF. This helps to examine the influence of preference
dataset on our proposed method, and (2) the utilization of high-quality chat
data leads to the improved performance of SFT models (Dettmers et al., 2023).
For SFT step, we employ linear learning rate schedule with starting learning
rate of $2\times 10^{-5}$, effective batch size of 64, number of epochs of 2,
weight decay of 0.1, and a sequence length of 4096 tokens. We do not use LoRA
(Hu et al., 2021) finetuning in the SFT step.

In the response generation step during the PDGRS (1) of our proposed method,
we generate a total of $k=16$ responses for each prompt, with the following
decoding parameters: a maximum of 512 new tokens, a top-k value of 50, a top-p
value of 0.98, and a sampling temperature of 1. We applied PDGRS on 10,300
samples from Anthropic/HH-RLHF, and 12,193 samples from WebGPT.

To assess the quality of generated responses, we employ the pythia-6.9B reward
model developed by Open Assistant, denoted as pythia-6.9B-RM-OA in our
experiments (OpenAssistant, 2023). This reward model is trained on a diverse
set of datasets, including Open Assistant preference (Köpf et al., 2023),
Anthropic (Bai et al., 2022; Ganguli et al., 2022), SHP (Ethayarajh et al.,
2022), hellaswag (Zellers et al., 2019), WebGPT (Nakano et al., 2021), and
summary pairs (Stiennon et al., 2020a). To control the impact of reward model
preference data with our proposed method, we also trained a pythia-6.9B reward
model using only WebGPT preference dataset, denoted as pythia-6.9B-RM-WG in
our experiments. We use pythia-6.9B (Biderman et al., 2023) as a base model
and train it for 1 epoch with learning rate of $1\times 10^{-5}$ with linear
learning rate schedule.

For DPO training in our experiments, we use cosine learning rate schedule with
an initial learning rate of $1\times 10^{-6}$, effective batch size of 64,
number of epochs of 4, $\beta=0.1$, and a sequence length of 4096 tokens. We
use LoRA with $\text{rank}=8$ to enable training Llama-2-7B models with
limited GPU resources.

For PPO (Schulman et al., 2017) training in our experiments, we use LoRA with
$\text{rank}=8$ and 8-bit quantization for both policy and reward models. We
adopt effective batch size of 64, learning rate of $2\times 10^{-5}$, and
Kullback-Leibler (KL) coefficient of 0.2. We train the policy model between
150-200 steps to converge.

###  3.3 Evaluation

Assessing alignment to human preference is challenging, but recent
developments have introduced specialized benchmarks like MT-Bench (Zheng et
al., 2023) and AlpacaEval (Li et al., 2023) to address this issue. These
benchmarks leverage strong LLM judges like GPT-4, providing a score that
strongly correlates with human preference ratings. We use the following
benchmarks to evaluate model’s performance on instruction following and
alignment to user intent:

MT-Bench: MT-Bench evaluation is based on GPT-4 judgement and achieves over
80% agreement with human preference. MT-bench is designed to test multi-turn
conversation and instruction-following ability of LLMs, covering 8 common
categories including writing, roleplay, extraction, reasoning, math, coding,
knowledge I (STEM), and knowledge II (humanities/social science). MT-Bench has
10 multi-turn questions for each category, and GPT-4 rates each turn’s
response on a scale of 1-10, with the final score being the mean over two
turns (Zheng et al., 2023).

AlpacaEval: It is an LLM-based automatic evaluation judged by GPT-4, where it
measures the pairwise win-rate against a baseline model (text-davinci-003). We
use 300 questions mostly focused on helpfulness from this benchmark in our
evaluations (Li et al., 2023).

##  4 Results and Ablations

This section presents our main results. We show sample model completions in
appendix C. To comprehensively assess the effectiveness of our proposed
method, we employ a comparative analysis of various preference data generation
policies. These policies guide the selection of the superior model response,
denoted as $y_{w}$, and the inferior model response, denoated as $y_{l}$, from
a set of $k$ generated answers. The following preference data generation
policies are considered:

Best-vs-worst: This policy ranks the $k$ responses according to their
respective rewards and selects the response with the highest reward as $y_{w}$
and the response with the lowest reward as $y_{l}$.

Best-vs-random: This policy selects the response with the highest reward as
$y_{w}$, while $y_{l}$ is chosen randomly from the remaining $k-1$ responses.

Original annotation: This policy chooses $y_{w}$ and $y_{l}$ from the original
preference data annotated by humans or larger models.

Rejection Sampling: This method utilizes only the response with the highest
reward as $y_{w}$ for each prompt $x$ and performs 1-step SFT using samples
$(x,y_{w})$.

PPO: This method dynamically generates responses $y$ for a batch of prompts
and employs a reward model for their assessment. Subsequently, it maximizes
the cumulative reward during RLHF training. PPO does not use any pre-generated
responses.

Proposed method: We consider all possible combinations of $y_{w}$ and $y_{l}$
from the $k$ answers. We keep all combinations with reward gap larger than
predefined threshold $\eta$.

We also investigate the performance of our proposed method under varying
thresholds $\eta$, maintaining a constant temperature $\tau$. Generally, lower
values of $\eta$ and $\tau$ lead to an increased size of preference data in
our proposed method. In contrast to other policies that limit the sample size
to the original preference data $\mathcal{D_{\text{RM}}}$ size, our proposed
method considers the reward distribution per prompt and identifies more
contrastive samples $(y_{w},y_{l})$, thereby resulting in enhanced
performance. Table 1 and Table 2 summarizes the results on the MT-Bench and
AlpacaEval benchmarks for Anthropic/HH-RLHF and WebGPT datasets, respectively.

Policy | Reward Model | Sample Size | Threshold | MT-Bench (Avg score) | AlpacaEval (win %)
---|---|---|---|---|---
SFT | - | 9,000 | - | 5.12 | $60.20_{2.84}$
Best-vs-worst | pythia-6.9B-RM-OA | 10,300 | - | 5.34 | $72.48_{2.59}$
Best-vs-random | pythia-6.9B-RM-OA | 10,300 | - | 5.07 | $70.00_{2.64}$
Original annotation | - | 10,300 | - | 5.26 | $65.33_{2.75}$
Rejection Sampling | pythia-6.9B-RM-OA | 10,300 | - | 4.84 | $60.20_{2.84}$
PPO | pythia-6.9B-RM-OA | 10,300 | - | 5.22 | $69.23_{2.67}$
Proposed method | pythia-6.9B-RM-OA | 12,795 | 0.90 | 5.44 | $73.75_{2.54}$
Proposed method | pythia-6.9B-RM-OA | 32,640 | 0.85 | $\mathbf{5.49}$ | $74.17_{2.53}$
Proposed method | pythia-6.9B-RM-OA | 63,938 | 0.80 | 5.36 | $\mathbf{79.67_{2.33}}$

Table 1: Performance of competing methods on Anthropic/HH-RLHF dataset using
different policies on MT-Bench and AlpacaEval benchmarks. A dash (-) sign
indicates that the specific parameters is not needed. The SFT model is trained
on Open Assistant conversation dataset. The base LLM for all experiments is
Llama-2-7B. The temperature $\tau$ is set to be 1 in our proposed method. The
subscript in the AlpacaEval win rate indicates the standard error.

Policy | Reward Model | Sample Size | Threshold | MT-Bench (Avg score) | AlpacaEval (win %)
---|---|---|---|---|---
SFT | - | 9,000 | - | 5.12 | $60.20_{2.84}$
Best-vs-worst | pythia-6.9B-RM-WG | 12,193 | - | 5.24 | $69.17_{2.67}$
Best-vs-random | pythia-6.9B-RM-WG | 12,193 | - | 5.04 | $69.90_{2.66}$
Original annotation | - | 12,193 | - | 5.14 | $65.55_{2.75}$
Rejection Sampling | pythia-6.9B-RM-WG | 12,193 | - | 5.15 | $68.17_{2.69}$
PPO | pythia-6.9B-RM-WG | 12,193 | - | 4.95 | $65.17_{2.75}$
Proposed method | pythia-6.9B-RM-WG | 3,449 | 0.90 | 5.13 | $68.90_{2.68}$
Proposed method | pythia-6.9B-RM-WG | 11,458 | 0.85 | 5.24 | $72.33_{2.59}$
Proposed method | pythia-6.9B-RM-WG | 29,698 | 0.80 | 5.31 | $72.91_{2.57}$
Rejection Sampling | pythia-6.9B-RM-OA | 12,193 | - | 5.23 | $71.00_{2.62}$
PPO | pythia-6.9B-RM-OA | 12,193 | - | 5.11 | $69.83_{2.65}$
Proposed method | pythia-6.9B-RM-OA | 12,611 | 0.90 | $\mathbf{5.35}$ | $71.91_{2.60}$
Proposed method | pythia-6.9B-RM-OA | 33,755 | 0.85 | $\mathbf{5.35}$ | $\mathbf{74.92_{2.51}}$
Proposed method | pythia-6.9B-RM-OA | 70,510 | 0.80 | 5.20 | $67.56_{2.71}$

Table 2: Performance of competing methods on WebGPT dataset using different
policies on MT-Bench and AlpacaEval benchmarks. A dash (-) sign indicates that
the specific parameters is not needed. The SFT model is trained on Open
Assistant conversation dataset. The base LLM for all experiments is
Llama-2-7B. The temperature $\tau$ is set to be 1 in our proposed method. The
subscript in the AlpacaEval win rate indicates the standard error.

In Table 1 and Table 2, our proposed method consistently demonstrates superior
performance compared to other methods on the Anthropic/HH-RLHF and WebGPT
datasets. All policies exhibit better performance than the SFT model, except
for the best-vs-random policy and PPO on MT-Bench benchmark. This can be
attributed to the best-vs-random policy’s random selection of $y_{l}$, which,
if it happens to select a high-quality response as $y_{l}$, can make
optimization process challenging and noisy.

The best-vs-worst policy consistently outperforms other policies except our
proposed method, primarily because it consistently selects high-quality pairs
of contrastive samples. Furthermore, the best-vs-worst policy also outperforms
the original annotation policy, despite both policies utilizing the same
amount of data. This observation holds true even for pythia-6.9B-RM-WG, which
is trained on the same original annotation dataset. The enhanced performance
of the best-vs-worst policy can be attributed to the fact that both $y_{l}$
and $y_{w}$ are sampled from the SFT model, as opposed to utilizing responses
from another language model or human annotation.

Rejection sampling method is not performing very well which can be attributed
the following factors: (1) it only utilizes $y_{w}$ for alignment and does not
take advantage of the remaining $k-1$ responses, (2) it applies 1-step SFT
which can be susceptible to overfitting issues.

The performance of PPO on Anthropic/HH-RLHF surpasses that of other methods,
with the exception of our proposed approach and the best-vs-worst policy.
However, the performance of PPO on MT-Bench average scores declines when
applied to WebGPT, primarily attributed to a low 2-turn score on MT-Bench, as
detailed in Tables 4 and 5 in appendix. This can be attributed to the prompt
types in the datasets, where the Anthropic/HH-RLHF dataset comprises prompts
featuring multi-turn conversations between humans and AI assistants, while the
WebGPT dataset exclusively involves single-turn questions. Consequently, PPO
indicates an enhancement in second-turn performance on the Anthropic/HH-RLHF
dataset in comparison to WebGPT within the MT-Bench benchmark.

How does changing the threshold $\eta$ affect our performance of proposed
method? Our proposed method takes into account the reward distribution per
prompt to determine pairs of $y_{l}$ and $y_{w}$ by assessing the reward gap.
Lower values of $\eta$ lead to an increased generation of preference data
within our proposed method because it allows selection of samples with smaller
reward gaps. However, setting $\eta$ too low may lead to $y_{l}$ and $y_{w}$
being similar in quality, potentially impeding the optimization process and
the convergence. In both datasets, reducing $\eta$ from 0.90 to 0.85 yields
improved performance as it increases preference data generation without
compromising quality. However, lowering $\eta$ further, from 0.85 to 0.80,
results in a slight performance decline in two cases when using the
pythia-6.9B-RM-OA reward model on MT-Bench bench. This can be attributed to a
substantial increase in sample size, preventing the convergence of the
optimization process and reduced quality of generated preference data.

How does the reward model impact the results? In our experiments, we employ
two reward models with identical architectures but trained on different amount
of preference data. Specifically, pythia-6.9B-RM-OA is trained on a larger
preference dataset, while pythia-6.9B-RM-WG is exclusively trained on the
WebGPT portion of preference datasets (detailed information is provided in
section 3.2). As a result, pythia-6.9B-RM-OA exhibits superior performance in
evaluating response quality in line with human preferences. Typically, a more
effective reward model tends to have a higher variance in its reward
distribution with longer tails, as it can differentiate the good and bad
responses in a broader range. In contrast, lower quality reward models often
have most rewards concentrated around the mean. Figure 2 shows the reward gap
distribution for both reward models on WebGPT dataset. The red dashed line
represents the threshold for preference data selection in the histograms. As
depicted in Figure 2, the histogram for pythia-6.9B-RM-OA exhibits longer
tails and greater variance, leading to a higher number of preference samples
falling in after the dashed line. As shown in the reward model ablation study
in Table 2, the pythia-6.9B-RM-OA reward model enhances the performance of our
proposed method, PPO, and the rejection sampling method, underscoring the
significance of a high-quality reward model. Nevertheless, the results
demonstrate the robustness of our proposed method to variations in reward
model quality, as it outperforms other methods even when employing the
pythia-6.9B-RM-WG reward model. Additionally, results indicates that the PPO
method is more sensitive to the quality of the reward model as transitioning
from the pythia-6.9B-RM-OA to the pythia-6.9B-RM-WG reward model notably
diminishes model performance across both benchmarks.

Figure 2: Histograms of reward gap for WebGPT datasets with different reward
models. The red dashed line represents the threshold value of 0.85 for
preference data selection. Mean and standard deviation values of reward gaps
are shown in the histograms.

How do multi-turn prompts influence performance?

The Anthropic/HH-RLHF dataset includes prompts comprising multi-turn
conversations between humans and AI assistants, while the WebGPT dataset
exclusively consists of single-turn questions. Through a comparative analysis
of MT-bench multi-turn scores presented in Table 4 and 5, it is evident that
the incorporation of multi-turn prompts enhances the 2-turn scores for both
our proposed method and PPO. Notably, our proposed method outperforms PPO.
However, there is no significant impact on performance observed in the
AlpacaEval benchmark as it employs only single-turn evaluation prompts.
Consequently, the inclusion of multi-turn prompts in RLHF is crucial for
improving the multi-turn capabilities of language models.

How does changing the temperature $\tau$ affect our method’s performance? To
analyze the impact of temperature on our proposed method, we design an
ablation study where we keep the threshold $\eta=0.85$ the same and change the
value of temperature. Decreasing $\tau$ leads to a heavy-tailed reward gap
distribution, generating more preference samples. Conversely, increasing
$\tau$ creates a thin-tailed reward gap distribution, resulting in fewer
preference samples being generated. We conduct this ablation study on
Anthropic/HH-RLHF dataset and Table 3 summarizes the results. A lower
temperature value increases the sample size, enhancing overall performance. As
the sample size increases, a diverse variety of preference pairs emerges,
encompassing both easy (with a higher reward gap) and hard (with a lower
reward gap) instances. The inclusion of easy preference pairs significantly
helps with the convergence of the DPO optimization process, facilitating the
attainment of a superior model. In contrast, solely using hard samples may
impede the optimization process, resulting in a failure to converge and
yielding a policy model of inferior quality.

Policy | Sample Size | Threshold | Temperature | MT-Bench (score) | AlpacaEval (win %)
---|---|---|---|---|---
Proposed method | 63,796 | 0.85 | 0.8 | 5.31 | $\mathbf{77.33_{2.42}}$
Proposed method | 45,668 | 0.85 | 0.9 | $\mathbf{5.51}$ | $76.92_{2.44}$
Proposed method | 32,640 | 0.85 | 1 | 5.49 | $74.17_{2.53}$
Proposed method | 22,951 | 0.85 | 1.1 | 5.40 | $71.00_{2.62}$
Proposed method | 16,160 | 0.85 | 1.2 | 5.43 | $71.33_{2.62}$

Table 3: Performance of our proposed method on Anthropic/HH-RLHF dataset using
different temperature $\tau$ on MT-Bench and AlpacaEval benchmarks. The base
LLM for all experiments is Llama-2-7B. The reward model for all methods is
pythia-6.9B-RM-OA. The subscript in the AlpacaEval win rate indicates the
standard error.

##  5 Discussion and Conclusion

In this paper, we proposed RS-DPO method that generates responses from the
large language model directly, and leverages RS to sample synthetic preference
pairs, and DPO for RLHF training. Extensive experiments show the effectiveness
of RS-DPO compared to existing methods including rejection sampling (RS),
proximal policy optimization (PPO) and direct preference optimization (DPO).
Additionally, RS-DPO is stable, and is not as sensitive to the quality of the
reward model as other methods. Our proposed method also offers a more
efficient and less time-consuming solution for the alignment task as compared
to PPO, minimizing resource requirements.

During RLHF training, PPO conducts online sampling from the policy model and
evaluates them using the loaded reward model in real-time. Consequently, PPO
necessitates loading three models during training: the initial SFT, policy
model, and reward model, demanding a significant amount of GPU memory and
decelerating the training process. Furthermore, the online sampling from the
policy model incurs increased memory consumption as the generated sequences
lengthen. In practical terms, even with 1-2 moderate GPUs, training a small-
scale (e.g., 7B) LLM using PPO is unfeasible. In our experiments, we had 8
A-100 GPUs each having 40G memory, but we resorted to 8-bit quantization of
both the policy and reward model to circumvent GPU memory constraints. Our
proposed method conducts response sampling offline from SFT and constructs a
dataset of synthetic preference data to bypass the high computational cost of
PPO, while remaining viable on 1-2 moderate GPUs. Notably, the operational
cost of running DPO and RS-DPO is identical; the sole disparity lies in RS-DPO
performing offline SFT sampling, rendering our proposed method an on-policy
reinforcement learning approach.

Moreover, as emphasized by prior researches (Singhal et al., 2023), PPO
represents an unstable process prone to sensitivity towards reward model
quality and hyperparameters, necessitating multiple runs to converge to a
satisfactory model. For instance, in Table 2, training two models using PPO
while altering the reward model quality from high (pythia-6.9B-RM-OA) to lower
(pythia-6.9B-RM-WG) significantly impacted the resulting model’s quality,
underscoring PPO’s sensitivity. Conversely, our proposed method exhibits
robustness against reward model quality, requiring only a single run to train
each model successfully.

##  6 Limitations

A limitation of our work is its primary focus on the helpfulness objective
derived from open-source preference datasets. Consequently, the
generalizability of our findings to other objectives, such as harmlessness may
be constrained. While we have demonstrated the efficacy of our proposed method
on language models at 7B scale, we acknowledge that we have yet to subject our
method to larger or close-source models. Despite these limitations, we
maintain confidence that our proposed method demonstrates robustness towards
reward model quality, and needs fewer resources compared to existing methods
of RLHF training.

## References

  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023.  Pythia: A suite for analyzing large language models across training and scaling.  In _International Conference on Machine Learning_ , pages 2397–2430. PMLR.
  * Bradley and Terry (1952) Ralph Allan Bradley and Milton E Terry. 1952.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.  Qlora: Efficient finetuning of quantized llms.  _arXiv preprint arXiv:2305.14314_.
  * Dong et al. (2023) Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023.  Raft: Reward ranked finetuning for generative foundation model alignment.  _arXiv preprint arXiv:2304.06767_.
  * Ethayarajh et al. (2022) Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2022.  Understanding dataset difficulty with $\mathcal{V}$-usable information.  In _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 5988–6008. PMLR.
  * Ganguli et al. (2022) Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022\.  Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.  _arXiv preprint arXiv:2209.07858_.
  * Gulcehre et al. (2023) Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. 2023.  Reinforced self-training (rest) for language modeling.  _arXiv preprint arXiv:2308.08998_.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_.
  * Köpf et al. (2023) Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. 2023.  Openassistant conversations–democratizing large language model alignment.  _arXiv preprint arXiv:2304.07327_.
  * Lee et al. (2023) Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. 2023.  Rlaif: Scaling reinforcement learning from human feedback with ai feedback.  _arXiv preprint arXiv:2309.00267_.
  * Li et al. (2023) Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Alpacaeval: An automatic evaluator of instruction-following models.  https://github.com/tatsu-lab/alpaca_eval.
  * Liu et al. (2023) Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. 2023.  Statistical rejection sampling improves preference optimization.  _arXiv preprint arXiv:2309.06657_.
  * Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021.  Webgpt: Browser-assisted question-answering with human feedback.  _arXiv preprint arXiv:2112.09332_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774.
  * OpenAssistant (2023) OpenAssistant. 2023.  Openassistant/oasst-rm-2-pythia-6.9b-epoch-1.  Accessed: 2023.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_.
  * Rajbhandari et al. (2020) Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020.  Zero: Memory optimizations toward training trillion parameter models.  In _SC20: International Conference for High Performance Computing, Networking, Storage and Analysis_ , pages 1–16. IEEE.
  * Santacroce et al. (2023) Michael Santacroce, Yadong Lu, Han Yu, Yuanzhi Li, and Yelong Shen. 2023.  Efficient rlhf: Reducing the memory usage of ppo.  _arXiv preprint arXiv:2309.00754_.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017\.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_.
  * Singhal et al. (2023) Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2023.  A long way to go: Investigating length correlations in rlhf.  _arXiv preprint arXiv:2310.03716_.
  * Stiennon et al. (2020a) Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020a.  Learning to summarize from human feedback.  In _NeurIPS_.
  * Stiennon et al. (2020b) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020b.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. 2023.  Zephyr: Direct distillation of lm alignment.  _arXiv preprint arXiv:2310.16944_.
  * von Werra et al. (2020) Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. 2020.  Trl: Transformer reinforcement learning.  https://github.com/huggingface/trl.
  * Wang et al. (2022) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022.  Self-instruct: Aligning language model with self generated instructions.  _arXiv preprint arXiv:2212.10560_.
  * Wang et al. (2023a) Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023a.  Aligning large language models with human: A survey.  _arXiv preprint arXiv:2307.12966_.
  * Wang et al. (2023b) Ziqi Wang, Le Hou, Tianjian Lu, Yuexin Wu, Yunxuan Li, Hongkun Yu, and Heng Ji. 2023b.  Enable language models to implicitly learn self-improvement from data.  _arXiv preprint arXiv:2310.00898_.
  * Williams (2004) Ronald J. Williams. 2004.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Machine Learning_ , 8:229–256.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  Hellaswag: Can a machine really finish your sentence?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023.  Judging llm-as-a-judge with mt-bench and chatbot arena.
  * Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_.

##  Appendix A MT-Bench Benchmark Multi-turn Results

MT-bench consists of a multi-turn question set which is deigned for testing
the multi-turn conversation and instruction-following ability of LLMs. In the
section, we present the MT-bench scores for all individual turns in Tables 4
and 5.

Policy | Reward Model | Sample Size | Threshold | Turn-1 | Turn-2 | Average
---|---|---|---|---|---|---
SFT | - | 9,000 | - | 5.70 | 4.54 | 5.12
Best-vs-worst | pythia-6.9B-RM-OA | 10,300 | - | 6.06 | 4.61 | 5.34
Best-vs-random | pythia-6.9B-RM-OA | 10,300 | - | 5.77 | 4.38 | 5.07
Original annotation | - | 10,300 | - | 5.89 | 4.62 | 5.26
Rejection Sampling | pythia-6.9B-RM-OA | 10,300 | - | 5.54 | 4.13 | 4.84
PPO | pythia-6.9B-RM-OA | 10,300 | - | 6.03 | 4.41 | 5.22
Proposed method | pythia-6.9B-RM-OA | 12,795 | 0.90 | 5.96 | 4.91 | 5.44
Proposed method | pythia-6.9B-RM-OA | 32,640 | 0.85 | 6.18 | 4.81 | 5.49
Proposed method | pythia-6.9B-RM-OA | 63,938 | 0.80 | 6.07 | 4.63 | 5.36

Table 4: Performance of competing methods on Anthropic/HH-RLHF dataset using
different policies on MT-Bench benchmark. We report turn-1, turn-2, and
average score from MT-Bench judged by GPT-4. A dash (-) sign indicates that
the specific parameters is not needed. The SFT model is trained on Open
Assistant conversation dataset. The base LLM for all experiments is
Llama-2-7B.

Policy | Reward Model | Sample Size | Threshold | Turn-1 | Turn-2 | Average
---|---|---|---|---|---|---
SFT | - | 9,000 | - | 5.70 | 4.54 | 5.12
Best-vs-worst | pythia-6.9B-RM-WG | 12,193 | - | 5.85 | 4.63 | 5.24
Best-vs-random | pythia-6.9B-RM-WG | 12,193 | - | 5.61 | 4.45 | 5.04
Original annotation | - | 12,193 | - | 5.80 | 4.48 | 5.14
Rejection Sampling | pythia-6.9B-RM-WG | 12,193 | - | 5.66 | 4.63 | 5.15
PPO | pythia-6.9B-RM-WG | 12,193 | - | 5.64 | 4.26 | 4.95
Proposed method | pythia-6.9B-RM-WG | 3,449 | 0.90 | 5.83 | 4.44 | 5.13
Proposed method | pythia-6.9B-RM-WG | 11,458 | 0.85 | 5.86 | 4.63 | 5.24
Proposed method | pythia-6.9B-RM-WG | 29,698 | 0.80 | 5.87 | 4.73 | 5.31
Rejection Sampling | pythia-6.9B-RM-OA | 12,193 | - | 5.94 | 4.49 | 5.23
PPO | pythia-6.9B-RM-OA | 12,193 | - | 5.94 | 4.27 | 5.11
Proposed method | pythia-6.9B-RM-OA | 12,611 | 0.90 | 5.91 | 4.77 | 5.35
Proposed method | pythia-6.9B-RM-OA | 33,755 | 0.85 | 6.13 | 4.55 | 5.35
Proposed method | pythia-6.9B-RM-OA | 70,510 | 0.80 | 5.96 | 4.42 | 5.20

Table 5: Performance of competing methods on WebGPT dataset using different
policies on MT-Bench benchmark. We report turn-1, turn-2, and average score
from MT-Bench judged by GPT-4. A dash (-) sign indicates that the specific
parameters is not needed. The SFT model is trained on Open Assistant
conversation dataset. The base LLM for all experiments is Llama-2-7B.

##  Appendix B DPO Reward Accuracy and Reward Margin

By employing our proposed PDGRS methodology to generate preference datasets,
we leverage the DPO method to fine-tune the policy model, enhancing its
alignment with human preferences. Figures 3 and 4 illustrate the reward
margins and accuracies achieved through DPO training across various methods on
the hold-out evaluation datasets for Anthropic/HH-RLHF and WebGPT,
respectively. According to the results, we observe significant correlation
between increased reward margins, accuracies, and improved model performance.
Our proposed preference data generation method indicates superior reward
accuracy and margin in the plots, thereby underscoring the high data quality
in our preference data generation approach.

Figure 3: The left and right plots depict the reward accuracy and reward
margin, respectively, of competing methods during DPO training on the
Anthropic/HH-RLHF dataset.  Figure 4: The left and right plots display the
reward accuracy and reward margin, respectively, of competing methods during
DPO training on the WebGPT dataset.

##  Appendix C Qualitative Examples

To conduct a qualitative comparison of model responses trained through various
methods, we select sample prompts from two benchmark datasets, namely MT-Bench
and AlpacaEval. Subsequently, responses are generated across all candidate
models. The results of this comparative analysis are presented in Tables 6, 8
and 7.

Prompt: You can see a beautiful red house to your left and a hypnotic
greenhouse to your right, an attractive heated pink place in the front. So,
where is the White House?
---
Policy |  Response
Proposed Method |  The White House is the official residence and workplace of the president of the United States. It is located at 1600 Pennsylvania Avenue NW in Washington, D.C. It is a complex of buildings that includes the West Wing, the East Wing, the Residence, and the Eisenhower Executive Office Building. The White House is often referred to as "the People’s House" because it is the symbol of the American people and the presidency. It is a historic landmark and a popular tourist destination. The red house, greenhouse, and heated pink place are not located near the White House, but they are likely other attractions or landmarks in the area. The White House can be easily recognized by its iconic white exterior and its prominent location in Washington, D.C.
SFT |  The White House is the large, white building in the center of the picture. It is located in Washington, D.C., and is the official residence and workplace of the president of the United States. The red house to the left is the Blair House, which is used for official guest accommodations. The greenhouse to the right is the United States Botanic Garden, which is a research and educational institution. The attractive heated pink place in the front is the Ellipse, a park located between the White House and the Washington Monument.
Original Annotation |  The White House is the building in the center of the picture. It is the building with the American flag flying from the roof. It is the building with the balcony and the columns. It is the building with the lawn and the trees. It is the building with the fence and the gate. It is the building with the flagpoles and the flag. It is the building with the windows and the doors. It is the building with the roof and the chimneys. It is the building with the stairs and the walkway. It is the building with the lights and the sign. It is the building with the name "The White House" written on it. It is the building that is the home of the President of the United States.
Rejection Sampling |  The White House is the large, white building at the center of the image, directly in front of the greenhouse. The red house to the left is known as the Blair House, and it is located directly across the street from the White House. The greenhouse is the United States Department of Agriculture (USDA) Research Center, and the attractive heated pink place in the front is the USDA South Building. The White House is the official residence and principal workplace of the president of the United States. It is located at 1600 Pennsylvania Avenue NW in Washington, D.C., and has been the residence of every U.S. president since John Adams in 1800. The Blair House is a historic house located at 1651 Pennsylvania Avenue NW, across the street from the White House. It serves as the official guest house for the president of the United States and is used for official entertaining and visiting dignitaries. The USDA Research Center is a federal research facility that is part of the U.S. Department of Agriculture. It is located at 10301 Baltimore Avenue in College Park, Maryland, and is dedicated to research in agriculture, food, and natural resources. The USDA South Building is a federal office building that is part of the U.S. Department of Agriculture. It is located at 14th Street and Independence Avenue SW in Washington, D.C., and houses various USDA offices and programs.
PPO |  The White House is the building in the center of the picture. It is the building with the flag flying from the roof. It is the building with the balcony and the columns. It is the building with the lawn and the trees. It is the building with the fence and the gate. It is the building with the flagpoles and the flag. It is the building with the windows and the doors. It is the building with the roof and the chimneys. It is the building with the columns and the arches. It is the building with the dome and the clock. It is the building with the flag and the stars. It is the building with the name "The White House" written on it. It is the building that is the home of the President of the United States.
Table 6: Comparing generations obtained for a cherry-picked MT-Bench prompt
from Llama-2-7B trained based on different methods. Prompt: What are the five
most common causes of stress among people?
---
Policy |  Response
Proposed Method |  The five most common causes of stress among people are: Work: Job-related stress is one of the most common causes of stress, as it can involve long hours, tight deadlines, and demanding workloads. Money: Financial stress is another common cause of stress, as it can involve worries about debt, income, and financial stability. Relationships: Conflict with partners, family members, or friends can cause stress, as can the pressure to maintain healthy relationships. Health: Illness, chronic pain, or concerns about one’s own health or the health of a loved one can cause stress. Personal problems: Events such as divorce, bereavement, or other personal difficulties can cause stress. It’s worth noting that these causes of stress can be interconnected, and that different people may experience stress for different reasons. It’s important to take steps to manage stress and maintain good mental health.
SFT |  The five most common causes of stress among people are: 1\. Work or career-related issues 2\. Money problems 3\. Relationship difficulties 4\. Health concerns 5\. Personal problems It is important to note that stress can also be caused by other factors such as personal beliefs, values, and experiences. Additionally, the perception of stress can vary from person to person, and what may be a source of stress for one person may not be a source of stress for another.
Original Annotation |  The five most common causes of stress among people are: 1\. Work or school 2\. Money or financial concerns 3\. Family responsibilities or relationships 4\. Health concerns 5\. Personal or emotional issues It’s important to note that the causes of stress can vary from person to person and may also be influenced by cultural and environmental factors. Additionally, it’s possible for more than one cause to contribute to an individual’s stress.
Rejection Sampling |  Stress is a complex issue with many possible causes. The most common causes of stress are often related to a person’s work or family life, and include: financial difficulties, work-related stress, relationship problems, and health concerns. Other common causes of stress include: personal problems, such as a difficult relationship, divorce, or the death of a loved one; and life changes, such as moving, getting married, or having a baby. Stress can also be caused by a person’s environment, including: living in an unsafe neighborhood, being a victim of crime, or experiencing discrimination.
PPO |  The five most common causes of stress among people are: 1\. Work-related stress 2\. Money problems 3\. Relationship difficulties 4\. Health issues 5\. Personal problems It is important to note that the causes of stress can vary from person to person and may be influenced by a variety of factors, including personal experiences, lifestyle, and cultural background.
Table 7: Comparing generations obtained for a cherry-picked AlpacaEval prompt
from Llama-2-7B trained based on different methods. Prompt: Why does it feel
like less effort to watch 3 - hour long TV episodes back to back than a film?
---
Policy |  Response
Proposed Method |  There are a few reasons why it might feel like less effort to watch several hours of TV back to back than a single film: 1\. Scope: TV shows often have a longer scope than films, with more time to develop characters, world-building, and plot. This can make the experience of watching several hours of TV feel more immersive and less like a single event. 2\. Pacing: The pacing of TV shows is often more episodic, with each episode having a self-contained story that builds towards a larger arc over the course of the season. This can make it easier to follow the story and feel less overwhelmed by the amount of content. 3\. Investment: Watching several hours of TV can also create a sense of investment in the characters and story, which can make it feel less like a one-time event and more like an ongoing experience. 4\. Comfort: Watching TV can also be a comfortable and familiar activity, with a familiar format and rhythm that can be soothing and relaxing. This can make it easier to engage with the content for extended periods of time. However, it’s important to note that both films and TV shows can be enjoyable and engaging experiences, and the format preference is a personal one that depends on individual preferences and circumstances. Some people might prefer the immersive experience of a film, while others might find the longer format of TV more conducive to their lifestyle or entertainment preferences. Ultimately, the choice between films and TV shows is a matter of personal preference and can vary depending on the specific content, genre, and format preferences of the individual.
Original Annotation |  There are several reasons why it may feel like less effort to watch three-hour long TV episodes back to back than a film. Firstly, the shorter duration of a film may make it seem more daunting to commit to watching it in one sitting, while the longer duration of a TV episode may feel more manageable. Additionally, the episodic format of TV shows often allows for natural breaks and pacing, which can make it easier to stay engaged and avoid feeling overwhelmed. Another reason may be the structure of the content itself. TV shows tend to have more character development, plot twists, and subplots than films, which can make them feel more complex and engaging. On the other hand, films often have a more straightforward narrative structure and focus on a single story arc, which can make them feel more straightforward and easier to follow. Finally, the cultural context in which we consume media may also play a role. With the rise of binge-watching and streaming services, it has become more common for people to watch multiple episodes of a TV show in one sitting. This has created a cultural expectation that may make it feel more normal and acceptable to watch longer periods of time, while the traditional film-going experience may still feel like a more significant investment of time and effort. In summary, the combination of shorter duration, episodic structure, complex content, and cultural expectations may all contribute to the feeling that it is less effort to watch multiple TV episodes back to back than a film. However, it’s important to note that this is a subjective experience and everyone’s preferences and priorities may differ.
PPO |  There are several possible reasons why it might feel like less effort to watch three-hour long TV episodes back to back than a film. One reason could be that the longer format of the TV episodes allows for more breaks and pauses, which can make the viewing experience feel less taxing. Additionally, the pacing of TV episodes can be more leisurely, which can make the content feel less overwhelming and easier to digest. Finally, the longer time commitment required for a TV series might make it feel like a more manageable task than a two-hour film, which can feel like a significant investment of time and attention. Ultimately, the perception of effort involved in watching content may be influenced by a variety of factors beyond the inherent length or format of the content itself.
Table 8: Comparing generations obtained for a cherry-picked AlpacaEval prompt
from Llama-2-7B trained based on different methods.


††thanks: These authors contributed equally.††thanks: These authors
contributed equally.††thanks: These authors contributed equally.††thanks:
These authors contributed equally.

# Quantum paramagnetism in the decorated square-kagome antiferromagnet
Na6Cu7BiO4(PO4)4Cl3
– Supplemental Material –

Nils Niggemann  Dahlem Center for Complex Quantum Systems and Fachbereich
Physik, Freie Universität Berlin, 14195 Berlin, Germany  Helmholtz-Zentrum für
Materialien und Energie, Hahn-Meitner-Platz 1, 14109 Berlin, Germany
Department of Physics and Quantum Centre of Excellence for Diamond and
Emergent Materials (QuCenDiEM), Indian Institute of Technology Madras, Chennai
600036, India  Nikita Astrakhantsev  Department of Physics, University of
Zürich, Winterthurerstrasse 190, CH-8057 Zürich, Switzerland  Arnaud Ralko
Institut Néel, UPR2940, Université Grenoble Alpes, CNRS, Grenoble, FR-38042
France  Department of Physics and Quantum Centre of Excellence for Diamond and
Emergent Materials (QuCenDiEM), Indian Institute of Technology Madras, Chennai
600036, India  Francesco Ferrari  Institut für Theoretische Physik, Goethe
Universität Frankfurt, Max-von-Laue-Straße 1, 60438 Frankfurt am Main, Germany
Department of Physics and Quantum Centre of Excellence for Diamond and
Emergent Materials (QuCenDiEM), Indian Institute of Technology Madras, Chennai
600036, India  Atanu Maity  Department of Physics and Quantum Centre of
Excellence for Diamond and Emergent Materials (QuCenDiEM), Indian Institute of
Technology Madras, Chennai 600036, India  Tobias Müller  Institut für
Theoretische Physik und Astrophysik, Julius-Maximilians-Universität Würzburg,
Am Hubland, D-97074 Würzburg, Germany  Johannes Richter  Institut für Physik,
Otto-von-Guericke-Universität Magdeburg, P.O. Box 4120, 39016 Magdeburg,
Germany  Max-Planck-Institut für Physik Komplexer Systeme, Nöthnitzer Straße
38, D-01187 Dresden, Germany  Ronny Thomale  Institut für Theoretische Physik
und Astrophysik, Julius-Maximilians-Universität Würzburg, Am Hubland, D-97074
Würzburg, Germany  Department of Physics and Quantum Centre of Excellence for
Diamond and Emergent Materials (QuCenDiEM), Indian Institute of Technology
Madras, Chennai 600036, India  Titus Neupert  Department of Physics,
University of Zürich, Winterthurerstrasse 190, CH-8057 Zürich, Switzerland
Johannes Reuther  Dahlem Center for Complex Quantum Systems and Fachbereich
Physik, Freie Universität Berlin, 14195 Berlin, Germany  Helmholtz-Zentrum für
Materialien und Energie, Hahn-Meitner-Platz 1, 14109 Berlin, Germany
Department of Physics and Quantum Centre of Excellence for Diamond and
Emergent Materials (QuCenDiEM), Indian Institute of Technology Madras, Chennai
600036, India  Yasir Iqbal  Department of Physics and Quantum Centre of
Excellence for Diamond and Emergent Materials (QuCenDiEM), Indian Institute of
Technology Madras, Chennai 600036, India  Harald O. Jeschke  Research
Institute for Interdisciplinary Science, Okayama University, Okayama 700-8530,
Japan  Department of Physics and Quantum Centre of Excellence for Diamond and
Emergent Materials (QuCenDiEM), Indian Institute of Technology Madras, Chennai
600036, India

## Additional DFT information

Figure S1: Quality of energy mapping in Na6Cu7BiO4(PO4)4Cl3. 68 calculated DFT energies are very precisely matched by the Heisenberg Hamiltonian with 13 exchange interactions.  Table S1: Exchange interactions of Na6Cu7BiO4(PO4)4Cl3 obtained by DFT energy mapping as described in the main text. The line in bold face corresponds to the set of couplings that match the experimental Curie-Weiss temperature [1]. $U$ (eV) |  $J_{1}$ (K) |  $J_{2}$ (K) |  $J_{3}$ (K) |  $J_{4}$ (K) |  $J_{5}$ (K) |  $J_{6}$ (K) |  $J_{8}$ (K) |  $J_{9}$ (K) |  $J_{10}$ (K) |  $J_{13}$ (K) |  $J_{14}$ (K) |  $J_{15}$ (K) |  $J_{20}$ (K) |  $\theta_{\rm CW}$ (K)
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
6.5 | 113.6(8) | 191.5(7) | 158.7(1.4) | 48.1(4) | 5.8(4) | 0.6(4) | -0.8(7) | 3.4(8) | 66.6(2) | -2.2(6) | 2.0(4) | 0.3(4) | -0.1(4) | -218.7
6.66 | 109.1(8) | 186.2(7) | 155.3(1.4) | 46.9(4) | 5.3(4) | 0.6(4) | -1.1(7) | 3.2(8) | 64.6(2) | -2.2(6) | 1.9(4) | 0.3(4) | 0.0(4) | -212
7 | 100.1(7) | 175.5(6) | 148.1(1.1) | 44.6(4) | 4.4(4) | 0.5(3) | -1.5(5) | 2.9(7) | 60.5(2) | -2.1(5) | 1.8(3) | 0.2(4) | 0.0(3) | -198.6
7.5 | 88.2(6) | 160.8(5) | 137.8(9) | 41.3(3) | 3.2(3) | 0.5(3) | -1.9(4) | 2.6(6) | 54.9(2) | -2.1(4) | 1.6(3) | 0.2(3) | 0.0(2) | -180.5
8 | 77.7(5) | 147.3(4) | 128.0(7) | 38.2(2) | 2.3(3) | 0.4(2) | -2.2(4) | 2.2(5) | 49.8(1) | -2.0(3) | 1.5(2) | 0.2(2) | 0.0(2) | -164.1
8.5 | 68.3(4) | 134.8(3) | 118.6(6) | 35.3(2) | 1.6(2) | 0.4(2) | -2.4(3) | 2.0(4) | 45.3(1) | -1.8(3) | 1.4(2) | 0.1(2) | 0.0(2) | -149.1
$d$ (Å) | 3.11405 | 3.27801 | 4.40394 | 5.0088 | 5.26566 | 5.33981 | 5.72746 | 5.76923 | 6.01815 | 6.55602 | 7.08351 | 7.71676 | 8.22822 |

We show the full result of the energy mapping procedure for
Na6Cu7BiO4(PO4)4Cl3 in Table S1. For every considered value of the onsite
interaction strength $U$, a total of 68 energies were calculated with $4\times
4\times 4$ $\mathbf{k}$-point meshes. The line in bold face is interpolated to
yield the experimental Curie-Weiss temperature $\theta_{\rm CW}=-212$ K [1].
As discussed in the main text, only five of the thirteen determined exchange
interactions are substantial and are considered in the many-body calculations
for Na6Cu7BiO4(PO4)4Cl3. The only interlayer coupling $J_{15}$ we determined
is small enough that Na6Cu7BiO4(PO4)4Cl3 can be treated as a 2D system.

In Fig. S1, we show that the energy mapping approach works very well for
Na6Cu7BiO4(PO4)4Cl3. In $P1$ symmetry, all 14 Cu2+ moments can be set
independently, and for the 68 chosen spin configurations, the comparison
between DFT energy and fit to the Heisenberg Hamiltonian is excellent.

## Lattice Convention

Each layer of the compound Na6Cu7BiO4(PO4)4Cl3 is represented by a square-
kagome lattice with an additional site at the center of each shuriken (square
surrounded by four triangles). More precisely, we define the unit vectors

$\mathbf{a}_{1}=(1,0),\quad\mathbf{a}_{2}=(0,1),$ (S1)

and a unit cell consisting of seven sites, labelled from $A$ to $G$ as shown
in Fig. S2, whose relative coordinates are

$\displaystyle\bm{\delta}_{A}$ $\displaystyle=(0,1/2),\
\bm{\delta}_{B}=(1/4,3/4)$ $\displaystyle\bm{\delta}_{C}$
$\displaystyle=(1/4,1/4),\ \bm{\delta}_{D}=(3/4,1/4)$
$\displaystyle\bm{\delta}_{E}$ $\displaystyle=(3/4,3/4),\
\bm{\delta}_{F}=(1/2,1)$ $\displaystyle\bm{\delta}_{G}$
$\displaystyle=(2/4,2/4)$ (S2)

Figure S2: Idealized square-kagome unit cell. Sites A and F correspond to
Cu(1) ions, sites B, C, D, E to Cu(2) ions and site G to the Cu(3) ion in the
unit cell of Na6Cu7BiO4(PO4)4Cl3.

In this convention, all sites lie at rational fractions of the basis vectors,
which ensures a finite extended Brillouin zone over which the structure factor
is periodic. The lattice for this compound features three types of symmetry
inequivalent sites, namely those that lie on the vertices of each shuriken
($A,F$), the corners of each square ($B,C,D,E$) and the center sites $G$ [see
Fig. S2]. Although, the geometry of the lattice does not alter the physics
(once the couplings are fixed), for comparison with experiments, it is
required to compute Fourier-resolved quantities with respect to the lattice of
the original compound. As most numerical methods do not explicitly employ
real-space crystallographic coordinates but rather the more abstract graph of
sites and respective bonds, this amounts to a choice in the post-processing of
real-space data. In the following, we outline this step in detail. We define
the Fourier transform of the spin susceptibility as

$\displaystyle\chi(\mathbf{q})=\frac{1}{N}\sum_{i,j}\chi(\mathbf{r}_{i},\mathbf{r}_{j})e^{i\mathbf{q}\cdot(\mathbf{r}_{i}-\mathbf{r}_{j})}.$
(S3)

Here, the vectors $\mathbf{q},\mathbf{r}_{i}$ are three-dimensional,
accounting for the true crystallographic coordinates, and allowing for
distortions outside of the square-kagome planes which occur in the crystal
structure. We must therefore map each pair of sites of the the true 3D lattice
to a unique site in the idealized lattice.

Upon noting that the layers are decoupled (as we neglect the tiny $J_{15}$
exchange interaction) and correlations between two sites from different layers
are zero, we can immediately interpret the summation to go over sites in a
single layer only.

Figure S3: Mapping from 3D lattice to 2D lattice. To compute Eq. (S3), sites
are mapped to the idealized lattice convention. Visualization done using VESTA
[2].

The mapping we need to perform can then be written as
$f(n_{1},n_{2},0,n_{b})=(n_{1}^{\prime},n_{2}^{\prime},n_{b}^{\prime})$, where
we define
$\mathbf{r}_{i}=n_{1}\mathbf{a}_{1}+n_{2}\mathbf{a}_{2}+\mathbf{b}_{n_{b}}$.
Since the unit cell of the three-dimensional lattice can be enlarged compared
to the idealized one, it may be cumbersome to find this mapping directly. A
convenient strategy is to work in cartesian coordinates, mapping
$\mathbf{r}_{i}$ to a $2D$ point $(x,y)$ which lies on the idealized lattice.
Then, the cartesian coordinates $(x,y)$ can be transformed back into either a
lattice position $(n_{1},n_{2},n_{b})$, or a site index $i$. This is shown in
Fig. S3.

To compare with powder samples, we compute the powder averaged structure
factor

$S(Q)\equiv\frac{1}{4\pi}\int d\Omega S(\mathbf{q}),\quad Q=|\mathbf{q}|$ (S4)

and multiply by the form factor to account for nuclear scattering on Cu2+ ions
[3]

$\displaystyle f(Q)$
$\displaystyle=Ae^{-a\frac{Q^{2}}{4\pi}}+Be^{-b\frac{Q^{2}}{4\pi}}+C_{1}e^{-c\frac{Q^{2}}{4\pi}}+D_{1}$
(S5) $\displaystyle A$ $\displaystyle=0.0232,\ B=0.4023$ $\displaystyle C_{1}$
$\displaystyle=0.5882,\ D_{1}=-0.0137$ $\displaystyle a$
$\displaystyle=34.969,\ b=11.564,\ c=3.843.$

## Methods

Figure S4: Momentum-resolved equal-time spin structure factor calculated from
mVMC on the $6\times 6\times 6$ lattice for different values of
$J_{10}/J_{2}$. The Fourier transform is performed with respect to the ideal
lattice coordinates of Eq. (Lattice Convention). The wave functions were
obtained with the symmetrization technique, enforcing the trivial
representation. The solid and dashed lines denote the first and the extended
Brillouin zones, respectively. The corresponding energies of these wave
functions are given in Table S2.

###  .1 Many-variable variational Monte Carlo (mVMC)

In this work, we employ the many-variable variational Monte Carlo
implementation presented in Refs. [4, 5]. To apply mVMC, spin operators are
first mapped to pseudo-fermionic bilinears through the Abrikosov
representation

$\hat{\mathbf{S}}_{i}=\frac{1}{2}\sum_{\alpha,\beta=\downarrow,\uparrow}\hat{c}_{i,\alpha}^{\dagger}\bm{\sigma}_{\alpha,\beta}\hat{c}_{i,\beta}\,.$
(S6)

Inspired by Anderson’s resonating valence-bond wave function, the mVMC Ansatz
has the form

$|\phi_{\mbox{\footnotesize
pair}}\rangle=\hat{\mathcal{P}}^{\infty}_{\mbox{\footnotesize
G}}\exp\left(\sum\limits_{i,j}f_{i,j}\hat{c}^{\dagger}_{i,\uparrow}\hat{c}^{\dagger}_{j\downarrow}\right)|0\rangle,$
(S7)

where a fermionic wave function is constrained to a space of singly occupied
sites by means of the Gutzwiller projector
$\hat{\mathcal{P}}^{\infty}_{\mbox{\footnotesize
G}}=\prod_{i}(n_{i,\uparrow}-n_{i,\downarrow})^{2}$, where
$n_{i,\sigma}=\hat{c}^{\dagger}_{i,\sigma}\hat{c}_{i,\sigma}$. The wave-
function value $\langle\bm{\sigma}|\phi_{\mbox{\footnotesize pair}}\rangle$
for a specific spin configuration $|\bm{\sigma}\rangle$ is evaluated using the
Slater determinant of the matrix with elements $f_{i,j}$. The parameters
$f_{i,j}$ are optimized using the stochastic reconfiguration optimization
technique [6, 7, 8].

$J_{10}/J_{2}$ | $0.0$ | $0.117$ | $0.223$ | $0.350$ | $0.466$ | $0.583$
---|---|---|---|---|---|---
$E/J_{2}$ | $-0.4128$ | $-0.4128$ | $-0.4080$ | $-0.4305$ | $-0.4436$ | $-0.4687$
Table S2: Ground state energy $E/J_{2}$ per site obtained on the $6\times 6$
lattice within mVMC for different ratios of $J_{10}/J_{2}$. The statistical
error is kept around $\delta E/J_{2}=10^{-4}$.

We may force a wave function to transform as a specific irreducible
representation of the symmetry group. To this end, we apply its generators
until the symmetry orbit is exhausted

$|\Psi_{\xi}\rangle=\hat{P}|\Psi\rangle=\sum\limits_{n}\xi^{n}\hat{G}^{n}|\Psi\rangle,$
(S8)

where $\xi$ is the desired projection quantum number and $|\Psi_{\xi}\rangle$
the resulting symmetrized state. In the mVMC package, the projection onto the
total spin $S$ is performed by superposing the $SU(2)$–rotated wave functions
[5].

In Fig. S4, we show the equal-time spin structure factor for different values
of $J_{10}/J_{2}$ measured within mVMC on the $6\times 6$ lattice. The
corresponding wave function energies are given in Table S2. One may see that
the frustration grows until $J_{10}/J_{2}\approx 0.3$, which is manifested by
growing energy and progressively diffuse character of the structure factor,
and then rapidly decays with further increase in $J_{10}/J_{2}$.

###  .2 Variational Monte Carlo (VMC)

Analogously to mVMC, the fermionic variational Monte Carlo calculations of
this work make use of a Gutzwiller-projected wave function to approximate the
ground state of the spin Hamiltonian by minimization of the variational
energy. The VMC trial state reads

$|\Psi_{0}\rangle=\hat{\mathcal{J}}\hat{\mathcal{P}}^{\infty}_{\mbox{\footnotesize
G}}|\Phi_{0}\rangle,$ (S9)

where ${\hat{\mathcal{P}}}^{\infty}_{\mbox{\footnotesize G}}$ is the
Gutzwiller projector, $|\Phi_{0}\rangle$ is a fermionic Slater determinant [in
the Abrikosov fermion representation of spins Eq. (S6)] and
$\hat{\mathcal{J}}$ is a spin-spin Jastrow factor. The definition of the
fermionic state $|\Phi_{0}\rangle$ is done by the introduction of an auxiliary
fermionic Hamiltonian

$\hat{\mathcal{H}}_{\rm
0}=\sum_{i,j,\alpha}\chi_{ij}\hat{c}_{i,\alpha}^{\dagger}\hat{c}_{j,\alpha},$
(S10)

whose hopping amplitudes, $\chi_{ij}$, play the role of variational
parameters. The presence of the spin-spin Jastrow factor

$\hat{\mathcal{J}}=\exp\left(\sum_{i,j}v_{i,j}\hat{S}^{z}_{i}\hat{S}^{z}_{j}\right)$
(S11)

can enhance/suppress spin-spin correlations between lattice sites. We consider
a long-range Jastrow factor with pseudopotential parameters that depend on the
distance between sites, i.e., $v_{i,j}=v(\|\mathbf{r}_{i}-\mathbf{r}_{j}\|)$.
The variational parameters defining $|\Psi_{0}\rangle$ are optimized by means
of the stochastic reconfiguration method [6, 7]. The VMC calculations are
performed on finite-size lattices with periodic boundary conditions.
Specifically, we employ two kinds of fully-symmetric clusters, one defined by
the translation vectors $\mathbf{T}_{1}=L\mathbf{a}_{1}$ and
$\mathbf{T}_{2}=L\mathbf{a}_{2}$ (containing $N_{s}=7L^{2}$ sites), and the
other defined by the vectors $\mathbf{T}_{1}=L(\mathbf{a}_{1}+\mathbf{a}_{2})$
and $\mathbf{T}_{2}=L(\mathbf{a}_{1}-\mathbf{a}_{2})$ (containing
$N_{s}=14L^{2}$ sites). The biggest lattice employed in our VMC calculations
contains $N_{s}=700$ spins.

The best variational energy is obtained by a parametrization of the auxiliary
Hamiltonian in which the translational symmetry is broken. In particular, in
agreement with mVMC observations, the optimal hopping pattern requires a
doubling of the unit cell along $\mathbf{a}_{1}$ and $\mathbf{a}_{2}$
($2\times 2$ periodicity). Concerning point group symmetries, the VMC wave
function turns out to preserve the $C_{4}$ rotational symmetry around the
center of the $2\times 2$ supercell. The small $C_{4}$ to $C_{2}$ symmetry
breaking found by mVMC calculations is not captured by the VMC variational
ansatz. The spin structure factor $S(\mathbf{q})$ corresponding to the best
variational state for the Heisenberg Hamiltonian of Na6Cu7BiO4(PO4)4Cl3 is
shown in Fig. S5. Due to reflection symmetry-breaking, the absolute maxima of
$S(\mathbf{q})$ are located at $\mathbf{q}=\pm 2\pi(3.75,2.25)$ with the
follow-up maxima at $\mathbf{q}=\pm 2\pi(1.75,0.25)$. Compared to mVMC
results, we find a larger intensity of the peaks of the structure factor,
which can be ascribed to the presence of the Jastrow factor that enhances
antiferromagnetic correlations.

Figure S5: Equal-time spin structure factor $S(\mathbf{q})$ from VMC for the
Hamiltonian of Na6Cu7BiO4(PO4)4Cl3. The results are obtained on a cluster with
$N_{s}=448$ sites, defined by the translation vectors
$\mathbf{T}_{1}=8\mathbf{a}_{1}$ and $\mathbf{T}_{2}=8\mathbf{a}_{2}$, and
plotted within the reciprocal space of the idealized lattice. The solid and
dashed lines denote the first and the extended Brillouin zones, respectively.

###  .3 PMFRG

The PMFRG formalism expresses spin operators by three different flavors
($x,y,z$) of Majorana fermions

$S^{x}_{i}=-i\eta^{y}_{i}\eta^{z}_{i}\text{,}\qquad
S^{y}_{i}=-i\eta^{z}_{i}\eta^{x}_{i}\text{,}\qquad
S^{z}_{i}=-i\eta^{x}_{i}\eta^{y}_{i}\text{.}$ (S12)

This representation has the advantage that no unphysical states are
introduced, allowing for quantitatively correct predictions at finite
temperature [9, 10]. One of the key advantages of PMFRG is its high momentum
space resolution which allows for an easy detection of incommensurate order.
The only restriction is a cut-off of all correlations beyond a numerically
chosen maximum distance $L$. In a paramagnet, correlation lengths are
typically small and effectively zero beyond a characteristic correlation
length $\xi$, which makes this approximation virtually exact. When the
correlation length diverges, for instance at the critical point of a phase
transition, a finite-size scaling analysis may be performed to give accurate
estimates of the critical temperature [10].

Figure S6: Evolution of the maxima of the equal-time spin structure factor as
a function of temperature obtained from PMFRG. The color of the line
corresponds to the temperature, i.e., the yellow part corresponds to the
highest simulated temperature ($T=2.5J_{2}$, while the blue part indicates the
lowest temperatures. The background shows the low temperature structure factor
as a reference. Figure S7: From PMFRG, we show the evolution of the sublattice
dependent susceptibilities defined in Eq. (S13) as a function of $J_{10}$. The
top panel shows the respective maxima in momentum space, while the heat maps
show the full momentum dependence. Note that points are shown for three
different maximal correlation lengths of $L=14,16,18$ nearest neighbor bonds,
which all coincide, indicating absence of finite size effects.

Figure S6 shows the equal-time spin structure factor at low temperature,
featuring soft maxima at wave-vectors $(q_{x},q_{y})\sim(3\pi,\pi/2)$. We find
the position of these maxima to be temperature dependent. At high temperatures
$T\gg J_{2}$, the maxima reside at $(4\pi,\pi)$ and symmetry related points,
while the position of the peak shifts as temperature is lowered. In addition
to the spin susceptibility $\chi$ defined in Eq. (S3), one may also define a
sublattice-resolved susceptibility $\chi_{\alpha\beta},\
\alpha,\beta=1,2,\dots N_{\textrm{UC}}$, where $N_{\textrm{UC}}=7$ such that
$\chi(\mathbf{k})=\frac{1}{N_{\textrm{UC}}}\sum_{\alpha\beta}{\chi_{\alpha\beta}(\mathbf{k})}$.
To study the influence of the Cu(3) sites, Fig. S7 further displays the
susceptibilities

$\displaystyle\chi_{12}(\mathbf{k})$
$\displaystyle\equiv\frac{1}{N_{\textrm{UC}}}\sum_{\alpha\beta\neq
Cu(3)}\chi_{\alpha\beta}(\mathbf{k})$ $\displaystyle\chi_{3}(\mathbf{k})$
$\displaystyle\equiv\chi(\mathbf{k})-\chi_{12}(\mathbf{k}).$ (S13)

Here, $\chi_{12}$ quantifies the correlations between all non-Cu(3) sites, and
$\chi_{3}$ the contribution to the susceptibility upon adding the Cu(3) site.
The correlations between the Cu(1) and Cu(2) sites decrease upon strengthening
the $J_{10}$ bond while maxima of intensity become diffuse, indicating
increased frustration. To see this, note that the conservation of spin
magnitude dictates a sum rule within the extended BZ:
$\sum_{i}\mathbf{\hat{S}}_{i}^{2}=\sum_{\mathbf{q}}\langle\mathbf{\hat{S}}(\mathbf{-q})\mathbf{\hat{S}}(\mathbf{q}\rangle)=\mathrm{N}\frac{3}{4}$.
As a consequence, the susceptibility $\chi_{\alpha\beta}(\mathbf{q})$ for non-
interacting spins must be a non-zero constant (see, e.g., $\chi_{3}$ in Fig.
S13 at $J_{10}=0$). Switching on the couplings between sites, leaves the sum
rule intact for each sublattice but intensities may shift in reciprocal space
and develop features. Therefore, the observed decrease in the maximum of
$\chi_{12}$ indicates a more uniform distribution of different ordering
wavevectors $\mathbf{q}$ within the system –a signature of the effects of
frustration.

While the correlations between Cu(3) sites and the rest of the system
naturally grow as they are increasingly coupled, ordering tendencies are
absent as visible from the independence of our data on the maximal correlation
length $L$ above the accessible temperature range of $T\gtrsim 0.3J_{2}$.

In Fig. S8, we also show the pattern of equal time spin-spin correlations in
real space for the system with $J_{10}=0$ and $J_{10}\approx 0.42J_{2}$. It is
visible that adding the additional Cu(3) site does not lead to increased
correlations between the other sites and, in fact, slightly decreases them.
Overall, we find good agreement to mVMC in Fig. 2 of the main text.

Figure S8: Real space correlations
$\langle\hat{\mathbf{S}}_{i}\cdot\hat{\mathbf{S}}_{j}\rangle$ computed from
PMFRG by taking Cu(1) (left), Cu(2) (middle), and Cu(3) (right) as reference
sites $i$. The radii of the circle indicates the strength of correlations,
while the color red (blue) indicates a positive (negative) sign of the
correlations. Figure S9: Equal time structure factor $S(q,t=0)$ at
$T=0.3J_{2}$ for a cut at $q_{z}=0$ computed with respect to the true, three
dimensional crystallographic unit cell. Shown is the first BZ (solid), and the
approximate extended BZ (dashed). Figure S10: Powder average of the equal time
structure factor $S(Q,t=0)$ and the static magnetic suscepibility
$\chi(Q,i\nu_{n}=0)$ at $T=0.1J_{2}$ obtainde from PMFRG. As in Fig. S9, the
Fourier transform is performed with respect to the actual crystal lattice site
coordinates.

For comparison with experimental data, we compute the spin structure factor
with respect to the actual crystal lattice site coordinates, i.e., allowing
for atoms to be positioned outside of a pure 2D layer. Figure S9 shows the
corresponding momentum resolved equal-time spin structure factor projected
onto the $q_{x}-q_{y}$ plane taking $q_{z}=0$. We note that due to the
irrational coordinates of the atoms within the unit cell, the structure factor
is no longer periodic within any extended Brillouin zone. The structure factor
permits points of high intensity away from $q_{z}=0$, although they will
naturally be broad, as length of correlations is limited in $z$ direction due
to the layered structure of the material. If we restrict ourselves to a finite
box in momentum space between $|q_{x,y,z}|<8\pi/a$, the maxima are positioned
at incommensurate positions $\mathbf{k}=\left(0.61,6.95,0\right)\pi/a$. For
experimental comparison, in Fig. S10, we further provide the powder averaged
structure factor Eq. S4.

## Schwinger Boson mean field theory

As in the VMC section, it is possible to make a parton construction of the
spin by introducing Schwinger bosons instead of Abrikosov fermions. We thus
consider the decoupling:

$\hat{\mathbf{S}}_{i}=\frac{1}{2}\sum_{\alpha,\beta=\downarrow,\uparrow}\hat{b}_{i,\alpha}^{\dagger}\bm{\sigma}_{\alpha,\beta}\hat{b}_{i,\beta},$
(S14)

where $\hat{b}^{(+)}$ are now bosonic operators. The advantage of dealing with
bosons as opposed to fermions is the possibility of having Bose condensation
and thus to easily access quantum magnetic orders as well as
($\mathds{Z}_{2}$) quantum spin liquids, which are both treated on an equal
footing. Here, we recall the main lines of the approach noting that more
details can be found in [11, 12, 13, 14, 15]. In this approach, it is possible
to write Heisenberg terms as function of two SU(2) invariant operators
$\hat{A}$ and $\hat{B}$

$\displaystyle\hat{A}_{ij}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\left[\hat{b}_{i\uparrow}\hat{b}_{j\downarrow}-\hat{b}_{i\downarrow}\hat{b}_{j\uparrow}\right]$
$\displaystyle\hat{B}_{ij}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\left[\hat{b}_{i\uparrow}^{+}\hat{b}_{j\uparrow}+\hat{b}_{i\downarrow}^{+}\hat{b}_{j\downarrow}\right]$

as

$\displaystyle\hat{\mathbf{S}}_{i}\hat{\mathbf{S}}_{j}$ $\displaystyle=$
$\displaystyle:\hat{B}_{ij}^{+}\hat{B}_{ij}:-\hat{A}_{ij}^{+}\hat{A}_{ij},$
(S15)

where $::$ denotes the normal ordering. At the mean field level, the
Hamiltonian then reads

$\displaystyle\cal{H}_{\text{SB}}$ $\displaystyle=$
$\displaystyle=\sum_{i,j}J_{ij}\left[\hat{B}_{ij}^{+}{B}_{ij}+\hat{B}_{ij}{B}_{ij}^{*}-\hat{A}_{ij}^{+}{A}_{ij}-\hat{A}_{ij}{A}_{ij}^{*}\right]$
(S16) $\displaystyle-$
$\displaystyle\sum_{i,j}J_{ij}\left[|B_{ij}|^{2}-|A_{ij}|^{2}\right]+\sum_{i}\lambda_{i}(\hat{n}_{i}-2S),$

with the mean field parameters
$A_{ij}=\langle\phi_{0}|\hat{A}_{ij}|\phi_{0}\rangle$ and
$B_{ij}=\langle\phi_{0}|\hat{B}_{ij}|\phi_{0}\rangle$ computed in
$|\phi_{0}\rangle$ – the boson vacuum at $T=0$ for each pair of interacting
spins $(i\to j)$. Because the Hilbert space is enlarged by the mapping, it is
necessary to fulfill the constraint
$\hat{n}_{i}=\hat{b}_{i\uparrow}^{+}\hat{b}_{i\uparrow}+\hat{b}_{i\downarrow}^{+}\hat{b}_{i\downarrow}=2S$
for a spin $S$. Thus, we have also introduced Lagrange multipliers
$\lambda_{i}$ to account for this on average. Another advantage of the method
is that $S$ can be treated as an external parameter and by reducing it, it is
possible to enhance quantum fluctuations. This is particularly interesting if
one wants to focus on phase transitions between a magnetically ordered state
and its quantum spin liquid parent. Also, the flexibility of the method allows
to compute the dynamical structure factor

$\displaystyle S(\mathbf{q},\omega)$ $\displaystyle=$
$\displaystyle\frac{1}{n_{s}}\sum_{i,j}e^{i\bf{q}\cdot(\bf{r}_{i}-\bf{r}_{j})}\int_{-\infty}^{\infty}dte^{-i\omega
t}\langle\hat{\mathbf{S}}_{i}(t)\hat{\mathbf{S}}_{j}(0)\rangle,$

and to extract relevant magnon features and study Bose condensations of
specific branches. Here, $n_{s}$ is the total number of sites given by
$n_{u}\times 2\times l\times l$, with a $n_{u}$ the number of sites per unit-
cell (here 14 in the presence of the Cu(3) atoms), and $l$ is the linear size
of the system. This allows for comparison with neutron experiments.

In Fig. 4 of the main text, we show the dynamical structure factor for two
representative spin values $S=0.12,0.15$ at which a quantum spin liquid (QSL)
can be achieved, and for various values of Cu(3) coupling $J_{10}$ for a
system size of $l=12$ with $4032$ spins, showing the proximity of the a phase
transition between a quantum spin liquid state and its Bose condensate
counterpart.

As seen in the fermion approaches, the effect of projecting the wave function
onto exact physical states increases the quantum fluctuations and helps the
system to remain disordered even in the presence of the Cu(3) atoms. In the
Schwinger boson mean field theory, since magnetic orders are more competitive
by construction, they are favoured at $S=1/2$. Thus, in order to reach the
QSL, one has to reduce the spin value. In Fig. 4 of the main text, we can see
that the Bose condensation arises on the Cu(3) spins while the others on the
square-kagome lattice remains mainly disordered. This is reflected by a clear
gap in the lower panels between the condensed branches and the excitations in
the continuum. To have a better view of this feature, we have plotted in Fig.
S11 the real-space (equal-time) spin-spin correlations using three different
reference sites on a Cu(3) spin (top panel), on a corner of a triangle (middle
panel) and on a corner of a square plaquette (bottom panel).

As one can see, a clear magnetic order appears on the Cu(3) spins while the
ones on the remaining square-kagome lattice are disordered. This last feature
is in good agreement with the observations of increased correlations between
the Cu(3) sites in PMFRG, although no order could be detected in the
accessible temperature regime.

Figure S11: Real space (equal-time) spin-spin correlations considering 3
reference sites: (top) Cu(3) spin, (middle) corner of triangles and (bottom)
corner of square plaquette. The parameters are the ones extracted in the main
text, $S\simeq 0.366$ here, for which the sum rule of the static structure
factor is satisfied [15]. The system size is $n_{u}\times 2\times 6\times 6$.
The colors correspond to positive (blue) and negative (red) correlations. The
reference site is always the largest red disk of the figure.

## References

  * Yakubovich  _et al._ [2021] O. V. Yakubovich, L. V. Shvanskaya, G. V. Kiriukhina, A. S. Volkov, O. V. Dimitrova, and A. N. Vasiliev, Hydrothermal Synthesis and a Composite Crystal Structure of Na6Cu7BiO4(PO4)4[Cl,(OH)]3 as a Candidate for Quantum Spin Liquid, Inorg. Chem. 60, 11450 (2021).
  * Momma and Izumi [2011] K. Momma and F. Izumi, VESTA 3 for three-dimensional visualization of crystal, volumetric and morphology data, J. Appl. Crystallogr. 44, 1272 (2011).
  * Ghosh  _et al._ [2023] P. Ghosh, T. Müller, Y. Iqbal, R. Thomale, and H. O. Jeschke, Effective spin-1 breathing kagome Hamiltonian induced by the exchange hierarchy in the maple leaf mineral bluebellite, arxiv preprint 2301.05224 (2023).
  * Misawa  _et al._ [2019] T. Misawa, S. Morita, K. Yoshimi, M. Kawamura, Y. Motoyama, K. Ido, T. Ohgoe, M. Imada, and T. Kato, mVMC—Open-source software for many-variable variational Monte Carlo method, Comput. Phys. Commun 235, 447 (2019).
  * Tahara and Imada [2008] D. Tahara and M. Imada, Variational Monte Carlo Method Combined with Quantum-Number Projection and Multi-Variable Optimization, J. Phys. Soc. Jpn. 77, 114701 (2008).
  * Sorella [1998] S. Sorella, Green Function Monte Carlo with Stochastic Reconfiguration, Phys. Rev. Lett. 80, 4558 (1998).
  * Becca and Sorella [2017] F. Becca and S. Sorella,  _Quantum Monte Carlo Approaches for Correlated Systems_ (Cambridge University Press, Cambridge, United Kingdom, 2017).
  * Carleo and Troyer [2017] G. Carleo and M. Troyer, Solving the quantum many-body problem with artificial neural networks, Science 355, 602 (2017).
  * Niggemann  _et al._ [2021] N. Niggemann, B. Sbierski, and J. Reuther, Frustrated quantum spins at finite temperature: Pseudo-Majorana functional renormalization group approach, Phys. Rev. B 103, 104431 (2021).
  * Niggemann  _et al._ [2022] N. Niggemann, J. Reuther, and B. Sbierski, Quantitative functional renormalization for three-dimensional quantum Heisenberg models, SciPost Phys. 12, 156 (2022).
  * Auerbach [1998] A. Auerbach,  _Interacting electrons and quantum magnetism_ (Springer Science & Business Media, 1998).
  * Halimeh and Punk [2016] J. C. Halimeh and M. Punk, Spin structure factors of chiral quantum spin liquids on the kagome lattice, Phys. Rev. B 94, 104413 (2016).
  * Schaffer  _et al._ [2017] R. Schaffer, Y. Huh, K. Hwang, and Y. B. Kim, Quantum spin liquid in a breathing kagome lattice, Phys. Rev. B 95, 054410 (2017).
  * Lugan  _et al._ [2019] T. Lugan, L. D. C. Jaubert, and A. Ralko, Topological nematic spin liquid on the square kagome lattice, Phys. Rev. Research 1, 033147 (2019).
  * Lugan  _et al._ [2022] T. Lugan, L. D. C. Jaubert, M. Udagawa, and A. Ralko, Schwinger boson theory of the ${J}_{1},{J}_{2}={J}_{3}$ kagome antiferromagnet, Phys. Rev. B 106, L140404 (2022).