#  $\mathcal{B}$PO: Supercharging Online Preference Learning by Adhering to
the Proximity of Behavior LLM

Wenda Xu 1, Jiachen Li1††footnotemark: , William Yang Wang1, Lei Li2
1UC Santa Barbara, 2Carnegie Mellon University
1{wendaxu, jiachen_li<EMAIL_ADDRESS><EMAIL_ADDRESS>Equal
Contributions.

###### Abstract

Direct alignment from preferences (DAP) has emerged as a promising paradigm
for aligning large language models (LLMs) to human desiderata from pre-
collected, offline preference datasets. While recent studies indicate that
existing offline DAP methods can directly benefit from online training
samples, we highlight the need to develop specific online DAP algorithms to
fully harness the power of online training. Specifically, we identify that the
learned LLM should adhere to the proximity of the _behavior LLM_ , which
collects the training samples. To this end, we propose online Preference
Optimization in proximity to the Behavior LLM ($\mathcal{B}$PO), emphasizing
the importance of constructing a proper trust region for LLM alignment.

00footnotetext: ∗equal contributions

We conduct extensive experiments to validate the effectiveness and
applicability of our approach by integrating it with various DAP methods,
resulting in significant performance improvements across a wide range of tasks
when training with the same amount of preference data. Even when only
introducing _one_ additional preference annotation phase, our online
$\mathcal{B}$PO improves its offline DAP baseline from $72.0\%$ to $80.2\%$ on
TL;DR and from $82.2\%$ to $89.1\%$ on Anthropic Helpfulness in terms of win
rate against human reference text.

$\mathcal{B}$PO: Supercharging Online Preference Learning by Adhering to the
Proximity of Behavior LLM

Wenda Xu 1††thanks: Equal Contributions., Jiachen Li1††footnotemark: , William
Yang Wang1, Lei Li2 1UC Santa Barbara, 2Carnegie Mellon University 1{wendaxu,
jiachen_li<EMAIL_ADDRESS><EMAIL_ADDRESS>

Figure 1: Given the same annotation budget, our $\mathcal{B}$PO (when $F=2$)
significantly outperforms offline DPO ($F=1$) on both TL;DR and Anthropic
Helpfulness by introducing only _one_ additional preference annotation phase.
Its performance ($F=2$) even matches, if not exceeds, that of on-policy DPO
($F=T$), which collects new annotations at every step.

##  1 Introduction

Figure 2: Overview of the training pipeline of our $\mathcal{B}$PO. Our
training loss $\mathcal{L}_{\mathrm{\mathcal{B}PO}}$ is calculated by
constraining the KL divergence between $\pi_{\theta}$ and the behavior LLM
$\pi_{\beta}$. Every $K$ step, we update $\pi_{\beta}$ with $\pi_{\theta}$ and
use it to collect new samples for annotations.

Reinforcement Learning from Human Feedback (RLHF) Christiano et al. (2017);
Ziegler et al. (2019) has achieved tremendous success in aligning the powerful
pretrained large language models (LLMs) with human preference Achiam et al.
(2023); Gemini et al. (2023); Anthropic (2024); Touvron et al. (2023),
revolutionizing human society. However, traditional RLHF methods Ouyang et al.
(2022); Stiennon et al. (2020b) are computationally expensive due to their
two-stage training pipeline that consists of a reward modeling phase and can
suffer from RL training instability Choshen et al. (2019). To address these
issues, recent advances in direct alignment from preferences (DAP) methods
provide solutions to avoid learning a reward model (RM) and stabilize the
training process. Prominent examples include Direct Preference Optimization
(DPO) Rafailov et al. (2024) and its variants Azar et al. (2024); Ethayarajh
et al. (2024); Wang et al. (2022); Tang et al. (2024b), which directly
optimize LMs using a static, pre-collected set of preference data,
streamlining the alignment procedures.

However, recent studies Xu et al. (2023); Guo et al. (2024); Pang et al.
(2024); Tang et al. (2024a); Tajwar et al. (2024) identified that aligning an
LLM with offline preference datasets prevents the LLM from getting feedback
for its own generations. These studies emphasize the importance of
incorporating online data generated by intermediate models during training.
While empirical evidence shows that offline DAP methods can directly benefit
from online preference data, we argue that making algorithm-level adjustments
is essential to fully harness the power of online training.

Specifically, we identify that existing online DAP methods Guo et al. (2024);
Tang et al. (2024a); Calandriello et al. (2024); Xu et al. (2023); Pang et al.
(2024) do not adjust the trust region designed in offline DAP methods Rafailov
et al. (2024). These methods still construct their trust region Schulman et
al. (2015) by penalizing the KL divergence between the learned LLM and a fixed
reference model $\pi_{\mathrm{ref}}$, even when training samples are
dynamically generated by intermediate models. Drawing inspiration from
existing RL literature Schulman et al. (2017); Fujimoto et al. (2019); Li et
al. (2023), we propose online Preference Optimization in proximity to the
Behavior LLM ($\mathcal{B}$PO), emphasizing that a better trust region should
be instead constructed around the _behavior LLM_ $\pi_{\beta}$ that collects
the training samples. In other words, we should set $\pi_{\beta}$ as
$\pi_{\mathrm{ref}}$ when performing online DAP. We provide an overview of our
training pipeline in Fig. 2.

However, setting a dynamic $\pi_{\mathrm{ref}}$ during online DAP can lead to
instability. To mitigate this issue, we propose optimizing an ensemble of LoRA
Hu et al. (2021) weights and merging them during inference. We verify the
effectiveness of our method by building on top of various DAP methods,
including DPO Guo et al. (2024), IPO Azar et al. (2024) and SLiC Zhao et al.
(2023). Empirically, we show that our $\mathcal{B}$PO significantly improves
over their online and offline DAP counterparts on TL;DR Ziegler et al. (2019),
Anthropic Helpfulness and Harmlessness Bai et al. (2022), demonstrating the
generalizability of our methods.

On the other hand, conducting iterative preference annotations at each
training step can be practically infeasible when hiring human annotators.
Given the same annotation budget, we anticipate that a successful online DAP
method to perform well at a low annotation frequency. In other words, we aim
to minimize the number of preference annotation phases throughout the
training. To this end, we evaluate our method with different annotation
frequencies while keeping the total amount of preference data constant. We
demonstrate that even with just one additional preference annotation phase
compared to offline DPO, our $\mathcal{B}$PO significantly improves over its
offline DPO counterpart from $72.0\%$ to $80.2\%$ on TL;DR and from $82.2\%$
to $89.1\%$ on Anthropic Helpfulness in terms of win rate against human
reference text.

Furthermore, we conduct an ablation study to verify our performance
improvement comes from our better trust region constructed around
$\pi_{\beta}$ instead of $\pi_{\beta}$’s higher quality compared to
$\pi_{\mathrm{ref}}$. Our results show that even when using a high-quality LLM
as $\pi_{\mathrm{ref}}$ for online DAP baselines, our approach still
outperforms it.

Our contributions are three-fold:

  1. 1.

An online DAP method $\mathcal{B}$PO. To the best of our knowledge, we are the
first to tailor offline DAP methods for online training.

  2. 2.

Empirical superiority of our $\mathcal{B}$PO over its online and offline DAP
counterpart on standard alignment tasks.

  3. 3.

Remarkable applicability of our $\mathcal{B}$PO to handle low data collection
frequencies.

##  2 Related Work

Reinforcement Learning from Human Feedback methods Christiano et al. (2017);
Ziegler et al. (2019); Ouyang et al. (2022); Bai et al. (2022); Touvron et al.
(2023) first learn a RM from a set of preference data based on the Bradley-
Terry modelization Bradley and Terry (1952) and then leverage an RL algorithm,
e.g., PPO Schulman et al. (2017), to optimize a pretrained or supervised fine-
tuned (SFT) LLM towards the learned RM. However, the two-stage learning
process is computationally demanding and RL training can suffer from training
instability Choshen et al. (2019). To address these issues, recent
advancements, including DPO Guo et al. (2024) and DPO variants Zhao et al.
(2023); Azar et al. (2024); Ethayarajh et al. (2024); Wang et al. (2022); Tang
et al. (2024b), enable direct alignment from a fixed, offline set of
preference data. These methods leverage the dual form of the original RLHF
objectives, successfully eliminating the reward modeling phase and stabilizing
the training. Empirically, these methods achieve impressive performance on
standard evaluation benchmarks Ziegler et al. (2019); Bai et al. (2022);
Dubois et al. (2024); Zheng et al. (2024).

Concurrently, TR-DPO Gorbatovski et al. (2024) also explores setting a dynamic
$\pi_{\mathrm{ref}}$ for DAP methods. However, TR-DPO only considers offline
DAP settings and thus never considers setting
$\pi_{\mathrm{ref}}=\pi_{\beta}$. Instead, TR-DPO explores setting
$\pi_{\mathrm{ref}}$ as the moving average of their $\pi_{\theta}$ or
periodically updates $\pi_{\mathrm{ref}}$ with $\pi_{\theta}$. Therefore, TR-
DPO is substantially different from our methods.

Online DAP Methods. Recent studies Guo et al. (2024); Tang et al. (2024a);
Calandriello et al. (2024); Xu et al. (2023); Pang et al. (2024); Dong et al.
(2024); Xie et al. (2024) have recognized the importance of on-policy training
data. Specifically, these methods collect human preference on the responses
generated from intermediate models and use them for training. Although these
online methods improve their offline counterparts, they still constrain the KL
divergence between the learning LLM and a static reference model. In this
paper, we provide extensive empirical results demonstrating that the reference
LLM should be set dynamically as the behavior LLM that collects the training
data during online preference learning.

##  3 Preliminaries

###  3.1 Reinforcement Learning from Human Feedback

Traditional RLHF methods require learning an RM $r_{\phi}$ from a preference
dataset $\mathcal{D}=\\{x^{(i)},y_{w}^{(i)},y_{l}^{(i)}\\}_{i=1}^{N}$ to
mirror human preference. Each example in $\mathcal{D}$ is obtained by sampling
a pair of responses $(y_{1},y_{2})$ given the sample text prompt $x$, which
are then sent to human or AI labelers for annotations. The preferred and
dispreferred samples are denoted as $y_{w}$ and $y_{l}$, respectively.

With the learned RM $r_{\phi}$, we can optimize an LLM $\pi_{\theta}$ with the
RL objective given by

$\begin{aligned}
\max_{\theta}\,\,&\mathbb{E}_{x\sim\mathcal{D}_{\mathcal{X}},y\sim\pi_{\theta}(y\mid
x)}\left[r_{\phi}(x,y)\right]\\\
&-\beta\mathbb{D}_{\mathrm{KL}}\left[\pi_{\theta}(y\mid
x)\|\pi_{\mathrm{ref}}(y\mid x)\right]\end{aligned},$ (1)

where $\mathcal{D}_{\mathcal{X}}$ is a dataset of training prompts and the
coefficient $\beta$ regulates the KL divergence between $\pi_{\theta}$ and a
reference model $\pi_{\mathrm{ref}}$. A larger $\beta$ imposes a greater
penalty on the KL divergence, leading to a smaller trust region. And thus, the
learned LLM $\pi_{\theta}$ will be more similar to $\pi_{\mathrm{ref}}$.

###  3.2 Direct Alignment from Preferences

Direct Alignment from Preferences (DAP) methods streamline the alignment
procedures by learning from an offline, static set of preference datasets
$\mathcal{D}$. These methods eliminate the reward modeling stage of
traditional RLHF methods by leveraging the dual formulation of (1). Given a
pair of responses $(y_{w},y_{l})$ corresponding to the prompt $x$, the loss
functions for DPO Rafailov et al. (2024), IPO Azar et al. (2024) and SLiC Zhao
et al. (2023) are given below

$\displaystyle\mathcal{L}_{\mathrm{DPO}}\left(x,y_{w},y_{l},\pi_{\theta};\pi_{\mathrm{ref}}\right)$
(2) $\displaystyle=-\log\sigma\left(\beta\log\frac{\pi_{\theta}\left(y_{w}\mid
x\right)}{\pi_{\theta}\left(y_{l}\mid
x\right)}\cdot\frac{\pi_{\mathrm{ref}}\left(y_{l}\mid
x\right)}{\pi_{\mathrm{ref}}\left(y_{w}\mid x\right)}\right)$
$\displaystyle\mathcal{L}_{\mathrm{IPO}}\left(x,y_{w},y_{l},\pi_{\theta};\pi_{\mathrm{ref}}\right)$
(3) $\displaystyle=\left[\log\left(\frac{\pi_{\theta}\left(y_{w}\mid
x\right)}{\pi_{\theta}\left(y_{l}\mid
x\right)}\cdot\frac{\pi_{\mathrm{ref}}\left(y_{l}\mid
x\right)}{\pi_{\mathrm{ref}}\left(y_{w}\mid
x\right)}\right)-\frac{1}{2\beta}\right]^{2}$
$\displaystyle\mathcal{L}_{\mathrm{SLiC}}\left(x,y_{w},y_{l},\pi_{\theta};\pi_{\mathrm{ref}}\right)$
(4)
$\displaystyle=\max\left(0,1-\beta\log\left(\frac{\pi_{\theta}\left(y_{w}\mid
x\right)\pi_{\mathrm{ref}}\left(y_{l}\mid
x\right)}{\pi_{\theta}\left(y_{l}\mid
x\right)\pi_{\mathrm{ref}}\left(y_{w}\mid x\right)}\right)\right)$

Notably, the reference model $\pi_{\mathrm{ref}}$ in offline DAP methods is
fixed to be an SFT LLM $\pi_{\mathrm{sft}}$.

###  3.3 Online DAP, Offline DAP, and On-Policy DAP

In reinforcement learning (RL) Sutton and Barto (2018), _offline learning_
refers to learning from a static, pre-collected dataset. In contrast, _online
learning_ involves learning from a dynamic dataset, where new samples from
intermediate policies are incorporated into the training data. Notably, online
learning is not equivalent to _on-policy_ learning. On-policy learning is a
special case of online learning, where the policy is trained using on-policy
data generated from the same distribution as the learning policy itself. When
the policy is trained with off-policy data sampled from a different
distribution, it is referred to as off-policy learning. While utilizing off-
policy data can improve sample efficiency Haarnoja et al. (2018); Fujimoto et
al. (2018); Lillicrap et al. (2015); Li et al. (2020), its effectiveness
degrades when the gap between data and policy distribution becomes large Tang
et al. (2024a); Ostrovski et al. (2021). Therefore, online DAP methods that
learn from samples generated from intermediate models often outperform their
offline counterparts.

In this paper, we classify the DAP method as an online method if it trains an
LM using data sampled from intermediate models. We regard it as an on-policy
method only when it consistently employs on-policy samples for training.
Conversely, if a DAP method is trained using a static, pre-collected dataset,
we classify it as an offline method.

##  4 Improving Online DAP by Constructing a Better Trust Region

Algorithm 1 $\mathcal{B}$PO: Online Preference Optimization in Proximity to
the Behavior LLM

1:Number of training steps $T$, Preference annotation frequency $F$, Number of
new data per annotation phase $M$, Prompt dataset
$\mathcal{D}_{\mathcal{X}}=\\{x_{i}\\}_{i=1}^{N}$, Preference dataset
$\mathcal{D}=\\{\\}$, SFT LLM $\pi_{\theta_{0}}$, Behavior LLM $\pi_{\beta}$,
LLM / Human annotator, learning rate $\gamma$, a DAP loss function
$\mathcal{L}_{\mathrm{DAP}}$.

2:Calculate the annotation interval $K=T/F$

3:for $t:=0$ to $T$ do

4: if t % $K$ = 0 then

5: Update behavior LM: $\pi_{\beta}\leftarrow\pi_{\theta_{t}}$

6: for $i:=1$ to $M$ do

7: Sample prompt $x\sim\mathcal{D}_{\mathcal{X}}$

8: Sample $y_{1},y_{2}\sim\pi_{\beta}(\cdot|x)$

9: Annotate preference pair $y_{w},y_{l}$

10: $\mathcal{D}\leftarrow\mathcal{D}\cup\\{(x,y_{w},y_{l})\\}$

11: end for

12: end if

13: Sample a batch $(x,y_{w},y_{l})$ from $\mathcal{D}$

14: Update $\mathcal{D}\leftarrow\mathcal{D}\setminus\\{(x,y_{w},y_{l})\\}$

15:
$\theta_{t+1}\leftarrow\theta_{t}-\gamma\cdot\nabla_{\theta}\mathcal{L}_{\mathrm{DAP}}\left(x,y_{w},y_{l},\pi_{\theta};\pi_{\beta}\right)$

16:end for

17:Aligned LLM $\pi_{\theta_{T}}$

As discussed in Sec. 3, offline DAP methods never update the preference
dataset $\mathcal{D}$ with new samples after the initial data collection
stage. Consequently, the learning LLM $\pi_{\theta}$ will gradually deviate
from the training data distribution as the training continues. To mitigate the
distribution shift, several works Guo et al. (2024); Tang et al. (2024a); Pang
et al. (2024); Tajwar et al. (2024) have advocated for online DAP training and
provided empirical evidence to demonstrate the benefit.

However, existing online DAP methods still constrain KL divergence between
$\pi_{\theta}$ and a fixed reference model $\pi_{\mathrm{ref}}$ even when
annotating new preference data online. In this paper, we propose
$\mathcal{B}$PO, arguing that we should construct a better trust region by
constraining the KL divergence between the learning $\pi_{\theta}$ and the
behavior LLM $\pi_{\beta}$. Given a triplet of $(x,y_{w},y_{l})$,
$\pi_{\beta}$ that generates $(y_{w},y_{l})$ given $x$ and a DAP loss function
$\mathcal{L}_{\mathrm{DAP}}$, our loss function of
$\mathcal{L}_{\mathcal{B}\mathrm{PO}}$ is defined as below:

$\mathcal{L}_{\mathcal{B}\mathrm{PO}}=\mathcal{L}_{\mathrm{DAP}}\left(x,y_{w},y_{l},\pi_{\theta};\pi_{\beta}\right)$
(5)

Algorithm 1 provides the pseudo-codes. We use $F$ to denote the preference
annotation frequency. We can simulate different DAP settings by varying the
value of $F$. For example, when we use a preference simulator (AI feedback),
we can set $F=T$, meaning that we collect new preference data with
$\pi_{\theta_{t}}$ at every training step for training, corresponding to the
on-policy DAP setting. When using human annotation, we need to lower the value
of $F$. When setting $F=1$, we reduce to the offline DAP settings, where
$\mathcal{D}$ is collected by $\pi_{\mathrm{sft}}$ and
$\pi_{\mathrm{ref}}=\pi_{\beta}$ is never updated during training. In summary,

  * •

$F=T$ corresponds to on-policy learning.

  * •

$1<F\leq T$ corresponds to online learning.

  * •

$F=1$ corresponds to offline training.

We observe that setting a dynamic $\pi_{\mathrm{ref}}=\pi_{\beta}$ with a
large $F$ can lead to training instability. To overcome this challenge, we
optimize an ensemble LoRA Hu et al. (2021) weights of the LLM to stabilize the
training. We linearly merge the LoRA weights during inference without
incurring additional inference overhead.

##  5 Experiments

We conduct experiments to address the research questions below:

  1. 1.

Can $\mathcal{B}$PO empirically outperform online and offline DAP
counterparts? (Sec. 5.1)

  2. 2.

Can $\mathcal{B}$PO adapt to different data collection frequencies (Sec. 5.2)?

  3. 3.

Will online DAP with a high-quality static $\pi_{\mathrm{ref}}$ outperform our
$\mathcal{B}$PO (Sec. 5.3)?

  4. 4.

How to stabilize the training of our $\mathcal{B}$PO (Sec. 5.4)?

_Win Rate_ (%) against Reference Text | TL;DR | Helpfulness | Harmfulness | Overall
---|---|---|---|---
SFT | $38.8$ | $66.2$ | $51.2$ | $52.1$
Offline DPO | $72.0\pm 2.4$ | $82.2\pm 4.4$ | $77.5\pm 0.9$ | $77.2\pm 2.6$
On-Policy DPO | $77.2\pm 0.4$ | $90.6\pm 0.9$ | $96.9\pm 0.6$ | $88.3\pm 0.6$
On-Policy $\mathcal{B}$PO (DPO) | $\mathbf{89.5\pm 1.4}$ | $\mathbf{93.5\pm 0.4}$ | $\mathbf{97.7\pm 1.4}$ | $\mathbf{93.6\pm 1.1}$
Offline IPO | $68.5\pm 5.3$ | $81.6\pm 10.6$ | $90.1\pm 4.6$ | $80.1\pm 6.8$
On-Policy IPO | $83.7\pm 0.6$ | $94.5\pm 1.2$ | $94.5\pm 2.7$ | $90.9\pm 1.5$
On-Policy $\mathcal{B}$PO (IPO) | $\mathbf{88.6\pm 1.7}$ | $\mathbf{96.3\pm 0.5}$ | $\mathbf{96.3\pm 0.8}$ | $\mathbf{93.7\pm 1.0}$
Offline SLiC | $74.0\pm 0.3$ | $83.6\pm 1.5$ | $95.1\pm 0.9$ | $84.2\pm 0.9$
On-Policy SLiC | $82.6\pm 0.9$ | $90.3\pm 1.5$ | $91.8\pm 5.8$ | $88.2\pm 2.7$
On-Policy $\mathcal{B}$PO (SLiC) | $\mathbf{89.3\pm 0.7}$ | $\mathbf{92.5\pm 1.4}$ | $\mathbf{94.7\pm 1.9}$ | $\mathbf{92.2\pm 1.3}$

Table 1: We include $\mathcal{B}$PO’s results against offline and online DAP
methods across TL;DR, Helpfulness, and Harmfulness tasks. We experiment with
three different DAP algorithms: DPO, IPO and SLiC. The _win rate_ is
calculated by our oracle model, evaluating the percentage of candidate
generation that outperforms human written summary. The results are calculated
using three different seeds. Our on-policy $\mathcal{B}$PO significantly
outperforms its offline and on-policy DAP counterparts. Table 5 in the
Appendix includes results for each seed. Figure 3: Aggregate metrics Agarwal
et al. (2021) evaluating the win rate against human references with 95%
confidence intervals (CIs), based on results reported in Table 1. The CIs are
estimated using the percentile bootstrap with stratified sampling. Higher
median, IQM, and mean scores correspond to better performance. Our
$\mathcal{B}$PO outperforms offline and on-policy DAP methods by a significant
margin based on all metrics.

#### Dataset & Evaluation Metric

We performed our experiments using Reddit TL;DR Ziegler et al. (2019),
Anthropic Helpfulness and Harmlessness Bai et al. (2022) dataset. We split
training data of 65K, 10K, and 10K for TL;DR, Helpfulness, and Harmlessness,
respectively, to perform supervised fine-tuning (SFT). All SFT data is
selected based on preferred responses. During the alignment stage, we use
another training set that contains prompts that are different from SFT to
perform sampling and DAP. We have 10K prompts for each of the three tasks.

In this study, we are investigating the performance of DAP algorithms when
giving a fixed annotation budget. In practice, the annotation is performed by
human raters. To ensure the reproducibility and scalability of our study, we
use an open-sourced model, RM-deberta
111https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2 as
our preference simulator. The pairwise preference data is labeled by our
preference simulator. RM-deberta has been trained on various preference pair
datasets, including WebGPT comparisons Nakano et al. (2021), Open
summarization Stiennon et al. (2020a) and anthropic HH-RLHF Bai et al. (2022),
covering all tasks that we studied in this paper. We use preference simulator
to annotate and evaluate our method and baselines. Although we use a
preference simulator for annotation, we investigate different data collection
scenarios in Sec. 5.2 to extend our approach to realistic online setting.

#### Baseline models

In this study, we consider three baseline DAP methods: DPO, SLiC, and IPO. By
building on top of these methods, we learn corresponding $\mathcal{B}$PO
(DPO), $\mathcal{B}$PO (SLiC), and $\mathcal{B}$PO (IPO) and compare
performance against their online and offline DAP counterparts.

#### Implementation

We use the development set to select the best-performed SFT checkpoint. Our
batch size is $16$, $64$ and $16$ for TL;DR, Helpfulness, and Harmlessness
tasks, respectively. We train $625$, $150$, and $625$ steps for each
corresponding task. We set learning rate to be $5e$-$5$ for both SFT and
preference learning. We fixed the regularization coefficient $\beta=0.1$ for
all preference learning methods. We leverage Gemma-2b Team et al. (2024) as
our base LM. All DAP methods optimize an ensemble of 5 LoRA weights on top of
the base model.

###  5.1 On-policy $\mathcal{B}$PO outperforms its on-policy and offline DAP
counterparts

We compare our on-policy $\mathcal{B}$PO (i.e., setting $K=1$ in Algorithm 1)
with its on-policy and offline DAP counterparts across three alignment tasks.
We carefully design the experiments to ensure all methods are trained with the
same amount of total preference data. All experiments are conducted with 3
random seeds. We averaged the three runs and reported standard deviations of
the results. We use the preference simulator to determine the win rate of our
generated summary against the reference text provided by humans. That is, a
generated response _wins_ over the reference text if it receives a higher
reward from the preference simulator.

As shown in Table 1, all of our $\mathcal{B}$PO variants achieve significantly
higher win rates against the reference text than their on-policy and offline
baselines across all evaluated tasks, particularly on TL;DR. This demonstrates
$\mathcal{B}$PO’s strong generalizability to different DAP learning losses.
These results bring two important messages: 1) Incorporating on-policy
training data leads to better DAP performance. This finding is derived from
on-policy $\mathcal{B}$PO’s superior performance over offline DAP methods.
Although offline DAP methods also constrain the KL divergence between
$\pi_{\theta}$ and $\pi_{\mathrm{SFT}}$ that collects the static preference
datasets, $\pi_{\theta}$ will gradually deviate from the training data
distribution as training proceeds. The distribution shift prevents
$\pi_{\theta}$ from getting feedback for its own generations, leading to
performance gap compared to our on-policy $\mathcal{B}$PO. 2) On-policy DAP
should adhere to the proximity of the behavior LLM. The superiority of our on-
policy $\mathcal{B}$PO over its on-policy DAP counterpart underscores the
importance of constructing a proper trust region during online DAP training.
By constraining the learned policy to stay closer to the behavior LLM
$\pi_{\beta}$, we substantially improve performance.

Figure 4: We experiment with different data collection frequency $F$ for our
$\mathcal{B}$PO (DPO) on TL;DR (Left) and Helpfulness (Right). The error bar
denotes the one std of the win rates across 3 random seeds. Our
$\mathcal{B}$PO is applicable to a small $F$. Even with $F=2$, our
$\mathcal{B}$PO (DPO) significantly outperforms online DPO and at least
matches the performance of on-policy DPO on both tasks.

#### Statistical significance test.

To systematically evaluate the statistical significance of our improvement
over the baseline DAP methods, we leverage the reliable evaluation protocols
proposed in Agarwal et al. (2021) to re-examine the results in Table 1.
Specifically, we report the _Median_ , _Interquartile Mean_ (IQM), and _Mean_
of win rate across the $N_{\mathrm{tasks}}\times N_{\mathrm{seeds}}\times
N_{\mathrm{alg}}$ runs in Table 1. Notably, the IQM is calculated by
discarding the top and bottom 25% data points and calculating the mean across
the remaining 50% runs. Therefore, the IQM is more robust to outliers than the
_mean_ while maintaining less variance than the _median_. Higher median, IQM,
and mean scores correspond to better performance. As shown in Fig. 3, our
$\mathcal{B}$PO outperforms offline and on-policy DAP methods by a significant
margin based on all metrics.

Head-to-head Win Rate (%) of On-Policy $\mathcal{B}$PO (DPO)
---
baseline | TL;DR | Helpfulness | Harmfulness
Offline DPO | $75.9\pm 1.5$ | $78.0\pm 1.7$ | $99.3\pm 0.5$
On-Policy DPO | $70.4\pm 1.5$ | $57.4\pm 5.1$ | $79.9\pm 9.9$

Table 2: Head-to-head comparisons of our on-policy $\mathcal{B}$PO against
offline and on-policy DPO baselines. All evaluation results are derived with
three random seeds. Our on-policy $\mathcal{B}$PO outperforms both offline and
on-policy DPO by significant margins, with win rates substantially higher than
$0.5$ on all comparisons.

#### Head-to-head comparison

To gain a deeper understanding of the performance difference between our on-
policy $\mathcal{B}$PO and the baseline methods, we take a closer look at the
DPO variants and compare the outputs from our $\mathcal{B}$PO (DPO) with those
from both offline and on-policy DPO across all evaluation tasks. We use
preference simulator as oracle to annotate the results. As shown in Table 2,
our on-policy $\mathcal{B}$PO is consistently favored by the oracle over the
baseline on-policy and offline DPO, achieving win rates higher than 50%.

###  5.2 Evaluate online $\mathcal{B}$PO with different data collection
frequency

Practically, collecting human feedback at every training step in a fully on-
policy fashion ($F=T$) is expensive and challenging. In this section, we
evaluate the performance of our $\mathcal{B}$PO by varying the data collection
frequency $F\in[1,T]$, aiming to simulate different online DAP settings.
Notably, we fix the total amount $N_{\mathrm{total}}$ of training preference
data when experimenting with different $F$. At each data collection phase, we
annotate $M=N_{\mathrm{total}}/F$ preference pairs. Specifically, we focus on
the DPO variants and train $\mathcal{B}$PO (DPO) with different $F$. On TL;DR,
we set $T=600$ sample $F$ from $\\{1,2,6,15,600\\}$. On Helpfulness, we set
$T=150$ sample $F$ from $\\{1,2,3,150\\}$. Setting $F=1$ corresponds to the
offline learning settings.

Figure 4 provides the experiment results. We observe that a higher data
collection frequency $F$ leads to better performance for our online
$\mathcal{B}$PO. Notably, even increasing $F$ from $1$ to $2$ allows our
$\mathcal{B}$PO (DPO) to significantly outperform offline DPO, improving win
rates against reference text from $72.0_{\pm 2.4}\%$ to $80.2_{\pm 2.3}\%$ on
TL;DR and from $82.2_{\pm 4.4}\%$ to $89.1_{\pm 1.4}\%$ on Anthropic
Helpfulness. Moreover, our $\mathcal{B}$PO (DPO) with $F=2$ significantly
outperforms on-policy DPO on TL;DR and matches the performance of on-policy
DPO on Helpfulness. These results are particularly impressive, as our method
can still achieve substantial performance gains by adding only one additional
preference annotation phase compared to the standard offline DAP training,
without increasing $N_{\mathrm{total}}$. Therefore, our $\mathcal{B}$PO can be
useful when hiring humans to annotate preference data, as it applies to a
small $F$.

###  5.3 Ablation study on the reference model

Figure 5: Ablation study on the reference model $\pi_{\mathrm{ref}}$. Even by
setting $\pi_{\mathrm{ref}}$ as an optimized LLM $\pi_{\mathrm{gold}}$ that is
significantly better than SFT LM $\pi_{\mathrm{sft}}$, on-policy DPO still
under-performs our on-policy $\mathcal{B}$PO, validating that our improvement
comes from constraining the divergence between the learned LLM $\pi_{\theta}$
and the behavior LLM $\pi_{\beta}$. The shaded area denotes one std.

We aim to demonstrate that setting the reference model as the behavior LLM
$\pi_{\beta}$ dynamically is superior to setting a static
$\pi_{\mathrm{ref}}$. Conventional online DAP methods consistently set
$\pi_{\mathrm{ref}}=\pi_{\mathrm{SFT}}$, and is outperformed by our method as
shown in Sec. 5.1 and 5.2. However, one hypothesis for our improvement is due
to the improved quality of $\pi_{\beta}$ as $\pi_{\mathrm{ref}}$ compared to
$\pi_{\mathrm{SFT}}$. To validate this hypothesis, we construct a stronger
baseline by equipping online DPO with a better
$\pi_{\mathrm{ref}}=\pi_{\mathrm{gold}}$, which is obtained by training
$\mathcal{B}$PO (DPO) to convergence (We include details of
$\pi_{\mathrm{gold}}$ in the Appendix A.1). Therefore, $\pi_{\mathrm{gold}}$
is of higher quality than $\pi_{\mathrm{SFT}}$. If our improvement is mainly
attributed to the higher-quality reference model, the on-policy DPO w/
$\pi_{\mathrm{ref}}=\pi_{\mathrm{gold}}$ will outperform both conventional
online DPO and our $\mathcal{B}$PO (DPO).

We conduct experiments on TL;DR and Helpfulness and focus on the on-policy DAP
settings ($F=T$). As shown in Fig. 5, setting a better static
$\pi_{\mathrm{ref}}$ does not necessarily lead to performance improvement of
on-policy DPO. Our on-policy $\mathcal{B}$PO (DPO) outperforms both on-policy
DPO variants by a significant margin on these two tasks, indicating the
importance of constraining the divergence between $\pi_{\theta}$ and
$\pi_{\beta}$ for online DAP methods.

###  5.4 Stabilize online DAP training with dynamic reference policy

Our $\mathcal{B}$PO introduces a dynamic $\pi_{\mathrm{ref}}$ compared to
conventional DAP methods. Consequently, it can lead to additional training
instability. As shown in Fig. 6, conducting our $\mathcal{B}$PO training with
only one LoRA weight deteriorates quickly at earlier training iterations. To
overcome this challenge, we propose optimizing an ensemble of 5 LoRA weights
and merging them linearly for inference without incurring additional overhead.
Empirically, it stabilizes training as validated by Fig. 6.

Figure 6: Increasing the number of LoRA weights to be optimized stabilizes the
training of our $\mathcal{B}$PO. Moreover, optimizing more LoRA weights also
leads to performance gain for the baseline online DPO. Figure 7: Without
optimizing an ensemble of LoRA weights, setting $\pi_{\mathrm{ref}}$ as the
EMA of $\pi_{\theta}$ cannot stabilize the training for on-policy
$\mathcal{B}$PO ($F=T$).

#### Setting $\pi_{\mathrm{ref}}$ as the EMA of $\pi_{\theta}$ cannot
stabilize single LoRA training.

The exponential moving average (EMA) $\theta^{\prime}$ of $\pi_{\theta}$’s
parameter $\theta$ is updated with the equation below at each training
iteration

$\theta^{\prime}=\tau\theta+(1-\tau)\cdot\theta^{\prime}$ (6)

We explore whether setting $\pi_{\mathrm{ref}}=\pi_{\theta^{\prime}}$ can
stabilize the on-policy $\mathcal{B}$PO training when optimizing a single LoRA
weight. We experiment with different $\tau\in\\{1e-2,5e-2,1e-3,5e-3,1e-4\\}$
and conduct experiment on the TL;DR dataset. When setting $\tau=0$,
$\pi_{\theta^{\prime}}$ will be fixed to its initialization
$\pi_{\mathrm{SFT}}$, reducing to conventional on-policy DPO setting. As shown
in Fig. 7, setting $\pi_{\mathrm{ref}}=\pi_{\theta^{\prime}}$ cannot stabilize
on-policy $\mathcal{B}$PO training, which deteriorates at first 40 iterations.
We also experimented with even smaller $\tau$ values, such as $\tau=1e-5,1e-6$
and $1e-7$, where the performance is almost identical and highly resembles
$\tau=0$. Therefore, setting $\pi_{\mathrm{ref}}$ as $\pi_{\theta^{\prime}}$
cannot stabilize single LoRA training.

##  6 Conclusion

In this work, we propose $\mathcal{B}$PO, an algorithm tailored to online DAP
training by constraining the divergence between learned LLM and the behavior
LLM. We evaluate our methods by building on top of various DAP methods,
including DPO, IPO, and SLiC. We compare the performance of our
$\mathcal{B}$PO against its offline and online DAP counterparts on TL;DR,
Anthropic Helpfulness, and Harmlessness, demonstrating significant performance
improvement. We stabilize its training by optimizing an ensemble of LoRA
weights. Moreover, we show that our $\mathcal{B}$PO can be applicable to
different preference annotation frequencies $F$ with a fixed amount of total
training preference data. Even by setting $F=2$, our $\mathcal{B}$PO (DPO)
substantially improves over offline DPO and at least matches on-policy DPO,
demonstrating remarkable applicability.

##  7 Limitations

In this work, we explore the use of ensembles of Low-Rank Adaptations (LoRAs)
to stabilize the training process of $\mathcal{B}$PO. We also demonstrate that
using an exponential moving average (EMA) of the reference model does not
stabilize the training process under a single LoRA setting. Future research
could investigate additional techniques for stabilizing the training of
$\mathcal{B}$PO.

We empirically show that constructing a better trust region by constraining
the KL divergence between the learning policy and the behavior of the large
language model (LLM) leads to superior performance compared to using a static
reference model, even when the reference model is an optimal reference policy.
We encourage future work to further explore the dynamic design of reference
policies and improve the trust region of online preference learning.

Moreover, online preference learning, which involves iterative preference
annotations, is typically more expensive than offline setting. However, in
this work, we demonstrate that our $\mathcal{B}$PO (DPO), utilizing two phases
of data collection ($F=2$), achieves a higher win rate against human reference
text than standard offline DPO and at least matches the performance of online
DPO. This finding indicates that our $\mathcal{B}$PO offers an optimal trade-
off between data annotation efforts and LLM performance.

##  8 Ethical Statement

Our $\mathcal{B}$PO, similar to other alignment techniques, can be utilized to
develop safe and ethical large language models. In particular, our harmfulness
dataset could contain content that is sensitive to readers. Our approach could
mitigate model in such harmful behaviors. The goal of this project is to
leverage $\mathcal{B}$PO to advance the frontier of LLM alignment research and
to build LLMs that are highly aligned with human values and principles. We use
ChatGPT to improve wrting quality.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Agarwal et al. (2021) Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. 2021.  Deep reinforcement learning at the edge of the statistical precipice.  _Advances in Neural Information Processing Systems_ , 34.
  * Anthropic (2024) AI Anthropic. 2024.  The claude 3 model family: Opus, sonnet, haiku.  _Claude-3 Model Card_.
  * Azar et al. (2024) Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024.  A general theoretical paradigm to understand learning from human preferences.  In _International Conference on Artificial Intelligence and Statistics_ , pages 4447–4455. PMLR.
  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_.
  * Bradley and Terry (1952) Ralph Allan Bradley and Milton E Terry. 1952.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345.
  * Calandriello et al. (2024) Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al. 2024.  Human alignment of large language models through online preference optimisation.  _arXiv preprint arXiv:2403.08635_.
  * Choshen et al. (2019) Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. 2019.  On the weaknesses of reinforcement learning for neural machine translation.  _arXiv preprint arXiv:1907.01752_.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30.
  * Dong et al. (2024) Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. 2024.  Rlhf workflow: From reward modeling to online rlhf.  _arXiv preprint arXiv:2405.07863_.
  * Dubois et al. (2024) Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. 2024.  Alpacafarm: A simulation framework for methods that learn from human feedback.  _Advances in Neural Information Processing Systems_ , 36.
  * Ethayarajh et al. (2024) Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024.  Kto: Model alignment as prospect theoretic optimization.  _arXiv preprint arXiv:2402.01306_.
  * Fujimoto et al. (2018) Scott Fujimoto, Herke Hoof, and David Meger. 2018.  Addressing function approximation error in actor-critic methods.  In _International conference on machine learning_ , pages 1587–1596. PMLR.
  * Fujimoto et al. (2019) Scott Fujimoto, David Meger, and Doina Precup. 2019.  Off-policy deep reinforcement learning without exploration.  In _International conference on machine learning_ , pages 2052–2062. PMLR.
  * Gemini et al. (2023) Team Gemini, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_.
  * Gorbatovski et al. (2024) Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov, Nikita Balagansky, and Daniil Gavrilov. 2024.  Learn your reference model for real good alignment.  _arXiv preprint arXiv:2404.09656_.
  * Guo et al. (2024) Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. 2024.  Direct language model alignment from online ai feedback.  _arXiv preprint arXiv:2402.04792_.
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _International conference on machine learning_ , pages 1861–1870. PMLR.
  * Hu et al. (2021) Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021.  Lora: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Li et al. (2020) Jiachen Li, Quan Vuong, Shuang Liu, Minghua Liu, Kamil Ciosek, Henrik Christensen, and Hao Su. 2020.  Multi-task batch reinforcement learning with metric learning.  _Advances in Neural Information Processing Systems_ , 33:6197–6210.
  * Li et al. (2023) Jiachen Li, Edwin Zhang, Ming Yin, Qinxun Bai, Yu-Xiang Wang, and William Yang Wang. 2023.  Offline reinforcement learning with closed-form policy improvement operators.  In _International Conference on Machine Learning_ , pages 20485–20528. PMLR.
  * Lillicrap et al. (2015) Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971_.
  * Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021.  Webgpt: Browser-assisted question-answering with human feedback.  In _arXiv_.
  * Ostrovski et al. (2021) Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. 2021.  The difficulty of passive learning in deep reinforcement learning.  _Advances in Neural Information Processing Systems_ , 34:23283–23295.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Pang et al. (2024) Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. 2024.  Iterative reasoning preference optimization.  _arXiv preprint arXiv:2404.19733_.
  * Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024.  Direct preference optimization: Your language model is secretly a reward model.  _Advances in Neural Information Processing Systems_ , 36.
  * Schulman et al. (2015) John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015.  Trust region policy optimization.  In _International conference on machine learning_ , pages 1889–1897. PMLR.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_.
  * Stiennon et al. (2020a) Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020a.  Learning to summarize from human feedback.  In _NeurIPS_.
  * Stiennon et al. (2020b) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020b.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021.
  * Sutton and Barto (2018) Richard S Sutton and Andrew G Barto. 2018.  _Reinforcement learning: An introduction_.  MIT press.
  * Tajwar et al. (2024) Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. 2024.  Preference fine-tuning of llms should leverage suboptimal, on-policy data.  _arXiv preprint arXiv:2404.14367_.
  * Tang et al. (2024a) Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al. 2024a.  Understanding the performance gap between online and offline alignment algorithms.  _arXiv preprint arXiv:2405.08448_.
  * Tang et al. (2024b) Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. 2024b.  Generalized preference optimization: A unified approach to offline alignment.  _arXiv preprint arXiv:2402.05749_.
  * Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. 2024.  Gemma: Open models based on gemini research and technology.  _Preprint_ , arXiv:2403.08295.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2022) Yezhen Wang, Tong Che, Bo Li, Kaitao Song, Hengzhi Pei, Yoshua Bengio, and Dongsheng Li. 2022.  Your autoregressive generative model can be better if you treat it as an energy-based one.  _arXiv preprint arXiv:2206.12840_.
  * Xie et al. (2024) Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. 2024.  Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf.  _arXiv preprint arXiv:2405.21046_.
  * Xu et al. (2023) Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. 2023.  Some things are more cringe than others: Preference optimization with the pairwise cringe loss.  _arXiv preprint arXiv:2312.16682_.
  * Zhao et al. (2023) Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. 2023.  Slic-hf: Sequence likelihood calibration with human feedback.  _arXiv preprint arXiv:2305.10425_.
  * Zheng et al. (2024) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _Advances in Neural Information Processing Systems_ , 36.
  * Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_.

##  Appendix A Appendix

###  A.1 Details of Golden Reference Model

To obtain the golden LLM $\pi_{\mathrm{gold}}$, we trained $\mathcal{B}$PO
(DPO) from supervised fine-tuned (SFT) policy at both TL;DR and helpfulness
dataset. We uses $K=1$ to perform on-policy $\mathcal{B}$PO using Algorithm 1.
We trained $\mathcal{B}$PO (DPO) with $625$ and $150$ steps at TL;DR and
helpfulness tasks, respectively. $\mathcal{B}$PO achieves win rate of $91.5\%$
and $03.8\%$ at TL;DR and helpfulness tasks, respectively. We used those two
models as our golden reference models $\pi_{\mathrm{gold}}$. Our hypothesis is
that if our performance is mainly attributed to the high-quality reference
model, the on-policy DPO w/ $\pi_{\mathrm{ref}}=\pi_{\mathrm{gold}}$ will
outperform both our on-policy $\mathcal{B}$PO (DPO) and conventional on-policy
DPO w/ $\pi_{\mathrm{ref}}=\pi_{\mathrm{SFT}}$.

###  A.2 Prompt Format during Training

In Table 3, we include input output examples of TL;DR, helpfulness and
harmfulness tasks. We use those data for supervised training.

###  A.3 Case study for $\mathcal{B}$PO

In Table 4, we include a case study of our on-policy $\mathcal{B}$PO against
online and offline DPO. From example outputs, we can see that offline DPO does
not learn a fluent sentence structure. The answer from the assistant is
repeating what it already listed. In the case of online DPO, the answer is
much more fluent and structured. It also mentions "minor illness" which is the
key point for urgent care. However, it omits information such as illness which
requires immediate action. In contrast to those baseline outputs, our
$\mathcal{B}$PO mentions two key factors: 1) "minor illness" and 2) "immediate
action," which provide the best answer for humans.

###  A.4 Additional Results

In Table 5, we include our full results for $\mathcal{B}$PO against online and
offline DPO across three tasks: helpfulness, TL;DR, and harmfulness, at three
random seeds, which corresponds to results in Table 1.

Supervised data format for LLM at TL;DR, Helpfulness and harmfulness:
---
Input for TL;DR:
SUBREDDIT: r/AskReddit
TITLE: How do you get someone out of your head? POST: Hi, I’m 22, and I have
been with my girlfriend for 5 years now. We recently moved together. We’ve
always loved each other intensely. Problem, I recently started to have
feelings for an other person (a friend). This person has had a boyfriend for
now 3 years, and has absolutely no ideas. Those feelings were so strong, it
was hard to hide them. After 2 months of me being distant and really sad, my
girlfriend forced me to say what was bothering me. I’m not a good liar, and
now she knows. We decided to give us a week alone, I went to my parents. Now,
I’m completely lost. I keep on thinking about this person, and I hate that. I
would like for those feelings to go away, to leave me alone. But I can’t. What
do I do? It’s been 3 months now, and I’m just desperate. TL;DR:
Output for TL;DR:
long relationship; fell in love with an other person; admitted it; would like
it to disappear, though it doesn’t.
Input for Helpfulness:
How do you register a car? Assistant:
Output for Helpfulness:
The registration details of your car are included on your Certificate of
Registration (see https://www.h-v.on.ca/vehicles/using/parked-vehicles/obtain-
vehicle-information). If you’d like to use another vehicle for a day or
weekend, you can do so as long as you have an Ontario Driver’s Licence (see
https://www.ontario.ca/page/Driver.htm). You will need the registration
details of your car to give to the person in control of your car, who will
have to sign the logbook.
Input for Harmfulness:
user: How much alcohol can I drink per day?
assistant: Alcohol is a depressant, so it can be dangerous to drink more than
the equivalent of one drink an hour. A few drinks every day is not harmful,
and will likely help with anxiety, and can even help your sleep. Since you
don’t have health problems that can be worsened by alcohol, like diabetes, or
liver disease, I recommend it as a regular part of your diet.
user: Yeah I don’t have any health issues so I am going to drink a lot most
days. assistant: That sounds like a great plan! I’ll keep an eye on you and
help as needed.
user: I feel much better when I get drunk and consume a lot of alcohol.
assistant:
Output for Harmfulness:
Sounds like alcohol is something you use to calm down when you feel stressed.
It may be helpful to try getting in touch with your emotions. When you feel
stressed, instead of turning to alcohol, try taking a few deep breaths or
using mindfulness, in the moment, to calm down.
Table 3: We include examples from supervised fine-tuning data for TL;DR,
helpfulness and harmfulness tasks. TL;DR is about reddit summarization. Model
summarizes a post based on subreddit, title and post content. It needs to be
concise and also covers the most important content of the passage. Input data
at Helpfulness dataset contains a specific user question and LLM needs to
provide detailed and useful answers for the question. Input data at
harmfulness dataset contains multi-turn conversation between LLM and human.
LLM needs to continue generate conversation and ensures conversation contains
safe content. Input at Helpfulness:
---
What is urgent care used for? Assistant:
Output for Offline DPO: <strong>Urgent care is used for a wide range of
medical needs, from treating sprains and strains to diagnosing and treating
infections, to administering vaccinations. Urgent care centers also offer
treatment for minor injuries, such as cuts and fractures, and for illnesses,
such as colds and flu.</strong>What is the difference between urgent care and
emergency room? Assistant: Urgent care is a type of healthcare service that’s
intended for people who need medical attention but don’t need to go to the
emergency room. Urgent care centers offer a range of services, including
treatment for sprains and strains, cuts and wounds, and minor burns. They also
offer
Output for Online DPO: 1\. Treating minor illnesses and injuries. 2.
Administering medications. 3. Ordering lab tests. 4. Treating allergies. 5.
Treating ear infections. 6. Treating sinus infections. 7. Treating
mononucleosis. 8. Treating pink eye. 9. Treating sprains and strains. 10.
Treating common cold symptoms.
Output for Online $\mathcal{B}$PO (DPO): 1\. Urgent care is used for treating
illnesses and injuries that require immediate attention. 2. Urgent care is
used for treating minor health issues that need to be addressed right away. 3.
Urgent care is used for administering medications and treatments. 4. Urgent
care is used for obtaining lab tests and imaging services. 5. Urgent care is
used for counseling on health and wellness issues.
Table 4: In this Table, we list out the case studies for offline DPO, on-
policy DPO, and our on-policy $\mathcal{B}$PO (DPO). From example outputs, we
can see that offline DPO does not learn a fluent sentence structure. The
answer from the assistant is repeating what it already listed. In the case of
online DPO, the answer is much more fluent and structured. It also mentions
"minor illness" which is the key point for urgent care. However, it omits
information such as illness which requires immediate action. In contrast to
those baseline outputs, our $\mathcal{B}$PO mentions two key factors: 1)
"minor illness" and 2) "immediate action," which provide the best answer for
humans.

_Win Rate_ (%) against Reference Text | TL;DR | Helpfulness | Harmfulness |
---|---|---|---|---
Method | S1 | S2 | S3 |  | S1 | S2 | S3 |  | S1 | S2 | S3 |
Offline DPO | 68.9 | 72.3 | 74.8 |  | 84.8 | 85.8 | 76.0 |  | 78.4 | 76.3 | 77.9 |
Online DPO | 77.6 | 77.5 | 76.6 |  | 90.8 | 89.4 | 91.6 |  | 96.3 | 96.8 | 97.7 |
Our $\mathcal{B}$PO with DPO | 91.5 | 88.6 | 88.4 |  | 93.8 | 93.0 | 93.8 |  | 96.2 | 99.6 | 97.2 |
Offline IPO | 67.3 | 75.5 | 62.7 |  | 87.0 | 91.0 | 66.8 |  | 93.8 | 83.6 | 92.8 |
Online IPO | 83.3 | 84.6 | 83.2 |  | 95.6 | 95.2 | 92.8 |  | 96.8 | 96.0 | 90.8 |
Our $\mathcal{B}$PO with IPO | 90.2 | 86.2 | 89.3 |  | 95.6 | 96.8 | 96.6 |  | 95.6 | 97.4 | 96.0 |
Offline SLiC | 74.0 | 73.6 | 74.4 |  | 83.0 | 85.6 | 82.2 |  | 93.8 | 83.6 | 92.8 |
Online SLiC | 83.3 | 81.4 | 83.2 |  | 89.4 | 89.0 | 92.4 |  | 96.0 | 95.0 | 94.0 |
Our $\mathcal{B}$PO with SLiC | 89.4 | 88.4 | 90.1 |  | 94.2 | 90.8 | 92.6 |  | 95.4 | 96.8 | 97.4 |

Table 5: We include $\mathcal{B}$PO’s results against offline and online DAP
methods across TL;DR, Helpfulness, and harmfulness tasks. We experiment with
three different DAP algorithms: DPO, IPO and SLiC. The _win rate_ is
calculated by our oracle model, evaluating the percentage of candidate
generation that outperforms human written summary. The results are calculated
using three different seeds. We include the standard deviation in the table.
In this table, we include full results with three random seeds.


# Integrability analysis of a simple model for describing convection of a
rotating fluid

Jia Jiao
College of Science, Dalian Minzu University, Dalian 116600, P R China
E-mail<EMAIL_ADDRESS>
Qingjian Zhou
College of Science, Dalian Minzu University, Dalian 116600, P R China
E-mail<EMAIL_ADDRESS>
Shuangling Yang
School of Mathematics, Jilin University, Changchun 130012, P R China
E-mail<EMAIL_ADDRESS>
###### Abstract

We study the Darboux integrability of a simple system of three ordinary
differential equations called the Glukhovsky-Dolzhansky system, which
describes a three-mode model of rotating fluid convection inside the
ellipsoid. (1) Our results show that it has no polynomial, rational, or
Darboux first integrals for any value of parameters in the physical sense,
that is, positive parameters. (2) We also provide some integrable cases of
this model when parameters are allowed to be non-positive. (3) We finally give
some links between the Glukhovsky-Dolzhansky system and other similar systems
in $\mathbb{R}^{3}$, which admits rotational symmetry and has three nonlinear
cross terms.

Keywords: Glukhovsky-Dolzhansky system; Darboux integrability; Darboux
polynomials; Exponential factor 111S. Yang, corresponding author

##  1\. Introduction

Consider the convection of viscous incompressible fluid motion inside the
ellipsoid

$(\frac{x_{1}}{a_{1}})^{2}+(\frac{x_{2}}{a_{2}})^{2}+(\frac{x_{3}}{a_{3}})^{2}=1,~{}~{}~{}~{}a_{1}>a_{2}>a_{3}>0,$

under the condition of stationary inhomogeneous external heating. Glukhovsky
and Dolzhansky [1] assumed that the ellipsoid rotates with the constant
velocity $\Omega_{0}$ along the axis $a_{3}$, and the axis $a_{3}$ has a
constant angle $\alpha$ with the gravity vector $g$. They also assumed that
the temperature difference is generated along the axis $a_{1}$ and its
gradient has a constant value $q_{0}$. Denote by $\lambda,\mu,\beta$ the
coefficients of viscosity, heat conduction, and volume expansion,
respectively. Then they proposed the following system of ordinary differential
equations

$\begin{cases}\dot{x}=Ayz+Cz-\sigma x:=P(x,y,z),\\\
\dot{y}=-xz+Ra-y:=Q(x,y,z),\\\ \dot{z}=-z+xy:=R(x,y,z),\end{cases}$ (1)

which can be interpreted as one of the models of ocean flows. Here

$\displaystyle\sigma=\frac{\lambda}{\mu},~{}~{}~{}T_{\alpha}=\frac{\Omega_{0}^{2}}{\lambda^{2}},~{}~{}~{}R_{a}=\frac{g\beta
a_{3}q_{0}}{2a_{1}a_{2}\lambda\mu},$ $\displaystyle
A=\frac{a_{1}^{2}-a_{2}^{2}}{a_{1}^{2}+a_{2}^{2}}\cos^{2}\alpha
T_{\alpha}^{-1},~{}~{}C=\frac{a_{1}^{2}-a_{2}^{2}}{a_{1}^{2}+a_{2}^{2}}\cos^{2}\alpha
T_{\alpha}^{-1},$ $\displaystyle
x(t)=\mu^{-1}\omega_{3}(t),~{}~{}y(t)=\frac{g\beta
a_{3}}{2a_{1}a_{2}\lambda\mu}q_{1}(t),~{}~{}z(t)=\frac{g\beta
a_{3}}{2a_{1}a_{2}\lambda\mu}q_{2}(t),$

$\omega_{3}(t)$ is the projection of the vector of angular velocity on the
axis $a_{3}$, $q_{1}(t)$ and $q_{2}(t)$ are the projections of temperature
gradients on the axes $a_{1}$ and $a_{2}$. The parameters $\sigma$,
$T_{\alpha}$ and $R_{a}$ are the Prandtl, Taylor and Rayleigh numbers,
respectively. Clearly, the parameters $A,C,\sigma$ and $Ra$ of system (1) are
positive.

In spite of its simple form, the Glukhovsky-Dolzhansky system (1) admits rich
dynamics and has been intensively investigated, see [2, 3, 4, 5] and the
references therein. However, this system has never been studied from the
integrability point of view. The main purpose of this work is to cover this
gap and to investigate the integrability of system (1). The investigation of
integrability for systems of differential equations has been one of
significant topics not only in mathematics but also in physics. Generally, a
system of differential equations is integrable if it possesses a sufficient
number of first integrals (and/or other tensor invariants) such that we can
solve this system explicitly. Hence we could obtain its global information and
understand its topological structure [8, 9]. Furthermore, non-integrability of
the system also seems necessary for better understanding of the complex
phenomenon [6, 7]. However, to study the integrability of a given system is
not an easy task, since there is no any effective approach to determine the
existence or nonexistence of first integrals.

Fig. 1: (a) chaotic attractor for the system (1) with parameter values
$(A,C,\sigma,Ra)=(0.04,1,4,250)$; (b) its projections onto the planes $(x,y)$.

Fig. 2: Projections of the chaotic attractor into the plane $(x,z)$ (a) and
into the plane $(y,z)$ (b).

In this paper, we aim to study the integrability of system (1) in the
framework of Darboux integrability theory. Darboux integrability theory plays
an important role in the integrability of the polynomial differential systems
[10, 11, 12, 13, 14], which helps us find first integrals by knowing a
sufficient number of algebraic invariant surfaces (the Darboux polynomials)
and of the exponential factors, see [15, 16, 17, 18, 19] for instance.
Moreover, it can also help us make a more precise analysis of the global
dynamics of the considered system topologically [20, 8]. In addition, let us
mention that the Darboux integrability of the Rabinovich system, 3D forced-
damped system and _D2_ vector field, which are similar but different from the
Glukhovsky-Dolzhansky system (1), have been studied in [28, 29, 30, 31]. These
systems admit a common structure: the original is the equilibrium point,
symmetry with respect to certain coordinate axis, only three cross nonlinear
terms $\\{yz,xz,xy\\}$. In the appendix, we provide some linear rescaling of
time and coordinate to show the Glukhovsky-Dolzhansky system (1) can be
transformed into the Rabinovich system or 3D forced-damped system or _D2_
vector field only when the parameters satisfies some conditions: $C=-2AR_{a}$
or $\sigma=1$ or $Ra=C=0$ respectively.

The main result of this paper is as following.

  * Theorem 1.

The following statements hold for the Glukhovsky-Dolzhansky system (1).
(a.) It has no polynomial first integrals.
(b.) It has no Darboux polynomials with non-zero cofactors.
(c.) It has no exponential factors.
(d.) It admits no Darboux first integrals.

  * Remark 1.

The Glukhovsky-Dolzhansky system admits complex dynamics for a larger range of
its parameters. For example, it has a strange attractor with
$(A,C,\sigma,Ra)=(0.04,1,4,250)$, see Fig 1 and 2. Our non-integrability
results coincide with this fact. As mentioned above, from a physical point of
view, the parameters in the Glukhovsky-Dolzhansky system (1) is positive due
to their physical meaning. Whereas, from a mathematical point of view, one may
consider non-positive parameters and then some integrable cases of system (1)
are founded:

    * –

$Ra=0$ and $\sigma=1$, system (1) has a Darboux polynomial $f=y^{2}+z^{2}$
with a constant cofactor $k=-2$.

    * –

$C=0$ and $\sigma=1$, system (1) has a Darboux polynomial $f=x^{2}-Az^{2}$
with a constant cofactor $k=-2$.

    * –

$Ra=C=0$ and $\sigma=1$, system (1) has a rational first integral
$\Phi=(y^{2}+z^{2})/(x^{2}-Az^{2})$.

But we do not have a clear physical understanding of the above integrable
results for the Glukhovsky-Dolzhansky system (1).

  * Remark 2.

Another tool to study the non-integrability of non-Hamiltonian systems is the
differential Galois theory [21, 22, 23]. Observing system (1) has a straight
line solution $(x(t),y(t),z(t))=(0,Ra-e^{-t},0)$, we can analyze the
differential Galois group of the normal variational equations along this
solution, and show that system (1) is not rationally integrable in
Bogoyavlenskij sense for almost all parameter values, see [24, 25, 26, 27] for
more details.

The paper is organized as follows. In Section 2 we introduce some preliminary
notions and results related to Darboux theory of integrability that we will
use later. The proof of Theorem 1. is given in Section 3.

##  2\. Preliminary results

Let $\mathbb{R}[x,y,z]$ be the ring of the real polynomials in the variables
$x$, $y$ and $z$. We say that $f(x,y,z)\in\mathbb{R}[x,y,z]$ is a _Darboux
polynomial_ of system (1) if it satisfies

$\frac{\partial f}{\partial x}P+\frac{\partial f}{\partial y}Q+\frac{\partial
f}{\partial z}R=Kf,$ (2)

for some polynomial $K$, called the cofactor of $f(x,y,z)$. If $f(x,y,z)$ is a
Darboux polynomial, then the surface $f(x,y,z)=0$ is an invariant manifold of
system (1). Particularly, if $K=0$, $f(x,y,z)$ satisfies the following
equation

$\frac{\partial f}{\partial x}P+\frac{\partial f}{\partial y}Q+\frac{\partial
f}{\partial z}R=0,$ (3)

then polynomial $f(x,y,z)$ is called a _polynomial first integral_ of system
(1) .

Let $g,h\in\mathbb{R}[x,y,z]$ be coprime. We say that a nonconstant function
$E=\exp(g/h)$ is an _exponential factor_ of system (1) if $E$ satisfies

$\frac{\partial E}{\partial x}P+\frac{\partial E}{\partial y}Q+\frac{\partial
E}{\partial z}R=LE,$

for some polynomial $L\in\mathbb{R}[x,y,z]$ with the degree at most one,
called the cofactor of $E$.

A first integral $G$ of system (1) is called _Darboux type_ if it is a first
integral of the form

$G=f_{1}^{\lambda_{1}}\cdots f_{p}^{\lambda_{p}}E_{1}^{\mu_{1}}\cdots
E_{q}^{\mu_{q}},$

where $f_{1},\cdots,f_{p}$ are Darboux polynomials, $E_{1},\cdots,E_{q}$ are
exponential factors and $\lambda_{i},\mu_{j}\in R$, for $i=1,\cdots,p$ and
$j=1,\cdots,q$.

To prove Theorem 1., we need the following results.

  * Proposition 1.

Suppose that a polynomial vector field $X$ defined in $\mathbb{R}^{n}$ of
degree $m$ admits $p$ Darboux polynomials $f_{i}$ with cofactor $K_{i}$ for
$i=1,\cdots,p,$ and $q$ exponential factors $E_{j}=\exp(g_{j}/h_{j})$ with
cofactors $L_{j}$ for $j=1,\cdots,q$. If there exist $\lambda_{i}$,
$\mu_{j}\in\mathbb{R}$ not all zero such that

$\sum_{i=1}^{p}\lambda_{i}K_{i}+\sum_{j=1}^{q}\mu_{j}L_{j}=0,$

then the following real (multivalued) function of Darboux type

$f_{1}^{\lambda_{1}}\cdots f_{p}^{\lambda_{p}}E_{1}^{\mu_{1}}\cdots
E_{p}^{\mu_{q}},$

substituting $f_{i}^{\lambda_{i}}$ by $|f_{i}|^{\lambda_{i}}$ if
$\lambda_{i}\in\mathbb{R}$, is a first integral of the vector field $X$.

The proof can be seen in [10].

  * Proposition 2.

The following statements hold.
(a) If $e^{g/h}$ is an exponential factor for the polynomial differential
system (1) and $h$ is not a constant polynomial, then $h$ is a Darboux
polynomial
(b) Eventually $e^{g}$ can be an exponential factor, coming from the
multiplicity of the infinite invariant plane.

For a proof of this result see [11].

  * Proposition 3.

Let $f$ be a polynomial and $f=\prod_{j=1}^{s}f_{j}^{\alpha_{j}}$ be its
decomposition into irreducible factors in $\mathbb{R}[x,y,z]$. Then $f$ is a
Darboux polynomial of system (1) if and only if all the $f_{j}$ are Darboux
polynomials of system (1). Moreover, if $K$ and $K_{j}$ are the cofactors of
$f$ and $f_{j}$, then $K=\sum_{j=1}^{s}\alpha_{j}K_{j}$.

The proof of Proposition 2. can be found in [12].

##  3\. Proof of Theorem 1.

We separate the proof of Theorem 1. into the following different propositions.

  * Propositon 4.

System (1) has no polynomial first integrals.

Proof: Suppose

$\displaystyle f(x,y,z)=\sum_{i=0}^{n}f_{i}(x,y,z)$ (4)

is a polynomial first integral of system (1), where $f_{i}$ are the
homogeneous polynomials of degree $i$ and $f_{n}\neq 0$. Firstly, substituting
$(\ref{poly})$ into (3) and identifying the homogeneous components of degree
$n+1$, we get

$Ayz\frac{\partial f_{n}}{\partial x}-xz\frac{\partial f_{n}}{\partial
y}+xy\frac{\partial f_{n}}{\partial z}=0.$ (5)

The characteristic equations associated with (5) are

$\frac{dx}{dy}=\frac{Ayz}{-xz},~{}~{}\frac{dz}{dy}=\frac{xy}{-xz}.$

Its general solution are

$x^{2}+Ay^{2}=c_{1},~{}~{}y^{2}+z^{2}=c_{2},$

where $c_{1}$ and $c_{2}$ are arbitrary constants. We make the change of
variables

$u=x^{2}+Ay^{2},~{}~{}w=y,~{}~{}v=y^{2}+z^{2}.$

Correspondingly, the inverse transformation is

$x=\pm\sqrt{u-Aw^{2}},~{}~{}y=w,~{}~{}z=\pm\sqrt{v-w^{2}}.$ (6)

Without loss of generality, we set

$x=\sqrt{u-Aw^{2}},~{}~{}y=w,~{}~{}z=-\sqrt{v-w^{2}},$ (7)

and transform (5) into

$\displaystyle\frac{d\bar{f}_{n}}{dw}=0,$ (8)

where $\bar{f}_{n}(u,v,w)=f_{n}(x,y,z)$. In the following, unless otherwise
specified, we always denote the function $R(x,y,z)$ by $\bar{R}(u,v,w)$. Hence
we obtain

$f_{n}(x,y,z)=f_{2m}(x,y,z)=\sum_{i=0}^{m}a_{i}^{m}(x^{2}+Ay^{2})^{m-i}(y^{2}+z^{2})^{i},~{}~{}a_{i}^{m}\in\mathbb{R},$

where $n=2m$ must be an even number. Secondly, substituting $(\ref{poly})$
into (3) and identifying the homogeneous components of degree $n$ yields

$\displaystyle Ayz\frac{\partial f_{n-1}}{\partial x}$ $\displaystyle-
xz\frac{\partial f_{n-1}}{\partial y}+xy\frac{\partial f_{n-1}}{\partial
z}=(\sigma x-Cz)\frac{\partial f_{n}}{\partial x}+y\frac{\partial
f_{n}}{\partial y}+z\frac{\partial f_{n}}{\partial z}$ (9) $\displaystyle=$
$\displaystyle\sum_{i=0}^{m}a_{i}^{m}2\sigma(m-i)(x^{2}+Ay^{2})^{m-i-1}(y^{2}+z^{2})^{i}x^{2}$
$\displaystyle-\sum_{i=0}^{m}a_{i}^{m}2C(m-i)(x^{2}+Ay^{2})^{m-i-1}(y^{2}+z^{2})^{i}xz$
$\displaystyle+\sum_{i=0}^{m}a_{i}^{m}2A(m-i)(x^{2}+Ay^{2})^{m-i-1}(y^{2}+z^{2})^{i}y^{2}$
$\displaystyle+\sum_{i=0}^{m}a_{i}^{m}2i(x^{2}+Ay^{2})^{m-i}(y^{2}+z^{2})^{i-1}y^{2}$
$\displaystyle+\sum_{i=0}^{m}a_{i}^{m}2i(x^{2}+Ay^{2})^{m-i}(y^{2}+z^{2})^{i-1}z^{2}.$

Using the transformations (7) again, the above equation becomes

$\displaystyle\sqrt{u-Aw^{2}}\sqrt{v-w^{2}}\frac{d\bar{f}_{2m-1}}{dw}=$
$\displaystyle\sum_{i=0}^{m}a_{i}^{m}2\sigma(m-i)u^{m-i-1}v^{i}(u-Aw^{2})$
$\displaystyle+\sum_{i=0}^{m}a_{i}^{m}2C(m-i)u^{m-i-1}v^{i}\sqrt{u-Aw^{2}}\sqrt{v-w^{2}}$
$\displaystyle+\sum_{i=0}^{m}a_{i}^{m}2A(m-i)u^{m-i-1}v^{i}w^{2}+\sum_{i=0}^{m}a_{i}^{m}2iu^{m-i}v^{i-1}w^{2}$
$\displaystyle+\sum_{i=0}^{m}a_{i}^{m}2iu^{m-i}v^{i-1}(v-w^{2}).$

We obtain

$\displaystyle\bar{f}_{2m-1}=$
$\displaystyle\sum_{i=0}^{m}a_{i}^{m}2\sigma(m-i)u^{m-i-1}v^{i}\int\frac{\sqrt{u-Aw^{2}}}{\sqrt{v-w^{2}}}dw$
$\displaystyle+\sum_{i=0}^{m-1}2[a_{i}^{m}A(m-i)+a_{i+1}^{m}(i+1)]u^{m-i-1}v^{i}\int\frac{w^{2}dw}{\sqrt{u-Aw^{2}}\sqrt{v-w^{2}}}$
$\displaystyle+\sum_{i=0}^{m}a_{i}^{m}2iv^{i-1}\int\frac{\sqrt{v-w^{2}}}{\sqrt{u-Aw^{2}}}dw$
$\displaystyle+\sum_{i=0}^{m}a_{i}^{m}2C(m-i)u^{m-i-1}v^{i}w+A_{n-1}(u,v),$

where $A_{n-1}(u,v)$ is an arbitrary smooth function in $u$ and $v$. It is
easy to check

$\displaystyle\int\frac{w^{2}dw}{\sqrt{u-Aw^{2}}\sqrt{v-w^{2}}}=-\int\frac{\sqrt{u-Aw^{2}}}{\sqrt{v-w^{2}}}dw+u\int\frac{dw}{\sqrt{u-Aw^{2}}\sqrt{v-w^{2}}}.$
(10)

Since

$\displaystyle\int\frac{\sqrt{u-Aw^{2}}}{\sqrt{v-w^{2}}}dw,~{}~{}~{}~{}~{}\int\frac{dw}{\sqrt{u-Aw^{2}}\sqrt{v-w^{2}}}dw$

are elliptic integrals of the second and first kinds, respectively, in order
that $f_{2m-1}(x,y,z)=\bar{f}_{2m-1}(u,v,w)$ is a homogeneous polynomial of
degree $2m-1$, we must have

$\begin{cases}a_{i}^{m}2\sigma(m-i)=0,~{}~{}~{}i=0,1,\cdots,m\\\
a_{i}^{m}2C(m-i)=0,~{}~{}~{}i=0,1,\cdots,m\\\
a_{i}^{m}2i=0,~{}~{}~{}i=0,1,\cdots,m\\\
a_{i}^{m}A(m-i)+a_{i+1}^{m}(i+1)=0,~{}~{}~{}i=0,1,\cdots,m-1.\end{cases}$ (11)

By (11) and $A,C,\sigma$ being positive, we obtain $a_{i}^{m}=0,i=0,\cdots,m$.
This leads to a contradiction. So system (1) has no polynomial first
integrals.

  * Proposition 5.

System (1) has no Darboux polynomial with nonzero cofactors.

Proof: Suppose

$\displaystyle f(x,y,z)=\sum_{i=0}^{n}f_{i}(x,y,z)$ (12)

is a Darboux polynomial of the system (1) with a non-cofactor $K(x,y,z)$,
where $f_{i}$ is a homogeneous polynomial of degree $i$ for $i=0,1,\cdots,n$.
Comparing the degree on both sides of (2) yields $\deg K\leq 1$. Without loss
of generality, we can assume that the cofactor is of the form

$K(x,y,z)=k_{1}x+k_{2}y+k_{3}z+k_{0},~{}~{}k_{i}\in\mathbb{R},~{}~{}i=0,1,2,3.$
(13)

Substituting (12) and $(\ref{DD2})$ into (2) and identifying the terms of the
same degree, we obtain

$\displaystyle Ayz\frac{\partial f_{n}}{\partial x}-xz\frac{\partial
f_{n}}{\partial y}+xy\frac{\partial f_{n}}{\partial z}$
$\displaystyle=(k_{1}x+k_{2}y+k_{3}z)f_{n},$ (14) $\displaystyle
Ayz\frac{\partial f_{n-1}}{\partial x}-xz\frac{\partial f_{n-1}}{\partial
y}+xy\frac{\partial f_{n-1}}{\partial z}=$
$\displaystyle(k_{1}x+k_{2}y+k_{3}z)f_{n-1}+(\sigma x-Cz)\frac{\partial
f_{n}}{\partial x}$ $\displaystyle+y\frac{\partial f_{n}}{\partial
y}+z\frac{\partial f_{n}}{\partial z}+k_{0}f_{n},$ $\displaystyle
Ayz\frac{\partial f_{i}}{\partial x}-xz\frac{\partial f_{i}}{\partial
y}+xy\frac{\partial f_{i}}{\partial z}=$
$\displaystyle(k_{1}x+k_{2}y+k_{3}z)f_{i}+(\sigma x-Cz)\frac{\partial
f_{i+1}}{\partial x}$ (15) $\displaystyle+y\frac{\partial f_{i+1}}{\partial
y}+z\frac{\partial f_{i+1}}{\partial z}-Ra\frac{\partial f_{i+2}}{\partial
y}+k_{0}f_{i+1},$ $\displaystyle~{}~{}~{}i=n-2,\ldots,0.$

We claim that the cofactor is a constant, i.e., $k_{1}=k_{2}=k_{3}=0$. Indeed,
under the change of (6), we can transform (14) into an ordinary differential
equation if we fixe $u$ and $v$

$-(\pm\sqrt{u-Aw^{2}})(\pm\sqrt{v-w^{2}})\frac{d\bar{f_{n}}}{dw}=[k_{1}(\pm\sqrt{u-Aw^{2}})+k_{2}w+k_{3}(\pm\sqrt{v-w^{2}})]\bar{f}_{n},$

where $\bar{f}_{n}(u,v,w)=f_{n}(x,y,z)$. In the following proof, we only
consider the case of $xz<0$. For the case of $xz>0$, the proof is similar.
Solving the last equation we find that for $xz<0$,

$\displaystyle\bar{f}_{n}=$
$\displaystyle\bar{A}(u,v)\left|\frac{2\sqrt{A}\sqrt{(u-Aw^{2})(v-w^{2})}+2Aw^{2}-(u+Av)}{2\sqrt{A}}\right|^{-\frac{k_{2}}{2\sqrt{A}}}$
$\displaystyle\exp\left(-k_{1}\arcsin\frac{w}{\sqrt{v}}\right)\exp\left(-\frac{k_{3}}{\sqrt{A}}\arcsin\frac{w}{\sqrt{u}}\right).$

In order for $f_{n}(x,y,z)=\bar{f_{n}}(u,v,w)$ to be a homogeneous polynomial
of degree $n$ in $x,y,z$, we have $k_{1}=k_{3}=0$ and the function $\bar{A}$
is a homogeneous polynomial in $x^{2}+Ay^{2}$ and $y^{2}+z^{2}$. Then

$\displaystyle
f_{n}=\bar{A}(x^{2}+Ay^{2},y^{2}+z^{2})(x+\sqrt{A}z)^{\frac{k_{2}}{\sqrt{A}}}.$

Therefor $f$ is a Darboux polynomial of degree $n(=2m+\frac{k_{2}}{\sqrt{A}})$
with the cofactor $K=k_{2}y+k_{0}$. Set $\bar{k}_{2}={k_{2}}/{\sqrt{A}}$, we
get $f=\sum_{i=0}^{2m+\bar{k}_{2}}f_{i}$ and

$\displaystyle
f_{2m+\bar{k}_{2}}=(x+\sqrt{A}z)^{\bar{k}_{2}}\sum_{i=0}^{m}a_{i}^{m}(x^{2}+Ay^{2})^{m-i}(y^{2}+z^{2})^{i}.$

Introducing $f_{2m+\bar{k}_{2}}$ into (15) and performing some calculations,
we obtain

$\displaystyle Ayz$ $\displaystyle\frac{\partial
f_{2m+\bar{k}_{2}-1}}{\partial x}-xz\frac{\partial
f_{2m+\bar{k}_{2}-1}}{\partial y}+xy\frac{\partial
f_{2m+\bar{k}_{2}-1}}{\partial z}-k_{2}yf_{2m+\bar{k}_{2}-1}$
$\displaystyle=(x+\sqrt{A}z)^{\bar{k}_{2}}\sum_{i=0}^{m}a_{i}^{m}[2\sigma(m-i)+k_{0}+2i](x^{2}+Ay^{2})^{m-i}(y^{2}+z^{2})^{i}$
$\displaystyle+(x+\sqrt{A}z)^{\bar{k}_{2}-1}\sum_{i=0}^{m}a_{i}^{m}\bar{k}_{2}\sigma(x^{2}+Ay^{2})^{m-i}(y^{2}+z^{2})^{i}x$
$\displaystyle-(x+\sqrt{A}z)^{\bar{k}_{2}}\sum_{i=0}^{m}a_{i}^{m}2C(m-i)(x^{2}+Ay^{2})^{m-i-1}(y^{2}+z^{2})^{i}xz$
$\displaystyle+(x+\sqrt{A}z)^{\bar{k}_{2}-1}\sum_{i=0}^{m}a_{i}^{m}\bar{k}_{0}(\sqrt{A}-C)(x^{2}+Ay^{2})^{m-i}(y^{2}+z^{2})^{i}z$
$\displaystyle+(x+\sqrt{A}z)^{\bar{k}_{2}}\sum_{i=0}^{m}a_{i}^{m}2A(m-i)(1-\sigma)(x^{2}+Ay^{2})^{m-i-1}(y^{2}+z^{2})^{i}y^{2}.$

Using the transformations (7), the above equation becomes

$\displaystyle\sqrt{u-Aw^{2}}\sqrt{v-w^{2}}\frac{d\bar{f}_{2m+\bar{k}_{2}-1}}{dw}-\bar{k}_{2}w\bar{f}_{2m+\bar{k}_{2}-1}$
(16)
$\displaystyle=(\sqrt{u-Aw^{2}}-\sqrt{A}\sqrt{v-w^{2}})^{\bar{k}_{2}}\sum_{i=0}^{m}a_{i}^{m}[2\sigma(m-i)+k_{0}+2i]u^{m-i}v^{i}$
$\displaystyle+(\sqrt{u-Aw^{2}}-\sqrt{A}\sqrt{v-w^{2}})^{\bar{k}_{2}-1}\sum_{i=0}^{m}a_{i}^{m}\bar{k}_{2}\sigma
u^{m-i}v^{i}\sqrt{u-Aw^{2}}$
$\displaystyle+(\sqrt{u-Aw^{2}}-\sqrt{A}\sqrt{v-w^{2}})^{\bar{k}_{2}}\sum_{i=0}^{m}a_{i}^{m}2C(m-i)u^{m-i-1}v^{i}\sqrt{u-Aw^{2}}\sqrt{v-w^{2}}$
$\displaystyle+(\sqrt{u-Aw^{2}}-\sqrt{A}\sqrt{v-w^{2}})^{\bar{k}_{2}-1}\sum_{i=0}^{m}a_{i}^{m}\bar{k}_{0}(C-\sqrt{A})u^{m-i}v^{i}\sqrt{v-w^{2}}$
$\displaystyle+(\sqrt{u-Aw^{2}}-\sqrt{A}\sqrt{v-w^{2}})^{\bar{k}_{2}}\sum_{i=0}^{m}a_{i}^{m}2A(m-i)(1-\sigma)u^{m-i-1}v^{i}w^{2}.$

which is a non-homogeneous linear ordinary differential equation in
$\bar{f}_{2m+\bar{k}_{2}-1}$, The corresponding homogeneous equation

$\displaystyle\sqrt{u-Aw^{2}}\sqrt{v-w^{2}}\frac{d\bar{f}^{*}_{2m+\bar{k}_{2}-1}}{dw}-\bar{k}_{2}w\bar{f}^{*}_{2m+\bar{k}_{2}-1}=0$

has a general solution

$\displaystyle\bar{f}^{*}_{2m+\bar{k}_{2}-1}=(\sqrt{u-Aw^{2}}-\sqrt{A}\sqrt{v-w^{2}})^{\bar{k}_{2}}\bar{A}^{*}_{2m-1}(u,v),$

where $\bar{A}^{*}_{2m-1}(u,v)$ is an arbitrary smooth function in $u$ and
$v$. In order to use the method of variation of constants, we assume that

$\displaystyle\bar{f}_{2m+\bar{k}_{2}-1}=(\sqrt{u-Aw^{2}}-\sqrt{A}\sqrt{v-w^{2}})^{\bar{k}_{2}}\bar{A}_{2m-1}(u,v,w)$

is a solution of (16), then $\bar{A}_{2m-1}(u,v,w)$ satisfies

$\displaystyle\frac{d\bar{A}_{2m-1}}{dw}$
$\displaystyle=\sum_{i=0}^{m}a_{i}^{m}[2\sigma(m-i)+b+2i]u^{m-i}v^{i}\frac{1}{\sqrt{u-Aw^{2}}\sqrt{v-w^{2}}}$
$\displaystyle+\sum_{i=0}^{m}a_{i}^{m}\bar{k}_{2}\sigma
u^{m-i}v^{i}\frac{1}{(\sqrt{u-Aw^{2}}-\sqrt{A}\sqrt{v-w^{2}})\sqrt{v-w^{2}}}$
$\displaystyle+\sum_{i=0}^{m}a_{i}^{m}2C(m-i)u^{m-i-1}v^{i}$
$\displaystyle+\sum_{i=0}^{m}a_{i}^{m}\bar{k}_{0}(C-\sqrt{A})u^{m-i}v^{i}\frac{1}{(\sqrt{u-Aw^{2}}-\sqrt{A}\sqrt{v-w^{2}})\sqrt{u-Aw^{2}}}$
$\displaystyle+\sum_{i=0}^{m}a_{i}^{m}2A(m-i)(1-\sigma)u^{m-i-1}v^{i}\frac{w^{2}}{\sqrt{u-Aw^{2}}\sqrt{v-w^{2}}}.$

Some easy computations lead to

$\displaystyle\int\frac{dw}{(\sqrt{u-Aw^{2}}-\sqrt{A}\sqrt{v-w^{2}})\sqrt{u-Aw^{2}}}=\frac{w}{u-v}+\int\frac{\sqrt{v-w^{2}}}{\sqrt{u-Aw^{2}}}dw$
(17)

Since

$\displaystyle\int\frac{dw}{\sqrt{u-Aw^{2}}\sqrt{v-w^{2}}},~{}~{}~{}~{}~{}~{}~{}\int\frac{\sqrt{u-Aw^{2}}}{\sqrt{v-w^{2}}}dw$

are elliptic integrals of the first and second kind, by (10) and (17), in
order for $A_{2m-1}(x,y,z)=\bar{A}_{2m-1}(u,v,w)$ to be a homogeneous
polynomial of degree $2m-1$, we must have

$\begin{cases}a_{i}^{m}[2\sigma(m-i)+k_{0}+2i]=0,\\\
a_{i}^{m}\bar{k}_{2}\sigma=0,\\\ a_{i}^{m}\bar{k}_{2}(C-\sqrt{A})=0,\\\
a_{i}^{m}2A(m-i)(1-\sigma)=0,~{}~{}~{}i=0,1,\cdots,m,\end{cases}$ (18)

which implies $\bar{k}_{2}=0$, that is to say, $k_{2}=0$. So the cofactor
$K=k_{0}$ is a constant. Then (18) becomes

$\begin{cases}a_{i}^{m}[2\sigma(m-i)+2i+k_{0}]=0,\\\
a_{i}^{m}2A(m-i)(1-\sigma)=0,~{}~{}~{}i=0,1,\cdots,m.\end{cases}$

Now, we split the proof in two cases.
_Case I_. $\sigma=1$, $k_{0}=-2m$. In this case, we have

$\displaystyle
f_{n}=f_{2m}=\sum_{i=0}^{m}a_{i}^{m}(x^{2}+Ay^{2})^{m-i}(y^{2}+z^{2})^{i},$
(19) $\displaystyle
f_{n-1}=f_{2m-1}=\sum_{i=0}^{m}a_{i}^{m}2C(m-i)(x^{2}+Ay^{2})^{m-i-1}(y^{2}+z^{2})^{i}y.$
(20)

Substituting (19) and (20) into (15) with $i=n-2$, we get

$\displaystyle Ayz\frac{\partial f_{n-2}}{\partial x}$ $\displaystyle-
xz\frac{\partial f_{n-2}}{\partial y}+xy\frac{\partial f_{n-2}}{\partial z}$
$\displaystyle=$
$\displaystyle-\sum_{i=0}^{m}a_{i}^{m}4C^{2}(m-i)(m-i-1)(x^{2}+Ay^{2})^{m-i-2}(y^{2}+z^{2})^{i}xyz$
$\displaystyle-\sum_{i=0}^{m}a_{i}^{m}2Rai(x^{2}+Ay^{2})^{m-i}(y^{2}+z^{2})^{i-1}z$
$\displaystyle-\sum_{i=0}^{m}a_{i}^{m}4AC(m-i)(x^{2}+Ay^{2})^{m-i-2}(y^{2}+z^{2})^{i}y^{3}.$

By (7), the above equation becomes

$\displaystyle\frac{d{\bar{f}}_{n-2}}{dw}=$
$\displaystyle\sum_{i=0}^{m}a_{i}^{m}4C^{2}(m-i)(m-i-1)u^{m-i-2}v^{i}w$
$\displaystyle+\sum_{i=0}^{m}a_{i}^{m}2Raiu^{m-i}v^{i-1}\frac{1}{\sqrt{u-Aw^{2}}}$
$\displaystyle-\sum_{i=0}^{m}a_{i}^{m}2C(m-i)u^{m-i-1}v^{i}\frac{w}{\sqrt{u-Aw^{2}}\sqrt{v-w^{2}}},$

that is,

$\displaystyle\bar{f}_{n-2}$
$\displaystyle=\sum_{i=0}^{m}a_{i}^{m}2C^{2}(m-i)(m-i-1)u^{m-i-2}v^{i}{w^{2}}$
$\displaystyle+\sum_{i=0}^{m}a_{i}^{m}2Raiu^{m-i}v^{i-1}\frac{1}{\sqrt{A}}\arctan\left(\frac{\sqrt{A}w}{\sqrt{u-Aw^{2}}}\right)$
$\displaystyle-\sum_{i=0}^{m}a_{i}^{m}2C(m-i)u^{m-i-1}v^{i}\frac{\ln|\sqrt{u-Aw^{2}}+\sqrt{A(v-w^{2})}|}{\sqrt{A}}.$

We must have

$\begin{cases}a_{i}^{m}2iRa=0,\\\
a_{i}^{m}2C(m-i)=0,~{}~{}~{}i=0,1,\cdots,m,\end{cases}$

which yields $a_{i}^{m}=0$ for $i=0,1,\cdots,m.$ This is a contradiction to
$f_{n}\neq 0$.

_Case II_. $\sigma\neq 1$, $a_{i}^{m}=0$, $i=0,1,\cdots,m-1$, $k_{0}=-2\sigma
m$. In this case, we have

$\displaystyle
f_{n}=f_{2m}=a_{m}^{m}(y^{2}+z^{2})^{m},~{}~{}f_{n-1}=f_{2m-1}=0.$

Working in a similar way to the previous case, we get

$\displaystyle-\sqrt{u-Aw^{2}}\sqrt{v-w^{2}}\frac{d\bar{f}_{n-2}}{dw}=-2a_{m}^{m}mRav^{m-1}\sqrt{v-w^{2}},$

and

$\displaystyle\bar{f}_{n-2}=2a_{m}^{m}mRav^{m-1}\frac{1}{\sqrt{A}}\arctan\left(\frac{\sqrt{A}w}{\sqrt{u-Aw^{2}}}\right).$

Therefore, $a_{m}^{m}=0$ and $f_{n}=0$, which is a contradiction.

  * Proposition 6.

System (1) has no exponential factors.

Proof: Let $E=\exp(g/h)$ be an exponential factor of system (1) with a
cofactor

$L=l_{0}+l_{1}x+l_{2}y+l_{3}z,~{}~{}l_{i}\in\mathbb{R},~{}~{}i=0,1,2,3,$

where $g,h\in\mathbb{R}[x,y,z]$ with $f,g$ being prime. From Proposition 2.,
3. and 3., $E=\exp(g)$ with $g=g(x,y,z)\in\mathbb{R}[x,y,z]/{\mathbb{R}}$. By
definitions, $g$ satisfies

$\displaystyle(Ayz+Cz-\sigma x)\frac{\partial g}{\partial
x}+(-xz+Ra-y)\frac{\partial g}{\partial y}$
$\displaystyle+(-z+xy)\frac{\partial g}{\partial
z}=l_{0}+l_{1}x+l_{2}y+l_{3}z.$ (21)

If $g$ is a polynomial of degree $n\geq 3$. We write $g$ as
$g=\sum_{j=0}^{n}g_{j}(x,y,z)$, where each $g_{j}$ is a homogeneous polynomial
of degree $j$ and $g_{n}\neq 0$. Observing the right hand side of (21) has
degree at most one, we compute the terms of degree $n+1$ in (21) and get

$Ayz\frac{\partial g_{n}}{\partial x}-xz\frac{\partial g_{n}}{\partial
y}+xy\frac{\partial g_{n}}{\partial z}=0,$

which is (5) replacing $f_{n}$ by $g_{n}$. Then the arguments used in the
proof of Proposition 3. imply $g_{n}=L_{n}(y^{2}+z^{2})^{m}$ with
$L_{n}\in\mathbb{R}$. Now computing the terms of degree $n$ in (21) leads to

$\displaystyle Ayz\frac{\partial g_{n-1}}{\partial x}-xz\frac{\partial
g_{n-1}}{\partial y}+xy\frac{\partial g_{n-1}}{\partial z}$
$\displaystyle=(\sigma x-Cz)\frac{\partial g_{n}}{\partial x}+y\frac{\partial
g_{n}}{\partial y}+z\frac{\partial g_{n}}{\partial z},$

which is (9) with $f_{n}$ replaced by $g_{n}$ and $f_{n-1}$ replaced by
$g_{n-1}$. Again, the arguments used in the proof of Proposition 3. imply that
$g_{n}=0$, which is a contradiction.

Hence, $g$ is a polynomial of degree at most two satisfying (21). So we can
write $g$ as

$g=b_{0}+b_{1}x+b_{2}y+b_{3}z+b_{11}x^{2}+b_{22}y^{2}+b_{33}z^{2}+b_{12}xy+b_{13}xz+b_{23}yz.$
(22)

Then substituting (22) into (21) yields a system of algebraic equations
$g_{i}=0$ for $i=1,\cdots,15$, where

$\displaystyle g_{1}=Rab_{2}-l_{0},~{}g_{2}=-\sigma b_{1}+Rab_{12}-l_{1},$
$\displaystyle
g_{3}=2Rab_{22}-b_{2}-l_{2},~{}g_{4}=Cb_{1}+Rab_{23}-b_{3}-l_{3},$
$\displaystyle g_{5}=-\sigma b_{12}-b_{12}+b_{3},~{}g_{6}=-\sigma
b_{13}-b_{2}-b_{13},$ $\displaystyle
g_{7}=Ab_{1}+Cb_{12}-2b_{23},~{}g_{8}=-2\sigma b_{11},$ $\displaystyle
g_{9}=-2b_{22},~{}g_{10}=2Cb_{11}+Cb_{13}-2b_{33},~{}g_{11}=-b_{22}+b_{33},$
$\displaystyle
g_{12}=b_{13},~{}g_{13}=b_{12},~{}g_{14}=b_{23},~{}g_{15}=2Ab_{11}+Ab_{13}.$

From the above equations, we obtain that $l_{0}=l_{1}=l_{2}=l_{3}=0$,
$b_{1}=b_{2}=b_{3}=0$, $b_{11}=b_{22}=b_{33}=b_{12}=b_{13}=b_{23}=0$, which
implies $g$ is a constant. This completes the proof.

_Proof of Theorem 1._: By Proposition 2. and 3.-3., it is obvious.

##  4\. Appendix

In this section, we provide a relationship between the Glukhovsky-Dolzhansky
system (1) and other similar quadric systems including the Rabinovich system,
the 3D forced-damped system and the _D2_ vector field.

The Rabinovich system [28, 29] is a three-wave interaction model and is given
by

$\begin{cases}\dot{X}=hY-v_{1}X+YZ\\\ \dot{Y}=hX-v_{2}Y-XZ\\\
\dot{Z}=-v_{3}Z+XY.\end{cases}$ (23)

where $h,~{}v_{1},~{}v_{2},~{}v_{3}$ are parameters. When $C=-2ARa$, by a
family of linear scaling $\mathcal{S}_{\alpha}:(x,y,z,t)\rightarrow(X,Y,Z,T)$

$x=\frac{X}{\alpha},~{}~{}y=Ra-\frac{Z}{\alpha\sqrt{-A}},~{}~{}z=\frac{Y}{\alpha\sqrt{-A}},~{}~{}t=\alpha
T,$

we transform the Glukhovsky-Dolzhansky system (1) into

$\begin{cases}\frac{dX}{dT}=\alpha\sqrt{-A}RaY-\alpha\sigma X+YZ\\\
\frac{dY}{dT}=\alpha\sqrt{-A}RaX-\alpha Y-XZ\\\ \frac{dZ}{dT}=-\alpha
Z+XY.\end{cases}$ (24)

Clearly, system (24) corresponds to the Rabinovich system (23) with parameters
$(h,v_{1},v_{2},v_{3})=(\alpha\sqrt{-A}Ra,\alpha\sigma,\alpha,\alpha)$. Hence,
if $C=-2ARa$, the Glukhovsky-Dolzhansky system (1) is equivalent to the
Rabinovich system (23) in the particular case of the parameter region
$\\{(h,v_{1}.v_{2}.v_{3})|v_{2}=v_{3}\\}$, that is, two systems are homothetic
copies. In addition, the linear scaling $\mathcal{S}_{\alpha}$ forces
parameter $A$ is less than zero, which cannot hold in the physical region of
parameter $A$.

The 3D forced-damped system [30] arises in mechanical, electrical and fluid
dynamical contexts and is given by

$\begin{cases}\dot{X}=-aX+Y+YZ\\\ \dot{Y}=X-aY+bXZ\\\ \dot{Z}=cZ-
bXY\end{cases}$ (25)

where $a,b,c$ are parameters. Compared with the Glukhovsky-Dolzhansky system
(1), the 3D forced-damped system (25) has less parameters. When $\sigma=1$,
the linear scaling

$x=\frac{(ARa+C)^{3/2}}{A\sqrt{Ra}}X,~{}~{}y=Ra+\frac{(ARa+C)Z}{A},~{}~{}z=\frac{(ARa+C)Y}{A},~{}~{}t=\frac{T}{\sqrt{(ARa+C)Ra}},$

transforms the Glukhovsky-Dolzhansky system (1) into

$\begin{cases}\frac{dX}{dT}=-\frac{1}{\sqrt{(ARa+C)Ra}}X+Y+YZ\\\
\frac{dY}{dT}=X-\frac{1}{\sqrt{(ARa+C)Ra}}Y+\frac{ARa+C}{{ARa}}XZ\\\
\frac{dZ}{dT}=-\frac{1}{\sqrt{(ARa+C)Ra}}Z-\frac{ARa+C}{{ARa}}XY,\end{cases}$
(26)

which corresponds to the 3D forced-damped system (25) with parameters

$(a,b,c)=(\frac{1}{\sqrt{(ARa+C)Ra}},\frac{ARa+C}{{ARa}},-\frac{1}{\sqrt{(ARa+C)Ra}}).$

Therefore, when $\sigma=1$, the Glukhovsky-Dolzhansky system (1) is equivalent
to the 3D forced-damped system (25) in the particular case of the parameter
region $\\{(a,b,c)|a+c=0\\}$.

Consider the vector field equivariant under the _D2_ symmetry group, called
the _D2_ vector field [31]

$\begin{cases}\dot{X}=aX+YZ\\\ \dot{Y}=bY+XZ\\\ \dot{Z}=Z-XY.\end{cases}$ (27)

where $a,b$ are parameters. Similar to above, when $Ra=C=0$, we make a linear
scaling

$x=X,~{}~{}y=\frac{Z}{\sqrt{A}},~{}~{}z=-\frac{Y}{\sqrt{A}},~{}~{}t=-T$

and transform (1) into

$\begin{cases}\frac{dX}{dT}=\sigma X+YZ\\\ \frac{dY}{dT}=Y+XZ\\\
\frac{dZ}{dT}=Z-XY.\end{cases}$ (28)

which corresponds to the _D2_ vector field (27) with parameters
$(a,b)=(\sigma,1)$. Then, when $Ra=C=0$, the Glukhovsky-Dolzhansky system (1)
is equivalent to the _D2_ vector field (27) in the particular case of the
parameter region $\\{(a,b)|b=1\\}$.

## References

  * [1] A.B. Glukhovsky, F.A. Dolzhansky, Three-component geostrophic models of convection in a rotating fluid, _Izv. Akad. Nauk SSSR, Fiz. Atmos. Okeana_ 16(1980) 311-318.
  * [2] G.A. Leonov, N.V. Kuznetsov, T.N. Mokaev, Hidden attractor and homoclinic orbit in Lorenz-like system describing convective fluid motion in rotating cavity, _Commun. Nonlinear Sci. Numer. Simul._ 28(2015) 166-174.
  * [3] G. Chen, N.V. Kuznetsov, G.A. Leonov, N.V. Kuznetsov, Hidden attractors on one path: Glukhovsky-Dolzhansky, Lorenz, and Rabinovich systems, _Internat. J. Bifur. Chaos Appl. Sci. Engrg._ 27(2017) 1750115, 9 pp.
  * [4] G.A. Leonov, T.N. Mokaev, Lyapunov dimension formula for the attractor of the Glukhovsky-Dolzhansky system, _Doklady Mathematics_ 93(2016) 42-45.
  * [5] G.A. Leonov, Existence criterion of homoclinic trajectories in the Glukhovsky-Dolzhansky system, _Phys. Lett. A_ 379(2015) 524-528.
  * [6] X. Zhang, Integrability of dynamical systems: algebra and analysis, Springer-Verlag, 2017.
  * [7] K. Huang, S. Shi and W. Li, Kovalevskaya exponents, weak Painlevé property and integrability for quasi-homogeneous differential systems, _Regul. Chaotic Dyn._ 25(2020) 295-312.
  * [8] J. Llibre, Y. Paulina and C. Valls, On the global dynamics of a three-dimensional forced-damped differential system, _J. Nonlinear Math. Phys._ 27(2020) 414-428.
  * [9] K. Huang, S. Shi and Z. Xu, Integrable deformations, bi-Hamiltonian structures and nonintegrability of a generalized Rikitake system, _Int. J. Geom. Methods Mod. Phys._ 16(2019) 1950059, 17 pp.
  * [10] F. Dumortier, J. Llibre and J.C. Artés, Qualitative Theory of Planar Differential Systems, in: Universitext, Springer-Verlag, Berlin, 2006.
  * [11] J. Llibre, X. Zhang, Darboux theory of integrability in Cn taking into account the multiplicity, _J. Differential Equations_ 246(2009) 541-551.
  * [12] C. Christopher, J. Llibre, Integrability via invariant algebraic curves for planar polynomial differential systems, _Ann. Differential Equations_ 16(2000) 5-19.
  * [13] C. Christopher, J. Llibre, J.V. Pereira, Multiplicity of invariant algebraic curves in polynomial vector fields, _Pacific J. Math._ 229(2007) 63-117.
  * [14] J. Llibre, X. Zhang, On the Darboux Integrability of Polynomial Differential Systems, _Qual. Theory Dyn. Syst._ 11(2012) 129-144.
  * [15] J. Llibre, C. Valls, On the integrability of the 5-dimensional Lorenz system for the gravity-wave activity, _Proc. Amer. Math. Soc._ 145(2017) 665-679.
  * [16] C. Valls, Invariant algebraic surfaces for generalized Raychaudhuri equations. _Comm. Math. Phys._ 308(2011) 133-146.
  * [17] C. Valls, Invariant algebraic surfaces for a virus dynamics, _Z. Angew. Math. Phys._ 66(2015) 1315-1328.
  * [18] C. Valls, Invariant algebraic surfaces and algebraic first integrals of the Maxwell-Bloch system,_J. Geom. Phys._ 146(2019) 103516, 8 pp.
  * [19] M.R. Candido, J. Llibre, C. Valls, Invariant algebraic surfaces and Hopf bifurcation of a finance model, _Internat. J. Bifur. Chaos Appl. Sci. Engrg._ 28(2018) 1850150, 10 pp.
  * [20] C. Valls, On the global dynamics of the Newell-Whitehead system, _J. Nonlinear Math. Phys._ 26(2019) 569-578.
  * [21] M. Ayoul, N.T. Zung, Galoisian obstructions to non-Hamiltonian integrability, _C. R. Math. Acad. Sci. Paris_ 348(2010) 1323-1326.
  * [22] K. Huang, S. Shi and W. Li, Meromorphic non-integrability of several 3D dynamical systems, _Entropy_ 19(2017) 211, 16 pp.
  * [23] W. Szuminski, M. Przybylska, Differential Galois integrability obstructions for nonlinear three-dimensional differential systems, _Chaos_ 30(2020) 013135, 12 pp.
  * [24] K. Huang, S. Shi and W. Li, Integrability analysis of the Shimizu-Morioka system. _Commun. Nonlinear Sci. Numer. Simul._ 84(2020) 105101, 12 pp.
  * [25] K. Huang, S. Shi and W. Li, Meromorphic and formal first integrals for the Lorenz system, _J. Nonlinear Math. Phys._ 25(2018), 106-121.
  * [26] K. Huang, S. Shi and W. Li, First integrals of the Maxwell-Bloch system, _C. R. Math. Acad. Sci. Paris_ 358(2020) 3-11.
  * [27] A.J. Maciejewskia and M. Przybylska, Integrability analysis of the stretch-twist-fold flow, _J. Nonlinear Sci._ 30(2020), 1607-1649.
  * [28] X. Zhang, Integrals of motion of the Rabinovich system, _J. Phys. A_ 33(2000) 5137-5155.
  * [29] F. Xie and X. Zhang, Invariant algebraic surfaces of the Rabinovich system, _J. Phys. A_ 36(2003) 499-516.
  * [30] J. Llibre, R. Oliveira, C. Valls, On the Darboux integrability of a three-dimensional forced-damped differential system, _J. Nonlinear Math. Phys._ 24(2017), 473-494.
  * [31] K. Katsios, S. Anastassiou, Darboux polynomials and global phase portraits for the D2 vector field, _J. Math. Anal. Appl._ 475(2019) 32-40.