# Align on the Fly:
Adapting Chatbot Behavior to Established Norms

Chunpu Xu1,3,8 Steffi Chern4,8 Ethan Chern1,8 Ge Zhang5,6 Zekun Wang6
Ruibo Liu7 Jing Li3 Jie Fu6 Pengfei Liu1,2,811footnotemark: 1
1 Shanghai Jiao Tong University 2 Shanghai Artificial Intelligence Laboratory
3 The Hong Kong Polytechnic University 4 Carnegie Mellon University 5
University of Waterloo
6 The Hong Kong University of Science and Technology 7 Google DeepMind
8 Generative AI Research Lab (GAIR)
*Corresponding author

###### Abstract

In this paper, we aim to align large language models with the ever-changing,
complex, and diverse human values (e.g., social norms) across time and
locations. This presents a challenge to existing alignment techniques, such as
supervised fine-tuning, which internalize values within model parameters. To
overcome this, we propose an On-the-fly Preference Optimization (OPO) method,
which is a real-time alignment that works in a streaming way. It employs an
external memory to store established rules for alignment, which can constrain
LLMs’ behaviors without further training, allowing for convenient updates and
customization of human values. We also introduce a scalable evaluation to
assess the proposed method more effectively. Experimental results on both
human-annotated and auto-generated questions from legal and moral domains
indicate the effectiveness of the proposed OPO method. Our code and data are
released at https://github.com/GAIR-NLP/OPO.

##  1 Introduction

Large language models (LLMs) Raffel et al. (2020); Brown et al. (2020a); Sun
et al. (2021); Scao et al. (2022); Zhang et al. (2022); Sanh et al. (2022)
have shown transformative capabilities in various Natural Language Processing
(NLP) tasks. Despite their strengths, there is an increasing concern regarding
the potential risks and negative social impacts they may pose Gehman et al.
(2020); Weidinger et al. (2021). Recently, many efforts have been made to
align LLMs with human values and intentions Askell et al. (2021); Yang et al.
(2023); Cheng et al. (2023), where supervised fine-tuning (SFT) and
reinforcement learning from human feedback (RLHF) as well as its variants
Glaese et al. (2022); Bai et al. (2022a); Go et al. (2023); Zhu et al. (2023);
Rafailov et al. (2023) are two predominant alignment strategies used in the
latest LLMs OpenAI (2022a, 2023); Anil et al. (2023); Anthropic (2023); Google
(2023); Zhou et al. (2023a); Touvron et al. (2023). However, optimizing LLMs’
parameters for SFT requires a large amount of computational resources Li et
al. (2023b). At the same time, reinforcement learning in RLHF is often
demonstrated to be complex, unstable, and sensitive to hyperparameters;
collecting high-quality human feedback data for RLHF is also tedious and cost-
consuming Bai et al. (2022a); Casper et al. (2023).

More importantly, certain human values to be aligned (e.g., social norms)
often vary with time and place, and can be intricately diverse Watson (2001);
Forbes et al. (2020); Bakker et al. (2022), making it challenging to
_internalize_ them into LLMs’ parameters using existing alignment methods such
as SFT.

Figure 1: Comparisons between internalized alignment and externalized
alignment. The internalized alignment requires annotated data to align LLMs
with human values by training LLMs with SFT or RLHF methods. Our proposed
externalized alignment retrieves certain values from an external text corpus
to constrain LLMs’ behaviors without further training.

Meanwhile, ensuring large language models comply with these values is not an
optional matter; rather, it is an urgent safety alignment issue to be
addressed Gabriel (2020a); Liu et al. (2022). In this paper, we propose to
organize these values into a structured memory positioned externally to models
and demonstrate how to make the behavior of large language models more aligned
with human defined regulations.

Specifically, we introduce an On-the-fly Preference Optimization (OPO)
alignment framework to align LLMs with ever-changing, complicated, and diverse
human values in real time. The differences between our proposed externalized
alignment (i.e., OPO) and conventional internalized alignment (e.g., SFT and
RLHF) are shown in Figure 1. This framework consists of three key components:
(i) rule creation module: it aims to collect and establish the text corpus
containing certain human values; (ii) alignment module: it is utilized to
select the relevant values from the text corpus based on the input query and
channel behavior of LLMs, (iii) evaluation module: it will be responsible for
evaluating the reliability of aligning methods. These three components
highlight three main challenges of achieving on-the-fly alignment for LLMs.

(1) How to construct a broad-coverage, authoritative, value-reflecting rule
module that LLMs can adhere to? To address this question, we are driven by the
fact that law and morality serve as two guiding systems that manage and govern
actions within human society Moka-Mubelo (2016). Essentially, the law serves
as a regulatory mechanism, influencing behavior by imposing consequences for
transgressions of legal statutes, while moral principles operate through a
system of incentives: adverse actions might result in guilt and societal
disapproval, whereas commendable behaviors are often associated with feelings
of ethical righteousness and societal praise (Shavell, 2002). We collect legal
rules and moral rules in human community to control LLMs’ behaviors based on
the assumption that LLMs need to obey the rules designed for humans.

(2) How to effectively enable large models to dynamically choose the values to
align and generate replies under value constraints? To tackle this issue, we
first retrieve relevant values to the query from the collected text corpus,
and then constrain LLMs’ behaviors with the retrieved values by the designed
question-answering prompt. Specifically, given an input question, we obtain a
dense representation by embedding the question and use the Faiss Johnson et
al. (2019) as the retriever to search for similar values, which are further
used to guide the behaviors of LLMs to achieve the goal of dynamically
aligning LLMs with human values without further training.

(3) How to design flexible evaluation algorithms that can encompass a broader
range of aligned values in testing, thereby assessing the generalization
capability of the alignment method? Li et al. (2023a) For this challenge, we
not only reuse existing data from different types of examinations (e.g., law
exam) as our evaluation sets, but also propose a scalable evaluation
framework, which owns two major advantages: increasing the coverage of testing
rules and avoiding the potential benchmark leakage issue Zhou et al. (2023b).
The core idea of such a scalable evaluation module is to generate the test
samples automatically based on different rules to be aligned. Specifically, we
leverage powerful LLMs to generate multi-choice questions based on given rules
and carefully designed prompts. To ensure the quality, we employ well-educated
annotators to review the questions and filter out the questions with problems.
As a result, we obtain 411 valid legal auto-generated multi-choice questions
based on the laws that are not covered in Chinese law exam, 111Note that
Chinese law practice questions focus on only a small part of the laws (e.g.,
Constitution Law and Criminal Law) while lots of laws are not examined,
especially the local laws. and 264 valid professional moral questions based on
the collected professional moral rules.

Experimentally, we construct 957,778 rules, 4,447 human-labeled test samples,
and 675 auto-generated test samples. The samples are split into 5 evaluation
datasets based on the sample source and domain. To assess the effectiveness of
OPO, we conduct a comprehensive evaluation on 15 LLMs, including both closed-
source variants such as GPT-4, as well as open-source models of varying sizes.
The results demonstrate that our method consistently enhances the alignment of
most LLMs across five evaluation datasets.

Below we detail our contributions:

$\bullet$ We propose OPO, an on-the-fly preference optimization alignment
framework, which subtly leverages retrieval augmented generation technique to
address the challenges posed by the ever-changing, complex, and customized
nature of alignment in human values.

$\bullet$ We construct a law corpus containing all Chinese laws and a morality
corpus which is gathered from moral textbooks, professional moral guidelines,
and daily moral scenarios. We also collect human-annotated legal questions and
moral questions for evaluation.

$\bullet$ We develop a general scalable automatic question-generation method
that could easily expand the evaluation scope and amount of questions, and
mitigate the problem of benchmark leakage.

$\bullet$ We conduct extensive experiments on both human-annotated questions
and auto-generated questions. The results demonstrate the superiority of the
proposed OPO method.

##  2 Related Work

###  2.1 AI Alignment

AI alignment Gabriel (2020b), which calibrates LLMs with human values and
preferences, has become a critical prerequisite prior to their deployment in
real-world scenarios. Preliminary versions of LLMs like GPT-3 Brown et al.
(2020b) exhibit impressive zero-shot and few-shot generalization capabilities
on a variety of tasks. However, these models frequently exhibit unintended
behaviors due to the misaligned training objective, which is to predict the
next token instead of following human intentions Rae et al. (2021); Fedus et
al. (2022); Ouyang et al. (2022). This underscores the importance of
alignment. Among various techniques employed for this purpose, supervised
fine-tuning (SFT) Ouyang et al. (2022) stands out as a pioneering method. This
approach involves fine-tuning powerful LLMs using datasets composed of human-
written instructions. Significantly, SFT plays a crucial role in enhancing
LLMs’ capacity to comprehend and comply with human instructions, thereby
forming the foundation for many subsequent alignment strategies. In exactly
the same work, Ouyang et al. (2022) also proposes RLHF via PPO. This method
facilitates learning human intentions through reward signals derived from a
reward model, which is trained using human ranking data. The integration of
SFT with PPO demonstrates broad applicability, equipping the model with a
task-agnostic proficiency that enables it to address a diverse range of tasks
efficiently. In addition, DPO Rafailov et al. (2023) innovatively incorporates
feedback directly into the loss function, offering an alternative, reward-free
pathway for model alignment. Additionally, many efforts have been made to
explore the self-improvement for alignment with synthetic data Bai et al.
(2022b); Wang et al. (2023a); Huang et al. (2022); Sun et al. (2023); Dubois
et al. (2023); Liu et al. (2023). These approaches depend on data annotated by
humans or synthesized data to train expansive models for better alignment.
Different from prior works, we propose an OPO method to retrieve relevant
documents as guiding rules from our specially curated corpus. We present a
comparison between our OPO and existing alignment methods in Table 1. The
proposed OPO directs the behavior of LLMs without the need for additional
training. Additionally, the corpus is designed for easy updates and
customization, accommodating dynamic regulatory changes and regional
variations. It is noteworthy that OPO is training-free while adjusting
established rules, which demonstrates its higher flexibility compared to known
SFT and RLHF frameworks. 222For example, in this paper, we take into account
tons of distinct values. The cost of aligning them using PPO or DPO becomes
prohibitively expensive, making it a non-trivial task. Furthermore, given the
plug-and-play characteristic of OPO, it can be utilized with most of the
existing SFT and RLHF frameworks to further boost channeling LLM’s behaviors
for alignment.

Method |  | Training
---
-free
| Reward
---
-free
| LLM
---
-agnostic
| Value
---
-updating
SFT Wei et al. (2021) | ✗ | ✔ | ✔ | Hard
PPO Ouyang et al. (2022) | ✗ | ✗ | ✗ | Hard
DPO Rafailov et al. (2023) | ✗ | ✔ | ✗ | Hard
OPO (ours) | ✔ | ✔ | ✔ | Easy

Table 1: Compared with SFT, PPO, and DPO, OPO is free from training or the
reward model, and agnostic to any LLMs or easy to update the established
rules.

###  2.2 Retrieval-augmented Generation

LLMs have demonstrated the ability to store factual knowledge within their
parameters Petroni et al. (2019); Roberts et al. (2020), excelling state-of-
the-art results when meticulously fine-tuned for various downstream NLP tasks
Maiorino et al. (2023). Nevertheless, they exhibit limitations in accessing
and accurately manipulating knowledge. The effectiveness of retrieving related
contexts Lewis et al. (2020); Izacard and Grave (2021); Borgeaud et al.
(2022); Wang et al. (2023b); Asai et al. (2023) to the generation has been
demonstrated in a wide range of knowledge-intensive tasks Mialon et al.
(2023). However, most past retrieval settings mainly focused on general
knowledge corpora to solve conventional question-answering tasks. In this
work, we are the first study to use the collected high-quality legal and moral
rules as an external memory to dynamically constrain LLMs’ behaviors and align
LLMs with human values.

Figure 2: The proposed On-the-fly Preference Optimization (OPO) framework.

##  3 On-the-fly Preference Optimization

As illustrated in Figure 2, the proposed framework consists of a rule creation
module, an alignment module, and an evaluation module.

Figure 3: Examples of legal rule, basic moral rule, professional moral rule,
and social moral rule, along with their translations. Note that we don’t
translate the social moral rule because it is already in English.

Rule Data | Source | Size | Avg. Len. | Language
---|---|---|---|---
Legal rules | NDLR and NDGR | 953,747 | 160.7 | Chinese
Basic moral rules | Morality textbooks | 655 | 122.8 | Chinese
Social moral rules | NormBank | 1,000 | 57.1 | English
Professional moral rules | Professional guidelines | 2,376 | 274.1 | Chinese

Table 2: Analysis of the collected legal and moral rules.

Evaluation Dataset | Question Source | Question Type | Size | Language
---|---|---|---|---
H-Law | Human annotation | Law | 1,706 | Chinese
A-Law | Automatic generation | Law | 411 | Chinese
H-Basic-Morality | Human annotation | Morality | 741 | Chinese
A-Professional-Morality | Automatic generation | Morality | 264 | Chinese
H-Social-Morality | Human annotation | Morality | 2,000 | English

Table 3:  Statistical comparison of the collected and generated dataset. We
collect human-written questions for H-Law and H-Basic-Morality. Besides, we
also randomly collect daily scenarios about social norms annotated by humans
from the test set of NormBank, named H-Social-Morality. To conduct a
comprehensive evaluation, we utilize collected legal rules and professional
moral rules to automatically generate questions by GPT-4, forming the A-Law
and A-Professional-Morality datasets. All questions are multi-choice
questions.

###  3.1 Rule Creation Module

As demonstrated in Shavell (2002), both law and morality act as guides for
human behavior. Law refers to the legal rules established and enforced by the
government to guide behavior and resolve conflicts, while morality indicates
the moral rules that are connected to specific norms and social attributes. To
provide LLMs with the rules, we collect raw documents, then we format and
clean the data, resulting in the text corpus containing legal and moral rules.
The examples of the collected legal and moral rules are shown in Figure 3, and
the analysis of the collected rules can be found in Table 2. Here are the
details of rule creation.

##### Legal Rules.

We collect legal rules from the National Database of Laws and Regulations
(NDLR) 333https://flk.npc.gov.cn and National Database of Government
Regulations (NDGR),444https://www.gov.cn/zhengce/xxgk/gjgzk/ssgz.htm which
mostly cover all laws and regulations currently in force in China. The former
contains the Constitution, Laws, Administrative Regulations, Supervisory
Regulations, Judicial Interpretation, and Local Regulations while the latter
consists of the Departmental Rules of the State Council and Local Government
Regulations.

##### Moral Rules.

Unlike legal rules, which are explicitly defined and established by government
legislatures, moral rules are typically rooted in social norms and cultural
values. These moral guidelines are often composed of implicit soft
constraints. Inspired by the fact that children learn about basic moral
theories through the subject of Morality and Law, we utilize middle school
textbooks on this subject, sourced from the Smart Education Platform,
555https://basic.smartedu.cn to extract basic moral rules. Specifically, we
filter out the legal content from these textbooks, focusing solely on
extracting moral rules from the remaining material. The moral rules collected
from textbooks are mainly about personal morality and social norms. Since
professional morality is an important part of morality Turner and Durkheim
(2013), we collect professional moral rules from 57 different company and
association guidelines online. In addition to gathering explicit moral rules
from morality textbooks and guidelines, we also collect moral rules implicitly
contained in human-labeled data. We randomly sample 1,000 examples from the
training set of NormBank Ziems et al. (2023), a knowledge bank of abundant
human-annotated situational norms. Each example consists of a structured
scenario and a corresponding label indicating whether the behavior is
expected, okay, or unexpected. To extract the underlying rule from each
example, we first use ChatGPT to convert the structured scenario into a
neutral sentence, and then ask GPT-4 to explain why the behavior in the
example aligns with the assigned label. The explanation for each example is
collected and regarded as a social moral rule.

###  3.2 Alignment Module

Our alignment module is inspired by retrieval-augmented generation (RAG) Lewis
et al. (2020); Xu et al. (2022); Wang et al. (2023b). We utilize OpenAI’s
text-embedding-ada-002 model OpenAI (2022b), a top-performing text embedding
model, to obtain dense representations for the collected rules and create the
vector database to store the representations. Our retriever is based on Faiss
Johnson et al. (2019) for efficient similarity search. When given a query, we
also get the representation of the query by text-embedding-ada-002 model and
employ Faiss to search the top-$k$ similar rules from the vector database.
Then, the query and the retrieved relevant rules are fed into the LLM to
obtain the response.666The prompt used for acquiring the response is shown in
Figure 4.

###  3.3 Evaluation Module

The pre-training and SFT of LLMs often involve extensive text data that is
typically non-public, raising concerns about benchmark leakage Zhou et al.
(2023b), which refers to the inadvertent use of data from evaluation sets in
training, potentially disturbing the fairness in comparing LLMs’ performance.
Meanwhile, a small portion of Chinese laws (e.g., Constitution and Criminal
Law) are covered in the questions collected from law exams while many laws are
not included, particularly the local laws. It’s difficult to collect human-
annotated questions related to these laws. Besides, there are no evaluation
benchmarks on the professional ethics of LLMs, a critical aspect of morality.
Therefore, to alleviate the benchmark leakage problem and enhance the
comprehensiveness of evaluation, we propose an evaluation module to
automatically generate new legal questions and professional moral questions by
utilizing GPT-4. We first collect a set of questions as the seed questions,
each of which consists of a question stem, four options, a detailed analysis,
the rules applied in solving the question, and the correct answer. These seed
questions777An example of seed questions is shown in Figure 6. serve as
practical demonstrations for in-context learning Brown et al. (2020c).
Additionally, we leverage our collected text corpus to provide relevant
contexts for the generation of new questions.

Specifically, a rule is randomly selected from the text corpus and top-$l$
similar rules are obtained via k-Nearest-Neighbor (kNN) retrieval to enrich
the content. The randomly selected rule is then concatenated with the $l$
similar rules to form the final context for question generation. To improve
the diversity of the generated questions, we randomly select a question from
the seed question set. Then we meticulously design a question generation
prompt that includes 13 detailed requirements. GPT-4 is employed to generate a
question based on the provided seed question and combined context.
Additionally, Chain-of-thought (COT) reasoning Kojima et al. (2022); Wei et
al. (2022) is employed to improve the quality of the generated questions. The
generated question contains the question stem, options, analysis, and correct
answer.888We demonstrate the question generation prompt in Figure 8 and the
generated question example in Figure 7. To ensure the high quality of the
generated questions, we use another GPT-4 to evaluate each component of the
question. This evaluation process employs the same rules that are used to
generate the question. We collect the question only if it satisfies all
criteria in these four areas; otherwise, we will discard it. The quality
evaluation prompt is provided in Figure 9. Note that many efforts are made to
design both the question generation prompt and quality evaluation prompt. We
test a small portion of the data, manually check the results, and iteratively
revise the prompts based on the analysis of bad cases until we achieve
satisfactory results.

##  4 Experiments and Discussions

###  4.1 Experimental Setup

We assess LLMs in a zero-shot setting for all experiments. Few-shot settings
are not employed for fair comparisons, as the maximum context length of some
LLMs might be exceeded due to the long retrieved texts. Similar to Huang et
al. (2023), we report the answer-only results, as extracting answer choices
from zero-shot predictions is difficult when the responses generated by some
LLMs do not follow specific patterns. IndexFlatL2 Johnson et al. (2019) method
is used in Faiss for similarity search. We manually gather 38 legal questions
and 7 moral questions as seed questions for the legal domain and moral domain,
respectively. Note that we retrieve relevant rules from various vector
databases that correspond with the different evaluation datasets. For
instance, the rules are retrieved from basic moral rules when LLMs are
evaluated on H-Basic-Morality dataset. Especially, the law database consists
of regional and national laws. Humans need to obey both national laws and
corresponding regional laws. Therefore, we adopt regular expressions to
extract the location (i.e., province, municipality, or autonomous region)
related to the query. Subsequently, we retrieve similar laws from the sub-
database consisting of national laws, and local laws related to the extracted
location.

###  4.2 Evaluation Data Collection

The evaluation data consists of two parts, human-annotated data and auto-
generated data. We collect 1,706 human-annotated legal questions related to
the National Judicial Examination of China (NJEC) named H-Law, along with 741
basic moral questions in the subject of Morality and Law named H-Basic-
Morality. An illustrative example of a law question is presented in Figure 5.
Additionally, we randomly select 2,000 questions about social norms from the
test set of NormBank dataset Ziems et al. (2023) named H-Social-Morality.

For auto-generated data, we utilize the evaluation module to generate 462
legal questions and 272 professional moral questions based on the randomly
selected 500 legal rules and 300 professional moral rules. 999The evaluator in
the evaluation module automatically filters out 38 legal questions and 28
professional moral questions through quality checking. Furthermore, three
well-educated human annotators are employed to review each question of the
auto-generated datasets to further ensure the quality. Each annotator is
required to carefully review the three components of the question, including
the question stem, options, and answer. The question analysis is not taken
into consideration, as it is optional and not utilized in the assessment of
LLMs. We will remove any question that is deemed unreasonable by at least two
out of three annotators. After undergoing a thorough human review process, we
retain 411 legal questions, labeled as M-Law, along with 264 moral questions,
labeled as M-Professional-Morality. Each question is composed of a question
stem followed by four options, among which only one is the correct
answer.101010Examples of human-annotated and auto-generated legal questions
are in Figure 5 and Figure 7, respectively.

Models | H-Law | A-Law
---|---|---
Base |  OPO |  $\Delta_{1}$ |  Base |  OPO |  $\Delta_{1}$ |  Oracle |  $\Delta_{2}$
Closed-source models |  |  |  |  |  |  |  |
ChatGPT |  41.62 |  48.07 |  +6.45$\uparrow$ |  77.86 |  86.13 |  +8.27$\uparrow$ |  90.02 |  +12.16$\uparrow$
GPT-4 |  57.21 |  62.25 |  +5.04$\uparrow$ |  84.43 |  94.65 |  +10.22$\uparrow$ |  97.81 |  +13.38$\uparrow$
Open-source models |  |  |  |  |  |  |  |
ChatGLM2-6B |  47.71 |  45.96 |  -1.75$\downarrow$ |  79.08 |  85.40 |  +6.32$\uparrow$ |  85.16 |  +6.08$\uparrow$
ChatGLM3-6B |  49.53 |  51.58 |  +2.05$\uparrow$ |  79.32 |  84.18 |  +4.86$\uparrow$ |  86.62 |  +7.30$\uparrow$
InternLM-Chat-6B |  46.13 |  48.30 |  +2.17$\uparrow$ |  75.67 |  84.91 |  +9.24$\uparrow$ |  86.13 |  +10.46$\uparrow$
Llama2-Chat-7B |  31.48 |  28.37 |  -3.11$\downarrow$ |  60.58 |  63.02 |  +2.44$\uparrow$ |  64.48 |  +3.90$\uparrow$
Qwen-Chat-7B |  52.46 |  53.63 |  +1.17$\uparrow$ |  81.75 |  87.35 |  +5.60$\uparrow$ |  91.48 |  +9.73$\uparrow$
XVERSE-Chat-7B |  47.30 |  52.99 |  +5.69$\uparrow$ |  79.56 |  89.78 |  +10.22$\uparrow$ |  90.02 |  +10.46$\uparrow$
DISC-LawLLM-13B |  50.41 |  52.58 |  +2.17$\uparrow$ |  80.54 |  85.40 |  +4.86$\uparrow$ |  88.81 |  +8.27$\uparrow$
Llama2-Chat-13B |  30.30 |  35.99 |  +5.69$\uparrow$ |  68.13 |  77.86 |  +9.73$\uparrow$ |  81.02 |  +12.89$\uparrow$
XVERSE-Chat-13B |  48.01 |  54.57 |  +6.56$\uparrow$ |  77.86 |  88.81 |  +10.95$\uparrow$ |  91.48 |  +13.62$\uparrow$
Qwen-Chat-14B |  62.43 |  66.82 |  +4.39$\uparrow$ |  85.40 |  92.94 |  +7.54$\uparrow$ |  93.19 |  +7.79$\uparrow$
InternLM-Chat-20B |  47.60 |  51.93 |  +4.33$\uparrow$ |  77.62 |  88.32 |  +10.70$\uparrow$ |  90.51 |  +12.89$\uparrow$
LLaMA2-Chat-70B |  35.11 |  39.10 |  +3.99$\uparrow$ |  67.64 |  81.02 |  +13.38$\uparrow$ |  88.56 |  +20.92$\uparrow$
XuanYuan-70B |  65.42 |  70.22 |  +4.80$\uparrow$ |  84.67 |  94.16 |  +9.49$\uparrow$ |  95.38 |  +10.71$\uparrow$

Table 4: The accuracy of evaluated LLMs on H-Law and A-law. “Base” column
indicates the original performance of each LLM. “OPO” column represents the
performance of our proposed OPO method when applied to each LLM. Here we show
the results with a maximum retrieval context length of up to 1000 for fair
comparisons. “Oracle” column stands for the results of using the rules,
utilized to generate the questions, to replace the retrieved rules in OPO. The
$\Delta_{1}$ column represents the absolute improvements of “OPO" column
compared with “Base" column while $\Delta_{2}$ column indicates the absolute
improvements of “Oracle" column relative to “Base" column.

Models | H-Basic-Morality | H-Social-Morality | A-Professional-Morality
---|---|---|---
Base |  OPO |  $\Delta_{1}$ |  Base |  OPO |  $\Delta_{1}$ |  Base |  OPO |  $\Delta_{1}$ |  Oracle |  $\Delta_{2}$
Closed-source models |  |  |  |  |  |  |  |  |  |  |
ChatGPT |  64.37 |  63.43 |  -0.94$\downarrow$ |  52.45 |  53.60 |  +1.15$\uparrow$ |  90.91 |  93.18 |  +2.27$\uparrow$ |  94.32 |  +3.41$\uparrow$
GPT-4 |  84.62 |  86.37 |  +1.75$\uparrow$ |  56.10 |  56.55 |  +0.45$\uparrow$ |  96.59 |  97.73 |  +1.14$\uparrow$ |  99.62 |  +3.03$\uparrow$
Open-source models |  |  |  |  |  |  |  |  |  |  |
ChatGLM2-6B |  83.67 |  83.54 |  -0.13$\downarrow$ |  46.30 |  49.60 |  +3.30$\uparrow$ |  87.50 |  92.42 |  +4.92$\uparrow$ |  92.05 |  +4.55$\uparrow$
ChatGLM3-6B |  83.81 |  82.59 |  -1.22$\downarrow$ |  39.20 |  41.60 |  +2.40$\uparrow$ |  90.15 |  92.42 |  +2.27$\uparrow$ |  92.42 |  +2.27$\uparrow$
InternLM-Chat-6B |  87.85 |  88.26 |  +0.41$\uparrow$ |  47.45 |  48.35 |  +0.90$\uparrow$ |  90.15 |  92.42 |  +2.27$\uparrow$ |  95.45 |  +5.30$\uparrow$
Llama2-Chat-7B |  35.36 |  34.95 |  -0.41$\downarrow$ |  34.35 |  39.15 |  +4.80$\uparrow$ |  71.21 |  75.00 |  +3.79$\uparrow$ |  80.30 |  +9.09$\uparrow$
Qwen-Chat-7B |  90.15 |  90.96 |  +0.81$\uparrow$ |  47.85 |  47.10 |  -0.75$\downarrow$ |  91.29 |  94.70 |  +3.41$\uparrow$ |  96.59 |  +5.30$\uparrow$
XVERSE-Chat-7B |  82.46 |  84.62 |  +2.16$\uparrow$ |  37.05 |  44.75 |  +7.70$\uparrow$ |  92.05 |  95.45 |  +3.40$\uparrow$ |  95.45 |  +3.40$\uparrow$
DISC-LawLLM-13B |  70.72 |  68.96 |  -1.76$\downarrow$ |  49.10 |  49.75 |  +0.65$\uparrow$ |  89.77 |  92.42 |  +2.65$\uparrow$ |  91.67 |  +1.90$\uparrow$
Llama2-Chat-13B |  42.78 |  44.80 |  +2.02$\uparrow$ |  35.50 |  47.20 |  +11.70$\uparrow$ |  76.14 |  83.71 |  +7.57$\uparrow$ |  83.33 |  +7.19$\uparrow$
XVERSE-Chat-13B |  79.49 |  81.51 |  +2.02$\uparrow$ |  45.35 |  51.30 |  +5.95$\uparrow$ |  89.39 |  92.80 |  +3.41$\uparrow$ |  94.32 |  +4.93$\uparrow$
Qwen-Chat-14B |  93.79 |  94.47 |  +0.68$\uparrow$ |  46.70 |  52.55 |  +5.85$\uparrow$ |  95.45 |  98.11 |  +2.66$\uparrow$ |  98.48 |  +3.03$\uparrow$
InternLM-Chat-20B |  87.99 |  86.91 |  -1.08$\downarrow$ |  46.50 |  50.20 |  +3.70$\uparrow$ |  91.29 |  94.32 |  +3.03$\uparrow$ |  95.83 |  +4.54$\uparrow$
LLaMA2-Chat-70B |  48.72 |  53.58 |  +4.86$\uparrow$ |  38.50 |  49.85 |  +11.35$\uparrow$ |  89.39 |  92.05 |  +2.66$\uparrow$ |  94.70 |  +5.31$\uparrow$
XuanYuan-70B |  93.52 |  94.06 |  +0.54$\uparrow$ |  53.35 |  49.80 |  -3.55$\downarrow$ |  96.97 |  98.86 |  +1.89$\uparrow$ |  99.24 |  +2.27$\uparrow$

Table 5:  The accuracy of evaluated LLMs on H-Basic-Morality, H-Social-
Morality and A-Professional-Morality. “Base” column indicates the original
performance of each LLM. “OPO” column represents the performance of our
proposed OPO method when applied to each LLM. Here we show the results with a
maximum retrieval context length of up to 1000 for fair comparisons. “Oracle”
column stands for the results of using the rules, which are utilized to
generate the questions, to replace the retrieved rules in OPO. The
$\Delta_{1}$ column represents the absolute improvements of “OPO" column
compared with “Base" column while $\Delta_{2}$ column indicates the absolute
improvements of “Oracle" column relative to “Base" column.

###  4.3 Rule Processing Details

The data sourced from NDLR is accessible in multiple formats, mainly PDF and
Microsoft Word documents, while the data from NDGR mostly exists in web page
format. We convert PDF files into Word documents, extract legal statutes, and
systematically organize them by their respective law numbers. For web pages,
we directly obtain the laws from their raw text content.

The morality textbooks are cluttered with extraneous information, requiring
careful data-cleansing efforts. Initially, we remove chapters related to law,
as legal materials are covered by the NDLR and NDGR databases. This allows us
to concentrate on collecting basic moral rules from textbooks. Furthermore, we
refine the material by retaining only the principal texts and discarding
supplementary contents such as figures and discussions. Lastly, we treat each
paragraph in the main texts as an individual moral rule, categorizing and
numbering them by chapter. Similarly, we process and number the professional
moral rules from the collected professional moral guidelines.

###  4.4 Evaluation Models

We conduct evaluation experiments across five datasets using multiple LLMs
that support both Chinese and English. The LLMs under evaluation include
ChatGPT Ouyang et al. (2022), GPT-4 OpenAI (2023), ChatGLM3 and with its
earlier version ChatGLM2 Du et al. (2022), InternLM Team (2023) , Qwen Qwe
(2023), XVERSE XVERSE-AI (2023), Llama2 Touvron et al. (2023), and XuanYuan
Zhang and Yang (2023).111111Details about these LLMs are presented in Table 7.

###  4.5 Main results

The results of LLMs on H-Law and A-Law are shown in Table 4. Our OPO method
can improve the performance of most models on the two datasets, which reveals
the effectiveness of the proposed method. Particularly, a considerable number
of LLMs improve the accuracy by 10 percent on A-Law dataset. For instance, the
accuracy of GPT-4 increases significantly, from 84.43 to 94.65 on the A-Law.
The performance of Qwen-Chat-14B and XuanYuan-70B is competitive with or
superior to that of GPT-4. The possible explanation is that Qwen-Chat-14B and
XuanYuan-70B are trained on a massive Chinese corpus, in contrast to GPT-4,
which incorporates a relatively minor proportion of Chinese data in its
training dataset. Besides, GPT-4, Qwen-Chat-14B, and XuanYuan-70B, the three
strongest LLMs obtain consistent enhancement with OPO, and a few LLMs’
performance drop slightly. The possible reason is that the stronger the model,
the better its ability to utilize information and effectively utilize
retrieved content. Furthermore, most LLMs perform better when we replace the
retrieved rules with oracle rules in the OPO method, as shown by comparing the
“Oracle” and “OPO” columns. These oracle rules represent the rules utilized to
generate the questions.

Table 5 demonstrates the results on H-Basic-Morality, H-Social-Morality, and
A-Professional-Morality. These results are consistent with those on H-Law and
A-Law. Enhanced with our OPO method, all LLMs achieve a significant boost in
performance on A-Professional-Morality. An improved performance is also noted
in most LLMs on H-Basic-Morality and H-Social-Morality.

Model | Base | 200 | 500 | 1000 | 1500 | 2000
---|---|---|---|---|---|---
ChatGLM3 | 79.32 | 82.73 | 83.94 | 84.18 | 82.48 | 82.73
XuanYuan | 84.67 | 86.62 | 92.94 | 94.16 | 94.16 | 93.43

Table 6:  The effects of retrieval length for ChatGLM3 and XuanYuan-70B on
A-Law dataset.

###  4.6 The effects of retrieval length

To provide a quantitative analysis of the impact of varying retrieval length,
we conduct the experiments of LLMs utilizing context lengths of 200, 500,
1000, 1500, and 2000. The results of ChatGLM3-6b and XuanYuan-70B on A-Law are
shown in Table 6. It could be observed that the performance first increases
and then decreases with longer context length. The reason might be that
contexts are sorted by the cosine similarity. The similarity of the context
increases as it gets closer to the beginning. Shorter contexts provide useful
content for LLMs while longer contexts could potentially include more noisy or
irrelevant text.121212The full results of all LLMs on the five datasets,
demonstrating similar observation, are shown across Figure 10$\sim$14.

###  4.7 Discussions

####  4.7.1 Comparison between legal dataset and moral dataset

Focusing on the results of “Base” column, we note that LLMs tend to achieve
better results on Chinese moral datasets (i.e., H-Basic-Morality and
A-Professional-Morality) than Chinese legal datasets (i.e., H-Law and A-Law).
We analyze some legal and moral questions and observe that answering legal
questions requires more complex reasoning. Additionally, all LLMs, including
GPT-4, perform poorly on the H-Social-Morality dataset. This difficulty arises
from the complexity of the H-Social-Morality, which may challenge LLMs in
accurately distinguishing people’s behaviors in specific daily scenarios. Such
scenarios require consideration of various factors, including location,
individual attributes, roles, and behaviors. However, most LLMs can achieve
improvements on H-Social-Morality by leveraging the OPO method.

####  4.7.2 Comparison between human-annotated dataset and auto-generated
dataset

LLMs consistently exhibit superior performance on auto-generated legal
questions compared to those annotated by humans, as evidenced by the
comparative results in A-Law versus H-Law. We guess that questions annotated
by humans possess greater complexity and difficulty compared to those
generated by GPT-4. Interestingly, we also note that the results on H-Basic-
Morality and A-Professional-Morality for most LLMs are comparative. This
suggests that GPT-4 may possess the potential to replace human efforts in
generating questions within domains where reasoning is not complex.

##  5 Conclusion

We propose an OPO method to dynamically align LLMs with constantly changing,
intricate, and various human values. Different from previous alignment
methods, we employ collected rules as external memory to constrain LLMs’
behaviors without further training. To expand the scope of testing rules and
prevent potential benchmark leakage, we introduce a scalable evaluation module
to automatically generate questions based on the rules. Extensive experiments
on both human-annotated questions and auto-generated questions demonstrate the
effectiveness of the proposed method.

## Limitations

First, inference efficiency is one of the main limitations of this work. Our
method requires retrieving relevant rules from the text corpus for every input
query prior to alignment, thus demanding additional computational resources
and time. Second, better retrieval models could be developed and utilized in
the OPO method. Comparing the results of “OPO” column and “Oracle” column, we
find that the performance of most LLMs consistently yields better results when
aligned with the oracle texts. This indicates the potential for enhancement in
the retrieval models. Besides, it is worth noting that while our on-the-fly
method may not benefit all LLMs, it is evident that strong LLMs can easily
demonstrate the effectiveness of our approach. Furthermore, the legal rules we
have collected are limited to Chinese laws, and moral rules represent only a
small portion of the moral standards in society. Our aim in this work is to
demonstrate the potential for using these collected rules to guide the
behaviors of LLMs. However, it is necessary to gather additional rules to
serve as external memory for a wide range of diverse and regional scenarios.

## References

  * Qwe (2023) 2023\.  Qwen technical report.  _CoRR_ , abs/2309.16609.
  * Anil et al. (2023) Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernández Ábrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vladimir Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, and et al. 2023.  Palm 2 technical report.  _CoRR_ , abs/2305.10403.
  * Anthropic (2023) Anthropic. 2023.  Model card and evaluations for claude models.
  * Asai et al. (2023) Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023.  Self-rag: Learning to retrieve, generate, and critique through self-reflection.  _CoRR_ , abs/2310.11511.
  * Askell et al. (2021) Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021.  A general language assistant as a laboratory for alignment.  _arXiv preprint arXiv:2112.00861_.
  * Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. 2022a.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _CoRR_ , abs/2204.05862.
  * Bai et al. (2022b) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemí Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022b.  Constitutional AI: harmlessness from AI feedback.  _CoRR_ , abs/2212.08073.
  * Bakker et al. (2022) Michiel A. Bakker, Martin J. Chadwick, Hannah Sheahan, Michael Henry Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt M. Botvinick, and Christopher Summerfield. 2022.  Fine-tuning language models to find agreement among humans with diverse preferences.  In _NeurIPS_.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. 2022\.  Improving language models by retrieving from trillions of tokens.  In _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 2206–2240. PMLR.
  * Brown et al. (2020a) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020a.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Brown et al. (2020b) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020b.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Brown et al. (2020c) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020c.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Casper et al. (2023) Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Ségerie, Micah Carroll, Andi Peng, Phillip J. K. Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca D. Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. 2023.  Open problems and fundamental limitations of reinforcement learning from human feedback.  _CoRR_ , abs/2307.15217.
  * Cheng et al. (2023) Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. 2023.  Black-box prompt optimization: Aligning large language models without model training.  _CoRR_ , abs/2311.04155.
  * Du et al. (2022) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022.  Glm: General language model pretraining with autoregressive blank infilling.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 320–335.
  * Dubois et al. (2023) Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Alpacafarm: A simulation framework for methods that learn from human feedback.  _CoRR_ , abs/2305.14387.
  * Fedus et al. (2022) William Fedus, Barret Zoph, and Noam Shazeer. 2022.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  _J. Mach. Learn. Res._ , 23:120:1–120:39.
  * Forbes et al. (2020) Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020\.  Social chemistry 101: Learning to reason about social and moral norms.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 653–670. Association for Computational Linguistics.
  * Gabriel (2020a) Iason Gabriel. 2020a.  Artificial intelligence, values, and alignment.  _Minds Mach._ , 30(3):411–437.
  * Gabriel (2020b) Iason Gabriel. 2020b.  Artificial intelligence, values, and alignment.  _Minds Mach._ , 30(3):411–437.
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020\.  Realtoxicityprompts: Evaluating neural toxic degeneration in language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020_ , volume EMNLP 2020 of _Findings of ACL_ , pages 3356–3369. Association for Computational Linguistics.
  * Glaese et al. (2022) Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin J. Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Sona Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. 2022.  Improving alignment of dialogue agents via targeted human judgements.  _CoRR_ , abs/2209.14375.
  * Go et al. (2023) Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman. 2023.  Aligning language models with preferences through f-divergence minimization.  In _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 11546–11583. PMLR.
  * Google (2023) Google. 2023.  Bard.
  * Huang et al. (2022) Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022.  Large language models can self-improve.  _CoRR_ , abs/2210.11610.
  * Huang et al. (2023) Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023.  C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.  _CoRR_ , abs/2305.08322.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021_ , pages 874–880. Association for Computational Linguistics.
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019.  Billion-scale similarity search with GPUs.  _IEEE Transactions on Big Data_ , 7(3):535–547.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  In _NeurIPS_.
  * Lewis et al. (2020) Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-augmented generation for knowledge-intensive NLP tasks.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Li et al. (2023a) Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2023a.  Generative judge for evaluating alignment.  _arXiv preprint arXiv:2310.05470_.
  * Li et al. (2023b) Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. 2023b.  RAIN: your language models can align themselves without finetuning.  _CoRR_ , abs/2309.07124.
  * Liu et al. (2023) Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M. Dai, Diyi Yang, and Soroush Vosoughi. 2023.  Training socially aligned language models in simulated human society.  _CoRR_ , abs/2305.16960.
  * Liu et al. (2022) Ruibo Liu, Ge Zhang, Xinyu Feng, and Soroush Vosoughi. 2022.  Aligning generative language models with human values.  In _Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022_ , pages 241–252. Association for Computational Linguistics.
  * Maiorino et al. (2023) Antonio Maiorino, Zoe Padgett, Chun Wang, Misha Yakubovskiy, and Peng Jiang. 2023\.  Application and evaluation of large language models for the generation of survey questions.  In _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21-25, 2023_ , pages 5244–5245. ACM.
  * Mialon et al. (2023) Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. 2023.  Augmented language models: a survey.  _CoRR_ , abs/2302.07842.
  * Moka-Mubelo (2016) Willy Moka-Mubelo. 2016.  Reconciling law and morality in human rights discourse.  _Philosophy and Politics-Critical Explorations_ , 3.
  * OpenAI (2022a) OpenAI. 2022a.  Blog: Introducing chatgpt.
  * OpenAI (2022b) OpenAI. 2022b.  New and improved embedding model.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _NeurIPS_.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 2463–2473. Association for Computational Linguistics.
  * Rae et al. (2021) Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew J. Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2021.  Scaling language models: Methods, analysis & insights from training gopher.  _CoRR_ , abs/2112.11446.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023.  Direct preference optimization: Your language model is secretly a reward model.  _CoRR_ , abs/2305.18290.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 5418–5426. Association for Computational Linguistics.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022.  Multitask prompted training enables zero-shot task generalization.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. 2022.  BLOOM: A 176b-parameter open-access multilingual language model.  _CoRR_ , abs/2211.05100.
  * Shavell (2002) Steven Shavell. 2002.  Law versus morality as regulators of conduct.  _American law and economics review_ , 4(2):227–257.
  * Sun et al. (2021) Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, Weixin Liu, Zhihua Wu, Weibao Gong, Jianzhong Liang, Zhizhou Shang, Peng Sun, Wei Liu, Xuan Ouyang, Dianhai Yu, Hao Tian, Hua Wu, and Haifeng Wang. 2021.  ERNIE 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation.  _CoRR_ , abs/2107.02137.
  * Sun et al. (2023) Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David D. Cox, Yiming Yang, and Chuang Gan. 2023.  Principle-driven self-alignment of language models from scratch with minimal human supervision.  _CoRR_ , abs/2305.03047.
  * Team (2023) InternLM Team. 2023.  Internlm: A multilingual language model with progressively enhanced capabilities.  https://github.com/InternLM/InternLM.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023\.  Llama 2: Open foundation and fine-tuned chat models.  _CoRR_ , abs/2307.09288.
  * Turner and Durkheim (2013) Bryan S Turner and Emile Durkheim. 2013.  _Professional ethics and civic morals_.  Routledge.
  * Wang et al. (2023a) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023a.  Self-instruct: Aligning language models with self-generated instructions.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_ , pages 13484–13508. Association for Computational Linguistics.
  * Wang et al. (2023b) Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md. Rizwan Parvez, and Graham Neubig. 2023b.  Learning to filter context for retrieval-augmented generation.  _CoRR_ , abs/2311.08377.
  * Watson (2001) Alan Watson. 2001.  _Society And Legal Change 2Nd Ed_.  Temple University Press.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  In _NeurIPS_.
  * Weidinger et al. (2021) Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2021.  Ethical and social risks of harm from language models.  _CoRR_ , abs/2112.04359.
  * Xu et al. (2022) Jing Xu, Arthur Szlam, and Jason Weston. 2022.  Beyond goldfish memory: Long-term open-domain conversation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 5180–5197. Association for Computational Linguistics.
  * XVERSE-AI (2023) XVERSE-AI. 2023.  Xverse.  https://github.com/xverse-ai/XVERSE-13B.
  * Yang et al. (2023) Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. 2023.  Alignment for honesty.  _arXiv preprint arXiv:2312.07000_.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  OPT: open pre-trained transformer language models.  _CoRR_ , abs/2205.01068.
  * Zhang and Yang (2023) Xuanyu Zhang and Qing Yang. 2023.  Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters.  In _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21-25, 2023_ , pages 4435–4439. ACM.
  * Zhou et al. (2023a) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023a.  LIMA: less is more for alignment.  _CoRR_ , abs/2305.11206.
  * Zhou et al. (2023b) Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023b.  Don’t make your LLM an evaluation benchmark cheater.  _CoRR_ , abs/2311.01964.
  * Zhu et al. (2023) Banghua Zhu, Michael I. Jordan, and Jiantao Jiao. 2023.  Principled reinforcement learning with human feedback from pairwise or k-wise comparisons.  In _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 43037–43067. PMLR.
  * Ziems et al. (2023) Caleb Ziems, Jane Dwivedi-Yu, Yi-Chia Wang, Alon Y. Halevy, and Diyi Yang. 2023\.  Normbank: A knowledge bank of situational social norms.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_ , pages 7756–7776. Association for Computational Linguistics.

##  Appendix A Appendix

###  A.1 Details of evaluation models

Table 7 shows the details of the 15 LLMs for evaluation. We use version 0613
for both ChatGPT and GPT-4.

Model | Model Size | Access | Creator
---|---|---|---
ChatGPT | - | api | OpenAI
GPT-4 | - | api | OpenAI
ChatGLM2-6B | 6B | weights | Tsinghua & Zhipu
ChatGLM3-6B | 6B | weights | Tsinghua & Zhipu
InternLM-Chat-6B | 6B | weights | Shanghai AI Lab
InternLM-Chat-20B | 20B | weights | Shanghai AI Lab
DISC-LawLLM-13B | 13B | weights | Fudan
Llama2-Chat-7B | 7B | weights | Meta
Llama2-Chat-13B | 13B | weights | Meta
Llama2-Chat-70B | 70B | weights | Meta
Qwen-Chat-7B | 7B | weights | Alibaba Cloud
Qwen-Chat-14B | 14B | weights | Alibaba Cloud
XVERSE-Chat-7B | 7B | weights | XVERSE
XVERSE-Chat-13B | 13B | weights | XVERSE
XuanYuan-70B | 70B | weights | Duxiaoman

Table 7: The evaluated LLMs in this paper. “-” represents the model size is
undisclosed.

###  A.2 The prompt for obtaining responses

The prompt used to obtain responses based on the rules is demonstrated in
Figure 4.

Figure 4: The prompt used to obtain the response based on the retrieved rules.
“rules” indicates the retrieved similar rules while “question” represents the
input query.

###  A.3 Question example

We show the human-annotated legal question example in Figure 5, seed question
example in Figure 6, and auto-generated legal question example in Figure 7,
respectively.

Figure 5: An example of the human-annotated question. Figure 6: An example of
the seed question. Figure 7: An example of generated questions.

###  A.4 Question prompt

The prompts used in the evaluation module for automatical question generation
and question quality evaluation are shown in Figure 8 and Figure 9,
respectively.

Figure 8: The specially designed prompt for generating question stems,
options, explanations, and answers. The “{mode_type}" could be set to "law"
for generating legal questions and to "morality" for moral questions.
“{example}" indicates a seed question including principles, a question stem,
options, an explanation, and an answer. “{input_principle}" represents the
principle context provided to LLMs. "{element}" depends on the attribute of
the “{input_principle}", and is set to the special location (e.g., Shanghai
city) when generating legal questions and to specifical company or association
(e.g, L’Oreal). "{action}" is determined by the input seed question and
denotes the categorization of question formulation by LLMs, which consists of
two types, crafting a scenario and focusing on a concept. "{action_type}"
depends on "{action}" and is set to "concept" or "scenario". Figure 9: The
prompt used to evaluate the quality of generated questions.

###  A.5 The full experimental results of varying retrieved context length

The full experimental results related to varying retrieved context length on
the five evaluation datasets are shown across Figure 10$\sim$14. We could
observe that the performance of most LLMs initially increases, and then
decreases with an extension in the context length.

Figure 10: The experimental results of varying context length on H-Law
dataset. Figure 11: The experimental results of varying context length on
A-Law dataset. Figure 12: The experimental results of varying context length
on H-Basic-Morality dataset. Figure 13: The experimental results of varying
context length on H-Social-Morality dataset. Figure 14: The experimental
results of varying context length on A-Professional-morality dataset.