# Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation

Zhouhong Gu1, Xiaoxuan Zhu1 *, Haoning Ye1, Lin Zhang1, Jianchen Wang1, Yixin
Zhu1,
Sihang Jiang1, Zhuozhi Xiong1, Zihan Li1, Weijie Wu1, Qianyu He1, Rui Xu1,
Wenhao Huang1, Jingping Liu2, Zili Wang, Shusen Wang, Weiguo Zheng3, Hongwei
Feng1, Yanghua Xiao1,4 $\dagger$ Equal ContributionCorresponding
AuthorsYanghua Xiao is also a member of Research Group of Computational and AI
Communication at Institute for Global Communications and Integrated Media,
Fudan University.

###### Abstract

New Natural Langauge Process (NLP) benchmarks are urgently needed to align
with the rapid development of large language models (LLMs). We present Xiezhi,
the most comprehensive evaluation suite designed to assess holistic domain
knowledge. Xiezhi comprises multiple-choice questions across 516 diverse
disciplines ranging from 13 different subjects with 249,587 questions and
accompanied by Xiezhi-Specialty with 14,041 questions and Xiezhi-
Interdiscipline with 10,746 questions. We conduct evaluation of the 47
cutting-edge LLMs on Xiezhi. Results indicate that LLMs exceed average
performance of humans in science, engineering, agronomy, medicine, and art,
but fall short in economics, jurisprudence, pedagogy, literature, history, and
management. All the evaluation code and data are open sourced in
https://github.com/MikeGu721/XiezhiBenchmark

## Introduction

Domain knowledge encompasses an in-depth comprehension of the world,
necessitating the cultivation of various cognitive skills, such as
memorization, abstraction, logical thinking, reasoning, and imagination. Human
has exhibited unparalleled proficiency in domain knowledge, far exceeding any
machine learning models in a long time. Nevertheless, recent advancements in
Large Language Models (LLMs), including Bloom (Scao et al. 2022), Llama
(Touvron et al. 2023), ChatGLM (Du et al. 2022), GPT4 (OpenAI 2023b; Bubeck et
al. 2023) and so many other models, have shown remarkable capabilities in
domain text understanding (Wei et al. 2022). It is time to propose more
comprehensive and more prospective evaluations than before to explore whether
LLMs have actually acquired knowledge, or just acquired a better imitation
ability (Srivastava et al. 2022).

Constructing benchmarks is crucial for automatic evaluation as benchmarks
facilitate efficient, systematic, and scalable comparisons among models.
However, as LLMs continue to grow in size and complexity, they exhibit
outstanding performance across a wide range of domain-specific tasks. This
makes even the newly released benchmarks like MMLU (Hendrycks et al. 2021),
BIG-bench (Srivastava et al. 2022) or HELM (Liang et al. 2022) all lag behind
the capabilities of the LLMs quickily (Suzgun et al. 2022).

Considering LLMs’ performance, we conclude that the benchmark used to evaluate
LLMs should meet the following needs: (1) Needs to cover more tasks
(Srivastava et al. 2022): Cutting-edge LLMs have integrated multiple
capabilities into unified Text-to-Text transformer models (Raffel et al.
2020). Therefore, the evaluation of LLMs should focus on abilities in multiple
tasks. (2) Needs to manifest the disparities among LLMs (Huang et al. 2023):
Considering the emergent capacity of the models (Wei et al. 2022), it is
likely that the SoTA LLMs by learning knowledge in different domains, now have
a certain level of performance in all domains. To accurately evaluate the
distinctions of LLMs with varying capacities, the benchmark should consider
breaking down the evaluation dimensions into more detailed categories. This
will allow for a more precise assessment of each model’s capabilities and
provide valuable insights into their relative strengths and weaknesses. (3)
Needs to go ahead of the training set (Bubeck et al. 2023): As LLMs are
trained on increasingly extensive corpora, newly released benchmarks may
become part of the LLMs’ training data much sooner than before. A prerequisite
for effective evaluation is to ensure that the benchmarks are fresher than the
training data used by LLMs.

Figure 1:  In Chinese mythology, the Xiezhi is a legendary creature known for
its ability to discern right from wrong and uphold justice. Xiezhi Benchmark
encompasses 13 distinct disciplinary categories, 118 sub-disciplines, and 385
further fine-grained disciplines, aiming to provide an extensive domain
taxonomy and benchmark for fair, effective, and comprehensive domain
evaluation. The number adjacent to the first-level discipline signifies the
number of second-level disciplines that are further divided in Chinese
discipline taxonomy.

In light of the aforementioned needs, we propose a comprehensive, multi-
disciplinary, auto-updating benchmark for domain knowledge evaluation. We call
this benchmark Xiezhi, named after a mythical creature that symbolizes
fairness and judgement. Xiezhi consists of 249587 questions with 516
disciplines, ranging from 13 different categories: philosophy, economics, law,
education, literature, history, natural sciences, engineering, agriculture,
medicine, military science, management, and arts. These 516 disciplines are
derived from the Chinese Disciplinary Taxonomy, a comprehensive hierarchical
classification system of domain knowledge proposed by the Chinese Ministry of
Education and widely acknowledged in China. We manually selected and annotated
20,124 questions from the Chinese Graduate Entrance Examination covering these
516 labels to form the Xiezhi-Meta dataset. Xiezhi-Meta is used to train an
annotation model capable of estimating the relevance between questions and
disciplinary labels.  The annotation model subsequently tag disciplinary
labels to 170k multiple-choice questions originating from diverse
examinations, along with 80k multiple-choice questions auto-generated from
academic surveys. To facilitate the usage of Xiezhi and align with the
inclination that “consolidate increasing capabilities into single LLMs”, we
also present Xiezhi-Specialty and Xiezhi-Interdiscipline in both Chinese and
English verision, consisting of 14,041 and 10,746 respectively more balanced,
less sensitive, and less China-centric questions. Xiezhi-Specialty encompasses
questions solvable using knowledge from a single domain, while Xiezhi-
Interdiscipline incorporates questions necessitating knowledge from multiple
domains for resolution.

To give more precise evaluation results, we propose a new evaluation setting
in this paper. We set 50 options for each multiple-choice question, as
previous researchers use only 4 options, resulting in significantly reducing
the accuracy of random guessing and thus better revealing the model’s real
capabilities. We rank all options of each model in generation probability, as
previous researchers use instructions to query the choice made by each model,
to avoid inaccurate evaluations due to model’s inability in answering
multiple-choice questions or errors in the generated content extraction.

To provide a detailed analysis of current development status of LLMs, as well
as to demonstrate the effectiveness of the Xiezhi Benchmark and our proposed
evaluation process, we conduct experiments on 47 famous LLMs across four
benchmarks proposed in different works in our evaluation setting. The
experiments are conducted under in 0-shot, 1-shot, 3-shot demonstration
setting, which is using small number of examples to demonstrate how to solve a
question, with all LLMs being evaluated on both Chinese and English versions
of Xiezhi. This enables us to analyze the LLM results based on their optimal
performance. Results show that the best-performing LLMs, when tested via
multiple-choice questions, have surpassed the level of average practitioners
in science, engineering, agronomy, and medicine in multiple-choice form of .
But humans still greatly outperform all LLMs in domains of economics,
jurisprudence, pedagogy, literature, history, and management. We also examined
the differences in performance of various LLMs across different benchmarks.
Compared to existing knowledge evaluation benchmarks, Xiezhi covers the
broadest range of domains, incorporates the highest quantity of questions, and
consists of the most current data. As shown in our experiments, due to the
vast diversity of knowledge domains covered in Xiezhi and its fifty-to-one
evaluation method, even marginal improvements in any aspect of a model can be
accurately assessed. As such, it is most proficient in discerning the
capability differences among various LMs, spanning from GPT-4 to LLMs with
only 560M parameters. Consequently, it serves as the most appropriate
benchmark for evaluating LLMs of differing competencies.

## Related Works

### Large Language Models

Recently, various companies released their LLMs, such as BARD, ERNIE Bot,
Bloom (Scao et al. 2022), pythia (Biderman et al. 2023), Llama (Touvron et al.
2023), Claude, ChatGPT (OpenAI 2023a), GPT-4 (OpenAI 2023b), and ChatGLM (Du
et al. 2022). Apart from their outstanding performance on trained tasks,
researchers have also discovered that they emerge to have strong performance
on many unseen tasks (Zhou et al. 2023; Chung et al. 2022). Consequently, the
evaluation of LLMs’ capabilities should focus more on a wide range of tasks
over numerous diverse domains and contain samples with different difficulty
levels.

The development of LLMs has spurred the growth of a series of small-scale
conversational LLMs, such as Alpaca (Taori et al. 2023), Vicuna (Chiang et al.
2023), H2Ogpt (H2O.ai 2023), and Moss (Sun et al. 2023a). Most of these small
conversational LLMs are fine-tuned based on existing pre-trained LLMs through
high-quality dialog data generated from LLMs (Ji et al. 2023b; Xu et al. 2023)
by parameter-efficient tuning methods (Hu et al. 2021, 2023). In order to
achieve excellent performance, these models continuously acquire the latest
data from the internet, and their iteration speed is much faster than LLMs.
Any new benchmark will quickly become outdated as it is incorporated into the
model’s training data.

### Benchmarks for Knowledge Evaluation

A number of studies concentrate on assessing a model’s knowledge and reasoning
ability. Certain works, including HellaSwag (Zellers et al. 2019), Physical
IQA (Bisk et al. 2020), and CosmosQA (Huang et al. 2019), focus on evaluating
the understanding of LLMs’ commonsense knowledge. Meanwhile, other research,
such as MMLU (Hendrycks et al. 2021), AGI-Eval (Zhong et al. 2023), MMCU (Zeng
2023), C-Eval (Huang et al. 2023), M3KE (Liu et al. 2023), LexTreme (Niklaus
et al. 2023), Big-Bench (Srivastava et al. 2022) and BIG-Bench-Hard (Suzgun et
al. 2022) target at evaluating the models’ proficiency in domain knowledge.
However, whether these benchmarks provide effective evaluations for all
language models remains debatable. This is because only LLMs with super
abilities show disparities on their datasets, while small LLMs only perform at
a level close to random guessing, leading to different evaluation researches
having different or even contradictory results on small LLMs (Huang et al.
2023; Li et al. 2023). Furthermore, as the training corpora for models become
increasingly larger, these benchmarks might lose their evaluative significance
shortly after they are proposed, due to their incorporation into the training
sets of LLMs.

Moreover, the rise of the generative LLMs presents its own difficulties in
evaluation (Sai, Mohankumar, and Khapra 2022). Beginning with MMLU (Hendrycks
et al. 2021), numerous works have proposed to use of multiple-choice questions
to assess generative models. Recently, a variety of evaluation studies, such
as SuperClue 111https://github.com/CLUEbenchmark/SuperCLUE, employed an
identical prompt to query all LLMs and do extraction to obtain the choice made
by these LLMs. This approach requires models to have strong abilities in
instruction understanding especially in multiple-choice answering, as many
LLMs are unable to meet that needs, leading to unfair evaluation results.

## Xiezhi Benchmark

### Chinese Discipline Taxonomy

Chinese Discipline Taxonomy, developed by the Chinese Ministry of Education,
organizes disciplines of different domains in college education. The taxonomy
divides all domains into different disciplines categories and various levels
of disciplines. The meanings of these levels are as follows:

Discipline Categories: This is the highest level of discipline taxonomy,
divided according to the nature, characteristics of subjects. There are 14
subject categories in Chinese Discipline Taxonomy, including philosophy,
economics, law, education, literature, history, science, engineering,
agriculture, medicine, military science, management, art, and Inter-
discipline.

First-level disciplines: A discipline category is divided into numerous first-
level disciplines, each possessing relatively independent research content.
For example, the “Economics” category is divided into first-level disciplines
“Applied Economics” and “Theoretical Economics”, and “Art Studies” consist of
“Theatre & File Studies”, “Fine Art” and so on.

Second-level disciplines: These disciplines represent more subdivided areas of
study or topics within the first-level discipline. For example, within the
first-level discipline of “Applied Economics”, further divisions include
“Financial Markets”, “Banking”, “Insurance” and many other second-level
disciplines.

As shown in Fig. 1, Xiezhi Benchmark consists of a total of 13 disciplinary
categories, 118 first-level disciplines, and 385 second-level disciplines as
question labels. The detailed information on the disciplines and the question
amount used in Xiezhi Benchmark is listed in Tab. LABEL:table_data in
Appendix.

### Dataset Construction

#### Data collection

Xiezhi consists of 249,587 questions from mainly two different sources. The
first category includes nearly 170k multiple-choice questions collected from
six different examinations in China: elementary school exams, middle school
entrance exams, college entrance exams, undergraduate exams, graduate entrance
exams, and adult education exams. These questions are all open sourced and
many Chinese knowledge evaluation dataset have employed these questions (Huang
et al. 2023; Liu et al. 2023). The second category comprises of nearly 80k
multiple choice questions generated from Chinese open-source academic surveys
or reviews, which is a result come from our auto updating method.

Figure 2:  The figure on the right is the statistics of all questions
collected by Xiezhi. The middle figure shows statistics for Xiezhi-Specialty
and the left shows Xiezhi-Interdiscipline.

#### Auto Updating

Our auto-updating method comprises three primary components: the construction
of Xiezhi-Meta dataset, the generation of questions from open academic
documents, and the automated annotation process.

Xiezhi-Meta

We annotated 20,124 questions collected from the Graduate Entrance Examination
to form the meta version of Xiezhi through both manual efforts and chatGPT.
The aim of annotation is to remove unanswerable questions and to tag each
question with as many disciplines as possible.

We first used ChatGPT to tag each question with first or second-level
disciplines in Chinese. In the process of tagging, we construct a prompt by
concatenating the description of a question with its options, answers, and
exam information with the description of each discipline to increase chatGPT’s
understanding of the question so that the question could be better tagged. The
prompts we used is listed in Appendix Prompt, and the detail of the annotation
process is described in Appendix Mannual Annotation.

Question Generation

Xiezhi comprises nearly 80k multiple-choice questions generated from academic
surveys, as they frequently encompass well-established domain knowledge.  We
select Chinese academic papers across all disciplines that incorporate the
terms “survey” or “review” in their titles. Subsequently, we extract several
longest sentences from these surveys, which typically are the introductory
sentences that contain comprehensive descriptive information pertinent to a
particular field of knowledge.  We identify keywords using the OpenNER method
(Zhu et al. 2019) from these sentences, which are then masked to formulate the
questions. To assemble the set of options for each question, the answers to
all other questions in Xiezhi were sampled and combined with the standard
answers for each respective question.

Auto Annotation

The objectives of auto annotation include the elimination of unanswerable
questions and the assignment of relevant discipline labels to each question.
For unanswerable questions, we extracted keywords from the Xiezhi-Meta, such
as “as shown in the figure below” or “as listed in the table” and so on, and
exclude questions that contain any of these keywords from collected data.  We
use ChatGPT and an annotation model trained by Xiezhi-Meta to do the
discipline labels tagging. The annotation model, which is based on llama-7B,
is used to tag coarse-grained discipline labels (The Discipline Categories in
this paper) to the questions. Based on the tagged coarse-grained labels, we
employ ChatGPT to assign more fine-grained labels (First and Second-level
discipline labels) to the questions, in a similar manner to the labeling of
Xiezhi-Meta. The detail about the training process of the annotation model and
the performance of the auto annotation process is described in Appendix Auto
Annotator.

#### Xiezhi-Specialty & Xiezhi-Interdiscipline

To ensure the validity of the evaluation results, we further propose two
additional datasets, Xiezhi-Specialty and Xiezhi-Interdiscipline in both
Chinese and English version. The trajectory of LLM development tends to
consolidate multiple capabilities within individual LLMs, which may
consequently yield unanticipated interdisciplinary problem-solving
proficiencies. The division of Xiezhi into the Specialty and Interdiscipline
datasets is designed to correspond with this evolving trend. These datasets
are derived from the original Xiezhi Benchmark with the exclusion of some
sensitive questions (e.g., military science) and deeply Chinese-centric
questions (e.g., Literary Chinese QA, ancient Chinese poetry completion).
Based on a balanced sampling strategy, Xiezhi-Specialty is constructed by
selecting questions involved in 3 disciplines or less, while Xiezhi-
Interdiscipline includes questions tagged by 4 disciplines or more. The down-
right of Fig. 3 presents an instance of the Xiezhi-Specialty, while an
instance of the Xiezhi-Interdiscipline is depicted in top-right of Fig. 3.
The process of translation and annotation is delineated in Appendix Manual
Annotation. Furthermore, Appendix Bias, Ethical Problems and Social Impact
comprehensively discusses potential ethical challenges and our effort
undertaken to mitigate them.

Figure 3:  Examples of a 3-shot evaluation with Xiezhi-Interdiscipline, a
question from Xiezhi-Interdiscipline and a question from Xiezhi-Specialty. The
red text in the left figure is the autocompleted response from the model,
while the preceding text is the inputted prompt. English translations are
shown below the corresponding Chinese text for better readability.

## Experiments

### Setup

Models&Device: We conducted experiments on 47 cutting-edge LLMs, the detailed
descriptions of all tested LLMs are listed in Tab 11 in Appendix. Our
experiments cover 45 open-source LLMs based on eight different base models:
bloom, llama, moss, pythia, gpt-neox, stablelm, chatGLM and falcon.
Considering the legal issues, we only show the results of two publicly
recognized API-based LLMs, ChatGPT and GPT-4. Our experiment was carried out
on a DGX Station with 8 80G memory Tesla A100.

More options: All tested LLMs need to choose the best-fit answer from 50
options for each question. Each question is set up with 3 confusing options in
addition to the correct answer, and another 46 options are randomly sampled
from all options in all questions in Xiezhi. We obtain options from questions
that have different discipline categories and select options that do not have
any identical characters (for Chinese) or identical 4-gram characters (for
English) to the ground truth. It is worth noting that it is possible to use
WordNet, open source synonym databases, or other word construction methods to
generate more confusing options. However, our experiments show that the
performance of all LLMs declined dramatically when the number of options
increased, even when using so many non-confusing options. This achieves our
goal of exacerbating the performance gap between LLMs through new experimental
settings and also shows that the traditional 4-choice setting has room for
improvement.

Models | MMLU | CEval | M3KE | Xiezhi-Spec.-Chinese | Xiezhi-Inter.-Chinese | Xiezhi-Spec.-English | Xiezhi-Inter.-English
---|---|---|---|---|---|---|---
| 0-shot | 1-shot | 3-shot | 0-shot | 1-shot | 3-shot | 0-shot | 0-shot | 1-shot | 3-shot | 0-shot | 1-shot | 3-shot | 0-shot | 1-shot | 3-shot | 0-shot | 1-shot | 3-shot
Random-Guess | 0.089 | 0.089 | 0.089 | 0.089 | 0.089 | 0.089 | 0.089 | 0.089 | 0.089 | 0.089 | 0.089 | 0.089 | 0.089 | 0.089 | 0.089 | 0.089 | 0.089 | 0.089 | 0.089
| Generation Probability For Ranking
Bloomz-560m | 0.111 | 0.109 | 0.119 | 0.124 | 0.117 | 0.103 | 0.126 | 0.123 | 0.127 | 0.124 | 0.130 | 0.138 | 0.140 | 0.113 | 0.116 | 0.123 | 0.124 | 0.117 | 0.160
Bloomz-1b1 | 0.131 | 0.116 | 0.128 | 0.107 | 0.115 | 0.110 | 0.082 | 0.138 | 0.108 | 0.107 | 0.117 | 0.125 | 0.123 | 0.130 | 0.119 | 0.114 | 0.144 | 0.129 | 0.145
Bloomz-1b7 | 0.107 | 0.117 | 0.164 | 0.054 | 0.058 | 0.103 | 0.102 | 0.165 | 0.151 | 0.159 | 0.152 | 0.214 | 0.170 | 0.133 | 0.140 | 0.144 | 0.150 | 0.149 | 0.209
Bloomz-3b | 0.139 | 0.084 | 0.146 | 0.168 | 0.182 | 0.194 | 0.063 | 0.186 | 0.154 | 0.168 | 0.151 | 0.180 | 0.182 | 0.201 | 0.155 | 0.156 | 0.175 | 0.164 | 0.158
Bloomz-7b1 | 0.167 | 0.160 | 0.205 | 0.074 | 0.072 | 0.073 | 0.073 | 0.154 | 0.178 | 0.162 | 0.148 | 0.160 | 0.156 | 0.176 | 0.153 | 0.207 | 0.217 | 0.204 | 0.229
Bloomz-7b1-mt | 0.189 | 0.196 | 0.210 | 0.077 | 0.078 | 0.158 | 0.072 | 0.163 | 0.175 | 0.154 | 0.155 | 0.195 | 0.164 | 0.180 | 0.146 | 0.219 | 0.228 | 0.171 | 0.232
Bloomz-7b1-p3 | 0.066 | 0.059 | 0.075 | 0.071 | 0.070 | 0.072 | 0.081 | 0.177 | 0.198 | 0.158 | 0.183 | 0.173 | 0.170 | 0.130 | 0.130 | 0.162 | 0.157 | 0.132 | 0.134
Bloomz | 0.051 | 0.066 | 0.053 | 0.142 | 0.166 | 0.240 | 0.098 | 0.185 | 0.133 | 0.277 | 0.161 | 0.099 | 0.224 | 0.069 | 0.082 | 0.056 | 0.058 | 0.055 | 0.049
Bloomz-mt | 0.266 | 0.264 | 0.248 | 0.204 | 0.164 | 0.151 | 0.161 | 0.253 | 0.198 | 0.212 | 0.213 | 0.189 | 0.184 | 0.379 | 0.396 | 0.394 | 0.383 | 0.405 | 0.398
Bloomz-p3 | 0.115 | 0.093 | 0.057 | 0.118 | 0.137 | 0.140 | 0.115 | 0.136 | 0.095 | 0.105 | 0.086 | 0.065 | 0.098 | 0.139 | 0.097 | 0.069 | 0.176 | 0.141 | 0.070
llama-7b | 0.125 | 0.132 | 0.093 | 0.133 | 0.106 | 0.110 | 0.158 | 0.152 | 0.141 | 0.117 | 0.142 | 0.135 | 0.128 | 0.159 | 0.165 | 0.161 | 0.194 | 0.183 | 0.176
llama-13b | 0.166 | 0.079 | 0.135 | 0.152 | 0.181 | 0.169 | 0.131 | 0.133 | 0.241 | 0.243 | 0.211 | 0.202 | 0.303 | 0.154 | 0.183 | 0.215 | 0.174 | 0.216 | 0.231
llama-30b | 0.076 | 0.107 | 0.073 | 0.079 | 0.119 | 0.082 | 0.079 | 0.140 | 0.206 | 0.162 | 0.186 | 0.202 | 0.183 | 0.110 | 0.195 | 0.161 | 0.088 | 0.158 | 0.219
llama-65b | 0.143 | 0.121 | 0.100 | 0.154 | 0.141 | 0.168 | 0.125 | 0.142 | 0.129 | 0.084 | 0.108 | 0.077 | 0.077 | 0.183 | 0.204 | 0.172 | 0.133 | 0.191 | 0.157
baize-7b (lora) | 0.129 | 0.091 | 0.079 | 0.194 | 0.180 | 0.206 | 0.231 | 0.216 | 0.148 | 0.123 | 0.173 | 0.158 | 0.198 | 0.182 | 0.190 | 0.194 | 0.218 | 0.188 | 0.209
baize-7b-healthcare (lora) | 0.130 | 0.121 | 0.106 | 0.178 | 0.174 | 0.178 | 0.203 | 0.178 | 0.146 | 0.123 | 0.266 | 0.107 | 0.118 | 0.175 | 0.164 | 0.173 | 0.197 | 0.231 | 0.198
baize-13b (lora) | 0.131 | 0.111 | 0.171 | 0.184 | 0.178 | 0.195 | 0.155 | 0.158 | 0.221 | 0.256 | 0.208 | 0.200 | 0.219 | 0.176 | 0.189 | 0.239 | 0.187 | 0.185 | 0.274
baize-30b (lora) | 0.193 | 0.216 | 0.207 | 0.191 | 0.196 | 0.121 | 0.071 | 0.109 | 0.212 | 0.190 | 0.203 | 0.256 | 0.200 | 0.167 | 0.235 | 0.168 | 0.072 | 0.180 | 0.193
Belle-0.2M | 0.127 | 0.148 | 0.243 | 0.053 | 0.063 | 0.136 | 0.076 | 0.172 | 0.126 | 0.153 | 0.171 | 0.165 | 0.147 | 0.206 | 0.146 | 0.148 | 0.217 | 0.150 | 0.173
Belle-0.6M | 0.091 | 0.114 | 0.180 | 0.082 | 0.080 | 0.090 | 0.075 | 0.188 | 0.149 | 0.198 | 0.188 | 0.188 | 0.175 | 0.173 | 0.172 | 0.183 | 0.193 | 0.184 | 0.196
Belle-1M | 0.137 | 0.126 | 0.162 | 0.066 | 0.065 | 0.072 | 0.066 | 0.170 | 0.152 | 0.147 | 0.173 | 0.176 | 0.197 | 0.211 | 0.137 | 0.149 | 0.207 | 0.151 | 0.185
Belle-2M | 0.127 | 0.148 | 0.132 | 0.058 | 0.063 | 0.136 | 0.057 | 0.163 | 0.166 | 0.130 | 0.159 | 0.177 | 0.163 | 0.155 | 0.106 | 0.166 | 0.151 | 0.150 | 0.138
chatglm-6B | 0.099 | 0.109 | 0.112 | 0.084 | 0.074 | 0.114 | 0.115 | 0.082 | 0.097 | 0.147 | 0.104 | 0.111 | 0.144 | 0.106 | 0.120 | 0.124 | 0.099 | 0.079 | 0.097
doctorglm-6b | 0.093 | 0.076 | 0.065 | 0.037 | 0.085 | 0.051 | 0.038 | 0.062 | 0.068 | 0.044 | 0.047 | 0.056 | 0.043 | 0.069 | 0.053 | 0.043 | 0.106 | 0.059 | 0.059
moss-base-16B | 0.072 | 0.050 | 0.062 | 0.115 | 0.048 | 0.052 | 0.099 | 0.105 | 0.051 | 0.059 | 0.123 | 0.054 | 0.058 | 0.124 | 0.077 | 0.080 | 0.121 | 0.058 | 0.063
moss-sft-16B | 0.064 | 0.065 | 0.051 | 0.063 | 0.062 | 0.072 | 0.075 | 0.072 | 0.067 | 0.068 | 0.073 | 0.081 | 0.066 | 0.071 | 0.070 | 0.059 | 0.074 | 0.084 | 0.075
vicuna-7b | 0.051 | 0.051 | 0.029 | 0.063 | 0.071 | 0.064 | 0.059 | 0.169 | 0.171 | 0.165 | 0.134 | 0.201 | 0.213 | 0.182 | 0.209 | 0.195 | 0.200 | 0.214 | 0.182
vicuna-13b | 0.109 | 0.104 | 0.066 | 0.060 | 0.131 | 0.131 | 0.067 | 0.171 | 0.167 | 0.166 | 0.143 | 0.147 | 0.178 | 0.121 | 0.139 | 0.128 | 0.158 | 0.174 | 0.191
alpaca-7b | 0.135 | 0.170 | 0.202 | 0.137 | 0.119 | 0.113 | 0.142 | 0.129 | 0.139 | 0.123 | 0.178 | 0.104 | 0.097 | 0.189 | 0.179 | 0.128 | 0.200 | 0.185 | 0.149
pythia-1.4b | 0.124 | 0.127 | 0.121 | 0.108 | 0.132 | 0.138 | 0.083 | 0.125 | 0.128 | 0.135 | 0.111 | 0.146 | 0.135 | 0.158 | 0.124 | 0.124 | 0.166 | 0.126 | 0.118
pythia-2.8b | 0.103 | 0.110 | 0.066 | 0.064 | 0.089 | 0.122 | 0.086 | 0.114 | 0.120 | 0.131 | 0.091 | 0.113 | 0.112 | 0.126 | 0.118 | 0.112 | 0.110 | 0.145 | 0.107
pythia-6.9b | 0.115 | 0.070 | 0.084 | 0.078 | 0.073 | 0.094 | 0.073 | 0.086 | 0.094 | 0.092 | 0.097 | 0.098 | 0.085 | 0.091 | 0.088 | 0.083 | 0.099 | 0.099 | 0.096
pythia-12b | 0.075 | 0.059 | 0.066 | 0.077 | 0.097 | 0.078 | 0.098 | 0.102 | 0.126 | 0.132 | 0.125 | 0.147 | 0.159 | 0.079 | 0.098 | 0.110 | 0.094 | 0.120 | 0.120
gpt-neox-20b | 0.081 | 0.132 | 0.086 | 0.086 | 0.096 | 0.069 | 0.094 | 0.140 | 0.103 | 0.109 | 0.120 | 0.098 | 0.085 | 0.088 | 0.101 | 0.116 | 0.099 | 0.113 | 0.156
h2ogpt-12b | 0.075 | 0.087 | 0.078 | 0.080 | 0.078 | 0.094 | 0.070 | 0.065 | 0.047 | 0.073 | 0.076 | 0.061 | 0.091 | 0.088 | 0.050 | 0.065 | 0.105 | 0.063 | 0.067
h2ogpt-20b | 0.114 | 0.098 | 0.110 | 0.094 | 0.084 | 0.061 | 0.096 | 0.108 | 0.080 | 0.073 | 0.086 | 0.081 | 0.072 | 0.108 | 0.068 | 0.086 | 0.109 | 0.071 | 0.079
dolly-3b | 0.066 | 0.060 | 0.055 | 0.079 | 0.083 | 0.077 | 0.066 | 0.100 | 0.090 | 0.083 | 0.091 | 0.093 | 0.085 | 0.079 | 0.063 | 0.077 | 0.076 | 0.074 | 0.084
dolly-7b | 0.095 | 0.068 | 0.052 | 0.091 | 0.079 | 0.070 | 0.108 | 0.108 | 0.089 | 0.092 | 0.111 | 0.095 | 0.100 | 0.096 | 0.059 | 0.086 | 0.123 | 0.085 | 0.090
dolly-12b | 0.095 | 0.068 | 0.093 | 0.085 | 0.071 | 0.073 | 0.114 | 0.098 | 0.106 | 0.103 | 0.094 | 0.114 | 0.106 | 0.086 | 0.088 | 0.098 | 0.088 | 0.102 | 0.116
stablelm-3b | 0.070 | 0.085 | 0.071 | 0.086 | 0.082 | 0.099 | 0.096 | 0.101 | 0.087 | 0.091 | 0.083 | 0.092 | 0.067 | 0.069 | 0.089 | 0.081 | 0.066 | 0.085 | 0.088
stablelm-7b | 0.158 | 0.118 | 0.093 | 0.133 | 0.102 | 0.093 | 0.140 | 0.085 | 0.118 | 0.122 | 0.123 | 0.130 | 0.095 | 0.123 | 0.103 | 0.100 | 0.134 | 0.121 | 0.105
falcon-7b | 0.048 | 0.046 | 0.051 | 0.046 | 0.051 | 0.052 | 0.050 | 0.077 | 0.096 | 0.112 | 0.129 | 0.141 | 0.142 | 0.124 | 0.103 | 0.107 | 0.198 | 0.200 | 0.205
falcon-7b-instruct | 0.078 | 0.095 | 0.106 | 0.114 | 0.095 | 0.079 | 0.104 | 0.075 | 0.083 | 0.087 | 0.060 | 0.133 | 0.123 | 0.160 | 0.203 | 0.156 | 0.141 | 0.167 | 0.152
falcon-40b | 0.038 | 0.043 | 0.077 | 0.085 | 0.090 | 0.129 | 0.087 | 0.069 | 0.056 | 0.053 | 0.065 | 0.063 | 0.058 | 0.059 | 0.077 | 0.066 | 0.085 | 0.063 | 0.076
falcon-40b-instruct | 0.126 | 0.123 | 0.121 | 0.070 | 0.080 | 0.068 | 0.141 | 0.103 | 0.085 | 0.079 | 0.115 | 0.082 | 0.081 | 0.118 | 0.143 | 0.124 | 0.083 | 0.108 | 0.104
| Instruction For Ranking
GPT-3.5 | 0.240 | 0.298 | 0.371 | 0.286 | 0.289 | 0.360 | 0.290 | 0.218 | 0.352 | 0.414 | 0.266 | 0.418 | 0.487 | 0.217 | 0.361 | 0.428 | 0.305 | 0.452 | 0.517
GPT-4 | 0.402 | 0.415 | 0.517 | 0.413 | 0.410 | 0.486 | 0.404 | 0.392 | 0.429 | 0.490 | 0.453 | 0.496 | 0.565 | 0.396 | 0.434 | 0.495 | 0.463 | 0.506 | 0.576
| Statistic
Performance-Average | 0.120 | 0.117 | 0.125 | 0.113 | 0.114 | 0.124 | 0.111 | 0.140 | 0.140 | 0.145 | 0.144 | 0.148 | 0.152 | 0.145 | 0.145 | 0.150 | 0.156 | 0.157 | 0.166
Performance-Variance | 0.062 | 0.068 | 0.087 | 0.067 | 0.065 | 0.078 | 0.064 | 0.058 | 0.070 | 0.082 | 0.067 | 0.082 | 0.095 | 0.067 | 0.080 | 0.090 | 0.078 | 0.092 | 0.104

Table 1:  The overall performance of all language models are listed in this
table. All tested models are divided into broad groups sharing similar
character. Bolded font indicates the best performing result within a group,
and underlined font indicates the best performing result for the same data set
with the same settings.

Few-Shot Demonstration: Additionally, we aim to test the LLMs’ understanding
of demonstrations. Therefore, we evaluate the LLMs’ capabilities under 0-shot,
1-shot, and 3-shot settings. Although previous researches use a 5-shot
setting, our experiments have much bigger options number for each question,
taking the maximum input length of each LLM into consideration, we only use at
most 3 examples in our few-shot learning experiments. The examples used for
demonstration were obtained from Xiezhi-Train, a dataset containing 2,555
questions absent from Xiezhi-Speciality and Xiezhi-Interdiscipline, with a
minimum of two labels matching the test questions, an illustration is depicted
in Fig. 3.

Metrics: In this section, we present mainly two experiment results: the
overall performance of all LLMs across various benchmarks, and the ranking of
the top eight 0-shot LLMs in 12 non-sensitive domain categories of the Xiezhi-
Benchmark with the scores for top and average practitioners. For the 45 open-
source models assessed in our evaluation, we calculated the probability of
each model choosing every option using generative probabilities and then
ranked all options accordingly based on the probabilities. Due to legal
considerations, we only display the results of two publicly recognized API-
based LLMs: ChatGPT and GPT-4, and we ask them to rank all given options
through instructions. To represent the results of all ranking outcomes, we
employed the Mean Reciprocal Rank (MRR) as the metric in this section, which
calculates the reciprocal rank of the correct answer. MRR closer to 1
indicates that the model is more capable of placing the correct answer at the
front of the ranking, while it suggests that the LLM tends to place the
correct answer at the bottom if it is closer to 0. As a comparison, we also
employ four different metrics and detailed them in Appendix Results on Other
Metrics.

Randomness: To reduce the effect of randomness on our experiment, we set the
random seed of some python libraries used in our experiment, which are Numpy,
Random, and Torch, to 42. It is worth noting that since we used a generative
probability to rank each option, this generative probability is independent of
the hyperparameters to each LLMs. Nonetheless, in order to be consistent in
our experiments even for details we did not notice, we still set the
deterministic hyperparameters, as described in Appendix Detail Hyper-
parameters. Besides, Given that each question need to sample other 46 options,
we constructed the set of options for each question before we started our
experiment to ensure the consistency in our experiment. Also, we used string
similarity during sampling to select questions that were very unlikely to be
standard answers.

Category | Human | Language Models
---|---|---
Top | Average
Philosophy | 0.856✓ | 0.453✗ | ChatGPT | bloomz-mt | GPT-4 | pythia-1.4b | llama-7b-hf | BELLE-7B-0.2M | BELLE-7B-1M | vicuna-13b-delta-v1.1
0.477 | 0.453 | 0.413 | 0.321 | 0.241 | 0.228 | 0.226 | 0.223
Economics | 0.871✓ | 0.520✓ | GPT-4 | bloomz-mt | llama-65b-hf | BELLE-7B-1M | llama-7b-hf | falcon-7b | baize-lora-7B | falcon-7b-instruct
0.419 | 0.310 | 0.290 | 0.255 | 0.234 | 0.233 | 0.222 | 0.214
Jurisprudence | 0.761✓ | 0.460✓ | GPT-4 | llama-65b-hf | baize-lora-7B | BELLE-7B-0.2M | ChatGPT | llama-7b-hf | BELLE-7B-1M | alpaca-lora-7b
0.368 | 0.323 | 0.230 | 0.217 | 0.213 | 0.210 | 0.199 | 0.192
Pedagogy | 0.854✓ | 0.510✓ | GPT-4 | bloomz-mt | ChatGPT | BELLE-7B-0.2M | baize-lora-13B | pythia-1.4b | llama-65b-hf | BELLE-7B-1M
0.472 | 0.442 | 0.280 | 0.251 | 0.244 | 0.241 | 0.237 | 0.237
Literature | 0.825✓ | 0.560✓ | GPT-4 | bloomz-mt | baize-healthcare-lora-7B | baize-lora-13B | baize-lora-7B | alpaca-lora-7b | BELLE-7B-0.2M | bloomz-3b
0.417 | 0.405 | 0.284 | 0.249 | 0.213 | 0.206 | 0.194 | 0.187
History | 0.854✓ | 0.460✓ | GPT-4 | bloomz-mt | ChatGPT | BELLE-7B-0.2M | BELLE-7B-1M | baize-lora-7B | alpaca-lora-7b | baize-healthcare-lora-7B
0.437 | 0.272 | 0.233 | 0.214 | 0.207 | 0.202 | 0.192 | 0.181
Science | 0.926✓ | 0.394✗ | GPT-4 | bloomz-mt | ChatGPT | BELLE-7B-1M | bloomz-3b | BELLE-7B-0.6M | BELLE-7B-0.2M | vicuna-7b-delta-v1.1
0.436 | 0.408 | 0.220 | 0.210 | 0.200 | 0.197 | 0.191 | 0.188
Engineering | 0.928✓ | 0.380✗ | GPT-4 | ChatGPT | bloomz-mt | bloomz-7b1 | bloomz-7b1-mt | falcon-7b | alpaca-lora-7b | BELLE-7B-1M
0.420 | 0.412 | 0.387 | 0.274 | 0.253 | 0.228 | 0.224 | 0.215
Agronomy | 0.902✓ | 0.333✗ | GPT-4 | bloomz-mt | ChatGPT | bloomz-7b1-mt | BELLE-7B-0.2M | bloomz-7b1 | bloomz-3b | pythia-1.4b
0.515 | 0.366 | 0.311 | 0.224 | 0.216 | 0.215 | 0.200 | 0.193
Medicine | 0.805✓ | 0.430✗ | GPT-4 | baize-healthcare-lora-7B | ChatGPT | doctorglm-6b | BELLE-7B-0.2M | bloomz-7b1 | bloomz-7b1-mt | BELLE-7B-1M
0.469 | 0.279 | 0.265 | 0.253 | 0.223 | 0.222 | 0.219 | 0.210
Management | 0.857✓ | 0.513✓ | GPT-4 | baize-lora-30B | pythia-2.8b | bloomz-p3 | BELLE-7B-0.2M | baize-lora-7B | baize-healthcare-lora-7B | BELLE-7B-1M
0.390 | 0.375 | 0.367 | 0.280 | 0.268 | 0.268 | 0.263 | 0.259
Art Studies | 0.821✓ | 0.400✗ | GPT-4 | baize-healthcare-lora-7B | bloomz-mt | ChatGPT | BELLE-7B-0.2M | baize-lora-13B | alpaca-lora-7b | moss-moon-003-base
0.437 | 0.417 | 0.377 | 0.339 | 0.238 | 0.229 | 0.227 | 0.224
Xiezhi | GPT-4 | bloomz-mt | ChatGPT | BELLE-7B-0.2M | BELLE-7B-1M | bloomz-7b1 | baize-lora-7B | bloomz-7b1-mt | alpaca-lora-7b | vicuna-7b-delta-v1.1
Overall | 0.431 | 0.337 | 0.267 | 0.211 | 0.209 | 0.203 | 0.200 | 0.196 | 0.194 | 0.191
MMLU | GPT-4 | Bloomz-mt | ChatGPT | baize-30b (lora) | Bloomz-7b1-mt | Bloomz-7b1 | llama-13b | stablelm-7b | llama-65b | Bloomz-3b
Overall | 0.402 | 0.266 | 0.240 | 0.193 | 0.189 | 0.167 | 0.166 | 0.158 | 0.143 | 0.139
C-Eval | GPT-4 | ChatGPT | Bloomz-mt | baize-7b (lora) | baize-30b (lora) | baize-13b (lora) | baize-7b-healthcare (lora) | Bloomz-3b | llama-65b | llama-13b
Overall | 0.413 | 0.286 | 0.204 | 0.194 | 0.191 | 0.184 | 0.178 | 0.168 | 0.154 | 0.152
M3KE | GPT-4 | ChatGPT | baize-7b (lora) | baize-7b-healthcare (lora) | Bloomz-mt | llama-7b | baize-13b (lora) | alpaca-7b | falcon-40b-instruct | stablelm-7b
Overall | 0.404 | 0.290 | 0.231 | 0.203 | 0.161 | 0.158 | 0.155 | 0.142 | 0.141 | 0.140

Table 2: Ranking of all LLMs in each category in 0-shot setting. ✓ denotes
human performance exceeds the state-of-the-art LLMs, whereas ✗ signifies LLMs
have surpassed human performance.

Human Performance: Since we mainly collected questions from some of the most
important examinations in China, whose average scores will be released
annually. Furthermore, for various academic entrance examinations, each
institution will publish the average score of their recruit students. We
annotate each question using the average score of the available corresponding
examination and calculated the mean of all the questions within the benchmark
where examination scores can be obtained. Additionally, we used the average
scores publicized by several of China’s top institution as a representation of
a higher level of human performance. While this scoring method has its
limitations, which we thoroughly analyze in Appendix Bias, Ethical Problems
and Social Impact, it still provides usable human baselines for Xiezhi.

### Results of LLMs

The overall performance towards Xiezhi and baselines of all LLMs are listed in
Tab. 1. The ranking of all LLMs in each domain category is listed in Tab. 2.
And here we give the most intriguing observation in the experiments.

Note: (1) The results of GPT-4 and ChatGPT are acquired through instructions,
their real capabilities of them may be higher than the score listed in the
tables. (2) Tab. 2 displays the optimal outcomes, which are combined
performance of Xiezhi-Specialty and Xiezhi-Interdiscipline, in both Chinese
and English Xiezhi. (3) At the moment of writing this paper, M3KE has solely
released its training dataset. So we employed this dataset for conducting the
experiments, which allowed us to execute only 0-shot experimental setups.

Observation 1: Best Performance = Pretraining + Finetuning Examining the
overall results presented in Tab. 2, it is observed that all top-10 open-
source LLMs are built upon either the llama or bloom frameworks. This suggests
that obtaining the most exceptional performance is more likely through these
two base models, due to their substantial potential and superior performance
in domain text comprehension. Moreover, it is noted that all open-source
models within the top-10 overall performance in Tab. 2 are finetuned models,
which implies that only finetuned LLMs can attain the highest performance. As
a result, both effective pretraining and fine-tuning processes are crucial
components in attaining optimal performance in domain text comprehension.

Observation 2: Most LLMs are incapable of performing stably few-shot learning
from demonstrations As shown in the “Performance-Average” in Tab. 1, the
average performance of LLMs reveals that more quantity of examples results in
better model performance. However, it is not an absolute guarantee that each
LLM will exhibit enhanced performance in response to an increased number of
demonstrations. On the contrary, several LLMs exhibit a decline in performance
as the quantity of learning examples expands. In contrast, GPT-4 and ChatGPT
demonstrate a more stable improvement in their performance through few-shot
learning. This can be attributed to the extensive domain knowledge possessed
by GPT-4 and ChatGPT, enabling them to effectively comprehend the features
embedded within the learning samples.

Observation 3: More LLMs’ parameters don’t guarantee better performance
Numerous studies have posited that an increase in the number of model
parameters corresponds to an enhancement in model’s performance. This notion
holds true when comparing LLMs that exhibit an order of magnitude difference
in their parameters. For instance, Bloomz-mt with 146 billion parameters
significantly outperforms Bloomz-560m with 560 million parameters. However,
this argument does not consistently hold. For instance, Bloomz-7b1 surpasses
Bloomz-p3 in the majority of domain tasks, and Pythia-1.4b outperforms other
Pythia models with larger parameter counts across most benchmarks. A possible
explanation for this phenomenon could be that LLMs with different parameter
quantities are optimally suited to different amounts of pre-training and fine-
tuning data (Hoffmann et al. 2022).

Observation 4: Small LMs enhance domain capabilities at the expense of generic
capabilities In our experiments, we examined two medical LLMs: DoctorGLM and
Baize-Healthcare. DoctorGLM originated from ChatGLM-6B, and Baize-Healthcare
was derived from Llama-7B, with both models fine-tuned using medical domain
text. Although both models have lower MRR compared to other LLMs fine-tuned
based on the same base models, they each demonstrate high performance in
medical domain. This suggests the augmentation of LLMs with fewer parameters
in domain text comprehension, whether finetuned through exclusively domain-
specific data or combining domain-specific and generic data, will inevitably
lead to a trade-off in the understanding of generic text. This observation
aligns with the findings from previous research (Fu et al. 2023; Zhao et al.
2023).

### Results of Benchmarks

Based on the observations from Tab. 2, although the objective is to
comprehensively evaluate the domain capabilities of LLMs, the various
benchmarks still exhibit differing results, which indicates the different
emphases of each benchmark. GPT-4, ChatGPT, and Bloomz-mt consistently rank
within the top 10 across all four benchmarks, Baize-7b, and Bloomz-7b1
demonstrate remarkable abilities as they rank within the top 10 across three
of the benchmarks. Furthermore, Xiezhi exhibits the highest variance among all
LLMs in the ”Performance-Variance” of Tab. 1, while the score of GPT-4 doesn’t
always rank first like it was in other benchmark works. This indicates that
the Xiezhi Benchmark excels at discerning the competence disparities among
diverse LLMs and possesses the potential to appraise more potent LLMs.

## Conclusion

We introduced Xiezhi, a new benchmark that measures how well LLMs acquire and
apply domain knowledge. By covering 516 subjects ranging from 13 categories
with 249,587 questions, Xiezhi proposes a taxonomy of all human knowledge and
assesses language understanding of the cutting-edge 47 LLMs in greatest
breadth and depth among all previous benchmarks.  Our research has revealed
that the SOTA LLMs have outperformed practitioner experts in several domains
when evaluated by multiple-choice question answering tasks. Furthermore, there
is still a big gap in generic domain knowledge comprehension between larger
and smaller models. Our experimental findings and the Xiezhi Benchmark we
developed provide researchers with a more comprehensive understanding of their
capabilities across diverse domains.

## Acknowledgement

This work was supported by Science and Technology Commission of Shanghai
Municipality Grant (No. 22511105902). Shanghai Municipal Science and
Technology Major Project (No.2021SHZDZX0103). National Natural Science
Foundation of China (No.62102095). National Natural Science Foundation of
China (No. 62306112). National Natural Science Foundation of China (No.
U23A20496).

## References

  * Aribandi et al. (2021) Aribandi, V.; Tay, Y.; Schuster, T.; Rao, J.; Zheng, H. S.; Mehta, S. V.; Zhuang, H.; Tran, V. Q.; Bahri, D.; Ni, J.; et al. 2021.  Ext5: Towards extreme multi-task scaling for transfer learning.  _arXiv preprint arXiv:2111.10952_.
  * Bai et al. (2022) Bai, Y.; Kadavath, S.; Kundu, S.; Askell, A.; Kernion, J.; Jones, A.; Chen, A.; Goldie, A.; Mirhoseini, A.; McKinnon, C.; et al. 2022.  Constitutional AI: Harmlessness from AI Feedback.  _arXiv preprint arXiv:2212.08073_.
  * Biderman et al. (2023) Biderman, S.; Schoelkopf, H.; Anthony, Q.; Bradley, H.; O’Brien, K.; Hallahan, E.; Khan, M. A.; Purohit, S.; Prashanth, U. S.; Raff, E.; Skowron, A.; Sutawika, L.; and van der Wal, O. 2023.  Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling.  arXiv:2304.01373.
  * Bisk et al. (2020) Bisk, Y.; Zellers, R.; Gao, J.; Choi, Y.; et al. 2020.  Piqa: Reasoning about physical commonsense in natural language.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, 7432–7439.
  * Black et al. (2022) Black, S.; Biderman, S.; Hallahan, E.; Anthony, Q.; Gao, L.; Golding, L.; He, H.; Leahy, C.; McDonell, K.; Phang, J.; Pieler, M.; Prashanth, U. S.; Purohit, S.; Reynolds, L.; Tow, J.; Wang, B.; and Weinbach, S. 2022.  GPT-NeoX-20B: An Open-Source Autoregressive Language Model.
  * Bubeck et al. (2023) Bubeck, S.; Chandrasekaran, V.; Eldan, R.; Gehrke, J.; Horvitz, E.; Kamar, E.; Lee, P.; Lee, Y. T.; Li, Y.; Lundberg, S.; et al. 2023.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_.
  * Chiang et al. (2023) Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; Stoica, I.; and Xing, E. P. 2023.  Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality.
  * Chung et al. (2022) Chung, H. W.; Hou, L.; Longpre, S.; Zoph, B.; Tay, Y.; Fedus, W.; Li, E.; Wang, X.; Dehghani, M.; Brahma, S.; et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Conover et al. (2023) Conover, M.; Hayes, M.; Mathur, A.; Meng, X.; Xie, J.; Wan, J.; Shah, S.; Ghodsi, A.; Wendell, P.; Zaharia, M.; and Xin, R. 2023.  Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM.  https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm.
  * Du et al. (2022) Du, Z.; Qian, Y.; Liu, X.; Ding, M.; Qiu, J.; Yang, Z.; and Tang, J. 2022.  GLM: General Language Model Pretraining with Autoregressive Blank Infilling.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 320–335.
  * Fu, Peng, and Khot (2022) Fu, Y.; Peng, H.; and Khot, T. 2022.  How does gpt obtain its ability? tracing emergent abilities of language models to their sources.  _Yao Fu’s Notion_.
  * Fu et al. (2023) Fu, Y.; Peng, H.; Ou, L.; Sabharwal, A.; and Khot, T. 2023.  Specializing Smaller Language Models towards Multi-Step Reasoning.  _arXiv preprint arXiv:2301.12726_.
  * H2O.ai (2023) H2O.ai. 2023.  h2oGPT - The world’s best open source GPT.  https://github.com/h2oai/h2ogpt.
  * Hendrycks et al. (2021) Hendrycks, D.; Basart, S.; Kadavath, S.; Mazeika, M.; Arora, A.; Guo, E.; Burns, C.; Puranik, S.; He, H.; Song, D.; et al. 2021.  Measuring coding challenge competence with apps.  _arXiv preprint arXiv:2105.09938_.
  * Hoffmann et al. (2022) Hoffmann, J.; Borgeaud, S.; Mensch, A.; Buchatskaya, E.; Cai, T.; Rutherford, E.; de Las Casas, D.; Hendricks, L. A.; Welbl, J.; Clark, A.; et al. 2022.  An empirical analysis of compute-optimal large language model training.  _Advances in Neural Information Processing Systems_ , 35: 30016–30030.
  * Hu et al. (2021) Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_.
  * Hu et al. (2023) Hu, Z.; Lan, Y.; Wang, L.; Xu, W.; Lim, E.-P.; Lee, R. K.-W.; Bing, L.; and Poria, S. 2023.  LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models.  _arXiv preprint arXiv:2304.01933_.
  * Huang et al. (2019) Huang, L.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2019.  Cosmos QA: Machine reading comprehension with contextual commonsense reasoning.  _arXiv preprint arXiv:1909.00277_.
  * Huang et al. (2023) Huang, Y.; Bai, Y.; Zhu, Z.; Zhang, J.; Zhang, J.; Su, T.; Liu, J.; Lv, C.; Zhang, Y.; Lei, J.; et al. 2023.  C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models.  _arXiv preprint arXiv:2305.08322_.
  * Ji et al. (2023a) Ji, Y.; Deng, Y.; Gong, Y.; Peng, Y.; Niu, Q.; Zhang, L.; Ma, B.; and Li, X. 2023a.  Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases.  _arXiv preprint arXiv:2303.14742_.
  * Ji et al. (2023b) Ji, Y.; Gong, Y.; Deng, Y.; Peng, Y.; Niu, Q.; Ma, B.; and Li, X. 2023b.  Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluation.  _arXiv preprint arXiv:2304.07854_.
  * Köpf et al. (2023) Köpf, A.; Kilcher, Y.; von Rütte, D.; Anagnostidis, S.; Tam, Z.-R.; Stevens, K.; Barhoum, A.; Duc, N. M.; Stanley, O.; Nagyfi, R.; ES, S.; Suri, S.; Glushkov, D.; Dantuluri, A.; Maguire, A.; Schuhmann, C.; Nguyen, H.; and Mattick, A. 2023.  OpenAssistant Conversations – Democratizing Large Language Model Alignment.  arXiv:2304.07327.
  * Li et al. (2023) Li, H.; Zhang, Y.; Koto, F.; Yang, Y.; Zhao, H.; Gong, Y.; Duan, N.; and Baldwin, T. 2023.  CMMLU: Measuring massive multitask language understanding in Chinese.  _arXiv preprint arXiv:2306.09212_.
  * Liang et al. (2022) Liang, P.; Bommasani, R.; Lee, T.; Tsipras, D.; Soylu, D.; Yasunaga, M.; Zhang, Y.; Narayanan, D.; Wu, Y.; Kumar, A.; et al. 2022.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_.
  * Liu et al. (2023) Liu, C.; Jin, R.; Ren, Y.; Yu, L.; Dong, T.; Peng, X.; Zhang, S.; Peng, J.; Zhang, P.; Lyu, Q.; et al. 2023.  M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Models.  _arXiv preprint arXiv:2305.10263_.
  * Muennighoff et al. (2022) Muennighoff, N.; Wang, T.; Sutawika, L.; Roberts, A.; Biderman, S.; Scao, T. L.; Bari, M. S.; Shen, S.; Yong, Z.-X.; Schoelkopf, H.; Tang, X.; Radev, D.; Aji, A. F.; Almubarak, K.; Albanie, S.; Alyafeai, Z.; Webson, A.; Raff, E.; and Raffel, C. 2022.  Crosslingual Generalization through Multitask Finetuning.  arXiv:2211.01786.
  * Niklaus et al. (2023) Niklaus, J.; Matoshi, V.; Rani, P.; Galassi, A.; Stürmer, M.; and Chalkidis, I. 2023.  Lextreme: A multi-lingual and multi-task benchmark for the legal domain.  _arXiv preprint arXiv:2301.13126_.
  * OpenAI (2023a) OpenAI. 2023a.  ChatGPT: Optimizing Language Models for Dialogue.  https://openai.com/blog/chatgpt.
  * OpenAI (2023b) OpenAI. 2023b.  GPT-4 Technical Report.  arXiv:2303.08774.
  * Ouyang et al. (2022) Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35: 27730–27744.
  * Perez et al. (2022) Perez, E.; Ringer, S.; Lukošiūtė, K.; Nguyen, K.; Chen, E.; Heiner, S.; Pettit, C.; Olsson, C.; Kundu, S.; Kadavath, S.; et al. 2022.  Discovering Language Model Behaviors with Model-Written Evaluations.  _arXiv preprint arXiv:2212.09251_.
  * Raffel et al. (2020) Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1): 5485–5551.
  * Sai, Mohankumar, and Khapra (2022) Sai, A. B.; Mohankumar, A. K.; and Khapra, M. M. 2022.  A survey of evaluation metrics used for NLG systems.  _ACM Computing Surveys (CSUR)_ , 55(2): 1–39.
  * Scao et al. (2022) Scao, T. L.; Fan, A.; Akiki, C.; Pavlick, E.; Ilić, S.; Hesslow, D.; Castagné, R.; Luccioni, A. S.; Yvon, F.; Gallé, M.; et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Srivastava et al. (2022) Srivastava, A.; Rastogi, A.; Rao, A.; Shoeb, A. A. M.; Abid, A.; Fisch, A.; Brown, A. R.; Santoro, A.; Gupta, A.; Garriga-Alonso, A.; et al. 2022.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _arXiv preprint arXiv:2206.04615_.
  * StabilityAI (2023) StabilityAI. 2023.  StableLM: Stability AI Language Models.  https://github.com/Stability-AI/StableLM.
  * Sun et al. (2023a) Sun, T.; Xiaotian, Z.; Zhengfu, H.; Peng, L.; Qinyuan, C.; Hang, Y.; Xiangyang, L.; Yunfan, S.; Qiong, T.; Xingjian, Z.; Ke, C.; Yining, Z.; Zhejian, Z.; Ruixiao, L.; Jun, Z.; Yunhua, Z.; Linyang, L.; Xiaogui, Y.; Lingling, W.; Zhangyue, Y.; Xuanjing, H.; and Xipeng, Q. 2023a.  FudanNLP Moss.
  * Sun et al. (2023b) Sun, T.; Zhang, X.; He, Z.; Li, P.; Cheng, Q.; Yan, H.; Liu, X.; Shao, Y.; Tang, Q.; Zhao, X.; Chen, K.; Zheng, Y.; Zhou, Z.; Li, R.; Zhan, J.; Zhou, Y.; Li, L.; Yang, X.; Wu, L.; Yin, Z.; Huang, X.; and Qiu, X. 2023b.  MOSS: An open-source tool-augmented conversational language model from Fudan University.  https://github.com/OpenLMLab/MOSS.
  * Suzgun et al. (2022) Suzgun, M.; Scales, N.; Schärli, N.; Gehrmann, S.; Tay, Y.; Chung, H. W.; Chowdhery, A.; Le, Q. V.; Chi, E. H.; Zhou, D.; et al. 2022.  Challenging BIG-Bench tasks and whether chain-of-thought can solve them.  _arXiv preprint arXiv:2210.09261_.
  * Taori et al. (2023) Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023.  Stanford Alpaca: An Instruction-following LLaMA model.  https://github.com/tatsu-lab/stanford˙alpaca.
  * Touvron et al. (2023) Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Wang et al. (2023a) Wang, E. J.; Alexiuk, C.; Bo, Y.; Yang, Z.; Kwok, K.; Gusev, I.; Echavez, A.; et al. 2023a.  Alpaca-LoRA.  https://github.com/tloen/alpaca-lora.
  * Wang et al. (2023b) Wang, J.; Liang, Y.; Meng, F.; Shi, H.; Li, Z.; Xu, J.; Qu, J.; and Zhou, J. 2023b.  Is chatgpt a good nlg evaluator? a preliminary study.  _arXiv preprint arXiv:2303.04048_.
  * Wei et al. (2022) Wei, J.; Tay, Y.; Bommasani, R.; Raffel, C.; Zoph, B.; Borgeaud, S.; Yogatama, D.; Bosma, M.; Zhou, D.; Metzler, D.; et al. 2022.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_.
  * Xiong et al. (2023) Xiong, H.; Wang, S.; Zhu, Y.; Zhao, Z.; Liu, Y.; Huang, L.; Wang, Q.; and Shen, D. 2023.  DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task.  arXiv:2304.01097.
  * Xu et al. (2023) Xu, C.; Guo, D.; Duan, N.; and McAuley, J. 2023.  Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data.  _arXiv preprint arXiv:2304.01196_.
  * Zellers et al. (2019) Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019.  HellaSwag: Can a machine really finish your sentence?  _arXiv preprint arXiv:1905.07830_.
  * Zeng et al. (2023) Zeng, A.; Liu, X.; Du, Z.; Wang, Z.; Lai, H.; Ding, M.; Yang, Z.; Xu, Y.; Zheng, W.; Xia, X.; Tam, W. L.; Ma, Z.; Xue, Y.; Zhai, J.; Chen, W.; Liu, Z.; Zhang, P.; Dong, Y.; and Tang, J. 2023.  GLM-130B: An Open Bilingual Pre-trained Model.  In _The Eleventh International Conference on Learning Representations (ICLR)_.
  * Zeng (2023) Zeng, H. 2023.  Measuring Massive Multitask Chinese Understanding.  _arXiv preprint arXiv:2304.12986_.
  * Zhao et al. (2023) Zhao, W. X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; et al. 2023.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_.
  * Zhong et al. (2023) Zhong, W.; Cui, R.; Guo, Y.; Liang, Y.; Lu, S.; Wang, Y.; Saied, A.; Chen, W.; and Duan, N. 2023.  AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models.  _arXiv preprint arXiv:2304.06364_.
  * Zhou et al. (2023) Zhou, C.; Li, Q.; Li, C.; Yu, J.; Liu, Y.; Wang, G.; Zhang, K.; Ji, C.; Yan, Q.; He, L.; et al. 2023.  A comprehensive survey on pretrained foundation models: A history from bert to chatgpt.  _arXiv preprint arXiv:2302.09419_.
  * Zhu et al. (2019) Zhu, M.; Deng, Z.; Xiong, W.; Yu, M.; Zhang, M.; and Wang, W. Y. 2019.  Neural Correction Model for Open-Domain Named Entity Recognition.  _arXiv preprint arXiv:1909.06058_.

##  Appendix A Appendix

### Discussion

Large Language Models Need More Benchmarks

In summary, the capabilities of LLMs are manifested in three distinct aspects
(Ouyang et al. 2022). And all three of these categories require benchmarks for
automated evaluation. Although many benchmarks are constructed after the
release of ChatGPT or GPT-4, LLMs still faced the problem of insufficient
evaluation dimensions and insufficient evaluation detail because LLMs are more
expressive than ever. Thus, we call upon the academic and industrial sectors
to summarize human knowledge and values, providing LLM development with more
effective, comprehensive, and advanced benchmarks.

The first capability of LLMs is the understanding of knowledge, which
encompasses memorization, reasoning, and abstraction (Zhou et al. 2023).
Currently, most works focus on enhancing the knowledge and understanding of
LLMs through pre-training (Fu, Peng, and Khot 2022). The proposal of Xiezhi is
aiming at establishing a taxonomy for human knowledge and building evaluation
criteria for this field. Although Xiezhi is already the most dimensional
domain evaluation benchmark with largest volume of data, we currently offer
only Chinese and English language version and lacks comprehensive coverage of
knowledge from different cultures and industries. In the future, one of the
critical improvements for Xiezhi lies in collecting more thorough and in-depth
knowledge from various countries, nations, fields, and open source benchmarks
in more languages.

Except for knowledge evaluation, there are two other capabilities of LLMs that
are in great need of benchmarks. One capacity is to understand and execute
instructions, rendering LLM into a valuable artificial tool (Aribandi et al.
2021; Hoffmann et al. 2022). Instruction fine-tuning is greatly involved in
many works to enhance LLM’s instruction-following ability. However, the
evaluation of LLM functionality largely relies on manual verification at
present. Another is to align with human values, which is essential for LLMs to
evolve into artificial general intelligence (AGI) (Bai et al. 2022; Perez et
al. 2022). Numerous technical approaches for alignment have been proposed by
companies like OpenAI and Claude, but many works have not aligned their models
with human values due to the lack of direct improvement in downstream
applications.

Large Language Models Need Better Evaluation Methods

Current language models predominantly adopt generative approaches (Zhao et al.
2023), and naturally, assessing these models presents inherent challenges
(Wang et al. 2023b). Most existing evaluation methods utilize multiple-choice
questions to measure a generative model’s understanding of knowledge and
employ extraction techniques to obtain the model’s answers (Huang et al. 2023;
Liu et al. 2023; Hendrycks et al. 2021).

We argue that this evaluation approach is a sub-optimal approach. Since this
approach requires models to possess the capability to answer multiple-choice
questions, a skill seldom employed in real-world applications. For small LLMs
or LLMs that have not been fine-tuned with multiple-choice data, such
evaluation approaches fail to provide effective performance indicators.

In this paper, we propose evaluating models by using generative probability.
While generative probability increases computational costs in comparison to
directly answering questions, it yields a more accurate and effective
assessment for LLMs unable to answer multiple-choice questions. Our study
serves as an exploration of improved evaluation methodologies. In the future,
we will consider incorporating a wider variety and diversity of evaluation
approaches.

### Detail Hyper-parameters

Our experiments involved two types of hyperparameters. The first type pertains
to the seeds of random numbers used in various Python libraries, while the
second type refers to the hyperparameters used when invoking the AutoCausalLM
class from the transformers library for generation. The first type of
hyperparameters will impact our experiments to some extent, and hence we
ensured that the random number seeds were all set to 42, which are presented
in Table 3, Considering that we use the generation probabilities of each
options without actually generate new content, so the second type of
hyperparameters would not impact our experimental results since they all
effect the output from a AutoCausalLM Class. Nonetheless, to ensure
consistency on details we might not have noticed, we configured our settings
as demonstrated in Table 3. The code and data utilized in our study can be
accessed in the CodeAndDataAppendix. Reproduction of the experiments can be
achieved by simply executing the ./Tester/test.sh file contained in our code
repository.

Random Seed
---
torch.manual_seed | torch.cuda.manual_seed_all | numpy.random.seed | random.seed | torch.backends.cudnn.deterministirc
42 | 42 | 42 | 42 | True
AutoCausalLM
temperature | top_p | top_k | num_beams | max_new_token
0.95 | 0.95 | 5 | 2 | 1

Table 3: All the parameter setting in our experiments.

### Prompts

We employed customized prompts in two scenarios: During the “Discipline
Annotation” phase in the annotation of Xiezhi and during the “LLMs Output”
phase as part of our experiment setting.

During the Discipline Annotation, given the existence of over 500 discipline
labels, direct input of all labels will incur huge economical expense. So we
used the Discipline Taxonomy for hierarchical annotation. Initially, all 13
discipline categories were used to query models and determine which categories
the question should belong to. Subsequently, we selected the first-level
disciplines within the chosen categories for further query, a process repeated
with second-level disciplines. This annotation strategy was used to assign
discipline labels to Xiezhi-Meta courtesy of ChatGPT and Xiezhi-All via both
the annotation model and ChatGPT. The prompts used for these procedures are
illustrated in the “Chinese-Version” section of Table 4, with the English
version of the prompts also provided in the “English-Version” section of the
same table for better illustration.

Regarding the model’s prompt-based option output, we designed four distinctive
prompts to cater to Chinese and English languages, and possible
demonstrations, as depicted in the “LLMs Output” of Table 4.

Description | Prompts
---|---
Discipline Annotation
| Chinese-Version (Used
---
in annotation process.)
| 下面我会给你几个学科标签，然后会给你一道选择题的问题、选项和答案，请你从我给你的学科名称中为这
---
道选择题选择合适的学科标签，如果你认为没有合适的标签，你可以回答“N/A”。
学科标签：
xxx: yyyyyyyy
xxx: yyyyyyyy
……
选择题问题：
xxxxxx
选择题选项：
1.xxx
2.yyy
选择题答案：
xxx
请告诉我你认为这道题涉及了哪些学科标签，请严格使用给定的学科标签进行回复，不同的学科标签之间请用
中文顿号“、”进行分割：
| English-Version (Only
---
for illustration.)
| Below, I will provide you with some subject labels and present you with a
multiple-choice question, comprising of the
---
problem, options, and answer. Please allocate an appropriate subject label
from the given set to this multiple-choice
question. If you believe that there is no suitable label, you may reply with
”N/A”.
Subject Labels:
xxx: yyyyyyy
xxx: yyyyyyy
……
Question Description：
xxxxxx
Options:
1.xxx
2.yyy
Answer:
xxx
Please inform me of the subject labels you believe this question encompasses.
Make sure to strictly use the given subject
labels for your response, and separate different subject labels with the
Chinese punctuation mark “、”:
LLMs Output
English-0-shot |  | ### question description: “‘{question}”’
---
###all options: “‘{options}”’
### answer: “‘{answer}”’
{eos}
English-few-shot |  | “‘{demonstrations}”’
---
### question description: “‘{question}”’
### all options: “‘{options}”’
### answer: “‘{answer}”’
{eos}
Chinese-0-shot |  | ### 问题描述: “‘{question}”’
---
### 所有选项: “‘{options}”’
### 答案: “‘{answer}”’
{eos}
Chinese-few-shot |  | “‘{demonstrations}”’
---
### 问题描述: “‘{question}”’
### 所有选项: “‘{options}”’
### 答案: “‘{answer}”’
{eos}

Table 4: All the prompt we used in both annotation and experiments.

### Manual Annotation

Figure 4: An illustration of the construction process of Xiezhi.

Annotators

We hired and paid graduate students from various majors to annotate and clean
the questions we collected. We provided training for all potential annotators
and tested their understanding of the training content to ensure they fully
met our requirements. The training mainly about the requirements in
annotation, we will talk about these requirements in the next subsection. Our
annotators comprised of ten Chinese graduate students specializing in diverse
disciplines: Medicine, Literature, Economics, Science, Jurisprudence, History,
Management, and Engineering. The annotators possess expertise in their
respective fields of study and have written numerous academic papers in
English, thereby exhibiting a high level of proficiency in the English. We
keep communicate with them during the annotation process, and all of them are
paid above the local minimum wage. To ensure the quality of the annotations,
each sample is annotated by at least three annotators.

Annotation Process in Benchmark Construction

As shown in Fig. 4, we performed manual annotation at two points: the
construction of Xiezhi-Meta, and the Verification of Xiezhi-Specialty and
Xiezhi-Interdiscipline for different purposes.

For the construction of Xiezhi-Meta, the aim is to construct a high-quality
Chinese domain knowledge dataset, covering as many discipline labels as
possible. The goal of manual annotation primarily lies in ensuring the
correctness of the questions themselves. Therefore, we ask annotators to
filter the dataset of Xiezhi-Meta under the following requirements:

  1. 1.

Choose questions where answers can be determined solely based on textual
content.

  2. 2.

Select questions with correct answers.

  3. 3.

Choose questions with rationally set options.

  4. 4.

The subject annotated by ChatGPT for the question is correct, or it can be
modified to be correct (this requires annotators to make modifications).

In terms of the construction of Xiezhi-Specialty and Xiezhi-Interdiscipline,
the goal is to construct a dataset that meets the requirements of domain
knowledge evaluation for LLMs and conforms to human values. Consequently, for
these two datasets, we propose the following requirements for annotators:

  1. 1.

If the question needs non-textual information to be solved, it should be
removed.

  2. 2.

If issue that cannot be modified was introduced during the crawling of
questions, it should be removed.

  3. 3.

Questions with incorrect answers should be removed.

  4. 4.

Questions with unreasonably set options should be removed.

  5. 5.

Questions that contain gender-biased content should be removed.

  6. 6.

Questions involving sensitive content such as military matters and politics
should be removed.

  7. 7.

Questions that contain China’s ancient texts and contemporary political
content should be removed.

  8. 8.

If the subject annotation is incorrect, it should be removed.

  9. 9.

Questions, and options that highly replicate the content of other questions
should be removed.

  10. 10.

Questions with discriminatory content should be removed.

  11. 11.

If deletion of a question leads to imbalance, new questions should be re-
selected from Xiezhi-All to be included.

  12. 12.

The reference of male and female appellations in the dataset should be
balanced.

Examples in Annotation Process

In this section, we provide a detailed list of real examples encountered
during our manual annotation process. Firstly, we removed all the disciplines
included in Tab. 5 from Xiezhi-Specialty and Xiezhi-Interdiscipline. Some of
these disciplines require a super deep understanding of the Chinese cultural,
which does not align with current model development needs and is too difficult
even for Chinese LLMs. Others involve too much military-related content, which
is not suitable for open source dissemination. In addition, regarding all the
requirements we outlined in the prior section, specific examples are shown in
Table 6.

Deeply Chinese Related Disciplines
---
Chinese Classical Literature, History of the Chinese Communist Party, Ancient
Chinese Literature
Marxist Theory and Ideological and Political Education
Military Related Disciplines
Military Science, Military Political Work, Military Political Work Studies,
Military Logistics, Military logistics and military, Military Strategy,
Military tactics, Contract Tactics,
War Mobilisation equipment science, Military Logistics, Rear professional
logistics, Strategic Studies,
Military Equipment Studies, Military Thought and Military History, Military
Thought, Tactics,
Military History, Military Equipment Studies, Military Training, Military
Systems, Joint Warfare,
Military Organization, Military Management, Military Command, Operational
Command,
Military Operations Research, Military Communications, Military Intelligence,
Cryptology,
Military Education and Training, Campaign Studies, Military Service Campaign
Studies.

Table 5:  All the subjects we delete from Xiezhi-Speciality and Xiezhi-
Disciplinary.

Xiezhi-Meta
---
| 1\. Do not select questions that cannot be answered based solely on
---
the content of the text.
| Exclude
---
Question:
“Based on the given figure, ….”
| 2\. Do not select questions with incorrect answers.
---
| Exclude
---
Question:
“The theory of time is first proposed by ______”
Answer:
“Stephen Hawking”, Real Answer: “Albert Einstein”
| 3\. Do not select questions with improperly formed alternatives.
---
| Exclude
---
Question:
“Who wrote ‘Pride and Prejudice’?”
Options:
“Joanne Rowling, J.K. Rowling, Stephen King, …”
| 4\. Do not select questions where the subject markers are mislabelled
---
and cannot be corrected.
| Exclude
---
Question:
“If you go back to the Spring and Autumn Period and the Warring States Period,
what you can do is ______ ”
Labels:
[Ancient Literature, …]
Xiezhi-Speciality & Xiezhi-Interdiscipline
| 1\. If the question needs non-textual information to be solved,
---
it should be removed.
| Delete
---
Question:
“Based on the given figure, ….”
| 2\. If issue that cannot be modified was introduced during the
---
crawling of questions, it should be removed.
| Delete
---
Question:
“¡banana¿ swirling css { ¡chairs margin=1em auto¿ } upwardly sky blues.”
| 3\. Questions with incorrect answers should be removed.
---
| Delete
---
Question:
“The theory of time is first proposed by ______”
Answer:
“Stephen Hawking”,
Real Answer:
“Albert Einstein”
| 4\. Questions with unreasonably set options should be removed.
---
| Delete
---
Answer:
“Samurai”,
Options:
“Bushi, Anchor, Archer, ……”
| 4\. If the options are set improperly in the question,
---
it should be deleted.
|  Question:
---
“Who wrote ‘Pride and Prejudice’?”
Options:
“Joanne Rowling, J.K. Rowling, Stephen King, …”
| 5\. Questions that contain gender-biased content should be removed.
---
| Delete
---
Question:
“Since men usually cook better than women …”
| 6\. Questions involving sensitive content such as military matters
---
and politics should be removed.
| Delete
---
Question:
“The implications of the recent nuclear agreement between …”
| 7\. Questions that contain China’s ancient texts and contemporary
---
political content should be removed.
| Delete
---
Question:
“The next sentences of The Dao that can be spoken is not the eternal Dao is
______”
| 8\. If the subject annotation is incorrect, it should be removed.
---
| Delete
---
Question:
“If you go back to the Spring and Autumn Period and the Warring States Period,
what you can do is ______”
Labels:
[Ancient Literature, …]
| 9\. Questions, and options that highly replicate the content
---
of other questions should be removed.
| /
---
| 10\. Questions with discriminatory content should be removed.
---
| Delete
---
Question:
“People of a certain race smarter than others, …”
| 11\. If deletion of a question leads to imbalance, new questions
---
should be re-selected from Xiezhi-All to be included.
| /
---
| 12\. The reference of male and female appellations in the dataset
---
should be balanced.
| Replace the male appellations in the sentence with female ones, while paying
attention
---
to modification of personal names.

Table 6:  Real examples in our annotation process.

### Auto Annotator

Figure 5:  The precision, recall, and error rates of various annotation
policies.

As shown in Figure 4, during the construction of the benchmark, we selected
20,124 questions from the Chinese postgraduate examination to form Xiezhi-
Meta. After manual verification, these translated questions were used to train
an Annotation Model, which aided in annotating all data in Xiezhi-All. We
refer to the trained model as the “Annotation Model”, and elaborate on this
model and the training details within this subsection.

Model and Training Process: We directly utilized the most up-to-date
Llama-7B-Chinese model 222https://huggingface.co/LinkSoul/Chinese-Llama-2-7b,
which is a model enhanced through secondary pre-training using Chinese corpora
based on the basic Llama-7B. Therefore, it has robust capabilities in
processing Chinese Language as shown in Tab. 1 of the overall experiments. We
primarily fine-tuned the instruction on Llama-7B-Chinese. The code used for
this process is the EasyLLM 333https://github.com/MikeGu721/EasyLLM training
framework available on GitHub.

Training Data:  We constructed the data using the Chinese Version of the
prompts in Table 4. Although the input questions, discipline labels and
descriptions of discipline labels are mainly Chinese. We required the model to
output Chinese disciplines labels relevant to the given question.

Annotation Performance: Our experimental study involved the analysis of 20,124
questions derived from Xiezhi-Meta. Three distinct strategies, namely
Annotation Model, Annotation Model + ChatGPT, and ChatGPT, were employed to
annotate these questions. The annotated datasets were subsequently compared to
the manual validation results. In the Annotation Model + ChatGPT strategy, the
Annotation Model was employed to annotate the discipline categories, and
ChatGPT was applied to annotate the first and second-level disciplines by
leveraging the annotated discipline categories. The evaluation metrics used
were the Wrong Rate, Missing Rate, and Error Rate:

  * •

Wrong Rate indicates the number of incorrect annotated discipline labels,
calculated as “[SET(manual labeling results) - SET(annotation
results)]/#questions”.

  * •

Missing Rate is used to indicate the number of discipline labels omitted by
the annotation strategy; it was calculated as “[SET(annotation results) -
SET(manual labeling results)]/#questions”.

  * •

Error Rate is the summation of the Wrong Rate and Missing Rate metrics, which
denotes the probability of manual involvement in the annotation process.

It is important to note that multiple labels may be missed or incorrectly
labeled for each question, so all the rates used here are not 0 1 metrics.

The results presented in Table 5 reveal that the Annotation Model has good
performance in coarse-grained discipline classification but is not as
effective in fine-grained discipline classification. It is observed that all
three strategies aimed to ensure high precision in their outputs, resulting in
low Wrong Rates. However, the Annotation Model missed many discipline labels,
particularly in the first and second-level subjects. The Missing Rate of the
Annotation Model + ChatGPT policy is much higher is because the missiong of
the discipline categories annotated by Annotation Model

Annotation of Xiezhi-All: To ensure the quality of Xiezhi-All, we applied a
combined annotation approach using Annotation Model and ChatGPT. Specifically,
we used the Annotation Model to annotate the subject categories of each
question, and used ChatGPT to annotate the primary subject based on the
already annotated subject categories, then annotated secondary subjects based
on the annotated primary subjects. This iterative annotation method can reduce
the number of subjects input into ChatGPT, thereby reducing overhead.
Moreover, adopting this annotation approach has significantly saved the time
spent on manually filtering Xiezhi-Speciality and Xiezhi-Interdiscipline. In
the future, we plan to use the annotated data to further train the Annotation
Model, aiming for its performance in subject annotation tasks to approach or
surpass that of ChatGPT, thereby eliminating the need for ChatGPT for
annotation.

### Bias, Ethical Problems and Social Impact

Even though we strive to avoid all possible ethical issues, we still find it
challenging to guarantee the resolution of all ethical issues when it comes to
constructing the largest knowledge-based evaluation benchmark in the world,
with the largest number of discipline labels, largely compiled and translated
from a single language. In this section, we provide a detailed description of
all the potential ethical issues that may exist in our dataset, along with how
we alleviate them.

Bias From Multilingual Dataset Curation

As a dataset constructed from Chinese sources, with the annotators being
Chinese graduates, invariably, there is a level of Chinese bias. These bias
may come from a large number of Chinese related questions, Chinese Polical
Standards or Chinese Distinct Value, which may result in a better Chinese
understanding models will have better performance in Xiezhi. We have made
extensive efforts to eliminate such bias in Xiezhi. Measures include:

  * •

Selecting questions from Xiezhi-All that involve various countries and
regions, rather than those biased towards Chinese contexts.

  * •

Ensuring balanced distribution of virtual names in both Chinese and English
styles, such as Li Hua in Chinese and Mike in English.

  * •

Deleting questions involving political stances.

  * •

Adding modified text like “In China, …” to the beginning of questions
involving common Chinese values.

Translation Version of Xiezhi

In this paper, we used Google Translate API to translate Xiezhi-Speciality and
Xiezhi-Interdisciplinary into English, followed by extensive manual post-
processing. The focus of verification includes several aspects:

  * •

Correcting translation errors

  * •

Correcting sentences with unnatural expressions

  * •

Making precise expression for specific terms

We invited 10 annotators described in Appendix Manual Annotation to
participate in the revision of the English version of Xiezhi. As all
annotators are graduates who all deeply involved in writing several English
papers, we are confident in their proficiency in both professional knowledge
and English language use. Despite our high standards for the translated
version, we still believe potential issues may exist in the current version of
Xiezhi:

  * •

Annotators are not native speakers living in English-speaking countries, so
their expressions might not be perfectly idiomatic.

  * •

The dataset covers 516 different fields. Although the annotators are graduate
students with experience in English writing, precision in translating field-
specific terms could be lacking.

Human Performance

We elaborated on the statistical method for human scoring in the experiment
setting in Section Experiments. This scoring method compromises for the sheer
reason that simply acquiring questions from one source would result in
insufficient number and highly biased data. Moreover, as the content covered
by each dataset varies greatly, we also found it hard to invite human
participants from all 516 fields to provide human baselines. We believe the
human performance scores we provided to some extent reflect human performance
on Xiezhi-Specialty and Xiezzi-Interdisciplinary. However, the real-life
decision-makers should not solely rely on these scores for these scores are
only used as an comparison.

Gender, Race, Religion, National Discrimination or Prejudice

The proposed of a benchmark will act as an indicator for LLM training for a
period of time, if the benchmark itself harbors discriminatory or prejudiced
content, it may encourage poor development of LLMs. Therefore, in compliance
with the NeurIPS dataset review standards, we eliminated all content related
to gender, race, religion, and national discrimination or prejudice in the
posterior process of Xiezhi-Specility and Xiezhi-Interdisciplinary as far as
possible. Our efforts in eliminating prejudiced content and discrimination are
evident from the requirements listed in Table 6.

Despite this, considering most of our questions were extracted from Chinese
exam papers, and a small fraction generated from English papers, with
annotation undertaken by Chinese postgraduate students, there still might be
potential ethical issues in the Xiezhi:

  * •

As our questions come from Chinese exams and English published papers, and our
annotators are Chinese graduates, their annotation may unknowingly lean
towards Eastern or Western cultural notions.

  * •

In consideration of the uneven distribution of gender, race, faith, and
nationalities that can access the original questions, we resolved the issues
of gender distribution in Xiezhi-Speciality and Xiezhi-Interdisciplinay. The
implicit prejudice brought by race, faith, and nationality may be more severe,
and though we ensured that our existing questions do not include prejudiced
content in the question description and answers, we are unable to change the
overall bias in the question distribution.

The detail about how we follow the NeurIPS checklist and DataSheet is
described in Appendix Checklist and Appendix Datasheet.

### Results on Other Metrics

Aside from the MRR score metric championed in our paper for ranking options,
we have listed some other indicators to gauge the performance of different
models on Xiezhi-Speciality and Xiezhi-Interdisciplinay. The variants brought
about by the different indicators have also been analyzed.

We have also considered ranking in our metrics, employing Hit@1, Hit@4, and
Mean Rank as indicators. The descriptions of these are as follows: We also
utilized the conventional method of calculating accuracy, a method heavily
employed in other papers.

  * •

Mean Rank (MR): This measures the average rank position of a query concept’s
true parent among all candidates, divided by the total number of options.

  * •

Hit@k: This is the number of query concepts whose parent is ranked in the top
k positions, divided by the total number of queries.

  * •

Accuracy: A standard measure used in most research.

Given the extensive computational cost, we carried out this experiment using
only a subset of models. The results from different models, using different
indicators, are presented in Tables 7 8 9 10.

Our findings indicate that even when addressing the same dataset, different
evaluation metrics yield different rankings. We suspect this may be because
varying evaluation metrics unearth different characteristics encapsulated
within the models. This is a significant factor in model evaluation and should
be deeply investigated in order to draw comprehensive conclusions. Therefore,
in our future work, we anticipate thoroughly researching the varying ranking
results driven by different evaluation metrics.

Xiezhi-Speciality-English
---
MRR | Hit@1 | Hit@4 |  MR$\downarrow$
GPT-4 | 0.476 | GPT-4 | 0.188 | GPT-4 | 0.641 | GPT-4 | 0.089
GPT-3.5 | 0.254 | GPT-3.5 | 0.101 | GPT-3.5 | 0.285 | GPT-3.5 | 0.204
bloomz-mt | 0.386 | bloomz-mt | 0.096 | bloomz-mt | 0.268 | bloomz-7b1-mt | 0.317
bloomz-7b1 | 0.256 | bloomz-7b1 | 0.083 | bloomz-3b | 0.259 | bloomz-7b1 | 0.325
bloomz-7b1-mt | 0.242 | bloomz-7b1-mt | 0.080 | bloomz-7b1 | 0.250 | bloomz-mt | 0.333
falcon-7b | 0.223 | BELLE-7B-2M | 0.065 | bloomz-7b1-mt | 0.228 | llama-7b-hf | 0.339
baize-healthcare-lora-7B | 0.218 | bloomz-7b1-p3 | 0.061 | falcon-7b-instruct | 0.213 | falcon-40b-instruct | 0.394
bloomz-3b | 0.187 | baize-healthcare-lora-7B | 0.060 | BELLE-7B-2M | 0.204 | falcon-7b-instruct | 0.424
bloomz-7b1-p3 | 0.173 | falcon-7b-instruct | 0.059 | falcon-7b | 0.186 | BELLE-7B-2M | 0.460
falcon-7b-instruct | 0.167 | bloomz-1b1 | 0.059 | bloomz-1b1 | 0.153 | falcon-7b | 0.462
bloomz-1b1 | 0.160 | bloomz-1b7 | 0.056 | bloomz-7b1-p3 | 0.150 | bloomz-3b | 0.490
bloomz-1b7 | 0.156 | falcon-40b-instruct | 0.055 | bloomz-1b7 | 0.146 | bloomz-1b1 | 0.494
Random | 0.089 | Random | 0.020 | Random | 0.080 | Random | 0.510

Table 7:  The ranking of various models according to different metrics within
the Xiezhi-Speciality-English Benchmark.

Xiezhi-Interdiscipline-English
---
MRR | Hit@1 | Hit@4 |  MR$\downarrow$
GPT-4 | 0.509 | GPT-4 | 0.243 | GPT-4 | 0.861 | GPT-4 | 0.071
GPT-3.5 | 0.382 | GPT-3.5 | 0.091 | GPT-3.5 | 0.371 | GPT-3.5 | 0.164
bloomz-mt | 0.329 | bloomz-7b1 | 0.122 | bloomz-7b1 | 0.336 | bloomz-mt | 0.285
bloomz-7b1 | 0.203 | falcon-7b | 0.113 | bloomz-7b1-mt | 0.319 | bloomz-7b1 | 0.287
bloomz-7b1-mt | 0.195 | bloomz-7b1-mt | 0.107 | falcon-7b | 0.300 | bloomz-7b1-mt | 0.370
baize-healthcare-lora-7B | 0.187 | baize-healthcare-lora-7B | 0.106 | baize-healthcare-lora-7B | 0.267 | bloomz-3b | 0.412
BELLE-7B-2M | 0.161 | bloomz-mt | 0.086 | bloomz-mt | 0.210 | llama-7b-hf | 0.428
falcon-7b-instruct | 0.159 | bloomz-1b1 | 0.078 | bloomz-1b7 | 0.171 | falcon-7b | 0.442
falcon-7b | 0.148 | bloomz-1b7 | 0.069 | bloomz-1b1 | 0.170 | falcon-7b-instruct | 0.428
bloomz-1b1 | 0.140 | bloomz-7b1-p3 | 0.062 | bloomz-560m | 0.145 | bloomz-7b1-p3 | 0.442
bloomz-7b1-p3 | 0.139 | bloomz-560m | 0.055 | llama-65b-hf | 0.130 | bloomz-1b1 | 0.475
bloomz-1b7 | 0.138 | falcon-7b-instruct | 0.053 | llama-30b-hf | 0.119 | bloomz-1b7 | 0.476
Random | 0.089 | Random | 0.020 | Random | 0.080 | Random | 0.510

Table 8:  The ranking of various models according to different metrics within
the Xiezhi-Interdiscipline-English Benchmark.

Xiezhi-Speciality-Chinese
---
MRR | Hit@1 | Hit@4 |  MR$\downarrow$
GPT-4 | 0.371 | GPT-4 | 0.104 | GPT-4 | 0.537 | GPT-4 | 0.197
GPT-3.5 | 0.243 | GPT-3.5 | 0.064 | GPT-3.5 | 0.376 | GPT-3.5 | 0.261
bloomz-mt | 0.324 | BELLE-7B-0.6M | 0.089 | baize-lora-7B | 0.304 | llama-65b-hf | 0.344
baize-lora-7B | 0.198 | baize-lora-7B | 0.087 | BELLE-7B-0.6M | 0.260 | llama-7b-hf | 0.354
BELLE-7B-0.6M | 0.190 | vicuna-7b-delta-v1.1 | 0.083 | vicuna-13b-delta-v1.1 | 0.257 | bloomz-3b | 0.387
bloomz-3b | 0.188 | bloomz-7b1-p3 | 0.082 | bloomz-3b | 0.256 | BELLE-7B-0.6M | 0.410
bloomz-7b1-p3 | 0.187 | vicuna-13b-delta-v1.1 | 0.081 | bloomz-mt | 0.241 | bloomz-7b1-p3 | 0.419
vicuna-13b-delta-v1.1 | 0.184 | bloomz-mt | 0.076 | vicuna-7b-delta-v1.1 | 0.238 | gpt-neox-20b | 0.426
vicuna-7b-delta-v1.1 | 0.181 | BELLE-7B-0.2M | 0.075 | bloomz-1b7 | 0.235 | dolly-v2-3b | 0.427
BELLE-7B-0.2M | 0.178 | BELLE-7B-2M | 0.074 | baize-healthcare-lora-7B | 0.233 | bloomz-mt | 0.430
baize-healthcare-lora-7B | 0.177 | BELLE-7B-1M | 0.072 | bloomz-7b1-p3 | 0.232 | llama-13b-hf | 0.439
BELLE-7B-1M | 0.176 | baize-healthcare-lora-7B | 0.071 | BELLE-7B-1M | 0.231 | pythia-1.4b | 0.450
Random | 0.089 | Random | 0.020 | Random | 0.080 | Random | 0.510

Table 9:  The ranking of various models according to different metrics within
the Xiezhi-Speciality-Chinese Benchmark.

Xiezhi-Interdiscipline-Chinese
---
MRR | Hit@1 | Hit@4 |  MR$\downarrow$
GPT-4 | 0.413 | GPT-4 | 0.161 | GPT-4 | 0.714 | GPT-4 | 0.103
GPT-3.5 | 0.214 | GPT-3.5 | 0.084 | GPT-3.5 | 0.237 | GPT-3.5 | 0.188
baize-lora-13B | 0.209 | baize-lora-13B | 0.108 | BELLE-7B-0.6M | 0.269 | BLLE-7B-0.6M | 0.384
BELLE-7B-0.6M | 0.198 | bloomz-mt | 0.100 | baize-lora-13B | 0.252 | llama-13b-hf | 0.396
alpaca-lora-7b | 0.192 | bloomz-7b1-p3 | 0.095 | alpaca-lora-7b | 0.247 | bloomz-7b1-p3 | 0.405
bloomz-7b1-p3 | 0.188 | bloomz-7b1 | 0.083 | BELLE-7B-1M | 0.244 | stablelm-tuned-alpha-7b | 0.419
baize-lora-7B | 0.186 | BELLE-7B-0.6M | 0.081 | baize-lora-7B | 0.234 | bloomz-mt | 0.427
bloomz-mt | 0.185 | BELLE-7B-1M | 0.080 | BELLE-7B-0.2M | 0.229 | falcon-40b-instruct | 0.435
BELLE-7B-1M | 0.180 | BELLE-7B-0.2M | 0.077 | bloomz-mt | 0.220 | llama-30b-hf | 0.441
BELLE-7B-0.2M | 0.173 | bloomz-7b1-mt | 0.076 | bloomz-7b1-p3 | 0.207 | pythia-12b | 0.455
bloomz-7b1 | 0.166 | baize-lora-30B | 0.075 | BELLE-7B-2M | 0.204 | BELLE-7B-1M | 0.465
BELLE-7B-2M | 0.164 | bloomz-3b | 0.075 | falcon-40b-instruct | 0.200 | dolly-v2-7b | 0.465
Random | 0.089 | Random | 0.020 | Random | 0.080 | Random | 0.510

Table 10:  The ranking of various models according to different metrics within
the Xiezhi-Interdiscipline-Chinese Benchmark.

### Models

A comprehensive overview of the evaluated models is presented in Table 11. The
“Model” column specifies the names of the analyzed models, while the
“#Parameter” column indicates their respective parameters. The “Base Model”
column reveals the origins of the fine-tuned models and a dash (-) signifies
that it is not an instruction fine-tuned model. The number of Transformer
layers utilized in each model is denoted by the “#Layer” column, and the
individual encoder and decoder Transformer layers are indicated by the
“#Encoder” and “#Decoder” columns, respectively. Lastly, the “#IFT Sample”
column represents the quantity of instruction samples employed for instruction
fine-tuning.

Model | #Parameter | Base Model | #Layer | #Encoder | #Decoder | #Pretrain Tokens | #IFT Sample
---|---|---|---|---|---|---|---
BLOOM-560m (Scao et al. 2022) 444https://huggingface.co/bigscience/bloom-560m | 0.56B | - | 24 | - | 24 | 350B tokens | -
BLOOMZ-560m (Muennighoff et al. 2022) 555https://huggingface.co/bigscience/bloomz-560m | 0.56B | BLOOM-560m | 24 | - | 24 | - | 3.67B tokens
Pythia-1B (Biderman et al. 2023) 666https://huggingface.co/EleutherAI/pythia-1b | 1B | - | 16 | - | 16 | 300B tokens | -
BLOOM-1b7 (Scao et al. 2022) 777https://huggingface.co/bigscience/bloom-1b7 | 1.7B | - | 24 | - | 24 | 350B tokens | -
BLOOMZ-1b7 (Muennighoff et al. 2022) 888https://huggingface.co/bigscience/bloomz-1b7 | 1.7B | BLOOM-1b7 | 24 | - | 24 | - | 8.39B tokens
Dolly-v2-3b (Conover et al. 2023) 999https://huggingface.co/databricks/dolly-v2-3b | 2.8B | Pythia-2.8B | 32 | - | 32 | - | 15K
Pythia-2.8B (Biderman et al. 2023) 101010https://huggingface.co/EleutherAI/pythia-2.8b | 2.8B | - | 32 | - | 32 | 300B tokens | -
BLOOM-3b (Scao et al. 2022) 111111https://huggingface.co/bigscience/bloom-3b | 3B | - | 30 | - | 30 | 350B tokens | -
BLOOMZ-3b (Muennighoff et al. 2022) 121212https://huggingface.co/bigscience/bloomz-3b | 3B | BLOOM-3b | 30 | - | 30 | - | 8.39B tokens
StableLM-Tuned-Alpha-3B (StabilityAI 2023) 131313https://huggingface.co/stabilityai/stablelm-tuned-alpha-3b | 3B | StableLM-Base-Alpha-3B | 16 | - | 16 | - | 632K
ChatGLM-6B (Zeng et al. 2023; Du et al. 2022) 141414https://huggingface.co/THUDM/chatglm-6b | 6B | - | 28 | 28 | 28 | 1T tokens | ✓
DoctorGLM (Xiong et al. 2023) 151515https://github.com/xionghonglin/DoctorGLM | 6B | ChatGLM-6B | 28 | 28 | 28 | - | 6.38M
Dolly-v2-7b (Conover et al. 2023) 161616https://huggingface.co/databricks/dolly-v2-7b | 6.9B | Pythia-6.9B | 32 | - | 32 | - | 15K
h2ogpt-oig-oasst1-512-6.9b (H2O.ai 2023) 171717https://huggingface.co/h2oai/h2ogpt-oig-oasst1-512-6.9b | 6.9B | Pythia-6.9B | 32 | - | 32 | - | 398K
Pythia-6.9B (Biderman et al. 2023) 181818https://huggingface.co/EleutherAI/pythia-6.9b | 6.9B | - | 32 | - | 32 | 300B tokens | -
Alpaca-7B (Taori et al. 2023) 191919https://huggingface.co/tatsu-lab/alpaca-7b-wdiff | 7B | LLaMA-7B | 32 | - | 32 | - | 52K
Alpaca-LoRA-7B (Wang et al. 2023a) 202020https://huggingface.co/tloen/alpaca-lora-7b | 7B | LLaMA-7B | 32 | - | 32 | - | 52K
Baize-7B (Xu et al. 2023) 212121https://huggingface.co/project-baize/baize-lora-7B | 7B | LLaMA-7B | 32 | - | 32 | - | 263K
Baize Healthcare-7B (Xu et al. 2023) 222222https://huggingface.co/project-baize/baize-healthcare-lora-7B | 7B | LLaMA-7B | 32 | - | 32 | - | 201K
LLaMA-7B (Touvron et al. 2023) 232323https://huggingface.co/decapoda-research/llama-7b-hf | 7B | - | 32 | - | 32 | 1T tokens | -
StableLM-Tuned-Alpha-7B (StabilityAI 2023) 242424https://huggingface.co/stabilityai/stablelm-tuned-alpha-7b | 7B | StableLM-Base-Alpha-7B | 16 | - | 16 | - | 632K
Vicuna-7b-delta-v1.1 (Chiang et al. 2023) 252525https://github.com/lm-sys/FastChat#vicuna-weights | 7B | LLaMA-7B | 32 | - | 32 | - | 70K
BELLE-7B-0.2M 262626https://huggingface.co/BelleGroup/BELLE-7B-0.2M/0.6M 272727https://huggingface.co/BelleGroup/BELLE-7B-0.6M/1M 282828https://huggingface.co/BelleGroup/BELLE-7B-1M/2M 292929https://huggingface.co/BelleGroup/BELLE-7B-2M (Ji et al. 2023a) | 7.1B | Bloomz-7b1-mt | 30 | - | 30 | - | 0.2M/0.6M/1M/2M
BLOOM-7b1 (Scao et al. 2022) 303030https://huggingface.co/bigscience/bloom-7b1 | 7.1B | - | 30 | - | 30 | 350B tokens | -
BLOOMZ-7b1 313131https://huggingface.co/bigscience/bloomz-7b1/mt 323232https://huggingface.co/bigscience/bloomz-7b1-mt/p3 333333https://huggingface.co/bigscience/bloomz-7b1-p3 (Muennighoff et al. 2022) | 7.1B | BLOOM-7b1 | 30 | - | 30 | - | 4.19B tokens
Dolly-v2-12b (Conover et al. 2023) 343434https://huggingface.co/databricks/dolly-v2-12b | 12B | Pythia-12B | 36 | - | 36 | - | 15K
h2ogpt-oasst1-512-12b (H2O.ai 2023) 353535https://huggingface.co/h2oai/h2ogpt-oasst1-512-12b | 12B | Pythia-12B | 36 | - | 36 | - | 94.6K
Open-Assistant-SFT-4-12B (Köpf et al. 2023) 363636https://huggingface.co/OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5 | 12B | Pythia-12B-deduped | 36 | - | 36 | - | 161K
Pythia-12B (Biderman et al. 2023) 373737https://huggingface.co/EleutherAI/pythia-12b | 12B | - | 36 | - | 36 | 300B tokens | -
Baize-13B (Xu et al. 2023) 383838https://huggingface.co/project-baize/baize-lora-13B | 13B | LLaMA-13B | 40 | - | 40 | - | 263K
LLaMA-13B (Touvron et al. 2023) 393939https://huggingface.co/decapoda-research/llama-13b-hf | 13B | - | 40 | - | 40 | 1T tokens | -
Vicuna-13b-delta-v1.1 (Chiang et al. 2023) 404040https://github.com/lm-sys/FastChat#vicuna-weights | 13B | LLaMA-13B | 40 | - | 40 | - | 70K
moss-moon-003-sft (Sun et al. 2023b) 414141https://huggingface.co/fnlp/moss-moon-003-sft | 16B | moss-moon-003-base | 34 | - | 34 | - | 1.1M
moss-moon-003-sft-plugin (Sun et al. 2023b) 424242https://huggingface.co/fnlp/moss-moon-003-sft-plugin | 16B | moss-moon-003-base | 34 | - | 34 | - | 1.4M
GPT-NeoX-20B (Black et al. 2022) 434343https://huggingface.co/EleutherAI/gpt-neox-20b | 20B | - | 44 | - | 44 | 825GB | -
h2ogpt-oasst1-512-20b (H2O.ai 2023) 444444https://huggingface.co/h2oai/h2ogpt-oasst1-512-20b | 20B | GPT-NeoX-20B | 44 | - | 44 | - | 94.6K
Baize-30B (Xu et al. 2023) 454545https://huggingface.co/project-baize/baize-lora-30B | 33B | LLaMA-30B | 60 | - | 60 | - | 263K
LLaMA-30B (Touvron et al. 2023) 464646https://huggingface.co/decapoda-research/llama-30b-hf | 33B | - | 60 | - | 60 | 1.4T tokens | -
LLaMA-65B (Touvron et al. 2023) 474747https://huggingface.co/decapoda-research/llama-65b-hf | 65B | - | 80 | - | 80 | 1.4T tokens | -
BLOOM (Scao et al. 2022) 484848https://huggingface.co/bigscience/bloom | 176B | - | 70 | - | 70 | 366B tokens | -
BLOOMZ 494949https://huggingface.co/bigscience/bloomz/mt 505050https://huggingface.co/bigscience/bloomz-mt/p3 515151https://huggingface.co/bigscience/bloomz-p3 (Muennighoff et al. 2022) | 176B | BLOOM | 70 | - | 70 | - | 2.09B tokens
ChatGPT (2023.05.01) (OpenAI 2023a) 525252https://platform.openai.com/docs/models/gpt-3-5 | - | GPT-3.5 | - | - | - | - | ✓
GPT-4 (2023.05.01) (OpenAI 2023b) 535353https://platform.openai.com/docs/models/gpt-4 | - | - | - | - | - | - | ✓

Table 11: The models we evaluted in our experiments.

### Data Sheet

### Motivation

For what purpose was the dataset created? Was there a specific task in mind?
Was there a specific gap that needed to be filled? Please provide a
description

Xiezhi was created for the purpose of comprehensively evaluating the domain
knowledge capabilities of large language models (LLMs). Some key gaps and
needs that existing benchmarks did not adequately address:

  1. 1.

Existing benchmarks did not cover enough tasks or domains to fully assess the
breadth of knowledge and capabilities of advanced LLMs.

  2. 2.

Many existing benchmarks quickly became outdated as they got incorporated into
the training data of the latest LLMs. There was a need for benchmarks with
fresher data.

  3. 3.

Most benchmarks relied on 4-option multiple choice questions. This made it too
easy for models to guess correctly. More options were needed to better
differentiate model capabilities.

  4. 4.

Existing evaluation methods using multiple choice extractions had limitations
for generative models. A better evaluation approaches are needed and Xiezhi
propose to rank options by generative probability.

Who created this dataset (e.g., which team, research group) and on behalf of
which entity (e.g., company, institution, organization)?

The Knowledge Works Research Laboratory from Fudan University in China created
this dataset.

Who funded the creation of the dataset? If there is an associated grant,
please provide the name of the grantor and the grant name and number.

The grant come from Fudan University.

### Composition

What do the instances that comprise the dataset represent (e.g., documents,
photos, people, countries)? Are there multiple types of instances (e.g.,
movies, users, and ratings; people and interactions between them; nodes and
edges)? Please provide a description.

The instances that comprise the Xiezhi are multiple choice questions designed
to assess domain knowledge across a wide range of disciplines with the
following components:

  * •

Question text: The question or problem statement.

  * •

Answer options: 4 possible options to choose from, with 1 correct answer and 3
near misses.

  * •

Correct answer: The ground truth answer out of the 4 options.

  * •

Subject labels: One or more labels categorizing the discipline/domain of
knowledge required to answer the question correctly (516 total subjects
organized hierarchically into 13 top-level categories).

How many instances are there in total (of each type, if appropriate)?

Xiezhi-All consist of 249,587 questions, Xiezhi-Speciality consists of 14,041
questions, Xiezhi-Interdiscipline consists of 10,746 questions, Xiezhi-Meta
consists of 20,124 questions and Xiezhi-Trian consists of 2,555 questions.

Does the dataset contain all possible instances or is it a sample (not
necessarily random) of instances from a larger set? If the dataset is a
sample, then what is the larger set? Is the sample representative of the
larger set (e.g., geographic coverage)? If so, please describe how this
representativeness was validated/verified. If it is not representative of the
larger set, please describe why not (e.g., to cover a more diverse range of
instances, because instances were withheld or unavailable).

Xiezhi-All contain all possible instances, but we don’t tend to open source it
for it is not verfied by human. Xiezhi-Meta, Xiezhi-Train, Xiezhi-Speciality
and Xiezhi-Interdiscipline are subset of Xiezhi-All but undertook manual
verification.

What data does each instance consist of? “Raw” data (e.g., unprocessed text or
images)or features? In either case, please provide a description.

Multiple-choice questions with manual semantic annotations. Please refer to
Fig. 3 for more details.

Is any information missing from individual instances? If so, please provide a
description, explaining why this information is missing (e.g., because it was
unavailable). This does not include intentionally removed information, but
might include, e.g., redacted text.

Individual instances within the Xiezhi-Meta, Xiezhi-Train, Xiezhi-Speciality,
and Xiezhi-Interdiscipline datasets have undergone manual verification,
resulting in a complete data set with no instances missing information.
However, Xiezhi-All, containing all raw data from open-source exams, has a
limited amount of data missing since it has been annotated only through
automatic annotation models.

Are relationships between individual instances made explicit (e.g., users’
movie ratings, social network links)? If so, please describe how these
relationships are made explicit.

Individual instances are extracted from all kinds of Chinese Examinations, so
these instances may share the same level of difficulty or discipline labels to
some extent.

Are there recommended data splits (e.g., training, development/validation,
testing)?

Yes, we propose Xiezhi-Train of experiments in demonstration setting, and
Xiezhi-Meta for model training, we also propose Xiezhi-Interdiscipline and
Xiezhi-Specility for model testing.

Are there any errors, sources of noise, or redundancies in the dataset? If so,
please provide a description.

The dataset is carefully reviewed and checked automatically and manually with
a strict quality control protocol, so there will be few error or noise.

Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g.,websites, tweets, other datasets)?

The dataset is self-contained.

Does the dataset contain data that might be considered confidential (e.g.,
data that is protected by legal privilege or by doctor– patient
confidentiality, data that includes the content of individuals’ non-public
communications)? If so, please provide a description. No.

Does the dataset contain data that, if viewed directly, might be offensive,
insulting, threatening, or might otherwise cause anxiety? If so, please
describe why

The dataset is carefully reviewed, all data might be offensive, insulting,
threatening or might otherwise cause anxiety are excluded automatically and
manually.

Does the dataset identify any subpopulations (e.g., by age, gender)?

No.

Is it possible to identify individuals (i.e., one or more natural persons),
either directly or indirectly (i.e., in combination with other data) from the
dataset?

No, all the instances in Xiezhi are questions about domain knowledge, it is
impossible to identify individuals from the dataset.

Does the dataset contain data that might be considered sensitive in any way
(e.g., data that reveals race or ethnic origins, sexual orientations,
religious beliefs, political opinions or union memberships, or locations;
financial or health data; biometric or genetic data; forms of government
identification, such as social security numbers; criminal history)?

No.

### Collection Process

How was the data associated with each instance acquired? Was the data directly
observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey
responses), or indirectly inferred/derived from other data (e.g., part-of-
speech tags, model-based guesses for age or language)? If data was reported by
subjects or indirectly inferred/derived from other data, was the data
validated/verified? If so, please describe how.

All the questions are extracted from all kinds of Chinese examinations online
or generated from Chinese academic surveys, so the answers, options and
questions are all directly observable.

What mechanisms or procedures were used to collect the data (e.g., hardware
apparatus or sensor, manual human curation, software program, software API)?
How were these mechanisms or procedures validated?

We used a Python crawler to grab questions from an online site and used
ChatGPT to automatically annotate the questions.

Who was involved in the data collection has process (e.g., students,
crowdworkers, contractors) and how were they compensated (e.g., how much were
crowdworkers paid)?

The questions are collected and generated by the authors, and are manual
annoated by 10 Chinese graduates. All the annotators are paid above the local
minimum wage.

Over what timeframe was the data collected? Does this timeframe match the
creation time frame of the data associated with the instances (e.g., recent
crawl of old news articles)? If not, please describe the timeframe in which
the data associated with the instances was created.

The data was collected from March 2023 until May 2023, it match the created
time of our github.

Were any ethical review processes conducted (e.g., by an institutional review
board)?

We undertake a serious ethical review, please refer to Appendix Bias, Ethical
Problems and Social Impact for more details.

Did you collect the data from the individuals in question directly, or obtain
it via third parties or other sources (e.g., websites)?

We do not collect the data from any individuals.

Did the individuals in question consent to the collection and use of their
data?

N/A

If consent was obtained, were the consenting individuals provided with a
mechanism to revoke their consent in the future or for certain uses?

N/A

Has an analysis of the potential impact of the dataset and its use on data
subjects (e.g., a data protection impact analysis) been conducted?

N/A

### Preprocessing / Cleaning / Labeling

Was any preprocessing/cleaning/labeling of the data done (e.g., discretization
or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction,
removal of instances, processing of missing values)?

Yes, our preprocessing process is illustrated in Fig. 4.

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data
(e.g., to support unanticipated future uses)? If so, please provide a link or
other access point to the “raw” data

Xiezhi-All saved all the raw data, the github url will be released after the
reviewing of AAAI-2024.

Is the software used to preprocess/clean/label the instances available? If so,
please provide a link or other access point.

No.

### Uses

Has the dataset been used for any tasks already?

No, not yet.

Is there a repository that links to any or all papers or systems that use the
dataset?

No.

What (other) tasks could the dataset be used for?

The dataset could be used for instruction-tuning for boosting LLMs performance
in domain text understanding.

Is there anything about the composition of the dataset or the way it was
collected and preprocessed/cleaned/labeled that might impact future uses? For
example, is there anything that a dataset consumer might need to know to avoid
uses that could result in unfair treatment of individuals or groups (e.g.,
stereotyping, quality of service issues) or other risks or harms (e.g., legal
risks, financial harms)? If so, please provide a description. Is there
anything a dataset consumer could do to mitigate these risks or harms?

No.

Are there tasks for which the dataset should not be used? If so, please
provide a description.

No.

### Distribution

Will the dataset be distributed to third parties outside of the entity (e.g.,
company, institution, organization) on behalf of which the dataset was
created?

Yes, probably.

How will the dataset will be distributed (e.g., tarball on website, API,
GitHub)? Does the dataset have a digital object identififier (DOI)?

The dataset will be distributed at
https://github.com/MikeGu721/XiezhiBenchmark

When will the dataset be distributed?

The up-to-date dataset has been uploaded now.

Will the dataset be distributed under a copyright or other intellectual
property (IP) license, and/or under applicable terms of use (ToU)? If so,
please describe this license and/or ToU, and provide a link or other access
point to, or otherwise reproduce, any relevant licensing terms or ToU, as well
as any fees associated with these restrictions.

This dataset is released under the CC BY-SA 4.0 license for general research
purposes.

Have any third parties imposed IP-based or other restrictions on the data
associated with the instances? If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any relevant
licensing terms, as well as any fees associated with these restrictions.

No.

Do any export controls or other regulatory restrictions apply to the dataset
or to individual instances? If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any
supporting documentation.

No.

### Maintance

Who is supporting / hosting / maintaining the dataset?

Cognitive Understanding Group of Knowledge Works Research Laboratory from
Fudan University, China.

How can the owner / curator / manager of the dataset be contacted (e.g., email
address)?

The emails of the first authors are {zhgu22<EMAIL_ADDRESS>and the
corrsponding authors are {hwfeng<EMAIL_ADDRESS>
Is there an erratum?

No.

Will the dataset be updated (e.g., to correct labeling errors, add new
instances, delete instances)?

According to our current plans, the dataset will be updated twice a year.

If the dataset relates to people, are there applicable limits on the retention
of the data associated with the instances (e.g., were the individuals in
question told that their data would be retained for a fixed period of time and
then deleted)?

N/A.

Will older versions of the dataset continue to be supported / hosted /
maintained?

Yes, older version is still maintained and updated and will be communicated to
users via Github.

If others want to extend/augment/build on/contribute to the dataset, is there
a mechanism for them to do so?

We welcome people from all walks of life to use our data, and as we mentioned
in Sec Discussion, the rapid development of big models requires more
challenging datasets, more evaluation metrics and evaluation methods, and we
are more than willing to contribute Xiezhi to this great goal.

### Check List

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope?

[Yes]

    2. (b)

Did you describe the limitations of your work?

[Yes] We describe the limitations in Section Discuss.

    3. (c)

Did you discuss any potential negative societal impacts of your work?

[Yes] We describe the limitations in Appendix Bias, Ethical Problems and
Social Impact.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them?

[Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results?

[N/A]

    2. (b)

Did you include complete proofs of all theoretical results?

[N/A]

  3. 3.

If you ran experiments (e.g. for benchmarks)…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)?

[Yes] We describe our experiment setting in Section Experiments and the random
seed in Appendix Detail Hyper-parameters.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)?

[Yes] We describe the training of proposed annotation in Appendix Auto
Annotator.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)?

[No] The experiments consume substantial GPU resources; therefore, to ensure
consistency, all our experiments were conducted under the random seed of 42.

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)?

[Yes] Provided in Appendix Detail Hyper-parameters.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators?

[Yes] We use baseline models from Huggingface’s Transformers, and detail
describe and cite them in Appendix Models.

    2. (b)

Did you mention the license of the assets?

[Yes]

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL?

[Yes] We provide details in Appendix Models.

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?

[N/A]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content?

[N/A]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable?

[Yes] The instructions are included in Appendix Manual Annotation.

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable?

[N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation?

[Yes]