# Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics
to Reasoning-Memorization Correlations

Jiaxing Sun$1$, Weiquan Huang$2$11footnotemark: 1, Jiang Wu$3$11footnotemark:
1 , Chenya Gu$3$, Wei Li$3$,
Songyang Zhang$3$, Hang Yan$3$, Conghui He$3$
$1$ Wuhan University
$2$ Tongji University
$3$ Shanghai AI Laboratory
<EMAIL_ADDRESS>Equal contribution. Project lead. Corresponding author.

###### Abstract

We introduce CHARM, the first benchmark for comprehensively and in-depth
evaluating the commonsense reasoning ability of large language models (LLMs)
in Chinese, which covers both globally known and Chinese-specific commonsense.
We evaluated 7 English and 12 Chinese-oriented LLMs on CHARM, employing 5
representative prompt strategies for improving LLMs’ reasoning ability, such
as Chain-of-Thought. Our findings indicate that the LLM’s language orientation
and the task’s domain influence the effectiveness of the prompt strategy,
which enriches previous research findings. We built closely-interconnected
reasoning and memorization tasks, and found that some LLMs struggle with
memorizing Chinese commonsense, affecting their reasoning ability, while
others show differences in reasoning despite similar memorization performance.
We also evaluated the LLMs’ memorization-independent reasoning abilities and
analyzed the typical errors. Our study precisely identified the LLMs’
strengths and weaknesses, providing the clear direction for optimization. It
can also serve as a reference for studies in other fields. We will release
CHARM at https://github.com/opendatalab/CHARM.

Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to
Reasoning-Memorization Correlations

Jiaxing Sun$1$††thanks:  Equal contribution., Weiquan Huang$2$11footnotemark:
1, Jiang Wu$3$11footnotemark: 1 ††thanks:  Project lead., Chenya Gu$3$, Wei
Li$3$, Songyang Zhang$3$, Hang Yan$3$, Conghui He$3$††thanks:  Corresponding
author. $1$ Wuhan University $2$ Tongji University $3$ Shanghai AI Laboratory
<EMAIL_ADDRESS>

Figure 1: Construction of CHARM. CHARM encompasses both global and Chinese-
specific commonsense. CHARM consists closely-interconnected reasoning and
memorization tasks.

##  1 Introduction

Commonsense reasoning is important for the enhancement of the large language
models (LLMs) Bommasani et al. (2021); Achiam et al. (2023) towards artificial
general intelligence (AGI) Davis and Marcus (2015), therefore requires
thorough evaluations. Numerous benchmarks evaluate the commonsense reasoning
of LLMs, but most are English-based, limiting non-English evaluations Davis
(2023). This paper focuses on assessing LLMs’ commonsense reasoning in a
Chinese context. Currently, some commonsense reasoning benchmarks in Chinese
are simply English translations Conneau et al. (2018); Ponti et al. (2020);
Lin et al. (2022), which overlooks unique Chinese cultural, linguistic,
regional, and historical aspects. These factors matter when Chinese users use
the LLM, hence should be included in benchmarks. To effectively tackle this,
we introduce CHARM, the benchmark designed to thoroughly and in-depth assess
the abilities of LLMs in Chinese commonsense reasoning. It covers two domains:
globally accepted commonsense (global domain) and Chinese-specific commonsense
(Chinese domain). The latter includes 7 aspects: History (H), Traditional
Culture and Arts (CA), Daily Life and Customs (LC), Entertainment (E), Public
Figures (F), Geography (G), and Chinese Language (L). Therefore CHARM allows a
thorough evaluation of LLMs’ reasoning in a Chinese context.

Prompt strategies like Chain of Thought (CoT) Wei et al. (2022) can
significantly improve LLMs’ reasoning performance Wang et al. (2022, 2023b).
Particularly, as the training corpus of LLMs is primarily in English Touvron
et al. (2023a), studies Shi et al. (2022); Huang et al. (2023a); Zhang et al.
(2023a) have shown that for non-English reasoning tasks, some LLMs perform
better when reasoning in English than the native language. We evaluate 7
English and 12 Chinese-oriented LLMs on CHARM, employing 5 representative
prompt strategies. The result shows that prompt strategies’ effectiveness
depends on the LLMs’ orientation and the benchmark task’s domain, which
enriches prior research and guides performance assessment and strategy choice
for non-English LLMs.

LLMs’ commonsense reasoning relies on memorization. Exploring the correlation
between memorization and reasoning offers insights into LLMs, aiding deeper
understanding and suggesting ways to enhance these abilitiesBian et al.
(2023). Some benchmarks Yu et al. (2023); Wang et al. (2023a); Fei et al.
(2023) aid the research of memorization-reasoning relationships by incorporate
tasks for assessing knowledge memorization and application (like reasoning).
However, they often use the existing and disparate datasets for different
tasks, resulting in a lack of intrinsic connections between these tasks. For
instance, the question $Q_{rea}$ tests the LLM’s reasoning with the knowledge
piece $K$. However, in memorization tasks, there probably is not any matching
questions to determine if the LLM has effectively memorized $K$. Hence, if the
LLM fails on $Q_{rea}$, it’s unclear whether due to poor reasoning or
forgetfulness of $K$. This results in the disjointed evaluation of
memorization and reasoning, failing to uncover their intrinsic links. To
address this limitation, we selected suitable reasoning tasks from CHARM’s
Chinese domain, and built related memorization questions for each reasoning
question (see Figure 1). This design produces the closely-interconnected
reasoning and memorization tasks, therefore allows for not only the concurrent
evaluation of the two abilities, but also the assessment of memorization-
independent reasoning, providing the clear guidance for the LLMs’ enhancement.

The contributions of this paper are as follows:

  * •

We present CHARM, the first benchmark for comprehensively evaluating the LLMs’
commonsense reasoning ability in Chinese, by encompassing not only the global
but also the Chinese-specific commonsense.

  * •

We evaluated the representative prompt strategies on CHARM. Results showed
that LLMs’ orientation and the task’s domain affect prompt strategy
performance, which enriches previous research findings.

  * •

In CHARM, we built closely-interconnected reasoning and memorization tasks in
Chinese commonsense domain, allowing for in-depth understanding the
correlation between these abilities and precisely identifying the LLMs’
strengths and weaknesses. The design approach could serve as the reference for
other fields.

##  2 Related Work

Commonsense Reasoning Benchmarks  There are lots of English commonsense
reasoning benchmarks or datasets Davis (2023), some of which have been
translated to various languages, including Chinese Conneau et al. (2018);
Ponti et al. (2020); Lin et al. (2022). There are also some native Chinese
benchmarks related to commonsense reasoning. LogiQA Liu et al. (2020, 2023),
sourced from the Chinese civil servant exam questions, evaluates the LLMs’
reading comprehension and logical reasoning across various knowledge types,
including basic commonsense knowledges. CLUE Xu et al. (2020), the
comprehensive Chinese language understanding benchmark, includes the Natural
Language Inference (NLI) tasks, which require the LLMs’ commonsense reasoning
abilities. CMMLU Li et al. (2023) is the comprehensive Chinese benchmark that
encompasses exam questions from the 67 subjects. It includes tasks that test
elementary commonsense and logical reasoning abilities. CORECODE Shi et al.
(2023) is the dataset designed for commonsense reasoning and conflict
detection in Chinese, presented in the dialogue format.

Prompt Strategy  Prompt strategies such as CoT Wei et al. (2022) can
effectively boost the reasoning capabilities of LLMs Wang et al. (2022,
2023b). Notably, as the LLM training corpus is primarily in English Touvron et
al. (2023a), research reveals that for reasoning tasks in non-English
languages, some LLMs exhibit superior performance when reasoning in English as
opposed to the native language Shi et al. (2022); Zhang et al. (2023a); Huang
et al. (2023a). Kim et al. (2023) proposed a novel cross-language transfer
prompt method, which uses both the source and target languages to construct
examples.

Benchmarks on Correlations of Memorization and Reasoning  There are benchmarks
which assess both the knowledge memorization and reasoning capabilities of the
LLMs within specific domains. For instance, KoLA Yu et al. (2023), with its
focus on world knowledge, includes tasks related to knowledge memorization and
application (reasoning). SeaEval Wang et al. (2023a), emphasizing cross-
language consistency and multicultural reasoning, involves tasks for cultural
understanding and complex reasoning. There are also benchmarks aimed at
specialized fields, like LawBench Fei et al. (2023), which include tasks for
both memorization and application.

##  3 CHARM

CHARM is built for comprehensive and in-depth evaluation of LLMs in Chinese
commonsense reasoning and revealing the intrinsic correlation between
memorization and reasoning. Therefore, CHARM covers two domains, global and
Chinese, using carefully selected tasks for comprehensive coverage. In
addition, we chose reasoning tasks and constructed the closely-tied
memorization tasks. The construction and main features of CHARM are in Figure
1. The detailed composition of CHARM is in Table 1.

###  3.1 Commonsense Domain

Global commonsense domain consists of universally understood commonsense. It
covers objects and aspects of modern life that an individual should be aware
of. It includes foundational knowledge that someone with a basic modern
education is expected to know. When it involves individuals, they are globally
recognized figures.

Chinese commonsense domain encompasses Chinese-specific elements. We
categoried them into 7 aspects:

History (H) includes important events and figures in Chinese history, China’s
dynasties, and other basic facts and shared knowledge about the history of
China.

Traditional Culture and Arts (CA) encompasses Chinese traditional cultural
arts, literary works, and traditional lifestyles.

Daily Life and Customs (LC) includes modern Chinese daily routines, clothing,
food, housing, transportation festivals and so on.

Entertainment (E) includes the movies, television programs, music, and other
entertainments in modern Chinese daily life.

Public Figures (F) encompasses the public figures well-known in Chinese
society.

Geography (G) includes China’s geographical distribution, natural landscapes,
and characteristic regional cultures.

Chinese Language (L) includes the fundamentals of the Chinese language, such
as Chinese characters, idioms and so on.

For the two domains, especially for the above 7 aspects, we collected
corresponding entities, forming the lists as shown in the Figure 1 and 5. Most
of the entities are selected from Gaokao Bench111Gaokao Bench is the
collection of China’s university entrance exam questions, which contributes to
all the 7 aspects.Zhang et al. (2023b), Douban222https://www.douban.com/ is
the popular user-centric cultural review platform in China, which mainly
contributes to the Entertainment aspect., Hupu333https://www.hupu.com/ is the
large sports community popular in China, which mainly contributes to the
Public Figures aspect.. Some entities are collected with the help of searching
engines. We only collect the entities that are well-known in China. These
entities are then used to create the commonsense reasoning questions, which
belong to the corresponding domain and aspect.

Task Type | Task | Domain | Chinese aspects | Construction | Question Type | # Question
---|---|---|---|---|---|---
Reasoning | Anachronisms Judgment (AJ) | Chinese | H, AC, LC, F | [H] | 2-option MCQ | 150
global | - | [T][H] | 2-option MCQ | 150
Time Understanding (TU) | Chinese | H, AC, LC | [H] | 4-option MCQ | 100
global | - | [T] | 5or6-option MCQ | 100
Sequence Understanding (SqU) | Chinese | H, CA, LC, G, L | [H] | 4-option MCQ | 100
global | - | [T][H] | 4-option MCQ | 100
Movie and Music Recommendation (MMR) | Chinese | E | [H] | 4-option MCQ | 50
global | - | [T] | 4-option MCQ | 50
Sport Understanding (SpU) | Chinese | F | [H] | 2-option MCQ | 200
global | - | [H] | 2-option MCQ | 200
Natural Language Inference (NLI) | Chinese |  G, E, L, | [S][H] | 3-option MCQ | 100
global | - | [S] | 3-option MCQ | 100
Reading Comprehension (RC) | Chinese | all 7 aspects | [S] | 4-option MCQ | 200
global | - | [S] | 4-option MCQ | 200
Memorization | Anachronisms Judgment (AJ) | Chinese | H,AC,LC,F | [H] | Free-form QA | 150
Time Understanding (TU) | Chinese | H, AC, LC | [H] | Free-form QA | 83
Movie and Music Recommendation (MMR) | Chinese | E | [H] | Free-form QA | 399
Sport Understanding (SpU) | Chinese | F | [H] | Free-form QA | 127
Table 1: Overview of CHARM. The question numbers of reasoning and memorization
tasks are 1800 and 759.

###  3.2 Reasoning Tasks

When designing the reasoning tasks in CHARM, we bear two criteria in mind.
First, the tasks should span both commonsense domains, particularly the 7
Chinese aspects. Second, the global and Chinese tasks should have identical
types and settings, differing only in their commonsense domains. From the
existing English commonsense reasoning datasets Davis (2023); Suzgun et al.
(2022), we selected the following 7 tasks:

Anachronisms Judgment (AJ) necessitates the LLM to identify anachronisms in
provided sentences. This involves the LLM understanding the era associated
with well-known figures, items, and events to facilitate commonsense-based
reasoning. Global domain questions are the mix of
translations444https://github.com/google/BIG-
bench/tree/main/bigbench/benchmark_tasks/anachronisms and handcrafted, while
all Chinese domain questions are handcrafted.

Time Understanding (TU) requires the LLM infers a time (including year, date,
moment, etc.) based on a given context, which necessitates the fundamental
understanding of time-related commonsense and the capacity for mathematical
reasoning. All question in the global domain are
translations555https://github.com/google/BIG-
bench/tree/main/bigbench/benchmark_tasks/date_understanding and all in Chinese
domain are handcrafted.

Sequence Understanding (SqU) requires the LLM sort a series of entities
according to time or occurrence order, requiring logical reasoning based on
commonsense. The global domain questions are the mix of
translations666https://github.com/google/BIG-
bench/tree/main/bigbench/benchmark_tasks/logical_sequence and handcrafted;
while all in the Chinese domain are handcrafted.

Movie and Music Recommendation (MMR) necessitates the LLM identifies the most
similar matches to a variety of movies or music tracks, requiring the
understanding of these popular movies and music and ability to identify their
commonalities. All global domain questions are
translations777https://github.com/google/BIG-
bench/tree/main/bigbench/benchmark_tasks/movie_recommendation, and all in the
Chinese domain are handcrafted.

Sport Understanding (SpU)  involves a crafted sentence with a famous athlete
and a common sport action, and the LLM must assess its credibility, which
demands understanding of sports and commonsense judgement. The questions in
both domains are handcrafted, refering Suzgun et al. (2022) .

Natural Language Inference (NLI) gives two sentences and asks the LLM to
classify their relationship as entailment, contradiction, or neutral,
necessitating commonsense-based reasoning and judgement. All global domain
questions are selected from CLUE Xu et al. (2020); the questions in the
Chinese domain are partly from CLUE, and partly handcrafted.

Reading Comprehension (RC) gives a passage of text, and the LLM is required to
reason based on it. All question in both domains are selected from LogiQA Liu
et al. (2020, 2023).

The chosen tasks adequately cover both the commonsense domains, particularly
the 7 aspects of the Chinese commonsense domain. This coverage enables a
comprehensive assessment of LLMs’ commonsense reasoning ability in Chinese.
Moreover, the Chinese-domain questions can be created following the similar
types and settings as their global counterpart, facilitating the cleaner
comparison of the LLMs’ performance across the domains.

All questions in the CHARM reasoning tasks are multiple-choice questions.
Detailed information is in Table 1. Question examples of the tasks are in
Figure 6 in Appendix B. We use regular expressions to extract the preferred
choice from the generation of the LLMs Huang et al. (2023b); Li et al. (2023)
and use accuracy as the metric.

###  3.3 Memorization Task

Shared commonsense knowledge pieces serve as a link between reasoning and
memorization questions. From the 7 reasoning tasks, we chose 4 that can be
readily associated in this manner, AJ, TU, MMR, SpU, refered as the
Memorization-Reasoning-Interconnected (MRI) tasks, and built the reltated
memorization questions.

Construction  We first extract the commonsense knowledge pieces related to the
entities in the corresponding reasoning questions. Information about each
entity was collected to the degree sufficient to address the associated
reasoning question, and then used to formulate the memorization questions.
Following the Knowledge Memorization task in KoLA Yu et al. (2023), we choose
free-form QA instead of multiple-choice or true/false questions, which can
effectively avoid the impact of randomness. All memorization questions are
handcrafted. Question examples are in Figure 7 in Appendix C. The averaged
number of related memorization questions for each reasoning question are shown
in Table 2.

Task | AJ | TU | SpU | MMR
---|---|---|---|---
Avg. # related memorization questions | 2.1 | 3.2 | 2.0 | 8.0
Table 2: Averaged number of related memorization questions per reasoning
question for each task.

Judgement and Metric  For the MMR task, we use a rule-based matching method
for evaluation; for the other three tasks, we use Chat-GPT for judgement. We
use accuracy as the metric.

Figure 2: Distribution of CHARM construction.

###  3.4 Construction and Quality Assurance

The question construction of CHARM involves the following three methods:

Handcraft [H]  The questions are created mannualy by the authors, based on the
entities in §3.1. For reasoning questions in Chinese commonsense domain, we
usually refer to the global counter-part and construct them with the same
design idea and style, as shown in Figure 6.

Translate [T]  We translate the existing English questions and only keep those
consistent with the global commonsense domain.

Select [S]  We select commonsense reasoning questions from existing Chinese
datasets.

Detailed information of the construction of all tasks are shown in Table 1.
The distribution of the three methods is shown in Figure 2.

We ensure the quality through these steps: First, two annotators independently
verify each question, whether handcrafted or translated. Second, annotators
are given only questions, not our answers, and use external resources to
respond. They can also flag problematic questions. Last, a question passes if
both annotators’ answers match ours; if not, it’s reviewed and modified by
multiple authors.

##  4 Experimental Setup

###  4.1 Language Models

We evaluated the currently commonly used LLMs, which can be divided into two
categories: (1) 7 English LLMs, including GPT series Achiam et al. (2023),
LLaMA-2 Touvron et al. (2023b), and
Vicuna888https://huggingface.co/lmsys/vicuna-7b-v1.5-16k and
https://huggingface.co/lmsys/vicuna-13b-v1.5-16k. (2) 12 Chinese-oriented
LLMs, including ChatGLM3999https://huggingface.co/THUDM/chatglm3-6b-32k,
Baichuan2 Yang et al. (2023), InternLM2 Team (2023),
Yi101010https://github.com/01-ai/Yi, DeepSeekBi et al. (2024) and Qwen Bai et
al. (2023). For open-source models, we choose the chat version instead of the
base version. For closed-source models, we use the official API111111We used
the gpt-3.5-turbo-1106 version for GPT-3.5 and the gpt-4-1106-preview version
for GPT-4.. Detailed information is in Table 3.

Models | Open Source? | Model Size | Primary Language
---|---|---|---
LLaMA-2 | Yes | 7B, 13B, 70B | English
Vicuna | Yes | 7B, 13B | English
GPT-3.5 | No | undisclosed | English
GPT-4 | No | undisclosed | English
ChatGLM3 | Yes | 6B | Chinese
Baichuan2 | Yes | 7B, 13B | Chinese
InternLM2 | Yes | 7B, 20B | Chinese
Yi | Yes | 6B, 34B | Chinese
DeepSeek | Yes | 7B, 67B | Chinese
Qwen | Yes | 7B, 14B, 72B | Chinese
Table 3: LLMs evaluated in our experiments LLM | Chinese Commonsense Domain | Global Commonsense Domain
---|---|---
AJ |  TU |  SqU |  MMR |  SpU |  NLI |  RC | Avg. |  AJ |  TU |  SqU |  MMR |  SpU |  NLI |  RC | Avg.
GPT-3.5-1106 |  85.33 |  39.0 |  65.0 |  42.0 |  80.5 |  61.0 |  50.5 | 60.48 |  90.00 |  94.0 |  87.0 |  46.0 |  88.5 |  66.0 |  49.5 | 74.43
GPT-4-1106 |  96.67 |  60.0 |  85.0 |  74.0 |  86.0 |  77.0 |  62.5 | 77.31 |  95.33 |  98.0 |  97.0 |  66.0 |  90.0 |  72.0 |  72.0 | 84.33
LLaMA-2-7B |  51.33 |  36.0 |  11.0 |  14.0 |  49.5 |  52.0 |  8.0 | 31.69 |  62.67 |  17.0 |  14.0 |  16.0 |  49.5 |  22.0 |  13.0 | 27.74
LLaMA-2-13B |  56.00 |  33.0 |  38.0 |  30.0 |  58.0 |  47.0 |  38.0 | 42.86 |  66.67 |  24.0 |  39.0 |  50.0 |  53.5 |  57.0 |  33.5 | 46.24
LLaMA-2-70B |  57.33 |  37.0 |  52.0 |  32.0 |  55.0 |  56.0 |  41.5 | 47.26 |  72.67 |  84.0 |  73.0 |  42.0 |  64.0 |  61.0 |  41.5 | 62.60
Vicuna-7B-v1.5 |  52.00 |  29.0 |  34.0 |  32.0 |  51.0 |  49.0 |  35.5 | 40.36 |  45.33 |  64.0 |  37.0 |  26.0 |  58.5 |  52.0 |  32.5 | 45.05
Vicuna-13B-v1.5 |  64.67 |  25.0 |  32.0 |  26.0 |  51.5 |  60.0 |  40.0 | 42.74 |  72.67 |  74.0 |  41.0 |  50.0 |  68.0 |  61.0 |  36.0 | 57.52
ChatGLM3-6B |  66.00 |  40.0 |  59.0 |  38.0 |  77.0 |  72.0 |  37.5 | 55.64 |  34.00 |  69.0 |  71.0 |  28.0 |  75.5 |  63.0 |  34.0 | 53.50
Baichuan2-7B |  76.00 |  41.0 |  48.0 |  38.0 |  72.0 |  53.0 |  49.5 | 53.93 |  55.33 |  65.0 |  54.0 |  26.0 |  60.5 |  59.0 |  29.0 | 49.83
Baichuan2-13B |  85.33 |  40.0 |  48.0 |  46.0 |  72.5 |  66.0 |  51.5 | 58.48 |  77.33 |  74.0 |  58.0 |  40.0 |  71.0 |  61.0 |  39.0 | 60.05
InternLM2-7B |  88.00 |  38.0 |  58.0 |  38.0 |  76.0 |  81.0 |  25.0 | 57.71 |  74.67 |  80.0 |  62.0 |  20.0 |  78.0 |  76.0 |  23.5 | 59.17
InternLM2-20B |  88.00 |  55.0 |  54.0 |  44.0 |  74.5 |  80.0 |  23.0 | 59.79 |  82.67 |  83.0 |  61.0 |  14.0 |  74.5 |  72.0 |  27.0 | 59.17
Yi-6B |  70.67 |  32.0 |  47.0 |  32.0 |  75.0 |  50.0 |  42.0 | 49.81 |  79.33 |  63.0 |  43.0 |  14.0 |  70.5 |  57.0 |  33.5 | 51.48
Yi-34B |  96.00 |  55.0 |  89.0 |  76.0 |  88.5 |  72.0 |  51.5 | 75.43 |  88.67 |  92.0 |  87.0 |  56.0 |  89.0 |  70.0 |  47.5 | 75.74
DeepSeek-7B |  81.33 |  34.0 |  50.0 |  50.0 |  79.5 |  57.0 |  31.5 | 54.76 |  68.00 |  76.0 |  47.0 |  50.0 |  72.5 |  59.0 |  32.5 | 57.86
DeepSeek-67B |  96.67 |  57.0 |  83.0 |  92.0 |  87.5 |  77.0 |  34.5 | 75.38 |  90.00 |  95.0 |  86.0 |  22.0 |  88.0 |  73.0 |  39.0 | 70.43
Qwen-7B |  70.67 |  38.0 |  55.0 |  48.0 |  71.0 |  57.0 |  49.5 | 55.60 |  74.67 |  78.0 |  69.0 |  50.0 |  72.5 |  55.0 |  36.0 | 62.17
Qwen-14B |  87.33 |  54.0 |  77.0 |  60.0 |  82.5 |  66.0 |  55.0 | 68.83 |  84.00 |  83.0 |  83.0 |  44.0 |  84.5 |  71.0 |  40.0 | 69.93
Qwen-72B |  98.00 |  59.0 |  91.0 |  84.0 |  86.5 |  84.0 |  67.5 | 81.43 |  94.00 |  92.0 |  93.0 |  64.0 |  93.0 |  71.0 |  63.5 | 81.50
Table 4: Accuracy on CHARM reasoning tasks. We selected the empirically
optimal prompt strategy: XLT for English LLMs and ZH-CoT for Chinese-oriented
LLMs. Bold and underline represent the first and second place respectively.
Detailed results are in Table 8 and Table 9 of Appendix E.

###  4.2 Prompt Strategies

We selected 5 commonly used prompt strategies, and assessed the performance of
the 19 LLMs on CHARM reasoning task:

Direct: The LLM does not perform intermediate reasoning and directly predicts
the answer.

ZH-CoT: The LLM conducts intermediate reasoning Wei et al. (2022) in Chinese
before producing the answer.

EN-CoT: The reasoning process of CoT is in English for the Chinese
questionsShi et al. (2022).

Translate-EN: We use the DeepL api121212https://www.deepl.com/translator to
translate our benchmark into English, and then use English CoT for reasoning
Zhang et al. (2023a).

XLT: The template prompt Huang et al. (2023a) is used to change the original
question into an English request, solve it step by step, and finally format
the answer for output .

The examples for each prompt strategy are in Figure 8 in Appendix D. For all
prompt strategies, we use the 3-shot setting.

##  5 Results and Analysis

###  5.1 Integrated Reasoning Performance

We show the performance of the 19 LLMs on CHARM reasoning tasks in Table 4. We
only choose one representative prompt strategy: XLT for English LLMs and ZH-
CoT for Chinese-oriented LLMs, which is based on our empirical conclusion in
§5.2. The LLMs’ performance on the 7 aspects of the Chinese commonsense domain
are shown in Table 10 in Appendix F.

Commonsense Domain We found that the LLMs exhibit inconsistent performance in
the global and Chinese commonsense domains. The rankings of the English LLMs
dropped in the Chinese domain compared to the global domain. For instance,
GPT4 ranks first in the global domain, but in the Chinese domain, Qwen-72B
outperforms all, pushing GPT4 to the second. In the Chinese domain, the
performance of LLaMA-2-70B is even worse than many Chinese-oriented LLMs in
the 6B-7B size range.

However, in the global domain, LLaMA-2-70B is better than all Chinese-oriented
LLMs up to 20B in size, except for Qwen-14B.

###  5.2 Prompt Strategy Selection

| Prompt | Avg. all LLMs | Avg. CN-LLMs | Avg. EN-LLMs
---|---|---|---|---
Avg. | Direct | 46.28 | 48.41 | 42.64
all | ZH-CoT | 56.66 | 62.40 | 46.81
domains | EN-CoT | 54.46 | 58.19 | 48.06
| Translate-EN | 53.88 | 55.51 | 51.07
| XLT | 56.81 | 59.09 | 52.90
Avg. | Direct | 45.43 | 47.76 | 41.44
Chinese | ZH-CoT | 56.35 | 62.23 | 46.26
domains | EN-CoT | 52.06 | 56.36 | 44.68
| Translate-EN | 47.25 | 47.82 | 46.27
| XLT | 53.80 | 56.63 | 48.96
Avg. | Direct | 47.13 | 49.05 | 43.85
global | ZH-CoT | 56.96 | 62.57 | 47.35
domains | EN-CoT | 56.85 | 60.01 | 51.44
| Translate-EN | 60.50 | 63.20 | 55.87
| XLT | 59.82 | 61.56 | 56.84
Table 5: Averaged accuracy on CHARM reasoning tasks. “CN-LLMs” means the 12
Chinese-oriented LLMs, “EN-LLMs” means the 7 English LLMs.

We tested the combinations of the 19 LLMs and the 5 prompt strategies in CHARM
reasoning tasks. Detailed results are in Table 8 and Table 9 in Appendix E. To
draw some empirical conclusions, we analyze along the following two
dimensions:

• Dim1: global or Chinese commonsense domain.

• Dim2: English or Chinese-oriented LLMs.

We average the $19\times 5$ LLM-prompt combinations along the above two
dimensions, and the obtained results are in the Table 5. From the LLM
dimension, it’s clear that various LLMs prefer different prompt strategies:
XLT consistently excels for English LLMs among the 5 strategies, while for
Chinese-oriented LLMs, despite some complexity, ZH-CoT generally performs
best. From the commonsense domain dimension, strategies that use English for
reasoning (like XLT, Translate-EN, etc.) are suitable for the global domain;
however, ZH-CoT generally performs better in the Chinese domain.

The conclusion here differs from previous studies Shi et al. (2022); Huang et
al. (2023a), which suggested that employing English for non-English reasoning
tasks was more effective than using the native language. These previous
studies had limitations, focusing only on English LLMs and neglecting the many
Chinese-oriented LLMs developed since 2023. Furthermore, most benchmarks in
these studies were merely translations from English, lacking unique cultural
and linguistic characteristics in Chinese. The empirical findings with CHARM
in this paper have somewhat alleviated those limitations, leading to more
current and comprehensive conclusions, and of course still have the
limitations, which are detailed in section Limitations.

###  5.3 Integrated Reasoning vs Memorization

We evaluated the correlation between the integrated reasoning and the
memorization on the MRI tasks, as mentioned in §3.2. The average performance
of the LLMs on the 4 MRI tasks is in Figure 3. Detailed performance on each
task is in Figure 9 in Appendix G.1.

As shown in Figure 3, the 19 LLMs can be roughly divided into the three types:

• Type I: Low memorization and low integrated reasoning ability. We found that
apart from OpenAI’s GPT series, all other English LLMs belong to this type.

• Type II: High memorization and medium integrated reasoning ability. GPT3.5
and all Chinese-oriented LLMs below 30B belong to this type. It’s worth noting
that some LLMs have high memorization performance, but relatively poor
integrated reasoning ability.

• Type III: Ultra-high memorization and high integrated reasoning ability.
This category includes GPT4 and the three Chinese-oriented LLMs that exceed a
size of 30B.

Figure 3: Averaged accuracy across the 4 MRI tasks in the Chinese commonsense
domain.

The above findings offer clear guidance for the enhancement of LLMs’ reasoning
abilities in Chinese commonsense domain. For Type I, the limitation lies in
the memorization. For Type II, there should be further improvement in
understanding, applying knowledge, and reasoning abilities.

In addition, we also evaluated the correlation between memorization and
integrated reasoning during the LLM pre-training process, details can be found
in Figure 10 in Appendix G.2.

The results clearly indicate that strong memorization is the foundation of
integrated reasoning. Weak memorization leads to poor reasoning, as shown by
Type I LLMs. Also, factors other than memorization can cause significant
differences in reasoning abilities among LLMs with similar memorization.

###  5.4 Memorization-Independent Reasoning

Rank | Integrated Reasoning | Memorization-independent Reasoning
---|---|---
FRMM | MIB
1 | DeepSeek-67B | Yi-34B (↑3) | GPT-4-1106 (↑2)
2 | Qwen-72B | DeepSeek-67B (↓1) | Yi-34B (↑2)
3 | GPT-4-1106 | GPT-4-1106 (-) | Qwen-72B (↓1)
4 | Yi-34B | Qwen-72B (↓2) | DeepSeek-67B (↓3)
5 | Qwen-14B | GPT-3.5-1106 (↑2) | GPT-3.5-1106 (↑2)
6 | InternLM2-20B | Qwen-14B (↓1) | Qwen-14B (↓1)
7 | GPT-3.5-1106 | InternLM2-20B (↓1) | InternLM2-20B (↓1)
8 | InternLM2-7B | InternLM2-7B (-) | InternLM2-7B (-)
9 | DeepSeek-7B | Baichuan2-13B (↑1) | Baichuan2-13B (↑1)
10 | Baichuan2-13B | DeepSeek-7B (↓1) | DeepSeek-7B (↓1)
11 | Baichuan2-7B | Yi-6B (↑3) | Baichuan2-7B (-)
12 | ChatGLM3-6B | ChatGLM3-6B(-) | ChatGLM3-6B(-)
13 | Qwen-7B | Baichuan2-7B (↓2) | Qwen-7B (-)
14 | Yi-6B | Qwen-7B (↓1) | Yi-6B (-)
15 | LLaMA-2-70B | LLaMA-2-13B (↑1) | LLaMA-2-13B (↑1)
16 | LLaMA-2-13B | LLaMA-2-70B (↓1) | LLaMA-2-70B (↓1)
17 | Vicuna-13B-v1.5 | Vicuna-13B-v1.5 (-) | Vicuna-13B-v1.5 (-)
18 | Vicuna-7B-v1.5 | LLaMA-2-7B (↑1) | Vicuna-7B-v1.5(-)
19 | LLaMA-2-7B | Vicuna-7B-v1.5 (↓1) | LLaMA-2-7B (-)
Table 6: Leaderboard on the MRI tasks. We propose two methods, i.e. FRMM and
MIC, to compare the LLMs’ memorization-independent reasoning, as detailed in
Appendix H. The arrows and numbers in brackets in the last two columns
indicate changes in ranking order relative to the second column.

We propose two methods to compare the LLMs’ memorization-independent reasoning
on the MRI tasks. The detail of the Filtering Reasoning Questions based on
Mono-LLM-Memorization (FRMM) is in Appendix H.1, and Memorization-Independent
Battles among LLMs (MIB) in Appendix H.2. The results are in Table 6. When
comparing the leaderboards for integrated and memorization-independent
reasoning, Type III LLMs rank at the forefront and the Type I rank at the end
in all leaderboards. There is a slight variation in the ranking order within
the three types of LLMs.

For the in-depth analysis, we chose Vicuna-13B, Qwen-7B, Qwen-72B as
representatives for Type I, II, and III LLMs, and filtered out the reasoning
questions in the MRI tasks, only keeping those with correct answers to the
related memorization questions, same as the FRMM in Appendix H.1. This ensures
the LLM has sufficiently memorized the commonsense knowledge required for the
retained reasoning questions, thereby minimizing the impact of memorization on
reasoning. There are totally 500 reasoning questions in the 4 MRI tasks, and
the numbers of the retained are 106, 323 and 402 for Vicuna-13B, Qwen-7B and
Qwen-72B respectively, as shown in Table 7.

Error Types  If the LLMs provided incorrect answers for the retained reasoning
questions, these errors can be referred to as memorization-independent
reasoning errors. We conducted the manual review and analysis of their
reasoning process, and classified the errors into 4 main categories.

• Understanding Error  In this case, the LLM was unable to accurately
comprehend the question, including misunderstanding the content, ignoring or
even modifying important information in the premise, and failing to grasp the
core query of the question.

• Knowledge Error  The LLM incorporated inaccurate knowledge during the
reasoning process. It’s important to highlight that the knowledge pieces
related to the reasoning question were previously examined in the related
memorization questions, which the LLM answered correctly. However, the LLM
output incorrect information during the reasoning phase.

• Logical Error  The LLM made logical reasoning errors, such as mathematical
reasoning errors, inability to reach the correct conclusion based on
sufficient information, or reaching the correct conclusion but outputing the
wrong option.

• Other Errors  These are other scattered, relatively rare types of errors.

We show examples of each type of errors in Figure 12 in Appendix I. The
distribution of these error types are shown in Figure 4.

Discussions  Obviously, the majority of errors are from logical reasoning
mistakes and knowledge inaccuracies, which further provides the directions for
LLMs’ enhancement. As for knowledge errors, prior studies Bian et al. (2023);
Allen-Zhu and Li (2023) have indicated that the way LLMs remember and master
knowledge is a relatively complex topic. Simple memorization doesn’t guarantee
that LLMs can apply this knowledge accurately and skillfully during the
reasoning process.

Models |  LLM type | # Original | # Retained | # Incorrect
---|---|---|---|---
Vicuna-13B-v1.5 |  Type I | 500 | 106 | 54
Qwen-7B |  Type II | 500 | 323 | 117
Qwen-72B |  Type III | 500 | 402 | 63
Table 7: Memorization-based filtering of reasoning questions. “Incorrect”
means the incorrectly answered questions among the retained. Figure 4:
Distribution of the memorization-independent reasoning errors

##  6 Conclusion

This paper introduces CHARM, the first benchmark designed to comprehensively
and thoroughly evaluate LLMs’ commonsense reasoning in Chinese. CHARM
encompasses two counterpart commonsense domains, global and Chinese-specific,
with the carefully selected tasks. We evaluated the representative prompt
strategies for improving LLMs’ reasoning ability, and the empirical findings
significantly enhances and supplements the conclusions of previous studies.
CHARM comprises closely-interconnected reasoning and memorization tasks,
helping to reveal the intrinsic correlation between memorization and reasoning
of LLMs. We have evaluated the strengths and weaknesses of different LLMs and
conducted the detailed analysis of memorization-independent reasoning
abilities. We hope that CHARM’s approach to studying the correlations between
memoriztion and reasoning can serve as a reference for similar research in
other fields.

## Limitations

This study conducted tests on combinations of the 19 LLMs and the 5 prompt
strategies, resulting in empirical conclusions. However, many existing LLMs
and prompt strategies have not yet been tested. Furthermore, the best prompt
strategy for the commonsense reasoning task for the LLMs, particularly in
Chinese or other non-English languages, is not static and should progress with
LLM technology. This is influenced by three elements: (1) The new prompt
strategies are continuously proposed, which are likely more effective. (2) The
new LLMs may have different prompt strategy preference, or be less sensitive
to prompt. (3) For other non-English languages with high resources, future
LLMs would be continuously evolving and updating, and necessitate ongoing
updates in evaluation.

The automation of the construction and evaluation of CHARM needs further
improvement, including the following: (1) Most of the questions in CHARM
Chinese domain are manually constructed by the author. This limits the number
of benchmark questions and the range of knowledge pieces covered. (2)
Regarding memorization-independent reasoning, we chose only 3 LLMs as
representative and manually categorized the types of errors within CHARM. In
future research, we could employ robust LLMs, like GPT4, for automated error
classification and statistical analysis.

## Ethical Consideration

This work involves human annotation. We have provided appropriate compensation
for all annotators. The total cost of annotation for the project is about 2.2k
RMB. For all annotators, we explicitly inform them about the use of the data
and require them to ensure that the questions included in CHARM do not involve
any social bias, ethical issues or privacy concerns during the annotation
process.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Allen-Zhu and Li (2023) Zeyuan Allen-Zhu and Yuanzhi Li. 2023.  Physics of language models: Part 3.2, knowledge manipulation.  _arXiv preprint arXiv:2309.14402_.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_.
  * Bi et al. (2024) Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. 2024.  Deepseek llm: Scaling open-source language models with longtermism.  _arXiv preprint arXiv:2401.02954_.
  * Bian et al. (2023) Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, and Ben He. 2023.  Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models.  _arXiv preprint arXiv:2303.16421_.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_.
  * Conneau et al. (2018) Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  Xnli: Evaluating cross-lingual sentence representations.  _arXiv preprint arXiv:1809.05053_.
  * Davis (2023) Ernest Davis. 2023.  Benchmarks for automated commonsense reasoning: A survey.  _arXiv preprint arXiv:2302.04752_.
  * Davis and Marcus (2015) Ernest Davis and Gary Marcus. 2015.  Commonsense reasoning and commonsense knowledge in artificial intelligence.  _Communications of the ACM_ , 58(9):92–103.
  * Fei et al. (2023) Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. 2023.  Lawbench: Benchmarking legal knowledge of large language models.  _arXiv preprint arXiv:2309.16289_.
  * Huang et al. (2023a) Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023a.  Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting.  _arXiv preprint arXiv:2305.07004_.
  * Huang et al. (2023b) Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. 2023b.  C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.  _arXiv preprint arXiv:2305.08322_.
  * Kim et al. (2023) Sunkyoung Kim, Dayeon Ki, Yireun Kim, and Jinsik Lee. 2023.  Boosting cross-lingual transferability in multilingual models via in-context learning.  _arXiv preprint arXiv:2305.15233_.
  * Li et al. (2023) Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023.  Cmmlu: Measuring massive multitask language understanding in chinese.  _arXiv preprint arXiv:2306.09212_.
  * Lin et al. (2022) Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2022.  Few-shot learning with multilingual generative language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9019–9052.
  * Liu et al. (2023) Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng, Nan Duan, Ming Zhou, and Yue Zhang. 2023.  Logiqa 2.0—an improved dataset for logical reasoning in natural language understanding.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_.
  * Liu et al. (2020) Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020.  Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.  _arXiv preprint arXiv:2007.08124_.
  * Ponti et al. (2020) Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020.  Xcopa: A multilingual dataset for causal commonsense reasoning.  _arXiv preprint arXiv:2005.00333_.
  * Shi et al. (2023) Dan Shi, Chaobin You, Jiantao Huang, Taihao Li, and Deyi Xiong. 2023.  Corecode: A common sense annotated dialogue dataset with benchmark tasks for chinese large language models.  _arXiv preprint arXiv:2312.12853_.
  * Shi et al. (2022) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022.  Language models are multilingual chain-of-thought reasoners.  _arXiv preprint arXiv:2210.03057_.
  * Suzgun et al. (2022) Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022.  Challenging big-bench tasks and whether chain-of-thought can solve them.  _arXiv preprint arXiv:2210.09261_.
  * Team (2023) InternLM Team. 2023.  Internlm: A multilingual language model with progressively enhanced capabilities.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2023a) Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao, Yang Ding, Ai Ti Aw, and Nancy F Chen. 2023a.  Seaeval for multilingual foundation models: From cross-lingual alignment to cultural reasoning.  _arXiv preprint arXiv:2309.04766_.
  * Wang et al. (2023b) Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023b.  Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models.  _arXiv preprint arXiv:2305.04091_.
  * Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Xu et al. (2020) Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. 2020.  Clue: A chinese language understanding evaluation benchmark.  _arXiv preprint arXiv:2004.05986_.
  * Yang et al. (2023) Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023.  Baichuan 2: Open large-scale language models.  _arXiv preprint arXiv:2309.10305_.
  * Yu et al. (2023) Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, et al. 2023.  Kola: Carefully benchmarking world knowledge of large language models.  _arXiv preprint arXiv:2306.09296_.
  * Zhang et al. (2023a) Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. 2023a.  Don’t trust chatgpt when your question is not in english: A study of multilingual abilities and types of llms.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 7915–7927.
  * Zhang et al. (2023b) Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. 2023b.  Evaluating the performance of large language models on gaokao benchmark.  _arXiv preprint arXiv:2305.12474_.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _arXiv preprint arXiv:2306.05685_.

##  Appendix A Entity and Question Examples of the 7 Chinese Commonsense
Aspects

Figure 5 shows the number of questions and partial entities of each Chinese
commonsense aspect we propose, as well as corresponding question examples.

##  Appendix B Question Examples of the Reasoning Task in CHARM

Figure 6 shows the question examples of the 7 reasoning tasks in CHARM,
including both Chinese and global domains.

##  Appendix C Question Examples of the Memorization Task in CHARM

Figure 7 shows the questions examples of the memorization tasks in CHARM.

##  Appendix D Examples of Prompt Strategies

Figure 8 shows the examples of the 5 prompt strategies.

##  Appendix E Detailed Evaluation Results of 19 LLMs with 5 Prompt Strageties
on Reasoning Tasks

We conducted a detailed evaluation of 19 different LLMs using 5 distinct
prompt strategies. Table 8 and Table 9 respectively display the performance of
various prompt strategies on 7 reasoning tasks in the CHARM’s Chinese
commonsense domain and global commonsense domain.

##  Appendix F Performance of LLMs on Chinese Commonsense Knowledge Aspects

Table 10 displays the performance of LLMs in the 7 Chinese commonsense
aspects. We only choose one representative prompt strategy: XLT for English
LLMs and ZH-CoT for Chinese LLMs, which is based on our empirical conclusion
in §5.2.

##  Appendix G Correlation of Memorization and Integrated Reasoning

###  G.1 Detailed Correlations of Memorization and Integrated Reasoning on the
4 MRI Tasks

The detailed performances of the 19 LLMs on the 4 MRI tasks are in Figure 9.

###  G.2 Correlation of Memorization and Integrated Reasoning throughout the
LLM pretraining

We tested the intermediate checkpoint models of Baichuan2 and DeepSeek on the
memorization and reasoning questions on the 4 MRI tasks. The results are shown
in Figure 10.

With the increase in the number of tokens during the training process, the
model’s memorization ability quickly reaches a high level (in fact, there is
no particularly obvious difference between the results of the first checkpoint
and the final results). This is because the knowledge involved in our task
setting is the most basic commonsense, and thus widely and abundantly exists
in various Chinese training corpora.

However, the improvement in reasoning performance significantly lags behind
memorization. This is because to complete a reasoning task in CHARM is
actually a multi-step process, requiring memorization of relevant knowledge,
understanding of the question, use of knowledge for reasoning, and answering
according to the requirements of the question and the demonstration of few-
shot examples, etc. If an error occurs in any step of the above complex
process, the reasoning task will fail.

##  Appendix H Leaderboard of Memorization-Independent Reasoning

It is non-trivial to acquire and compare the memorization-independent
reasoning abilities of the LLMs. Intuitively, we can filter the reasoning
questions by only retaining those whose related memorization questions are all
correctedly answered by every LLMs. This approach ensures that each LLM has
memorized the commonsense knowledge necessary for the retained reasoning
questions. However, when we apply this process to all the 19 LLMs, only 28
reasoning questions remain out of the original 500 in the MRI tasks, which is
obviously insufficient in number and lacks diversity, thereby introducing a
high degree of uncertainty due to randomness.

Therefore, we propose two slightly more complex methods, one called Filtering
Reasoning Questions based on Mono-LLM-Memorization (FRMM), the other is
Memorization-Independent Battles among LLMs (MIB).

###  H.1 Filtering Reasoning Questions based on Mono-LLM-Memorization (FRMM)

This method is relatively simple, but has some flaws to a certain extent. For
each LLM, we select reasoning questions based on its performance in
memorization tasks: only retaining reasoning questions for which all related
memorization questions are answered correctly. It’s clear that, after
individual filtration, different LLMs will retain different reasoning
questions, and even differ in the number of retained reasoning questions, as
shown in the “# retained” column in Table 11.

Then, we calculate the accuracy of the retained reasoning questions for each
LLM, and the results are shown in the “Retained Acc” column in Table 11. The
LLMs are then ranked based on the accuracy, producing the leaderboard shown in
the penultimate column of Table 6.

As mentioned above, while this method can reflect the memorization-independent
reasoning abilities of LLMs to some extent, its drawback lies in that the
denominator used in calculating the final ranking accuracy differs for
different LLMs.

To overcome this, we have proposed the MIB method.

###  H.2 Memorization-Independent Battles among LLMs (MIB)

To overcome the shortcomings of the FRMM method, we refer to the pairwise
battle method adopted in LLM evaluation Zheng et al. (2023). By tallying each
LLM’s performance in a “round-robin” tournament of pairwise match-ups, we rank
the performance of the LLMs.

Specifically, we select two LLMs at a time and filter the MRI task’s reasoning
questions based on the performance of these two LLMs in memorization tasks. We
only retain the reasoning questions whose related memorization questions are
correctly answered by both LLMs. In this way, the two LLMs are battled under
fair conditions. We then calculate the accuracy of these two LLMs on the
retained reasoning questions separately, and compute the difference in
accuracy as the battle score between the two models.

As shown in Figure 11, the element $E_{ij}$ represents the accuracy of LLM $i$
minus the accuracy of LLM $j$ during the battle between the two LLMs. For a
total of 19 LLMs, we average each model’s scores from the 18 battles they
participated in as their final score, as shown in Table 12.

Finally, we rank the LLMs based on these scores to produce the leaderboard
shown in the last column of Table 6.

##  Appendix I Memorization-Independent Reasoning Errors

LLMs can answer memorization questions correctly, but they make mistakes when
it comes to reasoning problems composed of these knowledge points. Figure 12
shows the examples of three memorization-independent reasoning errors of LLMs.

Figure 5: Entity and question examples of the commonsense aspects. Figure 6: Examples of the reasoning tasks in CHARM. Figure 7: Examples of the memorization tasks in CHARM. Figure 8: Examples of prompt strategies. LLM | Prompt | AJ | TU | SqU | MMR | SpU | NLI | RC | Avg.
---|---|---|---|---|---|---|---|---|---
GPT-3.5-1106 | Direct | 32.00 | 33.0 | 46.0 | 48.0 | 60.0 | 62.0 | 52.5 | 47.64
| ZH-CoT | 17.33 | 49.0 | 69.0 | 46.0 | 69.0 | 66.0 | 55.0 | 53.05
| EN-CoT | 50.67 | 39.0 | 64.0 | 44.0 | 80.5 | 59.0 | 53.5 | 55.81
| Translate-EN | 87.33 | 44.0 | 54.0 | 40.0 | 76.5 | 64.0 | 49.5 | 59.33
| XLT | 85.33 | 39.0 | 65.0 | 42.0 | 80.5 | 61.0 | 50.5 | 60.48
GPT-4-1106 | Direct | 94.00 | 50.0 | 81.0 | 72.0 | 73.0 | 86.0 | 63.0 | 74.14
| ZH-CoT | 99.33 | 67.0 | 81.0 | 68.0 | 85.0 | 83.0 | 34.0 | 73.90
| EN-CoT | 98.00 | 65.0 | 79.0 | 70.0 | 83.0 | 83.0 | 50.5 | 75.50
| Translate-EN | 96.00 | 43.0 | 70.0 | 42.0 | 75.5 | 81.0 | 53.5 | 65.86
| XLT | 96.67 | 60.0 | 85.0 | 74.0 | 86.0 | 77.0 | 62.5 | 77.31
LLaMA-2-7B | Direct | 47.33 | 21.0 | 20.0 | 30.0 | 49.0 | 6.0 | 31.5 | 29.26
| ZH-CoT | 52.67 | 34.0 | 23.0 | 22.0 | 51.0 | 34.0 | 29.5 | 35.17
| EN-CoT | 44.67 | 34.0 | 20.0 | 4.0 | 50.5 | 40.0 | 26.5 | 31.38
| Translate-EN | 10.00 | 32.0 | 20.0 | 20.0 | 39.0 | 40.0 | 20.0 | 25.86
| XLT | 51.33 | 36.0 | 11.0 | 14.0 | 49.5 | 52.0 | 8.0 | 31.69
LLaMA-2-13B | Direct | 47.33 | 30.0 | 34.0 | 30.0 | 49.0 | 33.0 | 31.5 | 36.40
| ZH-CoT | 52.67 | 38.0 | 38.0 | 30.0 | 53.0 | 34.0 | 23.0 | 38.38
| EN-CoT | 52.67 | 34.0 | 38.0 | 2.0 | 49.0 | 1.0 | 35.5 | 30.31
| Translate-EN | 53.33 | 34.0 | 20.0 | 10.0 | 62.0 | 9.0 | 31.0 | 31.33
| XLT | 56.00 | 33.0 | 38.0 | 30.0 | 58.0 | 47.0 | 38.0 | 42.86
LLaMA-2-70B | Direct | 47.33 | 23.0 | 25.0 | 26.0 | 49.5 | 44.0 | 33.0 | 35.40
| ZH-CoT | 51.33 | 31.0 | 35.0 | 24.0 | 51.5 | 46.0 | 37.0 | 39.40
| EN-CoT | 55.33 | 27.0 | 31.0 | 26.0 | 59.0 | 56.0 | 41.0 | 42.19
| Translate-EN | 72.67 | 26.0 | 46.0 | 42.0 | 66.5 | 65.0 | 48.0 | 52.31
| XLT | 57.33 | 37.0 | 52.0 | 32.0 | 55.0 | 56.0 | 41.5 | 47.26
Vicuna-7B-v1.5 | Direct | 52.67 | 25.0 | 30.0 | 16.0 | 14.5 | 19.0 | 21.5 | 25.52
| ZH-CoT | 56.00 | 25.0 | 39.0 | 26.0 | 49.5 | 56.0 | 33.0 | 40.64
| EN-CoT | 53.33 | 28.0 | 26.0 | 18.0 | 40.5 | 55.0 | 40.5 | 37.33
| Translate-EN | 66.67 | 25.0 | 31.0 | 30.0 | 60.0 | 57.0 | 33.0 | 43.24
| XLT | 52.00 | 29.0 | 34.0 | 32.0 | 51.0 | 49.0 | 35.5 | 40.36
Vicuna-13B-v1.5 | Direct | 47.33 | 34.0 | 34.0 | 30.0 | 48.5 | 52.0 | 46.0 | 41.69
| ZH-CoT | 64.00 | 33.0 | 31.0 | 34.0 | 51.0 | 54.0 | 36.0 | 43.29
| EN-CoT | 62.67 | 30.0 | 32.0 | 24.0 | 50.0 | 50.0 | 33.0 | 40.24
| Translate-EN | 69.33 | 23.0 | 26.0 | 32.0 | 63.0 | 68.0 | 40.5 | 45.98
| XLT | 64.67 | 25.0 | 32.0 | 26.0 | 51.5 | 60.0 | 40.0 | 42.74
ChatGLM3-6B | Direct | 44.67 | 35.0 | 48.0 | 46.0 | 58.5 | 73.0 | 60.5 | 52.24
| ZH-CoT | 66.00 | 40.0 | 59.0 | 38.0 | 77.0 | 72.0 | 37.5 | 55.64
| EN-CoT | 55.33 | 36.0 | 58.0 | 36.0 | 76.0 | 75.0 | 32.5 | 52.69
| Translate-EN | 57.33 | 24.0 | 39.0 | 40.0 | 53.5 | 71.0 | 46.0 | 47.26
| XLT | 48.00 | 44.0 | 43.0 | 36.0 | 68.5 | 65.0 | 41.0 | 49.36
Baichuan2-7B | Direct | 44.67 | 31.0 | 37.0 | 24.0 | 59.0 | 35.0 | 56.0 | 40.95
| ZH-CoT | 76.00 | 41.0 | 48.0 | 38.0 | 72.0 | 53.0 | 49.5 | 53.93
| EN-CoT | 55.33 | 36.0 | 44.0 | 30.0 | 69.0 | 53.0 | 41.0 | 46.90
| Translate-EN | 55.33 | 21.0 | 26.0 | 24.0 | 41.5 | 52.0 | 33.5 | 36.19
| XLT | 56.00 | 35.0 | 44.0 | 28.0 | 68.0 | 48.0 | 44.0 | 46.14
Baichuan2-13B | Direct | 59.33 | 23.0 | 42.0 | 30.0 | 67.0 | 36.0 | 23.5 | 40.12
| ZH-CoT | 85.33 | 40.0 | 48.0 | 46.0 | 72.5 | 66.0 | 51.5 | 58.48
| EN-CoT | 72.00 | 40.0 | 50.0 | 34.0 | 68.0 | 64.0 | 42.5 | 52.93
| Translate-EN | 65.33 | 38.0 | 40.0 | 32.0 | 58.5 | 49.0 | 36.0 | 45.55
| XLT | 61.33 | 33.0 | 38.0 | 34.0 | 67.0 | 61.0 | 46.0 | 48.62
InternLM2-7B | Direct | 22.00 | 33.0 | 62.0 | 54.0 | 58.5 | 84.0 | 66.0 | 54.21
| ZH-CoT | 88.00 | 38.0 | 58.0 | 38.0 | 76.0 | 81.0 | 25.0 | 57.71
| EN-CoT | 77.33 | 42.0 | 59.0 | 38.0 | 73.0 | 78.0 | 38.5 | 57.98
| Translate-EN | 73.33 | 28.0 | 45.0 | 36.0 | 56.5 | 72.0 | 46.0 | 50.98
| XLT | 80.67 | 38.0 | 60.0 | 30.0 | 66.5 | 72.0 | 53.5 | 57.24
InternLM2-20B | Direct | 14.00 | 42.0 | 61.0 | 50.0 | 39.5 | 54.0 | 46.5 | 43.86
| ZH-CoT | 88.00 | 55.0 | 54.0 | 44.0 | 74.5 | 80.0 | 23.0 | 59.79
| EN-CoT | 68.67 | 40.0 | 48.0 | 42.0 | 67.0 | 68.0 | 25.0 | 51.24
| Translate-EN | 80.67 | 34.0 | 54.0 | 36.0 | 53.5 | 71.0 | 53.0 | 54.60
| XLT | 85.33 | 36.0 | 71.0 | 42.0 | 64.5 | 68.0 | 58.0 | 60.69
Yi-6B | Direct | 14.67 | 17.0 | 20.0 | 30.0 | 48.0 | 19.0 | 35.5 | 26.31
| ZH-CoT | 70.67 | 32.0 | 47.0 | 32.0 | 75.0 | 50.0 | 42.0 | 49.81
| EN-CoT | 58.67 | 18.0 | 34.0 | 30.0 | 58.0 | 52.0 | 48.5 | 42.74
| Translate-EN | 56.00 | 25.0 | 23.0 | 26.0 | 24.0 | 15.0 | 23.0 | 27.43
| XLT | 54.67 | 36.0 | 35.0 | 28.0 | 68.5 | 56.0 | 43.0 | 45.88
Yi-34B | Direct | 89.33 | 28.0 | 85.0 | 56.0 | 70.0 | 51.0 | 68.0 | 63.90
| ZH-CoT | 96.00 | 55.0 | 89.0 | 76.0 | 88.5 | 72.0 | 51.5 | 75.43
| EN-CoT | 90.00 | 42.0 | 78.0 | 66.0 | 84.5 | 67.0 | 50.0 | 68.21
| Translate-EN | 86.67 | 28.0 | 55.0 | 34.0 | 71.0 | 65.0 | 41.5 | 54.45
| XLT | 92.00 | 48.0 | 87.0 | 72.0 | 84.0 | 66.0 | 61.0 | 72.86
DeepSeek-7B | Direct | 46.67 | 27.0 | 21.0 | 30.0 | 48.0 | 40.0 | 27.5 | 34.31
| ZH-CoT | 81.33 | 34.0 | 50.0 | 50.0 | 79.5 | 57.0 | 31.5 | 54.76
| EN-CoT | 72.00 | 33.0 | 33.0 | 24.0 | 73.0 | 47.0 | 35.5 | 45.36
| Translate-EN | 68.67 | 18.0 | 28.0 | 36.0 | 59.0 | 72.0 | 40.5 | 46.02
| XLT | 55.33 | 32.0 | 39.0 | 36.0 | 51.0 | 37.0 | 35.0 | 40.76
DeepSeek-67B | Direct | 22.67 | 48.0 | 33.0 | 28.0 | 12.5 | 53.0 | 39.5 | 33.81
| ZH-CoT | 96.67 | 57.0 | 83.0 | 92.0 | 87.5 | 77.0 | 34.5 | 75.38
| EN-CoT | 84.00 | 53.0 | 73.0 | 58.0 | 82.0 | 73.0 | 35.0 | 65.43
| Translate-EN | 94.67 | 45.0 | 60.0 | 38.0 | 67.5 | 69.0 | 40.0 | 59.17
| XLT | 95.33 | 59.0 | 80.0 | 66.0 | 87.5 | 76.0 | 54.5 | 74.05
Qwen-7B | Direct | 50.67 | 28.0 | 41.0 | 50.0 | 60.5 | 56.0 | 55.5 | 48.81
| ZH-CoT | 70.67 | 38.0 | 55.0 | 48.0 | 71.0 | 57.0 | 49.5 | 55.60
| EN-CoT | 58.67 | 40.0 | 48.0 | 32.0 | 68.5 | 58.0 | 43.0 | 49.74
| Translate-EN | 62.67 | 23.0 | 31.0 | 26.0 | 60.0 | 54.0 | 45.0 | 43.10
| XLT | 62.67 | 26.0 | 47.0 | 40.0 | 63.0 | 50.0 | 50.5 | 48.45
Qwen-14B | Direct | 63.33 | 28.0 | 69.0 | 60.0 | 73.0 | 59.0 | 59.0 | 58.76
| ZH-CoT | 87.33 | 54.0 | 77.0 | 60.0 | 82.5 | 66.0 | 55.0 | 68.83
| EN-CoT | 85.33 | 48.0 | 68.0 | 56.0 | 77.5 | 76.0 | 53.0 | 66.26
| Translate-EN | 78.00 | 29.0 | 45.0 | 22.0 | 60.5 | 63.0 | 46.5 | 49.14
| XLT | 83.33 | 36.0 | 66.0 | 56.0 | 76.5 | 65.0 | 50.0 | 61.83
Qwen-72B | Direct | 90.67 | 36.0 | 85.0 | 78.0 | 80.0 | 84.0 | 77.5 | 75.88
| ZH-CoT | 98.00 | 59.0 | 91.0 | 84.0 | 86.5 | 84.0 | 67.5 | 81.43
| EN-CoT | 95.33 | 55.0 | 88.0 | 64.0 | 86.0 | 78.0 | 72.0 | 76.90
| Translate-EN | 92.00 | 37.0 | 53.0 | 32.0 | 73.0 | 76.0 | 57.0 | 60.00
| XLT | 93.33 | 50.0 | 86.0 | 70.0 | 83.0 | 75.0 | 58.5 | 73.69
Table 8: Accuracy of reasoning tasks in the Chinese commonsense domain of CHARM. LLM | Prompt | AJ | TU | SqU | MMR | SpU | NLI | RC | Avg.
---|---|---|---|---|---|---|---|---|---
GPT-3.5-1106 | Direct | 41.33 | 58.0 | 59.0 | 42.0 | 61.0 | 64.0 | 45.0 | 52.90
| ZH-CoT | 25.33 | 89.0 | 90.0 | 28.0 | 77.5 | 73.0 | 48.5 | 61.62
| EN-CoT | 59.33 | 85.0 | 84.0 | 42.0 | 86.0 | 68.0 | 50.0 | 67.76
| Translate-EN | 88.67 | 86.0 | 80.0 | 48.0 | 83.5 | 65.0 | 58.0 | 72.74
| XLT | 90.00 | 94.0 | 87.0 | 46.0 | 88.5 | 66.0 | 49.5 | 74.43
GPT-4-1106 | Direct | 90.67 | 83.0 | 92.0 | 70.0 | 88.0 | 78.0 | 74.0 | 82.24
| ZH-CoT | 92.67 | 100.0 | 90.0 | 34.0 | 88.0 | 76.0 | 50.5 | 75.88
| EN-CoT | 95.33 | 97.0 | 97.0 | 52.0 | 89.5 | 70.0 | 61.5 | 80.33
| Translate-EN | 92.67 | 93.0 | 91.0 | 48.0 | 74.0 | 68.0 | 62.0 | 75.52
| XLT | 95.33 | 98.0 | 97.0 | 66.0 | 90.0 | 72.0 | 72.0 | 84.33
LLaMA-2-7B | Direct | 43.33 | 20.0 | 20.0 | 28.0 | 51.5 | 18.0 | 20.5 | 28.76
| ZH-CoT | 51.33 | 18.0 | 22.0 | 26.0 | 50.0 | 31.0 | 22.5 | 31.55
| EN-CoT | 56.67 | 20.0 | 22.0 | 22.0 | 51.5 | 10.0 | 27.0 | 29.88
| Translate-EN | 4.67 | 20.0 | 35.0 | 2.0 | 51.5 | 38.0 | 28.0 | 25.60
| XLT | 62.67 | 17.0 | 14.0 | 16.0 | 49.5 | 22.0 | 13.0 | 27.74
LLaMA-2-13B | Direct | 53.33 | 21.0 | 35.0 | 24.0 | 51.0 | 35.0 | 20.5 | 34.26
| ZH-CoT | 54.67 | 15.0 | 35.0 | 14.0 | 51.5 | 32.0 | 27.0 | 32.74
| EN-CoT | 52.67 | 19.0 | 35.0 | 16.0 | 51.5 | 35.0 | 38.0 | 35.31
| Translate-EN | 34.67 | 19.0 | 35.0 | 20.0 | 57.0 | 37.0 | 28.0 | 32.95
| XLT | 66.67 | 24.0 | 39.0 | 50.0 | 53.5 | 57.0 | 33.5 | 46.24
LLaMA-2-70B | Direct | 48.67 | 33.0 | 33.0 | 30.0 | 50.0 | 58.0 | 20.5 | 39.02
| ZH-CoT | 46.00 | 20.0 | 35.0 | 32.0 | 51.5 | 38.0 | 35.5 | 36.86
| EN-CoT | 46.00 | 82.0 | 53.0 | 34.0 | 51.5 | 64.0 | 48.5 | 54.14
| Translate-EN | 84.67 | 76.0 | 58.0 | 52.0 | 71.0 | 64.0 | 57.5 | 66.17
| XLT | 72.67 | 84.0 | 73.0 | 42.0 | 64.0 | 61.0 | 41.5 | 62.60
Vicuna-7B-v1.5 | Direct | 15.33 | 22.0 | 33.0 | 12.0 | 49.5 | 22.0 | 16.5 | 24.33
| ZH-CoT | 52.00 | 50.0 | 50.0 | 10.0 | 50.5 | 53.0 | 31.0 | 42.36
| EN-CoT | 49.33 | 45.0 | 44.0 | 16.0 | 51.0 | 23.0 | 31.5 | 37.12
| Translate-EN | 76.00 | 58.0 | 47.0 | 36.0 | 66.5 | 57.0 | 37.5 | 54.00
| XLT | 45.33 | 64.0 | 37.0 | 26.0 | 58.5 | 52.0 | 32.5 | 45.05
Vicuna-13B-v1.5 | Direct | 55.33 | 58.0 | 39.0 | 28.0 | 48.5 | 59.0 | 30.0 | 45.40
| ZH-CoT | 71.33 | 71.0 | 49.0 | 14.0 | 53.0 | 62.0 | 33.0 | 50.48
| EN-CoT | 66.00 | 82.0 | 42.0 | 38.0 | 65.0 | 55.0 | 40.5 | 55.50
| Translate-EN | 84.00 | 66.0 | 60.0 | 66.0 | 71.0 | 60.0 | 42.0 | 64.14
| XLT | 72.67 | 74.0 | 41.0 | 50.0 | 68.0 | 61.0 | 36.0 | 57.52
ChatGLM3-6B | Direct | 44.00 | 33.0 | 57.0 | 42.0 | 63.0 | 80.0 | 38.0 | 51.00
| ZH-CoT | 34.00 | 69.0 | 71.0 | 28.0 | 75.5 | 63.0 | 34.0 | 53.50
| EN-CoT | 41.33 | 65.0 | 63.0 | 24.0 | 60.5 | 70.0 | 34.0 | 51.12
| Translate-EN | 66.67 | 59.0 | 70.0 | 42.0 | 66.5 | 71.0 | 39.5 | 59.24
| XLT | 52.67 | 58.0 | 70.0 | 46.0 | 66.0 | 66.0 | 42.5 | 57.31
Baichuan2-7B | Direct | 46.00 | 20.0 | 47.0 | 8.0 | 58.0 | 35.0 | 38.0 | 36.00
| ZH-CoT | 55.33 | 65.0 | 54.0 | 26.0 | 60.5 | 59.0 | 29.0 | 49.83
| EN-CoT | 44.00 | 64.0 | 49.0 | 20.0 | 58.5 | 56.0 | 31.5 | 46.14
| Translate-EN | 73.33 | 59.0 | 48.0 | 28.0 | 64.0 | 54.0 | 36.5 | 51.83
| XLT | 48.67 | 18.0 | 49.0 | 34.0 | 56.0 | 50.0 | 23.0 | 39.81
Baichuan2-13B | Direct | 64.00 | 17.0 | 55.0 | 20.0 | 58.0 | 37.0 | 23.5 | 39.21
| ZH-CoT | 77.33 | 74.0 | 58.0 | 40.0 | 71.0 | 61.0 | 39.0 | 60.05
| EN-CoT | 78.67 | 70.0 | 55.0 | 30.0 | 57.0 | 66.0 | 37.5 | 56.31
| Translate-EN | 73.33 | 68.0 | 51.0 | 36.0 | 61.5 | 61.0 | 42.0 | 56.12
| XLT | 70.67 | 75.0 | 49.0 | 42.0 | 69.5 | 61.0 | 31.0 | 56.88
InternLM2-7B | Direct | 46.67 | 61.0 | 65.0 | 46.0 | 67.0 | 79.0 | 53.5 | 59.74
| ZH-CoT | 74.67 | 80.0 | 62.0 | 20.0 | 78.0 | 76.0 | 23.5 | 59.17
| EN-CoT | 72.00 | 87.0 | 70.0 | 44.0 | 76.0 | 73.0 | 38.5 | 65.79
| Translate-EN | 70.67 | 81.0 | 75.0 | 60.0 | 78.0 | 73.0 | 48.5 | 69.45
| XLT | 66.00 | 87.0 | 72.0 | 52.0 | 76.5 | 66.0 | 43.5 | 66.14
InternLM2-20B | Direct | 81.33 | 54.0 | 78.0 | 50.0 | 63.5 | 46.0 | 48.0 | 60.12
| ZH-CoT | 82.67 | 83.0 | 61.0 | 14.0 | 74.5 | 72.0 | 27.0 | 59.17
| EN-CoT | 73.33 | 83.0 | 63.0 | 14.0 | 75.0 | 73.0 | 26.5 | 58.26
| Translate-EN | 82.00 | 84.0 | 89.0 | 40.0 | 76.0 | 68.0 | 46.5 | 69.36
| XLT | 87.33 | 84.0 | 80.0 | 70.0 | 79.0 | 70.0 | 47.0 | 73.90
Yi-6B | Direct | 47.33 | 17.0 | 47.0 | 14.0 | 25.0 | 11.0 | 23.0 | 26.33
| ZH-CoT | 79.33 | 63.0 | 43.0 | 14.0 | 70.5 | 57.0 | 33.5 | 51.48
| EN-CoT | 68.67 | 56.0 | 53.0 | 32.0 | 55.5 | 27.0 | 41.0 | 47.60
| Translate-EN | 72.67 | 57.0 | 62.0 | 32.0 | 69.0 | 37.0 | 35.5 | 52.17
| XLT | 54.67 | 44.0 | 60.0 | 62.0 | 70.5 | 59.0 | 41.5 | 55.95
Yi-34B | Direct | 82.67 | 67.0 | 85.0 | 58.0 | 53.5 | 45.0 | 64.0 | 65.02
| ZH-CoT | 88.67 | 92.0 | 87.0 | 56.0 | 89.0 | 70.0 | 47.5 | 75.74
| EN-CoT | 89.33 | 91.0 | 88.0 | 44.0 | 80.0 | 66.0 | 48.0 | 72.33
| Translate-EN | 78.00 | 85.0 | 83.0 | 48.0 | 76.5 | 64.0 | 54.5 | 69.86
| XLT | 88.00 | 88.0 | 86.0 | 70.0 | 93.5 | 60.0 | 58.0 | 77.64
DeepSeek-7B | Direct | 47.33 | 24.0 | 35.0 | 14.0 | 22.0 | 41.0 | 17.5 | 28.69
| ZH-CoT | 68.00 | 76.0 | 47.0 | 50.0 | 72.5 | 59.0 | 32.5 | 57.86
| EN-CoT | 75.33 | 74.0 | 40.0 | 16.0 | 53.5 | 47.0 | 35.5 | 48.76
| Translate-EN | 72.67 | 59.0 | 45.0 | 32.0 | 60.0 | 57.0 | 38.0 | 51.95
| XLT | 58.00 | 28.0 | 38.0 | 16.0 | 51.5 | 35.0 | 29.5 | 36.57
DeepSeek-67B | Direct | 37.33 | 83.0 | 18.0 | 2.0 | 39.5 | 49.0 | 37.0 | 37.98
| ZH-CoT | 90.00 | 95.0 | 86.0 | 22.0 | 88.0 | 73.0 | 39.0 | 70.43
| EN-CoT | 61.33 | 96.0 | 76.0 | 30.0 | 90.5 | 71.0 | 35.0 | 65.69
| Translate-EN | 90.67 | 87.0 | 81.0 | 52.0 | 81.0 | 54.0 | 58.5 | 72.02
| XLT | 86.00 | 93.0 | 72.0 | 60.0 | 93.0 | 64.0 | 46.0 | 73.43
Qwen-7B | Direct | 52.67 | 38.0 | 54.0 | 38.0 | 56.5 | 67.0 | 40.0 | 49.45
| ZH-CoT | 74.67 | 78.0 | 69.0 | 50.0 | 72.5 | 55.0 | 36.0 | 62.17
| EN-CoT | 74.00 | 81.0 | 65.0 | 36.0 | 73.5 | 66.0 | 35.5 | 61.57
| Translate-EN | 73.33 | 71.0 | 65.0 | 46.0 | 70.5 | 66.0 | 41.0 | 61.83
| XLT | 74.67 | 64.0 | 69.0 | 48.0 | 67.0 | 46.0 | 32.0 | 57.24
Qwen-14B | Direct | 70.00 | 58.0 | 82.0 | 36.0 | 78.0 | 55.0 | 47.5 | 60.93
| ZH-CoT | 84.00 | 83.0 | 83.0 | 44.0 | 84.5 | 71.0 | 40.0 | 69.93
| EN-CoT | 86.67 | 82.0 | 81.0 | 44.0 | 79.5 | 66.0 | 42.5 | 68.81
| Translate-EN | 86.67 | 72.0 | 85.0 | 48.0 | 78.0 | 64.0 | 48.5 | 68.88
| XLT | 80.00 | 79.0 | 83.0 | 48.0 | 79.0 | 65.0 | 45.0 | 68.43
Qwen-72B | Direct | 88.00 | 63.0 | 85.0 | 56.0 | 83.5 | 78.0 | 65.5 | 74.14
| ZH-CoT | 94.00 | 92.0 | 93.0 | 64.0 | 93.0 | 71.0 | 63.5 | 81.50
| EN-CoT | 90.00 | 92.0 | 86.0 | 60.0 | 92.5 | 66.0 | 58.0 | 77.79
| Translate-EN | 91.33 | 87.0 | 89.0 | 54.0 | 81.5 | 63.0 | 64.0 | 75.69
| XLT | 92.67 | 70.0 | 91.0 | 66.0 | 91.5 | 66.0 | 50.5 | 75.38
Table 9: Accuracy of reasoning tasks in the global commonsense domain of CHARM. LLM | Prompt | H | CA | LC | E | F | G | L | Avg.
---|---|---|---|---|---|---|---|---|---
GPT-3.5-1106 | XLT | 80.85 | 41.78 | 63.24 | 49.30 | 80.88 | 52.21 | 55.45 | 60.53
GPT-4-1106 | XLT | 92.55 | 58.22 | 86.76 | 73.24 | 86.27 | 71.68 | 67.27 | 76.57
LLaMA-2-7B | XLT | 45.74 | 16.44 | 22.06 | 26.76 | 49.02 | 26.55 | 21.82 | 29.77
LLaMA-2-13B | XLT | 51.06 | 38.36 | 39.71 | 36.62 | 57.84 | 46.02 | 30.91 | 42.93
LLaMA-2-70B | XLT | 55.85 | 38.36 | 51.47 | 39.44 | 55.39 | 43.36 | 49.09 | 47.57
Vicuna-7B-v1.5 | XLT | 49.47 | 32.19 | 33.82 | 38.03 | 50.49 | 44.25 | 32.73 | 40.14
Vicuna-13B-v1.5 | XLT | 56.91 | 38.36 | 25.00 | 36.62 | 52.45 | 50.44 | 36.36 | 42.31
ChatGLM3-6B | ZH-CoT | 64.89 | 37.67 | 54.41 | 49.30 | 75.98 | 53.98 | 48.18 | 54.92
Baichuan2-7B | ZH-CoT | 69.15 | 45.21 | 51.47 | 40.85 | 72.55 | 51.33 | 47.27 | 53.98
Baichuan2-13B | ZH-CoT | 79.26 | 43.84 | 54.41 | 47.89 | 72.55 | 59.29 | 49.09 | 58.05
InternLM2-7B | ZH-CoT | 76.60 | 33.56 | 61.76 | 49.30 | 75.49 | 49.56 | 45.45 | 55.96
InternLM2-20B | ZH-CoT | 78.72 | 40.41 | 47.06 | 54.93 | 74.02 | 48.67 | 49.09 | 56.13
Yi-6B | ZH-CoT | 61.70 | 42.47 | 54.41 | 38.03 | 75.00 | 44.25 | 36.36 | 50.32
Yi-34B | ZH-CoT | 89.36 | 56.16 | 82.35 | 73.24 | 88.73 | 63.72 | 60.91 | 73.50
DeepSeek-7B | ZH-CoT | 70.21 | 37.67 | 42.65 | 56.34 | 79.41 | 38.05 | 44.55 | 52.70
DeepSeek-67B | ZH-CoT | 87.23 | 55.48 | 75.00 | 87.32 | 86.76 | 52.21 | 52.73 | 70.96
Qwen-7B | ZH-CoT | 65.43 | 42.47 | 51.47 | 52.11 | 70.59 | 53.98 | 53.64 | 55.67
Qwen-14B | ZH-CoT | 81.91 | 62.33 | 70.59 | 60.56 | 82.84 | 58.41 | 56.36 | 67.57
Qwen-72B | ZH-CoT | 92.55 | 61.64 | 89.71 | 83.10 | 86.76 | 75.22 | 77.27 | 80.89
Table 10: Accuracy of reasoning questions on the 7 Chinese commonsense aspects of CHARM. Figure 9: Accuracy of reasoning and memorization on the 4 MRI tasks. Figure 10: Averaged accuracy of the intermediate checkpoint models throughout the LLM pretraining across the 4 MRI tasks. LLM | # Original | Original Acc. | # Retained | Retained Acc.
---|---|---|---|---
DeepSeek-67B | 500 | 84.6 | 409 | 87.04
Qwen-72B | 500 | 84.2 | 402 | 84.33
GPT-4-1106 | 500 | 82.8 | 350 | 86.86
Yi-34B | 500 | 82.8 | 355 | 89.86
Qwen-14B | 500 | 76.0 | 331 | 80.97
InternLM2-20B | 500 | 71.6 | 329 | 78.42
GPT-3.5-1106 | 500 | 69.8 | 226 | 81.86
InternLM2-7B | 500 | 68.2 | 282 | 78.37
DeepSeek-7B | 500 | 68.0 | 338 | 75.74
Baichuan2-13B | 500 | 67.2 | 320 | 76.56
Baichuan2-7B | 500 | 63.6 | 337 | 64.69
ChatGLM3-6B | 500 | 62.4 | 258 | 65.12
Qwen-7B | 500 | 62.0 | 323 | 63.78
Yi-6B | 500 | 60.8 | 285 | 68.07
LLaMA-2-70B | 500 | 49.8 | 107 | 53.27
LLaMA-2-13B | 500 | 49.6 | 70 | 55.71
Vicuna-13B-v1.5 | 500 | 47.6 | 106 | 49.06
Vicuna-7B-v1.5 | 500 | 45.0 | 62 | 40.32
LLaMA-2-7B | 500 | 43.8 | 49 | 46.94
Table 11: Filtering Reasoning questions based on Mono-LLM-Memorization (FRMM) on the MRI tasks. LLM | Final Score
---|---
GPT-4-1106 | 21.60
Yi-34B | 19.52
Qwen-72B | 18.43
DeepSeek-67B | 18.25
GPT-3.5-1106 | 12.65
Qwen-14B | 10.80
InternLM2-20B | 10.01
InternLM2-7B | 7.30
Baichuan2-13B | 5.71
DeepSeek-7B | 5.17
Baichuan2-7B | -0.77
ChatGLM3-6B | -3.11
Qwen-7B | -4.04
Yi-6B | -6.22
LLaMA-2-13B | -15.31
LLaMA-2-70B | -22.69
Vicuna-13B-v1.5 | -22.92
Vicuna-7B-v1.5 | -26.25
LLaMA-2-7B | -28.13
Table 12: Final results of the Memorization-Independent Battles among LLMs
(MIB) on the MRI tasks. Figure 11: Results of the Memorization-Independent
Battles among LLMs (MIB) on the MRI tasks. Figure 12: Examples of the 3 types
of memorization-independent reasoning errors of LLMs