2023 1

Tao et al. CUDRT

Corresponding author Dinghao Xi.

# CUDRT: Benchmarking the Detection of Human vs. Large Language Models
Generated Texts

Zhen Tao  Zhiyu Li  Dinghao Xi  Wei Xu  School of Information, Renmin
University of China, Beijing, China  Institute for Advanced Algorithms
Research, Shanghai, China  Department of Digital Economics, Shanghai
University of Finance and Economics, Shanghai, China
<EMAIL_ADDRESS>
(Date Month Year; Date Month Year; Date Month Year)

###### Abstract

[Abstract]The proliferation of large language models (LLMs) has significantly
enhanced text generation capabilities across various industries. However,
these models’ ability to generate human-like text poses substantial challenges
in discerning between human and artificial intelligence (AI) authorship.
Despite the effectiveness of existing AI-generated text detectors, their
development is hindered by the lack of comprehensive, publicly available
benchmarks. Current benchmarks are limited to specific scenarios, such as
question answering and text polishing, and predominantly focus on English
texts, failing to capture the diverse applications and linguistic nuances of
LLMs. To address these limitations, this paper constructs a comprehensive
bilingual benchmark in both Chinese and English to evaluate mainstream AI-
generated text detectors. We categorize LLM text generation into five distinct
operations: Creation, Updating, Deletion, Rewriting, and Translation (CUDRT),
encompassing all current LLMs activities. We also establish a robust benchmark
evaluation framework to support scalable and reproducible experiments. For
each CUDRT category, we have developed extensive datasets to thoroughly assess
detector performance. By employing the latest mainstream LLMs specific to each
language, our datasets provide a thorough evaluation environment. Extensive
experimental results offer critical insights for optimizing AI-generated text
detectors and suggest future research directions to improve detection accuracy
and generalizability across various scenarios. Source code is available at
GitHub111https://github.com/TaoZhen1110/CUDRT_Benchmark.

###### keywords:

AI-generated Text, Large Language Models, LLMs Operations, Text Detection,
Benchmarking

††articletype: Article Type††journal: Journal††volume: 0011footnotetext: *
These authors contributed equally to this work.

##  1 Introduction

The emergence and rapid growth of large language models (LLMs) such as ChatGPT
1, LLama 42, and Qwen 4 is a significant milestone in the field of natural
language processing (NLP), driving a revolution in language models’
understanding and generation of human language 56. From automatically
completing text and chatbots to complex question answering and article
generation, LLMs not only enhance the quality of text generation but are also
capable of managing a wide range of text generation tasks. However, as LLMs
increasingly empower various industries, they also reveal escalating
challenges in information security, copyright, and ethics, particularly in
their ability to produce text content whose authenticity is challenging to
discern 9.

Currently, there are numerous methods for detecting and identifying AI-
generated texts. Given that human performance in distinguishing between texts
generated by LLMs and those written by humans barely surpasses random
guessing, research has significantly shifted towards automated detection
methods. These methods aim to identify subtle cues that are typically
imperceptible to humans. The existing artificial intelligence (AI) text
detection methods can be divided into two categories based on their working
principles 21: metric-based methods and model-based methods. Metric-based
methods, such as GLTR 12, primarily rely on quantitative statistical
indicators to evaluate text characteristics, while model-based methods, like
RoBERTa 22, depend on deep learning models to identify generated texts. Each
type of method has its advantages and disadvantages. Metric-based methods
offer transparency and are computationally efficient but may lack the ability
to capture complex language patterns, leading to lower accuracy 14. In
contrast, model-based methods like RoBERTa provide high accuracy and
adaptability through the use of deep learning, but they act as black boxes for
detection, lack some interpretability, and require a large amount of labeled
data. Consequently, establishing a comprehensive benchmark for detecting and
identifying AI-generated texts has thus become a new research focus. Such a
benchmark includes evaluating the performance of existing detectors to ensure
the legitimacy and safety of LLMs applications.

At present, there are only a limited number of benchmarks available for
assessing the performance of AI-generated text detectors. He et al. 14
proposed a new English benchmark framework called MGTBench, which encompasses
a variety of text types from academic essays and creative writing to news
reporting, providing extensive testing scenarios for detecting machine-
generated texts. Yang et al. 51 introduced a dataset specifically designed for
AI-generated text detection, named Human-ChatGPT Polished Paired Abstracts
(HPPT), which includes pairs of original human-written texts and their
ChatGPT-polished versions. Although existing benchmark frameworks provide
essential references for evaluating text detection systems, they often fail to
fully capture the nuances and complexities of text generation in diverse and
intricate real-world environments. Consequently, evaluations based on these
benchmarks may not accurately reflect the detectors’ performance in actual
applications. Firstly, existing benchmarks demonstrate significant limitations
in covering the range of text generation tasks involving LLMs. They typically
concentrate on a single type of text generation, such as question answering 13
or text polishing 51. This narrow focus may not fully capture the extensive
range of text generation tasks that LLMs are increasingly capable of handling,
such as expansive writing, text completion, and summarization. Such limitation
could reduce the effectiveness of evaluation tools in real-world applications,
as they fail to capture the varying performance of LLMs across different text
generation scenarios. Secondly, current benchmarks are primarily designed for
English texts 48, 11, failing to address the linguistic diversity and unique
challenges presented by other languages, such as Chinese. Given that the
leading LLMs are predominantly developed in the United States and China, this
limitation in language coverage is particularly problematic. Furthermore,
these benchmarks often rely on datasets created using a limited number of
models 13, 45, such as ChatGPT, which may not adequately represent the
capabilities and nuances of other mainstream LLMs. Consequently, there is a
pressing need for a more comprehensive and versatile benchmark that addresses
the limitations of current frameworks, accurately simulating the multifaceted
and diverse environments in which LLMs operate. This inclusive benchmark will
facilitate a more accurate and effective evaluation of text generation
detection methods across a broader range of scenarios and languages.

Based on previous research and considering the habits of ordinary users in
generating text with large language models, as well as the degree of model
involvement, we categorize the tasks involving LLMs into five main operations:
Create, Update, Delete, Rewrite, and Translate. The "Create" operation
includes two forms: "Complete" and "Question Answering". The "Update"
operation is divided into "Polish" and "Expand". The "Delete" operation
consists of "Summary" and "Refine". In the "Rewrite" operation, LLMs enhance
clarity and fluency while maintaining the original meaning. In the "Translate"
operation, LLMs mainly handle English-Chinese translation. The specific task
details are shown in Table 1.

Table 1: Task composition statistics of CUDRT benchmark.

AI Generated text Operation | Scale | Text composition | Task Description
---|---|---|---
Create | Complete | 5000 | News, Thesis |  LLMs predict subsequent text based on the given text, ensuring that the completed text logically and contextually aligns with the initial input.
Question Answering | 5000 | Community, Wiki, Medicine, Finance |  LLMs act as an expert, providing a detailed answer to the following question.
Update | Polish | 5000 | News, Thesis |  LLMs improve the quality, fluency, and accuracy of the given text to make it more in line with linguistic standards and reader expectations
Expand | 5000 | News, Thesis |  LLMs generate new text related to the original content but in greater detail, richness, or with more expressed nuances.
Delete | Summary | 3000 | News |  LLMs summarize the given text into a concise and accurate abstract, highlighting the main information and key points of the text.
Refine | 5000 | News, Thesis |  LLMs simplify, compress, or refine the given text into a form that is more concise, clear, and easy to understand.
Rewrite | 5000 | News, Thesis |  LLMs rephrase the content of the given text to produce a new text that is semantically close to the original but differs in linguistic expression.
Translate | 5000 | News |  LLMs are primarily used to perform English-Chinese translation.

  * •

Note: The dataset size for each task only indicates the quantity for a single
language. Therefore, to calculate the total amount for both Chinese and
English, you need to multiply the indicated number by 2.

To evaluate the performance of existing AI-generated text detectors in these
text generation scenarios, we create a comprehensive benchmark CUDRT for AI-
generated text detection in both Chinese and English. CUDRT is divided into
five main sections, each corresponding to one of the five major text
generation scenarios involving LLMs. Given the widespread use of LLMs in
generating texts for news and academic publications 50, we have collected
texts comprising news articles and academic papers written by humans before
the advent of LLMs from both Chinese and English websites. Additionally, we
utilize several of the latest well-known LLMs to create AI-generated texts for
each task scenario in our dataset. Drawing on our experimental findings, we
provide valuable insights into the effective use of AI-generated text
detection methods.

The primary contributions of this paper are summarized as follows:

  * •

We define the most comprehensive suite of operations for AI-generated text to
date, encompassing "Create", "Update", "Delete", "Rewrite", and "Translate".
This framework significantly expands the scope of existing research on how
LLMs applied in text generation, offering a structured approach to exploring
and evaluating LLMs capabilities in diverse textual scenarios.

  * •

We develop a comprehensive Chinese and English dataset, constructed from news
and academic papers. This dataset is designed to assess the performance of
existing latest detectors across all scenarios of AI-involved text generation.

  * •

We conduct extensive experiments on our benchmark using various metrics to
evaluate the performance of existing AI-generated text detectors. Based on our
findings, we provide valuable guidance and recommendations for future
researchers and developers of AI-generated text detectors.

##  2 Related Work

###  2.1 Large Language Models

The evolutionary path of language models began with initial rule-based models,
developed into statistical-based models, followed by early neural language
models, then advanced models that incorporated pre-training strategies, and
has now culminated in today’s complex large language models 6. Each generation
of models has significantly pushed the boundaries of natural language
processing technology, enhancing both the depth and breadth of language
handling capabilities. Currently, large language models represent the cutting
edge in processing and generating natural language text. LLMs can be
categorized into open-source and closed-source based on the availability of
their parameters 19. Open-source models, such as Alibaba Cloud’s Qwen 4,
Meta’s LLaMA 42, and Tsinghua University’s ChatGLM 10, make their code and
training methods public, facilitating further development by researchers.
Conversely, closed-source models like OpenAI’s GPT series 1 do not disclose
their full code or data, instead offering services via APIs and other methods.
LLMs have been extensively applied across various domains of text generation,
including creative writing, automated customer service, legal document
drafting, and scientific research, demonstrating their versatility and
efficacy in producing coherent, contextually relevant, and high-quality text
across a wide range of applications. Li et al. 18 developed the FlexKBQA
framework based on LLMs, addressing the performance decline in Knowledge Base
Question Answering (KBQA) due to the lack of high-quality annotated data. Shu
et al. 36 proposed novel instruction tuning and reinforcement learning
strategies for large language models, aimed at optimizing their performance on
cross-sentence rewriting tasks. Wang et al. 44 proposed a reference-free
machine translation evaluation method, EvLP, inspired by post-editing, to
assess the quality of translations polished by large language models. Clusmann
et al. 8 found that LLMs enhanced nursing outcomes by translating medical
terminology into layman’s terms and summarizing patient information. Xie et
al. 47 demonstrated that by simply presenting a few example stories as prompts
to LLMs, they can generate new stories that rival those written by human
authors. Furthermore, LLMs show significant potential in accessing scientific
literature and aiding in programming tasks 28. In this paper, we explore how
LLMs are applied across nearly all text generation scenarios, examining the
differences in text generation among various open-source and closed-source
models.

###  2.2 AI-generated Text Detection Method

Due to the increasingly human-like level of text generated by LLMs, relying on
manual differentiation of AI-generated text is not only costly, but also
generally low in accuracy. As a variety of LLMs continue to emerge,
researchers have increasingly focused on developing targeted AI-generated text
detection algorithms to prevent the misuse of these powerful models 33.
Currently, detectors are primarily categorized based on their working
principles and internal architectures, into metric-based and model-based
methods 21, 32. Metric-based methods evaluate and identify whether text is
generated by artificial intelligence by utilizing a series of quantitative
metrics, such as textual consistency, complexity, word rank, and entropy 32.
Gehrmann et al. 12 proposed a tool GLTR for detecting and visualizing machine-
generated text. This method uses statistical approaches to examine the
generation probability of each word, its rank in model predictions, and the
entropy of the prediction distribution, helping users determine whether a text
was generated by an automated language model. Mitchell et al. 27 proposed
DetectGPT, a new method that does not require additional training or data
collection. It determines whether a text is generated by a language model by
analyzing the curvature in the model’s log probability function, thereby
enhancing the accuracy of detecting news articles generated by GPT-NeoX. Su et
al. 38 developed two zero-shot detection methods, DetectLLM-LRR and DetectLLM-
NPR, to identify machine-generated text. DetectLLM-LRR uses the Log-Likelihood
Log-Rank Ratio (LRR) and can detect text without perturbations efficiently. In
contrast, DetectLLM-NPR employs the Normalized Perturbed Log-Rank (NPR), which
improves detection accuracy by slightly rewriting the text. Model-based
methods involve initially training machine learning or deep learning models on
corpora of human-generated and machine-generated texts, and then using these
trained models to classify the two types of texts. Guo et al. 13 proposed the
ChatGPT Detector, a model that uses the RoBERTa architecture as its backbone.
The model is fine-tuned on texts generated under various scenarios to enable
its multifunctional detection capabilities. Wang et al. 43 proposed a method
called SeqXGPT for detecting AI-generated text at the sentence level. This
model uses a list of log probabilities of tokens extracted from a white-box
language model as features. It processes these features through convolutional
networks and self-attention networks to achieve fine-grained classification of
the text. Liu et al. 21 proposed a machine-generated text detection method
COCO, enhancing text coherence under low-resource conditions using contrastive
learning. This method optimizes text representation by constructing a graph
structure based on entity coherence and employs an improved contrastive loss
function, significantly enhancing detection performance on public datasets.
This paper selects mainstream models from these two types of detectors to
comprehensively assess their detection performance across various LLMs text
generation scenarios.

Table 2: Summary of existing AI-generated text detection benchmarks.

Benchmark |  Data Composition |  Scale |  Evaluation Metrics |  Detectors |  LLMs Operations |  Language
---|---|---|---|---|---|---
M4GTBench 46 |  Student Essays, Peer Reviews | 65,177 |  Accuracy, Precision, Recall, F1-score |  RoBERTa, XLM-R, GLTR, SVM |  QA, Create |  Arabic, Bulgarian, Chinese, Indonesian, Russian, Urdu, Italian, German
MGTBench 14 |  Essay, Story, Reuters | 3,000 |  Accuracy, Precision, Recall, F1-score, AUC |  Log-Likelihood, Rank, DetectGPT, GLTR, Entropy |  Create |  English
MULTITuDE 23 |  News | 44,786 |  Accuracy, Precision, Recall, F1-score, FPR,FNR, AUC |  RoBERTa, GPT-2, ELECTRA, GLTR, Rank, GPTZero |  Create |  English, Spanish, Russian
HC3 13 |  Baike, Reddit, Wiki, Community | 44,425 |  F1-score |  RoBERTa, GLTR |  QA |  Chinese, English
HPPT 51 |  Abstract | 6,050 |  Accuracy, AUC |  GPTZero, DetectGPT, GPT-2, RoBERTa |  Polish |  English
MixSet 11 |  News, Abstract, Email | 3,600 |  Accuracy |  Log-Likelihood, GLTR, DetectGPT, Radar, Entropy |  Polish, Complete, Rewrite |  English
M4 45 |  Wiki, Reddit, Abstract | 122,481 |  Accuracy, Precision, Recall, F1-score |  RoBERTa, GLTR, NELA |  Create |  English, Chinese, Russian
AIG-ASAP 31 |  Students Essays | 1,000 |  Accuracy, AUC |  RoBERTa, CheckGPT, ArguGPT |  Rewrite |  English
CUDRT(Ours) |  News, Paper, Baike, Wiki, Community, Reddit | 480,000 |  Accuracy, Precision, Recall, F1-score |  MPU, RoBERTa, XLNet |  Create, Update, Delete, Rewrite, Translate |  Chinese, English

###  2.3 AI-generated Text Detector Evaluation Benchmark

In the development and optimization of AI-generated text detectors, the
effective assessment of their performance is a critical and fundamental issue.
Following the initial discussion on performance assessment, significant
advancements have been made in refining algorithms and enhancing dataset
quality to improve the detection accuracy of AI-generated text detectors. This
section provides a comprehensive review of existing benchmark literature, with
a focusing on the development of benchmarks, the LLM-generated text tasks they
cover, and their data sources. Key details regarding these benchmarks are
summarized in Table 2 for ease of reference.

In the domain of AI-generated text detection, question answering (QA) datasets
were initially widely adopted as benchmarks 13, 46. Briefly, by utilizing LLMs
to act as question answering experts, responding to human-posed queries and
producing corresponding answers, creating pairs of AI-generated question and
answer datasets. Guo et al. 13 created a dataset named Human ChatGPT
Comparison Corpus (HC3), which included about 40,000 questions along with
responses from human experts and ChatGPT. These questions and answers cover
multiple domains, including open-domain, computer science, finance, medicine,
law, and psychology. The human responses were collected from various publicly
available question-and-answer datasets, which include highly rated responses
by internet users and wiki-based texts. Given the powerful capabilities of
LLMs, and with their widespread application in various text generation tasks,
researchers are exploring text generation from other scenarios. This
exploration aims to test and improve the performance of detectors, thereby
enhancing the models’ generalization capabilities and accuracy across multiple
contexts. MGTBench 14 generated essays and news articles by providing titles
for essays and news, using GPT3.5-turbo to create essays in response to
specific writing prompts, or using Claude to generate articles responding to
news headlines. Upon the existing HC3 dataset, Su et al. 39 further expanded
its scope, introducing a new version named HC3 Plus. This dataset includes
additional tasks such as translation, summarization, and paraphrasing, which
involve AI-generated text in complex contexts. Wang et al. 45 introduced a
multigenerator, multi-domain, and multi-lingual corpus M4, which not only
encompassed various text generation tasks of LLMs such as news writing,
question answering, and academic paper abstracts, but also included texts in
multiple languages, reflecting the text generation capabilities in
multilingual environments. Despite ongoing improvements and refinements in
current benchmarks for AI-generated text detectors, they still fall short in
covering the breadth of text generation operations, failing to comprehensively
encompass the extensive applications of LLMs. Therefore, to enhance the
adaptability and accuracy of AI-generated text detection models, there is an
urgent need to update and expand the benchmarks so that they can more
comprehensively reflect and test the performance of LLMs in practical
applications. The detection of AI-generated text can be considered a
classification problem. Consequently, the performance of AI-generated text
detectors is typically evaluated using common metrics from classification
tasks, including Accuracy, Precision, Recall, and the F1 Score. These metrics
collectively reflect the detector’s effectiveness in various aspects,
primarily providing comprehensive information about the model’s recognition
accuracy.

##  3 The CUDRT Benchmark

As LLMs technology advances and its performance improves, the texts generated
by LLMs have become highly similar to human writing. This high degree of
realism has led to the widespread application of LLMs in multiple fields,
which also introduces potential risks of misuse. To address the potential
misuse of highly realistic AI-generated texts, numerous studies have focused
on developing detection methods for AI-generated text, aiming to differentiate
between text written by humans and that produced by LLMs. However, it is
essential to systematically evaluate the effectiveness of these detection
methods to ensure their reliability and accuracy. This assessment should focus
on the performance of the detectors across various application scenarios of
LLMs, different texts generated by these models, and multiple languages.
Additionally, the adaptability of these methods to the continuously evolving
capabilities of LLMs presents significant challenges, necessitating ongoing
research and development. To enhance the accuracy and robustness of these
detection methods, especially against constantly evolving LLMs that strive to
mimic human-like writing, we must rigorously test them in various text-
generation scenarios that mimic real-world conditions. This also helps to
mitigate the risks associated with the misuse of LLMs.

Figure 1: Illustration of the CUDRT benchmark framework.

However, the shortcomings of the current benchmarks for AI-generated text
detectors are also quite apparent. Most existing benchmarks mainly focus on
fewer individual text generation tasks, such as question answering, text
polishing, and content creation. They less frequently address other LLMs-
involved text operations, such as automatic summarization or translation,
limiting a comprehensive evaluation of the detectors’ performance.
Additionally, with the rapid advancement of Chinese LLMs, there is an urgent
need for Chinese benchmark to evaluate the effectiveness of the latest
detectors on Chinese text. Given that Chinese is the most widely spoken
language, and most existing benchmarks prioritize English without fully
considering the unique characteristics of the Chinese language, this
requirement is particularly critical.

To address the shortcomings of current benchmarks as previously discussed,
this study proposes a comprehensive benchmarking framework named CUDRT,
designed to evaluate AI-generated text detectors across multiple text
generation scenarios and in both Chinese and English. Figure 1 shows the
details of our CUDRT benchmark framework. Recognizing the limitations of
existing benchmarks that focus narrowly on certain tasks and predominantly in
English, our benchmark extends to include operations such as "Create",
"Update", "Delete", "Rewrite", and "Translate", encompassing a wider range of
LLMs’ capabilities in text manipulation. We will describe the evaluation tasks
and datasets that we designed for each CUDRT application scenario type.

Table 3: Detailed information on original human-generated texts.

Language | LLMs Operation | Data Composition and source | Scale
---|---|---|---
Chinese |  Complete, Polish, Expand, Summary, Refine, Rewrite |  News: Sina${}^{{\textsuperscript{\bf a}}}$, Guangming${}^{{\textsuperscript{\bf b}}}$; Thesis: Baidu Academic${}^{{\textsuperscript{\bf c}}}$ | 65,324
QA |  Community: Baidu Tieba${}^{{\textsuperscript{\bf d}}}$; Wiki: Baidu Baike${}^{{\textsuperscript{\bf e}}}$; Medicine: Meddialog${}^{{\textsuperscript{\bf f}}}$; Finance: ChineseNlpCorpus${}^{{\textsuperscript{\bf g}}}$ | 16,389
Translate |  Translation: translation2019zh${}^{{\textsuperscript{\bf h}}}$ | 5,200,000
English |  Complete, Polish, Expand, Summary, Refine, Rewrite |  News: BBC${}^{{\textsuperscript{\bf i}}}$; Thesis: Arxiv${}^{{\textsuperscript{\bf j}}}$ | 171,302
QA |  Community: Reddit${}^{{\textsuperscript{\bf k}}}$; Wiki: Wikipedia${}^{{\textsuperscript{\bf l}}}$; Medicine: Meddialog; Finance: FiQA 24 | 25,861
Translate |  Translation: translation2019zh | 5,200,000

  * a

https://www.sina.com.cn/

  * b

https://www.gmw.cn/

  * c

https://xueshu.baidu.com/

  * d

https://tieba.baidu.com/

  * e

https://baike.baidu.com/

  * f

https://github.com/Toyhom/Chinese-medical-dialogue-data

  * g

https://github.com/SophonPlus/ChineseNlpCorpus

  * h

https://github.com/genjinshuaji/translation2019zh

  * i

https://www.bbc.com/news

  * j

https://arxiv.org/

  * k

https://www.reddit.com/

  * l

https://www.wikipedia.org/

###  3.1 Problem Definition

In this paper, we first curated a comprehensive dataset encompassing various
sub-operations of AI text generation. We then utilized the most commonly used
AI-generated text detectors to perform the detection tasks. Given a dataset
$\\{(x_{i},y_{i})\\}_{i=1}^{N}$, where $x_{i}$ represents text content, and
$y_{i}\in\\{0,1\\}$ denotes the ground truth label indicating the presence or
absence of AI-generated text, the AI-generated text detection can be defined
as follows:

$\min_{f}\mathbb{E}_{(x,y)\sim\mathcal{D}}\left[\mathbbm{1}\\{f(x)\neq
y\\}\right],$ (1)

where our objective is to find a classifier $f$ that minimizes the average
misclassification rate over the dataset. The indicator function
$\mathbbm{1}\\{f(x_{i})\neq y_{i}\\}$ takes the value 1 if the classifier $f$
incorrectly labels the input $x_{i}$, and 0 otherwise. The inputs $x_{i}$
follow different distributions conditional on $y_{i}$:

$x_{i}\mid y_{i}=0\sim\mathcal{T}_{H}\quad\text{and}\quad x_{i}\mid
y_{i}=1\sim\mathcal{T}_{A}$ (2)

where $\mathcal{T}_{H}$ and $\mathcal{T}_{A}$ represent the human and AI
generated texts, respectively.

###  3.2 Data Collection and Pre-processing

To create our dataset of AI and human text comparisons, we need to ensure the
purity of human texts before generating LLMs texts. This is crucial to
maintain the authenticity and fairness of our detection process. Therefore, to
ensure that the source texts for creating the dataset consist solely of human-
written content, without any AI-generated material, we have selected texts
from 2016 and earlier, predating the emergence of LLMs. For the operations of
"Complete", "Polish", "Expand", "Summary", "Refine", and "Rewrite", we choose
news articles and academic papers as our data sources, which are the two most
widely used fields of LLMs 37, 15. They not only have a large quantity but
also have broad applications and far-reaching influence. News reports
typically use vivid and straightforward language to present facts and events,
aiming to help readers quickly understand and absorb the information 25. In
contrast, academic papers employ specialized terminology to elaborate on
research findings, ensuring precision and scientific rigor 17. By
incorporating both, it allows for a comprehensive assessment of AI-generated
text detectors in handling language styles and complexities, ensuring broad
representativeness and credibility in evaluations. However, the tasks of
question answering and Chinese-English translation are relatively unique, so
we select different human texts from those used in the previous tasks. In the
question answering tasks, we select Chinese and English question answering
(QA) data from Community, Wiki, Medicine, and Finance scenarios 13, because
these areas are the most representative and prevalent in question answering
applications 3. Moreover, translation is one of the most classic natural
language processing operations, with a substantial body of existing research.
Therefore, we directly opt "translation2019zh" as our source of human texts.
For detailed information on the original human texts corresponding to each
operation, please refer to Table 3.

Figure 2: The process of generating text through the "Complete" operation of
LLMs.

To ensure the quality and applicability of the data, we filtered 3,000 news
articles with word counts between 1,000 and 2,000, and extracted 2,000
segments of academic papers with approximately 1,500 words each. In processing
the academic papers, we removed annotations, special symbols, and reference
materials, retaining only the core content of the papers. These steps were
taken to prevent texts from being excessively long or short, aligning with
typical human writing conventions. The preprocessing methods for Chinese and
English texts are basically the same. As Arxiv primarily contains LaTeX-
formatted texts, we utilized pydetex.pipelines
222https://github.com/ppizarror/PyDetex to process equations and removed
certain special symbols while preserving the text content as much as possible.
For the question answering texts, we selected 1,250 QA pairs from each of the
four fields and removed those with answers that were too long or too short to
ensure balance in quantity and quality.

###  3.3 "Create" Operation in AI Text Generation

The "Create" operation of LLMs refers to the process of generating new content
based on user input prompts 28. This operation relies on the deep
understanding of language and knowledge possessed by LLMs, utilizing multiple
layers of neural networks to process the input. By combining the language
patterns and knowledge learned during training, LLMs generate new text that is
both grammatically and semantically correct. The generated content is not only
relevant to the prompt but also typically exhibits a degree of creativity 20.
We categorize the "Create" operation into two sub-operations based on the
different needs and purposes of the user’s input: "Complete" and "Question
Answering".

####  3.3.1 "Complete" Sub-operation of "Create"

The "Complete" operation refers to LLMs predict subsequent text based on the
given text, ensuring that the completed text logically and contextually aligns
with the initial input. In the text completion process of LLMs, the unfinished
text is continued to generate a semantically coherent and complete passage.
Compared to having the LLMs generate complete text directly, providing an
incomplete sentence as a prompt to guide the model in completing the text has
a wider range of applications 26. This is because, when generating complete
text directly, the model may struggle to accurately grasp the user’s specific
intent and contextual background, leading to content that lacks relevance and
coherence. By offering a partial prompt, it is possible to more effectively
constrain and guide the model, resulting in generated text that better aligns
with the user’s expectations. Therefore, completing text is also one of the
most commonly used operations for LLMs.

Figure 3: The process of generating text through the "Polish" operation of
LLMs.

When performing "Complete" operations, considering the diversity in how people
use LLMs to extend text length, we established three different completion
ratios: 25%, 50%, and 75%. We utilized collected news and academic paper
texts, truncating them according to these three ratios: 75%, 50%, and 25%.
Subsequently, we employed LLMs to complete the truncated portions of the text,
with the number of completed characters equal to the total character count
minus the character count of the truncated text. This approach ensures that
the completed text has a character count similar to the original text, thereby
ensuring fairness in detecting AI-generated text. Figure 2 illustrates the
entire process of performing text completion operations with LLMs. Using a 50%
news completion ratio as an example, we split the initial text into two parts:
the beginning text and the following text, with roughly equal word counts. To
enable the LLMs to perform the completion task more effectively, our prompt
primarily includes task definition and task requirements. The task definition
aims to provide the LLMs with a clearer understanding of the task, preventing
any misinterpretation. The task requirements specify the details for executing
the task, including the beginning text and following text. The following text
defines the number of words that need to be completed, while the initial text
provides the content that the LLMs need to complete. Additionally, we
stipulated that the completed text must adhere to the norms and style of news
texts, ensuring that the completed text is appropriate for its respective
field. The process for completing academic papers is similar to this approach.

####  3.3.2 "Question Answering" Sub-operation of "Create"

The "Question Answering" operation refers to that LLMs act as an expert,
providing a detailed answer to the given question. Human utilization of
machine-based question answering has become a widely pervasive phenomenon.
From early rule-based dialogue systems to context-based querying, and now
advanced question answering systems built upon deep learning and LLMs, the
evolution of this technology has been remarkable 29. The extensive application
of such systems has demonstrated significant value and potential across
various domains. Whether in education, healthcare, finance, or other
industries, people increasingly rely on these systems to obtain accurate and
timely answers 2. Currently, question answering systems based on LLMs have
essentially become capable of replacing humans, directly answering various
human queries. Whether in complex professional domains or everyday life
inquiries, these systems can provide accurate and rapid responses.

When creating the LLMs’ question answering dataset, we opted to cover human
queries across four domains: Community, Wiki, Medicine, and Finance. These
domains essentially cover mainstream aspects of people’s daily lives,
including community interactions, access to wiki knowledge, medical health
consultations, and financial services. This selection contributes to building
a diverse, highly practical, and challenging question answering dataset. For
specific details, please refer to section 3.2. It’s worth noting that some
questions involve complex specialized knowledge and personal privacy security
7. Directly asking such questions to LLMs may result in the model being
unwilling to respond or providing lower quality answers 5. Therefore, it’s
important to handle these types of questions with caution and considerate
questioning. In the prompts, we instruct the LLMs to assume specific roles.
For instance, when asking medical questions, we include the following
instruction: "Imagine you are a doctor specialized in answering patient
inquiries. Please provide detailed answers to the following questions based on
your knowledge base, ensuring accuracy." For other question answering domains,
it is sufficient to adjust the roles relevant to those specific domains.
According to our extensive experiments, this approach inclines the LLMs to be
more willing to respond to questions, resulting in better answer quality that
aligns more closely with human reasoning logic.

Figure 4: The process of generating text through the "Expand" operation of
LLMs.

###  3.4 "Update" Operation in AI Text Generation

In this section, we define the "Update" operation for AI-generated text. This
operation refers to making modifications to existing text to make the
expression clearer, more complete, or the content richer. Depending on the
degree and purpose of the modifications, we mainly divide them into "Polish"
and "Expand" operations.

####  3.4.1 "Polish" Sub-operation of "Update"

The "Polish" operation refers to the process where, given input text, the LLMs
improve the quality, fluency, and accuracy of the given text to better adhere
to language conventions and reader expectations. Text polishing is a task
frequently undertaken by humans using LLMs after completing initial drafts 54.
The LLMs can assist authors in enhancing text quality by identifying and
rectifying spelling errors, grammatical mistakes, and inappropriate
expressions. Additionally, LLMs can offer suggestions such as alternative
vocabulary, optimized sentence structures, and adjustments to tone and style,
thereby making the text more engaging and readable. Therefore, in typical
scenarios, LLMs during the polishing process may only appear as the auxiliary
tools, with the degree of AI involvement not particularly prominent. AI-
generated texts are more often interspersed within human-generated texts. This
also raises higher requirements for methods to detect AI text.

Figure 3 shows the entire process of using LLMs for text polishing. Similarly,
the "Polish" operation prompts we designed also include two parts: task
definition and task requirements. This helps the LLMs understand and execute
tasks better, which aligns with our usual practice. The figure uses a news
excerpt to illustrate the changes before and after text polishing. It can be
seen that the model primarily performs word replacement and expansion,
resulting in higher quality, more fluent text that better aligns with the
language norms of the news domain. Additionally, it’s important to note that
there are slight differences between polishing news articles and academic
papers, which necessitates instructing the LLMs to adhere to the respective
field-specific terminologies in the prompts.

####  3.4.2 "Expand" Sub-operation of "Update"

The "Expand" operation refers to the process where, given an input text, the
LLMs generate new text related to the original content but in greater detail,
richness, or with more expressed nuances. This operation aims to enrich the
existing text by making it more comprehensive and detailed. With the
generation capabilities of LLMs, new viewpoints can be added, more examples or
details can be provided, all while maintaining the style and semantic
coherence of the text. This results in text content that is more substantial
and persuasive. Compared to text completion, text expansion focuses more on
adding additional information and content based on existing text. The purpose
of text expansion is to enrich the text and make it more persuasive, rather
than simply filling in missing parts. It can involve introducing new
viewpoints, providing more examples or details. In contrast, text completion
emphasizes filling in blank parts of the text within a given context to
maintain its integrity and coherence.

Similar to the process of text polishing, text expansion mainly focuses on
word level expansion, making the text information more rich. Figure 4 shows
the entire process of using LLMs for text extension. We also visually
demonstrate the two main functions of text extension: richer words and greater
details. Similarly, it is also necessary to prompt the LLMs to recognize the
differences between news and academic papers, as their linguistic styles may
vary during text expansion.

Figure 5: The process of generating text through the "Summary" operation of
LLMs.

###  3.5 "Delete" Operation in AI Text Generation

In modern text processing, the "Delete" operation of LLMs is a common
application 16. It enables the effective identification and removal of
redundant parts of the text through a deep understanding of the content. This
operation is not just a simple character deletion, but a semantic-based
precise editing aimed at enhancing the conciseness and readability of the
text. LLMs utilize their advanced natural language processing capabilities to
analyze the structure and semantic coherence of the text, automatically
identifying unnecessary repetitions, overly detailed sections, or deviations
from the main topic. Subsequently, the model will propose modifications or
directly perform deletion operations, making the final version of the text
more compact and clear, while maintaining the integrity of the information and
the fluency of expression. Based on the extent of text deletion, we categorize
"Delete" operation into "Summary" and "Refine".

####  3.5.1 "Summary" Sub-operation of "Delete"

The "Summary" operation refers to the process where, after inputting text, the
LLMs summarize the given text into a concise and accurate abstract,
highlighting the main information and key points. It helps readers quickly
understand the core content. This not only improves reading efficiency but
also makes information delivery more direct and focused. When dealing with
large amounts of data or lengthy articles, "Summary" operations are especially
useful for processing large amounts of data or lengthy articles, supporting
quick decision-making and knowledge absorption 34. This technology has gained
widespread application in mainstream news websites and various information-
intensive fields 30. However, as such technology proliferates, identifying
whether a summary is generated by LLMs has become essential to ensure the
source and reliability of information.

Due to the token limitations of LLMs, entire papers are usually not input into
them for summarization. Therefore, we mainly focus on the creation of news
summaries. Figure 5 illustrates the specific process of news summarization by
LLMs, using a news excerpt as an example. To ensure the fairness of the
detection experiment, we calculated the word count of the human-generated
summary corresponding to each text and specified that the LLMs generate
summaries with a similar word count. This approach allows us to more
accurately compare summaries generated by humans and large language models,
avoiding the influence of length differences on the comparison results.
Similarly, Figure 5 also shows the specific changes during text summarization.
It can be seen that the LLM reduces redundant information in some parts of the
text and directly summarizes other parts into key points. By doing so, LLMs
not only enhance the text’s fluency and readability but also ensure the
accurate conveyance of important information.

Figure 6: The process of generating text through the "Refine" operation of
LLMs.

####  3.5.2 "Refine" Sub-operation of "Delete"

The "Refine" operation refers to the process where, after inputting text, the
LLMs simplify, compress, or refine the given text into a form that is more
concise, clear, and easy to understand. Through a deep analysis of the
content, the model can identify unnecessary repetitions, verbose sections, or
irrelevant parts, thus improving the flow and readability of the text while
retaining key information. This operation not only enhances the quality of the
text but also effectively improves the precision of information delivery 35.
Especially in scenarios such as business reports or research papers, where
text quality is crucial, text refinement can significantly improve the final
presentation, making information delivery clearer and preventing readers from
losing focus due to lengthy or complex expressions. Additionally, refinement
helps avoid ambiguity in information dissemination, making it easier for
readers to grasp the core ideas of the text and improving information
absorption efficiency. Compared to "Summary", "Refine" involves a lower degree
of deletion. The refinement process retains most of the information in the
text, only removing redundant or irrelevant parts, making the overall content
more concise and fluid.

Figure 6 shows the specific details of the "Refine" operation. When creating a
refined text, we specify in the prompt that the length of the output text from
LLMs should be between 80% and 90% of the given text. This prevents the
refined text from being too short and resembling a summary, aligning with
people’s understanding and impression of text refinement. It should be noted
that we also specifically showcase the detailed changes in the refined text
within the figure. The text has not only removed redundancy but has also
simplified certain sentence structures. Furthermore, we produce news
refinement and paper refinement separately to better align with the semantic
styles and characteristics of each field.

Figure 7: The process of generating text through the "Rewrite" operation of
LLMs.

###  3.6 "Rewrite" Operation in AI Text Generation

The "Rewrite" operation refers to the process where, upon inputting a text,
the LLMs adjusts word selection, reorganizes paragraphs to change the given
text style to produce a new text that is close in meaning but different in
linguistic expression from the original. This technique can enhance the
fluidity and readability of an article, significantly improving the text’s
quality and diversity, and making the language more vivid or suited to
specific stylistic requirements. During the rewriting process, synonyms can be
substituted, sentence structures adjusted, and semantics strengthened to
optimize the text, while keeping the core information and main themes intact.
This method is particularly useful for enhancing the clarity of academic
papers, increasing the appeal of business copy, or adjusting the style of any
text that requires it. Compared to "Polish" operation, "Rewrite" operation
often involves more profound and fundamental changes. Text polish mainly
entails minor adjustments such as correcting grammatical errors, enhancing
sentence fluency, and optimizing word choice, with the goal of making the
original content more concise and appealing. On the other hand, text rewrite
may involve rethinking and re-expressing entire sections or articles,
including changing the way information is organized, rearticulating the theme,
and even altering the original viewpoint or tone.

Figure 7 shows the specific details of the "Rewrite" operation. During the
execution of the "Rewrite" operation, considering the characteristics of text
rewriting and ensuring the fairness of the detection experiment, we require
that the word count of the text rewritten by the LLMs be approximately the
same as the human text. Simultaneously, to more intuitively demonstrate the
specific changes in text rewriting, we take a news excerpt as an example and
show the rewriting process at the word and sentence levels in the figure. By
adjusting word selection and reorganizing paragraph structures, the LLMs
transform the language style of the news, providing a completely new reading
experience.

Table 4: Specific parameter details for Chinese and English LLMs.

Chinese Model | Parm. | Type | Publisher | Release | English Model | Parm. | Type | Publisher | Release
---|---|---|---|---|---|---|---|---|---
Baichuan2 49 | 13B | Chat | Baichuan Inc. | 2023.09 | Baichuan2 49 | 13B | Chat | Baichuan Inc. | 2023.09
ChatGLM3-32K 53 | 6B | Chat | Tsinghua | 2024.01 | ChatGLM3-32K 53 | 6B | Chat | Tsinghua | 2024.01
GPT3.5-Turbo 55 | 175B | Chat | OpenAI | 2023.03 | GPT3.5-Turbo 55 | 175B | Chat | OpenAI | 2023.03
GPT4-1106 1 | NaN | Chat | OpenAI | 2023.11 | Llama3 40 | 8B | Chat | Meta | 2024.04
Qwen1.5 4 | 32B | Chat | Alibaba | 2024.04 | Llama2 42 | 13B | Chat | Meta | 2023.07

  * •

Note: NaN denotes no public data available.

###  3.7 "Translate" Operation in AI Text Generation

The use of large language models for text translation has become a common and
core function among their many capabilities. In the field of translation,
these models do more than just replace words; they capture the context, tone,
and cultural nuances of the original text and reconstruct these elements in
the target language. This capability makes the translation results more
natural and accurate, greatly enhancing the quality of cross-language
communication. Additionally, the ongoing learning and self-optimization
features of LLMs ensure that they continuously improve over time with
accumulated data, better adapting to new linguistic usages and translation
challenges. Therefore, "Translate" operation has gradually matured, with the
translated texts being virtually indistinguishable from those translated by
humans.

Therefore, it is the simplest for LLMs to perform this operation. When
creating the dataset, we chose the most commonly used English-Chinese and
Chinese-English translation. Simply have the LLMs translate into English or
Chinese as needed.

###  3.8 Large Language Models for Text Generation

Considering that our benchmark includes both Chinese and English materials, we
have selected several of the most advanced LLMs from China and the U.S. This
approach not only allows our dataset to keep pace with the latest
technological developments but also ensures the high quality of the texts
within the dataset. The LLMs we have selected mainly include: the GPT series,
Llama series, Qwen series, ChatGLM series, and Baichuan series. The specific
model details are shown in Table 4.

GPT4 and Llama3 represent state-of-the-art developments in close-source and
open-source LLMs, respectively. In the process of constructing our dataset, we
employed GPT-4-1106-preview for the generation of Chinese text and
Llama-3-8b-instruct for the generation of English text. This methodological
choice was informed by performance evaluations indicating that
Llama-3-8b-instruct demonstrates suboptimal proficiency in generating Chinese
text. Consequently, to ensure a balanced number of Chinese and English texts
within our dataset, we relied on GPT-4-1106-preview for the generation of
Chinese text. GPT3.5 is currently the most versatile large language model with
the widest user base. Therefore, we used GPT-3.5-turbo to create datasets in
both Chinese and English. The Qwen series of models were developed by Alibaba
Company, characterized by multiple models with varying numbers of parameters.
We have chosen the Qwen1.5-32B-Chat model, which has 3.2 billion parameters,
for creating our Chinese dataset. Our experiments have shown that the Qwen
series performs poorly in generating English text, so we only use it to
generate Chinese text. Conversely, before the release of the Llama3 series
models, Llama2 was the most commonly used open-source English model. Since the
majority of its training data consists of English texts, we used this model
exclusively for generating English text. The ChatGLM series of models were
jointly developed by Tsinghua University and Zhichart AI Company. This
research utilizes the ChatGLM3-6b-32k model to generate Chinese and English
texts. The Baichuan series of models are a range of extended multilingual base
language models developed by "Baichuan Intelligence" company. This research
employs the open-source Baichuan2-13B chat model.

###  3.9 AI-generated Text Detection Methods

When conducting detection of AI-generated text in both Chinese and English, we
employed two different types of detectors to ensure comprehensive coverage
across multiple aspects. The model-based approach relies on deep learning
models trained on extensive data, enabling them to learn and identify features
and patterns specific to LLMs’ text. On the other hand, the metric-based
approach relies on a set of predefined rules or feature indicators, which may
involve aspects such as grammar structure, vocabulary usage, and logical
coherence. By establishing a reasonable set of indicators, potential AI-
generated text can be identified quickly and effectively. By integrating these
two methods, we are able to scrutinize texts from multiple dimensions, thereby
enhancing the accuracy and reliability of detection.

####  3.9.1 Metric-based Method

Due to most metric-based models being primarily designed for AI-generated text
detection in English, and some common models needing improvement in
effectiveness, the research outcomes are not sufficiently innovative. In
contrast, Multiscale Positive-Unlabeled (MPU) not only performs excellently in
detection but also supports both Chinese and English, making it suitable for
comprehensive detection of common languages. Therefore, we have chosen MPU,
the latest research achievement, as our metric-based detection method.

Multiscale Positive-Unlabeled 41. The MPU addresses the difficulty of short
text detection by reformulating the problem of AI-generated text detection as
a partial Positive-Unlabeled (PU) problem. This approach uses a length-
sensitive Multiscale PU loss function, combined with an abstract recurrent
model to estimate the prior probabilities of positive samples across different
text lengths. It also introduces a Text Multiscaling module that generates
texts of varying lengths through random sentence deletion, thereby enhancing
the model’s ability to detect short texts while maintaining effectiveness in
long text detection.

####  3.9.2 Model-based Method

For the model-based method, we primarily selected two representative studies:
RoBERTa and XLNet. We chose these models because they serve as foundational
architectures for many language models, showcasing advanced capabilities in
handling complex linguistic structures and context dependencies.

RoBERTa 22. The RoBERTa classifier is a text classification tool based on the
pretrained language model RoBERTa, which has been fine-tuned specifically for
detecting AI-generated text. It is a variant of the BERT model that improves
performance on a variety of natural language processing tasks by enhancing
data handling and model configuration during training. In the application of
detecting AI-generated text, the RoBERTa classifier learns from a large set of
features extracted from both human and AI-generated texts, allowing it to
distinguish which texts are produced by natural language models such as GPT or
other LLMs. In this study, the model is used to detect Chinese and English
texts.

XLNet 52. XLNet adopts the permutation language model, which allows the model
to consider all possible word order combinations during training, addressing
the issue of inconsistency between pre-training and fine-tuning stages in
BERT. Additionally, XLNet integrates Transformer-XL technology, enhancing its
capability to handle long-range dependencies. Similarly, XLNet learns to
distinguish whether AI is involved in text generation by extracting features
from both human and AI-generated text.

##  4 Experiments

###  4.1 Experimental Settings

In this section, we will describe how we conduct experiments to evaluate the
impact of each component in AI-generated text detection. The AI-generated text
detection system consists of the following components:

  * •

Detection Methods: The core of system is the detection model, which analyzes
input text to determine whether it is generated by LLMs. We explored two sets
of detection models with different principles to evaluate their effectiveness.

  * •

Training Data Composition: For model-based detection methods, the composition
of the training dataset plays a crucial role in detection performance. This
includes the diversity of data types, whether the data comes from a single
text generation operation, and whether it originates from a single LLM. We
have studied how variations in the composition of the training data affect the
generalization ability of the detection model. For model-based detection
method experiment based on the dataset, we set the ratio of the training set
to the test set at 4:1.

Figure 8: The schematic diagram of the specific experimental process.

  * •

LLMs Text Generation Operation: When utilizing large language models for
different text generation operations, the LLMs’ involvement, generation
strategies, user prompts, text quality, and semantics all vary. Consequently,
the generated texts from different operations can have significantly different
impacts on detection results.

  * •

Languages: As the two most widely spoken languages, Chinese and English
exhibit significant differences in language structure, vocabulary and
expression, syntax and grammar rules, and cultural background. These
differences result in varying outcomes when detecting texts in different
languages.

  * •

Large Language Models: The detection system examines texts generated by
various large language models. Different LLMs may produce texts with varying
degrees of complexity, coherence, and style, which can affect the detection
accuracy.

To better evaluate the impact of each component in the detection system, we
have developed a easy-to-extend and easy-to-use evaluation framework, as
illustrated in Figure 8. In the experiments, we explore the impact of
different component variations on detection results, using the training data
composition of the detection model as the main focus. We compare the following
values for each component:

  * •

Detection Methods: MPU, RoBERTa, XLNet.

  * •

Training Data Composition: Cross-Dataset, Cross-Operation, Cross-LLM.

  * •

LLMs Text Generation Operation: Complete, Question Answering; Polish, Expand;
Summary, Refine; Rewrite; Translate.

  * •

Languages: Chinese, English.

  * •

Large Language Models: Chinese: Baichuan2-13B, ChatGLM3-6B-32K, GPT3.5-Turbo,
GPT4-1106, Qwen1.5-32B; English: Baichuan2-13B, ChatGLM3-6B-32K, GPT3.5-Turbo,
Llama3-8B, Llama2-13B.

###  4.2 Evaluation Metrics

In this paper, the detectors utilized are all binary classification models,
determining whether AI was involved in text generation. Similar to other
methods for assessing classifier performance, we employ common metrics
including Accuracy, Precision, Recall, and F1-score as evaluation metrics.
Given the true positive (TP), true negative (TN), false positive (FP), false
negative (FN) count, we can calculate the metrics as following:

$Accuracy=\dfrac{TP+TN}{TP+TN+FP+FN}$ (3) $Precision=\dfrac{TP}{TP+FP}$ (4)
$Recall=\dfrac{TP}{TP+FN}$ (5) $F1-score=\dfrac{TP}{TP+\dfrac{1}{2}(FP+FN)}$
(6)

These metrics provide a comprehensive overview of the AI-generated text
detector’s performance and are commonly used to evaluate binary classification
models.

Table 5: Direct detection of the proposed dataset based on the MPU model. |  | Chinese |  | English
---|---|---|---|---
|  | Accuracy | Precision | Recall | F1-score |  | Accuracy | Precision | Recall | F1-score
Complete | Baichuan | 0.5552 | 0.9423 | 0.2954 | 0.4498 | Baichuan | 0.6080 | 0.7312 | 0.4780 | 0.5781
QA | 0.9580 | 0.9775 | 0.9392 | 0.9580 | 0.9148 | 0.9675 | 0.8670 | 0.9145
Polish | 0.5492 | 0.9695 | 0.2815 | 0.4363 | 0.5288 | 0.6552 | 0.3257 | 0.4351
Expand | 0.6336 | 0.9771 | 0.4114 | 0.5790 | 0.6460 | 0.8336 | 0.4880 | 0.6156
Summary | 0.6672 | 0.8460 | 0.5191 | 0.6434 | 0.6292 | 0.6617 | 0.5929 | 0.6254
Refine | 0.6392 | 0.9888 | 0.4167 | 0.5863 | 0.5652 | 0.7508 | 0.3706 | 0.4962
Rewrite | 0.6340 | 0.9855 | 0.4097 | 0.5788 | 0.6096 | 0.8128 | 0.4313 | 0.5636
Translate | 0.5908 | 0.6587 | 0.5041 | 0.5712 | 0.6024 | 0.6939 | 0.4967 | 0.5789
Operations Average |  | 0.6534 | 0.9182 | 0.4721 | 0.6003 |  | 0.6380 | 0.7633 | 0.5063 | 0.6009
Complete | ChatGLM | 0.6356 | 0.9669 | 0.4174 | 0.5831 | ChatGLM | 0.5972 | 0.7323 | 0.4533 | 0.5600
QA | 0.9636 | 0.9809 | 0.9469 | 0.9636 | 0.9040 | 0.9711 | 0.8447 | 0.9035
Polish | 0.5396 | 0.9626 | 0.2669 | 0.4179 | 0.6260 | 0.8288 | 0.4540 | 0.5867
Expand | 0.5840 | 0.9646 | 0.3376 | 0.5001 | 0.7204 | 0.8769 | 0.5934 | 0.7078
Summary | 0.6324 | 0.8371 | 0.4616 | 0.5951 | 0.6688 | 0.6891 | 0.6479 | 0.6678
Refine | 0.6468 | 0.9893 | 0.4280 | 0.5975 | 0.6284 | 0.8216 | 0.4621 | 0.5915
Rewrite | 0.6072 | 0.9926 | 0.3673 | 0.5362 | 0.6504 | 0.8443 | 0.4902 | 0.6203
Translate | 0.7308 | 0.7560 | 0.7060 | 0.7301 | 0.7220 | 0.7829 | 0.6643 | 0.7187
Operations Average |  | 0.6675 | 0.9313 | 0.4915 | 0.6155 |  | 0.6897 | 0.8184 | 0.5763 | 0.6696
Complete | GPT3.5 | 0.9488 | 0.9904 | 0.9104 | 0.9487 | GPT3.5 | 0.8584 | 0.8771 | 0.8403 | 0.8583
QA | 0.9816 | 0.9696 | 0.9939 | 0.9816 | 0.9676 | 0.9733 | 0.9620 | 0.9676
Polish | 0.5968 | 0.9802 | 0.3542 | 0.5204 | 0.6420 | 0.8281 | 0.4837 | 0.6107
Expand | 0.8356 | 0.9941 | 0.7143 | 0.8313 | 0.7032 | 0.8713 | 0.5673 | 0.6872
Summary | 0.6228 | 0.8344 | 0.4455 | 0.5808 | 0.4816 | 0.4635 | 0.4328 | 0.4476
Refine | 0.7360 | 0.9933 | 0.5606 | 0.7167 | 0.6156 | 0.8204 | 0.4389 | 0.5719
Rewrite | 0.8128 | 0.9924 | 0.6790 | 0.8064 | 0.6132 | 0.7954 | 0.4472 | 0.5725
Translate | 0.6116 | 0.6759 | 0.5365 | 0.5982 | 0.5760 | 0.6503 | 0.4741 | 0.5484
Operations Average |  | 0.7683 | 0.9288 | 0.6493 | 0.7480 |  | 0.6822 | 0.7849 | 0.5808 | 0.6580
Complete | GPT4 | 0.9412 | 0.9929 | 0.8944 | 0.9410 | Llama3 | 0.7984 | 0.8546 | 0.7469 | 0.7971
QA | 0.9860 | 0.9750 | 0.9973 | 0.9860 | 0.9376 | 0.9798 | 0.8986 | 0.9375
Polish | 0.8400 | 0.9942 | 0.7213 | 0.8360 | 0.6992 | 0.8739 | 0.5590 | 0.6818
Expand | 0.9600 | 0.9974 | 0.9252 | 0.9599 | 0.7696 | 0.8919 | 0.6680 | 0.7639
Summary | 0.6648 | 0.8552 | 0.5099 | 0.6389 | 0.7164 | 0.7138 | 0.7190 | 0.7164
Refine | 0.8512 | 0.9933 | 0.7399 | 0.8480 | 0.7124 | 0.8766 | 0.5799 | 0.6981
Rewrite | 0.9524 | 0.9991 | 0.9097 | 0.9523 | 0.7528 | 0.9051 | 0.6312 | 0.7437
Translate | 0.6316 | 0.7018 | 0.5554 | 0.6201 | 0.6036 | 0.6891 | 0.5041 | 0.5823
Operations Average |  | 0.8534 | 0.9386 | 0.7816 | 0.8478 |  | 0.7488 | 0.8481 | 0.6633 | 0.7401
Complete | Qwen | 0.9888 | 0.9919 | 0.9857 | 0.9888 | Llama2 | 0.7640 | 0.8381 | 0.6971 | 0.7611
QA | 0.9788 | 0.9650 | 0.9930 | 0.9788 | 0.9796 | 0.9762 | 0.9831 | 0.9796
Polish | 0.7160 | 0.9927 | 0.5308 | 0.6917 | 0.6540 | 0.8257 | 0.5069 | 0.6282
Expand | 0.9600 | 0.9931 | 0.9289 | 0.9600 | 0.6592 | 0.8554 | 0.5000 | 0.6311
Summary | 0.6036 | 0.8136 | 0.4195 | 0.5536 | 0.7976 | 0.7544 | 0.8427 | 0.7961
Refine | 0.7948 | 0.9933 | 0.6505 | 0.7861 | 0.7044 | 0.8785 | 0.5652 | 0.6879
Rewrite | 0.8156 | 0.9901 | 0.6847 | 0.8096 | 0.7328 | 0.8922 | 0.6053 | 0.7213
Translate | 0.6004 | 0.6529 | 0.5353 | 0.5883 | 0.7048 | 0.7753 | 0.6379 | 0.6999
Operations Average |  | 0.8073 | 0.9241 | 0.7160 | 0.7946 |  | 0.7496 | 0.8495 | 0.6673 | 0.7381

###  4.3 Experimental Results

To comprehensively evaluate the efficacy of AI-generated text detectors, it is
crucial to delve into specific scenarios where these detectors are employed.
This section will explore the impact of various components of AI-generated
text detection systems on detection based on the dataset proposed in this
paper. Due to the significant impact of different training dataset
compositions on the performance of detectors, especially model-based
detectors, our entire experiment centers around the composition of the
datasets. We focus on three primary scenarios: Cross-Dataset detection, Cross-
Operation detection, and Cross-LLM detection. These experiments provide a
robust framework for understanding the strengths and weaknesses of current AI-
generated text detectors, offering valuable insights for optimizing their
design and implementation. The comprehensive evaluation enabled by our
benchmark not only enhances the assessment accuracy but also guides future
research directions and practical applications in the field of AI-generated
text detection.

Table 6: Detection results of MPU on various operation generated texts. |  | Chinese |  | English
---|---|---|---|---
|  | Accuracy | Precision | Recall | F1-score |  | Accuracy | Precision | Recall | F1-score
Complete | All Chinese | 0.8139 | 0.9769 | 0.7007 | 0.7823 | All English | 0.7252 | 0.8067 | 0.6431 | 0.7109
QA | 0.9736 | 0.9736 | 0.9740 | 0.9736 | 0.9407 | 0.9736 | 0.9111 | 0.9405
Polish | 0.6483 | 0.9798 | 0.4309 | 0.5804 | 0.6300 | 0.8023 | 0.4659 | 0.5885
Expand | 0.7946 | 0.9853 | 0.6635 | 0.7661 | 0.6997 | 0.8658 | 0.5633 | 0.6811
Summary | 0.6382 | 0.8372 | 0.4711 | 0.6023 | 0.6587 | 0.6565 | 0.6471 | 0.6507
Refine | 0.7336 | 0.9916 | 0.5591 | 0.7069 | 0.6452 | 0.8296 | 0.4834 | 0.6091
Rewrite | 0.7644 | 0.9920 | 0.6101 | 0.7366 | 0.6718 | 0.8500 | 0.5211 | 0.6443
Translate | 0.6330 | 0.6891 | 0.5675 | 0.6216 | 0.6418 | 0.7183 | 0.5554 | 0.6256
All Average |  | 0.7500 | 0.9282 | 0.6221 | 0.7212 |  | 0.7016 | 0.8128 | 0.5988 | 0.6814

####  4.3.1 Cross-Dataset Detection

This paper selects three of the most commonly used and latest AI-generated
text detectors. These detectors were previously trained using the widely
adopted HC3 dataset. To evaluate their performance on unfamiliar texts, we
utilize the new dataset created in this paper without any fine-tuning. In
practical applications, detectors often encounter texts for the first time.
Therefore, testing directly with an unfamiliar dataset provides a more
objective evaluation of the detectors’ performance in real-world scenarios.
This approach allows us to accurately reflect the detectors’ actual
performance when handling previously unseen texts, thereby providing a precise
assessment of their effectiveness in various application contexts.

Table 5 presents the detection results of the MPU model on our proposed
dataset. This table presents the evaluation metrics for text generated by five
different large language models in both Chinese and English across various
operations, as well as the average performance of the MPU in detecting text
from each LLM. For Chinese texts, the MPU model shows consistent detection
performance across texts generated by five different LLMs. Notably, in the
detection of generated texts for the QA operation, the MPU model performs
exceptionally well, with detection accuracy exceeding 0.95. For other
operations involving the five LLMs, the MPU’s detection performance shows some
variation. Overall, MPU demonstrates superior performance in detecting texts
generated by the GPT series models. Notably, when detecting texts generated by
GPT4, MPU achieves detection accuracy exceeding 90% not only for "QA" but also
for "Complete", "Expand", and "Rewrite" operations. These results indicate
that the MPU detector has high accuracy and reliability in detecting Chinese
texts generated by GPT models, making it suitable for various text generation
tasks. From the average detection values of various LLMs’ texts, MPU shows
excellent performance in detecting texts generated by GPT series models,
ranking first in all metrics. The main reason is that the text in the HC3
dataset mainly comes from ChatGPT, which mixes the text content of GPT3.5 and
GPT4, making MPU more familiar with detecting GPT series texts. In contrast,
MPU performs relatively poorly in detecting texts generated by Baichuan and
ChatGLM. Except for "QA" texts, its detection accuracy for other operation
texts does not exceed 80%. For English texts, the overall performance of MPU
is unsatisfactory. In the detection of texts generated by the LLMs, except for
the "QA" operation, the detection results for other types of generated texts
are poor. This indicates that MPU is more suitable for detecting Chinese
texts, while its performance in detecting English texts is lacking. In terms
of detecting five types of LLMs’ texts, MPU performs better overall in Llama2
text detection. We speculate that it may be due to the existence of certain
specific patterns or features in the training data of the Llama series models,
which are preserved in the generated text, making it easier for the MPU model
to recognize these texts. Additionally, it is noteworthy that MPU performs
better in terms of precision compared to other metrics across the entire
table. We believe this is due to the imbalance between positive and negative
samples in the HC3 dataset. Table 6 shows the detection performance of MPU on
various operation generated texts without distinguishing LLMs’ texts. It is
evident that, whether for Chinese or English text, MPU performs best on "QA"
generated text. This is understandable, as the HC3 data consists entirely of
"QA" texts, making MPU more familiar with this form of text.

((a)) Chinese

((b)) English

Figure 9: Radar charts of RoBERTa direct detection average results for Chinese
and English AI texts of various LLMs.

((a)) Chinese

((b)) English

Figure 10: Column charts of RoBERTa direct detection average results for
Chinese and English AI texts of various operations.

RoBERTa is currently the most widely used AI-generated text detection model.
Figure 9 shows the radar charts of RoBERTa’s average detection results for
text generated by various LLMs. For Chinese, similar to the detection trend of
MPU, RoBERTa performs better on the generated text of GPT series models.
Similarly, the detection results on Baichuan texts remain at a low level,
which further demonstrates that the source of training data significantly
impacts the performance of the detector. However, its detection results are
inferior to those of MPU: the F1-score of GPT4’s text detection is less than
0.6, significantly lower than MPU’s 0.8475. This indicates that, in cross-
dataset scenarios, RoBERTa’s generalization ability for Chinese text detection
is not as robust as that of the MPU model. Overall, precision has achieved the
best value, while the performance of the other metrics lags significantly
behind. This is likely due to the imbalance between positive and negative
samples in the HC3 dataset. For English, RoBERTa performed the best in the
Llama series models, maintaining the same trend as MPU. From the perspective
of the four metrics, accuracy has achieved the best value, which aligns with
our expectations. However, the detection accuracy has not exceeded 0.65,
indicating a gap compared to the overall performance of MPU. Figure 10 shows
the column charts of RoBERTa’s average detection results for text generated by
various operations. From the Chinese test results, RoBERTa’s performance in
"QA" is much better than other operations. In comparison to other operations,
"Complete" performs slightly better. The main reason is that during "Create"
operations, LLMs have a higher involvement in text generation, making them
easier to detect. Additionally, from the detection results of all texts, it
can be seen that RoBERTa’s performance is slightly inferior to MPU. This
proves that in cross-dataset scenarios, MPU has higher detection accuracy and
generalization performance. Notably, the detection results for English texts
reveal that RoBERTa exhibits higher detection accuracy for texts generated in
the "Summary" operation, showing a different detection trend compared to MPU.
This may be because English "Summary" texts, due to their summarizing nature,
tend to have a more uniform and consistent generation style, which may help
RoBERTa better recognize the characteristics of generated texts. XLNet
performs the worst in cross-dataset scenarios, and due to the length of the
main text, specific experimental results can be found in the appendix.

Table 7: The detection results of the RoBERTa model trained on a single main operation data on each sub-operation text. Train Data | Test Data | Chinese | Test Data | English
---|---|---|---|---
Accuracy | Precision | Recall | F1-score | Accuracy | Precision | Recall | F1-score
Create | Complete | 0.8289 | 0.7669 | 0.8971 | 0.8266 | Complete | 0.9842 | 0.9928 | 0.9761 | 0.9842
QA | 0.9750 | 0.9535 | 0.9976 | 0.9750 | QA | 0.9898 | 0.9811 | 0.9987 | 0.9898
Polish | 0.5394 | 0.5210 | 0.3679 | 0.4312 | Polish | 0.8757 | 0.8130 | 0.9465 | 0.8744
Expand | 0.5407 | 0.5215 | 0.3623 | 0.4275 | Expand | 0.8823 | 0.8133 | 0.9607 | 0.8809
Summary | 0.5946 | 0.5534 | 0.4968 | 0.5235 | Summary | 0.5045 | 0.5022 | 0.2725 | 0.3533
Refine | 0.5409 | 0.5217 | 0.3647 | 0.4292 | Refine | 0.8813 | 0.8185 | 0.9521 | 0.8801
Rewrite | 0.5424 | 0.5225 | 0.3658 | 0.4303 | Rewrite | 0.8853 | 0.8172 | 0.9625 | 0.8839
Translate | 0.6211 | 0.5736 | 0.5810 | 0.5772 | Translate | 0.5675 | 0.5385 | 0.4636 | 0.4982
Update | Complete | 0.7198 | 0.7102 | 0.7298 | 0.7175 | Complete | 0.9197 | 0.9546 | 0.8894 | 0.9192
QA | 0.7126 | 0.6363 | 0.7489 | 0.6880 | QA | 0.7946 | 0.7147 | 0.8762 | 0.7872
Polish | 0.9434 | 0.9872 | 0.9063 | 0.9429 | Polish | 0.9884 | 0.9978 | 0.9785 | 0.9884
Expand | 0.9586 | 0.9875 | 0.9332 | 0.9584 | Expand | 0.9942 | 0.9987 | 0.9899 | 0.9942
Summary | 0.7835 | 0.7037 | 0.8623 | 0.7749 | Summary | 0.6891 | 0.6192 | 0.7066 | 0.6600
Refine | 0.9899 | 0.9899 | 0.9900 | 0.9899 | Refine | 0.9908 | 0.9989 | 0.9833 | 0.9908
Rewrite | 0.9853 | 0.9894 | 0.9815 | 0.9853 | Rewrite | 0.9962 | 0.9987 | 0.9932 | 0.9962
Translate | 0.6135 | 0.5667 | 0.5516 | 0.5590 | Translate | 0.6559 | 0.5960 | 0.6437 | 0.6189
Delete | Complete | 0.6624 | 0.7376 | 0.5653 | 0.6358 | Complete | 0.8298 | 0.7757 | 0.8892 | 0.8282
QA | 0.7033 | 0.6392 | 0.7425 | 0.6869 | QA | 0.7614 | 0.6961 | 0.8245 | 0.7548
Polish | 0.8182 | 0.9882 | 0.6881 | 0.8084 | Polish | 0.9831 | 0.9985 | 0.9678 | 0.9831
Expand | 0.8793 | 0.9812 | 0.7835 | 0.8769 | Expand | 0.9833 | 0.9990 | 0.9686 | 0.9833
Summary | 0.9920 | 0.9956 | 0.9885 | 0.9920 | Summary | 0.9972 | 0.9991 | 0.9944 | 0.9972
Refine | 0.9898 | 0.9978 | 0.9809 | 0.9898 | Refine | 0.9890 | 0.9980 | 0.9807 | 0.9890
Rewrite | 0.9295 | 0.9853 | 0.8693 | 0.9290 | Rewrite | 0.9954 | 0.9987 | 0.9914 | 0.9954
Translate | 0.6025 | 0.7020 | 0.4759 | 0.5652 | Translate | 0.7543 | 0.7014 | 0.8068 | 0.7501
Rewrite | Complete | 0.6974 | 0.7261 | 0.6591 | 0.6860 | Complete | 0.9038 | 0.9362 | 0.7963 | 0.9032
QA | 0.7134 | 0.6374 | 0.7510 | 0.6895 | QA | 0.8652 | 0.7928 | 0.7746 | 0.8631
Polish | 0.9269 | 0.9927 | 0.8734 | 0.9258 | Polish | 0.9830 | 0.9935 | 0.9439 | 0.9830
Expand | 0.9522 | 0.9910 | 0.9187 | 0.9518 | Expand | 0.9877 | 0.9992 | 0.9644 | 0.9877
Summary | 0.7600 | 0.6816 | 0.8297 | 0.7483 | Summary | 0.5544 | 0.5290 | 0.6653 | 0.4490
Refine | 0.9894 | 0.9940 | 0.9850 | 0.9894 | Refine | 0.9892 | 0.9985 | 0.9906 | 0.9892
Rewrite | 0.9801 | 0.9906 | 0.9705 | 0.9801 | Rewrite | 0.9958 | 0.9989 | 0.9863 | 0.9958
Translate | 0.6237 | 0.5726 | 0.5683 | 0.5704 | Translate | 0.6143 | 0.5677 | 0.5802 | 0.5630
Translate | Complete | 0.6535 | 0.7022 | 0.6010 | 0.6474 | Complete | 0.5135 | 0.5069 | 0.2871 | 0.3665
QA | 0.7815 | 0.7779 | 0.7858 | 0.7799 | QA | 0.6258 | 0.5738 | 0.5726 | 0.5732
Polish | 0.6445 | 0.6963 | 0.5854 | 0.6357 | Polish | 0.5004 | 0.5002 | 0.2513 | 0.3345
Expand | 0.6726 | 0.7189 | 0.6240 | 0.6677 | Expand | 0.5010 | 0.5005 | 0.2524 | 0.3356
Summary | 0.4469 | 0.4203 | 0.4479 | 0.4333 | Summary | 0.9320 | 0.9121 | 0.9539 | 0.9319
Refine | 0.5957 | 0.6378 | 0.5240 | 0.5747 | Refine | 0.5010 | 0.5005 | 0.2528 | 0.3359
Rewrite | 0.6912 | 0.7266 | 0.6541 | 0.6877 | Rewrite | 0.5007 | 0.5004 | 0.2520 | 0.3352
Translate | 0.9925 | 0.9973 | 0.9878 | 0.9925 | Translate | 0.9908 | 0.9896 | 0.9920 | 0.9908
Table 8: The detection results of the RoBERTa model trained on a single main operation data on each LLM text. Train Data | Test Data | Chinese | Test Data | English
---|---|---|---|---
Accuracy | Precision | Recall | F1-score | Accuracy | Precision | Recall | F1-score
Create | Baichuan | 0.6368 | 0.6124 | 0.5403 | 0.5686 | Baichuan | 0.8056 | 0.7796 | 0.7889 | 0.7781
ChatGLM | 0.6411 | 0.6123 | 0.5434 | 0.5697 | ChatGLM | 0.8245 | 0.7855 | 0.8196 | 0.7948
GPT3.5 | 0.6506 | 0.6184 | 0.5585 | 0.5810 | GPT3.5 | 0.8215 | 0.7841 | 0.8215 | 0.7947
GPT4 | 0.6578 | 0.6219 | 0.5671 | 0.5866 | Llama3 | 0.8256 | 0.7856 | 0.8242 | 0.7971
Qwen | 0.6532 | 0.6189 | 0.5614 | 0.5819 | Llama2 | 0.8295 | 0.7882 | 0.8288 | 0.8009
| Average | 0.6479 | 0.6168 | 0.5542 | 0.5776 | Average | 0.8213 | 0.7846 | 0.8166 | 0.7931
Update | Baichuan | 0.8112 | 0.8131 | 0.7935 | 0.7996 | Baichuan | 0.8609 | 0.8569 | 0.8497 | 0.8512
ChatGLM | 0.8169 | 0.8181 | 0.7998 | 0.8055 | ChatGLM | 0.8770 | 0.8612 | 0.8780 | 0.8671
GPT3.5 | 0.8400 | 0.8211 | 0.8406 | 0.8289 | GPT3.5 | 0.8835 | 0.8578 | 0.8950 | 0.8750
GPT4 | 0.8617 | 0.8281 | 0.8763 | 0.8503 | Llama3 | 0.8802 | 0.8587 | 0.8854 | 0.8706
Qwen | 0.8619 | 0.8264 | 0.8795 | 0.8506 | Llama2 | 0.8915 | 0.8645 | 0.9048 | 0.8829
| Average | 0.8383 | 0.8214 | 0.8379 | 0.8270 | Average | 0.8786 | 0.8598 | 0.8826 | 0.8694
Delete | Baichuan | 0.7824 | 0.8725 | 0.6997 | 0.7670 | Baichuan | 0.8933 | 0.8915 | 0.8952 | 0.8918
ChatGLM | 0.8124 | 0.8885 | 0.7430 | 0.8036 | ChatGLM | 0.9186 | 0.8978 | 0.9397 | 0.9167
GPT3.5 | 0.7983 | 0.8536 | 0.7286 | 0.7810 | GPT3.5 | 0.9173 | 0.8959 | 0.9391 | 0.9158
GPT4 | 0.8587 | 0.8859 | 0.8199 | 0.8499 | Llama3 | 0.9130 | 0.8937 | 0.9320 | 0.9116
Qwen | 0.8589 | 0.8914 | 0.8176 | 0.8512 | Llama2 | 0.9164 | 0.9002 | 0.9336 | 0.9148
| Average | 0.8221 | 0.8784 | 0.7617 | 0.8105 | Average | 0.9117 | 0.8958 | 0.9279 | 0.9101
Rewrite | Baichuan | 0.7983 | 0.8109 | 0.7688 | 0.7839 | Baichuan | 0.8389 | 0.8487 | 0.7934 | 0.8186
ChatGLM | 0.8001 | 0.8186 | 0.7660 | 0.7872 | ChatGLM | 0.8623 | 0.8535 | 0.8363 | 0.8418
GPT3.5 | 0.8227 | 0.8181 | 0.8071 | 0.8088 | GPT3.5 | 0.8698 | 0.8512 | 0.8543 | 0.8506
GPT4 | 0.8637 | 0.8346 | 0.8739 | 0.8526 | Llama3 | 0.8675 | 0.8519 | 0.8474 | 0.8475
Qwen | 0.8671 | 0.8340 | 0.8814 | 0.8559 | Llama2 | 0.8701 | 0.8547 | 0.8504 | 0.8502
| Average | 0.8304 | 0.8233 | 0.8195 | 0.8177 | Average | 0.8617 | 0.8520 | 0.8364 | 0.8417
Translate | Baichuan | 0.6627 | 0.6945 | 0.6190 | 0.6524 | Baichuan | 0.6328 | 0.6221 | 0.4783 | 0.5259
ChatGLM | 0.7076 | 0.7277 | 0.6834 | 0.7032 | ChatGLM | 0.6370 | 0.6239 | 0.4841 | 0.5297
GPT3.5 | 0.6419 | 0.6679 | 0.5971 | 0.6289 | GPT3.5 | 0.6247 | 0.6206 | 0.4618 | 0.5169
GPT4 | 0.7372 | 0.7466 | 0.7248 | 0.7343 | Llama3 | 0.6345 | 0.6239 | 0.4774 | 0.5261
Qwen | 0.6747 | 0.7115 | 0.6319 | 0.6681 | Llama2 | 0.6369 | 0.6244 | 0.4823 | 0.5287
| Average | 0.6848 | 0.7096 | 0.6513 | 0.6774 | Average | 0.6332 | 0.6230 | 0.4768 | 0.5255

####  4.3.2 Cross-Operation Detection

Since metric-based models like MPU cannot be further trained, we only use
RoBERTa and XLNet for testing in cross-text generation operations to verify
their performance in cross-operation detection. To verify the adaptability of
the detection models across operations, we first pre-train the models on texts
from each of the five main operations proposed in this paper. Then, we test
their detection results across all operations. Given the diverse and complex
nature of text generation in real-world scenarios, we aim to train on data
from a single primary operation to identify the pre-trained model with optimal
performance and generalization capabilities. This approach will also offer
valuable insights and recommendations for the development and utilization of
detectors.

We divide into five groups based on five main operations, using the text of
one operation for training and testing on all sub operations. Table 7 shows
the detection results of the RoBERTa model trained on a single main operation
and then tested on each sub-operation. This experiment do not consider the
differences in LLMs’ texts, only distinguishing the detection differences of
texts generated by different operations. For Chinese, after training RoBERTa
on "Create" and "Translate" operation texts, its detection results are
satisfactory only on texts generated by those operations. However, its
performance on other operation texts is not satisfactory. For example, after
training RoBERTa on "Create" texts, the detection accuracy on other operation
texts is almost all below 0.7. This indicates that RoBERTa, after being
trained on "Create" and "Translate" operation texts, does not generalize well.
This also demonstrates that the features of texts generated by "Create" and
"Translate" operations differ significantly from those generated by other
operations. Training RoBERTa on these two types of texts can easily lead to
overfitting, thereby affecting its generalization ability in detecting other
texts. In contrast, training Roberta with texts generated through "Update,"
"Delete," and "Rewrite" operations significantly improves its detection
performance. For example, the Roberta model trained on "Update" operation
texts performs well on other texts except for a slightly lower performance on
"Translate" texts, where the accuracy does not exceed 0.7. In other cases, the
accuracy on four operation texts is above 0.94, indicating satisfactory
performance. This may be because the RoBERTa model, by learning from texts
generated by "Update" and "Delete" operations, can more easily capture the
internal differences between AI-generated texts and human-written texts. These
operation-generated texts are likely closer to human language patterns,
allowing RoBERTa to better learn to identify and distinguish between AI-
generated texts and human texts during training. This learning process
enhances the model’s generalization ability in detection tasks, resulting in
higher accuracy across various types of texts. RoBERTa’s overall performance
on English texts is better than that on Chinese texts, but the overall trend
of detection results is similar to that of Chinese texts. Specifically, after
being trained on "Create" and "Translate" operation texts, RoBERTa’s
performance still lags behind models trained on other types of texts. We
suspect this is because the texts generated by the LLMs based on "Create" and
"Translate" operations differ significantly from human texts, which affects
RoBERTa’s judgment ability on other types of texts.

((a)) Chinese

((b)) English

Figure 11: Radar charts of average detection results of XLNet model trained on
a single main operation data.

((a)) Chinese

((b)) English

Figure 12: Column charts comparing the comprehensive performance of RoBERTa
and XLNet for Cross-Operation detection.

Table 8 shows RoBERTa’s detection performance on texts generated by each LLM
after being trained on a single main operation text. The main purpose of this
experiment is to observe the differences in detection results of the RoBERTa
model on different LLMs texts after training on a single operation text. In
Chinese text experiments, the detection results of the RoBERTa model, after
being trained on texts from GPT4 and Qwen, are significantly better than those
from other LLMs. This is mainly because the texts generated by GPT4 and Qwen
are closer to human-written texts, allowing the detector to better capture
subtle differences. By simulating the naturalness and fluency of human
language, these two LLMs perform better in terms of text quality. The
performance of the RoBERTa model varies significantly based on the type of
training data used for different text operations. Notably, the detection
capabilities of the RoBERTa model trained on "Create" and "Translate"
operations texts are markedly inferior compared to those trained on "Update,"
"Delete," and "Rewrite" operations texts. Among them, the RoBERTa model
trained on "Update" texts performs the best, achieving a detection accuracy of
over 0.8 across all LLMs’ texts, reaching a level suitable for practical
application. This finding provides valuable insights for the development and
training of detection models. The primary reason for this superior performance
is that the "Update" operation involves minimal LLMs intervention, making it
easier for the detection model to distinguish between human-written and AI-
generated texts. The RoBERTa model performs better in English than in Chinese.
Except for the model trained on "Translate" texts, which shows poorer
detection results, the performance of models trained on other types of texts
is satisfactory. Relatively speaking, the RoBERTa model performs well in
detecting texts generated by ChatGLM, GPT3.5, Llama2, and Llama3, with minimal
differences between them. This also demonstrates that these four LLMs exhibit
strong performance in generating English texts, gradually approaching human-
level quality. Furthermore, the best-performing model is the RoBERTa model
trained with "Delete" text. This also provides direction for the practical
application and training of RoBERTa.

Table 9: The detection results of the XLNet model trained on a single LLM data on each sub-operation text. Train Data | Test Data | Chinese | Train Data | Test Data | English
---|---|---|---|---|---
Accuracy | Precision | Recall | F1-score | Accuracy | Precision | Recall | F1-score
Baichuan | Complete | 0.8360 | 0.8164 | 0.8568 | 0.8358 | Baichuan | Complete | 0.9304 | 0.9431 | 0.9201 | 0.9302
QA | 0.9714 | 0.9712 | 0.9723 | 0.9714 | QA | 0.9640 | 0.9400 | 0.9892 | 0.9640
Polish | 0.5490 | 0.5263 | 0.3874 | 0.4463 | Polish | 0.9584 | 0.9620 | 0.9558 | 0.9584
Expand | 0.5490 | 0.5262 | 0.3837 | 0.4438 | Expand | 0.9674 | 0.9600 | 0.9751 | 0.9674
Summary | 0.9232 | 0.9542 | 0.9008 | 0.9213 | Summary | 0.9610 | 0.9841 | 0.9419 | 0.9606
Refine | 0.5530 | 0.5281 | 0.3835 | 0.4443 | Refine | 0.9587 | 0.9603 | 0.9580 | 0.9587
Rewrite | 0.5528 | 0.5281 | 0.3860 | 0.4460 | Rewrite | 0.9679 | 0.9616 | 0.9747 | 0.9679
Translate | 0.9618 | 0.9643 | 0.9596 | 0.9617 | Translate | 0.8995 | 0.8730 | 0.9277 | 0.8994
ChatGLM | Complete | 0.8190 | 0.7954 | 0.8439 | 0.8186 | ChatGLM | Complete | 0.9387 | 0.9276 | 0.9508 | 0.9387
QA | 0.9727 | 0.9666 | 0.9793 | 0.9727 | QA | 0.9498 | 0.9673 | 0.9345 | 0.9497
Polish | 0.5686 | 0.5376 | 0.4349 | 0.4808 | Polish | 0.8760 | 0.8170 | 0.9426 | 0.8749
Expand | 0.5696 | 0.5380 | 0.4335 | 0.4801 | Expand | 0.8789 | 0.8149 | 0.9509 | 0.8776
Summary | 0.9270 | 0.9761 | 0.8897 | 0.9245 | Summary | 0.9839 | 0.9969 | 0.9719 | 0.9839
Refine | 0.5778 | 0.5425 | 0.4471 | 0.4902 | Refine | 0.8775 | 0.8111 | 0.9528 | 0.8761
Rewrite | 0.5738 | 0.5401 | 0.4347 | 0.4817 | Rewrite | 0.8867 | 0.8184 | 0.9644 | 0.8854
Translate | 0.8310 | 0.9874 | 0.7162 | 0.8240 | Translate | 0.8475 | 0.9572 | 0.7621 | 0.8434
GPT3.5 | Complete | 0.8018 | 0.7503 | 0.8580 | 0.7997 | GPT3.5 | Complete | 0.9157 | 0.9831 | 0.8600 | 0.9144
QA | 0.9730 | 0.9730 | 0.9735 | 0.9730 | QA | 0.9722 | 0.9626 | 0.9820 | 0.9722
Polish | 0.6462 | 0.5913 | 0.6357 | 0.6125 | Polish | 0.9430 | 0.9864 | 0.9060 | 0.9425
Expand | 0.6428 | 0.5879 | 0.6234 | 0.6050 | Expand | 0.9609 | 0.9884 | 0.9356 | 0.9608
Summary | 0.9235 | 0.9610 | 0.8952 | 0.9216 | Summary | 0.9802 | 0.9778 | 0.9829 | 0.9802
Refine | 0.6671 | 0.6019 | 0.6591 | 0.6292 | Refine | 0.9534 | 0.9850 | 0.9263 | 0.9531
Rewrite | 0.6620 | 0.5990 | 0.6511 | 0.6240 | Rewrite | 0.9706 | 0.9899 | 0.9529 | 0.9705
Translate | 0.9259 | 0.9910 | 0.8726 | 0.9243 | Translate | 0.8658 | 0.8136 | 0.9231 | 0.8645
GPT4 | Complete | 0.8350 | 0.9008 | 0.7829 | 0.8222 | Llama3 | Complete | 0.9314 | 0.9410 | 0.9242 | 0.9312
QA | 0.9710 | 0.9888 | 0.9547 | 0.9709 | QA | 0.9527 | 0.9242 | 0.9830 | 0.9527
Polish | 0.8040 | 0.8890 | 0.7366 | 0.7921 | Polish | 0.9600 | 0.9671 | 0.9546 | 0.9599
Expand | 0.8575 | 0.9090 | 0.8186 | 0.8507 | Expand | 0.9737 | 0.9721 | 0.9755 | 0.9737
Summary | 0.9068 | 0.9399 | 0.8809 | 0.9056 | Summary | 0.9544 | 0.9683 | 0.9438 | 0.9541
Refine | 0.9382 | 0.9291 | 0.9487 | 0.9382 | Refine | 0.9661 | 0.9704 | 0.9627 | 0.9660
Rewrite | 0.9002 | 0.9218 | 0.8856 | 0.8990 | Rewrite | 0.9718 | 0.9663 | 0.9775 | 0.9718
Translate | 0.9266 | 0.9901 | 0.8750 | 0.9248 | Translate | 0.8431 | 0.7831 | 0.9092 | 0.8412
Qwen | Complete | 0.6972 | 0.8855 | 0.5200 | 0.6241 | Llama2 | Complete | 0.9422 | 0.9767 | 0.9124 | 0.9419
QA | 0.9079 | 0.9870 | 0.8465 | 0.9052 | QA | 0.9448 | 0.9456 | 0.9447 | 0.9448
Polish | 0.6886 | 0.9732 | 0.4993 | 0.6110 | Polish | 0.9556 | 0.9730 | 0.9409 | 0.9554
Expand | 0.7270 | 0.9800 | 0.5575 | 0.6718 | Expand | 0.9698 | 0.9757 | 0.9644 | 0.9698
Summary | 0.8200 | 0.8461 | 0.7995 | 0.8150 | Summary | 0.9567 | 0.9928 | 0.9274 | 0.9560
Refine | 0.8367 | 0.9934 | 0.7259 | 0.8236 | Refine | 0.9610 | 0.9741 | 0.9498 | 0.9609
Rewrite | 0.7611 | 0.9881 | 0.6107 | 0.7217 | Rewrite | 0.9699 | 0.9738 | 0.9668 | 0.9699
Translate | 0.9237 | 0.9622 | 0.8931 | 0.9223 | Translate | 0.8890 | 0.8622 | 0.9181 | 0.8888

The average test results of XLNet model after training on single-operation
texts are shown in Figure 11. From the Chinese detection results, XLNet
performs weaker after being trained on "Create" and "Translate" operation
texts, but shows stronger performance after being trained on "Update,"
"Delete," and "Rewrite" operation texts. This also validates our hypothesis
and conclusion from the RoBERTa experiments, namely that the texts generated
by "Create" and "Translate" operations differ significantly from human-
generated texts, leading to overfitting of the detector. XLNet’s detection
performance in English is similar to that of RoBERTa. Apart from the model
trained with "Translate" texts, which shows relatively poor performance, the
other models perform well and have comparable results. Among them, the XLNet
model trained on texts generated by the "Delete" operation still achieved the
best detection performance. Figure 12 provides a comprehensive comparison of
the overall performance of the two detection models in Cross-Operation
scenarios. As shown in this figure, RoBERTa outperforms XLNet in detecting
both Chinese and English texts. This indicates that in practical detection
model training and research, it is more advisable to prioritize the
development of the RoBERTa model when the variety of operation texts is
limited.

####  4.3.3 Cross-LLM Detection

Due to the significant differences in the texts generated by different LLMs,
we conducted cross-LLM detection experiments to explore the impact of these
differences on the detector. Specifically, we trained the detector on all
operation texts generated by a specific LLM and then tested its detection
accuracy and generalization performance on texts generated by other LLMs.
Similarly, we also employed two model-based detectors, RoBERTa and XLNet, for
the experiment.

Table 10: The detection results of the XLNet model trained on a single LLM data on each LLM text. Train Data | Test Data | Chinese | Train Data | Test Data | English
---|---|---|---|---|---
Accuracy | Precision | Recall | F1-score | Accuracy | Precision | Recall | F1-score
Baichuan | Baichuan | 0.7341 | 0.7138 | 0.6624 | 0.6826 | Baichuan | Baichuan | 0.9407 | 0.9416 | 0.9411 | 0.9405
ChatGLM | 0.7289 | 0.7267 | 0.6378 | 0.6762 | ChatGLM | 0.9375 | 0.9482 | 0.9287 | 0.9373
GPT3.5 | 0.7506 | 0.7336 | 0.6724 | 0.6976 | GPT3.5 | 0.9497 | 0.9481 | 0.9543 | 0.9494
GPT4 | 0.7501 | 0.7324 | 0.6692 | 0.6946 | Llama3 | 0.9631 | 0.9505 | 0.9763 | 0.9631
Qwen | 0.7217 | 0.7278 | 0.6270 | 0.6681 | Llama2 | 0.9638 | 0.9518 | 0.9761 | 0.9637
| Average | 0.7370 | 0.7269 | 0.6538 | 0.6838 |  | Average | 0.9509 | 0.9480 | 0.9553 | 0.9508
ChatGLM | Baichuan | 0.7237 | 0.7345 | 0.6402 | 0.6816 | ChatGLM | Baichuan | 0.8851 | 0.8873 | 0.8892 | 0.8842
ChatGLM | 0.7509 | 0.7282 | 0.6887 | 0.7052 | ChatGLM | 0.9221 | 0.8830 | 0.9663 | 0.9213
GPT3.5 | 0.7314 | 0.7363 | 0.6467 | 0.6845 | GPT3.5 | 0.8977 | 0.8906 | 0.9161 | 0.8957
GPT4 | 0.7383 | 0.7422 | 0.6571 | 0.6920 | Llama3 | 0.9045 | 0.8911 | 0.9274 | 0.9031
Qwen | 0.7055 | 0.7361 | 0.6044 | 0.6570 | Llama2 | 0.9152 | 0.8919 | 0.9447 | 0.9143
| Average | 0.7299 | 0.7355 | 0.6474 | 0.6841 |  | Average | 0.9049 | 0.8888 | 0.9287 | 0.9037
GPT3.5 | Baichuan | 0.7740 | 0.7534 | 0.7638 | 0.7580 | GPT3.5 | Baichuan | 0.9212 | 0.9623 | 0.8881 | 0.9206
ChatGLM | 0.7456 | 0.7490 | 0.7160 | 0.7280 | ChatGLM | 0.9180 | 0.9626 | 0.8809 | 0.9172
GPT3.5 | 0.8017 | 0.7578 | 0.8090 | 0.7818 | GPT3.5 | 0.9657 | 0.9485 | 0.9839 | 0.9652
GPT4 | 0.8024 | 0.7634 | 0.8034 | 0.7820 | Llama3 | 0.9676 | 0.9650 | 0.9714 | 0.9674
Qwen | 0.7779 | 0.7611 | 0.7632 | 0.7562 | Llama2 | 0.9537 | 0.9660 | 0.9437 | 0.9536
| Average | 0.7803 | 0.7569 | 0.7711 | 0.7612 |  | Average | 0.9452 | 0.9609 | 0.9336 | 0.9448
GPT4 | Baichuan | 0.8385 | 0.9226 | 0.7677 | 0.8304 | Llama3 | Baichuan | 0.9286 | 0.9377 | 0.9222 | 0.9282
ChatGLM | 0.7780 | 0.9073 | 0.6715 | 0.7649 | ChatGLM | 0.9304 | 0.9350 | 0.9269 | 0.9301
GPT3.5 | 0.9362 | 0.9455 | 0.9297 | 0.9356 | GPT3.5 | 0.9463 | 0.9377 | 0.9581 | 0.9459
GPT4 | 0.9645 | 0.9439 | 0.9868 | 0.9645 | Llama3 | 0.9557 | 0.9314 | 0.9815 | 0.9552
Qwen | 0.9450 | 0.9486 | 0.9461 | 0.9442 | Llama2 | 0.9599 | 0.9409 | 0.9803 | 0.9596
| Average | 0.8924 | 0.9336 | 0.8604 | 0.8879 |  | Average | 0.9442 | 0.9366 | 0.9538 | 0.9438
Qwen | Baichuan | 0.6991 | 0.9164 | 0.5301 | 0.6393 | Llama2 | Baichuan | 0.9270 | 0.9608 | 0.8973 | 0.9267
ChatGLM | 0.6198 | 0.9260 | 0.4018 | 0.5390 | ChatGLM | 0.9325 | 0.9593 | 0.9083 | 0.9323
GPT3.5 | 0.7821 | 0.9711 | 0.6529 | 0.7575 | GPT3.5 | 0.9517 | 0.9632 | 0.9444 | 0.9512
GPT4 | 0.8999 | 0.9806 | 0.8368 | 0.8980 | Llama3 | 0.9667 | 0.9616 | 0.9724 | 0.9667
Qwen | 0.9755 | 0.9656 | 0.9862 | 0.9754 | Llama2 | 0.9655 | 0.9515 | 0.9805 | 0.9653
| Average | 0.7953 | 0.9519 | 0.6816 | 0.7619 |  | Average | 0.9487 | 0.9593 | 0.9406 | 0.9484

Table 9 shows the results of training the XLNet model on a single LLM text and
testing it on text generated by various operations. After training on Chinese
texts generated by five different LLMs, the XLNet model shows significant
performance variation. The results in this table indicate that the detection
performance of XLNet trained on Baichuan and ChatGLM texts is notably weaker
than that of the models trained on texts from the other three LLMs. This may
be because the Chinese texts generated by GPT3.5, GPT4, and Qwen models are
closer to human-level, allowing XLNet to more effectively capture the subtle
differences between AI-generated and human-generated texts during training,
resulting in superior performance during testing. This is consistent with our
established understanding that GPT3.5, GPT4, and Qwen models excel in
generating Chinese text. In particular, GPT4 frequently ranks at the top of
various performance leaderboards for LLMs. Observing the detection performance
of the XLNet model on various sub-operation texts, it is found that the model
typically achieves higher accuracy on "QA", "Summary", and "Translate" texts,
and this consistency is seen across texts generated by different LLMs. This
could be because these operations often introduce unique structures and
language patterns in the Chines text. For example, translated Chinese texts
may contain unnatural expressions or vocabulary usage, summaries typically
simplify and condense information, and QA texts might display a specific
question-and-answer format. These features make it easier for the XLNet model
to distinguish between AI-generated and human-generated texts. From the
English detection results, the detection performance of XLNet trained on texts
generated by various LLMs shows little difference. This suggests that the
performance of these five LLMs in generating English text is quite impressive,
and they are gradually approaching human-level proficiency in English text
generation. From the detection results of each sub-operation text, it can be
seen that, except for the "Translate" operation, all other operations have a
detection accuracy above 0.9. This may be because the translated text in terms
of grammar and vocabulary is very similar to human-translated text, making it
difficult for the XLNet to distinguish between machine translation and human
translation. Machine translation technology is already quite mature and can
generate text that conforms to the grammar and style of the target language,
reducing the effectiveness of the XLNet.

((a)) Chinese

((b)) English

Figure 13: Radar charts of average detection results of RoBERTa model trained
on a single LLM data.

((a)) Chinese

((b)) English

Figure 14: Column charts comparing the comprehensive performance of RoBERTa
and XLNet for Cross-LLM detection.

To explore the generalization capability of the detector on texts from
different LLMs, this paper conducted the experiment: first training on texts
generated by a single LLM, and then testing on texts generated by other LLMs.
The specific experimental results are shown in Table 10. The Chinese detection
results indicate that texts generated by GPT4 and GPT3.5 have significantly
improved the performance and generalization ability of XLNet. We believe this
is mainly because the texts generated by the GPT series make it easier for
XLNet to capture the subtle differences between AI-generated and human-
generated texts. Specifically, the texts generated by GPT4 not only enable
XLNet to perform well on GPT series models but also achieve satisfactory
results in detecting texts from other types of LLMs. It is worth noting that
although training on Qwen texts resulted in a high overall detection rate for
XLNet, this are primarily due to its high accuracy in detecting its own texts.
The performance of XLNet trained on Qwen texts is not satisfactory on Baichuan
and ChatGLM texts, with detection accuracy even failing to reach 0.7.
Therefore, texts generated by Qwen are not suitable for use as training texts
in practical detector training. Similarly, XLNet performs better in detecting
English texts. The detection results show that the accuracy for almost all
LLMs generated English texts is above 0.9. This indicates that the English
texts generated by various LLMs are relatively consistent, and the technology
for generating English texts has become quite mature.

The average test results of RoBERTa model after training on single-operation
texts are shown in Figure 13. From the Chinese detection results, RoBERTa
maintains high performance after being trained on GPT4 and Qwen texts. Unlike
the XLNet model, RoBERTa shows a certain drop in detection accuracy after
being trained on GPT3.5 texts compared to the previous two LLMs. This may be
because GPT4, as a more advanced model, generates text with quality and
complexity closer to human level, providing clearer distinguishing signals.
The Qwen model may have similar advantages in some aspects. However, the texts
generated by GPT3.5 may not exhibit the same subtle differences as GPT4,
making it difficult for RoBERTa to effectively recognize these differences
during training. In terms of English text detection, RoBERTa also performs
exceptionally well. Except for the model trained on ChatGLM texts, which shows
slightly weaker performance, RoBERTa maintains the detection accuracy of
around 0.95 when trained on texts from other LLMs, achieving satisfactory
results. Figure 14 presents the average detection results of these two
detection models in cross-LLM scenarios. Experimental results show that XLNet
performs slightly better in Chinese text detection, while RoBERTa has an edge
in English text detection. Therefore, when training on texts from only one
LLM, the choice of detection model should be based on the actual language
environment and specific requirements of the application.

###  4.4 Discussions and Suggestions

Our comprehensive evaluation of AI-generated text detection systems highlights
several key findings that offer critical insights for the advancement of this
field. The experiments, focused on Cross-Dataset, Cross-Operation, and Cross-
LLM detection, provide a nuanced understanding of the strengths and
limitations of various detection models and configurations. The Cross-Dataset
experiments underscore the importance of training data diversity. The MPU
model, for instance, demonstrated superior performance in detecting texts
generated by GPT series models, likely due to the familiarity with the text
patterns from the HC3 dataset. This dataset’s predominance of ChatGPT
generated texts likely contributed to MPU’s higher accuracy. However, MPU
struggled with texts generated by Baichuan and ChatGLM, indicating a need for
more diverse training data that includes outputs from a broader range of LLMs.
Although RoBERTa is generally effective, its performance significantly
decreases when encountering unseen datasets, indicating that its
generalization ability is limited compared to MPU. This suggests that future
research should focus on developing training datasets that encompass a wider
variety of text sources to enhance generalization. The Cross-Operation
experiments revealed significant variability in detection performance based on
the type of text generation operation. Models trained on "Create" and
"Translate" tasks exhibited poorer generalization, likely due to the distinct
features of texts generated by these operations. In contrast, training on
"Update," "Delete," and "Rewrite" operations yielded better performance across
various operations. This suggests that future detectors should prioritize
training on operations that produce texts closer to human language patterns.
The Cross-LLM detection results highlight the significant impact of the source
LLM on detection accuracy. Models trained on Chinese texts from advanced LLMs
like GPT4 and Qwen showed superior performance, underscoring these models’
ability to generate texts that capture subtle differences from human-written
texts. However, the performance drop with texts from Baichuan and ChatGLM
indicates that current detectors may not be fully equipped to handle the
variability in text generation quality across different LLMs. This calls for
the development of more robust detection models that can adapt to the unique
characteristics of texts from diverse LLMs. Moreover, the relatively better
performance of models on English texts suggests a potential area for
improvement in handling Chinese texts, possibly by incorporating linguistic
and cultural nuances into the training process.

To enhance the robustness and accuracy of AI-generated text detection systems,
several key strategies should be pursued. First, training datasets should be
diversified to include texts from a broad array of LLMs and text generation
operations, which will enhance the generalization capabilities of detection
models. Incorporating multi-task learning approaches can help models learn
shared features across different tasks, improving their adaptability to varied
text generation scenarios. For non-English texts, especially Chinese, it’s
crucial to integrate linguistic and cultural nuances into the training data to
significantly enhance detection performance. Regular updates to detection
models with data from the latest LLMs are also essential to keep up with
evolving text generation technologies. Furthermore, expanding research into
cross-lingual detection methods can offer valuable insights for developing
models that perform robustly across different languages, potentially through
the use of transfer learning techniques and multilingual training datasets. By
addressing these areas, future research can foster the development of more
robust and versatile AI-generated text detection systems.

##  5 Conclusion and Future Work

In this paper, we propose CUDRT, a novel benchmark designed to rigorously
evaluate the performance of AI-generated text detectors across diverse text
generation scenarios in both Chinese and English. Our findings demonstrate
that CUDRT effectively addresses gaps in current evaluation methods by
incorporating a broader spectrum of text generation operations and linguistic
variations. The benchmark highlights the challenges in detecting AI-generated
text across various domains and languages, emphasizing the need for more
sophisticated detection methods. Our experimental results underscore the
importance of developing nuanced detection mechanisms that can adapt to the
evolving capabilities of large language models (LLMs). The diversity of CUDRT
scenarios from text completion to translation provides a comprehensive testing
ground for AI-generated text detectors, offering insights into their strengths
and limitations. This research opens new avenues for developing more robust,
language-inclusive detection systems. By incorporating real-world applications
and cross-linguistic data, CUDRT sets a new standard in evaluating text
generation detection technologies, encouraging a move towards more globally
inclusive AI technologies that recognize and adapt to linguistic diversity and
the nuanced nature of human language.

As AI technology continues to advance, the development of benchmarks like
CUDRT will be crucial for ensuring the safe, ethical, and effective use of
LLMs across industries and communities worldwide. Future research should focus
on expanding the scope of the benchmark to include more languages and LLMs
types, further enhancing the utility and relevance of detection tools in a
rapidly evolving digital landscape.

*Acknowledgments This work was supported by the National Natural Science Foundation of China (Grant No. 72271233), Suzhou Key Laboratory of Artificial Intelligence and Social Governance Technologies (SZS2023007), Smart Social Governance Technology and Innovative Application Platform (YZCXPT2023101).

*Conflict of interest

The authors declare no potential conflict of interests.

## References

  * Achiam . 2023 achiam2023gptAchiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, FL.others  2023\.  Gpt-4 technical report Gpt-4 technical report.  arXiv preprint arXiv:2303.08774.
  * Alowais . 2023 2023revolutionizingAlowais, SA., Alghamdi, SS., Alsuhebany, N., Alqahtani, T., Alshaya, AI., Almohareb, SN.others  2023\.  Revolutionizing healthcare: the role of artificial intelligence in clinical practice Revolutionizing healthcare: the role of artificial intelligence in clinical practice.  BMC medical education231689.
  * Bahak . 2023 2023evaluatingBahak, H., Taheri, F., Zojaji, Z. Kazemi, A.  2023\.  Evaluating ChatGPT as a Question Answering System: A Comprehensive Analysis and Comparison with Existing Models Evaluating chatgpt as a question answering system: A comprehensive analysis and comparison with existing models.  arXiv preprint arXiv:2312.07592.
  * J. Bai . 2023 bai2023qwenBai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X.others  2023\.  Qwen technical report Qwen technical report.  arXiv preprint arXiv:2309.16609.
  * Y. Bai . 2024 bai2024benchmarkingBai, Y., Ying, J., Cao, Y., Lv, X., He, Y., Wang, X.others  2024\.  Benchmarking foundation models with language-model-as-an-examiner Benchmarking foundation models with language-model-as-an-examiner.  Advances in Neural Information Processing Systems36.
  * Chang . 2024 chang2024surveyChang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K.others  2024\.  A survey on evaluation of large language models A survey on evaluation of large language models.  ACM Transactions on Intelligent Systems and Technology1531–45.
  * Chen . 2023 chen2023canChen, Y., Arunasalam, A. Celik, ZB.  2023\.  Can large language models provide security & privacy advice? measuring the ability of llms to refute misconceptions Can large language models provide security & privacy advice? measuring the ability of llms to refute misconceptions.  Proceedings of the 39th Annual Computer Security Applications Conference Proceedings of the 39th annual computer security applications conference ( 366–378).
  * Clusmann . 2023 clusmann2023futureClusmann, J., Kolbinger, FR., Muti, HS., Carrero, ZI., Eckardt, JN., Laleh, NG.others  2023\.  The future landscape of large language models in medicine The future landscape of large language models in medicine.  Communications medicine31141.
  * Derner Batistič 2023 derner2023beyondDerner, E. Batistič, K.  2023\.  Beyond the safeguards: Exploring the security risks of chatgpt Beyond the safeguards: Exploring the security risks of chatgpt.  arXiv preprint arXiv:2305.08005.
  * Du . 2022 glmDu, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z. Tang, J.  2022\.  GLM: General Language Model Pretraining with Autoregressive Blank Infilling GLM: General language model pretraining with autoregressive blank infilling.  Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers) ( 320–335).  Association for Computational Linguistics.
  * Gao . 2024 gao2024llmGao, C., Chen, D., Zhang, Q., Huang, Y., Wan, Y. Sun, L.  2024\.  Llm-as-a-coauthor: The challenges of detecting llm-human mixcase Llm-as-a-coauthor: The challenges of detecting llm-human mixcase.  arXiv preprint arXiv:2401.05952.
  * Gehrmann . 2019 gltrGehrmann, S., Strobelt, H. Rush, AM.  2019\.  Gltr: Statistical detection and visualization of generated text Gltr: Statistical detection and visualization of generated text.  arXiv preprint arXiv:1906.04043.
  * Guo . 2023 guo2023closeGuo, B., Zhang, X., Wang, Z., Jiang, M., Nie, J., Ding, Y.Wu, Y.  2023\.  How close is chatgpt to human experts? comparison corpus, evaluation, and detection How close is chatgpt to human experts? comparison corpus, evaluation, and detection.  arXiv preprint arXiv:2301.07597.
  * He . 2023 he2023mgtbenchHe, X., Shen, X., Chen, Z., Backes, M. Zhang, Y.  2023\.  Mgtbench: Benchmarking machine-generated text detection Mgtbench: Benchmarking machine-generated text detection.  arXiv preprint arXiv:2303.14822.
  * Heidt 2023 heidt2023artificialHeidt, A.  2023\.  Artificial-intelligence search engines wrangle academic literature Artificial-intelligence search engines wrangle academic literature.  Nature6207973456–457.
  * Kaddour . 2023 kaddour2023challengesKaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R. McHardy, R.  2023\.  Challenges and applications of large language models Challenges and applications of large language models.  arXiv preprint arXiv:2307.10169.
  * Khasawneh jadallah abed Khasawneh 2023 khasawneh2023rolesKhasawneh, MAS. jadallah abed Khasawneh, Y.  2023\.  The Roles Of Formulaic Sequences And Discourse Markers In Academic Writing; Insights From Lecturers And Other Researchers The roles of formulaic sequences and discourse markers in academic writing; insights from lecturers and other researchers.  Journal of Namibian Studies: History Politics Culture347102–7122.
  * Li . 2024 li2024flexkbqaLi, Z., Fan, S., Gu, Y., Li, X., Duan, Z., Dong, B.Wang, J.  2024\.  Flexkbqa: A flexible llm-powered framework for few-shot knowledge base question answering Flexkbqa: A flexible llm-powered framework for few-shot knowledge base question answering.  ( 18608–18616).  Proceedings of the AAAI Conference on Artificial Intelligence.
  * Liang . 2023 liang2023uhgevalLiang, X., Song, S., Niu, S., Li, Z., Xiong, F., Tang, B.others  2023\.  Uhgeval: Benchmarking the hallucination of chinese large language models via unconstrained generation Uhgeval: Benchmarking the hallucination of chinese large language models via unconstrained generation.  arXiv preprint arXiv:2311.15296.
  * G. Liu . 2024 liu2024semanticLiu, G., Du, H., Niyato, D., Kang, J., Xiong, Z., Kim, DI. Shen, X.  2024\.  Semantic communications for artificial intelligence generated content (AIGC) toward effective content creation Semantic communications for artificial intelligence generated content (aigc) toward effective content creation.  IEEE Network.
  * X. Liu . 2023 2023-cocoLiu, X., Zhang, Z., Wang, Y., Pu, H., Lan, Y. Shen, C.  2023\.  CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Low Resource With Contrastive Learning CoCo: Coherence-enhanced machine-generated text detection under low resource with contrastive learning.  Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing Proceedings of the 2023 conference on empirical methods in natural language processing ( 16167–16188).  Association for Computational Linguistics.
  * Y. Liu . 2019 liu2019robertaLiu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D.Stoyanov, V.  2019\.  Roberta: A robustly optimized bert pretraining approach Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692.
  * Macko . 2023 macko2023multitudeMacko, D., Moro, R., Uchendu, A., Lucas, JS., Yamashita, M., Pikuliak, M.others  2023\.  MULTITuDE: Large-Scale Multilingual Machine-Generated Text Detection Benchmark Multitude: Large-scale multilingual machine-generated text detection benchmark.  arXiv preprint arXiv:2310.13606.
  * Maia . 2018 maia201818Maia, M., Handschuh, S., Freitas, A., Davis, B., McDermott, R., Zarrouk, M. Balahur, A.  2018\.  Www’18 open challenge: financial opinion mining and question answering Www’18 open challenge: financial opinion mining and question answering.  ( 1941–1942).  Companion proceedings of the the web conference 2018.
  * Manovich 2002 manovich2002languageManovich, L.  2002\.  The language of new media The language of new media.  MIT press.
  * Marvin . 2023 marvin2023promptMarvin, G., Hellen, N., Jjingo, D. Nakatumba-Nabende, J.  2023\.  Prompt Engineering in Large Language Models Prompt engineering in large language models.  ( 387–402).  International Conference on Data Intelligence and Cognitive Informatics.
  * Mitchell . 2023 2023detectgptMitchell, E., Lee, Y., Khazatsky, A., Manning, CD. Finn, C.  2023\.  Detectgpt: Zero-shot machine-generated text detection using probability curvature Detectgpt: Zero-shot machine-generated text detection using probability curvature.  ( 24950–24962).  International Conference on Machine Learning.
  * Nam . 2024 nam2024usingNam, D., Macvean, A., Hellendoorn, V., Vasilescu, B. Myers, B.  2024\.  Using an llm to help with code understanding Using an llm to help with code understanding.  ( 1–13).  Proceedings of the IEEE/ACM 46th International Conference on Software Engineering.
  * Ni . 2023 ni2023recentNi, J., Young, T., Pandelea, V., Xue, F. Cambria, E.  2023\.  Recent advances in deep learning based dialogue systems: A systematic survey Recent advances in deep learning based dialogue systems: A systematic survey.  Artificial intelligence review5643055–3155.
  * Pan . 2023 pan2023riskPan, Y., Pan, L., Chen, W., Nakov, P., Kan, MY. Wang, WY.  2023\.  On the risk of misinformation pollution with large language models On the risk of misinformation pollution with large language models.  arXiv preprint arXiv:2305.13661.
  * Peng . 2024 peng2024hiddingPeng, X., Zhou, Y., He, B., Sun, L. Sun, Y.  2024\.  Hidding the Ghostwriters: An Adversarial Evaluation of AI-Generated Student Essay Detection Hidding the ghostwriters: An adversarial evaluation of ai-generated student essay detection.  arXiv preprint arXiv:2402.00412.
  * Rouf . 2024 rouf2024instantopsRouf, R., Rasolroveicy, M., Litoiu, M., Nagar, S., Mohapatra, P., Gupta, P. Watts, I.  2024\.  InstantOps: A Joint Approach to System Failure Prediction and Root Cause Identification in Microserivces Cloud-Native Applications Instantops: A joint approach to system failure prediction and root cause identification in microserivces cloud-native applications.  ( 119–129).  Proceedings of the 15th ACM/SPEC International Conference on Performance Engineering.
  * Sarzaeim . 2023 sarzaeim2023frameworkSarzaeim, P., Doshi, A. Mahmoud, Q.  2023\.  A Framework for Detecting AI-Generated Text in Research Publications A framework for detecting ai-generated text in research publications.  ( 121–127).  Proceedings of the International Conference on Advanced Technologies.
  * Sel . 2024 sel2024skinSel, B., Shanmugasundaram, P., Kachuee, M., Zhou, K., Jia, R. Jin, M.  2024\.  Skin-in-the-Game: Decision Making via Multi-Stakeholder Alignment in LLMs Skin-in-the-game: Decision making via multi-stakeholder alignment in llms.  arXiv preprint arXiv:2405.12933.
  * Shi . 2023 shi2023chatgraphShi, Y., Ma, H., Zhong, W., Tan, Q., Mai, G., Li, X.Huang, J.  2023\.  Chatgraph: Interpretable text classification by converting chatgpt knowledge to graphs Chatgraph: Interpretable text classification by converting chatgpt knowledge to graphs.  ( 515–520).  IEEE International Conference on Data Mining Workshops (ICDMW).
  * Shu . 2024 shu2024rewritelmShu, L., Luo, L., Hoskere, J., Zhu, Y., Liu, Y., Tong, S.Meng, L.  2024\.  Rewritelm: An instruction-tuned large language model for text rewriting Rewritelm: An instruction-tuned large language model for text rewriting.  ( 18970–18980).  Proceedings of the AAAI Conference on Artificial Intelligence.
  * Stokel-Walker Van Noorden 2023 stokel2023chatgptStokel-Walker, C. Van Noorden, R.  2023\.  What ChatGPT and generative AI mean for science What chatgpt and generative ai mean for science.  Nature6147947214–216.
  * J. Su . 2023 su2023detectllmSu, J., Zhuo, TY., Wang, D. Nakov, P.  2023\.  Detectllm: Leveraging log rank information for zero-shot detection of machine-generated text Detectllm: Leveraging log rank information for zero-shot detection of machine-generated text.  arXiv preprint arXiv:2306.05540.
  * Z. Su . 2023 su2023hc3Su, Z., Wu, X., Zhou, W., Ma, G. Hu, S.  2023\.  Hc3 plus: A semantic-invariant human chatgpt comparison corpus Hc3 plus: A semantic-invariant human chatgpt comparison corpus.  arXiv preprint arXiv:2309.02731.
  * Team 2024 llama3Team, MA.  2024\.  Meta Llama 3: The most capable openly available LLM to date Meta llama 3: The most capable openly available llm to date.  Hugging Face.  https://huggingface.co/meta-llama/Meta-Llama-3
  * Tian . 2023 tian2023multiscaleTian, Y., Chen, H., Wang, X., Bai, Z., Zhang, Q., Li, R.Wang, Y.  2023\.  Multiscale positive-unlabeled detection of ai-generated texts Multiscale positive-unlabeled detection of ai-generated texts.  arXiv preprint arXiv:2305.18149.
  * Touvron . 2023 touvron2023llamaTouvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, MA., Lacroix, T.others  2023\.  Llama: Open and efficient foundation language models Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971.
  * P. Wang . 2023 seqxgptWang, P., Li, L., Ren, K., Jiang, B., Zhang, D. Qiu, X.  202312\.  SeqXGPT: Sentence-Level AI-Generated Text Detection SeqXGPT: Sentence-level AI-generated text detection.  ( 1144–1156).  Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.
  * Y. Wang 2023 wang2023largeWang, Y.  2023\.  Large Language Models Evaluate Machine Translation via Polishing Large language models evaluate machine translation via polishing.  ( 158–163).  2023 6th International Conference on Algorithms, Computing and Artificial Intelligence.
  * Y. Wang . 2023 wang2023m4Wang, Y., Mansurov, J., Ivanov, P., Su, J., Shelmanov, A., Tsvigun, A.others  2023\.  M4: Multi-generator, multi-domain, and multi-lingual black-box machine-generated text detection M4: Multi-generator, multi-domain, and multi-lingual black-box machine-generated text detection.  arXiv preprint arXiv:2305.14902.
  * Y. Wang . 2024 wang2024m4gtWang, Y., Mansurov, J., Ivanov, P., Su, J., Shelmanov, A., Tsvigun, A.others  2024\.  M4GT-Bench: Evaluation Benchmark for Black-Box Machine-Generated Text Detection M4gt-bench: Evaluation benchmark for black-box machine-generated text detection.  arXiv preprint arXiv:2402.11175.
  * Xie . 2023 xienextXie, Z., Cohn, T. Lau, JH.  2023\.  The Next Chapter: A Study of Large Language Models in Storytelling The next chapter: A study of large language models in storytelling.  ( 323–351).  Proceedings of the 16th International Natural Language Generation Conference.
  * Xu . 2023 xu2023generalizationXu, H., Ren, J., He, P., Zeng, S., Cui, Y., Liu, A.Tang, J.  2023\.  On the generalization of training-based chatgpt detection methods On the generalization of training-based chatgpt detection methods.  arXiv preprint arXiv:2310.01307.
  * A. Yang . 2023 yang2023baichuanYang, A., Xiao, B., Wang, B., Zhang, B., Bian, C., Yin, C.others  2023\.  Baichuan 2: Open large-scale language models Baichuan 2: Open large-scale language models.  arXiv preprint arXiv:2309.10305.
  * J. Yang . 2024 yang2024harnessingYang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H.Hu, X.  2024\.  Harnessing the power of llms in practice: A survey on chatgpt and beyond Harnessing the power of llms in practice: A survey on chatgpt and beyond.  ACM Transactions on Knowledge Discovery from Data1861–32.
  * L. Yang . 2023 yang2023chatgptYang, L., Jiang, F. Li, H.  2023\.  Is chatgpt involved in texts? measure the polish ratio to detect chatgpt-generated text Is chatgpt involved in texts? measure the polish ratio to detect chatgpt-generated text.  APSIPA Transactions on Signal and Information Processing132.
  * Z. Yang . 2019 yang2019xlnetYang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, RR. Le, QV.  2019\.  Xlnet: Generalized autoregressive pretraining for language understanding Xlnet: Generalized autoregressive pretraining for language understanding.  Advances in neural information processing systems32.
  * Zeng . 2022 zeng2022glmZeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M.others  2022\.  Glm-130b: An open bilingual pre-trained model Glm-130b: An open bilingual pre-trained model.  arXiv preprint arXiv:2210.02414.
  * Zhang . 2023 zhang2023visarZhang, Z., Gao, J., Dhaliwal, RS. Li, TJJ.  2023\.  Visar: A human-ai argumentative writing assistant with visual programming and rapid draft prototyping Visar: A human-ai argumentative writing assistant with visual programming and rapid draft prototyping.  ( 1–30).  Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology.
  * Zhao . 2023 zhao2023surveyZhao, WX., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y.others  2023\.  A survey of large language models A survey of large language models.  arXiv preprint arXiv:2303.18223.
  * Zhou . 2024 zhou2024visionZhou, X., Liu, M., Yurtsever, E., Zagar, BL., Zimmer, W., Cao, H. Knoll, AC.  2024\.  Vision Language Models in Autonomous Driving: A Survey and Outlook Vision language models in autonomous driving: A survey and outlook.  IEEE Transactions on Intelligent Vehicles.

Other experimental data

Table 11: Direct detection of the proposed Dataset Based on the RoBERTa Model. |  | Chinese |  | English
---|---|---|---|---
|  | Accuracy | Precision | Recall | F1-score |  | Accuracy | Precision | Recall | F1-score
Complete | Baichuan | 0.5124 | 0.9189 | 0.2256 | 0.3623 | Baichuan | 0.5460 | 0.5251 | 0.3949 | 0.4508
QA | 0.9204 | 0.9897 | 0.8594 | 0.9200 | 0.6280 | 0.5899 | 0.6326 | 0.6105
Polish | 0.5128 | 0.9000 | 0.2279 | 0.3638 | 0.5212 | 0.5109 | 0.3072 | 0.3837
Expand | 0.5516 | 0.9510 | 0.2881 | 0.4422 | 0.5244 | 0.5126 | 0.3126 | 0.3884
Summary | 0.5712 | 0.9450 | 0.3211 | 0.4794 | 0.9332 | 0.8849 | 0.9865 | 0.9329
Refine | 0.5404 | 0.9391 | 0.2714 | 0.4211 | 0.5208 | 0.5107 | 0.3047 | 0.3816
Rewrite | 0.5408 | 0.9722 | 0.2676 | 0.4197 | 0.5228 | 0.5117 | 0.3080 | 0.3845
Translate | 0.5208 | 0.7321 | 0.2710 | 0.3956 | 0.6604 | 0.6334 | 0.6822 | 0.6569
Operations Average |  | 0.5838 | 0.9185 | 0.3415 | 0.4755 |  | 0.6071 | 0.5849 | 0.4911 | 0.5237
Complete | ChatGLM | 0.5448 | 0.9912 | 0.2716 | 0.4264 | ChatGLM | 0.5320 | 0.5176 | 0.3804 | 0.4385
QA | 0.9488 | 0.9930 | 0.9082 | 0.9487 | 0.6776 | 0.6210 | 0.7019 | 0.6590
Polish | 0.5080 | 0.7778 | 0.2310 | 0.3562 | 0.5216 | 0.5112 | 0.3134 | 0.3886
Expand | 0.5256 | 0.9324 | 0.2472 | 0.3908 | 0.5092 | 0.5048 | 0.2974 | 0.3742
Summary | 0.5700 | 0.9487 | 0.3184 | 0.4768 | 0.9180 | 0.8762 | 0.9635 | 0.9177
Refine | 0.5352 | 0.9490 | 0.2614 | 0.4099 | 0.5172 | 0.5090 | 0.3149 | 0.3890
Rewrite | 0.5348 | 0.9579 | 0.2597 | 0.4086 | 0.4964 | 0.4981 | 0.2924 | 0.3685
Translate | 0.6796 | 0.9212 | 0.5032 | 0.6509 | 0.6236 | 0.6081 | 0.6358 | 0.6217
Operations Average |  | 0.6059 | 0.9339 | 0.3751 | 0.5085 |  | 0.5995 | 0.5807 | 0.4875 | 0.5197
Complete | GPT3.5 | 0.5316 | 0.9759 | 0.2524 | 0.4011 | GPT3.5 | 0.5620 | 0.5331 | 0.4029 | 0.4590
QA | 0.9712 | 0.9950 | 0.9485 | 0.9712 | 0.6824 | 0.6217 | 0.7063 | 0.6613
Polish | 0.5204 | 0.9180 | 0.2397 | 0.3802 | 0.5204 | 0.5104 | 0.3006 | 0.3784
Expand | 0.6116 | 0.9729 | 0.3788 | 0.5452 | 0.5288 | 0.5148 | 0.3195 | 0.3943
Summary | 0.5272 | 0.8864 | 0.2557 | 0.3969 | 0.9136 | 0.8698 | 0.9614 | 0.9133
Refine | 0.5484 | 0.9690 | 0.2802 | 0.4347 | 0.5224 | 0.5115 | 0.3039 | 0.3813
Rewrite | 0.5812 | 0.9765 | 0.3308 | 0.4942 | 0.5228 | 0.5117 | 0.3087 | 0.3851
Translate | 0.5176 | 0.6864 | 0.2755 | 0.3932 | 0.6348 | 0.6163 | 0.6495 | 0.6325
Operations Average |  | 0.6012 | 0.9225 | 0.3702 | 0.5021 |  | 0.6109 | 0.5862 | 0.4941 | 0.5256
Complete | GPT4 | 0.7104 | 0.9962 | 0.5210 | 0.6842 | Llama3 | 0.5560 | 0.5300 | 0.3958 | 0.4532
QA | 0.9904 | 0.9968 | 0.9841 | 0.9904 | 0.6552 | 0.6058 | 0.6678 | 0.6353
Polish | 0.5620 | 0.9755 | 0.3009 | 0.4599 | 0.5268 | 0.5138 | 0.3192 | 0.3938
Expand | 0.6160 | 0.9899 | 0.3813 | 0.5506 | 0.5196 | 0.5102 | 0.3116 | 0.3869
Summary | 0.5688 | 0.9479 | 0.3167 | 0.4747 | 0.9296 | 0.8937 | 0.9682 | 0.9295
Refine | 0.5984 | 0.9695 | 0.3591 | 0.5241 | 0.5244 | 0.5125 | 0.3103 | 0.3866
Rewrite | 0.6356 | 0.9913 | 0.4106 | 0.5806 | 0.5252 | 0.5130 | 0.3130 | 0.3888
Translate | 0.5336 | 0.7625 | 0.2938 | 0.4241 | 0.6500 | 0.6285 | 0.6677 | 0.6475
Operations Average |  | 0.6519 | 0.9537 | 0.4459 | 0.5861 |  | 0.6109 | 0.5884 | 0.4942 | 0.5277
Complete | Qwen | 0.5928 | 0.9957 | 0.3448 | 0.5122 | Llama2 | 0.5540 | 0.5291 | 0.3965 | 0.4533
QA | 0.9556 | 0.9922 | 0.9215 | 0.9555 | 0.7084 | 0.6345 | 0.7433 | 0.6846
Polish | 0.5044 | 0.7200 | 0.2293 | 0.3478 | 0.5240 | 0.5123 | 0.3109 | 0.3870
Expand | 0.5272 | 0.9857 | 0.2444 | 0.3917 | 0.5280 | 0.5145 | 0.3222 | 0.3962
Summary | 0.5380 | 0.9130 | 0.2711 | 0.4181 | 0.9340 | 0.8834 | 0.9901 | 0.9337
Refine | 0.5340 | 0.9570 | 0.2584 | 0.4070 | 0.5212 | 0.5109 | 0.3041 | 0.3812
Rewrite | 0.5240 | 0.9286 | 0.2449 | 0.3876 | 0.5256 | 0.5132 | 0.3132 | 0.3890
Translate | 0.5216 | 0.7109 | 0.2792 | 0.4009 | 0.6736 | 0.6428 | 0.6991 | 0.6697
Operations Average |  | 0.5872 | 0.9004 | 0.3492 | 0.4776 |  | 0.6211 | 0.5926 | 0.5099 | 0.5368
Table 12: Direct detection of the proposed Dataset Based on the XLNet Model. |  | Chinese |  | English
---|---|---|---|---