# Disentangling Heterogeneous Knowledge Concept Embedding for Cognitive
Diagnosis on Untested Knowledge

Kui Xiao1, Runtian Xing1, Miao Zhang1,, Shunfeng Tan1,
Ziming Wang2, Xiaolian Zhu3
1School of Computer Science and Information Engineering, Hubei University
2Key Laboratory of Intelligent Sensing System and Security
3Hubei Key Laboratory of Big Data Intelligent Analysis and Application
{xiaokui<EMAIL_ADDRESS>
{xrt, tsf, wzm<EMAIL_ADDRESS>the corresponding author.

###### Abstract

Cognitive diagnosis is a fundamental and critical task in learning assessment,
which aims to infer students’ proficiency on knowledge concepts from their
response logs. Current works assume each knowledge concept will certainly be
tested and covered by multiple exercises. However, whether online or offline
courses, it’s hardly feasible to completely cover all knowledge concepts in
several exercises. Restricted tests lead to undiscovered knowledge deficits,
especially untested knowledge concepts(UKCs). In this paper, we propose a
novel Disentangling Heterogeneous Knowledge Cognitive Diagnosis framework on
untested knowledge(DisKCD). Specifically, we leverage course grades, exercise
questions, and resources to learn the potential representations of students,
exercises, and knowledge concepts. In particular, knowledge concepts are
disentangled into tested and untested based on the limiting actual exercises.
We construct a heterogeneous relation graph network via students, exercises,
tested knowledge concepts(TKCs), and UKCs. Then, through a hierarchical
heterogeneous message-passing mechanism, the fine-grained relations are
incorporated into the embeddings of the entities. Finally, the embeddings will
be applied to multiple existing cognitive diagnosis models to infer students’
proficiency on UKCs. Experimental results on real-world datasets show that the
proposed model can effectively improve the performance of the task of
diagnosing students’ proficiency on UKCs. Our anonymous code is available at
https://anonymous.4open.science/r/DisKCD.

##  1 Introduction

Cognitive diagnosis is an important component of intelligent tutoring
systems[1, 2, 3], which aims to assess students’ proficiency on specific
knowledge concepts. Cognitive diagnosis models (CDMs) analyze students’
interaction history, including correct and incorrect responses in exercise
logs, to infer their cognitive states. Presently, cognitive diagnosis outcomes
are integrated into various facets of intelligent tutoring, such as exercise
recommendation [4, 5] and adaptive testing [6]. As the array of learning
materials expands and students’ learning preferences vary more widely,
cognitive diagnosis models are becoming increasingly important. They furnish
instructors with potent psychometric instruments for educational appraisal,
enhancing teaching and learning support.

In cognitive diagnosis, researchers propose various CDMs, such as Item
Response Theory (IRT) [7], Multidimensional IRT (MIRT) [8], and Deterministic
Inputs, Noisy-And gate model (DINA) [9]. These models employ artificially
designed linear functions to simulate student-practice interactions. However,
in recent years, due to the proliferation of neural networks and deep learning
technologies, researchers have introduced novel cognitive diagnosis models
leveraging neural networks, such as Neural Cognitive Diagnosis (NeuralCD)
[10], and models based on graph convolutional networks like Relation map
driven Cognitive Diagnosis (RCD) [11]. These state-of-the-art models have
yielded cutting-edge outcomes in cognitive diagnosis.

Researchers typically assume that each knowledge concept is contained in
multiple exercises. In other words, these works focus on students’ mastery of
the tested knowledge concepts (TKCs). However, whether online or offline
courses, it’s hardly feasible to completely cover all knowledge concepts in
several exercises. The scope of exams is limited, which makes it challenging
for assessments to cover all course concepts comprehensively. Moreover, issues
like plagiarism further complicate the evaluation process, which hinders
instructors’ ability to accurately gauge students’ understanding of each
concept and provide timely guidance. Thus, effectively diagnosing students’
mastery of untested knowledge concepts (UKCs) is a critical concern within
cognitive diagnosis.

Figure 1: Examples of cognitive diagnosis. Students’ mastery of tested
knowledge components is often emphasized, while their grasp of untested
knowledge components is usually overlooked.

As shown In Figure 1, a course’s final exam contains a limited number of
exercises, covering only a fraction of the knowledge concepts of the course,
such as sections 1.3, 2.2 and 5.1. Instructors can assess student
understanding of TKCs in the final exam [12]. However, gauging students’
mastery of the UKCs, which are not assessed in the final exam, is impossible.
In this paper, we introduce a cognitive diagnosis framework to assess
students’ mastery of UKCs in exams. Specifically, knowledge concepts are
disentangled into TKCs and UKCs. Next, we construct a heterogeneous relation
graph network using entities like students, exercises, TKCs, and UKCs. Through
a hierarchical heterogeneous message-passing mechanism, the fine-grained
relations are incorporated into the entity embeddings. Finally, the entity
embeddings will be applied to multiple existing cognitive diagnosis models to
infer students’ proficiency on UKCs.

Our key contributions are summarized as follows:

  * •

We propose a novel cognitive diagnosis framework to assess students’
proficiency on UKCs, which effectively addresses the challenge of difficulty
in comprehensively examining traditional cognitive diagnosis.

  * •

We present a heterogeneous relation aware network for students, exercises,
TKCs and UKCs to ensure that the embeddings are highly sensitive to students’
proficiency and the diagnostic orientation of UKCs.

  * •

We create an authentic dataset from two university courses, facilitating
cognitive diagnosis research on UKCs. Diverging from previous datasets, our
dataset is meticulously curated from genuine data extracted from final exams
of university courses, which are better poised to address the complexities of
evaluating students’ proficiency.

##  2 Related Work

###  2.1 Cognitive Diagnosis

Cognitive diagnosis is a comprehensive approach to education assessment, which
involves two main tasks: cognitive diagnosis [13][14][15] and knowledge
tracking[16, 17, 18]. Knowledge tracking focuses on monitoring changes in
student performance to predict their responses to future questions. However,
it does not directly consider how well students grasp concepts throughout
their learning process. On the other hand, Cognitive diagnosis is designed to
evaluate students’ understanding of each knowledge concept, assuming that
their cognitive abilities remain consistent during tests[19, 20].

Researchers focus on evaluating students’ knowledge levels through their past
responses. IRT is a pivotal advancement [7], which predicts students’ answer
accuracy through manually crafted functions. MIRT then incorporates
multidimensional features of students [8]. However, the effectiveness of
traditional models hinges on the well-designed functions [21, 22]. To address
this issue, NeuralCD employs neural networks to simulate student-exercise
interactions [10, 23, 24]. RCD [11], explores the relation between students,
exercises, and knowledge concepts. Another widely used approach is DINA, which
uses discrete numerical values to assess knowledge mastery and consider errors
like slipping and guessing in the diagnosis process [9]. An improved model of
DINA is FuzzyCDM [25], which introduces fuzzy logic control and realizes that
the knowledge mastery state can be kept constant.

###  2.2 Knowledge Coverage Problem

The insufficient coverage of knowledge in exercises presents a significant
obstacle in cognitive assessment tasks[26, 27]. CDMs primarily focus on tests
structured around fixed exercises, where the number of knowledge concepts is
limited. Student responses about multiple related exercises are extremely
important in assessing mastery of each knowledge concept. However, in modern
education, there’s a vast array of knowledge concepts, yet student responses
are sparse, highlighting the challenge of ensuring comprehensive knowledge
coverage. Leighton [28] uses the hierarchical relations between knowledge
concepts for cognitive diagnosis assessment(AHM). De La Torre and Douglas [29]
propose HO-DINA, which incorporates low-dimensional high-order latent features
influencing students’ mastery of each knowledge concept to enhance cognitive
diagnosis tasks.

In addition, zero-shot cognitive diagnosis at the domain level is also a
recent hot topic, such as newly TechCD [30] and Zero-1-to-3 [31]. Unlike these
works that explore cognitive diagnosis across different domains, our cognitive
diagnosis tasks focus on UKCs in the examination. We aim to deduce students’
proficiency on these UKCs by examining their proficiency in TKCs and the
relation between them. To the best of our knowledge, this is the first attempt
to evaluate students’ proficiency on UKCs.

##  3 Preliminaries

###  3.1 Cognitive Diagnosis Model

We first outline the general form of CDMs, which typically consist of three
fundamental components: students, exercises, and knowledge concepts. Most CDMs
predict how well students will perform in practicing exercises, indirectly
quantifying students’ mastery of knowledge. The diagnosis core essentially
involves modeling student-exercise-concept interactions by predicting
students’ performance during practice sessions, as illustrated below:

${\hat{y}}_{uv}=F_{CDM}(u,v,K),$ (1)

Here, $u$ denotes the traits of a student, $v$ represents the traits of
exercise, and $K$ is the knowledge concept space of a domain. The primary aim
of CDMs is to minimize the disparity between the anticipated outcome
${\hat{y}}_{uv}$ and the actual score $y_{uv}$ of the student’s response.
$F_{CDM}(\cdot)$ is a diagnostic tool for predicting the response outcome.

Interpretability is significantly important for cognitive diagnosis for
improving teaching and learning [32]. CDMs typically follow the monotonicity
assumption in the literature [8]: "The probability of correct response to the
exercise is monotonically increasing at any dimension of the student’s
knowledge proficiency." In pursuit of the assumption, CDMs enforce
$\frac{\partial F}{\partial u}>0$ throughout the training phase.

###  3.2 Problem Setup

In the cognitive diagnosis task involving UKCs, we define several terms. Let
$S=\left\\{s_{1},\ s_{2},\ ...,\ s_{N}\right\\}$ represent the set of
students, and $E=\left\\{e_{1},\ e_{2},\ ...,\ e_{M}\right\\}$ denote the
exercises within a course exam. $K_{tested}$ signifies the knowledge concepts
corresponding to these exercises, and $K_{untested}$ denote the knowledge
concepts not covered in the exam. The union $K=K_{tested}\cup K_{untested}$
encompasses all knowledge concepts in the course, with the constraint
$K_{tested}\cap K_{untested}=\Phi$. Let
$R_{ht}=\left\\{(k_{h},k_{t},r_{ht})|k_{h},k_{t}\in K,\ r_{ht}\in
R_{k}\right\\}$ denotes the set of triples of knowledge concepts, where
$R_{k}$ contains all forms of relations among knowledge concepts. This study
specifically employs two types of relations: similarity and prerequisite
relations.

On the other hand, the observed interaction logs are a triple set
$R_{se}=\left\\{(s_{u},e_{v},y_{uv})|s_{u}\in S,e_{v}\in
E,y_{uv}\in{0,1}\right\\}$, where $y_{uv}$ indicates the response of student
$s_{u}$ to exercise $e_{v}$. A correct response is denoted by $y_{uv}=1$,
while an incorrect response is indicated by $y_{uv}=0$. Additionally, we have
an expert-annotated Q-matrix [33], where $Q_{ij}=1$ signifies that exercise
$e_{i}$ is associated with knowledge concept $k_{j}$, otherwise $Q_{ij}=0$.

PROBLEM DEFINITION. Given students’ exercising response logs $R_{se}$ on TKCs,
the set of triples of knowledge concepts $R_{ht}$, as well as the $Q-matrix$,
our goal is to assess students’ proficiency on UKCs.

###  3.3 Heterogeneous Relation Graphs

Depending on dynamic interactions between students, exercises, and knowledge
concepts, we define diverse heterogeneous relation subgraph.

student-exercise interaction subgraph. The interaction relations between
students and exercises listed on the test paper collectively form the student-
exercise interaction subgraph. This subgraph is bidirectional and can be
represented as $G_{se}(S\cup E,r_{se})$, where $r_{se}$ denotes the set of
interactions between students and the exercises. If $r_{s_{u}\leftrightarrow
e_{v}}=1$, it indicates that student $s_{u}$ attempted exercise $e_{v}$.

exercise-TKC correlation subgraph. The incorporation connection between TKCs
and UKCs collectively forms the Exercise-TKC correlation subgraph, denoted as
$G_{ek}(E\cup K_{tested},r_{ek})$. Here, all $r_{ek}$ derive from the
Q-matrix. If $r_{e_{v}\leftrightarrow k_{w}}$ within this set equals 1, it
indicates that exercise $e_{v}$ encompasses the knowledge concept $k_{w}$.

TKC-TKC dependency subgraph. The connections among the TKCs collectively form
the TKC-TKC dependency subgraph, denoted as $G_{kk}(K_{tested},r_{kk})$. We
solely contemplate the similarity and prerequisite relations. The similarity
relation allows the propagation of students’ cognitive states between concepts
in both directions. Conversely, the prerequisite relation functions
unilaterally.

TKC-UKC dependency subgraph. The connections among TKCs and UKCs, denoted as
$G_{ku}(K_{tested}\cup K_{untested},r_{ku})$. Just like $r_{kk}$, $r_{ku}$
encompasses the array of connections between TKCs and UKCs, encompassing
similarities and prerequisites. Moreover, the nodes linked by a relation must
embody a TKC and a UKC.

UKC-UKC dependency subgraph. The connections among all UKCs collectively form
the UKC-UKC dependency subgraph, represented as $G_{uu}(K_{untested},r_{uu})$.
Much like $r_{kk}$,$r_{uu}$ denotes the array of relations among UKCs.
Likewise, the connections between concept nodes here solely comprise
similarity and prerequisite relations.

##  4 Methodology

To address the challenge of accurately assessing students’ proficiency with
limited homework and test practice, this paper introduces a DisKCD framework
for UKCs. First, in the embedding layer, we disentangle knowledge concepts to
TKCs and UKCs. Then, in the heterogeneous relation aware layer, the fine-
grained relations are incorporated into the entity embeddings. Finally,
embeddings are applied to CDMs to infer students’ proficiency on UKCs.

Figure 2: Overview of DisKCD. The left side shows the disentangling process of
decomposing knowledge concepts into TKCs and UTCs. The middle part shows the
details of the heterogeneous relation-aware layer where fine-grained relations
are merged into entity embeddings. The right part shows the diagnostic process
of DisKCD framework.

###  4.1 Embedding Layer

The embedding layer traps semantic details to generate representations of
students, test questions, and knowledge concepts, providing a robust basis for
subsequent cognitive diagnosis tasks.

Student embedding. To assess students’ initial capabilities accurately, we
gather and normalize their grades using the standard score (z-score), which is
the number of standard deviations above or below the mean value of what is
being observed or measured [34]. The conversion operation is

$z_{c}^{(u)}=\frac{X_{c}^{(u)}-\mu_{c}}{\sigma_{c}},$ (2)

where $X_{c}^{(u)}$ is the grade of student $s_{u}$ in course $c$, $\mu_{c}$
denotes the average score for $c$. $\sigma_{c}$ stands for the standard
deviation of all scores within the class for this particular course. For each
student, we obtain an original embedding via a fully connected neural network:

$\bm{s}=FNN(z_{1},\ z_{2},\ ...,\ z_{C}),$ (3)

Here, $\bm{s}\in\mathbb{R}^{d_{k}\times 1}$ and $d_{k}$ is the number of all
knowledge concepts. $C$ represents the number of historical courses. s
incorporates all the student’s historical performance information.

Exercise embedding. We gather the text from the final exam exercises to
construct embeddings for the exercises based on textual content. Specifically,
we feed the word2vec [35] sequence $(\bm{w}_{1},\ \bm{w}_{2},\ ...,\
\bm{w}_{T})$ of exercise into each time step of Bi-LSTM [36]. Subsequently, we
concatenate the hidden state by the forward direction
$(\overleftarrow{\bm{h}}_{1},\ \overleftarrow{\bm{h}}_{2},\ ...,\
\overleftarrow{\bm{h}}_{T})$ and the backward direction
$(\overrightarrow{\bm{h}}_{1},\ \overrightarrow{\bm{h}}_{2},\ ...,\
\overrightarrow{\bm{h}}_{T})$, aligning them by position, and then yielding

$\bm{h}_{i}=[\overleftarrow{\bm{h}}_{i},\overrightarrow{\bm{h}}_{i}],$ (4)

where $[,]$ denotes the concatenation operation. Finally, the exercise vector
$\textbf{e}\in\mathbb{R}^{d_{k}\times 1}$ is updated by averaging operation
and a fully connected layer:

$\bm{e}=FNN(AVG(\bm{h}_{1},\ \bm{h}_{2},\ ...,\ \bm{h}_{T})).$ (5)

Concept embedding. The content-relevant pedagogical resources, such as course
slides, is employed to enhance the semantic representation of knowledge
concepts. The embedding construction process for the knowledge concept is the
same as Eq.4 and Eq.5. Specifically, we denote the embedding of TKC and UKC as
$\bm{k}\in\mathbb{R}^{d_{k}\times 1}$ and $\bm{u}\in\mathbb{R}^{d_{k}\times
1}$ respectively.

###  4.2 Heterogeneous Relation-aware Layer

DisKCD constructs a heterogeneous relation-aware graph via relations between
students, exercises, TKCs, and UKCs, linking four types of entities through a
heterogeneous attention mechanism. So that each entity node can iteratively
obtain updated information from neighbor nodes based on the attention weights.

Heterogeneous Relation Attention. The heterogeneous relation attention occurs
in a heterogeneous graph with five subgraphs, $G=(G_{se}$, $G_{ek}$, $G_{kk}$,
$G_{ku}$, $G_{uu})$. For example, in $G_{uu}$ for UKCs there are prerequisite
and similarity relations among them. Under both relations, each node contains
different neighbor nodes. Specifically, after the Embedding Layer, we obtain
the initial embeddings for four types of entities: $\bm{s}$, $\bm{e}$,
$\bm{k}$ and $\bm{u}$. Within subgraphs $g\in G$, we exemplify with entity
node $t\in(S,E,TKCs,UKCs)$. For entity node $t$, we calculate its attention
weights with neighbor nodes under various relation contexts. Through weighted
summation, we then derive the vector embedding of entity node $t$ in a
subgraph $G$ after separately fusing different relations (In the subgraphs
$G_{kk}$,$G_{ku}$, and $G_{uu}$, each one encompasses multiple types of
relations). The implementation of heterogeneous relation attention is executed
as follows:

${\widetilde{\bm{n}}}^{(l+1)}=\sum_{r\in R_{g}}\sum_{n\in
N_{t}^{r}}{\alpha_{n}^{(l+1)}\bm{W}^{(l)}\bm{n}^{(l)}},$ (6)
${\hat{\alpha}}_{n}^{\left(l+1\right)}=FNN([\bm{W}^{(l)}\bm{t}^{(l)},\bm{W}^{(l)}\bm{n}^{(l)}]),$
(7)
$\alpha_{n}^{(l+1)}=\frac{exp({\hat{\alpha}}_{n}^{\left(l+1\right)})}{\sum_{j\in
N_{t}^{r}}{exp({\hat{\alpha}}_{j}^{\left(l+1\right)})}},$ (8)

where $r\in R_{g}$ denotes the relation between node $t$ and neighbor node $n$
in heterogeneous subgraph $g$. $\alpha_{n}^{(l+1)}$ refers to the attention
weights assigned to node $t$’s neighbors, reflecting the significance of each
relation type $r$ in the heterogeneous subgraph. $\bm{W}^{(l)}$ is the
trainable matrix. $\bm{n}^{(l)}$ is the embeddings of node $t$ neighbor nodes
after the initial iteration. In the Heterogeneous Relation Attention section,
we obtain the embedding vector representations of the four types of entities
$(S,E,TKCs,UKCs)$ under various relations within a specific subgraph.

Heterogeneous Relation Aggregation. An entity node may appear in multiple
subgraphs. For example, node $u$ is present in both $G_{ku}$ and $G_{uu}$,
serving as distinct roles. Since the entity acquires different embeddings in
each subgraph, we aggregate these embeddings in the Heterogeneous Relation
Aggregation subsection according to attention weights. Specifically, for an
entity node $t\in(S,E,TKCs,UKCs)$, the embeddings from different subgraphs are
spliced with the original embedding of node $t$. Softmax normalization is then
applied to determine the respective weights, followed by a weighted summation
to perform the aggregation update. This process results in an integrated
embedding representation that encapsulates the diverse relations and
subgraphs. The $l+1$ aggregation iteration is as follows:

$\bm{t}^{(l+1)}=\bm{t}^{(l)}+\sum_{g\in
G}{\mu_{g}^{(l+1)}\widetilde{\bm{n}}}^{(l+1)},$ (9)
$\mu_{g}^{(l+1)}=FNN([\bm{t}^{(l)},{\widetilde{\bm{n}}}^{(l+1)}]),$ (10)

Here, $\mu_{g}^{(l+1)}$ represents the attention of the entity node $t$ in
different subgraphs. After aggregating from different subgraphs, we obtain the
final embedding representations of four types of entities:
$\bm{s}^{(l+1)}$,$\bm{e}^{(l+1)}$,$\bm{k}^{(l+1)}$ and $\bm{u}^{(l+1)}$. Next,
we feed embeddings generated through the DisKCD framework into a general CDM
for cognitive diagnosis.

###  4.3 Diagnosis Layer

In this section, we integrate DisKCD with popular CDMs to assess students’
mastery on UKCs and predict their outcomes on exercises unrelated to the TKCs.

Given that distinct CDM employ varying formats of student, exercise, and
knowledge concept features as input, transform functions to convert the
embeddings into the formats suitable for each CDM are necessary. Hence, we
redefine Eq.1 as follows:

${\hat{y}}_{uv}=F_{CDM}(\phi_{s}(\bm{u}),\phi_{e}(\bm{v}),\phi_{c}(\bm{K})).$
(11)

Here, $\phi_{s}(\cdot)$, $\phi_{e}(\cdot)$, and $\phi_{c}(\cdot)$ represent
transform functions. We utilize cross-entropy loss function to calculate the
difference between the predicted probability ${\hat{y}}_{uv}$ and the actual
response $y_{uv}$.

$L=-\sum_{(s_{u},e_{v},y_{uv})\in
R_{se}}{(y_{uv}{log{\widehat{y}}}_{uv}+(1-y_{uv})log{(}1-{\hat{y}}_{uv}))}.$
(12)

##  5 Experiments

We conduct comprehensive experiments to answer the following research
questions:

RQ1: Does DisKCD enhance the performance of CDMs in assessing students’
mastery of UKCs?

RQ2: Is DisKCD beneficial for CDMs in evaluating students’ mastery of TKCs?

RQ3: How well does DisKCD model entities like students, exercises, and
knowledge concepts?

RQ4: Are the cognitive diagnosis results produced by DisKCD interpretable?

###  5.1 Datasets

We evaluate our approach on three real-world datasets: one from an online
learning platform and the other from the final exams of two offline university
courses. Details are shown in Table 1.

JAD and SDP consist of student response data extracted from the final exams of
two courses, "Java Application Development" and "Software Design Patterns",
offered during the 2021-2022 academic year in the software engineering major
at a Chinese university. Note that the exercises in these exams are open
questions rather than fill-in-the-blank questions or multiple-choice
questions.

Junyi is sourced from the Chinese e-learning platform Junyi Academy [37],
which is widely applied in CDM. To more effectively differentiate between TKCs
and UKCs, we extracted response logs of 10,000 students in the Junyi dataset
for our experiments.

Table 1: The statistics of three datasets.

Dataset | # Student |  | # Tested
---
Exercise
| # Untested
---
Exercise
| # TKCs
---
| # UKCs
---
# Record |  | # Record per
---
student
JAD | 275 | 14 | 4 | 14 | 4 | 4950 | 18
SDP | 128 | 12 | 3 | 10 | 3 | 1920 | 15
Junyi | 10000 | 589 | 246 | 589 | 246 | 353835 | 35.38

###  5.2 Experiment Settings

Baselines. To evaluate the validity and generalizability, we employ DisKCD to
five widely adopted CDMs: 1)DINA [9]: DINA assesses the knowledge or skills by
answer specific test questions, 2)IRT [7]: IRT diagnoses a student’s cognitive
status by analyzing questions’ parameters (difficulty, discrimination, and
guessing parameters), 3)MIRT [8]: MIRT extends the parameters of student
learning abilities and exercises to multiple dimensions, 4)NeuralCD [10]: NCD
utilizes multilayer perceptrons to model complex higher-order interactions
between student and exercises, and 5)RCD [11]: RCD constructs multi-layer maps
of relations between students, exercises, and concepts.

Metrics. Following previous studies, we use various metrics from regression
and classification perspectives. From the regression perspective, we select
Root Mean Square Error (RMSE) [38] to measure the difference between predicted
and actual scores. From the classification perspective, we use prediction
Accuracy (ACC) [10] and Area Under an ROC Curve (AUC) [39] for model
evaluation.

Implementation Details. We set the dimensions of the student and exercise
vectors to the number of all concepts in the course. All network parameters
are initialized using Xavier. In addition, the upper batch was set to 8 for
JAD and SDP and 128 for Junyi. the optimizer uses Adam. the code is based on
the PyTorch implementation and runs on a Linux server with an RTX 4090(24G)
GPU.

###  5.3 Performance Comparison on UKCs (RQ1)

To investigate RQ1, we trained our models using data related to TKCs and made
predictions on data related to UKCs. Predictions use vectors from five popular
CDMs, including vectors updated through the original CDM and those updated via
DisKCD. The results, as presented in Table 2, led to the following
observations: (1) In nearly all the CDMs, vectors updated through DisKCD
perform better than those updated through the original CDMs. It indicates that
DisKCD is capable of diagnosing students’ mastery of UKCs with limited data.
(2)On three datasets, DisKCD is significantly better than baseline, suggesting
that disentangling TKCs from UKCs positively influences prediction tasks.
(3)The experimental results for RCD and NCD are generally superior to those of
DINA, IRT, and MIRT, demonstrating that modeling complex student-exercise-
concept relations is more effective.

Table 2: Performance comparison on Untested Knowledge Concept. The best
performance prediction is highlighted in bold.

Dataset | JAD |  | SDP |  | Junyi
---|---|---|---|---|---
| ACC | RMSE | AUC |  | ACC | RMSE | AUC |  | ACC | RMSE | AUC
DINA | baseline | 0.5936 | 0.5427 | 0.5944 |  | 0.5781 | 0.5124 | 0.5679 |  | 0.6076 | 0.4737 | 0.6523
DisKCD | 0.6409 | 0.4933 | 0.6478 |  | 0.6432 | 0.4965 | 0.6457 |  | 0.7504 | 0.4305 | 0.8051
IRT | baseline | 0.5916 | 0.5382 | 0.5979 |  | 0.5260 | 0.5718 | 0.5119 |  | 0.6462 | 0.5079 | 0.7235
DisKCD | 0.5932 | 0.4943 | 0.7086 |  | 0.5683 | 0.5386 | 0.6318 |  | 0.7545 | 0.4088 | 0.8147
MIRT | baseline | 0.5336 | 0.5426 | 0.6313 |  | 0.5495 | 0.5549 | 0.6454 |  | 0.6545 | 0.5353 | 0.7262
DisKCD | 0.5991 | 0.5148 | 0.6319 |  | 0.5885 | 0.5061 | 0.6663 |  | 0.7626 | 0.4071 | 0.8130
NCD | baseline | 0.5903 | 0.5759 | 0.6035 |  | 0.5260 | 0.5811 | 0.5776 |  | 0.7054 | 0.4287 | 0.7888
DisKCD | 0.6527 | 0.4702 | 0.6938 |  | 0.6225 | 0.5603 | 0.6042 |  | 0.7480 | 0.4115 | 0.7010
RCD | baseline | 0.5909 | 0.4895 | 0.6294 |  | 0.6068 | 0.4960 | 0.6580 |  | 0.6284 | 0.4765 | 0.6458
DisKCD | 0.6673 | 0.4606 | 0.6953 |  | 0.6771 | 0.4501 | 0.7546 |  | 0.7746 | 0.3936 | 0.7716

###  5.4 Performance Comparison on TKCs (RQ2)

In the experiments addressing RQ2, the exercises in training and testing
datasets are all related to the TKCs. According to the results shown in Table
3, the following observations were obtained: (1)Across all CDMs, vectors from
DisKCD demonstrated outstanding performance compared to the original vectors,
indicating that DisKCD can assess students’ mastery of TKCs more accurately.
(2)The experiments on Junyi showed generally higher AUC values than those
based on our custom datasets, JAD and SDP. This suggests that models trained
with the Junyi dataset have stronger predictive capabilities, underscoring the
necessity of disentangling the diagnosis tasks between the rich and
extensively tested and untested knowledge concepts.

Table 3: Performance comparison on Tested Knowledge Concept.

Dataset | JAD |  | SDP |  | Junyi
---|---|---|---|---|---
| ACC | RMSE | AUC |  | ACC | RMSE | AUC |  | ACC | RMSE | AUC
DINA | baseline | 0.7091 | 0.4492 | 0.7012 |  | 0.6172 | 0.4962 | 0.6857 |  | 0.6414 | 0.4737 | 0.7536
DisKCD | 0.7516 | 0.4371 | 0.7213 |  | 0.6589 | 0.4814 | 0.7121 |  | 0.6720 | 0.4582 | 0.8024
IRT | baseline | 0.7503 | 0.4323 | 0.6430 |  | 0.6823 | 0.4706 | 0.7102 |  | 0.6462 | 0.5079 | 0.7951
DisKCD | 0.7636 | 0.4077 | 0.7316 |  | 0.7109 | 0.4643 | 0.7285 |  | 0.7545 | 0.4089 | 0.8115
MIRT | baseline | 0.7066 | 0.4473 | 0.6080 |  | 0.6901 | 0.4689 | 0.7119 |  | 0.7626 | 0.4071 | 0.8131
DisKCD | 0.7697 | 0.4135 | 0.7562 |  | 0.7005 | 0.4618 | 0.7407 |  | 0.7705 | 0.3977 | 0.8273
NCD | baseline | 0.7401 | 0.4150 | 0.7103 |  | 0.6276 | 0.4790 | 0.6690 |  | 0.7393 | 0.4166 | 0.7963
DisKCD | 0.7507 | 0.4044 | 0.7526 |  | 0.6626 | 0.4783 | 0.6976 |  | 0.7495 | 0.4112 | 0.8031
RCD | baseline | 0.7585 | 0.4050 | 0.7306 |  | 0.6614 | 0.4589 | 0.7215 |  | 0.7694 | 0.3981 | 0.8262
DisKCD | 0.7755 | 0.4005 | 0.7387 |  | 0.7119 | 0.4572 | 0.7402 |  | 0.7727 | 0.3955 | 0.8302

###  5.5 Ability of Modeling Entity Embedding (RQ3)

To understand the influence of the embedding layer within DisKCD on overall
model performance, we conducted a series of comparative experiments on JAD. We
utilize the commonly one-hot vectors as a baseline, replacing the embedding
vectors generated by DisKCD. Table 4 indicates that using the embedding
vectors produced by DisKCD’s embedding layer improves diagnostic performance.
This suggests that refining semantics through a layered heterogeneous message-
passing mechanism can enhance the model’s accuracy in predicting students’
proficiency on UKCs.

Table 4: The performance of entity modeling.

Train/Test | DINA | IRT | MIRT | NCD | RCD
---|---|---|---|---|---
-Emb | OURS | -Emb | OURS | -Emb | OURS | -Emb | OURS | -Emb | OURS
ACC | 0.5913 | 0.6409 | 0.5926 | 0.5932 | 0.5433 | 0.5991 | 0.5904 | 0.6527 | 0.6273 | 0.6673
RMSE | 0.5442 | 0.4933 | 0.5386 | 0.4943 | 0.5542 | 0.5148 | 0.5097 | 0.4702 | 0.4772 | 0.4606
AUC | 0.5892 | 0.6478 | 0.5340 | 0.7086 | 0.5467 | 0.6319 | 0.6372 | 0.6938 | 0.6647 | 0.6953

###  5.6 Interpretability and Case Study (RQ4)

We assess the interpretability of DisKCD’s diagnostic results using Degree of
Agreement (DOA)[40], based on Monotonicity Assumption[8]. The assumption
states that a more proficient student on a concept should perform better on
its exercises. As Figure 3 shows, solid nodes(DisKCD) have higher DOA values
than hollow nodes(baseline), suggesting the diagnosed knowledge proficiency
levels by DisKCD are reasonable. Notably, the latent vectors in IRT and MIRT
lack direct correspondence to concepts, so their DOA is considered to be 0.

Figure 3: The performance of interpretability of DisKCD and baseline in JAD
and Junyi.

Figure 4: Case study: responses and diagnostic results of two students.

Figure 4 shows the cognitive status of students in the JAD dataset. Panel (a)
displays the relevance between exercises and TKCs, like web.xml and statement,
and UKCs, such as MVC and interface. Panel (b) shows the student’s responses.
Panel (c) displays the diagnostic results of the DisK-RCD (left) and the
original RCD (right) on the knowledge status of two students, with bars
representing exercise difficulty and lines depicting proficiency levels.
Observations indicate that students answer correctly when their proficiency
exceeds exercise difficulty. The analysis reveals that both DisKCD and
baseline perform well on TKCs, but only DisKCD effectively diagnoses UKCs,
affirming its ability to assess untested knowledge concepts.

##  6 Conclusions

In this paper, we propose a general knowledge diagnosis framework called
DisKCD. Specifically, we define four types of entities: students, exercises,
TKCs, and UKCs, along with their fine-grained relations. This allows us to
disentangle TKCs and UKCs. Then, using a heterogeneous relation-aware network,
we generate relation-aware entity embeddings, which can be used in various
CDMs to predict students’ performance. Experimental results on real-world
datasets show that DisKCD not only effectively predicts students’ proficiency
on UKCs but also helps assess their mastery of TKCs, with good
interpretability. Future research will explore more diverse educational
dependencies to enhance CDMs, infer students’ cognitive states, and customize
personalized learning paths. We hope our work will contribute to the field of
intelligent education.

##  7 Limitations

Limited by computational resources, we couldn’t compare with additional CDMs.
Training a CDM on a server with two 48G GPUs takes about six days on average.
Furthermore, we only use similarity and prerequisite relations in our work. In
the future, we aim to incorporate more diverse knowledge relations to enhance
the model’s performance.

## References

  * [1] Qi Liu.  Towards a new generation of cognitive diagnosis.  In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 4961–4964. ijcai.org, 2021.
  * [2] Ashton Anderson, Daniel P. Huttenlocher, Jon M. Kleinberg, and Jure Leskovec.  Engaging with massive online courses.  In 23rd International World Wide Web Conference, WWW ’14, Seoul, Republic of Korea, April 7-11, 2014, pages 687–698. ACM, 2014.
  * [3] Yingjie Liu, Tiancheng Zhang, Xuecen Wang, Ge Yu, and Tao Li.  New development of cognitive diagnosis models.  Frontiers Comput. Sci., 17(1):171604, 2023.
  * [4] Fei Liu, Xuegang Hu, Shuochen Liu, Chenyang Bu, and Le Wu.  Meta multi-agent exercise recommendation: A game application perspective.  In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6-10, 2023, pages 1441–1452. ACM, 2023.
  * [5] Zhi Mao and Mingfang Li.  Artificial intelligence and cognitive diagnosis based teaching resource recommendation algorithm.  PeerJ Comput. Sci., 9:e1594, 2023.
  * [6] Yan Zhuang, Qi Liu, Guanhao Zhao, Zhenya Huang, Weizhe Huang, Zachary A. Pardos, Enhong Chen, Jinze Wu, and Xin Li.  A bounded ability estimation for computerized adaptive testing.  In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.
  * [7] Frederic Lord.  A theory of test scores.  Psychometric monographs, 1952.
  * [8] Mark D. Reckase.  Multidimensional Item Response Theory Models, pages 79–112.  Springer New York, New York, NY, 2009.
  * [9] Jimmy De La Torre.  Dina model and parameter estimation: A didactic.  Journal of educational and behavioral statistics, 34(1):115–130, 2009.
  * [10] Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yuying Chen, Yu Yin, Zai Huang, and Shijin Wang.  Neural cognitive diagnosis for intelligent education systems.  In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020, pages 6153–6161. AAAI Press, 2020.
  * [11] Weibo Gao, Qi Liu, Zhenya Huang, Yu Yin, Haoyang Bi, Mu-Chun Wang, Jianhui Ma, Shijin Wang, and Yu Su.  RCD: relation map driven cognitive diagnosis for intelligent education systems.  In SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 501–510. ACM, 2021.
  * [12] Xiangzhi Chen, Le Wu, Fei Liu, Lei Chen, Kun Zhang, Richang Hong, and Meng Wang.  Disentangling cognitive diagnosis with limited exercise labels.  In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.
  * [13] Shanshan Wang, Zhen Zeng, Xun Yang, Ke Xu, and Xingyi Zhang.  Boosting neural cognitive diagnosis with student’s affective state modeling.  In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, February 20-27, 2024, Vancouver, Canada, pages 620–627. AAAI Press, 2024.
  * [14] Haiping Ma, Changqian Wang, Hengshu Zhu, Shangshang Yang, Xiaoming Zhang, and Xingyi Zhang.  Enhancing cognitive diagnosis using un-interacted exercises: A collaboration-aware mixed sampling approach.  In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, February 20-27, 2024, Vancouver, Canada, pages 8877–8885. AAAI Press, 2024.
  * [15] Yan Zhao, Huifang Ma, Jing Wang, Xiangchun He, and Liang Chang.  Question-response representation with dual-level contrastive learning for improving knowledge tracing.  Inf. Sci., 658:120032, 2024.
  * [16] Aritra Ghosh, Neil T. Heffernan, and Andrew S. Lan.  Context-aware attentive knowledge tracing.  In KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 2330–2339. ACM, 2020.
  * [17] Jianwen Sun, Fenghua Yu, Sannyuya Liu, Yawei Luo, Ruxia Liang, and Xiaoxuan Shen.  Adversarial bootstrapped question representation learning for knowledge tracing.  In Proceedings of the 31st ACM International Conference on Multimedia, MM 2023, Ottawa, ON, Canada, 29 October 2023- 3 November 2023, pages 8016–8025. ACM, 2023.
  * [18] Xinping Wang, Liangyu Chen, and Min Zhang.  Deep attentive model for knowledge tracing.  In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 10192–10199. AAAI Press, 2023.
  * [19] Jiatong Li, Fei Wang, Qi Liu, Mengxiao Zhu, Wei Huang, Zhenya Huang, Enhong Chen, Yu Su, and Shijin Wang.  Hiercdf: A bayesian network-based hierarchical cognitive diagnosis framework.  In KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 - 18, 2022, pages 904–913. ACM, 2022.
  * [20] Shuo Liu, Hong Qian, Mingjia Li, and Aimin Zhou.  QCCDM: A q-augmented causal cognitive diagnosis model for student learning.  In ECAI 2023 - 26th European Conference on Artificial Intelligence, (PAIS 2023), volume 372 of Frontiers in Artificial Intelligence and Applications, pages 1536–1543. IOS Press, 2023.
  * [21] Emiko Tsutsumi, Ryo Kinoshita, and Maomi Ueno.  Deep-irt with independent student and item networks.  In Proceedings of the 14th International Conference on Educational Data Mining, EDM 2021, virtual, June 29 - July 2, 2021. International Educational Data Mining Society, 2021.
  * [22] Lina Gao, Zhongying Zhao, Chao Li, Jianli Zhao, and Qingtian Zeng.  Deep cognitive diagnosis model for predicting students’ performance.  Future Gener. Comput. Syst., 126:252–262, 2022.
  * [23] Peichao Jiang and Xiaodong Wang.  Preference cognitive diagnosis for student performance prediction.  IEEE Access, 8:219775–219787, 2020.
  * [24] Ghodai Abdelrahman and Qing Wang.  Deep graph memory networks for forgetting-robust knowledge tracing.  IEEE Trans. Knowl. Data Eng., 35(8):7844–7855, 2023.
  * [25] Qi Liu, Run-ze Wu, Enhong Chen, Guandong Xu, Yu Su, Zhigang Chen, and Guoping Hu.  Fuzzy cognitive diagnosis for modelling examinee performance.  ACM Trans. Intell. Syst. Technol., 9(4):48:1–48:26, 2018.
  * [26] Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu.  EKT: exercise-aware knowledge tracing for student performance prediction.  IEEE Trans. Knowl. Data Eng., 33(1):100–115, 2021.
  * [27] Haiping Ma, Changqian Wang, Hengshu Zhu, Shangshang Yang, Xiaoming Zhang, and Xingyi Zhang.  Enhancing cognitive diagnosis using un-interacted exercises: A collaboration-aware mixed sampling approach.  CoRR, abs/2312.10110, 2023.
  * [28] Jacqueline P Leighton, Mark J Gierl, and Stephen M Hunka.  The attribute hierarchy method for cognitive assessment: A variation on tatsuoka’s rule-space approach.  Journal of educational measurement, 41(3):205–237, 2004.
  * [29] Jimmy De La Torre and Jeffrey A Douglas.  Higher-order latent trait models for cognitive diagnosis.  Psychometrika, 69(3):333–353, 2004.
  * [30] Weibo Gao, Hao Wang, Qi Liu, Fei Wang, Xin Lin, Linan Yue, Zheng Zhang, Rui Lv, and Shijin Wang.  Leveraging transferable knowledge concept graph embedding for cold-start cognitive diagnosis.  In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023, pages 983–992. ACM, 2023.
  * [31] Weibo Gao, Qi Liu, Hao Wang, Linan Yue, Haoyang Bi, Yin Gu, Fangzhou Yao, Zheng Zhang, Xin Li, and Yuanjing He.  Zero-1-to-3: Domain-level zero-shot cognitive diagnosis via one batch of early-bird students towards three diagnostic objectives.  In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, February 20-27, 2024, Vancouver, Canada, pages 8417–8426. AAAI Press, 2024.
  * [32] Blerta Abazi Chaushi, Besnik Selimi, Agron Chaushi, and Marika Apostolova.  Explainable artificial intelligence in education: A comprehensive review.  In Explainable Artificial Intelligence - First World Conference, xAI 2023, Lisbon, Portugal, July 26-28, 2023, Proceedings, Part II, volume 1902 of Communications in Computer and Information Science, pages 48–71. Springer, 2023.
  * [33] Kikumi K Tatsuoka.  Architecture of knowledge structures and cognitive diagnosis: A statistical pattern recognition and classification approach.  In Cognitively diagnostic assessment, pages 327–359. Routledge, 2012.
  * [34] Alexander E Curtis, Tanya A Smith, Bulat A Ziganshin, and John A Elefteriades.  The mystery of the z-score.  Aorta, 4(04):124–130, 2016.
  * [35] Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean.  Distributed representations of words and phrases and their compositionality.  In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 3111–3119, 2013.
  * [36] Alex Graves and Jürgen Schmidhuber.  Framewise phoneme classification with bidirectional LSTM and other neural network architectures.  Neural Networks, 18(5-6):602–610, 2005.
  * [37] Haw-Shiuan Chang, Hwai-Jung Hsu, and Kuan-Ta Chen.  Modeling exercise relationships in e-learning: A unified approach.  In Proceedings of the 8th International Conference on Educational Data Mining, EDM 2015, Madrid, Spain, June 26-29, 2015, pages 532–535. International Educational Data Mining Society (IEDMS), 2015.
  * [38] Hongbin Pei, Bo Yang, Jiming Liu, and Lei Dong.  Group sparse bayesian learning for active surveillance on epidemic dynamics.  In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 800–807. AAAI Press, 2018.
  * [39] Andrew P. Bradley.  The use of the area under the ROC curve in the evaluation of machine learning algorithms.  Pattern Recognit., 30(7):1145–1159, 1997.
  * [40] Francois Fouss, Alain Pirotte, Jean-michel Renders, and Marco Saerens.  Random-walk computation of similarities between nodes of a graph with application to collaborative recommendation.  IEEE Transactions on Knowledge and Data Engineering, 19(3):355–369, 2007.


# An Ensemble Method Based on the Combination of Transformers with
Convolutional Neural Networks to Detect Artificially Generated Text

Vijini Liyanage  Davide Buscaldi
LIPN , Université Sorbonne Paris Nord, CNRS UMR 7030
99 av. Jean-Baptiste Clément, 93430 Villetaneuse, France
<EMAIL_ADDRESS>
###### Abstract

Thanks to the state-of-the-art Large Language Models (LLMs), language
generation has reached outstanding levels. These models are capable of
generating high quality content, thus making it a challenging task to detect
generated text from human-written content. Despite the advantages provided by
Natural Language Generation, the inability to distinguish automatically
generated text can raise ethical concerns in terms of authenticity.
Consequently, it is important to design and develop methodologies to detect
artificial content. In our work, we present some classification models
constructed by ensembling transformer models such as Sci-BERT, DeBERTa and
XLNet, with Convolutional Neural Networks (CNNs). Our experiments demonstrate
that the considered ensemble architectures surpass the performance of the
individual transformer models for classification. Furthermore, the proposed
SciBERT-CNN ensemble model produced an F1-score of 98.36% on the ALTA shared
task 2023 data.

An Ensemble Method Based on the Combination of Transformers with Convolutional
Neural Networks to Detect Artificially Generated Text

Vijini Liyanage and Davide Buscaldi LIPN , Université Sorbonne Paris Nord,
CNRS UMR 7030 99 av. Jean-Baptiste Clément, 93430 Villetaneuse, France
<EMAIL_ADDRESS>

##  1 Introduction

Nowadays, people have access to state-of-the-art LLMs which help them simplify
some of their daily activities. One of the most notable breakthroughs in
recent years is the evolution of OpenAI’s GPT models which are capable of
generating text that looks as if they are written by a human. Especially, the
latest models such as ChatGPT and GPT4 OpenAI (2023) have won global attention
for providing solutions to any kind of question or concern that humans
possess. Moreover, these models produce outputs that appear to be written by a
human.

Thus there is a potential risk in determining the authenticity of textual
content that mankind refers to. Especially, in a domain such as academia,
leveraging generation models in composing articles might raise an ethical
concern. For example in ICML 2023, they have included a note under the
“Ethics” section prohibiting the use of text generated by ChatGPT and other
LLMs, unless “presented as part of the paper’s experiential analysis.”
111https://icml.cc/Conferences/2023/llm-policy. Accordingly, it is essential
to have mechanisms for detecting artificially composed text from human written
text.

Currently, a substantial amount of research has focused on the detection of
automatically generated text. Recent research (Zellers et al. (2019), Glazkova
and Glazkov (2022) and Liyanage and Buscaldi (2023)) mostly consider detection
as a binary classification task and leverage SOTA classification models to
distinguish machine-generated text from original text. Besides, some employ
statistical detection tools such as GLTR Gehrmann et al. (2019) or latest deep
learning based tools such as GPT2 output detector222https://openai-openai-
detector--5smxg.hf.space, DetectGPT Mitchell et al. (2023) or GPTZero
333https://gptzero.me/. Moreover, several researchers (Liyanage et al. (2022),
Kashnitsky et al. (2022)) have published corpora composed of machine-generated
content, which can be utilized by future research on detection.

Our work is based on the participation of our team in the ALTA shared task
2023 Molla et al. (2023) The objective of the task is to build automatic
detection systems that can discriminate between human-authored and synthetic
text generated by Large Language Models (LLMs). Their corpus is composed of
artificial contents that belong to a variety of domains (such as law, medical)
and are generated by models such as T5 Raffel et al. (2020) and GPT-X.

This paper is organized as follows. We provide the corpus and task description
in Section 2. In Section 3, we describe our methodology and Section 4, deliver
the experimental setup and the official results. Section 5 concludes this
paper.

##  2 Task Overview

###  2.1 Task Definition

The task at hand revolves around distinguishing between automatically
generated and human-written texts. In essence, it involves a binary
classification challenge where the goal is to categorize provided texts into
two distinct and exclusive groups. To outline this formally:

  * •

Input: We are presented with text segments.

  * •

Output: The objective is to assign one of two possible labels to each text
segment: either "human-written" or "machine-generated".

This undertaking aims to establish a clear boundary between texts created
through automated processes and those crafted by human authors. The primary
aim is to develop a model that can effectively differentiate between these two
categories based on the characteristics of the given excerpts.

###  2.2 Corpus

The dataset published for the ALTA shared task is a balanced one composed of
9000 original (human written) excerpts and 9000 fake (artificially generated)
excerpts. On average, the excerpts consist of 35 words each. To gain a deeper
comprehension of the corpus, category-wise (original vs generated) statistics
with respective example excerpts are provided in Table 1.

|  Original |  Generated
---|---|---
Min. word count |  10 |  1
Max. word count |  96 |  192
Avg. word count |  25 |  45
Example excerpt |  This is the data I collected so far (motorcycle standing on central stand, back wheel revolving, velocity comes from the back wheel, ABS LED blinking). |  In this sense, she emphasized that it was a mistake to tie development aid to times of economic booms, as it is a "permanent commitment".
Table 1: Statistics of the ALTA shared task corpus (The avg. figures are
rounded off to the nearest whole number)

##  3 Methodology

Given that the shared task frames detection as a binary classification
challenge, we utilized a range of classification models to address this
objective. In the subsequent subsections, in-depth explanations are provided
pertaining to the examined statistical, recurrent and transformer models, and
the corresponding ensemble architectures.

###  3.1 Statistical Models and their Respective Ensemble Architectures

In our work, we primarily employed Naive Bayes, Passive Aggressive and Support
Vector Machine (SVM), which are classification algorithms used in machine
learning to categorize data points into different classes Bishop and Nasrabadi
(2006). Naive Bayes is a probabilistic classification algorithm based on
Bayes’ theorem and it is widely used for tasks such as spam detection. It
assumes that the features are conditionally independent given the class label.
Passive Aggressive is a type of algorithm that aims to make aggressive updates
when it encounters a misclassified point and passive updates when the point is
correctly classified. SVM is a powerful supervised machine learning algorithm
used for classification and regression tasks. It is a popular algorithm in
text classification tasks. These algorithms were employed in conjunction with
the two text encoding methodologies, namely Bag of Words (BoW) and Term
Frequency-Inverse Document Frequency (TF-IDF).

Furthermore, we harnessed the capabilities of ensembles comprising the
aforementioned statistical models, applying various ensemble methodologies
such as voting, stacking, bagging, and boosting. By amalgamating the
predictions of multiple models, ensemble techniques aim to enhance the overall
predictive power of our system. Voting combines the outputs through a majority
or weighted decision, stacking involves training a meta-model on the
predictions of base models, bagging leverages bootstrapped subsets of data for
training individual models, and boosting iteratively adjusts model weights to
prioritize difficult-to-classify instances. Through these ensemble strategies,
we sought to extract richer insights from our data and attain improved
classification performance.

###  3.2 Recurrent Models and their Respective Ensemble Architectures

Recurrent models, a subset of neural network architectures, are models
designed to capture temporal dependencies and patterns within sequences. We
conducted experiments with LSTM and Bi-LSTM models, which are a type of RNN
architecture specifically designed to address the vanishing gradient problem
that can occur in traditional RNNs. To further improve classification
accuracies of these models, we ensembled them with a Convolutional Neural
Networks (CNNs) architecture. This approach helps in enhancing the predictive
capabilities overall model by capitalizing on their respective strengths in
capturing temporal dependencies and spatial features. We trained the entire
ensemble end-to-end, allowing the network to learn how to best combine the
features extracted by both LSTM and CNN components.

###  3.3 Transformer Models and their Respective Ensemble Architectures

For our classification experiments, we leveraged cutting-edge transformer
models, namely BERT, SciBERT, DeBERTa, and XLNet. These state-of-the-art
architectures have demonstrated exceptional proficiency in a wide spectrum of
natural language processing tasks, including classification. BERT
(Bidirectional Encoder Representations from Transformers) Devlin et al. (2018)
introduces bidirectional context by pretraining on a massive corpus and then
fine-tuning on task-specific data. SciBERT Beltagy et al. (2019) is
specialized for scientific text, adapting BERT’s embeddings to domain-specific
language. DeBERTa (Decoding-enhanced BERT with Disentangled Attention) He et
al. (2020) enhances attention mechanisms, capturing dependencies among words
more effectively. XLNet Yang et al. (2019) employs a permutation-based
training approach to capture bidirectional context and alleviate BERT’s
limitations.

Initially, we created ensembles by combining the capabilities of SciBERT and
DeBERTa models with the foundational BERT model. This process involves
channeling the data through each base model, which comprises the transformer
block along with a subsequent max pooling layer. Subsequently, the outcomes
derived from these individual models are concatenated to generate a unified
representation, which is then channeled into a linear classification layer for
making refined predictions.

Furthermore, we integrated the transformer models with Convolutional Neural
Networks (CNNs) to construct ensemble architectures that exhibit enhanced
performance. As depicted in the architectural diagram 1, the embeddings
produced by the transformer model are directed into a CNN layer. In our
approach, the labels won’t be actively employed since we’re solely utilizing
the BERT transformer without a distinct specialized component on the upper
layer. The CNN, placed as the upper layer, takes on the role of our main
component. We exclude the nn.Embedding layers, as the need for a lookup table
for embedding vectors is obviated. Instead, we directly infuse the embedding
vectors from BERT into the CNN architecture.

Figure 1: Architecture of Transformer-CNN Ensemble

##  4 Experiments and Results

The text underwent preliminary processing, involving the elimination of
stopwords and stemming, before being supplied to either statistical or neural
network architectures. The processed data was then transformed into numerical
vectors using Bag of Words (BoW) or tf-idf encoding techniques, which were
subsequently utilized as inputs for the statistical models. All of the
employed statistical models, as well as their corresponding ensemble methods,
were imported from the Scikit-learn library. For constructing LSTM and CNN
models, the relevant layers were imported from TensorFlow’s Keras module.
Training these recurrent models, including those combined with CNN ensembles,
involved running 10 epochs. The LSTM and Bi-LSTM architectures were trained
using batch sizes of 64 and 128, respectively.

Concerning transformer architectures and their associated ensembles, pre-
trained models from Hugging Face Wolf et al. (2020) were imported and
subsequently fine-tuned through the utilization of Simple Transformers
444https://simpletransformers.ai. The BERT tokenizer was consistently employed
across all models. The fine-tuning process involved 3 epochs, a batch size of
16, and a maximum sequence length of 128. Leveraging the T4 GPU Hardware
accelerator, the average training time for models was approximately 30
minutes. For standalone models, the input consisted of unprocessed text, while
ensembles underwent pre-processing involving punctuation removal and
conversion to lowercase. As represented in Figure 1, the CNN block of the
ensembles was composed of three convolutional layers.

The dataset was split in 80:20 ratio for training and testing. To assess the
classification performance of the models under consideration, the F1 score was
employed. This score, being a balanced combination of precision and recall,
offers a comprehensive evaluation. Each model underwent a total of five
experimental iterations, and the resultant average F1 scores are presented in
Table 2.

Model | F1
---|---
Statistical Models |
NB + BoW | 89.04
PA + BoW | 84.07
SVM + BoW | 87.51
NB + tf-idf | 89.02
NB + tf-idf | 91.00
NB + tf-idf | 91.42
Ensembles of Statistical Models |
Voting (NB + PA + SVM) + BoW | 90.29
Stacking (NB + PA + SVM) + BoW | 88.23
Bagging (NB + PA + SVM) + BoW | 91.56
Boosting (NB + PA + SVM) + BoW | 90.28
Recurrent Models |
LSTM | 49.08
Bi-LSTM | 90.58
Ensembles of RNNs |
LSTM + CNN | 49.08
Bi-LSTM + CNN | 90.02
Transformer Models |
BERTbase | 90.81
SciBERT | 94.89
DeBERTalarge | 96.67
XLNetlarge | 93.62
Ensembles of BERT models |
BERTbase \+ SciBERT | 97.80
BERTbase \+ DeBERTalarge | 97.47
Ensembles of transformers with CNN |
BERTbase \+ CNN | 97.42
SciBERT + CNN | 97.56
DeBERTalarge \+ CNN | 98.36
XLNetbase \+ CNN | 97.44
Table 2: Classification Scores

In general, the ensemble architectures have exhibited superior performance
compared to their corresponding original models. Our best-performing solution
is the combination of DeBERTalarge with CNN, achieving an F1 score of 98.36%.

##  5 Conclusion

In this work, we have explored the application of different SOTA
classification models on the detection of automatically generated text from
human written text. Moreover, we have created various ensemble methods with
the aforementioned models and examined their performance on the detection
task. Our results on the test data showed that generally the ensemble
architectures outperform the considered original models.

As future work, we plan to examine the applicability of our ensemble
architectures in detecting artificially generated text in multilingual
corpora. Another potential research direction involves assessing the
effectiveness of knowledge-based approaches for detecting artificial text.

## References

  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  Scibert: A pretrained language model for scientific text.  _arXiv preprint arXiv:1903.10676_.
  * Bishop and Nasrabadi (2006) Christopher M Bishop and Nasser M Nasrabadi. 2006.  _Pattern recognition and machine learning_ , volume 4.  Springer.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Gehrmann et al. (2019) Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. 2019.  Gltr: Statistical detection and visualization of generated text.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 111–116.
  * Glazkova and Glazkov (2022) Anna Glazkova and Maksim Glazkov. 2022.  Detecting generated scientific papers using an ensemble of transformer models.  In _Proceedings of the Third Workshop on Scholarly Document Processing_ , pages 223–228.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020.  Deberta: Decoding-enhanced bert with disentangled attention.  _arXiv preprint arXiv:2006.03654_.
  * Kashnitsky et al. (2022) Yury Kashnitsky, Drahomira Herrmannova, Anita de Waard, Georgios Tsatsaronis, Catriona Fennell, and Cyril Labbé. 2022.  Overview of the dagpap22 shared task on detecting automatically generated scientific papers.  In _Third Workshop on Scholarly Document Processing_.
  * Liyanage and Buscaldi (2023) Vijini Liyanage and Davide Buscaldi. 2023.  Detecting artificially generated academic text: The importance of mimicking human utilization of large language models.  In _International Conference on Applications of Natural Language to Information Systems_ , pages 558–565.
  * Liyanage et al. (2022) Vijini Liyanage, Davide Buscaldi, and Adeline Nazarenko. 2022.  A benchmark corpus for the detection of automatically generated text in academic publications.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 4692–4700.
  * Mitchell et al. (2023) Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023.  Detectgpt: Zero-shot machine-generated text detection using probability curvature.  _arXiv preprint arXiv:2301.11305_.
  * Molla et al. (2023) Diego Molla, Haolan Zhan, Xuanli He, and Qiongkai Xu. 2023.  Overview of the 2023 alta shared task: Discriminate between human-written and machine-generated text.  In _Proceedings of the 21st Annual Workshop of the Australasian Language Technology Association (ALTA 2023)_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_ , pages 38–45.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in neural information processing systems_ , 32.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019.  Defending against neural fake news.  _Advances in neural information processing systems_ , 32.