# Clues Before Answers: Generation-Enhanced Multiple-Choice QA

Zixian Huang  Ao Wu  Jiaying Zhou
State Key Laboratory for Novel Software Technology, Nanjing University,
Nanjing, China
<EMAIL_ADDRESS>
Yu Gu
The Ohio State University, Columbus, USA
<EMAIL_ADDRESS>
Yue Zhao  Gong Cheng
State Key Laboratory for Novel Software Technology, Nanjing University,
Nanjing, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

A trending paradigm for multiple-choice question answering (MCQA) is using a
text-to-text framework. By unifying data in different tasks into a single
text-to-text format, it trains a generative encoder-decoder model which is
both powerful and universal. However, a side effect of twisting a generation
target to fit the classification nature of MCQA is the under-utilization of
the decoder and the knowledge that can be decoded. To exploit the generation
capability and underlying knowledge of a pre-trained encoder-decoder model, in
this paper, we propose a generation-enhanced MCQA model named GenMC. It
generates a clue from the question and then leverages the clue to enhance a
reader for MCQA. It outperforms text-to-text models on multiple MCQA datasets.

##  1 Introduction

Multiple-choice question answering (MCQA) aims at selecting the correct answer
from a set of options given a question. This long-standing challenge in
natural language processing (NLP) requires machines to have a wealth of
knowledge, such as commonsense knowledge Talmor et al. (2019); Mihaylov et al.
(2018) and scientific knowledge Clark et al. (2018); Khot et al. (2020); Huang
et al. (2019); Li et al. (2021), and have reasoning skills such as multi-hop
reasoning Khot et al. (2019) and logical reasoning Yu et al. (2020); Liu et
al. (2020b); Li et al. (2022).

MCQA has made great progress with the development of pre-trained language
models (PLMs). Basically there are two types of PLMs that are suitable for
different tasks. BERT Devlin et al. (2019) and its variants such as RoBERTa
Liu et al. (2019) and ALBERT Lan et al. (2020) are encoder-only models, being
more suitable for natural language understanding (NLU) tasks including MCQA
and other classification and regression tasks. T5 Raffel et al. (2020) and
BART Lewis et al. (2020) are encoder-decoder models, being more suitable for
natural language generation (NLG) tasks. However, encoder-decoder models can
also be applied to MCQA Khashabi et al. (2020); Zhou et al. (2021). This is
enabled by the text-to-text framework, which transforms data in different
tasks into a unified text-to-text format so that knowledge spanning many and
various tasks can be learned, aggregated, and used by a single model.

##### Research Question

To fit MCQA, existing implementations of the text-to-text framework take all
the options as input and are trained to generate one of the options, i.e., to
copy some tokens from the input. However, this is inconsistent with how
encoder-decoder models are pre-trained so that their underlying knowledge may
not be sufficiently exploited. Indeed, Liu et al. (2021) have found that in
classification and regression tasks, the decoder layer is often under-
utilized. One research question is _how to apply pre-trained encoder-decoder
models in a more natural way to MCQA_ , in particular, to exploit their NLG
capabilities.

Figure 1: An example MCQA task and a generated clue. Bold underline indicates
the correct answer.

##### Our Contribution

Our idea is inspired by human behavior. When reading a question, humans are
sometimes triggered to associate the question with their background knowledge
to form some _clues_ even before reading the options. For simple questions, a
clue may be exactly the correct answer, while for complex questions, clues may
play an auxiliary role to help humans connect the question with the correct
answer. For example, for the question shown in Figure 1, the clue “paper”
forms an intermediate concept between “notebook” in the question and “tree” in
the correct answer.

With this idea, we propose to employ a pre-trained encoder-decoder model to
_generate_ a clue from the question by exploiting its underlying knowledge,
without seeing and being strictly confined to the options as in the text-to-
text framework. The clue representation is then leveraged by an encoder-based
model to read the options and make prediction. We refer to this generation-
enhanced MCQA model as GenMC. It significantly outperforms comparable models,
in particular, text-to-text models, on five MCQA datasets.

##### Outline

We discuss related work in Section 2, introduce GenMC in Section 3, describe
the experimental setup in Section 4, report the results in Section 5, and
conclude in Section 6.

##### Code

Our code is available on GitHub111https://github.com/nju-websoft/GenMC under
the Apache Licence 2.0.

##  2 Related Work

###  2.1 Text-to-Text Paradigm for MCQA

Recently, the text-to-text paradigm has achieved breakthrough results on many
NLP tasks Raffel et al. (2020); Lewis et al. (2020). As illustrated in Figure
2a, adopting this paradigm for MCQA, the question $Q$ and all the options
$\\{O_{1},O_{2},O_{3},O_{4}\\}$ are spliced into a text as input, and the
correct answer $O_{1}$ is used as the generation target. One benefit is that
extensive training data can be shared across different tasks. Using such a
framework, UnifiedQA Khashabi et al. (2020) integrates 20 QA datasets into a
unified format for training, and achieves state-of-the-art results on multiple
MCQA datasets. Similarly, CALM Zhou et al. (2021) learns concept-centric
knowledge from text for commonsense QA.

However, it might be debatable whether it is appropriate to train a
classification task via a generation target. Liu et al. (2021) point out that
the decoder layers of T5 are under-utilized when fine-tuning on classification
and regression tasks. Therefore, they propose a method to reduce the number of
T5 parameters to improve efficiency without reducing accuracy. By contrast, we
address this issue from a different perspective of how to exploit the NLG
capability of pre-trained encoder-decoder models for MCQA to improve accuracy.

Some other works propose new pre-trained models for unified generation and
classification tasks by designing universal encoders and task-specific
decoders Shao et al. (2021); Sun et al. (2021). They are orthogonal to our
work as we leverage existing pre-trained encoder-decoder models instead of
pre-training new models at an additional cost.

(a) Text-to-Text

(b) Encoder-Only

Figure 2: Paradigms for MCQA.

###  2.2 Encoder-Only Paradigm for MCQA

Benefiting from the powerful NLU capabilities of BERT-style PLMs Devlin et al.
(2019); Liu et al. (2019); Lan et al. (2020), the encoder-only paradigm has
been popular for MCQA. As illustrated in Figure 2b, in this paradigm, the
question $Q$ and each option in $\\{O_{1},O_{2},O_{3},O_{4}\\}$ are interacted
to calculate a score, and the option with the highest score is chosen as the
answer. Building on this, some works study how to design better attention-
based models to identify evidence Chen et al. (2019); Zhang et al. (2020); Zhu
et al. (2020). Other efforts mimic human behavior of reading evidence and
answering questions Ran et al. (2019); Tang et al. (2019); Sun et al. (2019).
There, evidence is derived from the given passage or retrieved from external
corpora. By contrast, we aim at exporting clues from pre-trained models
without resorting to extra sources.

Figure 3: Architecture of GenMC. To make the prediction $O_{p}\in\mathbb{O}$,
the clue generator first takes $Q$ as input and outputs a clue representation
$\mathbf{H}^{QC}$ which is indicative of the correct answer. The enhanced
reader then relies on the generated clue representation to better attend to
options from $\mathbb{O}$ and makes the final prediction. The whole model is
trained in an end-to-end manner with both the generation loss
$\mathcal{L}^{\text{GEN}}$ and the classification loss
$\mathcal{L}^{\text{READ}}$.

###  2.3 Knowledge in PLMs

Recently, PLMs have been used as knowledge bases Petroni et al. (2019), and
the knowledge in parameters can be exported via methods such as Prompt Jiang
et al. (2020); Shin et al. (2020). Exploiting the knowledge in PLMs for QA
tasks has come into play in many forms including question expansion Mao et al.
(2021) and question generation Shwartz et al. (2020).

There is also research on MCQA trying to exporting knowledge from PLMs before
answering. Rajani et al. (2019) propose CAGE as a framework for generating
explanations for commonsense QA. However, CAGE relies on explanations
annotated by humans, which are not available in many real scenarios and
datasets. Latcinnik and Berant (2020) propose a joint generator-classifier
model where the generator produces a human-readable textual hypothesis.
Although it somewhat improves the explainability of MCQA, in terms of accuracy
of MCQA there is little advancement. CEGI Liu et al. (2020c) is probably the
most similar work to ours. It first uses a generative model to generate
evidence, and then uses a reading model to incorporate the evidence and
predict the answer, both using answer supervision. However, the generative
model and the reading model are separate steps in a pipeline and are connected
only via the evidence text. Such token-level interaction can lead to
significant losses in accuracy as we will see in our experiments, where our
representation-level interaction exhibits better performance.

##  3 GenMC Model

In MCQA, a question $Q$ is given together with a set of $n$ options
$\mathbb{O}=\\{O_{1},\ldots,O_{n}\\}$ with exactly one option being the
correct answer. The key to finding the correct answer is to capture and deeply
understand the connection between $Q$ and each $O_{i}\in\mathbb{O}$, which
oftentimes is beyond the lexical level and requires a non-trivial entailment
process. We follow the trend of building on a pre-trained encoder-decoder
model and use the encoder to jointly encode $Q$ and each $O_{i}$. However,
previous works directly use the decoder to generate an option in $\mathbb{O}$,
i.e., using the decoder as a classifier, which may have under-exploited the
model’s NLG capability Liu et al. (2021). Moreover, a simple joint encoding of
$Q$ and each $O_{i}$ can only enable lexical-level reasoning Zellers et al.
(2019) which is insufficient for MCQA tasks.

Our proposed model GenMC overcomes these limitations. Building on a pre-
trained encoder-decoder model, GenMC firstly generates a clue which is
indicative of the correct answer, thereby exploiting the NLG capability and
underlying knowledge of the pre-trained encoder-decoder model. Then GenMC
employs the generated clue representation as intermediate knowledge connecting
the question and the correct answer to interact with and enhance a reader for
solving MCQA. Our model design mimics how humans solve an MCQA task, i.e.,
after reading a question, humans may firstly associate it with some of their
background knowledge (i.e., looking for clues) that helps them to later
identify the correct answer.

The overall architecture of GenMC is shown in Figure 3. The clue generator
(Section 3.1) first generates a clue representation only given $Q$. Then the
enhanced reader (Section 3.2) uses the generated clue to augment question-
option understanding.

###  3.1 Clue Generator

The clue generator takes the question $Q$ as input and autoregressively
outputs a clue $C={c_{1},\dots,c_{|C|}}$ using a pre-trained encoder-decoder
model.222For efficiency, we decode the clue greedily without performing beam
search. Note that not the clue text $C$ but its representation
$\mathbf{H}^{C}$ will be used in our model, although one could output $C$ as
evidence for explainability.

Specifically, we obtain the question representation
$\mathbf{H}^{Q}\in\mathbb{R}^{d\times|Q|}$ and the clue representation
$\mathbf{H}^{C}\in\mathbb{R}^{d\times|C|}$ from the last layer of the encoder
and of the decoder, respectively, where $d$ denotes the representation
dimension. $\mathbf{H}^{C}_{j}$, denoting the representation of the $j$-th
token $c_{j}\in C$, is computed as follows:

$\mathbf{p}^{C}_{j},\mathbf{H}^{C}_{j}=\mathtt{Decoder}(c_{j-1},\mathbf{H}_{<j}^{C},\mathbf{H}^{Q})\,,$
(1)

where $\mathtt{Decoder}$ $(\cdot,\cdot,\cdot)$ takes the last token $c_{j-1}$,
the representation for the decoding history $\mathbf{H}_{<j}^{C}$, and
$\mathbf{H}^{Q}$ as input, and outputs the hidden state $\mathbf{H}_{j}^{C}$
together with the probability distribution $\mathbf{p}^{C}_{j}$ over the
decoding vocabulary at the $j$-th step.

To encourage the tokens in $C$ to thoroughly interact with each other and with
$Q$, we strengthen the clue representation by passing it to a transformer
layer Vaswani et al. (2017) and obtain $\mathbf{H}^{QC}$:

$\mathbf{H}^{QC}=\mathtt{Transformer}([\mathbf{H}^{Q};\mathbf{H}^{C}])\,,$ (2)

where $[\cdot;\cdot]$ denotes concatenation. $\mathbf{H}^{QC}$ carries the
information of $C$ which can be helpful to better understand and answer $Q$.

###  3.2 Enhanced Reader

Previous works often directly model the relevance of each $O_{i}\in\mathbb{O}$
to $Q$ via joint encoding using a pre-trained encoder, which largely performs
superficial lexical reasoning Zellers et al. (2019). By contrast, we use the
previously generated clue representation to enhance our reader for a deeper
understanding of each question-option pair.

Specifically, we first concatenate $Q$ and each $O_{i}$ independently333A
delimiter ”$\backslash n$” is inserted between $Q$ and each $O_{i}$. and feed
the concatenated input into the pre-trained encoder (which is shared with our
clue generator) to obtain $O_{i}$’s contextualized representation
$\mathbf{H}_{i}^{QO}$, which constitutes a column of
$\mathbf{H}^{QO}\in\mathbb{R}^{d\times n}$ where $n=|\mathbb{O}|$.

Next, based on the clue representation $\mathbf{H}^{QC}$, our model
intensively reads each question-option pair and obtains the matching signal
between the clue and the option. Specifically, inspired by Huang et al.
(2021), we first use dual-attention Liu et al. (2020a) to fuse information
from $\mathbf{H}^{QO}_{i}$ to $\mathbf{H}^{QC}$ and from $\mathbf{H}^{QC}$ to
$\mathbf{H}^{QO}_{i}$. Then we perform max-pooling to aggregate the matching
features:

$\displaystyle\begin{split}(\hat{\mathbf{H}}_{i}^{QO},\hat{\mathbf{H}}_{i}^{QC})&=\text{DualAttention}(\mathbf{H}_{i}^{QO},\mathbf{H}^{QC})\,,\\\
\mathbf{f}^{QO}_{i}&=\text{Max-Pooling}(\hat{\mathbf{H}}_{i}^{QO})\,,\\\
\mathbf{f}^{QC}_{i}&=\text{Max-
Pooling}(\hat{\mathbf{H}}_{i}^{QC})\,.\end{split}$ (3)

To obtain the final score $s_{i}$ for each $O_{i}$, we concatenate the dual
matching features $\mathbf{f}^{QO}_{i}$ and $\mathbf{f}^{QC}_{i}$ and feed
them into a two-layer multi-layer perceptron (MLP):

$s_{i}=\text{Linear}(\text{ReLU}(\text{Linear}([\mathbf{f}^{QO}_{i};\mathbf{f}^{QC}_{i}])))\,.\\\
$ (4)

We select the option with the highest score as the predicted answer, denoted
as $O_{p}$.

###  3.3 Training Objective

We jointly train the clue generator and the enhanced reader in an end-to-end
fashion with a combined loss:

$\mathcal{L}=\mathcal{L}^{\text{GEN}}+\mathcal{L}^{\text{READ}}\,.$ (5)

##### Generator Loss

For $\mathcal{L}^{\text{GEN}}$, assuming that $O_{t}\in\mathbb{O}$ is the
correct answer containing $m$ tokens $a_{1},\dots,a_{m}$, we first use $O_{t}$
as the target to calculate our clue generator loss with teacher forcing:

$\displaystyle\begin{split}\mathbf{p}^{O_{t}}_{j},\mathbf{H}^{O_{t}}_{j}&=\mathtt{Decoder}(a_{j-1},\mathbf{H}_{<j}^{O_{t}},\mathbf{H}^{Q})\,,\\\
\mathcal{L}^{\text{GEN}}&=-\frac{1}{m}\sum_{j=1}^{m}\log\mathbf{p}^{O_{t}}_{j,a_{j}}\,,\end{split}$
(6)

where $\mathbf{p}^{O_{t}}_{j}$ denotes the probability distribution over the
decoding vocabulary at the $j$-th step, and $\mathbf{p}^{O_{t}}_{j,a_{j}}$ is
the probability of token $a_{j}$.

|  | Train set
---
size
| Dev set
---
size
| Test set
---
size
| Option
---
number
| Question
---
average length
| Option
---
average length
CSQA | 8,500 | 1,241 | 1,221 | 5 | 13.38 | 1.52
OBQA | 4,957 | 500 | 500 | 4 | 10.65 | 2.85
ARC-Easy | 2,241 | 567 | 2,365 | 4 | 19.36 | 3.73
ARC-Challenge | 1,117 | 295 | 1,165 | 4 | 22.30 | 4.93
QASC | 7,320 | 814 | 926 | 8 | 8.12 | 1.64
Table 1: Dataset statistics. For CSQA and QASC, their official dev sets are
used as our test sets, and our dev sets are in-house split from their official
training sets.

##### Reader Loss

For $\mathcal{L}^{\text{READ}}$, we simply calculate a cross-entropy loss
given the correct answer $O_{t}\in\mathbb{O}$ as follows:

$\mathcal{L}^{\text{READ}}=-\log\frac{\exp(s_{t})}{\sum_{i=1}^{n}\exp(s_{i})}\,.$
(7)

Note that we update the encoder using the joint loss $\mathcal{L}$, while we
do not allow $\mathcal{L}^{\text{READ}}$ to be backpropagated to the decoder
part to reduce the memory consumption.

The above training objective exploits the double properties of the correct
answer $O_{t}$ in MCQA: as a text and as an index. We use $O_{t}$ as a text to
supervise our clue generator, and as an index (i.e., classification label) to
supervise our enhanced reader. Such usage is more natural than the text-to-
text paradigm Khashabi et al. (2020); Zhou et al. (2021), thus having the
potential to outperform.

##  4 Experimental Setup

###  4.1 Data

We conducted experiments on five popular MCQA datasets spanning from
commonsense questions to scientific questions. The former requires commonsense
knowledge and reasoning, and the latter requires inference over scientific
facts.

##### Datasets

CSQA Talmor et al. (2019) and OBQA Mihaylov et al. (2018) are two commonsense
MCQA datasets created by crowd workers based on commonsense facts. Each
question is given with 5 options in CSQA and 4 options in OBQA. ARC-Easy and
ARC-Challenge, denoting two disjointed subsets of ARC Clark et al. (2018),
contain natural grade-school science questions with 4 options, where ARC-
Challenge comprises difficult questions which require more advanced reasoning.
QASC Khot et al. (2020) is collected from elementary and middle school level
science with 8 options for each question.

##### Train-Dev-Test Split

For OBQA, ARC-Easy, and ARC-Challenge we used their official train, dev, and
test sets. For CSQA and QASC, since the correct answers in the official test
set are not public, we took their official dev set as our test set for
experiments and randomly held out an in-house dev set from the training set.
The dataset statistics are shown in Table 1.

##### External Knowledge

For all these datasets, our experiments did not rely on any provided documents
or external corpora; a question was solely provided with its options to form
the input. It means that pre-trained models were used as the primary source of
knowledge in the experiments.

###  4.2 Implementation Details

We used two popular encoder-decoder models as a basis, BART Lewis et al.
(2020) and T5 Raffel et al. (2020). For each model, we experimented with its
Base and Large versions.

We used PyTorch 1.7. We used the Adam optimizer and set $\text{warmup
fraction}=0.1$, $\text{weight decay}=0.01$, $\text{maximum source length}=64$,
$\text{maximum target length}=32$, $\text{epoch}=30$, and early stop training
when there was no better result on the dev set after 5 epochs. For each model,
we searched for the best learning rate from $\\{1e-4,5e-5,1e-5\\}$, and for
the best batch size out of $\\{8,64\\}$.

Because neural models are known to be sensitive to different random seeds,
especially when the training set is small, we performed multiple experiments
for all models with different random seeds, and reported the mean and standard
deviation. For CSQA, OBQA, ARC-Easy, and QASC, we used three random seeds
$\\{1,10,20\\}$. For the smallest dataset ARC-Challenge, we used five random
seeds $\\{1,10,20,30,40\\}$.

All the experiments were performed on a GeForce RTX 3090 with 24G memory.

###  4.3 Evaluation Metric

For each model, we reported its proportion of correctly answered questions in
each dataset.

##  5 Experimental Results

CSQA OBQA ARC-Easy ARC-Challenge QASC dev test dev test dev test dev test dev
test BART${}_{\textsc{Base}}$ Text2Text${}_{\text{vanilla}}$ 51.62 ($\pm$0.04)
53.26 ($\pm$0.57) 54.93 ($\pm$0.83) 52.73 ($\pm$1.00) 51.55 ($\pm$1.38) 50.51
($\pm$1.82) 30.05 ($\pm$1.25) 24.95 ($\pm$1.10) 46.72 ($\pm$1.21) 26.78
($\pm$1.21) Text2Text${}_{\text{enc}}$ 50.63 ($\pm$0.66) 52.22 ($\pm$1.64)
55.87 ($\pm$1.10) 51.00 ($\pm$1.83) 49.03 ($\pm$1.86) 49.94 ($\pm$1.49) 32.32
($\pm$4.87) 26.24 ($\pm$2.01) 48.08 ($\pm$1.35) 17.06 ($\pm$0.39) GenMC 54.82
($\pm$0.61) 56.40 ($\pm$0.61) 58.53 ($\pm$0.31) 57.53 ($\pm$2.91) 59.38
($\pm$1.60) 56.80 ($\pm$0.28) 38.64 ($\pm$0.90) 33.82 ($\pm$1.66) 57.70
($\pm$0.43) 35.96 ($\pm$1.70) T5${}_{\textsc{Base}}$
Text2Text${}_{\text{vanilla}}$ 57.59 ($\pm$0.81) 60.93 ($\pm$0.73) 59.53
($\pm$0.81) 57.53 ($\pm$0.70) 52.20 ($\pm$0.31) 51.75 ($\pm$0.89) 29.38
($\pm$2.63) 23.69 ($\pm$2.47) 54.55 ($\pm$1.01) 37.94 ($\pm$1.47)
Text2Text${}_{\text{enc}}$ 58.96 ($\pm$1.21) 59.49 ($\pm$1.41) 60.67
($\pm$2.86) 57.07 ($\pm$3.03) 56.55 ($\pm$1.17) 52.92 ($\pm$0.29) 29.49
($\pm$5.13) 26.09 ($\pm$0.23) 56.84 ($\pm$0.84) 39.60 ($\pm$2.38) GenMC 60.65
($\pm$0.47) 63.45 ($\pm$0.29) 62.07 ($\pm$1.01) 61.67 ($\pm$0.58) 62.38
($\pm$0.67) 58.82 ($\pm$0.37) 43.62 ($\pm$0.52) 39.00 ($\pm$0.30) 58.93
($\pm$1.76) 41.72 ($\pm$1.18) BART${}_{\textsc{Large}}$
Text2Text${}_{\text{vanilla}}$ 65.58 ($\pm$2.72) 66.91 ($\pm$2.14) 62.66
($\pm$1.18) 61.46 ($\pm$1.74) 63.49 ($\pm$1.89) 62.81 ($\pm$2.15) 29.94
($\pm$2.32) 28.55 ($\pm$4.97) 64.57 ($\pm$2.21) 47.80 ($\pm$2.22)
Text2Text${}_{\text{enc}}$ 65.00 ($\pm$0.66) 67.35 ($\pm$0.90) 63.80
($\pm$1.44) 62.47 ($\pm$1.53) 68.20 ($\pm$2.04) 65.33 ($\pm$1.74) 35.37
($\pm$6.07) 31.13 ($\pm$5.86) 65.07 ($\pm$0.94) 47.19 ($\pm$0.71) GenMC 69.57
($\pm$0.89) 72.26 ($\pm$0.70) 68.93 ($\pm$1.17) 68.07 ($\pm$1.70) 72.43
($\pm$0.54) 68.68 ($\pm$0.34) 48.93 ($\pm$0.98) 45.52 ($\pm$1.54) 68.39
($\pm$0.68) 55.90 ($\pm$0.92) T5${}_{\textsc{Large}}$
Text2Text${}_{\text{vanilla}}$ 67.53 ($\pm$0.43) 70.63 ($\pm$0.74) 66.80
($\pm$0.87) 63.53 ($\pm$1.10) 65.61 ($\pm$0.18) 62.55 ($\pm$0.54) 43.05
($\pm$1.69) 42.83 ($\pm$2.00) 64.13 ($\pm$1.47) 57.74 ($\pm$0.82)
Text2Text${}_{\text{enc}}$ 68.41 ($\pm$0.73) 70.30 ($\pm$0.82) 65.93
($\pm$1.03) 63.67 ($\pm$0.46) 69.61 ($\pm$0.20) 66.65 ($\pm$0.34) 30.73
($\pm$3.15) 28.76 ($\pm$4.85) 65.27 ($\pm$1.55) 55.65 ($\pm$0.45) GenMC 71.10
($\pm$0.41) 72.67 ($\pm$1.02) 71.60 ($\pm$0.92) 66.87 ($\pm$1.33) 72.49
($\pm$0.77) 69.01 ($\pm$1.97) 49.83 ($\pm$2.06) 47.41 ($\pm$2.00) 67.61
($\pm$1.14) 58.06 ($\pm$0.92)

Table 2: Comparison with text-to-text models.

###  5.1 Main Results: Comparison with Text-to-Text Models

To empirically evaluate GenMC in terms of whether it better exploits the
potential of pre-trained encoder-decoder models for MCQA, we compare GenMC
with a standard text-to-text implementation and with a variant thereof for
analysis.

####  5.1.1 Baselines

##### Text2Text${}_{\text{vanilla}}$

The vanilla usage of pre-trained encoder-decoders for MCQA is to reform the
input and output in a way that can be directly processed by a encoder-decoder
model. Specifically, following Raffel et al. (2020), we concatenate the input
question with all candidate options, where each option is also preceded by its
option ID, and then prepend the sequence with a dataset name. The concatenated
sequence is fed into the encoder part to get a joint representation for the
question and all options. Based on the joint representation, the decoder
finally outputs an option ID. In this setting, the decoder is basically used
as a classifier.

##### Text2Text${}_{\text{enc}}$

Similar to Liu et al. (2021), we use only the encoder part of a pre-trained
encoder-decoder model. Each option is independently paired with the question
to obtain a joint representation using the encoder. Then the representation is
fed into a scorer (i.e., an MLP) to obtain a matching score for each question-
option pair. The model then predicts the option with the highest score. In
this setting, the decoder is totally unused. Though Liu et al. (2021) find
that their encoder-only model performs comparably to using the decoder as a
classifier, we argue that the decoder part can further improve the
performance, if being properly used.

####  5.1.2 Results

The main results (see Table 2) show that GenMC consistently and significantly
(with p-value < 0.01) outperforms Text2Text${}_{\text{vanilla}}$ and
Text2Text${}_{\text{enc}}$ on all datasets. For several settings, GenMC even
obtains an absolute gain of over 10%. For example, on the test set of the
challenging scientific MCQA dataset ARC-Challenge, T5${}_{\textsc{Base}}$ \+
GenMC improves T5${}_{\textsc{Base}}$ \+ Text2Text${}_{\text{vanilla}}$ from
an accuracy of 23.69% to 39.00%, suggesting a relative gain of around 65%.
These results demonstrate that GenMC is a more effective usage of pre-trained
encoder-decoder models than existing ones.

Moreover, we interestingly find that the decoder-free baseline
Text2Text${}_{\text{enc}}$ outperforms Text2Text${}_{\text{vanilla}}$ on over
half of the experiments. This indicates that the decoder’s general language
knowledge gained from pre-training is largely wasted by only using it as a
classifier, which may further explain the superior performance of our model
because GenMC can exploit the pre-trained decoder more effectively. In
addition, all Large models significantly outperform their Base counterparts.
This suggests that the embedded knowledge gained from pre-training is critical
to MCQA tasks, strengthening our point to make full use of pre-trained
encoders and decoders.

###  5.2 Comparison with Other Models

| CSQA | OBQA | ARC-Easy | ARC-Challenge | QASC
---|---|---|---|---|---
| dev | test | dev | test | dev | test | dev | test | dev | test
Base |  |  |  |  |  |  |  |  |  |
RoBERTa | 56.51 ($\pm$0.34) | 58.91 ($\pm$0.79) | 58.67 ($\pm$1.03) | 49.67 ($\pm$0.76) | 56.56 ($\pm$0.91) | 52.32 ($\pm$0.70) | 38.64 ($\pm$0.90) | 34.85 ($\pm$2.20) | 55.28 ($\pm$0.12) | 34.38 ($\pm$1.72)
ALBERT | 53.16 ($\pm$0.58) | 53.95 ($\pm$0.49) | 54.53 ($\pm$1.10) | 49.20 ($\pm$2.27) | 48.32 ($\pm$0.88) | 45.84 ($\pm$1.94) | 34.80 ($\pm$1.53) | 30.21 ($\pm$1.74) | 40.99 ($\pm$(1.78) | 24.55 ($\pm$1.23)
UnifiedQA${}_{\text{T5}}$ $\ast$ | - | 45.00 ($\pm$0.00) | - | 59.00 ($\pm$0.00) | - | 53.00 ($\pm$0.00) | - | 42.40 ($\pm$0.00) | - | 25.80 ($\pm$0.00)
UnifiedQA${}_{\text{T5}}$ | 41.02 ($\pm$0.00) | 44.80 ($\pm$0.00) | 59.20 ($\pm$0.00) | 59.60 ($\pm$0.00) | 54.85 ($\pm$0.00) | 53.66 ($\pm$0.00) | 44.75 ($\pm$0.00) |  42.58 ($\pm$0.00) | 17.94 ($\pm$0.00) | 25.70 ($\pm$0.00)
UnifiedQA${}_{\text{T5-FT}}$ | 56.81 ($\pm$0.49) | 62.35 ($\pm$0.80) | 60.80 ($\pm$0.72) | 58.47 ($\pm$0.64) | 54.97 ($\pm$0.20) | 53.88 ($\pm$0.39) |  45.31 ($\pm$0.39) | 42.43 ($\pm$0.47) | 55.57 ($\pm$0.58) |  43.20 ($\pm$0.57)
GenMC${}_{\text{T5}}$ |  60.65 ($\pm$0.47) |  63.45 ($\pm$0.29) |  62.07 ($\pm$1.01) |  61.67 ($\pm$0.58) |  62.38 ($\pm$0.67) |  58.82 ($\pm$0.37) | 43.62 ($\pm$0.52) | 39.00 ($\pm$0.30) |  58.93 ($\pm$1.76) | 41.72 ($\pm$1.18)
Large |  |  |  |  |  |  |  |  |  |
RoBERTa | 68.92 ($\pm$0.76) | 71.88 ($\pm$0.26) | 67.80 ($\pm$1.22) | 64.47 ($\pm$1.41) | 65.73 ($\pm$0.80) | 62.40 ($\pm$0.89) | 38.08 ($\pm$1.99) | 35.97 ($\pm$1.74) | 67.32 ($\pm$0.58) | 50.22 ($\pm$1.88)
ALBERT | 60.62 ($\pm$0.57) | 59.32 ($\pm$0.91) | 54.50 ($\pm$1.40) | 49.27 ($\pm$0.64) | 54.03 ($\pm$0.45) | 53.77 ($\pm$1.81) | 33.90 ($\pm$1.22) | 31.19 ($\pm$3.79) | 51.11 ($\pm$1.72) | 33.12 ($\pm$1.24)
UnifiedQA${}_{\text{T5}}$ $\ast$ | - | 60.90 ($\pm$0.00) | - | 68.40 ($\pm$0.00) | - | 65.90 ($\pm$0.00) | - | 54.40 ($\pm$0.00) | - | 43.30 ($\pm$0.00)
UnifiedQA${}_{\text{T5}}$ | 55.28 ($\pm$0.00) | 61.34 ($\pm$0.00) | 70.40 ($\pm$0.00) | 68.40 ($\pm$0.00) | 69.31 ($\pm$0.00) | 66.43 ($\pm$0.00) | 56.61 ($\pm$0.00) | 54.33 ($\pm$0.00) | 29.24 ($\pm$0.00) | 43.74 ($\pm$0.00)
UnifiedQA${}_{\text{T5-FT}}$ | 69.00 ($\pm$0.51) |  73.60 ($\pm$0.45) | 70.53 ($\pm$0.23) |  68.80 ($\pm$0.69) | 69.72 ($\pm$0.71) | 66.92 ($\pm$0.85) |  56.84 ($\pm$0.39) |  54.42 ($\pm$0.15) | 66.63 ($\pm$1.56) |  58.71 ($\pm$0.90)
GenMC${}_{\text{T5}}$ |  71.10 ($\pm$0.41) | 72.67 ($\pm$1.02) |  71.60 ($\pm$0.92) | 66.87 ($\pm$1.33) |  72.49 ($\pm$0.77) |  69.01 ($\pm$1.97) | 49.83 ($\pm$2.06) | 47.41 ($\pm$2.00) |  67.61 ($\pm$1.14) | 58.06 ($\pm$0.92)

Table 3: Comparison with other models. (* indicates the results reported by
Khashabi et al. (2020).)

CSQA OBQA ARC-Easy ARC-Challenge QASC dev test dev test dev test dev test dev
test Base UnifiedQA${}_{\text{T5-FT}}$ 56.81 ($\pm$0.49) 62.35 ($\pm$0.80)
60.80 ($\pm$0.72) 58.47 ($\pm$0.64) 54.97 ($\pm$0.20) 53.88 ($\pm$0.39) 45.31
($\pm$0.39) 42.43 ($\pm$0.47) 55.57 ($\pm$0.58) 43.20 ($\pm$0.57)
GenMC${}_{\text{T5-U}}$ 61.24 ($\pm$0.45) 63.45 ($\pm$0.76) 62.33 ($\pm$0.81)
59.20 ($\pm$1.91) 61.73 ($\pm$0.35) 59.35 ($\pm$0.43) 45.54 ($\pm$0.20) 43.98
($\pm$0.36) 60.16 ($\pm$0.07) 45.43 ($\pm$0.87) Large
UnifiedQA${}_{\text{T5-FT}}$ 69.00 ($\pm$0.51) 73.60 ($\pm$0.45) 70.53
($\pm$0.23) 68.80 ($\pm$0.69) 69.72 ($\pm$0.71) 66.92 ($\pm$0.85) 56.84
($\pm$0.39) 54.42 ($\pm$0.15) 66.63 ($\pm$1.56) 58.71 ($\pm$0.90)
GenMC${}_{\text{T5-U}}$ 71.58 ($\pm$0.25) 72.26 ($\pm$0.31) 71.67 ($\pm$0.46)
69.00 ($\pm$0.69) 73.90 ($\pm$0.47) 72.87 ($\pm$0.50) 59.55 ($\pm$1.09) 55.97
($\pm$0.62) 68.55 ($\pm$0.81) 58.75 ($\pm$0.56)

Table 4: Comparison with UnifiedQA after unifying training sets.

####  5.2.1 Baselines

##### UnifiedQA

Existing methods that rely on external documents or corpora have achieved
state-of-the-art performance on several MCQA datasets. However, to enable a
fair comparison, we only compare with models that adopt the same setting as
ours, where a question and its options are the only input to the model. Among
these models, UnifiedQA Khashabi et al. (2020) is the current best model.
While UnifiedQA reports the best score using its T5-11B version, since for T5
we experiment with its Base and Large versions, we only report and compare
under T5${}_{\textsc{Base}}$ and T5${}_{\textsc{Large}}$. Note that instead of
training on each dataset separately, UnifiedQA converts a line of popular QA
datasets with four formats (e.g., retrieval-based QA, MCQA) into a unified
format, and trains a single model over all training data, while GenMC only
uses each dataset’s own training data.

##### RoBERTa and ALBERT

In addition, we compare with two encoder-only models, RoBERTa Liu et al.
(2019) and ALBERT Lan et al. (2020), which have served as the basis of many
MCQA models.

All models are of comparable model size to ours.

CSQA OBQA ARC-Easy ARC-Challenge QASC dev test dev test dev test dev test dev
test BART${}_{\textsc{Base}}$ GenMC 54.82 ($\pm$0.61) 56.40 ($\pm$0.61) 58.53
($\pm$0.31) 57.53 ($\pm$2.91) 59.38 ($\pm$1.60) 56.80 ($\pm$0.28) 38.64
($\pm$0.90) 33.82 ($\pm$1.66) 57.70 ($\pm$0.43) 35.96 ($\pm$1.70) Weak Clue
53.96 ($\pm$1.01) 54.35 ($\pm$1.97) 55.53 ($\pm$1.27) 54.27 ($\pm$0.92) 57.20
($\pm$1.80) 55.42 ($\pm$1.26) 39.89 ($\pm$0.20) 32.62 ($\pm$0.31) 54.05
($\pm$0.21) 25.99 ($\pm$0.82) Token Clue 45.53 ($\pm$1.28) 46.41 ($\pm$1.79)
54.07 ($\pm$1.72) 52.93 ($\pm$1.10) 48.97 ($\pm$0.91) 48.87 ($\pm$1.29) 31.19
($\pm$0.59) 27.64 ($\pm$0.69) 49.06 ($\pm$0.39) 21.31 ($\pm$1.03)
T5${}_{\textsc{Base}}$ GenMC 60.65 ($\pm$0.47) 63.45 ($\pm$0.29) 62.07
($\pm$1.01) 61.67 ($\pm$0.58) 62.38 ($\pm$0.67) 58.82 ($\pm$0.37) 43.62
($\pm$0.52) 39.00 ($\pm$0.30) 58.93 ($\pm$1.76) 41.72 ($\pm$1.18) Weak Clue
58.80 ($\pm$0.70) 60.88 ($\pm$1.89) 61.47 ($\pm$0.95) 59.73 ($\pm$0.90) 58.97
($\pm$0.54) 57.10 ($\pm$0.72) 42.26 ($\pm$2.21) 37.54 ($\pm$0.64) 57.37
($\pm$1.40) 36.29 ($\pm$1.66) Token Clue 50.55 ($\pm$0.44) 48.79 ($\pm$0.87)
56.00 ($\pm$1.25) 54.93 ($\pm$1.63) 46.50 ($\pm$0.83) 46.65 ($\pm$0.54) 32.66
($\pm$0.20) 26.01 ($\pm$1.28) 43.69 ($\pm$1.52) 27.50 ($\pm$1.56)
BART${}_{\textsc{Large}}$ GenMC 69.57 ($\pm$0.89) 72.26 ($\pm$0.70) 68.93
($\pm$1.17) 68.07 ($\pm$1.70) 72.43 ($\pm$0.54) 68.68 ($\pm$0.34) 48.93
($\pm$0.98) 45.52 ($\pm$1.54) 68.39 ($\pm$0.68) 55.90 ($\pm$0.92) Weak Clue
67.28 ($\pm$2.39) 69.64 ($\pm$2.76) 66.20 ($\pm$0.53) 64.47 ($\pm$1.40) 70.66
($\pm$1.50) 65.71 ($\pm$1.47) 27.80 ($\pm$2.06) 24.92 ($\pm$2.06) 65.68
($\pm$1.31) 52.02 ($\pm$1.44) Token Clue 53.85 ($\pm$0.47) 55.23 ($\pm$0.62)
61.20 ($\pm$3.14) 59.20 ($\pm$0.69) 58.02 ($\pm$0.98) 54.22 ($\pm$1.27) 41.81
($\pm$1.19) 37.60 ($\pm$0.90) 48.65 ($\pm$1.23) 32.47 ($\pm$1.11)
T5${}_{\textsc{Large}}$ GenMC 71.10 ($\pm$0.41) 72.67 ($\pm$1.02) 71.60
($\pm$0.92) 66.87 ($\pm$1.33) 72.49 ($\pm$0.77) 69.01 ($\pm$1.97) 49.83
($\pm$2.06) 47.41 ($\pm$2.00) 67.61 ($\pm$1.14) 58.06 ($\pm$0.92) Weak Clue
68.33 ($\pm$1.62) 71.66 ($\pm$1.28) 69.27 ($\pm$0.42) 65.87 ($\pm$0.90) 69.66
($\pm$0.77) 66.24 ($\pm$0.79) 47.57 ($\pm$2.04) 46.24 ($\pm$1.29) 64.99
($\pm$0.74) 53.35 ($\pm$1.35) Token Clue 59.47 ($\pm$0.08) 60.74 ($\pm$0.29)
62.80 ($\pm$1.44) 57.73 ($\pm$1.10) 48.85 ($\pm$1.62) 48.36 ($\pm$2.15) 37.97
($\pm$0.90) 30.50 ($\pm$1.46) 49.22 ($\pm$0.62) 38.77 ($\pm$1.74)

Table 5: Influence of clues.

####  5.2.2 Results

The results in Table 3 show that GenMC${}_{\text{T5}}$ significantly (with
p-value < 0.01) outperforms the two encoder-only strong baselines RoBERTa and
ALBERT. More interestingly, GenMC${}_{\text{T5}}$ also performs better than
UnifiedQA${}_{\text{T5}}$ on most datasets. Moreover, for
UnifiedQA${}_{\text{T5-FT}}$, which further fine-tunes the model on the
training set of the target dataset, GenMC${}_{\text{T5}}$ outperforms it on
the test sets of CSQA, OBQA, and ARC-Easy for the base models and ARC-Easy for
the large models. It also achieves comparable results on the remaining
datasets. These results are impressive because UnifiedQA uses more datasets
(i.e., eight different QA datasets) for training. The promising results of
GenMC further reveals that our model can learn to effectively extract
knowledge from pre-trained encoder-decoders with limited training data.

As a fairer comparison in Table 4, by unifying the training sets of all the
five datasets, our GenMC${}_{\text{T5-U}}$ outperforms
UnifiedQA${}_{\text{T5-FT}}$ on all datasets except for CSQA with large
models.

###  5.3 Ablation Study: Influence of Clues

Our main results in Section 5.1 have demonstrated the effectiveness of our
model. To better understand its superior results and the influence of our clue
generation, we compare with two variants.

####  5.3.1 Variants of GenMC

##### Weak Clue

We train this variant only using the classification loss
$\mathcal{L}^{\text{READ}}$, so only the encoder part is updated, while the
decoder part is left untouched from pre-training. Intuitively, under this
setting, the generated clue is weaker than GenMC which learns how to generate
a clue with supervision.

##### Token Clue

In this setting, we separately train a clue generator and a reader. We first
collect the generated clue text $C$ (instead of its representation) from the
decoder. We then directly concatenate $C$ with $Q$ and $O_{i}$ to compute a
score for $O_{i}$ using the model’s encoder part stacked with an MLP layer.
This variant is indeed very similar to Liu et al. (2020c), which also adopts a
pipeline framework to first generate a token-level evidence and then use the
evidence to expand the question.

Clue Type Percentage Example Instance Clue Irrelevant 23.60% Which would you
likely find inside a beach ball? (A) cheese (B) steam (C) water (D) air a
squid Relevant but unhelpful 52.40% What may have been formed by a volcano?
(A) Mt. McKinley (B) Lake Pontchartrain (C) The great lakes (D) Niagara Falls
a lake Helpful 24.00% Where would there be an auditorium with only a single
person speaking? (A) lights (B) crowd (C) university campus (D) theater (E)
park school

Table 6: Distribution of clue types in negative cases with examples. Bold
underline indicates the correct answer, and italic indicates the predicted
label.

####  5.3.2 Results

Table 5 shows that masking out generation loss leads to substantial
performance drops across all datasets, demonstrating that fine-tuning the
decoder with generation loss $\mathcal{L}^{\text{GEN}}$ helps to derive useful
clues from pre-trained encoder-decoder models. We also observe that the
performance of using token-level clues lags much behind GenMC. This
demonstrates that naively using explicit knowledge in plain text, instead of
using implicit clues from the decoder’s hidden state, is inferior as it may
unnecessarily bring information loss and noise.

###  5.4 Error Analysis

We analyze the clues generated by GenMC using T5${}_{\textsc{Large}}$ with a
focus on instances that are correctly predicted by the baseline in our main
experiments (i.e., T5${}_{\textsc{Large}}$ \+ Text2Text${}_{\text{vanilla}}$),
while our GenMC fails. The intuition is that in these _negative cases_ , the
clues generated by GenMC may play a negative role. By studying these
potentially negative clues, we can gain more insights into how GenMC fails and
discuss venues for future improvement.

Specifically, we randomly sample 50 negative cases from
T5${}_{\textsc{Large}}$ \+ GenMC for each dataset. We show six graduate
students of computer science444They are volunteers recruited from the contact
author’s research group. They know and agree that their annotations will be
used for error analysis in a research paper. an instance along with the
generated clue, correct answer, and predicted answer. We then ask them to
categorize clues into the following families:555We follow a similar definition
by Shwartz et al. (2020).

  * •

Irrelevant: The clue is off topic or is not understandable.

  * •

Relevant but unhelpful: Though relevant, the clue makes a factually incorrect
statement, often on the contrary of the main question, or the clue contributes
relevant but insufficient knowledge for prediction, such as repetition of the
question or other distractors.

  * •

Helpful: The clue adds helpful information to answer the question.

| CSQA | OBQA | ARC-Easy | ARC-Challenge | QASC
---|---|---|---|---|---
T5${}_{\textsc{Base}}$ |  |  |  |  |
Text2Text${}_{\text{vanilla}}$ | 0.040 ($\pm$0.007) | 0.035 ($\pm$0.002) | 0.035 ($\pm$0.002) | 0.039 ($\pm$0.004) | 0.035 ($\pm$0.002)
UnifiedQA | 0.059 ($\pm$0.041) | 0.089 ($\pm$0.047) | 0.097 ($\pm$0.055) | 0.129 ($\pm$0.075) | 0.068 ($\pm$0.027)
GenMC | 0.069 ($\pm$0.019) | 0.107 ($\pm$0.046) | 0.113 ($\pm$0.060) | 0.121 ($\pm$0.053) | 0.072 ($\pm$0.027)
T5${}_{\textsc{Large}}$ |  |  |  |  |
Text2Text${}_{\text{vanilla}}$ | 0.077 ($\pm$0.008) | 0.083 ($\pm$0.012) | 0.081 ($\pm$0.012) | 0.084 ($\pm$0.014) | 0.078 ($\pm$0.011)
UnifiedQA | 0.108 ($\pm$0.037) | 0.178 ($\pm$0.096) | 0.190 ($\pm$0.107) | 0.257 ($\pm$0.127) | 0.130 ($\pm$0.052)
GenMC | 0.105 ($\pm$0.027) | 0.178 ($\pm$0.078) | 0.219 ($\pm$0.120) | 0.242 ($\pm$0.112) | 0.127 ($\pm$0.048)
Table 7: Inference time for answering a question (seconds).

To ensure the annotation quality, we aggregate annotated results from three
students for every dataset using majority vote. If all three students annotate
differently from each other for an instance, we introduce a fourth student to
arbitrate.

Table 6 shows the percent of each clue type across all datasets with an
example for each type. Figure 4 breaks down by dataset. Though the majority of
our clues are relevant (i.e., 76.4% of them are relevant across all datasets),
which seems positive, only 24% of the clues are deemed as helpful. This
suggests a great room for improvement. In our future research, we will focus
on how to generate more helpful clues from questions.

Figure 4: Distribution of clue types in negative cases on each dataset.

###  5.5 Inference Time and Model Size

Table 7 shows the inference time for answering a question. GenMC is slower
than Text2Text${}_{\text{vanilla}}$, but their inference time has the same
scale, suggesting that GenMC is more cost-effective considering its superior
accuracy. GenMC and UnifiedQA are comparable in inference time.

Among T5${}_{\textsc{Base}}$ based models, Text2Text${}_{\text{vanilla}}$ and
UnifiedQA have 223 M parameters, while GenMC is slightly larger with 234 M
parameters. Among T5${}_{\textsc{Large}}$ based models,
Text2Text${}_{\text{vanilla}}$ and UnifiedQA have 738 M parameters, while
GenMC has 757 M parameters.

##  6 Conclusion

We present GenMC, a simple yet effective model which tailors pre-trained
encoder-decoders for MCQA tasks. Compared with existing usages of pre-trained
encoder-decoders for MCQA, our model fully exploits the pre-trained encoder-
decoders’ NLG capabilities to generate a clue from the input question, which
facilitates deep understanding of question-option pairs. Experimental results
further verify the superiority of GenMC over existing usages. Notably, our
model achieves promising results without using any provided documents or
external corpora, showing an interesting application of PLMs by directly
inducing either commonsense or scientific knowledge from them through clue
generation.

In the future, we will focus on how to further improve the clue generation
quality, which remains a bottleneck of GenMC. We hope this work will spur more
research in how to better use pre-trained encoder-decoders for not only MCQA,
but also beyond; for tasks with divergent structures from the pre-training, a
smarter use of PLMs can boost the performance significantly.

## Acknowledgments

This work was supported in part by the NSFC (62072224) and in part by the
Beijing Academy of Artificial Intelligence (BAAI).

## References

  * Chen et al. (2019) Zhipeng Chen, Yiming Cui, Wentao Ma, Shijin Wang, and Guoping Hu. 2019.  Convolutional spatial attention model for reading comprehension with multiple-choice questions.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , pages 6276–6283. AAAI Press.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the AI2 reasoning challenge.  _CoRR_ , abs/1803.05457.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Huang et al. (2019) Zixian Huang, Yulin Shen, Xiao Li, Yuang Wei, Gong Cheng, Lin Zhou, Xinyu Dai, and Yuzhong Qu. 2019.  Geosqa: A benchmark for scenario-based question answering in the geography domain at high school level.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 5865–5870. Association for Computational Linguistics.
  * Huang et al. (2021) Zixian Huang, Ao Wu, Yulin Shen, Gong Cheng, and Yuzhong Qu. 2021.  When retriever-reader meets scenario-based multiple-choice questions.  In _Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021_ , pages 985–994. Association for Computational Linguistics.
  * Jiang et al. (2020) Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020.  How can we know what language models know.  _Trans. Assoc. Comput. Linguistics_ , 8:423–438.
  * Khashabi et al. (2020) Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020.  Unifiedqa: Crossing format boundaries with a single QA system.  In _Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020_ , volume EMNLP 2020 of _Findings of ACL_ , pages 1896–1907. Association for Computational Linguistics.
  * Khot et al. (2020) Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020\.  QASC: A dataset for question answering via sentence composition.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 8082–8090. AAAI Press.
  * Khot et al. (2019) Tushar Khot, Ashish Sabharwal, and Peter Clark. 2019.  What’s missing: A knowledge gap guided approach for multi-hop question answering.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 2814–2828. Association for Computational Linguistics.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  ALBERT: A lite BERT for self-supervised learning of language representations.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Latcinnik and Berant (2020) Veronica Latcinnik and Jonathan Berant. 2020.  Explaining question answering models through text generation.  _CoRR_ , abs/2004.05569.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 7871–7880. Association for Computational Linguistics.
  * Li et al. (2022) Xiao Li, Gong Cheng, Ziheng Chen, Yawei Sun, and Yuzhong Qu. 2022.  Adalogn: Adaptive logic graph network for reasoning-based machine reading comprehension.  _CoRR_ , abs/2203.08992.
  * Li et al. (2021) Xiao Li, Yawei Sun, and Gong Cheng. 2021.  TSQA: tabular scenario based question answering.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_ , pages 13297–13305. AAAI Press.
  * Liu et al. (2020a) Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, and Nan Duan. 2020a.  RikiNet: reading Wikipedia pages for natural question answering.  In _ACL_ , pages 6762–6771.
  * Liu et al. (2021) Frederick Liu, Siamak Shakeri, Hongkun Yu, and Jing Li. 2021.  Enct5: Fine-tuning T5 encoder for non-autoregressive tasks.  _CoRR_ , abs/2110.08426.
  * Liu et al. (2020b) Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020b.  Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020_ , pages 3622–3628. ijcai.org.
  * Liu et al. (2020c) Ye Liu, Tao Yang, Zeyu You, Wei Fan, and Philip S. Yu. 2020c.  Commonsense evidence generation and injection in reading comprehension.  In _Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGdial 2020, 1st virtual meeting, July 1-3, 2020_ , pages 61–73. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Mao et al. (2021) Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2021.  Generation-augmented retrieval for open-domain question answering.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 4089–4100. Association for Computational Linguistics.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a suit of armor conduct electricity? A new dataset for open book question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018_ , pages 2381–2391. Association for Computational Linguistics.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 2463–2473. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Rajani et al. (2019) Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Explain yourself! leveraging language models for commonsense reasoning.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 4932–4942. Association for Computational Linguistics.
  * Ran et al. (2019) Qiu Ran, Peng Li, Weiwei Hu, and Jie Zhou. 2019.  Option comparison network for multiple-choice reading comprehension.  _CoRR_ , abs/1903.03033.
  * Shao et al. (2021) Yunfan Shao, Zhichao Geng, Yitao Liu, Junqi Dai, Fei Yang, Li Zhe, Hujun Bao, and Xipeng Qiu. 2021.  CPT: A pre-trained unbalanced transformer for both chinese language understanding and generation.  _CoRR_ , abs/2109.05729.
  * Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020.  Autoprompt: Eliciting knowledge from language models with automatically generated prompts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 4222–4235. Association for Computational Linguistics.
  * Shwartz et al. (2020) Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020\.  Unsupervised commonsense question answering with self-talk.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 4615–4629. Association for Computational Linguistics.
  * Sun et al. (2019) Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2019.  Improving machine reading comprehension with general reading strategies.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 2633–2643. Association for Computational Linguistics.
  * Sun et al. (2021) Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, Weixin Liu, Zhihua Wu, Weibao Gong, Jianzhong Liang, Zhizhou Shang, Peng Sun, Wei Liu, Xuan Ouyang, Dianhai Yu, Hao Tian, Hua Wu, and Haifeng Wang. 2021.  ERNIE 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation.  _CoRR_ , abs/2107.02137.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4149–4158. Association for Computational Linguistics.
  * Tang et al. (2019) Min Tang, Jiaran Cai, and Hankz Hankui Zhuo. 2019.  Multi-matching network for multiple choice reading comprehension.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , pages 7088–7095. AAAI Press.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Yu et al. (2020) Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020.  Reclor: A reading comprehension dataset requiring logical reasoning.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  HellaSwag: Can a machine really finish your sentence? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4791–4800, Florence, Italy. Association for Computational Linguistics.
  * Zhang et al. (2020) Shuailiang Zhang, Hai Zhao, Yuwei Wu, Zhuosheng Zhang, Xi Zhou, and Xiang Zhou. 2020\.  DCMN+: dual co-matching network for multi-choice reading comprehension.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 9563–9570. AAAI Press.
  * Zhou et al. (2021) Wangchunshu Zhou, Dong-Ho Lee, Ravi Kiran Selvam, Seyeon Lee, and Xiang Ren. 2021\.  Pre-training text-to-text transformers for concept-centric common sense.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Zhu et al. (2020) Pengfei Zhu, Hai Zhao, and Xiaoguang Li. 2020.  Dual multi-head co-attention for multi-choice reading comprehension.  _CoRR_ , abs/2001.09415.


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

IberLEF 2023, September 2023, Jaén, Spain

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]

# UPB at IberLEF-2023 AuTexTification: Detection of Machine-Generated Text
using Transformer Ensembles

Andrei-Alexandru Preda  Dumitru-Clementin Cercel  Traian Rebedea  Costin-
Gabriel Chiru  Computer Science and Engineering Department, Faculty of
Automatic Control and Computers, University Politehnica of Bucharest,
Bucharest 060042, Romania

(2023)

###### Abstract

This paper describes the solutions submitted by the UPB team to the
AuTexTification shared task, featured as part of IberLEF-2023. Our team
participated in the first subtask, identifying text documents produced by
large language models instead of humans. The organizers provided a bilingual
dataset for this subtask, comprising English and Spanish texts covering
multiple domains, such as legal texts, social media posts, and how-to
articles. We experimented mostly with deep learning models based on
Transformers, as well as training techniques such as multi-task learning and
virtual adversarial training to obtain better results. We submitted three
runs, two of which consisted of ensemble models. Our best-performing model
achieved macro F1-scores of $66.63\%$ on the English dataset and $67.10\%$ on
the Spanish dataset.

###### keywords:

Machine-Generated Text Transformer Multi-Task Learning Virtual Adversarial
Training

##  1 Introduction

Recently, computer-generated content started growing in presence on the
Internet. With the public release of powerful Large Language Models (LLMs)
such as the Generative Pre-trained Transformer [1], and its derivative systems
such as ChatGPT [2], manufacturing texts is easier than ever and probably
harder to detect than ever. This phenomenon has already raised several ethical
issues that society must answer soon. This effort can be helped by finding
mechanisms to automatically and reliably detect computer-generated text.

The AuTexTification: Automated Text Identification shared task [3] is a
natural language processing (NLP) competition at IberLEF-2023 [4]. Its main
focus is detecting and understanding the computer-generated text, especially
that of LLMs. The competition presents two subtasks: (1) Subtask 1 is a binary
classification problem in which we have to detect whether a human or an
artificial intelligence model wrote a document, and (2) Subtask 2 is a multi-
class classification problem in which you have to select which LLM generated a
given document from a list of several LLMs. To address these subtasks, the
organizers made available a bilingual dataset of documents produced by humans
and computers in English and Spanish, covering several domains.

Our team participated only in the first subtask. We first experimented with
more standard machine learning methods before moving to deep learning models,
where we explored techniques such as multi-task learning (MTL) [5] and virtual
adversarial training (VAT) [6]. Finally, we combined multiple models trained
independently to form ensembles, which we used to generate our submissions,
since they performed the best.

##  2 Related Work

While text classification is one of NLP’s fundamental and most well-
established tasks, detecting computer-generated text is a relatively novel
task. This is probably because, until recently, few systems could produce text
realistic enough to fool humans. Creating such texts is commonly called
natural language generation [7].

Currently, there seem to be various ways of addressing this problem, which can
be classified into black-box and white-box methods [8]. White-box techniques
require access to the target language model, and they can involve concepts
such as watermarks which the models could embed into their outputs to make
detection easier. As such, black-box methods are more relevant to the
previously mentioned task since we only have access to the model’s output, but
we do not even know which model produced it.

Black-box methods can involve both classical machine learning classification
algorithms, as well as ones based on deep learning [8]. To make predictions,
traditional algorithms combine statistical features and linguistic patterns
with classifiers such as Support Vector Machines (SVMs) [9]. On the other
hand, deep learning methods usually involve fine-tuning pre-trained language
models using supervised learning in order to make predictions. These deep
learning approaches often obtain state-of-the-art results but are harder to
interpret, which means they are also harder to trust, as well.

##  3 Methods

This section describes the different classification methods we tried and the
final ensemble architectures we submitted.

###  3.1 Shallow Learning Models

####  3.1.1 Readability Scores

Similar to Stodden and Venugopal [10], we combined several linguistic features
with pre-trained embeddings. Specifically, we computed the following
readability scores: the Flesch reading ease score [11], the Gunning-Fog index
[12], and the SMOG index [13]. The intuition behind this choice was that LLMs
might not consider the ease of comprehension when generating texts. For
example, the generated legal texts might be harder to understand than those
written by humans. To compute the aforementioned scores, we used the
Readability Python library [14], which offered $35$ such features.

Then, we concatenated the readability scores with document-level pre-trained
embeddings offered by the spaCy library [15], which are $300$-dimensional and
language-specific. The English embeddings are based on GloVe [16], while the
Spanish ones are based on FastText [17]. Finally, all features were scaled to
have zero mean and unit variance with scikit-learn’s
`StandardScaler}~\citep`scikit-learn, before using them to train two
classifiers, namely XGBoost [19] and k-Nearest Neighbors (kNN) [18].

####  3.1.2 String Kernels

We also experimented with string kernels [20], which are kernel functions that
measure the degree of similarity between two strings. An example of a simple
string kernel counts the number of n-grams shared by the two strings without
considering duplicates. Such a function can be computed for multiple sizes of
n-grams, and used as the kernel function of a classifier such as an SVM. We
performed common natural language preprocessing operations on the input text:
removing punctuation, removing stopwords, lowercasing all letters, and
stemming the words. We used n-gram sizes between $3$ and $5$, and the SVM
classifier implemented in scikit-learn [18]. Since custom kernels might need
to be computed between each pair of input samples, using the entire training
dataset would have taken a long time, so we tested the method only on a small
slice of it, comprising several thousand samples.

###  3.2 Deep Learning Models

####  3.2.1 Transformers

The Transformer architecture was introduced in 2017 by Vaswani et al. [21] and
is currently powering numerous state-of-the-art solutions for many tasks.
Transformers usually feature two main components: an encoder and a decoder.
However, these two parts can be useful by themselves as well. One example is
the Bidirectional Encoder Representations from Transformers (BERT) [22] model
family, which can encode input text into contextual embeddings. We
experimented with several BERT versions: multilingual ones (i.e., XLM-RoBERTa
[23] and multilingual BERT [22]), and one pre-trained on tweets (i.e., TwHIN-
BERT [24]). Since BERT models can be large and typically require large amounts
of data to train from scratch, we utilize transfer learning [25] instead, by
fine-tuning pre-trained models. We experimented with Transformer-based models
to encode the raw input text into embeddings, which we then connected to a
linear layer, followed by a dropout layer [26] before the final prediction
head. The last layer produces a probability of the document being computer-
generated, and the binary prediction is chosen by comparing it with a
threshold.

####  3.2.2 Multi-Task Learning

As an additional method of preventing overfitting, we used the technique of
multi-task learning. MTL refers to training a model to solve multiple tasks
simultaneously. As such, these models typically feature a set of parameters
shared for all tasks, and separate prediction heads for each task.
Intuitively, multi-task learning should make the task harder to solve, thus
adding extra complexity, which the model has to adapt to. In our case, an
extra task that is easy to derive is predicting the language of a given input
document. More precisely, apart from predicting the human/computer label of a
document, the model has to detect whether it is written in English or Spanish.
This means that, for training, we combined the two datasets supplied for
Subtask 1. However, we did not use any of the data provided for Subtask 2. The
MTL architecture is very similar to the one presented in the previous section,
only adding an extra classification head. The architecture can be seen in
Figure 3.3. Since both tasks involve binary classification, we compute a
binary cross-entropy loss for each of them, namely
$\mathcal{L}_{\mathrm{bot}}$ for the human/computer classification task, and
$\mathcal{L}_{\mathrm{lang}}$ for the language detection task. The final loss
of the model is a combination of these two losses, as given by the following
formula:

$\mathcal{L}=\alpha\mathcal{L}_{\mathrm{bot}}+(1-\alpha)\mathcal{L}_{\mathrm{lang}}$
(1)

where hyperparameter $\alpha$ controls how much attention is paid to each
task.

####  3.2.3 Virtual Adversarial Training

VAT [27] is another regularization technique for deep learning models. It aims
to help models generalize better by perturbing the inputs to maximize the loss
function. For our models, the inputs refer to the embeddings of the raw
documents, not to the token IDs. In our case, this method implies performing
the forward and backward passes multiple times in order to compute the
gradients. Then, the loss function specific to VAT gets added to the regular
loss function, the final loss being the sum of the two. We added VAT to our
models using the VAT-pytorch Python library [28], which implements the
distributional smoothing technique described by Miyato et al. [6].

###  3.3 Ensemble Learning

Ensemble techniques combine multiple different models to make better
predictions. Intuitively, they should make the weaknesses of each model matter
less since if one model happens to have poor performance on a certain edge
case, all the other models will probably give better results, thus negating
the impact of the incorrect prediction. While there are multiple ways of
combining models into ensembles (such as majority voting or bagging) [29], we
decided to use the stacking technique inspired by the work of Gaman [20].
Thus, we train an extra meta-learner model, which learns to make predictions
based on the outputs of each model in the ensemble. We experimented with an
XGBoost classifier, which takes as input the probabilities produced by each
model, and the binary predictions they make. The submitted final ensembles can
be seen in Figure 1(b).

(a) The MTL architecture used for `.`

(b) Summary of the ensemble architectures used for runs 2 and 3.

(c) The architectures used to create the submissions. `f`eatured an MTL model,
while the other two runs used ensembles. The difference between `a`nd `i`s
that the latter misses the model pre-trained on Tweets. The Multilayer
Perceptron (MLP) architecture is described in Subsection 3.2.1.

##  4 Experiments

###  4.1 Dataset

The training dataset provided for Subtask 1 consists of approximately
$33{,}845$ English documents, and $32{,}062$ Spanish documents. For both
languages, the ratio of computer-generated to human-generated texts was
roughly $50\%$. This suggested that the dataset we worked with was fairly
well-balanced, and we did not attempt to use any techniques for dealing with
imbalanced data. While the individual BERT classifier (MLP) was trained for
the two languages separately, the multi-task learning experiments merged the
two slices into a single dataset. In both cases, we used $70\%$ of the labeled
data to train the models, and we set aside the other $30\%$ for validation.
This split was fixed at the beginning of the project to allow us to compare
the models’ performance. However, when creating the final predictions, we
retrained all models on the full labeled dataset, only using $2\%$ of it
(around $1{,}300$ samples) as a validation set to monitor the training
process. As seen in Figure 1, there is a relatively large number of documents
with fewer than $200$ characters. Since the task organizers described one of
the domains in the dataset as social media, and Twitter usually limits user
posts to approximately $300$ characters at most, we assumed these samples to
be tweets. For this reason, we experimented with a language model pre-trained
specifically on tweets, namely TwHIN-BERT.

Figure 1: The length distribution of the documents in the training set,
grouped by label and language.

We did not perform any data preprocessing for the deep learning models, using
the raw documents as input instead.

###  4.2 Hyperparameters

For the deep learning models, we used a hidden layer of size $64$, and a
dropout rate of $0.2$. For MTL, we assigned the same weight to the two loss
values, i.e., $\alpha=0.5$. We used the default values for VAT, namely
$\alpha=1$, $\epsilon=1$, and $\xi=10$. One of the parameters we did not set
to a fixed value was the threshold for turning the probabilities into binary
predictions. To choose this threshold, we compute the true positive rate (TPR)
and false positive rate (FPR) of the Receiver Operating Characteristic curve
for the validation set. Since the dataset was balanced, we did not want to
favor either precision or recall, so we picked the threshold which brings the
sum of TPR and FPR closest to $1$. In our experiments, the threshold was often
greater than $0.9$, sometimes over $0.95$. We used the AdamW optimizer [30]
implemented in PyTorch [31] with a learning rate of $10^{-5}$, and $2$-$4$
epochs for training. In order to avoid overfitting, we used both a dropout
layer and the early stopping technique. We used batch sizes between $24$ and
$48$, depending on the model size. For the kNN shallow model, we set the
parameter $k$ to $10$. For the final ensembles, we performed a grid search to
find the hyperparameters of the XGBoost model, which maximized the macro
F1-score on a small validation set for each of the two languages. We searched
for estimators between $2$ and $30$, depths between $3$ and $10$, and learning
rates between $10^{-5}$ and $10^{-1}$. The best hyperparameters were:
$\mathrm{n\\_estimators}=3$, $\mathrm{max\\_depth}=5$, and
$\mathrm{learning\\_rate}=10^{-3}$.

###  4.3 Results

The results obtained for Subtask 1 can be seen in Tables 1 and 2, alongside
the baselines provided by the task organizers, the best results obtained by
other participant teams, and our other experiments, which we did not submit.

Table 1: Macro-F1 (F1) scores obtained on the English dataset for Subtask 1. Our experiments are highlighted in bold. The validation set refers to the one we used, while the organizers provided the test set. Model | Rank | Validation Set F1 | Test Set F1
---|---|---|---
TALN-UPF Hybrid Plus | 1 | - | 80.91
TALN-UPF Hybrid | 2 | - | 74.16
Full ensemble (our `` & 18 & \- & 66.63 \\\ \textbfEnsemble without TwHIN-BERT
(our

    run3})}
          & 19   & -             & 66.40 \\
    Logistic Regression (baseline)
          & 23   & -             & 65.78 \\
    \textbf

MTL (`xlm-roberta-base}) (our \mintinline`textrun1) | 25 | 93.30 | 65.53
Symanto Brain (Few-shot) (baseline) | 37 | - | 59.44
DeBERTa V3 (baseline) | 51 | - | 57.10
Random (baseline) | 69 | - | 50.00
Symanto Brain (Zero-shot) (baseline) | 73 | - | 43.47
MTL (`` & \- & 92.70 & \- \\\ \textbfMLP (

    bert-base-multilingual-cased})}
          & -    & 91.80         & - \\
    \textbf

XGBoost + Readability + GloVe | - | 79.80 | 59.22
kNN + Readability + GloVe | - | 74.90 | 56.31
Table 2: Macro-F1 (F1) scores obtained on the Spanish dataset for Subtask 1. Our experiments are highlighted in bold. The validation set refers to the one we used, while the organizers provided the test set. Model | Rank | Validation Set F1 | Test Set F1
---|---|---|---
TALN-UPF Hybrid Plus | 1 | - | 70.77
Linguistica_F-P_et_al | 2 | - | 70.60
RoBERTa (BNE) (baseline) | 3 | - | 68.52
Ensemble without TwHIN-BERT (our `` & 6 & \- & 67.10 \\\ \textbfFull ensemble
(our

    run2})}
          & 7    & -             & 66.97 \\
    \textbf

MTL (`xlm-roberta-base}) (our \mintinline`textrun1) | 12 | 92.30 | 65.01
Logistic Regression (baseline) | 25 | - | 62.40
Symanto Brain (Few-shot) (baseline) | 39 | - | 56.05
Random (baseline) | 46 | - | 50.00
Symanto Brain (Zero-shot) (baseline) | 50 | - | 34.58
MTL (`` & \- & 91.00 & \- \\\ \textbfMLP (

    bert-base-multilingual-cased})}
          & -    & 90.90         & - \\
    \textbf

XGBoost + Readability + FastText | - | 80.90 | 63.70
kNN + Readability + FastText | - | 72.20 | 59.59

As expected, the ensembles performed better than the MTL model (i.e.,

    run1}) on both datasets. However, the scores obtained on the test set are much smaller than those obtained on our validation set. This fact could indicate that our chosen validation set was too small or poorly chosen, or that the distribution of the test set is different from that of the training set. It would have been interesting to see if methods such as cross-validation would have produced other validation scores closer to the real performance.
    Our experiments indicated that the choice of Transformer mattered as well, with multilingual models being generally better for this use case. Similarly, the fine-tuned embeddings produced during training were better than the pre-trained ones provided by spaCy, and hence, we did not explore the shallow learning direction more.
    %
    We did not perform a grid search or any other type of hyperparameter search for the deep learning models, so our approaches could obtain better results simply by choosing appropriate hyperparameter values. Another thing to note is that while the full ensemble performed better on the English dataset, removing the TwHIN-BERT Transformer slightly improved the results on the Spanish dataset.
    \section

Conclusions In this paper, we proposed multiple methods for addressing the
task of detecting LLM-generated text. While we experimented briefly with more
classical machine learning classifiers, we saw that Transformer-based models
performed better for this task. We described our experiments with several
classification models powered by BERT and detailed some regularization
techniques, which improved their performance slightly. Finally, we stacked
multiple such models to form ensembles, leading to even better performance and
achieving our best macro F1-scores of 66.63% on the English dataset, and
$67.10\%$ on the Spanish dataset of the AuTexTification shared task, Subtask
1. Regarding future work, we could improve the choice of hyperparameters since
they are often crucial for achieving good performance, and techniques such as
grid search should find better values. Similarly, the training process can be
improved by using better methods to avoid overfitting, and increasing the
number of training epochs.

## References

  * Radford et al. [2018] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al.,  Improving language understanding by generative pre-training,  OpenAI (2018).
  * Team [2022] O. Team, Chatgpt: Optimizing language models for dialogue, 2022.
  * Sarvazyan et al. [2023] A. M. Sarvazyan, J. Á. González, M. Franco Salvador, F. Rangel, B. Chulvi, P. Rosso,  Overview of autextification at iberlef 2023: Detection and attribution of machine-generated text in multiple domains,  in: Procesamiento del Lenguaje Natural, Jaén, Spain, 2023.
  * Jiménez-Zafra et al. [2023] S. M. Jiménez-Zafra, F. Rangel, M. Montes-y Gómez,  Overview of IberLEF 2023: Natural Language Processing Challenges for Spanish and other Iberian Languages,  Procesamiento del Lenguaje Natural 71 (2023).
  * Collobert and Weston [2008] R. Collobert, J. Weston,  A unified architecture for natural language processing: Deep neural networks with multitask learning,  in: Proceedings of the 25th international conference on Machine learning, 2008, pp. 160--167.
  * Miyato et al. [2015] T. Miyato, S.-i. Maeda, M. Koyama, K. Nakae, S. Ishii,  Distributional smoothing with virtual adversarial training,  arXiv preprint arXiv:1507.00677 (2015).
  * Paris et al. [2013] C. L. Paris, W. R. Swartout, W. C. Mann, Natural language generation in artificial intelligence and computational linguistics, volume 119, Springer Science & Business Media, 2013.
  * Tang et al. [2023] R. Tang, Y.-N. Chuang, X. Hu,  The science of detecting llm-generated texts,  arXiv preprint arXiv:2303.07205 (2023).
  * Hearst et al. [1998] M. A. Hearst, S. T. Dumais, E. Osuna, J. Platt, B. Scholkopf,  Support vector machines,  IEEE Intelligent Systems and their applications 13 (1998) 18--28.
  * Stodden and Venugopal [2021] R. Stodden, G. Venugopal,  Rs_gv at semeval-2021 task 1: Sense relative lexical complexity prediction,  in: Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), 2021, pp. 640--649.
  * Flesch [1948] R. Flesch,  A new readability yardstick.,  Journal of applied psychology 32 (1948) 221.
  * Gunning [1952] R. Gunning,  Technique of clear writing,  McGraw-Hill (1952).
  * Mc Laughlin [1969] G. H. Mc Laughlin,  Smog grading-a new readability formula,  Journal of reading 12 (1969) 639--646.
  * van Cranenburgh [2022] A. van Cranenburgh, Readability, https://github.com/andreasvc/readability, 2022\. Accessed 15 Jun 2023.
  * Honnibal et al. [2020] M. Honnibal, I. Montani, S. Van Landeghem, A. Boyd,  spaCy: Industrial-strength Natural Language Processing in Python (2020). doi:10.5281/zenodo.1212303.
  * Pennington et al. [2014] J. Pennington, R. Socher, C. D. Manning,  Glove: Global vectors for word representation,  in: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532--1543.
  * Mikolov et al. [2018] T. Mikolov, E. Grave, P. Bojanowski, C. Puhrsch, A. Joulin,  Advances in pre-training distributed word representations,  in: Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018), 2018.
  * Pedregosa et al. [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al.,  Scikit-learn: Machine learning in python,  Journal of Machine Learning Research 12 (2011) 2825--2830.
  * Chen and Guestrin [2016] T. Chen, C. Guestrin,  Xgboost: A scalable tree boosting system,  in: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 2016, pp. 785--794.
  * Gaman [2023] M. Gaman,  Using ensemble learning in language variety identification,  in: Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), 2023, pp. 230--240.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin,  Attention is all you need,  Advances in neural information processing systems 30 (2017).
  * Devlin et al. [2019] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova,  Bert: Pre-training of deep bidirectional transformers for language understanding,  in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 4171--4186.
  * Conneau et al. [2020] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, É. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov,  Unsupervised cross-lingual representation learning at scale,  in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 8440--8451.
  * Zhang et al. [2022] X. Zhang, Y. Malkov, O. Florez, S. Park, B. McWilliams, J. Han, A. El-Kishky,  Twhin-bert: A socially-enriched pre-trained language model for multilingual tweet representations,  arXiv preprint arXiv:2209.07562 (2022).
  * Weiss et al. [2016] K. Weiss, T. M. Khoshgoftaar, D. Wang,  A survey of transfer learning,  Journal of Big data 3 (2016) 1--40.
  * Srivastava et al. [2014] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov,  Dropout: a simple way to prevent neural networks from overfitting,  The journal of machine learning research 15 (2014) 1929--1958.
  * Miyato et al. [2018] T. Miyato, S.-i. Maeda, M. Koyama, S. Ishii,  Virtual adversarial training: a regularization method for supervised and semi-supervised learning,  IEEE transactions on pattern analysis and machine intelligence 41 (2018) 1979--1993.
  * Yokoo [2018] S. Yokoo, Vat-pytorch, https://github.com/lyakaap/VAT-pytorch, 2018\. Accessed 15 Jun 2023.
  * Galar et al. [2011] M. Galar, A. Fernandez, E. Barrenechea, H. Bustince, F. Herrera,  A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches,  IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 42 (2011) 463--484.
  * Kingma and Ba [2015] D. P. Kingma, J. Ba,  Adam: A method for stochastic optimization,  in: Y. Bengio, Y. LeCun (Eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * Paszke et al. [2017] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, A. Lerer,  Automatic differentiation in pytorch,  in: NIPS 2017 Workshop on Autodiff, 2017\. URL: https://openreview.net/forum?id=BJJsrmfCZ.