# Wickets in 3-uniform Hypergraphs

Jozsef Solymosi  Department of Mathematics, University of British Columbia,
Vancouver, Canada, and Obuda University, Budapest, Hungary
<EMAIL_ADDRESS>
###### Abstract.

In these notes, we consider a Turán-type problem in hypergraphs. What is the
maximum number of edges if we forbid a subgraph? Let $H_{n}^{(3)}$ be a
3-uniform linear hypergraph, i.e. any two edges have at most one vertex
common. A special hypergraph, called wicket, is formed by three rows and two
columns of a $3\times 3$ point matrix. We describe two linear hypergraphs –
both containing a wicket – that if we forbid either of them in $H_{n}^{(3)}$,
then the hypergraph is sparse, and the number of its edges is $o(n^{2})$. This
proves a conjecture of Gyárfás and Sárközy.

##  1\. Introduction

In this note, we answer a question of Gyárfás and Sárközy. To formulate it,
let’s review some definitions, notations and a brief history of the problem.

######  Definition.

A hypergraph is linear if two edges intersect at most one vertex.

We will work with 3-uniform linear hypergraphs, i.e. hypergraphs where every
edge has three vertices. We use the notations $H_{n}^{3}$ and $F_{n}^{3}$ for
3-uniform hypergraphs on $n$ vertices and $G_{n}$ for graphs on $n$ vertices.
The number of edges in $H$ is denoted by $e(H)$.

It is a notoriously difficult problem to characterize unavoidable subgraphs in
dense 3-uniform linear hypergraphs. One of the central conjectures in extremal
combinatorics is the following:

######  Conjecture 1.1 (Brown, Erdős, and Sós).

For every $\ell\geq 3$ and $c>0$ there exists an $n_{0}=n_{0}(c)$ such that if
$n>n_{0}$ and $e(H_{n}^{3})\geq cn^{2}$ then there exists a
$F_{\ell+3}^{3}\subseteq H_{n}^{3}$ such that $e(F_{\ell+3}^{3})\geq\ell$.

Although the conjecture does not specify that the hypergraph is linear, it is
enough to consider it for linear hypergraphs.

The $\ell=3$ case of Conjecture 1.1 was proved by Ruzsa and Szemerédi [8], but
for larger values, the conjecture is still open. We refer to works in [9, 10,
1] for partial results and related bounds.

In a closely related line of research, the following question was raised. For
a given 3-uniform linear hypergraph, $F$, what is the maximum number of edges
in a 3-uniform linear hypergraph without $F$ as a subgraph?

######  Definition.

The Turán number of a linear 3-uniform hypergraph $F$, denoted by
$ex_{L}(n,F)$ is the maximum number of edges of a 3-uniform linear hypergraph
not containing a subgraph isomorphic to $F$.

In [6] Gyárfás and Sárközy investigated $ex_{L}(n,F)$ for any $F$ with at most
five edges. They had good estimates with the exception of one configuration,
which they called wicket. The wicket, denoted by $W$, is formed by three rows
and two columns of a $3\times 3$ point matrix (fig. 1).

Figure 1. The wicket. If we add the edge spanned by vertices $E,I,H$, then it
is a grid, $GR$.

In their excellent paper, titled The linear Turán number of small triple
systems or why is the wicket interesting? [6] they review the relevant
questions and explain the importance of deciding whether
$ex_{L}(n,W)=o(n^{2})$.

If we add the third column to the wicket to have six edges on the nine
vertices, then this configuration can be avoided in dense linear hypergraphs.
Let’s call this hypergraph a grid, denoted by $GR$ (see in fig. 1). A
construction of Gishboliner and Shapira shows that
$ex_{L}(n,GR)=\Omega(n^{2})$ [4].

##  2\. The Main Result

In this section, we give two proofs of the conjecture of Gyárfás and Sárközy
(Theorem 2.1). In the first proof, we are using the graph regularity lemma
only. The second uses hypergraph regularity. We don’t know if such powerful
tools are needed to prove the conjecture. Both proofs guarantee the existence
of a larger graph – containing a wicket as a subgraph – for which a $o(n^{2})$
type lower bound is known. No similar lower bound is known for wickets. A
related conjecture of Gowers and Long in [5] states that there is a $c>0$ such
that if no nine vertices span at least five edges in an $n$-vertex 3-uniform
linear hypergraph (like in a wicket), then the number of edges is
$O(n^{2-c})$.

######  Theorem 2.1.

If a linear hypergraph contains no wickets, it is sparse,
$ex_{L}(n,W)=o(n^{2})$.

The following quantitative version of the Ruzsa-Szemerédi theorem [8] is used
in both proofs:

######  Theorem 2.2.

For every $c>0$ there exists a $\delta>0$ such that if the number of edges in
a 3-uniform hypergraph $H_{n}^{3}$ is at least $cn^{2}$ then there exists
$\delta n^{3}$ subgraphs $F_{6}^{3}\subseteq H_{n}^{3}$ such that
$e(F_{6}^{3})\geq 3$.

Since we are considering linear hypergraphs only, we can select a specific
subgraph from $F_{6}^{3}$ if $e(F_{6}^{3})\geq 3$. It is a 6-vertex hypergraph
with three edges such that every edge has one degree-one vertex and two
degree-two vertices. We call it a $(6,3)$-configuration.

The second is the “induced matching” variant. A matching $M$ in a graph is an
induced matching if the vertices of $M$ do not span edges other than the edges
of $M$.

######  Theorem 2.3.

For every $c>0$ there is an $N(c)$ such that if a bipartite graph on $2n$
vertices is the union of $n$ induced matchings, and it has at least $cn^{2}$
edges, then $n\leq N(c)$.

###  2.1. First Proof of Theorem 2.1

###### Proof.

In our first proof, we will use the above two results only. Let us suppose
that for a $c>0$, we have infinitely many linear 3-uniform hypergraphs with at
least $cn^{2}$ edges without a wicket. We assume that $n$ is divisible by six
to avoid carrying fractions. In the proof, we will use constants, $c_{i}>0$,
which only depend on the initial $c$.

We label the main steps of the proof for easier reference.

  1. (1)

We can assume that the graph is three-partite with equal vertex classes
$V_{1},V_{2},V_{3}$, where $|V_{i}|=n/3$. Theorem 2.2 shows that there are
$c_{1}n^{3}$ $(6,3)$-configurations, each with two vertices in the classes,
one of degree one and one of degree two. In a three-partite graph, the wicket
looks like the graph in Fig. 2.

Figure 2. The wicket in a 3-partite graph

  2. (2)

Take random partitions of the $V_{i}$-s into two equal parts. The partition
classes are denoted as $V^{\prime}_{i},V^{\prime\prime}_{i}\subset V_{i}$
where $|V_{i}^{\prime}|=|V_{i}^{\prime\prime}|=n/6$. From this point, we only
consider $(6,3)$-configurations which have the degree-one vertices in
$V^{\prime}_{i}$-s and the degree-two vertices in the $V_{i}^{\prime\prime}$-s
(fig. 3). There are $c_{2}n^{3}$ such $(6,3)$-configurations.

Figure 3. Partitioning the vertices of the $(6,3)$-configurations

  3. (3)

Let’s select a perfect matching between the vertices of $V_{1}^{\prime}$ and
$V_{3}^{\prime}$ out of the $(n/6)!$ possible perfect matchings at random
(every matching has probability $1/(n/6)!$ to be selected). The matching is
denoted by $M$, and the pairs are indexed as
$M=\\{m_{1},m_{2},\ldots,m_{s}\\}$ where $s=n/6$. For any matching between the
vertices of $V_{1}^{\prime}$ and $V_{3}^{\prime}$ of size $k$ ($3\leq k\leq
s$) the probability that at least one edge is in $M$ can be calculated by
simple counting. The inclusion-exclusion formula gives the number of perfect
matchings containing at least one edge

$k(s-1)!-\binom{k}{2}(s-2)!+\binom{k}{3}(s-3)!-\ldots\geq\frac{k(s-1)!}{2}.$

The probability that at least one edge is in $M$ is at least $k/(2s)$.

  4. (4)

Define an auxiliary bipartite graph $G_{M}$ between $V_{1}^{\prime\prime}$ and
$V_{3}^{\prime\prime}$ as follows: Two vertices, $v_{1}\in
V_{1}^{\prime\prime}$ and $v_{3}\in V_{3}^{\prime\prime}$ are connected by an
edge if there is a pair $m_{i}\in M$ such that the set of the four vertices,
$\\{\\{v_{1},v_{2}\\}\cup m_{i}\\}$ are from a $(6,3)$-configuration.

  5. (5)

For any pair $v_{1}\in V_{1}^{\prime\prime},v_{3}\in V_{3}^{\prime\prime}$ the
probability that $(v_{1},v_{3})\in E(G_{M})$ depends on the number of
$(6,3)$-configuration containing these two vertices. This number, denoted by
$M(v_{1},v_{3})$ is between $0$ and $s$. Note that the other pairs of vertices
forming a $(6,3)$-configuration with $v_{1},v_{3}$ give a matching between the
vertices of $V_{1}^{\prime}$ and $V_{3}^{\prime}$. The probability that
$(v_{1},v_{3})\in E(G_{M})$ is at least $M(v_{1},v_{3})/2s$.

By the linearity of expectations, the expected number of edges in $G_{M}$ is
at least

$\sum_{v_{1}\in V_{1}^{\prime\prime},v_{3}\in
V_{3}^{\prime\prime}}\frac{M(v_{1},v_{3})}{2s}={c_{3}n^{2}}.$

Let us select a matching, $M^{*}$ where $G_{M^{*}}$ has at least
${c_{3}n^{2}}$ edges. The pairs are indexed as
$M^{*}=\\{m_{1},m_{2},\ldots,m_{s^{\prime}}\\}$.

  6. (6)

In the last step, we partition $G_{M^{*}}$ into matchings. For every pair
$m_{i}\in M^{*}$, we select a set of edges from $G_{M^{*}}$ following the
order of the indices of $m_{i}$. First, we select all $(v,w)\in E(G_{M^{*}})$
so that $\\{\\{v,w\\}\cup m_{1}\\}$ are from a $(6,3)$-configuration. The set
of selected edges forms a matching, denoted by $GM_{1}$. We remove these edges
and continue the selection of matchings. In the $i$-th step, the matching
$GM_{i}$ is given by selecting all pairs from the remaining edges

$(v,w)\in\\{G_{M^{*}}\setminus\\{\cup_{1\leq j<i}GM_{j}\\}\\}$

where the points in $\\{\\{v,w\\}\cup m_{i}\\}$ are from a
$(6,3)$-configuration.

Now we are ready to complete the proof. The set of edges of $G_{M^{*}}$ is the
union of disjoint matchings, $GM_{i}$ ($1\leq i\leq s^{\prime}$). If $n$ is
large enough, then by Theorem 2.3, there is a matching, $GM_{i}$, which is not
induced; its edges span another edge, $e$. Now, $m_{i}$ and the two edges of
$M_{i}$ spanning $e$ together with $e$ span a wicket (fig. 4)

Figure 4. The vertices of $m_{i}=\\{A,D\\}$, $\\{(C,B),(E,F)\in GM_{i}\\}$ and
$e=(C,F)$ span a wicket with the additional vertices $J,K,G$.

The wicket we’ve found is the subgraph of a larger hypergraph on 11 or 10
vertices (depending on whether $I$ and $H$ are disjoint vertices) with seven
edges. Our second proof results in a larger graph on 13 or 12 vertices and
nine edges. ∎

###  2.2. Second Proof of Theorem 2.1

In this proof, we use the hypergraph regularity lemma by Frankl and Rödl [3].

######  Theorem 2.4.

For any $c>0$, there is a $\delta>0$ such that the following holds for large
enough $n$: If $H^{3}_{n}$ contains at least $cn^{3}$ pairwise edge-disjoint
complete subgraphs $K^{3}_{4}$, then it has at least $\delta n^{4}$ complete
subgraphs $K^{3}_{4}$.

###### Proof.

We follow the first two steps of the previous proof. Define a 4-partite
3-uniform hypergraph, $H^{3}_{m}$ on the vertex sets
$V_{1}^{\prime},V_{1}^{\prime\prime},V_{3}^{\prime},V_{3}^{\prime\prime}$. The
edges are defined by the $(6,3)$-configurations; each triple of the four
vertices span and edge. Every $(6,3)$-configuration spans a $K^{3}_{4}$ clique
in $H^{3}_{m}$. Since every edge uniquely determines the
$(6,3)$-configuration, $H^{3}_{m}$ contains at least $c_{2}n^{3}$ pairwise
edge-disjoint $K_{4}^{3}$ cliques on $m=4n/6$ vertices. By Theorem 2.4, we
know that the number of $K_{4}^{3}$ cliques is at least $c_{4}n^{4}$ in the
graph. If two edges in a $K_{4}^{3}$ are from the same $(6,3)$-configuration,
the other two are also from there since any pair of edges determines the four
vertices. If $c_{4}n^{4}>c_{2}n^{3}$ then there is a $K_{4}^{3}$ where every
edge is from a different $(6,3)$-configuration. Let’s denote the vertices of
such a $K_{4}^{4}$ by
$w_{1}^{\prime},w_{1}^{\prime\prime},w_{3}^{\prime},w_{3}^{\prime\prime}$,
where $w_{1}^{\prime}\in V_{1}^{\prime},w_{1}^{\prime\prime}\in
V_{1}^{\prime\prime},w_{3}^{\prime}\in V_{3}^{\prime},w_{3}^{\prime\prime}\in
V_{3}^{\prime\prime}$.

Figure 5. The edges spanned by the vertices
$W1^{\prime},W1^{\prime\prime},$$W3^{\prime},W3^{\prime\prime},$$V1^{\prime\prime},V3^{\prime\prime}$
contain a wicket.

For every triple, the fourth vertex of the $(6,3)$-configuration is denoted as
$v_{1}^{\prime}\in V_{1}^{\prime},v_{1}^{\prime\prime}\in
V_{1}^{\prime\prime},v_{3}^{\prime}\in V_{3}^{\prime},v_{3}^{\prime\prime}\in
V_{3}^{\prime\prime}$. For example, the four vertices
$w_{1}^{\prime},w_{1}^{\prime\prime},v_{3}^{\prime},w_{3}^{\prime\prime}$ span
a $(6,3)$-configuration. The eight vertices, the $w$-s and $v$-s, are all
disjoint. The six vertices
$w_{1}^{\prime},w_{1}^{\prime\prime},w_{3}^{\prime},w_{3}^{\prime\prime},v_{1}^{\prime\prime},v_{3}^{\prime\prime}$
span the edges of a wicket with the two extra edges spanned by
$v_{1}^{\prime\prime},w_{3}^{\prime\prime}$ and by
$w_{1}^{\prime\prime},v_{3}^{\prime\prime}$ (fig 5). ∎

##  3\. Further Remarks

In the two proofs, we did find two hypergraphs that are unavoidable in dense
3-uniform hypergraphs. The two graphs are shown in Fig. 6 and 7.

Figure 6. The first unavoidable hypergraph in dense hypergraphs has 14
vertices and 9 edges, or 13 vertices and 9 edges if $I$ and $K$ are the same
vertices Figure 7. The second unavoidable hypergraph in dense linear
hypergraphs has 13 vertices and 9 edges, or 13 vertices and 9 edges if $N$ and
$P$ are the same vertices

The main question remains to decide the actual Turán number of wickets in
3-uniform linear hypergraphs. Here we proved that if $H_{n}^{(3)}$ contains no
wickets, then it is sparse. On the other hand, the best-known construction
avoiding wickets is an arrangement avoiding $C_{4}$-s, resulting in the
$\Omega(n^{3/2})$ lower bound, as noted in [6].

##  4\. Acknowledgements

The research was partly supported by an NSERC Discovery grant and OTKA K grant
no.119528. The author is thankful to András Gyárfás and Gábor Sárközy for
introducing the problem and the valuable remarks.

## References

  * [1] David Conlon, Lior Gishboliner, Yevgeny Levanzov, Asaf Shapira, A new bound for the Brown–Erdős–Sós problem, Journal of Combinatorial Theory, Series B, Volume 158, Part 2, 2023, Pages 1–35,
  * [2] Paul Erdős William G. Brown and Vera Sós. On the existence of triangulated spheres in 3-graphs, and related problems. Period. Math. Hung., 3:221–228.
  * [3] Peter Frankl and Vojtech Rödl.  Extremal problems on set systems.  Random Structures and Algorithms, 20(2):131–164, 2002.
  * [4] Lior Gishboliner and Asaf Shapira, Constructing dense grid-free linear 3-graphs Proc. Am. Math. Soc., 150 (2022), pp. 69–74
  * [5] William Gowers and Jason Long, The length of an $s$-increasing sequence of $r$-tuples. Combinatorics, Probability and Computing, (2021) 30(5), 686–721.
  * [6] András Gyárfás and Gábor N. Sárközy, The linear Turán number of small triple systems or why is the wicket interesting?, Discrete Mathematics, Volume 345, Issue 11, 2022,
  * [7] András Gyárfás and Gábor N. Sárközy, Turán and Ramsey numbers in linear triple systems, Discrete Mathematics, Volume 344, Issue 3, 2021,
  * [8] Imre Z. Ruzsa and Endre Szemerédi.  Triple systems with no six points carrying three triangles.  In Colloq. Math. Soc. J. Bolyai Combinatorics II: 939–945, 1978
  * [9] Gábor Sárközy and Stanley Selkow.  An extension of the Ruzsa-Szemerédi theorem.  Combinatorica, 25(1):77–84, 2004.
  * [10] David Solymosi and Jozsef Solymosi, Small cores in 3-uniform hypergraphs, Journal of Combinatorial Theory, Series B, Volume 122, 2017, Pages 897–910,


# Audio-based cough counting using independent subspace analysis ††thanks:
Identify applicable funding agency here. If none, delete this.

Paul Leamy  Biomedical Research Group
Technological University Dublin1
Dublin, Ireland
<EMAIL_ADDRESS>Ted Burke1 Biomedical Research Group
Dublin, Ireland
<EMAIL_ADDRESS>Dan Barry  QxLab
University College Dublin
Dublin, Ireland
<EMAIL_ADDRESS>David Dorran1 Biomedical Research Group
Dublin, Ireland
<EMAIL_ADDRESS>
###### Abstract

In this paper, an algorithm designed to detect characteristic cough events in
audio recordings is presented, significantly reducing the time required for
manual counting. Using time-frequency representations and independent subspace
analysis (ISA), sound events that exhibit characteristics of coughs are
automatically detected, producing a summary of the events detected. Using a
dataset created from publicly available audio recordings, this algorithm has
been tested on a variety of synthesized audio scenarios representative of
those likely to be encountered by subjects undergoing an ambulatory cough
recording, achieving a true positive rate of 76% with an average of 2.85 false
positives per minute.

###### Index Terms:

cough counting, independent subspace analysis, epidemiology, health monitoring

##  I Introduction

The information contained within the sounds produced by the human vocal tract,
such as speech, moaning, sighing, and coughing, present an opportunity to
facilitate remote and non-contact monitoring of an individual’s health [1]. In
relation to physical health, coughing is a common symptom for which patients
seek medical advice [2], especially in the acute and chronic categories [3],
where a persistent cough can severely impair an individual’s quality of life.
In determining the frequency and severity of a person’s cough, a clinician can
make a suitable diagnosis relating to a person’s cough, and this is the
objective of a cough detection/monitoring system [4]. Having an objective
measure of a cough is useful when tracking the progression of an illness [5]
especially in the process of early detection and monitoring [6], highlighting
the importance of cough monitoring systems to aid with tracking the progress
of a disease.

Presented here is an algorithm designed to detect the presence of candidate
cough sounds in audio recordings. The area of cough detection has received
notable attention in recent years. Semi-automatic approaches using hand-
crafted features as inputs to probabilistic neural networks [7] and
statistical models [8, 9] achieved satisfactory results but rely on input from
operators to successfully detect cough sounds. More recent algorithms have
used eigenvalue decomposition with random forest classifiers [10], Hu moments
with k-nearest neighbours [11], spectral features with time-delay neural
networks [12], convolutional and recurrent neural networks [13], and spectral
features with support vector machines [14].

The nature of cough sounds, as described in [15], share characteristics with
the signals produced by percussive and harmonic musical instruments. Cough
sounds typically begin with a quick onset of wideband noise, stretching across
a spectrum up to $20\text{\,}\mathrm{kHz}$, followed by a short stationary
period dependent upon the cause of the cough. An example of the phases
associated with a cough can be seen in Figure 1.

Figure 1: Phases of a cough sound showing explosive phase, intermediate phase,
and decaying voiced phase.

The onset of coughs share similar properties to the percussive characteristics
of drums, and in [16] the transient nature of drum sounds were exploited using
independent subspace analysis (ISA) [17] to automatically transcribe drum
tracks from audio recordings. Due to the nature of ambulatory cough
recordings, cough sounds produced by a person are likely to be more prominent
within these recordings. The repeated occurrences of a person’s cough and
proximity to a microphone means that the cough sound is likely to be one of
the sources contributing most variance to the recorded signal. Given the
variance based nature of ISA, it is expected to produce candidate time-
activation functions that coincide with cough events. An example of this
algorithm’s output is illustrated in Figure 2, which highlights the time-
activation functions and corresponding frequency spectra for a short drum loop
containing three sounds (kick drum, snare, hi-hat). The approach presented
here builds on and refines the ISA approach for the purpose of cough event
detection.

Figure 2: Illustration of ISA being applied to a short drum loop, highlighting
the significant time-activation functions and corresponding frequency spectra
of each drum.

##  II Methodology

Many existing approaches to cough detection are trained in a manner that aims
to achieve generalisation such that algorithms can correctly identify cough
sounds in a wide variety of recording scenarios. While generalisation is
desirable in detection algorithms, it may not be required to achieve
satisfactory results when detecting the presence of repeating events in more
constrained scenarios. In ambulatory cough recordings, the coughs present are
likely to come from a single person, which presents an opportunity, namely the
characteristics of the coughs to be detected in an audio recording are likely
to share similar time-frequency domain features.

The algorithm proposed in this paper is designed to summarise an ambulatory
recording, producing a number of short audio clips, each of which is likely to
contain a cough event. Using SVD to analyse the spectrogram of an audio
signal, a set of time-activation functions is produced which then undergoes
independent component analysis (ICA). Together, SVD followed by ICA is known
as independent subspace analysis (ISA). Significant peaks in these time-
activation functions are then used as markers for candidate cough events since
these peaks correspond to the presence of high variance events in the audio
recording. Using a suitable threshold, candidate events can be marked and
extracted from the input audio signal and presented to the clinician for
further analysis and verification. The algorithm overview is illustrated in
Figure 3, and a MATLAB implementation of the algorithm is available at [18].

Figure 3: Block diagram illustrating the process of the proposed algorithm.
Further detail of each block in this process is presented in Section II of
this paper.

###  II-A Short-time Fourier transform

The first task of this algorithm is to compute the complex short-time Fourier
transform (STFT) $X$ of the input data $x$. This is computed using (1).

$X(k,m)=\sum_{n=0}^{N-1}w(n)x(n+mH)e^{-j2\pi nk/N}$ (1)

where $k$ is the discrete-frequency index, $m$ is the hop number for the
analysis window, $N$ is the frame size (2048 samples), and $H$ is the hop size
(512 samples). A Hanning window is used for the window function $w$, and the
sampling frequency is $44.1\text{\,}\mathrm{kHz}$. $X(k,m)$ is evaluated for
$k=0,...,N/2$ and the magnitude of these components are retained.

Figure 4: Illustration of the ISA algorithm on a sample magnitude spectrum.
Input data (left) undergoes SVD to produce the frequency basis spectra $U$ and
time-activation functions $V$. Applying ICA to the frequency-basis spectra and
time-activation functions results in improved separation of the events, as can
be seen in the four plots to right.

###  II-B Independent subspace analysis

SVD first decomposes the input matrix $X$ into three matrices $U$, $V$, and
$S$, to identify subspaces in order of the variance they contribute towards
the original data [19]. In the context of the magnitude spectrum $X$, where
columns contain the magnitudes of frequency content for a single analysis
window, $U$ are the frequency basis spectra and $V$ are the corresponding
time-activation functions. The outer product of each spectra and time-
activation pair produces a subspace of the original input data, and the input
data can be reconstructed using

$X=USV^{T}$ (2)

For this algorithm, the columns of $V$ will act as time-activation functions
for cough events in the input data $X$, and only the first nine singular
values are computed (determined experimentally to be optimal).

Figure 4 illustrates the input and output of SVD on an example magnitude
spectrum. These pairs of frequency-basis spectra and time-activation functions
($U_{1},U_{2},V_{1},V_{1}$) both contribute towards reconstructing elements of
both “sources” in the magnitude spectrogram. Ideally a single time-activation
function would coincide with each single type of event has occurred, but in
practice this is not the case. This can be overcome by applying ICA to the
time-activation function to transform the decorrelated time-activations into
statistically independent activation functions [20, 21]. This is the principle
underlying ISA. In Figure 4, the plots on the right show the output of the ICA
stage. The frequency-basis spectrum and time-activation function pair’s
contributions towards reconstructing both sources in the input magnitude
spectrum has decreased significantly.

###  II-C Time-activation selection and thresholding

Recall that in Section II-B the first nine singular values were computed. In
determining which time-activation functions to retain, the kurtosis function
$k(v)$ is used,

$k(v)=\frac{\sum_{m=0}^{M-1}(V_{m,v}-\bar{V_{v}})^{4}/M}{\sigma_{V_{v}}^{4}}$
(3)

where $V_{m,v}$ is the $m^{th}$ sample in $v^{th}$ column of $V$,
$\bar{V_{v}}$ and $\sigma_{V_{v}}$ are mean and standard deviation of $v^{th}$
column of $V$, respectively. When normalised, kurtosis is 0 for a Gaussian
distribution, positive for a “peakier” or leptokurtic distribution, and
negative for a “flatter” or mesokurtic distribution. Sparse events are likely
to occur sporadically in the time-activation functions, meaning a majority of
near-zero values, resulting in a leptokurtic distribution. The time-activation
functions with the three highest kurtosis measurements are retained and
rectified to produce three candidate time-activation functions $c_{1}$,
$c_{2}$, and $c_{3}$, for further analysis.

Peaks in each time-activation function correspond to moments in the input
signal where the contribution of the corresponding frequency-basis spectrum is
greater. A threshold $\tau=a.\sigma_{c}$ is applied to the candidate time-
activation functions to identify peaks, where $\sigma_{c}$ is the standard
deviation of $c$, and $a$ is a constant, with $4<a<8$.

Retained peaks from the candidate time-activation function are used to
generate a summary of the input signal, where $1\text{\,}\mathrm{s}$ windows
around each peak are extracted from the input signal and concatenated to
produce the short summary of the input signal.

###  II-D Dataset and evaluation

The dataset used in evaluating the proposed algorithm was constructed from a
number of publicly available cough, non-cough, and background sources
collected from YouTube videos [22] and the DCASE 2016 Challenge [23]. Each
test signal comprises 20 coughs produced by one individual and alternative
foreground sounds (door knocks, table banging, speech, laughing, etc.). In
total, 10 test signals were created with a duration of 10 minutes each.
Instructions for reproducing these test signals, including annotations and URL
links, can be accessed at [18].

The evaluation framework used here is adapted from [24], a framework for
polyphonic sound-event detection which overcomes the limitations of collar-
based event decisions and labelling subjectivity by annotators. True positive
(TP) or false positive (FP) decisions rely on the degree of overlap with
annotated events.

The overlap $t_{o}$ between an annotated event and detected event (see Figure
4) is measured and expressed as a ratio of a $500\text{\,}\mathrm{ms}$ window,
based on the average duration of coughs presented in [25]. When this ratio
exceeds the detection tolerance criteria $\rho_{DTC}$, the detected event is
labelled a TP, otherwise FP is declared,

$D=\begin{cases}TP,&\text{if}\ \frac{t_{o}}{500ms}>\rho_{DTC}\\\
FP,&\text{otherwise}\end{cases}\hskip 2.84526pt,0<\rho_{DTC}\leq 1$ (4)

where $D$ is a single detection and $\rho_{DTC}=0.3$. Undetected annotated
events are classed as false negatives (FN).

Figure 5: Overlap between the time of a detected event $t_{p}$ and time of an
annotated event $t_{a}$ is used to determine if a detected event is classified
as a TP or FP. The overlap time $t_{o}$ is expressed as a ratio of a
$500\text{\,}\mathrm{ms}$ window. When this ratio exceeds the detection
tolerance criteria $\rho_{DTC}$ the detected event is marked as a TP.

The performance of this algorithm is quantified using the true positive ratio
($r_{TP}$) and the false positive rate ($R_{FP}$) [24],

$r_{TP}=\frac{N_{TP}}{P}\hskip 11.38109ptR_{FP}=\frac{N_{FP}}{T_{dur}}$ (5)

where $N_{TP}$ and $N_{FP}$ are the number of true and false positives, $P$ is
the number of annotated cough events, and $T_{dur}$ is the duration of the
signal being analysed.

##  III Results and discussion

The mean values for the true positive ratio, false positive rate, and
summarised duration across all test signals are presented in Table I. These
results were produced for each time-activation function. Activation function
$c_{1}$ achieved the best performance in each metric across all mean values,
suggesting this is the appropriate activation function to use. The increase in
false positive rate with $c_{2}$ and $c_{3}$ suggests that these activation
functions encapsulate features common to both cough and non-cough events after
SVD.

TABLE I: Summary of results from all candidate time-activation functions ($c_{1}$, $c_{2}$, and $c_{3}$) showing the mean results across all test signals.  c | $r_{TP}(\%)$ | $R_{FP}(min^{-1})$ | $T(min)$
---|---|---|---
1 | 76.00 | 2.85 | 0.73
2 | 67.00 | 4.17 | 0.92
3 | 39.00 | 6.26 | 1.17

Table II highlights a best-case scenario which includes the metrics for each
test signal and the associated activation function that achieved this result.
For the majority of results, $c_{1}$ produced the best case which aligns well
with the results from Table I. For signal #3 an #5, the activation functions
that achieved the most desirable results are $c_{3}$ and $c_{2}$,
respectively. Assuming that the appropriate time-activation functions are
used, an improvement in the mean true positive ratio and false positive rate
is observed.

TABLE II: A best-case scenario representation of the results, where $c$ is the $c^{th}$ activation function, $r_{TP}$ is the true positive ratio, $R_{FP}$ the false positive rate, and $T$ is the summarised signal duration.  # | c | $r_{TP}(\%)$ | $R_{FP}(min^{-1})$ | $T(min)$
---|---|---|---|---
1 | 1 | 70 | 4.5 | 0.98
2 | 1 | 90 | 3.4 | 0.87
3 | 3 | 95 | 1.4 | 0.55
4 | 1 | 100 | 4.00 | 1
5 | 2 | 100 | 0.6 | 0.43
6 | 1 | 90 | 2.2 | 0.67
7 | 1 | 95 | 0.5 | 0.4
8 | 1 | 90 | 6.7 | 1.42
9 | 1 | 95 | 3.6 | 0.92
10 | 1 | 100 | 1.2 | 0.53
Mean |  | 92.50 | 2.81 | 0.77

From the results presented here, ISA has shown to be a reliable method for
detecting the presence of cough sounds in audio recordings. Referring to Table
II, it can be seen that the majority of cough sounds are accurately
represented within a single time-activation function, $c_{1}$ in this case,
which supports the use of kurtosis as a suitable statistical measure for
determining which activation function to use to achieve the most desirable
results. On average, kurtosis produces the best activation function to use as
shown in Table I. The mean results can be improved by selecting the time-
activation function containing the coughs manually. An automatic method of
determining which activation functions could be used to complement this
algorithm and improve overall results, as shown by the mean calculations in
Table II.

##  IV Conclusions

Building on an approach used for automatically transcribing drum sounds, the
proposed algorithm uses ISA for detecting the presence of an individual’s
cough within audio recordings, reducing the human effort required by medical
staff to identify cough events within long recordings.

An average true positive ratio of $76\text{\,}\mathrm{\char 37\relax}$, and a
false positive rate of 2.85 false positives per minute was achieved, with a
significant reduction in the duration of signals. This algorithm’s ability to
detect cough sounds is evident, and a reduction in false positives will
improve the overall performance. A solution to consider in future work
utilises multiple analysis windows during the SVD stage, taking advantage of
the varying spectral characteristics observed during different phases of a
single cough event. Also planned is a more detailed comparison against other
state-of-the-art cough-detection approaches.

## References

  * [1] M. El Ayadi, M. S. Kamel, and F. Karray, “Survey on speech emotion recognition: Features, classification schemes, and databases,” _Pattern Recognition_ , vol. 44, no. 3, pp. 572–587, 2011.
  * [2] S. M. Schappert, “National ambulatory medical care survey: 1991 summary.” _Vital and health statistics. Series 13, Data from the National Health Survey_ , no. 116, pp. 1–110, 1994.
  * [3] R. S. Irwin, L.-P. Boulet, M. M. Cloutier, R. Fuller, P. M. Gold, V. Hoffstein, A. J. Ing, F. D. McCool, P. O’Byrne, R. H. Poe _et al._ , “Managing cough as a defense mechanism and as a symptom: a consensus panel report of the american college of chest physicians,” _Chest_ , vol. 114, no. 2, pp. 133S–181S, 1998.
  * [4] J. Smith, “Ambulatory methods for recording cough,” _Pulmonary pharmacology & therapeutics_, vol. 20, no. 4, pp. 313–318, 2007.
  * [5] C. T. French, R. S. Irwin, K. E. Fletcher, and T. M. Adams, “Evaluation of a cough-specific quality-of-life questionnaire,” _Chest_ , vol. 121, no. 4, pp. 1123–1131, 2002.
  * [6] C. Sohrabi, Z. Alsafi, N. O’Neill, M. Khan, A. Kerwan, A. Al-Jabir, C. Iosifidis, and R. Agha, “World health organization declares global emergency: A review of the 2019 novel coronavirus (covid-19),” _International Journal of Surgery_ , 2020.
  * [7] S. J. Barry, A. D. Dane, A. H. Morice, and A. D. Walmsley, “The automatic recognition and counting of cough,” _Cough_ , vol. 2, no. 1, p. 8, 2006.
  * [8] S. Matos, S. S. Birring, I. D. Pavord, and H. Evans, “Detection of cough signals in continuous audio recordings using hidden markov models,” _IEEE Transactions on Biomedical Engineering_ , vol. 53, no. 6, pp. 1078–1083, 2006.
  * [9] A. Barton, P. Gaydecki, K. Holt, and J. A. Smith, “Data reduction for cough studies using distribution of audio frequency content,” _Cough_ , vol. 8, no. 1, p. 12, 2012.
  * [10] E. C. Larson, T. Lee, S. Liu, M. Rosenfeld, and S. N. Patel, “Accurate and privacy preserving cough sensing using a low-cost microphone,” in _Proceedings of the 13th international conference on Ubiquitous computing_ , 2011, pp. 375–384.
  * [11] J. Monge-Álvarez, C. Hoyos-Barceló, K. Dahal, and P. Casaseca-de-la Higuera, “Audio-cough event detection based on moment theory,” _Applied Acoustics_ , vol. 135, pp. 124–135, 2018.
  * [12] Y. A. Amrulloh, U. R. Abeyratne, V. Swarnkar, R. Triasih, and A. Setyati, “Automatic cough segmentation from non-contact sound recordings in pediatric wards,” _Biomedical Signal Processing and Control_ , vol. 21, pp. 126–136, 2015.
  * [13] J. Amoh and K. Odame, “Deep neural networks for identifying cough sounds,” _IEEE transactions on biomedical circuits and systems_ , vol. 10, no. 5, pp. 1003–1011, 2016.
  * [14] M. You, Z. Liu, C. Chen, J. Liu, X.-H. Xu, and Z.-M. Qiu, “Cough detection by ensembling multiple frequency subband features,” _Biomedical Signal Processing and Control_ , vol. 33, pp. 132–140, 2017.
  * [15] W. Thorpe, M. Kurver, G. King, and C. Salome, “Acoustic analysis of cough,” in _The Seventh Australian and New Zealand Intelligent Information Systems Conference, 2001_. IEEE, 2001, pp. 391–394.
  * [16] D. FitzGerald, “Sub-band independent subspace analysis for drum transcription,” in _International Conference on Digital Audio Effects, Germany, 2002_ , 2002.
  * [17] M. A. Casey and A. Westner, “Separation of mixed audio sources by independent subspace analysis.” in _ICMC_ , 2000, pp. 154–161.
  * [18] P. Leamy, “paulleamy/isa-cough-detection,” Apr 2020. [Online]. Available: https://github.com/paulleamy/ISA-cough-detection.git
  * [19] C. Uhle, C. Dittmar, and T. Sporer, “Extraction of drum tracks from polyphonic music using independent subspace analysis,” in _Proc. ICA_ , 2003, pp. 843–847.
  * [20] A. Hyvärinen and E. Oja, “Independent component analysis: algorithms and applications,” _Neural networks_ , vol. 13, no. 4-5, pp. 411–430, 2000.
  * [21] P. Comon, “Independent component analysis, a new concept?” _Signal processing_ , vol. 36, no. 3, pp. 287–314, 1994.
  * [22] L. Kvapilova, V. Boza, P. Dubec, M. Majernik, J. Bogar, J. Jamison, J. C. Goldsack, D. J. Kimmel, and D. R. Karlin, “Continuous sound collection using smartphones and machine learning to measure cough,” _Digital Biomarkers_ , vol. 3, no. 3, pp. 166–175, 2019.
  * [23] E. Benetos, G. Lafay, and M. Lagrange, “DCASE2016 task 2 baseline,” DCASE2016 Challenge, Tech. Rep., September 2016.
  * [24] C. Bilen, G. Ferroni, F. Tuveri, J. Azcarreta, and S. Krstulovic, “A framework for the robust evaluation of sound event detection,” _arXiv preprint arXiv:1910.08440_ , 2019.
  * [25] P. Leamy, D. Berry, T. Burke, and D. Dorran, “Re-annotation of cough events in the AMI corpus,” in _2019 30th Irish Signals and Systems Conference (ISSC)_. IEEE, 2019, pp. 1–5.


# Assessing Distractors in Multiple-Choice Tests

Vatsal Raina
ALTA Institute, Cambridge University
<EMAIL_ADDRESS>
&Adian Liusie
ALTA Institute, Cambridge University
<EMAIL_ADDRESS>
Mark Gales
ALTA Institute, Cambridge University
<EMAIL_ADDRESS>

###### Abstract

Multiple-choice tests are a common approach for assessing candidates’
comprehension skills. Standard multiple-choice reading comprehension exams
require candidates to select the correct answer option from a discrete set
based on a question in relation to a contextual passage. For appropriate
assessment, the distractor answer options must by definition be incorrect but
plausible and diverse. However, generating good quality distractors satisfying
these criteria is a challenging task for content creators. We propose
automated assessment metrics for the quality of distractors in multiple-choice
reading comprehension tests. Specifically, we define quality in terms of the
incorrectness, plausibility and diversity of the distractor options. We assess
incorrectness using the classification ability of a binary multiple-choice
reading comprehension system. Plausibility is assessed by considering the
distractor confidence - the probability mass associated with the distractor
options for a standard multi-class multiple-choice reading comprehension
system. Diversity is assessed by pairwise comparison of an embedding-based
equivalence metric between the distractors of a question. To further validate
the plausibility metric we compare against candidate distributions over
multiple-choice questions and agreement with a ChatGPT model’s interpretation
of distractor plausibility and diversity.

##  1 Introduction

Figure 1: Distractor Assessment Framework (DAF) filtration pipeline for
generated distractors.

Multiple-choice tests are an efficient and effective way of assessing
candidates’ comprehension skills (Alderson, 2000) with key advantages such as
being a standardized format, eliminating subjective grading and being easy to
grade. These advantages make them a highly popular assessment method widely
adopted in a range of settings (Kurz, 1999), such as university exams, job
screening and qualification accreditation. A challenging aspect of generating
suitable multiple-choice questions is in selecting the incorrect options, i.e
the distractors (Gierl et al., 2017). Selecting good distractors is a subtle
process, which requires the option to possess several properties (Qiu et al.,
2020); 1) The distractor should not be a possible correct answer, as this
would make marking the question subjective. 2) the distractor option should
not be too obviously invalid, as then candidates may easily avoid them. 3) The
questions should have relatively diverse distractors, as this would better
allow questions to gauge more information from candidates.

Currently, test creators conduct a pre-test phase where questions are
internally reviewed and then tested on a subset of real candidates (Liusie et
al., 2023b), an evaluation process that is very manual and can be both
subjective and expensive. Automating the process to evaluate distractors would
lead to improved efficiency in the test creation process, and may aid test
designers to create high-quality questions. However, currently, assessing the
quality of distractors is a challenging task. To the best of our knowledge,
there are no existing datasets targeted towards assisting automated distractor
evaluation (beyond sequence overlap measures (Gao et al., 2019)), and
therefore any approach has to port information from other resources. Further,
validating the efficacy of approaches is a challenging task, especially
without manual labels of distractor quality, which themselves due to the
nature of the task are at risk of being subjective.

In this paper, we propose the Distractor assessment framework (DAF), a
collection of systems that can be used to automatically determine the quality
of distractors. Our framework provides automatic scores for the 3 previously
mentioned important properties of the distractors: incorrectness, plausibility
and diversity. The incorrectness detector is a binary machine reading
comprehension system that predicts whether a given distractor could be the
correct answer, the plausibility evaluator leverages system confidence, while
the diversity assessor considers the average similarity score between all
pairs of distractors. We further propose several methods to probe existing
large-scale foundation models, specifically ChatGPT instruction fine-tuned
(Ouyang et al., 2022) from GPT-3 (Brown et al., 2020), to validate the
suitability of our quality metrics and demonstrate that our methods do
reasonably capture elements of the considered properties. Additionally, we
validate the plausibility metric against human candidate distributions on
multiple-choice questions. Our contributions can be summarized as follows:

  * •

Proposed assessment metrics for the challenging task of distractor assessment
in terms of incorrectness, plausibility and diversity.

  * •

Verification of the assessment metrics including probing ChatGPT and
comparison with real candidate distribution scores.

##  2 Related Work

Previous automatic distractor assessment methods proposed to compare the
similarity of generated distractors with the ground-truth distractors present
in the dataset (Gao et al., 2019) or consider rule-based approaches (Pho et
al., 2015). Following standard reference-based evaluation, n-gram overlap
metrics such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004) and METEOR
(Banerjee and Lavie, 2005) have been considered, where these metrics measure
the overlap between generated distractors and the distractors from a set of
human-annotated ground truth sequences. However, having reference-based
distractor evaluation approaches has notable shortcomings (Moon et al., 2022).
In particular, for a given multiple-choice question, the set of annotated
distractors is unlikely to span the set of all possible good distractors, and
some options may get unfairly penalised simply because no similar ones exist
in the annotated set.

We therefore focus on decomposing the distractors in terms of individual
qualities (incorrectness, plausibility and diversity) and consider quality as
an amalgamation of the above. These approaches have been considered for the
assessment of alternate qualities of questions. Dugan et al. (2022)
investigate answer-agnostic generated questions in terms of the qualities of
relevance, interpretability and acceptability with comparison against human
markers. Raina and Gales (2022b) assess multiple-choice questions in terms of
grammatical fluidity, answerability, diversity and complexity. Our work
specifically explores distractor assessment in multiple-choice questions with
a focus on automated assessment.

##  3 Multiple-Choice Comprehension

In this section, we describe the multiple-choice reading comprehension task,
and the architecture of standard machine reading comprehension systems. Note
that the machine reading comprehension system will later be leveraged in
several components of the DAF (see Section 4).

###  3.1 Multiple-choice comprehension task

Multiple-choice reading comprehension is a common examination format that aims
to measure the reading comprehension abilities of candidates. Given question
$Q$ and passage of textual information, context $C$, candidates have to select
the correct answer from a discrete set of options $\\{O\\}$. The correct
answer $y_{ans}$ is then the option where the information in the passage is
consistent with the question.

###  3.2 Machine reading comprehension

Figure 2: Model architectures of multi-class and binary multiple-choice
machine reading comprehension systems with context, $C$, question, $Q$ and
options, $\\{O\\}$.

Machine reading comprehension (MRC) refers to building automatic systems for
performing the reading comprehension task. For multiple-choice reading
comprehension, state-of-the-art machine reading systems (Zhang et al., 2021;
Yamada et al., 2020; Zaheer et al., 2020; Wang et al., 2022) have demonstrated
human-level performance on public benchmarks (Clark et al., 2018; Lai et al.,
2017; Trischler et al., 2017; Yang et al., 2018). In this work, we consider
two variations of the approach:

Multi-class MRC: A standard approach for machine reading comprehension (Yu et
al., 2020; Raina and Gales, 2022a) is to predict a probability distribution
over the options, as shown in Figure 2. For this method, the context, question
and a particular option are concatenated together and fed through a standard
transformer encoder (Vaswani et al., 2017). The hidden representation output
by the transformer encoder is then passed through a linear layer to return a
scalar score. This process is repeated for each option in turn, and a softmax
function then returns a discrete probability distribution over the answer
options. Note that the weights are shared across each of the four versions of
the transformer encoder and linear layers. During inference, the answer option
with the largest probability mass is selected as the system prediction.

Binary MRC: As an alternative to the multi-class approach, we also consider a
binary multiple-choice machine reading comprehension system, as suggested by
Ghosal et al. (2022). This approach is similar to the multi-class approach,
however instead of the softmax at the final stage, the binary approach applies
a sigmoid to the logit scalar score. This approach therefore determines
whether a given option is the correct answer, instead of determining which of
the options is the correct one (which the multi-class approach does). At
inference time, the answer option with the greatest probability is selected as
the output from the model.

Both the binary MRC and the multi-class MRC systems have identical fundamental
structures, however differ in how the options are normalised before the final
probability output. The multi-class approach leverages the fact that only one
of the options is correct, and so the probability of the four options are
normalised relative to each other. The binary MRC system does not. Therefore
for tasks that are objective in nature and depend on the single option only,
such as incorrectness detection, we leverage the binary MRC system. However,
the multi-class MRC system is better suited for tasks where we want to assess
options relative to each other and the correct answer, such as plausibility
assessment. Note, the binary MRC system can be built from a single unit of the
multi-class MRC system.

##  4 Distractor Assessment Framework

As discussed in Section 2, n-gram reference-based assessment metrics may not
be appropriate for distractor assessment as 1) the number of valid distractors
for a given multiple choice reading question is vast, which a limited set of
references may fail to capture; 2) they are only valid when a set of reference
distractors are available, which requires human intervention and limits the
advantages of automatic distractor evaluation. The next section discusses the
DAF, which uses three different reference-free methods to estimate the quality
of distractors’ incorrectness, plausibility and diversity independently.

Incorrectness For multiple-choice reading comprehension questions, distractors
must by definition be incorrect, and the answer option must be the only valid
answer in the set of options. For automated multiple-choice question
generation pipelines, it is particularly important to ensure that any
generated distractors satisfy the requirement of being incorrect and do not
cause a subjective interpretation of the question. Here, we detail the
approach used to assess whether distractors satisfy the incorrectness
requirement.

The binary MRC system from Section 3.1 is used to assess incorrectness. The
system returns a probability score, $p_{c}$, which is the probability that the
system thinks the given option is correct. An appropriate threshold, $\tau$
can then be selected, such if the probability score is less than the
threshold, a distractor is deemed to satisfy the incorrectness requirement, as
indicated in Equation 1 (where $y\in\\{\texttt{incorrect, correct}\\}$ denotes
the binary output decision).

$y=\begin{cases}\texttt{incorrect},&\text{if }p_{c}<\tau\\\
\texttt{correct},&\text{otherwise}\end{cases}$ (1)

The selected threshold is a design choice of the test creator depending on how
stringent the incorrectness criteria should be. For example, the incorrectness
detector can be the first stage of a test creation pipeline to filter out the
generated questions with multiple options that could be valid. The test
creator may then select an operating point with low precision and high recall
(in terms of incorrectness) in order to capture a larger pool of questions
which should be considered in the subsequent stages of the evaluation
pipeline. Conversely, for high-stakes educational settings, an operating point
which leads to higher precision at the cost of recall may be preferred.

Plausibility As emphasised by Qiu et al. (2020), good quality distractors
should be both incorrect yet also plausible and not obviously invalid. Unlike
the binary incorrectness metric, plausibility is a continuous property, and
distractors can be plausible to different degrees. For assessing the
plausibility of a distractor within a distractor set, one can consider the
model confidence of the multi-class multiple-choice machine reading
comprehension system. The motivation for this approach is that a high
confidence score is one that the MRC system finds more plausible, which one
can assume would be similar for real candidates. We further define the
plausibility score as the sum of confidence scores corresponding to each of
the distractors in a question, which can similarly be calculated as the
difference between 1 and the confidence score associated with the correct
answer. This is expressed in Equation 2, where $P_{\theta}$ denotes the
probability distribution learnt by the multi-class machine reading
comprehension system.

$\texttt{plausibility}=1-\max_{y}P_{\theta}\left(y|C,Q,\\{O\\}\right)$ (2)

Diversity As a human candidate, when attempting a multiple-choice question all
four options are considered together. If distractors are similar or identical,
then one can eliminate multiple options simultaneously using the same
information, limiting the amount of comprehension that a question may require.
Therefore, it becomes increasingly important to ensure the distractors are
diverse. Particularly, diversity has been demonstrated to be a concern for
automated question generation systems (Raina and Gales, 2022b), where systems
are quite susceptible to frequently generating repeated distractors. This
demands a need for automated approaches to determine the diversity amongst
distractors to select, or at least be aware of, the distractor set with the
maximum diversity.

In this work, the BERT Equivalence Metric (BEM) (Bulian et al., 2022) is
leveraged for assessing the diversity of the distractors. BEM is a semantic
similarity measure for question answering, where the equivalence score between
an answer and the reference is returned. BEM takes the text of a predicted
answer, the text of the answer option and the question, concatenates them
together and a BERT system then returns a scalar score, $0\leq e\leq 1$. This
score captures the equivalence between the candidate and the reference, where
a score of $e=1$ indicates the candidate and the reference are identical while
$e=0$ indicates the candidate and the reference are completely semantically
different. BEM is trained explicitly on an answer equivalence dataset and has
been shown to out-perform zero-shot equivalence measures such as the BERTScore
from Zhang et al. .

In this work, BEM is applied pair-wise to all possible pairs of distractors in
a given question. The context is not concatenated to the question since
initial experiments demonstrated that the long contexts diluted the
differences between pairs of distractors. Since BEM is not order invariant, we
average the output from BEM with both orderings for the pair of distractors
considered. The overall diversity is quoted as the 1 minus the average
pairwise BEM scores between the distractors, as indicated by Equation 3 where
the $K$ distractors associated with a given question are denoted as
$\\{d_{1},d_{2},\ldots,d_{K}\\}$.

$\texttt{diversity}=1.-\sum_{i=1}^{K}\sum_{j=1,j\neq
i}^{K}\frac{\texttt{BEM}[d_{i},d_{j},Q]}{K^{2}-K}$ (3)

##  5 Experiments

###  5.1 Data

RACE++: RACE++ (Liang et al., 2019) is a large-scale machine reading
comprehension dataset of real questions used in middle school (RACE-M), high
school (RACE-H) and college level (RACE-C). There are 4 options per question
with a single option as the correct for each. Table 1 details the train,
validation and test splits used for training and testing of the multiple-
choice reading comprehension datasets.

subset | train | valid | test
---|---|---|---
RACE-M | 25,241 | 1,436 | 1,436
RACE-H | 62,445 | 3,451 | 3,498
RACE-C | 12,702 | 712 | 708
RACE++ | 100,388 | 5,599 | 5,642
Table 1: Data splits for RACE++. RACE++ is composed of questions at the middle
school (M), high school (H), and college (C) level.

CMCQRD111https://www.englishlanguageitutoring.com/.: The Cambridge Multiple-
Choice Questions Reading Dataset (CMCQRD) (Mullooly et al., 2023) is a small-
scale multiple-choice reading comprehension evaluation dataset from the pre-
testing stage partitioned into grade levels B1 to C2 on the Common European
Framework of Reference for Languages (CEFR). Additionally, a subset of the
CMCQRD dataset has candidate distributions available. We perform our
experiments only on this subset of questions as analyzed in Liusie et al.
(2023b). The statistics of these questions are given in Table 2.

subset | contexts | questions
---|---|---
B1 | 23 | 115
B2 | 37 | 222
C1 | 12 | 72
C2 | 6 | 39
CMCQRD | 78 | 448
Table 2: Splits of CMCQRD subset (with candidate distribution) of data between
CEFR levels.

###  5.2 Training

For multi-class MRC, we take the ELECTRA pretrained language model (Clark et
al., 2020) (specifically ELECTRA-large 222Available at:
https://huggingface.co/google/electra-large-discriminator) and train the
system with cross-entropy loss on the train split of RACE++, with the best
epoch selected using the RACE++ validation split. Following Raina and Gales
(2022a), the model is trained using the AdamW optimizer, a batch size of 4,
learning rate of 2e-6 and a maximum of 3 training epochs. All inputs are
truncated to 512 tokens, and all processing is performed on NVIDIA V100
graphical processing units. We consider ensembles of 3 models for each system.
For the binary MRC system, a single unit of the trained multi-class MRC system
is used with the softmax layer removed and a sigmoid at the output instead
(mimics Figure 2). 333Initial experiments trained a separate system for binary
MRC where each option was reformatted as individual data points with either a
label of correct (answer) or incorrect (distractor). However, this system
generalized poorly to CMCQRD despite good performance on the RACE++ dataset.

##  6 Results

In this section, we present results for assessing incorrectness, plausibility
and diversity as part of the DAF for standard multiple-choice reading
comprehension datasets.

Table 3 presents the baseline performance of the MRC system on the RACE++ and
CMCQRD test sets. Overall, the MRC system ports across well from RACE++ to
CMCQRD, getting an accuracy of 85% on RACE++ and 74% on CMCQRD. It is also
apparent that for both datasets, the accuracy of the MRC system degrades for
more challenging questions by approximately 7% from RACE-M to RACE-C and 25%
from CEFR level B1 to C2.

Table 3 further presents the newly proposed incorrectness, plausibility and
diversity scores using the described approaches applied to both multiple-
choice reading comprehension datasets. For each question in each dataset, the
distractors for the question are considered to be the set of ‘generated’
distractors (first stage of distractor generation in the pipeline of Figure 1)
for which the incorrectness, plausibility and diversity scores need to be
calculated. For incorrectness, each distractor is classified as either
incorrect or correct based on the optimal operating point threshold of
performance (see Table 4) which is a value of $\tau=0.25$ for RACE++ and
$\tau=0.04$ for CMCQRD. Hence, the overall incorrectness score is the
percentage of distractors that are categorized as incorrect (higher is
better). Plausibility (Equation 2) and diversity (Equation 3) scores are
averaged across all the questions in the dataset.

Dataset | Acc. | Incorr. | Plaus. | Divers.
---|---|---|---|---
RACE++ | 85.0 | 91.8 | 15.0 | 74.1
RACE-M | 88.1 | 93.8 | 11.8 | 66.8
RACE-H | 84.4 | 91.0 | 15.7 | 75.7
RACE-C | 81.6 | 91.7 | 18.0 | 81.0
CMCQRD | 74.3 | 86.7 | 27.7 | 78.2
B1 | 90.4 | 85.5 | 11.9 | 75.7
B2 | 73.4 | 86.9 | 30.0 | 78.0
C1 | 56.9 | 87.5 | 40.9 | 80.3
C2 | 64.1 | 87.2 | 37.0 | 82.8
Table 3: Ported accuracy of the MRC system trained on RACE++. For proposed
distractors (in the dataset), incorrectness rate, average plausibility and
diversity scores are reported as percentages.

It is observed that the incorrectness rate remains consistent across all the
splits for RACE++. A similar consistency is evident on the CMCQRD dataset. In
general, it can be seen that the plausibility scores tend to be higher for
more challenging questions for both RACE++ and CMCQRD. This is potentially
explainable by the fact that more challenging questions can expect to have a
greater probability mass attributed to the distractors compared to the correct
answer option. Loosely, the average diversity score follows a similar pattern
where more challenging questions can expect to have more diverse distractors.
Possibly a low diversity in the distractors offers fewer opportunities to
distract the candidates.

We have presented the incorrectness, plausibility and diversity scores on the
RACE++ and CMCQRD datasets. The subsequent sections aim to provide a form of
verification for each of these metrics to demonstrate they are suitable for
the respective qualities that they are assessing.

###  6.1 Assessing correctness detector

This section assesses the accuracy of the correctness detector which is used
for measuring the incorrectness rate. To assess the accuracy, we assume that
the allocation of answer options as either distractors or the correct answer
are the ground-truth binary labels. Table 4 assesses how well the correctness
detector performs on RACE++ and CMCQRD datasets using the optimal F1 score for
this binary classification task.

| Precision | Recall | F1
---|---|---|---
RACE++ | 80.1 | 72.7 | 76.2
CMCQRD | 62.2 | 65.8 | 64.0
Table 4: Performance for the correctness detector.

Figure 3 presents the precision-recall curve of the correctness detector on
both the RACE++ and CMCQRD datasets. From both Figure 3 and Table 4, the
performance of the correctness detector is sensible, with performance on
CMCQRD lagging RACE++ demonstrated by top F1 scores of 76% and 64% on RACE++
and CMCQRD respectively. In line with these single-value summaries, the CMCQRD
precision-recall curve undercuts the RACE++ precision-recall curve for all
recall rates.

Figure 3: Precision-Recall curve for correctness detector on RACE++ and
CMCQRD.

Figure 4 further presents an operating chart for the correctness detector. The
chart sweeps the threshold for the binary MRC system from 0 to 1 and
identifies the fraction of distractors and answer options that are captured
cumulatively. As expected, for both the RACE++ and CMCQRD operating charts,
the ‘distractor’ curve significantly leads the ‘answer’ curve. The operating
charts offers content creators a means to choose an operating threshold; a low
threshold on correctness may guarantee that only real distractors are captured
but also reduces the pool of distractors that are considered in the review
process.

(a) RACE++.

(b) CMCQRD.

Figure 4: Operating chart for correctness detector on RACE++ and CMCQRD.

###  6.2 Verification of plausibility/diversity via ChatGPT

Recently, generative large-scale foundation models (Brown et al., 2020;
Chowdhery et al., 2022; Scao et al., 2022), such as the popularized ChatGPT,
have demonstrated state-of-the-art performance across a large range of natural
language tasks in zero-shot and few-shot settings. These models are
particularly impressive at successfully completing tasks that they have never
seen before.

Figure 5: Prompting ChatGPT for more/less plausible/diverse distractors.

However, there remain several practical challenges in using foundation models
such as ChatGPT. 1\. There are concerns with security of the data as
confidential data cannot be taken off-site. This is an important consideration
in the educational setting as often the trial questions to be assessed will
form a core component of a live standardized test. 2. Access via an API
(Application Program Interface) means the input/output is restricted as well
as the risk of evolving with limited warning, jeopardizing an exterior
infrastructure developed to interact with the API. 3. There is a continual
cost of interacting with API access which may limit the scalability in
deployment. 4. Zero-shot performance can be challenging to tune to specific
tasks(s) of interest. Therefore, we do not employ ChatGPT as a direct
assessment approach for the DAF. It is necessary for any automated assessment
approach to be local so that there is complete control over the model.
Instead, ChatGPT is considered here as a validation process for the DAF that
in itself bypasses ChatGPT’s challenges.

Here ChatGPT, specifically gpt-3.5-turbo
444https://platform.openai.com/docs/models/gpt-3-5, is employed as an approach
to verify the proposed plausibility and diversity assessment metrics. ChatGPT
is given a standard multiple-choice reading comprehension question. The
foundation model is then requested to refine the choice of the distractors to
make them more/less plausible or diverse. Figure 5 presents the prompts.

By probing ChatGPT to create alternatives for the distractors, it is useful to
check the agreement of ChatGPT’s interpretation of plausibility and diversity
with the proposed assessment metrics.

System | All | M | H | C
---|---|---|---|---
Vanilla | 85.0 | 88.1 | 84.4 | 81.6
Increase plausibility | 74.6 | 77.5 | 73.6 | 73.7
Decrease plausibility | 84.0 | 85.3 | 83.3 | 85.2
Increase diversity | 74.6 | 78.5 | 73.7 | 71.3
Decrease diversity | 62.0 | 68.3 | 59.7 | 60.9
Table 5: Accuracy of ensemble on test split of RACE++ (RACE-M, RACE-H, RACE-C)
using the multi-MRC system after probing ChatGPT to refine the distractors in
terms of plausibility and diversity.

From Table 5, the accuracy of RACE++ trained system is impacted by exchanging
the distractors with variants provided by ChatGPT. Prompting ChatGPT to
generate more plausible distractors leads to the accuracy of the MRC system
dropping by up to 10% as the altered questions on average are more
challenging. In contrast, prompting ChatGPT to decrease the plausibility has
less of an impact on the behaviour of the MRC system’s accuracy. By prompting
ChatGPT to increase or decrease the diversity of the distractors, there is an
observed drop in the MRC system accuracy, particularly for less diverse
distractors of more than 20%.

System | Standard | Context-free
---|---|---
Vanilla | 85.0 | 57.0
Increase plausibility | 74.6 | 39.3
Decrease plausibility | 84.0 | 54.4
Increase diversity | 74.6 | 42.3
Decrease diversity | 62.0 | 38.7
Table 6: Impact of world knowledge after probing ChatGPT to refine the
distractors in terms of plausibility and diversity on the RACE++ test set.

In Table 6, the impact of world knowledge in reading comprehension (Liusie et
al., 2023a) is explored for the ChatGPT generated distractors. Here, a
context-free system (no access to the context) measures to what extent a
question relies on using knowledge outside the context to determine the
correct answer. With all values substantially above the random performance of
25%, for both the original questions and the probed version of the questions,
there is significant scope to leverage world knowledge to answer the
questions.

(a) Plausibility.

(b) Diversity.

Figure 6: Impact on distribution of plausibility and diversity by probing
ChatGPT on the RACE++ dataset. See Equations 2 and 3 for the definitions of
plausibility and diversity respectively.

The distribution of plausibilities and diversities in Figure 6 further
demonstrates that there is an upward shift with the increased
plausibility/diversity variants of the distractors.

setA | setB |  $>$ plaus. |  $>$ div.
---|---|---|---
Increase | Vanilla | 57.6 | 60.7
Vanilla | Decrease | 63.2 | 53.2
Increase | Decrease | 69.1 | 63.3
Table 7: Fraction of examples for which plausibility and diversity of
distractors in a question for setA is $>$ setB.

Finally, Table 7 presents the impact of refining the distractors at an
individual question level. For example, the increased plausibility versions of
the distractors compared to the decreased plausibility versions have a high
plausibility score for 69% of the RACE++ questions. It seems it is challenging
to be able to increase plausibility and decrease diversity, while it is
relatively easier to decrease plausibility and increase diversity. With all
scores above 50%, it suggests that there is alignment between ChatGPT’s
interpretation of plausibility/diversity and the assessment approaches for
these qualities.

###  6.3 Verification of plausibility via candidate distribution

As in Section 4, the plausibility of distractors is assessed using the
probability confidence scores distribution output from a multiple-choice
machine reading comprehension system. The claim is that a higher confidence
score suggests that a distractor is more plausible. In a practical sense, the
plausibility scores for the distractors should correspond with how likely a
candidate taking a test is to select the distractors. CMCQRD (see Section 5.1)
includes candidate distributions over multiple-choice questions. Therefore,
the human candidate distributions are used to verify whether the plausibility
scores from a standard multiple-choice reading comprehension system correspond
with candidates’ interpretation of the plausibility of distractors.

We consider two comparison methods for validating the plausibility scores.
Intra-question: compare the ranking of distractors by system confidence and
human confidence for each question. Inter-question: compare the ranking across
questions of distractor confidence (see Equation 2) by the system and the
candidates.

The intra-question verification informs whether the individual distractor
plausibility scores by the system can be used to identify which distractors
are more convincing while the inter-question verification informs whether the
system’s distractor confidence is a universal measure of how convincing the
distractors are for a question as a collective.

For intra-question rankings, the averaged (across questions) Spearman’s rank
correlation coefficients between the candidate probabilities for a set of
distractors per question and the system’s probabilities for the same set of
distractors is 0.25. For the inter-question case, the global Spearman’s rank
correlation between candidate plausibility (sum of individual distractor
confidences) and system plausibility is 0.22. Despite not being strong
correlations (potentially due to human noise from learners taking the test),
the positive values indicate that human understanding of distractor
plausibility is somewhat aligned with the system’s understanding.

##  7 Conclusions

This work proposes the distractor assessment framework, an automatic approach
for assessing the quality of distractors on three key properties:
incorrectness, plausibility and diversity. By leveraging multi-class and
binary machine reading comprehension systems, and semantic similarity metrics,
we propose intuitive methods for calculating automatic scores for the 3
properties. We validate the metrics by refining distractors with ChatGPT.
Further there is a positive correlation indicated between candidate and system
plausibilities.

##  8 Acknowledgements

This research is partially funded by the EPSRC (The Engineering and Physical
Sciences Research Council) Doctoral Training Partnership (DTP) PhD studentship
and supported by Cambridge University Press & Assessment (CUP&A), a department
of The Chancellor, Masters, and Scholars of the University of Cambridge.

## References

  * Alderson (2000) J. Charles. Alderson. 2000.  _Assessing Reading_ , 1 edition.  Cambridge University Press,, Cambridge :.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Bulian et al. (2022) Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Boerschinger, and Tal Schuster. 2022.  Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation.  _arXiv preprint arXiv:2202.07654_.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  In _International Conference on Learning Representations_.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _ArXiv_ , abs/1803.05457.
  * Dugan et al. (2022) Liam Dugan, Eleni Miltsakaki, Shriyash Upadhyay, Etan Ginsberg, Hannah Gonzalez, DaHyeon Choi, Chuning Yuan, and Chris Callison-Burch. 2022.  A feasibility study of answer-agnostic question generation for education.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1919–1926, Dublin, Ireland. Association for Computational Linguistics.
  * Gao et al. (2019) Yifan Gao, Lidong Bing, Piji Li, Irwin King, and Michael R Lyu. 2019.  Generating distractors for reading comprehension questions from real examinations.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 6423–6430.
  * Ghosal et al. (2022) Deepanway Ghosal, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2022.  Two is better than many? binary classification as an effective approach to multi-choice question answering.  _arXiv preprint arXiv:2210.16495_.
  * Gierl et al. (2017) Mark J Gierl, Okan Bulut, Qi Guo, and Xinxin Zhang. 2017.  Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review.  _Review of Educational Research_ , 87(6):1082–1116.
  * Kurz (1999) Terri Barber Kurz. 1999.  A review of scoring algorithms for multiple-choice tests.
  * Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and E. Hovy. 2017.  Race: Large-scale reading comprehension dataset from examinations.  In _EMNLP_.
  * Liang et al. (2019) Yichan Liang, Jianheng Li, and Jian Yin. 2019.  A new multi-choice reading comprehension dataset for curriculum learning.  In _Proceedings of The Eleventh Asian Conference on Machine Learning_ , volume 101 of _Proceedings of Machine Learning Research_ , pages 742–757, Nagoya, Japan. PMLR.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Liusie et al. (2023a) Adian Liusie, Vatsal Raina, and Mark Gales. 2023a.  World knowledge in multiple choice reading comprehension.  In _Proceedings of the Sixth Fact Extraction and VERification Workshop (FEVER)_ , pages 49–57, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Liusie et al. (2023b) Adian Liusie, Vatsal Raina, Andrew Mullooly, Kate Knill, and Mark J. F. Gales. 2023b.  Analysis of the Cambridge multiple-choice questions reading dataset with a focus on candidate response distribution.
  * Moon et al. (2022) Hyeongdon Moon, Yoonseok Yang, Hangyeol Yu, Seunghyun Lee, Myeongho Jeong, Juneyoung Park, Jamin Shin, Minsam Kim, and Seungtaek Choi. 2022.  Evaluating the knowledge dependency of questions.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10512–10526.
  * Mullooly et al. (2023) Andrew Mullooly, Øistein Andersen, Luca Benedetto, Paula Buttery, Andrew Caines, Mark J.F. Gales, Yasin Karatay, Kate Knill, Adian Liusie, Vatsal Raina, and Shiva Taslimipoor. 2023.  The Cambridge Multiple-Choice Questions Reading Dataset.  Cambridge University Press and Assessment.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Pho et al. (2015) Van-Minh Pho, Anne-Laure Ligozat, and Brigitte Grau. 2015.  Distractor quality evaluation in multiple choice questions.  In _Artificial Intelligence in Education: 17th International Conference, AIED 2015, Madrid, Spain, June 22-26, 2015. Proceedings 17_ , pages 377–386. Springer.
  * Qiu et al. (2020) Zhaopeng Qiu, Xian Wu, and Wei Fan. 2020.  Automatic distractor generation for multiple choice questions in standard tests.  _arXiv preprint arXiv:2011.13100_.
  * Raina and Gales (2022a) Vatsal Raina and Mark Gales. 2022a.  Answer uncertainty and unanswerability in multiple-choice machine reading comprehension.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1020–1034, Dublin, Ireland. Association for Computational Linguistics.
  * Raina and Gales (2022b) Vatsal Raina and Mark Gales. 2022b.  Multiple-choice question generation: Towards an automated assessment framework.  _arXiv preprint arXiv:2209.11830_.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Trischler et al. (2017) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017.  Newsqa: A machine comprehension dataset.  In _Rep4NLP@ACL_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2022) Siyuan Wang, Wanjun Zhong, Duyu Tang, Zhongyu Wei, Zhihao Fan, Daxin Jiang, Ming Zhou, and Nan Duan. 2022.  Logic-driven context extension and data augmentation for logical reasoning of text.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1619–1629.
  * Yamada et al. (2020) Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020\.  Luke: Deep contextualized entity representations with entity-aware self-attention.  In _EMNLP_.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _EMNLP_.
  * Yu et al. (2020) Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020.  Reclor: A reading comprehension dataset requiring logical reasoning.  In _International Conference on Learning Representations_.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020.  Big bird: Transformers for longer sequences.  _ArXiv_ , abs/2007.14062.
  * (34) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhang et al. (2021) Zhuosheng Zhang, Junjie Yang, and Hai Zhao. 2021.  Retrospective reader for machine reading comprehension.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 14506–14514.

##  Appendix A Limitations

This work explores automated approaches to assess the incorrectness,
plausibility and diversity of distractors encompassed in a DAF. A limitation
is that the current research focuses specifically on the RACE++ and CMCQRD
datasets. Further work should investigate the applicability of the DAF for
other multiple-choice datasets. Second, the proposed assessment methods are
verified using both candidate distributions and agreement with ChatGPT’s
interpretation of the qualities. Explicit at-scale human evaluation may help
provide further evidence for the validity of the assessment approaches.
Finally, a more extensive and rigorous approach could have been taken to
determine optimal prompts for increasing/decreasing the plausibility or the
diversity of the distractors.

##  Appendix B Ethics Statement

There are no ethical concerns with this work.