# Language Agnostic Multilingual Information Retrieval
with Contrastive Learning

Xiyang Hu1, Xinchi Chen2∗, Peng Qi2∗, Deguang Kong2, Kunlun Liu2,
William Yang Wang2, Zhiheng Huang2
1Carnegie Mellon University
2AWS AI Labs
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Multilingual information retrieval (IR) is challenging since annotated
training data is costly to obtain in many languages. We present an effective
method to train multilingual IR systems when only English IR training data and
some parallel corpora between English and other languages are available. We
leverage parallel and non-parallel corpora to improve the pretrained
multilingual language models’ cross-lingual transfer ability. We design a
_semantic contrastive loss_ to align representations of parallel sentences
that share the same semantics in different languages, and a new _language
contrastive loss_ to leverage parallel sentence pairs to remove language-
specific information in sentence representations from non-parallel corpora.
When trained on English IR data with these losses and evaluated zero-shot on
non-English data, our model demonstrates significant improvement to prior work
on retrieval performance, while it requires much less computational effort. We
also demonstrate the value of our model for a practical setting when a
parallel corpus is only available for a few languages, but a lack of parallel
corpora resources persists for many other low-resource languages. Our model
can work well even with a small number of parallel sentences, and be used as
an add-on module to any backbones and other tasks.

††footnotetext: 1 Work done during an internship at AWS AI Labs.
††footnotetext: * These authors contributed equally to this
work.††footnotetext: Code https://github.com/xiyanghu/multilingualIR.

##  1 Introduction

Information retrieval (IR) is an important natural language processing task
that helps users efficiently gather information from a large corpus (some
representative downstream tasks include question answering, summarization,
search, recommendation, etc.), but developing effective IR systems for all
languages is challenging due to the cost of, and therefore lack of, annotated
training data in many languages. While this problem is not unique to IR
research Joshi et al. (2020), constructing IR data is often more costly due to
the need to either translate a large text corpus or gather relevancy
annotations, or both, which makes it difficult to generalize IR models to
lower-resource languages.

One solution to this is to leverage the pretrained multilingual language
models to encode queries and corpora for multilingual IR tasks Zhang et al.
(2021); Sun and Duh (2020). One series of work on multilingual representation
learning is based on training a masked language model, some with the next
sentence prediction task, on monolingual corpora of many languages, such as
mBERT and XLM-R Conneau et al. (2020). They generally do not explicitly learn
the alignment across different languages and do not perform effectively in
empirical IR experiments. Other works directly leverage multilingual parallel
corpora or translation pairs to explicitly align the sentences in two
languages, such as InfoXLM Chi et al. (2021) and LaBSE Feng et al. (2022).

Figure 1:  (a) The semantic contrastive loss encourages the embeddings of
parallel pairs, i.e. sentences that have the same semantics but from different
languages, to be close to each other, and away from the rest negative samples
— sentences with different semantics. (b) The language contrastive loss
incorporates the non-parallel corpora in addition to the parallel ones. It
encourages the distances from a sentence representation, which can be a sample
from both the parallel corpora and the non-parallel corpora, to the two
embeddings of a paralleled pair to be the same.

In this work, we propose to use the semantic contrastive loss and the language
contrastive loss to jointly train with the information retrieval objective,
for learning cross-lingual representations that encourage efficient lingual
transfer ability on retrieval tasks. Our semantic contrastive loss aims to
align the embeddings of sentences that have the same semantics. It is similar
to the regular InfoNCE Oord et al. (2018) loss, which forces the
representations of parallel sentence pairs in two languages to be close to
each other, and away from other negative samples. Our language contrastive
loss aims to leverage the non-parallel corpora for languages without any
parallel data, which are ignored by the semantic contrastive loss. It
addresses the practical scenario wherein parallel corpora are easily
accessible for a few languages, but the lack of such resources persists for
many low-resource languages. The language contrastive loss encourages the
distances from a sentence representation to the two embeddings of a paralleled
pair to be the same. Figure 1 illustrates how the two losses improve language
alignment. In experiments, we evaluate the zero-shot cross-lingual transfer
ability of our model on monolingual information retrieval tasks for 10
different languages. Experimental results show that our proposed method
obtains significant gains, and it can be used as an add-on module to any
backbones. We also demonstrate that our method is much more computationally
efficient than prior work. Our method works well with only a small number of
parallel sentence pairs and works well on languages without any parallel
corpora.

Figure 2:  Our model framework contains two parts: the main task (IR), and the
parallel corpora task. For the main task part, we use a dual-encoder dense
passage retrieval module for information retrieval. For the parallel corpora
task part, we adopt the semantic contrastive loss to improve cross-lingual
domain adaptation with parallel corpora. We also use the language contrastive
loss by leveraging parallel corpora and non-parallel corpora altogether.

##  2 Background: Multilingual DPR

Dense Passage Retriever (DPR) Karpukhin et al. (2020) uses a dual-encoder
structure to encode the queries and passages separately for information
retrieval. To generalize to multilingual scenarios, we replace DPR’s original
BERT encoders with a multilingual language model XLM-R Conneau et al. (2020)
to transfer English training knowledge to other languages.

Concretely, given a batch of $N$ query-passage pairs ($\boldsymbol{p}_{i}$,
$\boldsymbol{q}_{i}$), we consider all other passages
$\boldsymbol{p}_{j},j\neq i$ in the batch irrelevant (negative) passages, and
optimize the retrieval loss function as the negative log-likelihood of the
gold passage:

$\displaystyle\mathcal{L}_{\text{IR}}$
$\displaystyle=-\frac{1}{N}\sum_{i=1}^{N}$ (1)
$\displaystyle\log\frac{\exp\left(\operatorname{sim}\left(\boldsymbol{q}_{i},\boldsymbol{p}_{i}\right)\right)}{\exp\left(\operatorname{sim}\left(\boldsymbol{q}_{i},\boldsymbol{p}_{i}\right)\right)+\sum_{j=1,j\neq
i}^{N}\exp\left(\operatorname{sim}\left(\boldsymbol{q}_{i},\boldsymbol{p}_{j}\right)\right)}$

where the similarity of two vectors is defined as
$\operatorname{sim}(\boldsymbol{u},\boldsymbol{v})=\frac{\boldsymbol{u}^{\top}\boldsymbol{v}}{\|\boldsymbol{u}\|\|\boldsymbol{v}\|}$.

##  3 Contrastive Learning for Cross-Lingual Generalization

The multilingual dense passage retriever only uses English corpora for
training. To improve the model’s generalization ability to other languages, we
leverage two contrastive losses, semantic contrastive loss and language
contrastive loss. Figure 2 shows our model framework.

Specifically, the semantic contrastive loss Chen et al. (2020a) pushes the
embedding vectors of a pair of parallel sentences close to each other, and at
the same time away from other in-batch samples that have different semantics.
The language contrastive loss focuses on the scenario when there is no
parallel corpora for some languages, which encourages the distance from a
sentence embedding to paralleled embedding pairs to be the same.

###  3.1 Semantic Contrastive Loss

To learn a language-agnostic IR model, we wish to encode the sentences with
the same semantics but from different languages to have the same embeddings.
For each parallel corpora batch, we do not limit our sample to just one
specific language pair. We randomly sample different language pairs for a
batch. For example, a sampled batch could contain multiple language pairs of
En-Ar, En-Ru, En-Zh, etc. This strategy can increase the difficulty of our
contrastive learning and make the training more stable.

Concretely, we randomly sample a mini-batch of $2N$ data points ($N$ here does
not have to be the same value as the $N$ in Section 2). The batch contains $N$
pairs of parallel sentences from multiple different languages. Given a
positive pair $\boldsymbol{z}_{i}$ and $\boldsymbol{z}_{j}$, the embedding
vectors of a pair of parallel sentences $(i,j)$ from two languages, the rest
$2(N-1)$ samples are used as negative samples. The semantic contrastive loss
for a batch is:

$\displaystyle\mathcal{L}_{\text{semaCL}}=$
$\displaystyle-\frac{1}{2N}\sum_{(i,j)}$ (2) $\displaystyle{\big{[}}$
$\displaystyle\log\frac{\exp\left(\operatorname{sim}\left(\boldsymbol{z}_{i},\boldsymbol{z}_{j}\right)/\tau\right)}{\sum_{k=1,k\neq
i}^{2N}\exp\left(\operatorname{sim}\left(\boldsymbol{z}_{i},\boldsymbol{z}_{k}\right)/\tau\right)}+$
$\displaystyle\log\frac{\exp\left(\operatorname{sim}\left(\boldsymbol{z}_{j},\boldsymbol{z}_{i}\right)/\tau\right)}{\sum_{k=1,k\neq
j}^{2N}\exp\left(\operatorname{sim}\left(\boldsymbol{z}_{j},\boldsymbol{z}_{k}\right)/\tau\right)}{\big{]}}$

where $\tau$ is a temperature hyperparameter.

###  3.2 Language Contrastive Loss

When training multilingual IR systems, we might not always have parallel
corpora for all languages of interest. In a realistic scenario, we have easy
access to a few high-resource languages’ parallel corpora, but no such
availability for many low-resource languages. We propose a language
contrastive loss to generalize the model’s ability to the languages which do
not have any parallel corpora. For a batch $B$ consisting of both parallel
corpora $\mathbb{P}$ and non-parallel corpora $\mathbb{Q}$, we denote
$\boldsymbol{z}_{i}$ and $\boldsymbol{z}_{j}$ as the embeddings of a pair of
parallel sentences $(i,j)$ from two languages. We wish the cosine similarity
from any other sentence embedding $\boldsymbol{z}_{k}$ to the two embeddings
of a parallel pair to be the same. Therefore, we minimize the following loss.

$\displaystyle\mathcal{L}_{\text{langCL}}=$
$\displaystyle-\frac{1}{N(N-2)}\sum_{(i,j)\in\mathbb{P}}\sum_{k\in(\mathbb{P}\cup\mathbb{Q})\setminus\\{i,j\\}}$
(3) $\displaystyle{\big{[}}$
$\displaystyle\log\frac{\exp(\operatorname{sim}\left(\boldsymbol{z}_{i},\boldsymbol{z}_{k}\right))}{\exp(\operatorname{sim}\left(\boldsymbol{z}_{i},\boldsymbol{z}_{k}\right))+\exp(\operatorname{sim}\left(\boldsymbol{z}_{j},\boldsymbol{z}_{k}\right))}+$
$\displaystyle\log\frac{\exp(\operatorname{sim}\left(\boldsymbol{z}_{j},\boldsymbol{z}_{k}\right))}{\exp(\operatorname{sim}\left(\boldsymbol{z}_{i},\boldsymbol{z}_{k}\right))+\exp(\operatorname{sim}\left(\boldsymbol{z}_{j},\boldsymbol{z}_{k}\right))}\big{]}$

The optimum can be reached when
$\operatorname{sim}\left(\boldsymbol{z}_{i},\boldsymbol{z}_{k}\right)=\operatorname{sim}\left(\boldsymbol{z}_{j},\boldsymbol{z}_{k}\right)$
for all $i,j,k$. Note that the parallel corpus involved is not the target
language’s parallel corpus. For example, in Formula 3, $i$ and $j$ are two
languages that are parallel with each other, and $k$ is a third language
(target language) that does not have any parallel corpus with other languages.

###  3.3 Semantic vs Language Contrastive Losses

While both the semantic contrastive loss and language contrastive loss can
serve to align the representations of parallel sentences and remove language
bias, they achieve this goal differently, one via contrasting against in-batch
negative samples, the other using in-batch parallel examples to constrain the
target language embeddings. Moreover, a key property of the language
contrastive loss is that as long as there is some parallel corpus, we can use
this loss function to remove the language bias from representations of
sentences where no parallel data exists, which makes it more broadly
applicable.

##  4 Training

The two contrastive losses are applied to the passage encoder only.
Experiments show that applying them to both the passage encoder and the query
encoder would result in unstable optimization, where we see weird jumps in the
training loss curves.

The joint loss with the information retrieval loss, the semantic contrastive
loss, and the language contrastive loss is

$\mathcal{L}=\mathcal{L}_{\text{IR}}+w_{s}\mathcal{L}_{\text{semaCL}}+w_{l}\mathcal{L}_{\text{langCL}},$
(4)

where $w_{s}$ and $w_{l}$ are hyperparameters for the semantic contrastive
loss and the language contrastive loss weights which need to be tuned
adaptively in different tasks.

We train our model using 8 Nvidia Tesla V100 32GB GPUs. We use a batch size of
48. We use the AdamW optimizer with $\beta_{1}=0.9,\beta_{2}=0.999$ and a
learning rate of $10^{-5}$. For the three losses
$\mathcal{L}_{\text{IR}},\mathcal{L}_{\text{semaCL}},\mathcal{L}_{\text{langCL}}$,
we sequentially calculate the loss and the gradients. We use $w_{s}=0.01$ and
$w_{l}=0.001$. The hyperparameters are determined through a simple grid
search.

##  5 Experiments

###  5.1 Datasets

Our IR experiments involve two types of datasets: IR datasets and the parallel
corpora.

####  5.1.1 Information Retrieval

In our experiments, we only use English information retrieval corpora (Natural
Questions), and we evaluate the model’s zero-shot transfer ability on other
target languages (Mr.TyDi).

  * •

Natural Questions Kwiatkowski et al. (2019) is an English QA dataset.
Following Zhang et al. (2021), we use NQ dataset to train IR.

  * •

Mr.TyDi is a multilingual dataset for monolingual retrieval Zhang et al.
(2021), which is constructed from a question answering dataset TyDi Clark et
al. (2020). It contains eleven typologically diverse languages, i.e., Arabic
(Ar), Bengali (Bn), English (En), Finnish (Fi), Indonesian (Id), Japanese
(Ja), Korean (Ko), Russian (Ru), Swahili (Sw), Telugu (Te), Thai (Th). We do
not use Mr.TyDi for IR training.

####  5.1.2 Parallel Corpora

WikiMatrix parallel corpora contains extracted parallel sentences from the
Wikipedia articles in 85 different languages Schwenk et al. (2021). For those
languages involved in the Mr.Tydi dataset, the number of parallel pairs
between them and the English of the WikiMatrix dataset ranges from 51,000 and
1,019,000. During training, we sample the same number of parallel pairs (50K)
for them.

###  5.2 Baseline Models

We apply our contrastive loss functions on three multilingual pretrained
language models:

  * •

XLM-R Conneau et al. (2020) is a pre-trained transformer-based multilingual
language model. It is trained on a corpus from 100 languages only with the
Masked language Model (MLM) objective in a Roberta way.

  * •

InfoXLM Chi et al. (2021) uses 42GB parallel corpora to pre-train XLM-R by
maximizing mutual information between multilingual-multi-granularity texts.

  * •

LaBSE Feng et al. (2022) pre-trains BERT with Masked Language Model and
Translation Language Model on the monolingual data and bilingual translation
pairs. They train the model by 1.8M steps using a batch size of 8192.

In Table 1, we compare the computational efforts needed by each model to
improve the language transfer ability. Both InfoXLM and LaBSE require a large-
scale pre-training which needs a larger batch size and a larger number of
training steps than ours. Our model only requires "co-training" on the
parallel corpora along with the main task. In Table 1, we list our model’s
training steps on the information retrieval task. This comparison indicates
that for the retrieval task, our model does not need the costly pre-training
as InfoXLM and LaBSE.

| InfoXLM | LaBSE | Our Model
---|---|---|---
Batch Size | 2,048 | 8,192 | 48
Training Steps | 200K | 1.8M | 24.54K
Training Compute | 347x | 12,518x | 1x
Table 1: A comparison of our model and baseline models’ pre-training for lingual adaptation. Ours actually uses a "co-training" mode rather than "pre-training", so our training steps are the same as the main task.  Model |  Metric  | Ar | Bn | En | Fi | Id | Ja | Ko | Ru | Sw | Te | Th | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---
XLM-R |  RR  | 0.335 | 0.345 | 0.275 | 0.302 | 0.368 | 0.274 | 0.275 | 0.287 | 0.231 | 0.298 | 0.403 | 0.308
\+ semaCL |  RR  | 0.399 | 0.465 | 0.332 | 0.355 | 0.445 | 0.360 | 0.338 | 0.345 | 0.281 | 0.550 | 0.482 | 0.396
\+ langCL |  RR  | 0.402 | 0.437 | 0.338 | 0.335 | 0.425 | 0.339 | 0.320 | 0.329 | 0.265 | 0.600 | 0.453 | 0.386
\+ semaCL + langCL |  RR  | 0.404 | 0.465 | 0.338 | 0.346 | 0.430 | 0.333 | 0.320 | 0.341 | 0.266 | 0.516 | 0.477 | 0.385
Table 2: MRR@100 on the monolingual information retrieval task of Mr.TyDi dataset. Model |  Metric  | Ar | Bn | En | Fi | Id | Ja | Ko | Ru | Sw | Te | Th | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Results reported by Wu et al. (2022)
XLM-R |  RR  | 0.365 | 0.374 | 0.275 | 0.318 | 0.395 | 0.299 | 0.304 | 0.306 | 0.274 | 0.346 | 0.401 | 0.333
InfoXLM |  RR  | 0.373 | 0.354 | 0.325 | 0.300 | 0.380 | 0.310 | 0.299 | 0.313 | 0.351 | 0.311 | 0.400 | 0.338
LABSE |  RR  | 0.372 | 0.504 | 0.314 | 0.309 | 0.376 | 0.271 | 0.309 | 0.325 | 0.394 | 0.465 | 0.374 | 0.365
CCP |  RR  | 0.426 | 0.457 | 0.359 | 0.372 | 0.462 | 0.377 | 0.346 | 0.360 | 0.392 | 0.470 | 0.489 | 0.410
Results reported by Zhang et al. (2021)
BM25 (default) |  RR  | 0.368 | 0.418 | 0.140 | 0.284 | 0.376 | 0.211 | 0.285 | 0.313 | 0.389 | 0.343 | 0.401 | 0.321
BM25 (tuned) |  RR  | 0.367 | 0.413 | 0.151 | 0.288 | 0.382 | 0.217 | 0.281 | 0.329 | 0.396 | 0.424 | 0.417 | 0.333
Our implementation |  |  |  |  |  |  |  |  |  |  |  |  |
XLM-R |  RR  | 0.335 | 0.345 | 0.275 | 0.302 | 0.368 | 0.274 | 0.275 | 0.287 | 0.231 | 0.298 | 0.403 | 0.308
\+ semaCL |  RR  | 0.399 | 0.465 | 0.332 | 0.355 | 0.445 | 0.360 | 0.338 | 0.345 | 0.281 | 0.550 | 0.482 | 0.396
InfoXLM |  RR  | 0.371 | 0.337 | 0.284 | 0.272 | 0.343 | 0.311 | 0.271 | 0.298 | 0.338 | 0.306 | 0.385 | 0.320
\+ semaCL |  RR  | 0.375 | 0.413 | 0.331 | 0.314 | 0.406 | 0.321 | 0.292 | 0.318 | 0.299 | 0.304 | 0.427 | 0.345
LaBSE |  RR  | 0.321 | 0.419 | 0.240 | 0.283 | 0.347 | 0.224 | 0.290 | 0.296 | 0.428 | 0.387 | 0.322 | 0.323
\+ semaCL |  RR  | 0.333 | 0.485 | 0.300 | 0.313 | 0.395 | 0.216 | 0.265 | 0.329 | 0.374 | 0.330 | 0.308 | 0.332
Table 3: MRR@100 on the monolingual information retrieval task of Mr.TyDi
dataset.

###  5.3 Information Retrieval - All languages have parallel corpora with
English

For the information retrieval training, we follow the previous literature
Zhang et al. (2021); Wu et al. (2022) to use an English QA dataset – the
Natural Questions dataset Kwiatkowski et al. (2019) for both training and
validation.

We evaluate our model performance on the Mr.TyDi dataset Zhang et al. (2021)
for monolingual query passage retrieval in eleven languages. We follow Zhang
et al. (2021) to use MRR@100 and Recall@100 as metrics.

In this section, we experimented with the setting when we have parallel
corpora from English to all other target languages. We tested three different
variants of our model using the XLM-R as the backbone:

  1. 1.

we only include the semantic contrastive loss for the parallel corpora:
$\mathcal{L}_{\text{IR}}+w_{s}\mathcal{L}_{\text{semaCL}}$;

  2. 2.

we only include the language contrastive loss for the parallel corpora:
$\mathcal{L}_{\text{IR}}+w_{l}\mathcal{L}_{\text{langCL}}$;

  3. 3.

we use both the semantic contrastive loss and the language contrastive loss:
$\mathcal{L}_{\text{IR}}+w_{s}\mathcal{L}_{\text{semaCL}}+w_{l}\mathcal{L}_{\text{langCL}}$.

Table 2 shows the results of our model and the baseline XLM-R model. We also
report the results of Wu et al. (2022), which propose a model called
contrastive context prediction (CCP) to learn multilingual representations by
leveraging sentence-level contextual relations as self-supervision signals.
For our analysis, we mainly focus on MRR, since MRR is more aligned with our
retrieval loss function, which aims to rank relevant passages at higher
orders. We also report Recall@100 in Table 7 in Appendix A. We find that
overall our model performs significantly better than the basic XLM-R. For our
different model variants, we find that: (1) using only the semantic
contrastive loss for the parallel corpora would achieve the best average
performance; (2) using only the language contrastive loss for the parallel
corpora also achieves a significant performance improvement, which is lower
than but close to using only the semantic contrastive loss; (3) using both
semantic contrastive loss and language contrastive loss would only contribute
to a few languages like Ar, but does not improve the overall performance. Our
assumption is that the semantic contrastive loss has already efficiently
removed the language embedding shifts by leveraging the parallel pairs, so it
is not helpful to use additional language contrastive loss when we have
parallel corpora for all the languages. In Section 5.4, we experiment with a
more practical scenario when we only have parallel corpora for some of the
target languages but non-parallel corpora for the rest. And we find our
language contrastive loss brings significant performance gains in that case.

We then further compare the performance of our best model — XLM-R + semantic
contrastive loss, with those of other strong baselines, i.e. InfoXLM and
LaBSE. We also examine if the semantic contrastive loss can be used as an add-
on module to InfoXLM and LaBSE to further boost their performance. Table 3
shows the MRR@100 results of XLM-R, InfoXLM, LaBSE themselves — all of them
are trained with the IR loss, and the results trained jointly with the
semantic contrastive loss. We find that our best model — XLM-R with only
semantic contrastive loss — significantly outperforms these strong baselines.
Note that both InfoXLM and LaBSE involve a large-scale pre-training to improve
the lingual transfer ability, which is not required in our method. Our model
only requires joint training with the contrastive loss, which needs much less
computational effort as in Table 1. We also find that the semantic contrastive
loss can be used as an add-on module to effectively boost the performance of
InfoXLM and LaBSE. But such an add-on module’s improvements on InfoXLM and
LaBSE are not as large as that on XLM-R. We speculate that this phenomenon
could be attributed to that InfoXLM and LaBSE have already been pre-trained on
other datasets, which have some distribution shifts away from the WikiMatrix
dataset we used for the semantic contrastive loss add-on module. We also
report the Recall@100 results in Table 8 of Appendix A. In addition to the
above results output by our own runs, we also list the results reported by Wu
et al. (2022) in Table 3 as a reference. The difference in the baseline model
performances may be due to the randomness during model training. We also
present the performance of the traditional BM25 method. The average MRR@100 of
BM25 is significantly lower than that of our method.

####  5.3.1 Effect of the Size of Parallel Dataset

We further investigate the effect of the size of the parallel dataset on the
multilingual retrieval performance. We train our model by varying the parallel
dataset size using the XLM-R with only semantic contrastive loss. Figure 3
shows the results. We find that: (1) using parallel corpora can significantly
boost the retrieval performance, compared with the dashed horizontal line when
we do not have parallel corpora at all (the basic XLM-R); (2) even when we
only have a small parallel corpus of 500 pairs for each language, we can
already achieve a good performance<EMAIL_ADDRESS>When we gradually increase
the parallel corpora to 50,000, the MRR@100 grows gradually to 0.396. But the
increase is not very large. This suggests that our model framework can work
well even with a small parallel corpora dataset. This makes our method
promising for those low-resource languages which lack parallel corpora with
English.

Figure 3:  The effect of the size of the parallel dataset for each language,
with the 95% CI in shadow.

####  5.3.2 Effect of Language Pair Connection

In order to understand how different language pair connections affect
performance, we conduct experiments using different language pairs on En, Fi,
Ja, Ko. We experimented with six different settings:

  1. 1.

Basic setting: Train XLM-R without using any parallel corpora, which is the
same as the first row in Table 2;

  2. 2.

Setting 1: Train XLM-R with parallel corpora between English and all other
languages, i.e. En-Fi, En-Ja, En-Ko;

  3. 3.

Setting 2: Train XLM-R with parallel corpora between English and Korean, and
between Korean and the rest languages, i.e. En-Ko, Ko-Fi, Ko-Ja;

  4. 4.

Setting 3: Train XLM-R with parallel corpora between English and Korean, and
between Japanese to Finnish, i.e. En-Ko, Ja-Fi;

  5. 5.

Setting 4: Train XLM-R with parallel corpora between English and Korean, i.e.
En-Ko;

  6. 6.

Setting 5: Train XLM-R with parallel corpora between Japanese to Finnish, i.e.
Ja-Fi.

Table 4 shows the MRR@100 results. We find that all settings 1 to 5
significantly surpass the basic setting. This echoes our previous finding that
it is helpful to leverage parallel corpora. Among settings 1 to 5, the
differences are not large — the minimum MRR of them is 0.325, and the maximum
one is 0.343. This suggests that the connection among language pairs is not a
deterministic factor for our method. We also report the Recall@100 in Table 9
of Appendix A.

|  Metric  | En | Fi | Ja | Ko | Avg
---|---|---|---|---|---|---
Basic Setting |  RR  | 0.275 | 0.302 | 0.274 | 0.275 | 0.281
Setting 1 |  RR  | 0.330 | 0.340 | 0.325 | 0.331 | 0.331
Setting 2 |  RR  | 0.334 | 0.344 | 0.354 | 0.320 | 0.338
Setting 3 |  RR  | 0.318 | 0.341 | 0.348 | 0.365 | 0.343
Setting 4 |  RR  | 0.336 | 0.354 | 0.340 | 0.345 | 0.343
Setting 5 |  RR  | 0.323 | 0.352 | 0.317 | 0.311 | 0.325
Table 4: MRR@100 on different language pair connections.

Model |  Metric  |  Ar |  Bn |  En |  Fi |  Id |  Ja |  Ko | Avg$\parallel$ |  Ru |  Sw |  Te |  Th | Avg$\nparallel$ | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
XLM-R |  RR  |  0.335 |  0.345 |  0.275 |  0.302 |  0.368 |  0.274 |  0.275 | 0.307 |  0.287 |  0.231 |  0.298 |  0.403 | 0.305 | 0.308
\+ semaCL |  RR  |  0.389 |  0.427 |  0.310 |  0.333 |  0.404 |  0.323 |  0.316 | 0.365 |  0.319 |  0.252 |  0.423 |  0.448 | 0.360 | 0.358
\+ langCL (WikiMatrix) |  RR  |  0.385 |  0.394 |  0.314 |  0.340 |  0.411 |  0.319 |  0.299 | 0.351 |  0.316 |  0.227 |  0.474 |  0.430 | 0.361 | 0.355
\+ langCL (Mr.TyDi) |  RR  |  0.347 |  0.378 |  0.287 |  0.306 |  0.371 |  0.292 |  0.277 | 0.322 |  0.294 |  0.221 |  0.248 |  0.395 | 0.289 | 0.311
\+ semaCL + langCL (WikiMatrix) |  RR  |  0.396 |  0.421 |  0.342 |  0.357 |  0.438 |  0.357 |  0.336 | 0.384 |  0.350 |  0.274 |  0.420 |  0.476 | 0.380 | 0.378
\+ semaCL + langCL (Mr.TyDi) |  RR  |  0.408 |  0.470 |  0.336 |  0.362 |  0.438 |  0.339 |  0.332 | 0.391 |  0.347 |  0.291 |  0.449 |  0.471 | 0.389 | 0.385

  * Note: Avg for languages with ($\parallel$) and without ($\nparallel$) parallel data.

Table 5: Experiment results when Ru, Sw, Te, Th do NOT have parallel data
(MRR@100).

###  5.4 Information Retrieval - Some languages do not have parallel data

In this section, we investigate the scenario when we have parallel corpora
only for some of the target languages, but not for the rest languages. This
scenario emphasizes a realistic constraint that we lack parallel corpora for
many low-resource languages. To test it, we leave Ru, Sw, Te, Th as languages
that do not have parallel corpora, and keep these parallel corpora for all
other languages, i.e. Ar, Bn, Fi, Id, Ja, Ko. We experimented with three
different settings:

  1. 1.

XLM-R + Semantic CL: we only use the semantic contrastive loss on languages
which have parallel corpora (Ar, Bn, Fi, Id, Ja, Ko):
$\mathcal{L}_{\text{IR}}+w_{s}\mathcal{L}_{\text{semaCL}}$;

  2. 2.

XLM-R + Semantic CL + Language CL (WikiMatrix): we use the semantic
contrastive loss on languages which have parallel corpora (Ar, Bn, Fi, Id, Ja,
Ko), and the language contrastive loss on these parallel corpora (Ar, Bn, Fi,
Id, Ja, Ko) along with the non-parallel WikiMatrix corpora:
$\mathcal{L}_{\text{IR}}+w_{s}\mathcal{L}_{\text{semaCL}}+w_{l}\mathcal{L}_{\text{langCL}}$.

  3. 3.

XLM-R + Semantic CL + Language CL (Mr.TyDi): we use the semantic contrastive
loss on languages which have parallel corpora (Ar, Bn, Fi, Id, Ja, Ko), and
the language contrastive loss on these parallel corpora (Ar, Bn, Fi, Id, Ja,
Ko) along with the non-parallel Mr.TyDi corpora:
$\mathcal{L}_{\text{IR}}+w_{s}\mathcal{L}_{\text{semaCL}}+w_{l}\mathcal{L}_{\text{langCL}}$;

Table 5 shows the MRR@100 results of our experiments. The language contrastive
loss can effectively leverage the non-parallel corpora to improve the
information retrieval performance. For the XLM-R + Semantic CL + Language CL
(Mr.TyDi) setting, the language contrastive loss boosts the average MRR@100
from 0.358 to 0.385. We also calculate the average performance on the
languages with parallel corpora (Ar, Bn, Fi, Id, Ja, Ko), and the languages
without parallel corpora (Ru, Sw, Te, Th). The Avg (withParallel) column and
the Avg (noParallel) column in Table 5 are their corresponding results. We
find that the language contrastive loss can improve the performance on both
types of languages. For languages with parallel corpora (Ar, Bn, Fi, Id, Ja,
Ko), the MRR@100 increases from 0.365 to 0.391; for languages without parallel
corpora (Ru, Sw, Te, Th), the MRR@100 increase from 0.360 to 0.389. This
result suggests our model can be effectively deployed in situations when we
have no parallel corpora for low-resource languages. Appendix A Table 10
reports the Recall@100 results.

Since using the Mr.TyDi corpora brings in the target domain information, we
also examine the XLM-R + Semantic CL + Language CL (WikiMatrix) setting. This
setting uses the WikiMatrix non-parallel corpora for Ru, Sw, Te, Th — it does
not introduce the target domain information, and reflects the clean gain from
the language contrastive loss. We find that using the WikiMatrix non-parallel
corpora achieves a little lower but close performance than the one using the
Mr.TyDi corpora. This suggests that the introduction of the target domain
information is very minor in improving IR performance.

####  5.4.1 Effect of the Size of Non-Parallel Dataset

We further investigate the effect of the size of the non-parallel dataset on
the multilingual retrieval performance. We train our model by varying the non-
parallel dataset size using XLM-R with both the semantic contrastive loss and
the language contrastive loss. We keep the size of the parallel dataset fixed
at 50,000. Figure 4 shows the results. The dashed horizontal line is the one
using only parallel corpora, i.e. the first row in Table 5. We find that: (1)
using non-parallel corpora can significantly boost the retrieval performance,
compared with the most left point when we do not use the non-parallel corpora
at all; (2) when the non-parallel corpora dataset size increases from 0 to
10,000, the MRR@100 improves quickly; (3) when the non-parallel dataset size
increases from 10,000 to 50,000, the MRR@100 has minor changes, but its
variance decreases.

Figure 4:  The effect of the size of the non-parallel dataset for each
language, with the 95% CI in shadow.

###  5.5 BUCC: Bitext Retrieval

The information retrieval task above is not common to see in multilingual NLP
papers. A closely related task they often work on is the BUCC111BUCC (Building
and Using Comparable Corpora) is a dataset for bitext mining — identify
sentence pairs that are translations in two different languages Zweigenbaum et
al. (2018). The task aims to mine parallel sentences between English and four
other languages, i.e., German, French, Russian, and Chinese. About 2–3% of the
sentences in the whole dataset are gold parallel. task Zweigenbaum et al.
(2018). The BUCC task has been tested in the LaBSE benchmark model we used in
the previous section Feng et al. (2022), and in many other multilingual NLP
works, such as Artetxe and Schwenk 2019, etc; Yang et al. 2019, etc; Schwenk
2018, etc; Reimers and Gurevych 2020, etc. Therefore, following these works,
we also investigate our model’s performance on the BUCC task.

For the BUCC bitext mining task, we follow previous work Artetxe and Schwenk
(2019); Reimers and Gurevych (2020) to first encode texts, and then use the
equation below to calculate the score of two sentence embeddings
$\boldsymbol{u},\boldsymbol{v}$:

$\displaystyle\operatorname{score}(\boldsymbol{u},\boldsymbol{v})=\frac{\operatorname{sim}(\boldsymbol{u},\boldsymbol{v})}{\sum_{\boldsymbol{z}\in\operatorname{NN}_{k}(\boldsymbol{u})}\frac{\operatorname{sim}(\boldsymbol{u},\boldsymbol{z})}{2k}+\sum_{\boldsymbol{z}\in\operatorname{NN}_{k}(\boldsymbol{v})}\frac{\operatorname{sim}(\boldsymbol{v},\boldsymbol{z})}{2k}}$
(5)

where $\operatorname{NN}_{k}(\boldsymbol{u})$ denotes $\boldsymbol{u}$’s $k$
nearest neighbors in another language. The training set is used to find a
threshold value of the score, for which pairs with scores above this threshold
are predicted as parallel sentences. We use F1 to measure the model
performance on BUCC.

Table 6 shows F1 score of our model based on the XLM-R, InfoXLM, and LaBSE. We
first examine the vanilla XLM-R, InfoXLM, and LaBSE as the text encoder. LaBSE
and InfoXLM outperform XLM-R a lot due to their large-scale pre-training on
improving lingual adaptation using parallel datasets. When we add our semantic
contrastive loss to XLM-R, we get a large improvement across all four
languages. We find that our model (XLM-R + Semantic CL) outperforms XLM-R, but
underperforms InfoXLM and LaBSE. We attribute LaBSE’s great performance to its
much larger pre-training than ours, and LaBSE’s training involves Translation
Language Model Conneau and Lample (2019) with translation corpora. This is
exactly the same type of corpora as BUCC’s translation parallel pairs. When we
add our semantic contrastive loss to InfoXLM, we obtain performance gain for
all languages. The gain is smaller than that of XLM-R because InfoXLM has
already been trained on parallel corpora. When we add our semantic contrastive
loss module to LaBSE, we obtained a small increase in the average performance
— the performance on Zh has a significant increase.

Model | De | Fr | Ru | Zh | Avg
---|---|---|---|---|---
XLM-R |  17.90 |  12.12 |  21.84 |  15.06 |  16.73
\+ semaCL |  72.86 |  69.44 |  73.10 |  66.27 |  70.42
InfoXLM |  63.32 |  54.36 |  69.92 |  66.19 |  63.45
\+ semaCL |  80.40 |  75.48 |  78.20 |  75.04 |  77.28
LaBSE Feng et al. (2022) |  92.50 |  88.70 |  88.90 |  88.90 |  89.75
LaBSE |  93.03 |  89.75 |  89.75 |  85.93 |  89.62
\+ semaCL |  92.95 |  89.50 |  89.82 |  89.77 |  90.51
Table 6: F1 score on the BUCC task. Row LaBSE Feng et al. (2022) is the
results reported in Feng et al. (2022). All other rows are the output of our
own implementation.

One important insight we get from comparing Table 6 and Table 3 is that a
model’s better performance in NLP tasks like BUCC does not necessarily mean
better performance in information retrieval. Most existing multilingual NLP
papers only examine the BUCC bi-text retrieval task, and we highlight the
inconsistency between models’ performances on the two types of retrieval
tasks.

##  6 Related Work

Dense monolingual / multilingual information retrieval study recently attracts
great attention, which mainly benefits from (1) supervised finetuning based on
large pre-trained language models and (2) self-supervised contrastive
learning.

Dense Passage Retrieval Karpukhin et al. (2020) is the framework first
proposed for monolingual superivsed finetuning on information retrieval. It
uses a BERT-based dual-encoder structure to encode the query and the candidate
passages into embeddings. Similar to monolingual IR, supervised finetuning can
also be applied to multilingual pretrained language models (LMs) for
multilingual IR. Commonly uesd multilingual pretrained LMs include
multilingual BERT (mBERT, Devlin et al., 2019) and XLM-R Conneau et al.
(2020), both of which are trained on large corpora representing about 100
languages primarily with the masked language modeling task. These models do
not use any explicit objective to improve the alignment between language
sentences. Recent efforts in NLP field have provided easy access to parallel
corpora, e.g. Schwenk et al. (2021). Many multilingual language models use
additional parallel data to improve lingual transfer ability. InfoXLM Chi et
al. (2021) uses parallel corpora to pre-train XLM-R by maximizing mutual
information between multilingual multi-granularity texts. LaBSE Feng et al.
(2022) pre-trains BERT with Masked Language Model and Translation Language
Model on the monolingual data and bilingual translation pairs.

Self-supervised contrastive learning is another way used to improve the cross-
lingual alignment. Contrastive learning maximizes the agreement between
positive samples, and minimizes the similarity of positive and negative ones
(He et al., 2020; Chen et al., 2020a, b, c). For language representation
learning, Clark et al. (2019) apply contrastive learning to train a
discriminative model to learn language representations. For multilingual
representation learning, contrastive learning has been used to improve cross-
lingual transfer ability by using additional parallel data Hu et al. (2021) or
by leveraging other self-supervision signals Wu et al. (2022).

##  7 Conclusion

In this paper, we present a model framework for multilingual information
retrieval by improving lingual adaptation through contrastive learning. Our
experiments demonstrate the effectiveness of our methods in learning better
cross-lingual representations for information retrieval tasks. The two
contrastive losses can be used as an add-on module to any backbones and many
other tasks besides information retrieval.

##  8 Limitations

In this work, we did not conduct a detailed analysis of how language-specific
characteristics contribute to our model’s cross-lingual generalization
capabilities. Future work may address this question through extensive matrix
experiments — traverse the training on each possible language pair combination
and evaluate on all languages.

## References

  * Artetxe and Schwenk (2019) Mikel Artetxe and Holger Schwenk. 2019.  Margin-based parallel corpus mining with multilingual sentence embeddings.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3197–3203, Florence, Italy. Association for Computational Linguistics.
  * Chen et al. (2020a) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020a.  A simple framework for contrastive learning of visual representations.
  * Chen et al. (2020b) Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. 2020b.  Big self-supervised models are strong semi-supervised learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 22243–22255.
  * Chen et al. (2020c) Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. 2020c.  Improved baselines with momentum contrastive learning.  _arXiv preprint arXiv:2003.04297_.
  * Chi et al. (2021) Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021.  InfoXLM: An information-theoretic framework for cross-lingual language model pre-training.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3576–3588, Online. Association for Computational Linguistics.
  * Clark et al. (2020) Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages.  _Transactions of the Association for Computational Linguistics_ , 8:454–470.
  * Clark et al. (2019) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2019.  Electra: Pre-training text encoders as discriminators rather than generators.  In _International Conference on Learning Representations_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  _Advances in neural information processing systems_ , 32.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Feng et al. (2022) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022\.  Language-agnostic BERT sentence embedding.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 878–891, Dublin, Ireland. Association for Computational Linguistics.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9729–9738.
  * Hu et al. (2021) Junjie Hu, Melvin Johnson, Orhan Firat, Aditya Siddhant, and Graham Neubig. 2021\.  Explicit alignment objectives for multilingual bidirectional encoders.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3633–3643, Online. Association for Computational Linguistics.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020.  The state and fate of linguistic diversity and inclusion in the NLP world.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6282–6293, Online. Association for Computational Linguistics.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4512–4525, Online. Association for Computational Linguistics.
  * Schwenk (2018) Holger Schwenk. 2018.  Filtering and mining parallel data in a joint multilingual space.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 228–234, Melbourne, Australia. Association for Computational Linguistics.
  * Schwenk et al. (2021) Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2021.  WikiMatrix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1351–1361, Online. Association for Computational Linguistics.
  * Sun and Duh (2020) Shuo Sun and Kevin Duh. 2020.  CLIRMatrix: A massively large collection of bilingual and multilingual datasets for cross-lingual information retrieval.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4160–4170, Online. Association for Computational Linguistics.
  * Wu et al. (2022) Ning Wu, Yaobo Liang, Houxing Ren, Linjun Shou, Nan Duan, Ming Gong, and Daxin Jiang. 2022.  Unsupervised context aware sentence representation pretraining for multi-lingual dense retrieval.  _arXiv preprint arXiv:2206.03281_.
  * Yang et al. (2019) Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2019.  Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_ , pages 5370–5378. International Joint Conferences on Artificial Intelligence Organization.
  * Zhang et al. (2021) Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021.  Mr. TyDi: A multi-lingual benchmark for dense retrieval.  In _Proceedings of the 1st Workshop on Multilingual Representation Learning_ , pages 127–137, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zweigenbaum et al. (2018) Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2018.  Overview of the third bucc shared task: Spotting parallel sentences in comparable corpora.  In _Proceedings of 11th workshop on building and using comparable corpora_ , pages 39–42.

##  Appendix A Appendix

Table 7 and Table 8 show the Recall@100 results of experiments in Section 5.3.
Table 9 shows the Recall@100 results of experiments in Section 5.3.2. Table 10
shows the Recall@100 results of experiments in Section 5.4.

Model |  Metric  | Ar | Bn | En | Fi | Id | Ja | Ko | Ru | Sw | Te | Th | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---
XLM-R |  ecall  | 0.782 | 0.797 | 0.754 | 0.755 | 0.840 | 0.741 | 0.691 | 0.741 | 0.614 | 0.820 | 0.852 | 0.762
\+ semaCL |  ecall  | 0.799 | 0.851 | 0.777 | 0.773 | 0.867 | 0.779 | 0.730 | 0.763 | 0.597 | 0.862 | 0.886 | 0.789
\+ langCL |  ecall  | 0.799 | 0.820 | 0.782 | 0.769 | 0.858 | 0.769 | 0.723 | 0.750 | 0.629 | 0.892 | 0.871 | 0.788
\+ semaCL + langCL |  ecall  | 0.806 | 0.864 | 0.798 | 0.784 | 0.858 | 0.780 | 0.736 | 0.743 | 0.626 | 0.867 | 0.877 | 0.794
Table 7: Recall@100 on the monolingual information retrieval task of Mr.TyDi dataset. Model |  Metric  | Ar | Bn | En | Fi | Id | Ja | Ko | Ru | Sw | Te | Th | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Results reported by Wu et al. (2022)
XLM-R |  ecall  | 0.813 | 0.842 | 0.776 | 0.782 | 0.886 | 0.785 | 0.727 | 0.774 | 0.633 | 0.875 | 0.882 | 0.798
InfoXLM |  ecall  | 0.806 | 0.860 | 0.804 | 0.749 | 0.869 | 0.788 | 0.717 | 0.767 | 0.724 | 0.867 | 0.874 | 0.802
LABSE |  ecall  | 0.762 | 0.910 | 0.783 | 0.760 | 0.852 | 0.669 | 0.644 | 0.744 | 0.750 | 0.889 | 0.834 | 0.782
CCP |  ecall  | 0.820 | 0.883 | 0.801 | 0.787 | 0.875 | 0.800 | 0.732 | 0.772 | 0.751 | 0.888 | 0.889 | 0.818
Results reported by Zhang et al. (2021)
BM25 (default) |  RR  | 0.793 | 0.869 | 0.537 | 0.719 | 0.843 | 0.645 | 0.619 | 0.648 | 0.764 | 0.758 | 0.853 | 0.732
BM25 (tuned) |  RR  | 0.800 | 0.874 | 0.551 | 0.725 | 0.846 | 0.656 | 0.797 | 0.660 | 0.764 | 0.813 | 0.853 | 0.758
Our implementation |  |  |  |  |  |  |  |  |  |  |  |  |
XLM-R |  ecall  | 0.782 | 0.797 | 0.754 | 0.755 | 0.840 | 0.741 | 0.691 | 0.741 | 0.614 | 0.820 | 0.852 | 0.762
\+ semaCL |  ecall  | 0.799 | 0.851 | 0.777 | 0.773 | 0.867 | 0.779 | 0.730 | 0.763 | 0.597 | 0.862 | 0.886 | 0.789
InfoXLM |  ecall  | 0.797 | 0.900 | 0.785 | 0.725 | 0.843 | 0.790 | 0.717 | 0.753 | 0.711 | 0.873 | 0.875 | 0.797
\+ semaCL |  ecall  | 0.790 | 0.842 | 0.791 | 0.731 | 0.829 | 0.805 | 0.708 | 0.753 | 0.646 | 0.800 | 0.866 | 0.778
LaBSE |  ecall  | 0.769 | 0.887 | 0.760 | 0.773 | 0.854 | 0.652 | 0.649 | 0.764 | 0.832 | 0.862 | 0.824 | 0.784
\+ semaCL |  ecall  | 0.725 | 0.865 | 0.762 | 0.770 | 0.845 | 0.575 | 0.604 | 0.734 | 0.739 | 0.724 | 0.697 | 0.731
Table 8: Recall@100 on the monolingual information retrieval task of Mr.TyDi dataset. |  Metric  | En | Fi | Ja | Ko | Avg
---|---|---|---|---|---|---
Basic Setting |  ecall  | 0.754 | 0.755 | 0.741 | 0.691 | 0.735
Setting 1 |  ecall  | 0.776 | 0.770 | 0.777 | 0.706 | 0.757
Setting 2 |  ecall  | 0.785 | 0.778 | 0.781 | 0.710 | 0.763
Setting 3 |  ecall  | 0.767 | 0.762 | 0.766 | 0.723 | 0.754
Setting 4 |  ecall  | 0.779 | 0.785 | 0.764 | 0.722 | 0.762
Setting 5 |  ecall  | 0.765 | 0.759 | 0.722 | 0.703 | 0.737
Table 9: Recall@100 on different language pair connections.

Model |  Metric  |  Ar |  Bn |  En |  Fi |  Id |  Ja |  Ko | Avg$\parallel$ |  Ru |  Sw |  Te |  Th | Avg$\nparallel$ | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
XLM-R |  ecall  |  0.782 |  0.797 |  0.754 |  0.755 |  0.840 |  0.741 |  0.691 | 0.766 |  0.741 |  0.614 |  0.820 |  0.852 | 0.757 | 0.762
\+ semaCL |  ecall  |  0.759 |  0.824 |  0.752 |  0.738 |  0.826 |  0.745 |  0.715 | 0.767 |  0.724 |  0.598 |  0.851 |  0.859 | 0.758 | 0.762
\+ langCL (WikiMatrix) |  RR  |  0.778 |  0.797 |  0.782 |  0.757 |  0.840 |  0.752 |  0.726 | 0.776 |  0.734 |  0.550 |  0.861 |  0.854 | 0.749 | 0.766
\+ langCL (Mr.TyDi) |  RR  |  0.767 |  0.842 |  0.739 |  0.749 |  0.819 |  0.761 |  0.713 | 0.770 |  0.724 |  0.605 |  0.773 |  0.857 | 0.739 | 0.759
\+ semaCL + langCL (WikiMatrix) |  ecall  |  0.784 |  0.856 |  0.776 |  0.774 |  0.866 |  0.759 |  0.738 | 0.796 |  0.761 |  0.592 |  0.828 |  0.887 | 0.767 | 0.783
\+ semaCL + langCL (Mr.TyDi) |  ecall  |  0.792 |  0.806 |  0.768 |  0.782 |  0.871 |  0.782 |  0.737 | 0.795 |  0.755 |  0.619 |  0.850 |  0.885 | 0.777 | 0.786

  * Note: Avg for languages with ($\parallel$) and without ($\nparallel$) parallel data.

Table 10: Experiment results when Ru, Sw, Te, Th do NOT have parallel data
(Recall@100).


# Using Language Models to Detect Alarming Student Responses

Chirstopher Ormerod, Milan Patel, and Harry Wang  Cambium Assessment
1000 Thomas Jefferson St NW, Washington
District of Columbia, 20007, United States
<EMAIL_ADDRESS>
###### Abstract.

This article details the advances made to a system that uses artificial
intelligence to identify alarming student responses. This system is built into
our assessment platform to assess whether a student’s response indicates they
are a threat to themselves or others. Such responses may include details
concerning threats of violence, severe depression, suicide risks, and
descriptions of abuse. Driven by advances in natural language processing, the
latest model is a fine-tuned language model trained on a large corpus
consisting of student responses and supplementary texts. We demonstrate that
the use of a language model delivers a substantial improvement in accuracy
over the previous iterations of this system.

##  1\. Introduction

Automated Text Scoring (ATS) refers to using artificial intelligence (AI) to
approximate the assessment of constructed text responses. Despite its
potential for reducing costs, ensuring consistent scores, and minimizing bias,
there are still very real and valid concerns about the complete removal of
human oversight from the scoring process. In particular, this article concerns
instances where the constructed response suggests that the student poses a
risk to themselves or others. These are situations where it is necessary for a
person to intervene to ensure the safety of everyone in the school community.
We refer to responses of this nature as an Alarming Student Response (ASR),
which may include threats of violence, severe depression, suicide risks, and
descriptions of abuse [2]. This program is incredibly important, especially
given the regularity and severity of school shootings.

While some testing agencies ensure that all responses are reviewed by a person
to detect ASRs, screening for these types of responses can be very time-
consuming, especially when millions of responses are received daily.
Furthermore, the concerning situations associated with these responses can be
time-sensitive. This study concerns the advancements made to our human-AI
hybrid system in which the same AI used in ATS systems are used to prioritize
a small collection of responses for human review. By integrating AI into the
detection of ASRs, the time required to provide an appropriate response can be
significantly reduced. The program in place, and the infrastructure and
protocols around this piece of AI, aim to provide the fastest possible
response to life-threatening situations.

We typically categorize ATS into two main classes; Automated Essay Scoring
(AES) and Automated Short Answer Scoring (ASAS). The development of AES can be
traced back to the 1960s when researchers started exploring the use of
computers to analyze and evaluate natural language. The first AES system,
Project Essay Grade (PEG), was developed in 1966 by Ellis Page and used a set
of rules to analyze the structure and content of an essay [16]. The PEG system
is based on a concept known as Bag-of-Words (BoW), which uses the frequencies
of keywords in addition to hand-crafted features. A BoW model was the first
instance of ATS methods used to detect ASRs.

It is well known that frequency-based approaches, such as BoW, can be very
brittle when handling ASAS. This is why reliable systems for ASAS were
developed much later [9]. Over the years, researchers have refined the
algorithms and techniques used in AES and ASAS. With the increasing
sophistication of natural language processing (NLP) researchers began to use
machine learning techniques such as neural networks [22, 6]. In 2018, a
systematic study was conducted on the effectiveness of recurrent neural
network architectures in the detection of ASRs [14]. The study considered two
main recurrent neural network (RNN) architectures; Long-Short-Term Memory
(LSTM) networks and Gated Recurrent Units [3] with and without attention [1].
As a result of [14], the BoW model used to detect ASRs was replaced with a
two-layer bidirectional LSTM with attention.

More recently, the field has embraced the use of language models in both AES
[15, 23] and ASAS [12, 13]. It stands to reason that the detection of ASRs
could greatly benefit from language models. This article demonstrates
significant improvements from using language models to detect ASRs over
previous methods.

This article is organized into the following sections: we present the data and
give a brief overview of how the model works, and how we train it to detect
ASRs in §2. In §3 we detail the improvements over two baselines used in
previous generations of the program.

##  2\. Method

To understand the context of the model, we give a brief overview of how the
model fits into the broader system used to detect ASRs. We then consider the
data used to train and validate the model, we then consider the model itself,
why we chose it, and how it was trained.

###  2.1. The system

Given the nature of the problem, it is not reasonable to expect that a
machine-learning model is solely responsible for the detection of alarming
student responses. The model that detects alarming student responses is part
of a larger program that is built into an online assessment program. When a
student submits a response, our system divides the response into multiple
fragments, each of which is processed by the model, which classifies the
response as either a normal student response or a response that should be
routed to another system for human review. During the 2018-2019 school year,
this system processed almost 82 million fragments, which, during the peak
testing season can reach up to 7 million responses a day. This means that
efficiency is a key concern. The faster the system can process responses, the
faster the human review can be completed, and ultimately, the faster human
intervention can take place.

Category |  Details |  Examples
---|---|---
Harm to self |  Suicidal thoughts or actions |  I wanna kill myself
|  Self harming thoughts or actions |  I cut a lot
|  Eating disorder |
|  Drug Use |
Harm to another |  Threat or admission of violence |  I hit my girlfriend
|  Threat of sexual assault |  All (PC) must die
|  Threatening hate speach |  I want a sniper rifle
Harm from another |  Report of abuse |  My dad beats me
|  Report of sexual assault |  I get bullied
|  Bullying |
Severe depression and/or trauma |  Ongoing or unresolved |  Please kill me
|  |  I want to die
|  |  I wish I was dead
Specific serious request for help |  Not test related |  I hate my life, please help
|  |  help me or kill me
Table 1. A rubric for the detection of ASRs as seen in [2]. PC stands for
Protected Class.

A trained team of reviewers is responsible for reading the response and
determining whether the response is a true ASR or simply a false positive. A
full rubric for how they are assessed was featured in [2]. A summary of this
rubric is presented in Table 1. Once a response is deemed to be an ASR, the
response and any identifying information are sent via a secure platform to the
appropriate authorities.

There are some responses that may be genuine ASRs that are not correctly
classified by the model, however, depending on where and how the response was
entered, those responses are also scored by hand-scorers and are subjected to
a set of protocols in case they are alarming in nature.

###  2.2. Data

For any modeling to be effective, it is crucial that the data is carefully
considered and appropriate for the problem at hand. We use an updated version
of the corpus that was used in [14]. The system in place assesses texts from
various sources, such as short answers, essay responses, and comments made by
students. We estimate that true ASRs are approximately 0.012% of all responses
[14], which creates an imbalanced and difficult classification problem. To
address this issue, we supplement actual ASRs with responses from a diverse
set of open online forums and other texts that are similar to student
responses. By including this supplementary data, our dataset heavily
oversamples the ASRs, resulting in approximately 2.08% of all responses being
ASRs in our training set, which is still about 100 times the frequency that
they appear in production, but makes the classification problem more
manageable. An outline of this data is detailed in Table 2.

In this project, we do not adhere to standard machine learning practices for
segmenting our data into different splits because our objective is not to
enhance accuracy. Rather, in the operational setting, the engine must classify
a conservatively defined fixed percentage of the overall population for
review. In practice, our aim is to increase the number of true positives
within that fixed percentage even if it means that we have a very large number
of false positives. As a result, the model validation process is very
different. For a given response, we use the model’s output probability as a
measure of the severity of the response and establish thresholds that align
with the fixed percentages.

Category |  | ASR | Normal | Total
---|---|---|---|---
Training | Student | 20,409 | 1,214,381 | 1,234,790
| Supplementary | 5,476 | 4,122 | 9,598
| Total | 25,885 | 1,218,503 | 1,244,388
Table 2. This data represents the training data used in this study.

The model was trained using supervised learning using the labeled data. This
corpus is segmented into a training set which is 80% of the data, and a random
development set, which is 20% of the data. In addition to the training data,
we curated a collection of exactly 1 million unlabeled texts which we call the
threshold data. The threshold data is a collection of data randomly chosen and
representative of typical responses. The motivation for using the threshold
data is that we obtain the distribution of model outputs found in live
operations. We can set a cutoff point corresponding to a fixed percentage of
all responses we wish to send for review.

Our validation sample consists of a set of one thousand ASRs that have been
reviewed by human experts. Using the threshold data, an associated cutoff
value, and the percentage of responses that we can review, we aim to determine
the number of ASRs that will be correctly classified by the model as being for
review. The percentage of correctly classified ASRs from the validation sample
is our measure of the effectiveness of the model for that percentage.
Typically we want to know the effectiveness of the model for a number of
viable percentage values, which informs the decision regarding what percentage
of all responses we should review.

###  2.3. Previous Benchmarks

The Bag-of-Words (BoW) model is based on the term-frequency inverse-document-
frequency ($\mathrm{tf}$-$\mathrm{idf}$) matrix. Suppose we have a set of
training data, $D$, which contains words from a vocabulary, $V$. We can
summarize the word-frequency information in $D$ in a matrix, $X=(x_{v,d})$,
where $d\in D$ and $v\in V$. We define the term frequency matrix

(1) $\mathrm{tf}(v,d)=x_{v,d}/\left(\sum_{u}x_{u,d}\right)$

which encodes the proportions of each word in the document. The other
component is the inverse document frequency term, given by

(2) $\mathrm{idf}(v)=\mathrm{log}\dfrac{|D|}{|\\{d\in D:v\in D\\}|}.$

Together, the $\mathrm{tf}$-$\mathrm{idf}$ matrix is given by

(3) $T_{v,d}=\mathrm{tf}(v,d)\cdot\mathrm{idf}(v)$

This matrix still has far too many rows for a meaningful analysis, so what is
often done is we perform a latent semantic analysis (LSA), where a fixed
number of dominant eigenvectors of the $\mathrm{tf}$-$\mathrm{idf}$ matrix
summarizes the key features of the space. By discarding all but the fixed
number of dominant eigenvectors, we obtain a transformation from the set of
documents to some fixed dimensional vector space. The classification pipeline
that uses this vector space as the input into a traditional logistic
regression is called a Bag-of-Words classifier. What this approach operates on
is a collection of important words in the student response, which does not
capture the true semantics of the response. We trained a BoW model using 500
eigenvectors on the training data for this study.

The other classifier, a recurrent neural network, is based on the Long-Short-
Term-Memory unit [8]. Firstly, all words are mapped to a vector space via an
embedding, such as the GloVe embedding [17], and then are sequentially used as
input into the recurrent neural network. Within the neural network, then input
of a recurrent unit is a word vector and the memory state from the previous
iteration where the initial memory state is zero. The memory state within a
recurrent neural network allows for information to persist and be used in any
final classification. We use the final output of the LSTM as input into a
linear classifier, whose outputs are interpreted as log probabilities for a
classification.

a$0$${}_{a_{1}}$${}_{a_{2}}$${}_{a_{n-1}}$${}_{a_{n}}$$\ldots$$\ldots$$\ldots$x${}_{x_{1}}$${}_{x_{2}}$${}_{x_{n-1}}$${}_{x_{n}}$${}_{h_{n}}$${}_{h_{1}}$${}_{h_{2}}$${}_{h_{n-1}}$${}_{h_{n}}$
Figure 1. When we unfold an RNN, we express it as a sequence of cells each
accepting, as input, an element of the sequence. The output of the RNN is the
output of the last state.

The first modification of the basic RNN architecture is to use the states of
the RNN as input into another layer of recurrent units. This practice is known
as stacking RNNs [8]. A second modification is to segment the recurrent units
into two groups; one in which we use the sequence of inputs and the other in
which we use the reversed sequence. For obvious reasons, the resulting
structure is called a bidirectional RNN. Lastly, we can subject the outputs of
the RNN to an attention mechanism [1], which is a way we can appropriately
focus on important aspects of the input to classification and disregard those
aspects that are not as important. All these modifications to the basic RNN
structure are beneficial to many downstream tasks. An attempt to quantify the
benefit of these modifications to the detection of ASRs appears in [14] in
addition to the performance of the final RNN model we use to compare the
language model. We used the training sample to train a bidirectional two-layer
LSTM model with 512 hidden units (in each direction) with attention for this
study.

###  2.4. Modeling Details

The introduction of transformer-based language models has sparked a revolution
in natural language processing. Among the first of these models was the GPT
model [18] and the BERT model [5]. These models made waves by establishing new
state-of-the-art benchmarks on a standard set of tasks designed to push the
limits of natural language processing models [25]. The underlying premise is
that one can improve downstream tasks on supervised data, where the corpora
are limited in size, by pretraining the model on unsupervised data, where the
corpora can be as large as one needs. One uses pretraining to endow the model
with a basic understanding of language based on a large dataset of text.
Examples of datasets used are the Book Corpus [28], which is about 4.5GB,
Wikipedia dumps, which has 40GB in English alone, and the C4 dataset derived
from the common crawl, which is about 750GB. Training such models is a huge
investment of time, money, and computing power, which ultimately equates to a
sizable carbon footprint [11].

Models are typically pretrained on these huge corpora to be one of two types
of models: generative models that are trained to perform next-word prediction,
like GPT [18], or masked models that are trained to predict masked words, like
BERT [5]. There are some variations on this, such as the adding sentence
ordering to the loss function, or adversarial training mechanisms like Electra
[4] or DeBERTa [7], however, the vast majority of language models available in
standard libraries are masked-word models 111See
https://huggingface.co/docs/transformers.. These models are all variations of
the same transformer architecture described in [24]. This architecture is
presented in Figure 2.

EmbeddingPosition$+$ Multi-head
---
Attention
Add & Norm Feed
---
Forward
Add & NormEmbeddingPosition$+$ Masked
---
Multi-head
Attention
Add & Norm Multi-head
---
Attention
Add & Norm Feed
---
Forward
Add & Norm$N\times$$N\times$ Figure 2. This is the basic architecture of a
transformer-based model [24]. The left block of transformers is the encoder
while the right block of $N$ layers is the decoder.

At the heart of the transformer-based model is the idea of an attention
mechanism. From a mathematical perspective, we define a query matrix, $Q$, a
key matrix, $K$ and a value matrix, $V$, which are all normalized linear
transformations of the input matrix. Attention is defined as

(4) $Attention(Q,K,V)=\mathrm{softmax}\left(\dfrac{QK^{T}}{\sqrt{d}}\right)V,$

where $d$ is the dimension of the query and key values. The softmax component
of (4) is referred to as the attention matrix. The multi-headed attention that
features in Figure 2 operates on a decomposition of the input space into a
disjoint union of subspaces and computes attention as it applies to each
subspace [5].

Intuitively, attention and multi-headed attention are mechanisms that enable a
model to selectively focus on certain parts of the input data while
disregarding others. This mechanism is inspired by how humans selectively
attend to relevant information and ignore irrelevant information. In neural
networks, attention works by assigning weights to different parts of the input
data, which indicates how important each part is to the training task. These
weights are learned during the training process. During inference, the
attention weights are used to compute a weighted sum of the input features,
where the weights reflect the relevance of each feature for the task. This
enables the model to selectively attend to the most informative features,
which can improve its performance on the task. Self-attention is a key
component of the Transformer architecture, which allows the model to capture
long-range dependencies and contextual information within a sequence of input
tokens, such as words in a sentence. In a transformer-based language model,
multiple layers of self-attention are used on the entire sequence of input
tokens at once, rather than attending to a fixed-size window of tokens as in
traditional recurrent neural networks.

The model we use for this task is a small and very efficient language model
based on the Electra architecture [4]. The model follows a very similar
architecture to the base BERT model where the hidden size, the feed-forward
layer, and the number of attention heads are a third of those found in the
base BERT model. Secondly, the small Electra model has a much smaller
embedding facilitated by a linear layer between the embedding and the
transformer layers. The more substantial change is the adversarial training
scheme. The model is part of a pair of models, one trained to generate masked
tokens, like BERT, and the other trained to distinguish between generated and
original tokens.

In classification tasks, such as the Corpus of Linguistic Acceptability (CoLA)
[26] and the Stanford Sentiment Treebank (SST) [21], the small Electra model
delivers strong performance despite its small size. In fact, the small Electra
model has a better performance on the Kaggle Automated Student Assessment
Prize essay dataset [15, 19] and the Short Answer dataset [12, 20]. Among the
collection of all pretrained language models available using standard
libraries [27], very few models are as computationally efficient while still
delivering strong performance in classification tasks. A comparison with some
of the relevant models to this study is presented in Table 3.

| Params | CoLA | SST | MRPC | STS | QQP | MNLI | QNLI | RTE | Avg
---|---|---|---|---|---|---|---|---|---|---
BiLSTM + Attn |  | 15.7 | 85.8 | 68.5 | 59.3 | 83.5 | 74.2 | 77.2 | 51.9 | 63.9
GPT | 117M | 45.4 | 91.3 | 75.7 | 80.0 | 88.5 | 82.1 | 88.1 | 56.0 | 75.9
BERT (base) | 110M | 52.1 | 93.5 | 84.8 | 85.8 | 89.2 | 84.6 | 90.5 | 66.4 | 80.9
Electra (small) | 13M | 54.6 | 89.1 | 83.7 | 80.3 | 88.0 | 79.7 | 87.7 | 60.8 | 78.0
Table 3. Baseline performance on the GLUE task test sets. For more information
regarding these tests, we refer to [25].

After choosing a model, the next step is to apply it. Electra has input
limits, so the number of operations required to compute attention grows
quadratically with length. As a result, most models are limited to 512 tokens.
We can train the engine on segments of 256 sub-word tokens to get around this,
where ASRs are given a label of 1 and normal text is given a label of 0. In
this way, we can interpret the model’s output for a fragment as a measure of
the level of concern of the response.

Specific references to self-harm or threats made to other teachers do not
necessarily require long-term dependencies in the same way that an essay
might. This is one of the key observations regarding ASRs. We can use this to
our advantage by training the engine on segments of 256 sub-word tokens. This
is because what typically makes ASRs alarming is isolated to a few sentences.

We fine-tuned the pretrained Electra model on our labeled text using the Adam
optimizer with a weight decay mechanism [10], a learning rate of $2.5\times
10^{-5}$, a batch size of 32, and a linear learning rate scheduler. Given the
size of the data, we trained for 2 epochs on a T4 graphics card. At inference,
we divide any fragment into segments of length 256 with an overlap of 32 sub-
word tokens. The final score is the maximum model output over all the
segments. We combine this with the Open Neural Network Exchange (ONNX)
222https://github.com/onnx/onnx, which is an optimized execution platform.
With the advancements in GPU technology, and the availability of standard
libraries supported by ONNX, our system is able to classify all responses even
with heavy loads within 1-4 hours, whereas larger systems like BERT might take
approximately 3 times longer on equivalent hardware with comparable efficacy.

##  3\. Results

To evaluate the results, recall that our measure of efficacy is the percentage
of ASRs in the validation sample that would be sent for review if some fixed
percentage of all responses are sent for review. If we denote the efficacy by
$E$, then $E$ is a function of the fixed percentage, which we denote $p$. We
evaluate $E$ at values that make sense from an operational standpoint, which
range from 0.05 to 4 percent. A table of efficacy values for our latest
Electra model in addition to the bidirectional LSTM with attention and a BoW
model has been presented in Table 4. Figure 3 is a graphical representation of
the efficacy as a function of the percentage, $p$ (on a logarithmic scale).

Model | 0.05 | 0.1 | 0.3 | 0.5 | 1 | 2 | 4
---|---|---|---|---|---|---|---
BoW + LSA + Logistic Regression | 61.3 | 69.8 | 80.0 | 83.7 | 88.0 | 91.5 | 94.1
Bidirectional LSTM + Attention | 92.2 | 93.6 | 95.1 | 96.6 | 96.9 | 97.4 | 98.0
Electra small | 96.3 | 97.6 | 98.7 | 99.2 | 99.6 | 99.9 | 99.9
Table 4. The efficacy values of the three different iterations of the systems
trained for this study.

0.050.10.30.512480.084.088.092.096.0100.0

ElectraBoWLSTM

Figure 3. A graph of the approximate percentage of ASRs caught against the
logarithm of the approximate percent of responses flagged.

We see that the efficacy of the Electra model is on par at $p=0.1$ with the
efficacy of the RNN at $2\%$. This would signify a twenty-fold reduction in
the number of fragments that are required for review to catch the same number
of ASRs.

##  4\. Discussion

In this article, we have outlined an application of the advancements in
natural language processing to a real-world problem that arises in automated
scoring. Decreasing the number of fragments that are required to be reviewed
is not about decreasing costs, it is about significant reductions in the
amount of time required to review serious and critical threats to the safety
of students. Keeping the percentage low, while maintaining a certain efficacy,
and focusing on efficient models is a way of ensuring that the appropriate
authorities are made aware of these life-threatening situations in a timely
manner.

## References

  * [1] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural Machine Translation by Jointly Learning to Align and Translate, May 2016.  arXiv:1409.0473 [cs, stat].
  * [2] Amy Burkhardt, Susan Lottridge, and Sherri Woolf.  A Rubric for the Detection of Students in Crisis.  Educational Measurement: Issues and Practice, 40(2):72–80, 2021\.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/emip.12410.
  * [3] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation, September 2014.  arXiv:1406.1078 [cs, stat].
  * [4] Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning.  ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators.  Technical Report arXiv:2003.10555, arXiv, March 2020.  arXiv:2003.10555 [cs] type: article.
  * [5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  Technical Report arXiv:1810.04805, arXiv, May 2019.  arXiv:1810.04805 [cs] type: article.
  * [6] Fei Dong, Yue Zhang, and Jie Yang.  Attention-based Recurrent Convolutional Neural Network for Automatic Essay Scoring.  In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 153–162, Vancouver, Canada, August 2017. Association for Computational Linguistics.
  * [7] Pengcheng He, Jianfeng Gao, and Weizhu Chen.  DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing, December 2021.  Number: arXiv:2111.09543 arXiv:2111.09543 [cs].
  * [8] Sepp Hochreiter and Jürgen Schmidhuber.  Long Short-Term Memory.  Neural Computation, 9(8):1735–1780, November 1997.
  * [9] Claudia Leacock and Martin Chodorow.  C-rater: Automated Scoring of Short-Answer Questions.  Computers and the Humanities, 37(4):389–405, November 2003.
  * [10] Ilya Loshchilov and Frank Hutter.  Decoupled Weight Decay Regularization, January 2019.  arXiv:1711.05101 [cs, math].
  * [11] Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat.  Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model, November 2022.
  * [12] Christopher Ormerod.  Short-answer scoring with ensembles of pretrained language models, February 2022.  arXiv:2202.11558 [cs].
  * [13] Christopher Ormerod, Susan Lottridge, Amy E. Harris, Milan Patel, Paul van Wamelen, Balaji Kodeswaran, Sharon Woolf, and Mackenzie Young.  Automated Short Answer Scoring Using an Ensemble of Neural Networks and Latent Semantic Analysis Classifiers.  International Journal of Artificial Intelligence in Education, June 2022.
  * [14] Christopher M. Ormerod and Amy E. Harris.  Neural network approach to classifying alarming student responses to online assessment.  Technical Report arXiv:1809.08899, arXiv, September 2018.  arXiv:1809.08899 [cs, stat] type: article.
  * [15] Christopher M. Ormerod, Akanksha Malhotra, and Amir Jafari.  Automated essay scoring using efficient transformer-based language models, February 2021.  Number: arXiv:2102.13136 arXiv:2102.13136 [cs].
  * [16] Ellis Batten Page.  Project Essay Grade: PEG.  In Automated essay scoring: A cross-disciplinary perspective, pages 43–54. Lawrence Erlbaum Associates Publishers, Mahwah, NJ, US, 2003.
  * [17] Jeffrey Pennington, Richard Socher, and Christopher Manning.  GloVe: Global Vectors for Word Representation.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar, October 2014. Association for Computational Linguistics.
  * [18] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving Language Understanding by Generative Pre-training.
  * [19] Mark D. Shermis.  State-of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstration.  Assessing Writing, 20:53–76, April 2014.
  * [20] Mark D. Shermis.  Contrasting State-of-the-Art in the Machine Scoring of Short-Form Constructed Responses.  Educational Assessment, 20(1):46–65, January 2015.  Publisher: Routledge _eprint: https://doi.org/10.1080/10627197.2015.997617.
  * [21] Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank.  page 12.
  * [22] Kaveh Taghipour and Hwee Tou Ng.  A Neural Approach to Automated Essay Scoring.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1882–1891, Austin, Texas, November 2016. Association for Computational Linguistics.
  * [23] Masaki Uto and Yuto Uchida.  Automated Short-Answer Grading Using Deep Neural Networks and Item Response Theory.  AIED, 2020.
  * [24] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is All you Need.  In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [25] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.  Technical Report arXiv:1804.07461, arXiv, February 2019.  arXiv:1804.07461 [cs] type: article.
  * [26] Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman.  Neural Network Acceptability Judgments, October 2019.  arXiv:1805.12471 [cs].
  * [27] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.  HuggingFace’s Transformers: State-of-the-art Natural Language Processing.  Technical Report arXiv:1910.03771, arXiv, July 2020.  arXiv:1910.03771 [cs] type: article.
  * [28] Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.  Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books, June 2015.  arXiv:1506.06724 [cs].