#  AGReE: A system for generating Automated Grammar Reading Exercises

Sophia Chan
Educational Testing Service Canada
<EMAIL_ADDRESS>
Swapna Somasundaran
Educational Testing Service
<EMAIL_ADDRESS>
Debanjan Ghosh
Educational Testing Service
<EMAIL_ADDRESS>
Mengxuan Zhao
Educational Testing Service Canada
<EMAIL_ADDRESS>
###### Abstract

We describe the AGReE system, which takes user-submitted passages as input and
automatically generates grammar practice exercises that can be completed while
reading. Multiple-choice practice items are generated for a variety of
different grammar constructs: punctuation, articles, conjunctions, pronouns,
prepositions, verbs, and nouns. We also conducted a large-scale human
evaluation with around 4,500 multiple-choice practice items. We notice for 95%
of items, a majority of raters out of five were able to identify the correct
answer and for 85% of cases, raters agree that there is only one correct
answer among the choices. Finally, the error analysis shows that raters made
the most mistakes for punctuation and conjunctions.

##  1 Introduction

Acquiring a language necessitates learning its grammar. In the United States,
the Common Core standards for K-12 English literacy reflect this by including
grammar as a learning outcome across all grade
levels.111http://www.corestandards.org/ELA-Literacy While both students and
educators acknowledge that learning grammar is “necessary and effective” for
language acquisition, it is “not something they enjoy doing” (Jean and Simard,
2011). Given the sheer amount of grammar constructs and rules, creating
exercises that target each of these rules can be tedious. Likewise, for
students, completing these exercises can become repetitive. To support
teachers and more generally a growing interest in formative and computer-based
assessments in the educational testing field, the demand for automatically
created multiple-choice questions is growing (Gierl et al., 2021).

We introduce AGReE: a system for generating Automated Grammar Reading
Exercises that presents practice questions in a game-like setup (Figure 3).
While gamification is not an explicit part of our system, AGReE offers
immediate feedback, similar to a more formal incentive system that can boost
user engagement and motivation by rewarding correct answers (Plass et al.,
2015). In related research Yip and Kwan (2006); Hung et al. (2018) find that
games which focus on drilling and practice have a positive effect on language
acquisition. The grammar questions are embedded within a text in the form of
masked sentences, and users are encouraged to solve each question to uncover
the whole text. Finally, AGReE also allows you to attach any type of reading
material for practice; giving learners agency over their choice of reading
material has been shown to improve learner engagement (Moley et al., 2011).

Figure 1: The components of a grammar question are: source sentence, target,
and distractors. The contextual log probability score from BERT is shown to
the right of each choice. In the actual task the target word is replaced by a
blank.

Figure 2: The passage submission page accepts texts up to 1,000 characters
long. Since AGReE designs to generate items for only one construct per
sentence, learners can re-order the grammar constructs as per their choice
above the text box by dragging and dropping the numbered boxes.

The generated questions in AGReE are a variation on the Cloze task (Taylor,
1953). In this task, part of a sentence is removed and replaced with a blank
and the goal is to recover the missing portion. While originally introduced to
evaluate a text’s readability, it has since been widely adapted for language
practice and assessment. Figure 1 illustrates an example of a practice item in
AGReE. In the running example in the paper, we use the source sentence “There
were …in the Americas”. The sentence contains the correct target word
“anywhere” which has been replaced with a blank in the Cloze task. We use the
pretrained language model BERT Devlin et al. (2018) to generate the options
for the blank via masked language modeling. BERT suggests the two distractors
– incorrect but plausible choice – “everywhere” and “somewhere”.

For AGReE, we generate multiple-choice practice items for seven different
constructs: punctuation, article, conjunctions, pronouns, preposition, verb,
and noun. Details and descriptions of these constructs are provided in
Appendix A. Given a sentence, we identify the token corresponding to the above
constructs and mask it from the sentence to form the question item. Next, we
use BERT to identify the distractor choices (Section 3.5). Note, we did not
use any custom tokenizer here but instead used the default tokenizer from
BERT.

After we design the grammar practice tool AGReE, we conducted a thorough user-
study via the crowdsourcing platform Amazon Mechanical Turk (MTurk). We ask
the crowd-raters to attempt around 4,500 grammar items and then asked them to
provide feedback on the quality of the items (see Section 4). We notice that
in 95% of the items, a majority of crowd-raters were able to identify the
correct answer whereas around 85% of responses agree that the items have only
one correct answer. On the contrary, in our error analysis we found that
crowd-raters often made mistakes for punctuation and conjunction items showing
these two constructs are harder than the rest. The grammar practice tool AGReE
is available online for practice and learning.222https://grammarcloze.nlplab-
dev.c.ets.org

##  2 Overview

Figure 3: After a user clicks on a masked sentence on the left side, a grammar
question is revealed to the right of the window. The masked sentences are
color-coded depending on the grammar constructs, and the short name for each
construct can be found in the banner at the top of the page. See Appendix A
for the full names of the grammar constructs, which also appear under the
banner when hovered over. The passage shown is an excerpt from the TASA corpus
we describe in Section 2.

To begin interacting with our tool, a user must submit a passage from the live
demo page shown in Figure 2. The available grammar constructs are shown above
the free text box, and can be reordered in terms of priority from left to
right. This enables the teacher or student to customize the experience based
on learning goals. After the passage is submitted, the text is sent to a back-
end implemented using Amazon Web Services (AWS) components.

After submission, the system will create grammar questions from the passage
(see Figure 3). Grammar item generation happens at the sentence level. Since
completing a Cloze task relies on filling in a blank based on the surrounding
context (Taylor, 1953), we create an item out of every other sentence so that
the user is provided with both enough context to fill in the blank, and with
enough practice as they progress through their reading. Only one grammar item
is generated per sentence, and this item will correspond to the first grammar
construct that can be found in the sentence that our distractor generation
algorithm successfully generates an item for.

Next, the grammar items will appear to the user in the form of masked
sentences. When a sentence is masked, it becomes clickable to the user. And
once clicked on, the question choices will appear on the right side of the
window. If the student selects an incorrect answer they are prompted to try
again. But when the correct answer is selected the sentence becomes unmasked
so that they may proceed with their reading. For each correct answer AGReE
provides feedback to encourage and motivate the learner.

The system has been tested thoroughly on informational texts such as the
Touchstone Applied Science Associates (TASA) corpus which “consists of
representative random samples of text of all kinds read by students in each
grade through first year of college” Zeno et al. (1995); Landauer et al.
(1998). Note that the system itself does not have any dependency to a
particular corpus, and so in theory can be used with texts from any domain.

##  3 Grammar item generation

Creating a grammar item from a sentence involves the following steps:

  * 1.

Token matching

  * 2.

Syntactic pattern matching

  * 3.

Sentence validation

  * 4.

Sentence filtering

  * 5.

Distractor generation

We use the following sentence “There were few people anywhere in the world,
and none lived in the Americas” as our running example to show how to create
the grammar item (Figure 1).

###  3.1 Token matching

Let’s say the first grammar construct on the priority list is indefinite
pronoun. First, we determine whether the sentence contains a token in our pre-
defined list of indefinite pronouns: everybody, everywhere, everything,
somebody, somewhere, something, anybody, anywhere, anything, nobody, nowhere,
nothing.333https://www.gingersoftware.com/content/grammar-rules

###  3.2 Syntactic pattern matching

The system finds anywhere, and this prompts a check to make sure the syntactic
properties of the token match those of the grammar construct. Each grammar
construct has a pattern defined using dependency parse tags and part of speech
tags from spaCy.444https://spacy.io

###  3.3 Sentence validation

The syntactic properties of anywhere matches the pattern defined for the
indefinite pronoun construct, so at this point we have identified the grammar
construct in the sentence and know its location. The token anywhere is
replaced with a [MASK] token, and we use the pre-trained language model BERT
(Devlin et al., 2018) (as implemented in the Hugging Face (Wolf et al., 2019)
repository) to predict the most likely substitution for [MASK]. We currently
only handle words that are common enough to exist in the vocabulary of the
BERT tokenizer; our closed classes of distractor choices can all be found, but
we may skip over an open class word if it turns out that the noun or verb is
tokenized into separate wordpieces. The other tokens from the set of
indefinite pronouns (Section 3.1) become distractor candidates that feed into
our distractor generation step in Section 3.5.

###  3.4 Sentence filtering

If the word that was originally in the sentence (here, anywhere) does not rank
the highest in terms of probability score among the construct type (here,
indefinite pronouns) as predicted by BERT in the previous step, it indicates
that there is ambiguity in the correct answer to fill the [MASK] location
with. In other words, multiple indefinite pronouns may suit the context. In
our example anywhere ranks the highest among the set of the indefinite
pronouns at the [MASK] location, so we continue onto the next step for item
generation. Otherwise, we discard the item from further processing.

###  3.5 Distractor generation

It is important for a Cloze task that the distractors are plausible yet
incorrect. Gao et al. (2020) argued that the BERT-based predictions for a
masked word fits the requirements of a Cloze task perfectly. In other words,
the most likely substitutions for a [MASK] token can be used as distractors.
Thus, similar to Gao et al. (2020), based on the probability scores of the
indefinite pronouns, we select the highest ranking candidates (except the
token anywhere) as the distractor.

For nouns and verbs, the process remains the same except parts-of-speech tags
are used in the place of a pre-defined list of tokens. We use the lemminflect
package to create the list of possible candidates by inflecting the stem
word.555https://lemminflect.readthedocs.io

For comma items, the candidates are all possible relocations of the comma. We
create the distractor candidates by inserting a [MASK] token in-between every
word.

##  4 Evaluation

Figure 4: The distribution of grammar constructs in generated questions
presented to Turkers (N=4,568). Figure 5: The distribution of grammar
constructs in generated questions with two or fewer correct answers (N=455).

We are interested in evaluating the quality of the multiple-choice items in
AGReE. To that end, we conduct a large-scale MTurk study where we present
crowd-raters with our generated grammar exercises as shown in Figure 3. We ask
the raters to select their choice for each item, and to answer followup
questions along four aspects of item quality. We paid raters $1.70 USD for
completing our prerequisite task and $2.00 USD for completing our main task,
which is approximately equivalent to $20-$24 USD (per hour). In total, the
evaluation cost $12,575 USD.

###  4.1 Prerequisite task

We conducted a set of prerequisite tasks to select the raters for our main
task on AGReE. A rater must have at least 10,000 approved HITs, $\geq$ 95% of
HIT approval, a Masters qualification, and reside in the United States. We
also ask the raters to answer a set of 20 publicly available TOEFL Junior
reading comprehension questions (typically used for assessing English language
skills of students 11 or older) so that we can gauge raters’ English
proficiency.666https://www.ets.org/toefl_junior/prepare/standard_sample_questions/reading_comprehension

The mean score on this set of reading comprehension questions among 79 raters
is 96%, and 95% of raters received a score of 90% or higher, meaning they got
at most two questions out of 20 incorrect. Responses from this high-performing
group make up 99.8% of our responses. Given that the raters have some
competence in the language, we expect that they will be able to identify the
correct answer among the choices. If they are unable to do so, it will provide
a strong indication that our generated items are incorrectly keyed.

###  4.2 Main task

Raters are presented with two passages containing grammar questions from
AGReE. The setup is identical to how we present the questions on the front-end
(Figure 3), and the method for generating grammar items is the same as
described previously (Section 3). Each passage is around 10 sentences long,
taken from the beginning of either a TASA (Zeno et al., 1995) or a TIPSTER
(Harman, 1993) document. Altogether 4,568 grammar constructs items were
presented to the raters. Five different raters responded to each item to
select the correct answer. The distribution of the items is shown in Figure 4.

Among the 4,568 items that we generated, for 76% of items all five raters
answered correctly. If we consider items where the majority of raters answered
correctly (i.e, three or more selected the target), then the proportion
increases to 95%. This is an indication that nearly all our items are
correctly keyed. On the contrary, there were 455 instances where only a
minority of raters chose the correct answer. Among these items, the article,
verb, and noun items are less frequent whereas the punctuation and
conjunctions items are the most frequent (see Figure 5).

To shed some light on why raters made mistakes on certain items, we randomly
select one example each of punctuation and conjunctions items where no raters
answered correctly. Note that we present the choices here in order of their
ranking from BERT (the log probability is in parentheses), but when presented
to raters the choices are randomly shuffled. Here is an example of a
problematic punctuation (comma) item:

> source sentence: With so many children in the family___ there___ was a
> constant___ buzz of activity
> target: … family, … (12.76)
> distractor: … constant, … (7.17)
> distractor: … there, … (6.07)

And here is an example of a problematic conjunctions (coordinating
conjunction) item:

> source sentence: Many of these people did not go to the theater, of course,
> ___ to keep playgoers happy, acting troupes had to provide a variety of
> plays.
> target: but (13.93)
> distractor: and (13.39)
> distractor: so (12.18)

In both items, all five raters selected the highest-ranking distractor as
their answer. This supports the idea that the contextual probabilities are
useful predictors of distractor-context fit. While the placement of the comma
in the target is the only proper usage in the punctuation example, one might
argue that both but and and are grammatical in the conjunctions source
sentence. However, the use of the connective but more accurately describes the
relationship between the first and second clause. Given that the probability
of the distractor and target are so close in the conjunctions example, we
suspect that the distance between the probabilities provides some signal about
item difficulty, but future work is needed to investigate and calibrate the
difficulty of the grammar items.

][t].235

(a) I felt that there was more than one correct answer.

][t].195

(b) I felt that some response options were too obviously wrong.

][t].195

(c) The question was likely created by a teacher.

97pt][t].3

(d) The question was easy.

Figure 6: Bar charts displaying the distribution of responses for each item
quality aspect over the five Likert scale options (N=22,840).

The purpose of our MTurk experiment is not only to measure how many items are
solvable by the raters, but also to know specific aspects about them. Such as
whether (1) the items are correctly keyed (i.e., we have accurately and
unambiguously identified the target) (2) the item contains nonfunctional
distractors (Gierl et al., 2017), (3) the item can be distinguished from a
human-generated one, and finally, (4) the item was difficult. To answer these
four questions we also asked the same group of raters to respond a five-point
Likert scale (Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree) to
the following statements, respectively.

  1. 1.

I felt that there was more than one correct answer

  2. 2.

I felt that some response options were too obviously wrong

  3. 3.

The question was likely created by a teacher

  4. 4.

The question was easy

Figure 6 presents four bar charts displaying the distribution of the responses
to the above four questions from left to right. We find corroborating evidence
that the items are correctly keyed: 85% of responses agree that the items have
only one correct answer (Figure 6(a)). As for whether some options are too
obviously wrong, the results appear to be mixed; no clear pattern in the
responses can be observed (Figure 6(b)). We find that more often than not,
raters found our items indistinguishable from those created by teachers, but
here we also do not observe a clear pattern (Figure 6(c)). Finally, nearly all
responses say that the items are easy. Gathering data from second language
learners could help clarify whether the items are easy for English learners as
well (Figure 6(d)).

Overall, the MTurk results paint a promising picture for the utility of our
items. Since 95% of items were answered correctly by a majority of raters, and
85% agree that there was only one correct answer, we have strong indicators
that nearly all our items are correctly keyed.

##  5 Related work

Tools that enhance authentic texts in support of grammar acquisition include
WERTi (Meurers et al., 2010), FLAIR (Chinkina and Meurers, 2016), and
GrammarTagger (Hagiwara et al., 2021). Except for a verb practice activity in
WERTi, these tools provide few opportunities for immediate feedback. Our
system fills this gap by generating multiple-choice practice questions from
authentic texts.

On the other hand, there are also systems that generate multiple-choice
grammar questions. FAST (Chen et al., 2006) covers nine grammatical categories
and Lee et al. (2016) create a system for learning preposition usage. We
extend these works by not only covering a range of grammatical categories and
collecting perception responses, but also conducting a large-scale evaluation
on item performance (i.e., whether the item can be solved by the user with
some competence in the language).

##  6 Conclusion

We describe AGReE, a system and procedure for converting an informational
passage into game-like grammar practice exercises that can be completed while
reading. We find in human evaluations that nearly all the multiple-choice
questions we generate for the exercises are correctly keyed, and can therefore
be used to provide immediate feedback to students. We also observe for almost
95% of items that a majority of the raters were able to identify the correct
target. On the contrary, raters made the most mistakes for punctuation and
conjunctions. Although we did not design our system to include gamification
explicitly, it is set up in a way that makes it easy to incorporate in the
future. The system can automatically generate a wealth of questions for which
the correct answer is identified, and these questions can be used by whomever
(e.g., game designers) to create a game that rewards learners when they
correctly solve them.

##  7 Future work

AGReE is a proof of concept system we built with the intent of allowing users
to personalize grammar items according to their unique goals. The user
interface does not currently prevent learners from progressing through the
passage without providing answer for each question. In fact, it is likely that
they can still complete the reading since it is usually a weakly semantic
element missing from the sentence. A truly gamified system may choose to block
out the rest of the passage before the current question is answered, or make
it so that the next question cannot be clicked on until the current question
is answered.

As it stands, the system allows teachers some control over the generated
items; the reading material itself and the ranking of grammar constructs can
be customized. However, teachers do not have control at the level of
individual items. We may want to build in this finer-grained control in the
future so that the exercises can be adapted more closely to their needs.
Currently, the number of questions generated for a given construct (e.g.,
preposition) depends on that construct’s ranking, its frequency in the text,
and how well it is covered in the manually created list of distractor choices.
For example, there may be fewer preposition items generated than expected due
to the fact that we use a reduced set of prepositions in AGReE. If we expand
the set of distractor choices to cover all possible prepositions, we would
likely run into latency issues.

While increasing the coverage of existing constructs is one potential line of
future work, it may be more important to find ways to align our constructs to
existing EFL (English as a Foreign Language) curricula if we want to create
efficacious questions. Since the constructs that a learner struggles with is
influenced by—among other factors—aspects of their learner profile such as
English level (Hawkins and Buttery, 2010) and language background, an
efficacious system should take these aspects into account. Doing so may allow
AGReE to provide more personalized feedback and generate distractors that can
be calibrated. For example, we can imagine using token-level probabilities to
make filtering decisions about whether an item is appropriate for a certain
language level. The distance between the target log probability and the log
probability of the hardest distractor can be smaller for a learner whose
language level is higher, assuming that as language ability improves, so does
the ability to discriminate between a correct choice and a plausible yet
incorrect one.

This raises the question about the threshold at which a token can be
considered grammatical versus not, as illustrated in the coordinating
conjunction example in Section 4.2. According to Larsen-Freeman (2001), one
way to think about grammar is to see it as an interaction between
form/structure, meaning/semantics, and use/pragmatics. Schneider and Gilquin
(2016) also argue that, when it comes to learner English, there is no “clear
dichotomy between innovation and error”. Following this line of thinking, we
view the threshold for grammaticality as context-specific and tied to
pedagogical goals.

More experimentation is also needed to determine how the quality of the
questions changes when individual components are altered or replaced. To
increase the flexibility of the system, we can think of replacing the token
matching and syntactic pattern matching components with a more specialized
model that identifies the gap locations, such as the one described in Felice
et al. (2022). As such, making the system more functional would involve
building in the ability to evaluate the output of individual components in
addition to the final output.

## References

  * Chen et al. (2006) Chia-Yin Chen, Hsien-Chin Liou, and Jason S Chang. 2006.  Fast–an automatic generation system for grammar tests.  In _Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions_ , pages 1–4.
  * Chinkina and Meurers (2016) Maria Chinkina and Detmar Meurers. 2016.  Linguistically aware information retrieval: Providing input enrichment for second language learners.  In _Proceedings of the 11th Workshop on Innovative use of NLP for Building Educational Applications_ , pages 188–198.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Felice et al. (2022) Mariano Felice, Shiva Taslimipoor, and Paula Buttery. 2022.  Constructing open cloze tests using generation and discrimination capabilities of transformers.  _arXiv preprint arXiv:2204.07237_.
  * Gao et al. (2020) Lingyu Gao, Kevin Gimpel, and Arnar Jensson. 2020.  Distractor analysis and selection for multiple-choice cloze questions for second-language learners.  In _Proceedings of the 15th Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 102–114.
  * Gierl et al. (2017) Mark J Gierl, Okan Bulut, Qi Guo, and Xinxin Zhang. 2017.  Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review.  _Review of Educational Research_ , 87(6):1082–1116.
  * Gierl et al. (2021) Mark J Gierl, Hollis Lai, and Vasily Tanygin. 2021.  _Advanced Methods in Automatic Item Generation_.  Routledge.
  * Hagiwara et al. (2021) Masato Hagiwara, Joshua Tanner, and Keisuke Sakaguchi. 2021.  Grammartagger: A multilingual, minimally-supervised grammar profiler for language education.  _arXiv preprint arXiv:2104.03190_.
  * Harman (1993) Donna Harman. 1993.  Overview of the first trec conference.  In _Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 36–47.
  * Hawkins and Buttery (2010) John A Hawkins and Paula Buttery. 2010.  Criterial features in learner corpora: Theory and illustrations.  _English Profile Journal_ , 1.
  * Hung et al. (2018) Hsiu-Ting Hung, Jie Chi Yang, Gwo-Jen Hwang, Hui-Chun Chu, and Chun-Chieh Wang. 2018\.  A scoping review of research on digital game-based language learning.  _Computers & Education_, 126:89–104.
  * Jean and Simard (2011) Gladys Jean and Daphnée Simard. 2011.  Grammar teaching and learning in l2: Necessary, but boring?  _Foreign Language Annals_ , 44(3):467–494.
  * Landauer et al. (1998) Thomas K Landauer, Peter W Foltz, and Darrell Laham. 1998.  An introduction to latent semantic analysis.  _Discourse Processes_ , 25(2-3):259–284.
  * Larsen-Freeman (2001) Diane Larsen-Freeman. 2001.  Teaching grammar.  _Teaching English as a Second or Foreign Language_ , 3:251–266.
  * Lee et al. (2016) John SY Lee, Donald Sturgeon, and Mengqi Luo. 2016.  A call system for learning preposition usage.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 984–993.
  * Meurers et al. (2010) Detmar Meurers, Ramon Ziai, Luiz Amaral, Adriane Boyd, Aleksandar Dimitrov, Vanessa Metcalf, and Niels Ott. 2010.  Enhancing authentic web pages for language learners.  In _Proceedings of the Fifth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 10–18.
  * Moley et al. (2011) Pauline F Moley, Patricia E Bandré, and John E George. 2011.  Moving beyond readability: Considering choice, motivation and learner engagement.  _Theory into Practice_ , 50(3):247–253.
  * Plass et al. (2015) Jan L Plass, Bruce D Homer, and Charles K Kinzer. 2015.  Foundations of game-based learning.  _Educational Psychologist_ , 50(4):258–283.
  * Schneider and Gilquin (2016) Gerold Schneider and Gaëtanelle Gilquin. 2016.  Detecting innovations in a parsed corpus of learner english.  _International Journal of Learner Corpus Research_ , 2(2):177–204.
  * Taylor (1953) Wilson L Taylor. 1953.  Cloze procedure: A new tool for measuring readability.  _Journalism Quarterly_ , 30(4):415–433.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Yip and Kwan (2006) Florence WM Yip and Alvin CM Kwan. 2006.  Online vocabulary games as a tool for teaching and learning english vocabulary.  _Educational Media International_ , 43(3):233–249.
  * Zeno et al. (1995) Susan Zeno, Stephen H Ivens, Robert T Millard, and Raj Duvvuri. 1995.  _The Educator’s Word Frequency Guide_.  Touchstone Applied Science Associates.

##  Appendix A Grammar constructs

Construct |  |  | Distractor candidates
---|---|---|---
Punctuation | Comma location | CMA | All possible relocations
Punctuation | PCT | : : ,
Article |  | ART | the, a, an
Conjunctions | Coordinating conjunction | COO | for, nor, but, or, yet, so, and
Subordinating conjunction | SUB | after, although, because, before, if, once, since,
|  | than, unless, until, when, whenever, while, as
Correlative conjunction | COR | either/or, neither/nor, both/and, as/so,
|  | whether/or
Pronouns | Indefinite pronoun | IDP | everybody, everywhere, everything, somebody,
|  | somewhere, somewhere, something, anybody,
|  | anywhere, anything, nobody, nowhere, nothing
Interrogative pronoun | ITP | who, which, what, whose, whom
Possessive pronoun | POS | my, mine, your, yours, our, ours, their, theirs
Reflexive pronoun | REL | myself, yourself, herself, himself, itself,
|  | yourselves, ourselves, themselves
Preposition |  | PRP | to, toward, on, onto, in, into
Noun |  | NOU | NN, NNS
Verb |  | VRB | VB, VBD, VBG, VBN, VBP, VBZ
Table 1: The available grammar constructs and their distractor candidates. All
the possible tokens are enumerated except for comma location where the
candidates are all possible relocations of the comma, and noun and verb items
where candidates are inflections of the word stem.

Each grammar construct and its sub-constructs are listed in Table 1, along
with the set of distractor candidates used for the token matching step from
Section 3.1 and distractor candidate generation step from Section 3.5.


0.2pt 55 0.2pt 55

# QMUL-SDS @ DIACR-Ita: Evaluating Unsupervised Diachronic Lexical Semantics
Classification in Italian

Rabab Alkhalifa Queen Mary University of London, United Kingdom  Imam
Abdulrahman bin Faisal University, Saudi Arabia  Adam Tsakalidis Queen Mary
University of London, United Kingdom  Alan Turing Institute, United Kingdom
Arkaitz Zubiaga Queen Mary University of London, United Kingdom  Maria Liakata
Queen Mary University of London, United Kingdom  Alan Turing Institute, United
Kingdom

###### Abstract

In this paper, we present the results and main findings of our system for the
DIACR-Ita 2020 Task. Our system focuses on using variations of training sets
and different semantic detection methods. The task involves training, aligning
and predicting a word’s vector change from two diachronic Italian corpora. We
demonstrate that using Temporal Word Embeddings with a Compass C-BOW model is
more effective compared to different approaches including Logistic Regression
and a Feed Forward Neural Network using accuracy. Our model ranked 3rd with an
accuracy of 83.3%.

††footnotetext: Copyright © 2020 for this paper by its authors. Use permitted
under Creative Commons License Attribution 4.0 International (CC BY 4.0).

##  1 Introduction

The quantitative analysis of language evolution over time is a new emerging
research area within the domain of Natural Language Processing [Turney and
Pantel, 2010, Hamilton et al., 2016, Dubossarsky et al., 2017]. The study of
Diachronic Lexical Semantics [Tahmasebi et al., 2018, Kutuzov et al., 2018],
which contributes towards detecting word-level language evolution, brings
together researchers with broadly varying backgrounds from computational
linguistics, cognitive science, statistics, mathematics, and historical
linguistics, since the identification of words whose lexical semantics have
changed over time has numerous downstream applications in various domains such
as historical linguistics and NLP. Despite the increase in research interest,
few tasks that track word meaning change over time have focused on non-English
languages, while the comparison of different approaches in the same
experimental and evaluation setting is still limited [Schlechtweg et al.,
2020]. The DIACR-Ita 2020 Task [Basile et al., 2020] aims to fill these gaps
by focusing on the Italian language used during two different time periods and
providing a single evaluation framework to researchers for testing their
methods.

This work presents our approach towards detecting Italian words with altered
lexical semantics during the two distinct time periods studied in the DIACR-
Ita 2020 Shared Task. Our contribution focuses on evaluating findings from
previous studies, exploring evaluation approaches for different methods and
comparing their performance. We contrast several variants of training-testing
words with different alignment approaches across two word embedding models,
namely Skip-gram and Continuous Bag-of-Words [Mikolov et al., 2013]. Our
submission consisted of four models that showed the best average cosine
similarity, calculated on the basis of their ability to accurately reconstruct
the representations of Italian stop-words across the two periods of time under
study. Our best performing model uses a Continuous Bag-of-Words temporal
compass model, adapted from the model introduced by [Carlo et al., 2019]. Our
system ranked third in the task.

##  2 Related Work

Work related to unsupervised diachronic lexical semantics detection can be
divided into different approaches depending on the type of word
representations used in a diachronic model (e.g., based on graphs or
probability distributions [Frermann and Lapata, 2016, Azarbonyad et al.,
2017], temporal dimensions [Basile and McGillivray, 2018], frequencies or co-
occurrence matrices [Sagi et al., 2009, Cook and Stevenson, 2010], neural- or
Transformer-based [Hamilton et al., 2016, Boleda et al., 2019, Shoemark et
al., 2019, Schlechtweg et al., 2019, Giulianelli et al., 2020], etc.). In our
work, we focus on dense word representations [Mikolov et al., 2013], due to
their high effectiveness that has been demonstrated in prior work.

Systems operating on representations such as those derived from Skip-gram or
Continuous Bag-of-Words leverage in most cases deterministic approaches using
mathematical matrix transformations [Hamilton et al., 2016, Azarbonyad et al.,
2017, Tsakalidis et al., 2019], such as Orthogonal Procrustes [Schönemann,
1966], or machine learning models [Tsakalidis and Liakata, 2020]. The goal of
these approaches is to learn a mapping between the word vectors that have been
trained independently by leveraging textual information from two or more
different periods of time. The common standard for measuring the level of
diachronic semantic change of a word under this setting is to use a similarity
measure (e.g., cosine distance) on the aligned space – i.e., after the mapping
step is complete [Turney and Pantel, 2010].

[Dubossarsky et al., 2017] argue that using cosine distance introduces bias in
the system triggered by word frequency variations. [Tan et al., 2015] only use
the vectors of the top frequent terms to find the transformation matrix, and
then they calculate the similarity for the remaining terms after applying the
transformation to the source matrix. Incremental update [Kim et al., 2014,
Boleda et al., 2019] used the intersection of words between datasets in each
time frame by initializing the word embedding from the previous time slice to
compare the word shift cross different years instead of using matrix
transformation. Temporal Word Embeddings with a Compass (TWEC) [Carlo et al.,
2019] approach uses an approach of freezing selected vectors based on model’s
architecture, it learn a parallel embedding for all time periods from a base
embedding frozen vectors.

Our approaches, detailed in Section 4, follow and compare different
methodologies from prior work based on (a) Orthogonal Procrustes alignment,
(b) machine learning models and (c) aligned word embeddings across different
time periods.

##  3 Task Description

The task was introduced by [Cignarella et al., 2020] and is defined as
follows:

Given two diachronic textual data, an unsupervised diachronic lexical
semantics classifier should be able to find the optimal mapping to compare the
diachronic textual data and classify a set of test words to one of two
classes: 0 for stable words and 1 for words whose meaning has shifted.

We were provided with the two corpora in the Italian language, each from a
different time period, and we developed several methods in order to classify a
word in the given test set as “semantically shifted” or “stable” across the
two time periods. The test set included 18 observed words – 12 stable and 6
semantically shifted examples.

##  4 Our Approach

Here we outline our approaches for detecting words whose lexical semantics
have changed.

###  4.1 Generating Word Vectors

Word representations $W_{i}$ at the period $T_{i}$ were generated in two ways:

(a) IND: via Continuous Bag of Words (CBOW) and Skip-gram (SG) [Mikolov et
al., 2013] applied to each year independently;

(b) CMPS: via the Temporal Word Embeddings with a Compass (TWEC) approach
[Carlo et al., 2019], where a single model (CBOW or SG) is first trained over
the merged corpus; then, SG (or CBOW) is applied on the representations of
each year independently, by initialising and freezing the weights of the model
based on the output of the first base model pass and learning only the
contextual part of the representations for that year.

In both cases, we used gensim with default
settings.111https://radimrehurek.com/gensim/ Sentences were tokenised using
the simple split function for flattened sentences provided by the organisers,
without any further pre-processing. Although there are many approaches to
generate word representations (e.g., using syntactic rules), we focused on
1-gram representations using CBOW and SG, without considering words lemmas and
Part-of-Speech tags.

###  4.2 Measuring Semantic Change

We employ the cosine similarity for measuring the level of semantic change of
a word. Given two word vectors $w^{T0}$, $w^{T1}$, semantic change between
them is defined as follows:

$\cos(w^{T0},w^{T1})=\frac{w^{T0}\cdot
w^{T1}}{\|w^{T0}\|\|w^{T1}\|}=\frac{\sum_{i=1}w^{T0}_{i}w^{T1}_{i}}{\sqrt{\sum_{i=1}{w^{T0}_{i}}^{2}}\sqrt{\sum_{i=1}{w^{T1}_{i}}^{2}}}$
(1)

Though alternative methods have been introduced in the literature (e.g.,
neighboring by pivoting the top five similar words [Azarbonyad et al., 2017]),
we opted for the similarity metric which is most widely used in related work
[Hamilton et al., 2016, Shoemark et al., 2019, Tsakalidis et al., 2019].

###  4.3 Evaluation Sets

The challenge is expecting the lexical change detection to be done in an
unsupervised fashion (i.e., no word labels have been provided). Thus, we
considered stop words222https://github.com/stopwords-iso/stopwords-it (SW) and
all of the other common words (CW) in $T_{0}$ and $T_{1}$ as our training and
evaluation sets interchangeably.

###  4.4 Semantic Change Detection Methods

We employed the following approaches for detecting words whose lexical
semantics have changed:

(a) Orthogonal Procrustes (OP): Due to the stochastic nature of CBOW/SG, the
resulting word vectors $W_{0}$ and $W_{1}$ in IND were not aligned. Orthogonal
Procrustes [Hamilton et al., 2016] tackles this issue by aligning $W_{1}$
based on $W_{0}$. The level of semantic shift of a word is calculated by
measuring the cosine similarity between the aligned vectors. For evaluation
purposes, we measured the cosine similarity of the stop words between the two
aligned matrices. Higher values indicate a better model (i.e., stop words
retain their meaning over time).

(b) Feed-Forward Neural Network (FFNN): We trained a FFNN that leverages IND
to predict $W_{1}$ based on $W_{0}$. The level of semantic shift of a word in
a test set is calculated by measuring the cosine similarity between the
predicted $W^{*}_{1}$ and $W_{1}$. For evaluation purposes, we measure the
cosine similarity between the actual and predicted representations of words in
$T_{1}$. Higher values for stop-words indicate a better model.

(c) Linear Regression (LR): We employed an ordinary linear mapping with least
square error objective function.333https://scikit-learn.org/stable/ The task
and the evaluation setting was identical to FFNN.

(d) Temporal Word Embeddings with a Compass (TWEC) [Carlo et al., 2019]:
Working on the CMPS vectors, the level of semantic shift of a word is
calculated by measuring the cosine similarity between $T_{0}$ and $T_{1}$
directly.

#### Notation

In the rest of this paper, we denote a model $M$ trained on CW (SW) as
$M\\_CW$ ($M\\_SW$). For the case of $OP$, the training process involves
learning an alignment based on a specific word set ($CW$ or $SW$). Note that
this notation does not apply for $TWEC$, since the word vectors in the two
time periods can be directly compared against each other – thus the level of
semantic change can be calculated directly (i.e., there is no need to learn
any mapping between $W_{0}$ and $W_{1}$). Finally, we add a subscript CBOW or
SG to our models, denoting the type of algorithm that was used for generating
the respective embeddings that are fed to our model.

#### Model Selection

We select to apply the models on the test set providing high average cosine
similarity with stop words.

###  4.5 Word Classification

As per the task guidelines [Cignarella et al., 2020], words can fall into one
of the two categories: 0: the target word does not change meaning between
$T_{0}$ and $T_{1}$ and 1: the target word changes its meaning between $T_{0}$
and $T_{1}$. For all of our submitted models, we considered all the words with
cosine similarity below the mean as shifted words and labelled them with 1. We
further investigate the model’s ability to detect words laying two standard
deviations below the mean ($\mu-2\sigma$), a.k.a variance. Interestingly, some
of the models including LR and FFNN_CWCBOW showed an increase in accuracy.

##  5 Results

IND SG C-BOW Accuracy Ranking Accuracy Ranking train. M $CS_{avg}^{SW}$ %$\mu$
%$\mu-2\sigma$ %$\mu_{rank}$ $R_{p50}$ $R_{\downarrow 6}$ $CS_{avg}^{SW}$
%$\mu$ %$\mu-2\sigma$ %$\mu_{rank}$ $R_{p50}$ $R_{\downarrow 6}$ SW OP 0.748
0.778 0.667 0.222 1.000 0.667 0.784 0.778 0.667 0.270 1.000 0.833 LR 0.854
0.333 0.389 0.373 0.833 0.500 0.795 0.500 0.778 0.278 0.833 0.500 FFNN 0.769
0.333 0.333 0.373 0.833 0.500 0.709 0.556 0.722 0.341 0.833 0.500 CW OP 0.464
0.389 0.778 0.381 0.667 0.500 0.289 0.611 0.667 0.397 0.833 0.333 LR 0.409
0.333 0.444 0.508 0.500 0.333 0.146 0.333 0.444 0.381 0.667 0.667 FFNN 0.658
0.333 0.389 0.317 1.000 0.500 0.621 0.333 0.722 0.317 0.833 0.500 TWEC 0.722
0.722 0.667 0.317 0.833 0.667 0.833 0.833∗ 0.667 0.286 1.000 0.667

Table 1: Performance of our models using different evaluations methods. (∗)
best submission. (a)  Figure 1: Frequency of stop words by their cosine
similarity scores, where each subfigure pertains to a different model.

The results are shown in Table 1, where we split our results based on model #M
architecture, SG and CBOW and model’s training word sets, Stop-Words (SW) and
Common-Words (CW). For models based on linear transformation, our top
performing models scored below average cosine similarity, TWECCBOW ($0.833$),
OP_SWSG($0.778$), OP_SWCBOW ($0.778$), TWECSG($0.722$). As shown in Figure 1a,
we observe that these models tend to have skewed distributions for stop words,
where the vast majority of stop words are assigned high cosine similarity
scores. However, other models did not show this skewness, e.g.
OP_CWSG($0.389$) and OP_CWCBOW($0.611$). When labeling the change based on
variance ($\mu-2\sigma$), as in outlier detection, some models showed an
increase from the dummy classifier’s performance. For instance, OP_CWsg showed
an increase on performance from ($0.389$) to ($0.778$) showing that those with
low average cosine similarity lay out in the tail from majority similarity.
Similarly, models based on reducing the similarity error between the predicted
and actual vectors, e.g. LR and FFNN considering the outlier detection
methodology, tend to achieve better performance, including LR_SWCBOW,
FFNN_SWCBOW and FFNN_CWCBOW where LR_SWCBOW showed an increase from frequency
classifier’s baseline ($0.500$) to ($0.778$), and LR_SWCBOW showed an increase
from dummy classifier performance ($0.333$) to ($0.722$).

Ranking methods, average ranking ($\mu_{rank}$) and Recall ($R$), expect prior
knowledge about the evaluation labels to make them useful for evaluating the
reliability of the model of interest. For that, we further investigate the
reliability of our experiment models, using $\mu\\_{rank}$ and $R$ at $\%50$
(Rp50) and $\%30$ (R↓6). Although using (Rp50) signal OP_SWSG, OP_SWCBOW,
FFNN_CWSG, TWECCBOW as equalliy good, $\mu_{rank}$ ranked top models as
OP_SWSG, OP_SWCBOW, LR_SWCBOW then TWECCBOW with (0.222, 0.270, 0.278 and
0.286), respectively. Additionally, under extreme conditions, OP_SWCBOW ranked
better than all including TWECCBOW. This shows that under extreme conditions,
a good method is the one which keeps providing out of distribution signals to
changing words and that needs to take a careful consideration about the
distribution of the words before and after the alignments as in OP. In
general, CBoW-based models showed better performance than SG-based models with
average accuracy of (%$\mu$ 0.564 and %$\mu-2\sigma$ 0.667) compared to
(%$\mu$ 0.460 and $\mu-2\sigma$ 0.524) for words labelled by mean and
variance, respectively. Further, alignment using non-changing words (e.g.
stop-words) yields higher performance than using all common words with average
cosine similarity for stop words as ($CS_{avg}^{SW}$ 0.777) compared to
($CS_{avg}^{SW}$ 0.431), which is expected because SW-based models learns the
optimal mapping with less noise than CW-based models.

##  6 Discussion

Our work provides a comprehensive analysis for Italian lexical diachronic
methods introduced from previous work. For models that are based on matrix
linear transformation including TWEC and OP, we find a relation between high
average stop words similarity and accuracy. Further, C-BOW tends to achieve
better results than the SG architecture for most experiments. Visually, we
find that a visibly skewed distribution showing the tendency of stop words to
have high cosine similarity scores leads to effective means for capturing
semantic shift. We also showed that by evaluating the models using different
methods, TWECCBOW achieved top performance. Followed by OP_SW and OP_CWSG, and
LR using outlier detection methodology. Further, FFNN showed high recall
(Rp50) by ranking changed words with lowest cosine similarity on testing set
similar to OP_SW and TWECCBOW. This provides promising insights encouraging
further investigation of neural network models using different languages and
larger datasets.

##  7 Conclusions

In this report, we describe and compare our models submitted to the DIACR-Ita
2020 shared task, which assessed the ability to classify semantic-shift of
words in Italian. We show that the TWEC model yields better performance than
Orthogonal Procrustes, labelling all words scored below average cosine
similarity as semantically shifted words, i.e. words with altered semantics
over the two time periods. Additionally, we showed that using an outlier
detection methodology yields better results in prediction-based models such as
Linear Regression and Feed-Forward Neural Network, boosting the performance
significantly compared to the baselines and dummy classifier.

In the future we aim to focus on fine tuning SoTa pre-trained language models
such as ELMo and BERT for word level semantics-shift detection as well as
investigating the ability of dynamic graph models on capturing word evolution.

##  8 Acknowledgments

This research utilised Queen Mary’s Apocrita HPC facility, supported by QMUL
Research-IT.

## References

  * [Azarbonyad et al., 2017]  Hosein Azarbonyad, Mostafa Dehghani, Kaspar Beelen, Alexandra Arkut, Maarten Marx, and Jaap Kamps.  2017\.  Words are malleable: Computing semantic shifts in political and media discourse.  International Conference on Information and Knowledge Management, Proceedings, Part F1318(3):1509–1518.
  * [Basile and McGillivray, 2018]  Pierpaolo Basile and Barbara McGillivray.  2018\.  Exploiting the web for semantic change detection.  In International Conference on Discovery Science, pages 194–208. Springer.
  * [Basile et al., 2020]  Pierpaolo Basile, Annalina Caputo, Tommaso Caselli, Pierluigi Cassotti, and Rossella Varvara.  2020\.  Overview of the EVALITA 2020 Diachronic Lexical Semantics (DIACR-Ita) Task.  In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors, Proceedings of the 7th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA 2020), Online. CEUR.org.
  * [Boleda et al., 2019]  Gemma Boleda, Marco Del Tredici, and Raquel Fernández.  2019\.  Short-term meaning shift: a distributional exploration.  Proceedings of the 2019; 2019 Jun 2-7; Minneapolis, United States of America. Stroudsburg (PA): ACL; 2019. p. 2069–75.
  * [Carlo et al., 2019]  Valerio Di Carlo, Federico Bianchi, and Matteo Palmonari.  2019\.  Training temporal word embeddings with a compass.  CoRR, abs/1906.02376.
  * [Cignarella et al., 2020]  Alessandra Teresa Cignarella, Mirko Lai, Cristina Bosco, Viviana Patti, and Paolo Rosso.  2020\.  Overview of the EVALITA 2020 Task on Stance Detection in Italian Tweets (SardiStance).  In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors, Proceedings of the 7th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian (EVALITA 2020). CEUR-WS.org.
  * [Cook and Stevenson, 2010]  Paul Cook and Suzanne Stevenson.  2010\.  Automatically Identifying Changes in the Semantic Orientation of Words.  In Proceedings of the Seventh conference on International Language Resources and Evaluation.
  * [Dubossarsky et al., 2017]  Haim Dubossarsky, Daphna Weinshall, and Eitan Grossman.  2017\.  Outta control: Laws of semantic change and inherent biases in word representation models.  In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 1136–1145.
  * [Frermann and Lapata, 2016]  Lea Frermann and Mirella Lapata.  2016\.  A bayesian model of diachronic meaning change.  Transactions of the Association for Computational Linguistics, 4:31–45.
  * [Giulianelli et al., 2020]  Mario Giulianelli, Marco Del Tredici, and Raquel Fernández.  2020\.  Analysing Lexical Semantic Change with Contextualised Word Representations.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, July. Association for Computational Linguistics.
  * [Hamilton et al., 2016]  William L Hamilton, Jure Leskovec, and Dan Jurafsky.  2016\.  Diachronic word embeddings reveal statistical laws of semantic change.  arXiv preprint arXiv:1605.09096.
  * [Kim et al., 2014]  Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov.  2014\.  Temporal analysis of language through neural language models.  In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 61–65.
  * [Kutuzov et al., 2018]  Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski, and Erik Velldal.  2018\.  Diachronic word embeddings and semantic shifts: a survey.  In Proceedings of the 27th International Conference on Computational Linguistics, pages 1384–1397.
  * [Mikolov et al., 2013]  Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  2013\.  Distributed representations of words and phrases and their compositionality.  In Advances in neural information processing systems, pages 3111–3119.
  * [Sagi et al., 2009]  Eyal Sagi, Stefan Kaufmann, and Brady Clark.  2009\.  Semantic Density Analysis: Comparing Word Meaning across Time and Phonetic Space.  In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pages 104–111. Association for Computational Linguistics.
  * [Schlechtweg et al., 2019]  Dominik Schlechtweg, Anna Hätty, Marco Del Tredici, and Sabine Schulte im Walde.  2019\.  A wind of change: Detecting and evaluating lexical semantic change across times and domains.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 732–746.
  * [Schlechtweg et al., 2020]  Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi.  2020\.  Semeval-2020 task 1: Unsupervised lexical semantic change detection.  arXiv preprint arXiv:2007.11464.
  * [Schönemann, 1966]  Peter H Schönemann.  1966\.  A Generalized Solution of the Orthogonal Procrustes Problem.  Psychometrika, 31(1):1–10.
  * [Shoemark et al., 2019]  Philippa Shoemark, Farhana Ferdousi Liza, Dong Nguyen, Scott Hale, and Barbara McGillivray.  2019\.  Room to glo: A systematic comparison of semantic change detection approaches with word embeddings.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 66–76.
  * [Tahmasebi et al., 2018]  Nina Tahmasebi, Lars Borin, and Adam Jatowt.  2018\.  Survey of computational approaches to lexical semantic change.  arXiv preprint arXiv:1811.06278.
  * [Tan et al., 2015]  Luchen Tan, Haotian Zhang, Charles Clarke, and Mark Smucker.  2015\.  Lexical comparison between wikipedia and twitter corpora by using word embeddings.  In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 657–661.
  * [Tsakalidis and Liakata, 2020]  Adam Tsakalidis and Maria Liakata.  2020\.  Autoencoding word representations through time for semantic change detection.  arXiv preprint arXiv:2004.13703.
  * [Tsakalidis et al., 2019]  Adam Tsakalidis, Marya Bazzi, Mihai Cucuringu, Pierpaolo Basile, and Barbara McGillivray.  2019\.  Mining the UK web archive for semantic change detection.  In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 1212–1221.
  * [Turney and Pantel, 2010]  Peter D Turney and Patrick Pantel.  2010\.  From frequency to meaning: Vector space models of semantics.  Journal of Artificial Intelligence Research, 37:141–188.