Preprint: [gray]0.5 1cm 90 20cm

# Survey in Characterization of Semantic Change

Jader Martins Camboim de Sá  FSTM - University of Luxembourg
2 place de l’Université, L-4365, Esch-sur-Alzette, Luxembourg  Luxembourg
Institute of Science and Technology,
5 avenue des Hauts-Fourneaux, L-4362, Esch-sur-Alzette, Luxembourg
<EMAIL_ADDRESS>Marcos Da Silveira  Luxembourg Institute of Science and
Technology,
5 avenue des Hauts-Fourneaux, L-4362, Esch-sur-Alzette, Luxembourg
<EMAIL_ADDRESS>Cédric Pruski  Luxembourg Institute of Science and
Technology,
5 avenue des Hauts-Fourneaux, L-4362, Esch-sur-Alzette, Luxembourg
<EMAIL_ADDRESS>
###### Abstract

Live languages continuously evolve to integrate the cultural change of human
societies. This evolution manifests through neologisms (new words) or semantic
changes of words (new meaning to existing words). Understanding the meaning of
words is vital for interpreting texts coming from different cultures
(regionalism or slang), domains (e.g., technical terms), or periods. In
computer science, these words are relevant to computational linguistics
algorithms such as translation, information retrieval, question answering,
etc. Semantic changes can potentially impact the quality of the outcomes of
these algorithms. Therefore, it is important to understand and characterize
these changes formally. The study of this impact is a recent problem that has
attracted the attention of the computational linguistics community. Several
approaches propose methods to detect semantic changes with good precision, but
more effort is needed to characterize how the meaning of words changes and to
reason about how to reduce the impact of semantic change. This survey provides
an understandable overview of existing approaches to the characterization of
semantic changes and also formally defines three classes of characterizations:
if the meaning of a word becomes more general or narrow (change in dimension)
if the word is used in a more pejorative or positive/ameliorated sense (change
in orientation), and if there is a trend to use the word in a, for instance,
metaphoric or metonymic context (change in relation). We summarized the main
aspects of the selected publications in a table and discussed the needs and
trends in the research activities on semantic change characterization.

##  1 Introduction

Language seems to be one of the most dynamic skills that humans have acquired
through their evolution. It has constantly evolved to adapt the communication
to new purposes and needs Pinker, (2003). While there is no single unified
theory about its origins Allan, (2013), it is clear that language has
undergone significant modifications since the early days of human history.
These modifications have been driven by the need for humans to communicate
complex ideas, interact with each other, pass on history and experiences, and
build social bond McWhorter, (2011). One of the prevailing theories of
language creation and evolution is based on the assumption of “continuity of
change” Rilling et al., (2012). This theory holds that language is too complex
to have arisen suddenly, and must have gradually evolved from earlier pre-
linguistic systems used by our ancestors Pinker and Bloom, (1990). In this
context, comprehending the process of language change is an contribution to
the fields of history, sociology, linguistics, philosophy, and computer
science ( Natural Language Processing (NLP)), as it is essential to
understanding the origins of language, cultural development, grammatical
theories, and the relation between language and thought Sherwood et al.,
(2008); Bowern, (2019).

Many unplanned factors drive language change. For example, linguistic
advances, social, cultural, and historical influences, as well as technical
improvements. There is a variety of grammatical factors that also promote
changes, including:

  * •

Phonological Changes changes in the creation, perception, and representation
of sounds in language. For instance, the pronunciation of vowel sounds
significantly changed due to the Great Vowel Shift in Medieval English
Nevalainen and Traugott, (2016).

  * •

Lexical Changes alterations in the written form of words to form meaningful
sentences and to pair with phonological changes, e.g., panem to pain, ‘bread’
in Latin to French.

  * •

Syntactic Changes alterations to the arrangement of words to create coherent
sentences. For example, Modern English emphasizes word order and auxiliary
verbs more than the intricate inflectional system used in Old English to
identify grammatical connections Nevalainen and Traugott, (2016).

  * •

Semantic Changes phrases or words that have meaning changed over time, e.g.,
notebook “a book for notes”, to “a portable computer.”

While alterations in sound and form are crucial elements shaping our language,
semantics plays a pivotal role in driving these changes Allan and Robinson,
(2012). Semantic drift influences how words adapt to evolving cultural,
technological, and societal contexts, creating new meanings, and preserving
the language’s fundamental purpose of effective communication. Despite being a
recurring topic in scientific research, semantic drifts remain less understood
than other forms of linguistic evolution due to their often subtle and less
observable nature, such as those influenced by grammar reforms.

Historically, their approach to studying this process primarily relied on
manual methods. Linguists would painstakingly analyze historical texts,
dictionaries, and linguistic corpora, tracing the shifting meanings of words
through centuries of written records. These manual studies provided invaluable
insights into the intricacies of semantic change but were labor-intensive,
time-consuming, and difficult to reproduce.

However, with the advent of computational linguistics and the development of
advanced computer models, the field underwent a transformative shift.
Linguists began harnessing the power of machine learning and natural language
processing to automate the analysis of vast amounts of textual data. So the
research community gradually shifted from anthropological observation to
statistical methods for language analysis Allan and Robinson, (2012).

These computer models, fueled by large corpora, and sophisticated algorithms,
enabled researchers to detect subtle properties in language with unprecedented
accuracy and efficiency. Enabling studying languages without any previous
knowledge of it’s syntax or semantics, comparing multiple languages between
themselves in the pursuit for universal properties Vulic et al., (2020).

By embracing a model-based approach, linguistics can now unravel the
intricacies of language phenomena and cope with the complexity of linguistics
from different cultures and backgrounds Nivre et al., (2020). This scientific
tooling provides a clear pathway for progress by facilitating direct
comparisons between hypothesis, mathematical models, their refinement, and
testing against emerging linguistic phenomena Hammarström et al., (2019).

While initially only observed with a theoretical motivation, the practical
importance in computer science became very relevant. For example, in sentiment
analysis, where the orientation of a word can change over time. Consider the
word “sick”, while it traditionally meant “ill” or “unwell”, we can observe
young people frequently using this word to mean “impressive” or “cool”
Merriam-Webster, (2023). Another type of change is observed with the word
“baby”. The frequent use of this word as a metaphor has impacted on its
semantic interpretation in musics and poems, moving from “child” (as in the
14th century) to include the meaning “young woman” (sweetheart) in the 20th
century.

The impact on applications like social media monitoring could be significant,
as misinterpretation of slang terms, metaphors, or users’ domains could lead
to unsuccessful outcomes. If an NLP algorithm is not aware about these
differences, it may incorrectly interpret a sentence and attribute, for
instance, a positive sentiment instead of a negative one (or vice versa),
leading to an inaccurate analysis of the text Buechel et al., (2016).

Addressing this issue involves continuously updating NLP models and algorithms
to account for semantic change, incorporating real-time data sources, and
staying aware of evolving language trends Lu et al., (2022). It underscores
the need for adaptability, ongoing maintenance, and explain decisions of NLP
applications that rely on natural language understanding.

The field of computational Lexical Semantic Change (LSC) emerged as more
focused effort to improve methods to computationally identify semantic change
in lexicons. Employing computer models that infer statistics from data has
shown reliable capabilities to investigate lexicon meaning, enabling scholars
to keep track of these variances in a theoretical and structured manner and
many algorithms with diverse assumptions were proposed to improve meaning
representation capabilities Tahmasebi et al., (2021), and consequently NLP
systems.

The development of better tools, that are cost effective, and can handle the
many variation in language is crucial for many business. However, while the
linguistic field has broadly studied what phenomena cause change Traugott,
(2017), and many works provide deep analysis and theories on types of change.
The computational linguistics community performed considerable effort to
detect these drifts, nonetheless, to the best of our knowledge, less effort
has been made to formalize and understand what kind of drift a word has
undergone Hengchen et al., (2021).

Another important development in this field is a proper language and formalism
to describe the problems in semantic change Tahmasebi et al., (2021). Having a
formal language is of paramount importance in a scientific field for several
reasons. It provides a structured framework that allows researchers to
precisely articulate and communicate their hypotheses and findings. It also
ensures that ideas are expressed in a clear and unambiguous manner,
facilitating collaboration and knowledge dissemination within the field.
Furthermore, employing a mathematical language can significantly enhance our
understanding of semantic change, advancing our comprehension of one of the
most intriguing aspects of human communication.

The objective of this survey is to cover the lack of mapping and formalization
in semantic change characterization. Being able to characterize such changes
will open new perspectives in many areas. This is particularly the case in
communication where semantic change creates potential for misunderstandings
and ambiguity, especially in contexts where both meanings of a word are
relevant, leading to a politically or socially misinterpretation of the
context where the word was employed.

As both computer science and linguistic communities, value understanding the
computational aspects of language evolution. On this survey, we will focus on
three factors of semantic changes: the orientation (i.e., if the words meaning
is getting more pejorative or positive), the relation (i.e., if the words
meaning change through a figurative usage), and the dimension (i.e., if the
words meaning is becoming more abstract or more specific). Some of these
properties, as we detail later, are crucial, for instance, for understanding
and explaining the intention of a sentence, for writing tools to assist
translation or to interact with users (e.g. in social robot). Finally, it will
open new opportunities to improve reasoning capabilities for NLP tools
Danilevsky et al., (2020).

Drawing from a comprehensive review of the extensive literature across related
areas, this survey makes a significant contribution by consolidating,
categorizing, and pinpointing gaps in existing approaches. Furthermore, it
introduces formal descriptions and proposes enhancements towards a more robust
characterization of semantic change. This endeavor represents a pivotal
advancement in our comprehension of this intricate phenomenon. By synthesizing
the collective knowledge from prior research and offering systematic solutions
to the identified limitations, this survey not only helps streamline the
existing body of work but also paves the way for a more efficient and
comprehensive understanding of the multifaceted nature of semantic change.

The remainder of the paper is structured as follows: Section 2 reviews
existing surveys and their difference with respect to this work. In Section 3
we investigate available corpora of documents for semantic change
identification, characterization and their limitations. In Section 4 we
explore existing methods to represent meaning change characterization and how
they attempt to solve the characterization problem. Section 6 presents our
analysis and open questions in characterization. Finally, Section 7 we
conclude this survey and outline future work.

##  2 Lexical Semantic Change

Concerning the characterization of change, we adopt in this survey the
mainstream typology in linguistic literature for semantic change Koch, (2016);
Blank et al., (2003); Hock and Joseph, (2019), (i) broadening and narrowing,
(ii) amelioration and pejoration, and (iii) metonymization and
metaphorization. Before starting our study on change characterization, we
reviewed the works published on change identification. This is a well-
developed topic that focuses on evaluating whether the meaning of a word has
changed or not. Papers addressing change identification often claim the need
for further understand and characterization of the type of change that
occurred, but the current state of works on computational change
characterization is unclear. This conclusion motivated the work presented in
this paper and differentiate our survey from existing ones on LSC.

In the following subsections we present existing surveys dealing with the
semantic change problem and identify the gaps of the current literature.
Inspired by the definitions introduced in these studies, we propose our own
typology of changes. The last section introduces the objectives of this survey
and how we plan to address the gap in semantic change characterization in our
future work.

###  2.1 Related Works on Identification of Lexical Semantic Changes

Surveys for semantic change identification are regularly published, with a
main focus on methodologies for change detection. In a survey about “semantic
change computation” Tang, (2018), Tang proposes to split LSC analysis into 4
steps: obtaining and preparing a diachronic corpus, identifying word senses,
modeling semantic change, followed by presentation and validation of
hypotheses. This framework sets the structure of the survey, describing the
studies related to each of these steps. The characterization of changes is
mainly discussed in the “modeling semantic change” section, where the author
explains the difference between Cambridge Change Ardley, (1968), and McTaggart
Change Broad, (1938) and cites the approaches following these definitions.
However, the approaches that can classify semantic changes as sense
broadening, sense narrowing, pejoration, amelioration, metaphorical change and
metonymic change are not included.

The survey about computational approaches to LSC Tahmasebi et al., (2018)
focuses on word-level differentiation, sense differentiation, and lexical
replacement. While it includes approaches with several types of change
categories (e.g., metaphor, metonymy), the main focus of change categories are
broadening, narrowing and correlated types (split, join, birth). However, one
recommendation of the authors is to have deepen studies on methods to
automatically detecting other types of change and to use it to build the
history of the words. Auxiliary to the methodologies discussed, the authors
provide an in-depth analysis of linguistic literature that grounds each
domain, followed by evaluation strategies and applications. The difficulty to
compare approaches is highlighted in the survey and, according to the authors,
is the result of an absence of standard metrics/practices and reference
datasets.

Kutuzov et al., (2018) survey word representation models and data for semantic
change identification, and describe a framework to compare these
methodologies. The authors focus on defining a consistent terminology, and
coherent practices to evaluate these models. In this work, they claim that
there’s still a gap in studying sub-classifications of change:

> “However, more detailed analysis of the nature of the shift is needed. […]
> This problem was to some degree addressed by Mitra et al., (2015), but much
> more work is certainly required to empirically test classification schemes
> proposed in much of the theoretical work […]”

First studies on the field employed graph- or frequency-based methods to
analyse LSC, later approaches benefited from the advances in the state-of-the-
art to replace the analytical methods by those based on embeddings and
contextual embeddings. Although contextual embeddings represented a huge
improvement for a series of NLP tasks Devlin et al., (2019), the improvement
for LSC did not follow this trend Schlechtweg et al., (2020), showing that
there is still room for improvement in this domain Zhou et al., (2021);
Kutuzov et al., (2022).

In this context, Dubossarsky et al., (2017) investigate the quality of laws
addressing the properties of change. The authors observed that laws proposed
in previous studies are affected by the bias in the model or in the data. When
this bias was reduced, the ‘laws’ either did not hold or were less precise
than originally reported.

Research in this field also delves into ongoing LSC methodological issues.
These issues involve the challenges of characterizing semantic shifts. As
pointed out in Hengchen et al., (2021), there are few studies that detail the
nature of these changes, mainly because it is difficult to model meaning from
textual data. It further concludes that no particular measures for quantifying
category change exist and that a consensus on taxonomy for classifying types
of change is required.

In the work of Montanelli and Periti, (2023), the authors review methodologies
based on contextualized word embeddings. They categorize methodologies for
training contextualized embeddings based on how they represent meaning, how
the time information is presented to the learning model, and what kind of
learning procedure is performed (unsupervised or supervised). In this paper,
the authors also discuss the importance of characterization:

> “Most of the literature papers do not investigate the nature of the detected
> shifts, meaning that they do not classify the semantic shifts according to
> the existing linguistic theory (e.g., amelioration, pejoration, broadening,
> narrowing, metaphorisation, and metonymisation) […]. These studies could be
> crucial to detect “laws” of semantic shift that describe the condition under
> which the meanings of words are prone to change. […]”

For this survey, we argue that semantic change characterization needs to have
appropriated methods and tools to analyse each fine-grained category. We also
defend that observing variations in a single property can not describe well
all forms of LSC. For instance, changes in angles between vectors representing
words Dubossarsky et al., (2017) is not the best solution to identify
narrowing/broadening of sense Tahmasebi et al., (2018). Specific methods for
identifying the different types of change could led to a better understanding
of LSC than straightforward dissimilarity measures.

This survey addresses the need to depict the characterization problem in
semantic change and present recent works dealing with this problem. In the
next section we define the survey objectives and search criteria to select
papers for change characterization.

###  2.2 Objectives

Different terms are used to refer to Lexical Semantic Change (LSC), such as
semantic drift or semantic shift. In this paper we make the distinction
between identification of a change and characterization of the change. The
former refers to the process through which the meaning of a word gradually or
significantly changes over time in a language. It is the process through which
a language unit’s initial semantic content changes, leading to a shift in its
usage, connotation, or meaning. Cambridge Ardley, (1968) and McTaggart Broad,
(1938) change approaches are different on how to use temporal information to
identify changes. While Cambridge proposes to compare two different corpus
(from different time period), McTaggart prefers to study the evolution of the
usage of a word over a period of time. The analysis of works dealing with the
identification problem has been well covered by Tahmasebi et al., (2018),
Kutuzov et al., (2018), and Tang, (2018). We formally define the semantic
change identification in Section 5.1.

The focus of our literature review is on the characterization of change, where
we look for common properties that allows for grouping identified semantic
changes into categories. Semantic change characterization occurs by the need
to reduce fuzziness when combining meanings and forms Hock and Joseph, (2019).
While some authors propose to use the cause of the changes to group them,
other authors categorize the type of change by language figures that describe
the change, or by the cultural process. From a computation perspective,
determining the cause of a semantic change becomes very complex to implement
and, to the best of our knowledge, none computational approach was implemented
yet. For this review we followed the proposition of using language figures as
bases to create categories of changes. We adopt the most predominant
categories in literature, following the typology proposed in Juvonen and
Koptjevskaja-Tamm, (2016) and Tahmasebi et al., (2021):

  * •

Broadening gaining a new meaning, that is not related or not, to the previous
meaning, such that a symbol represents more concepts, e.g., ‘cloud’ a
computing infrastructure.

  * •

Narrowing restriction of meaning where a symbol represents fewer concepts, as
in ‘gay’ that used to have senses as festive or happy, now it generally is
used in a single sense of homossexual.

  * •

Amelioration a word gained a more positive sense with respect to the previous
sense, nice, ‘foolish, innocent’ changed to ‘pleasant.’

  * •

Pejoration the word is used with a worse connotation with respect to the
previous usage, stincan, ‘smell (sweet or bad)’ changed to stink.

  * •

Metonymization association between terms, e.g., board ‘table’, changed to
“people sitting around a table, governing body.”

  * •

Metaphorization conceptualizing one thing in terms of another, e.g., the use
of Latin ad ’to’ + mit ’send’ for locution (admit) Traugott, (2017).

As the types in the above typology are directly correlated (involves the same
computational challenge), e.g., broadening-narrowing, requires sense counting,
amelioration-pejoration, requires sentiment analysis, metaphorization-
metonymization, requires some form of relatedness measure Koch, (2016); Blank
et al., (2003); Hock and Joseph, (2019), we group them in poles to better
compare, categorize, and generalize studies exploring these types of change.
Nevertheless, we have included only approaches that consider changes between
corpora as this typology was designed for diachronic studies, not changes
related to a specific domain or to a specific context Koch, (2016); Blank et
al., (2003); Hock and Joseph, (2019). In Figure 1 we illustrate the
hierachical taxonomy for LSC.

Figure 1: Taxonomy for the poles of Lexical Semantic Change, based on the work
of Koch, (2016); Blank et al., (2003); Hock and Joseph, (2019).

We assign studies that track sense broadening or narrowing in the dimension
($\mathcal{D}$) pole and we assume that words that are getting more or fewer
senses are increasing or decreasing their dimension value (see Section 5.2).
This pole is self-complementary as the change in the value of the dimension
addresses both senses. A change in dimension could work as a level of
carefulness with translations for example, that broader words should be
translated with more care Mohammed, (2009).

We regroup in the relation ($\mathcal{R}$) pole changes increasing the
relation between one word sense and other senses for this word (see Section
5.3). This kind of changes are commonly related to metaphorization or
metonymization, where the meaning of a word depends on the link built by a
conceptual (abstract) or material (direct) similarity between concepts.
Relations like hypernymy, homonymy, and synonymy are also factors for semantic
change Koch, (2016); Hock and Joseph, (2019) as they are material relation and
can potentially be encompassed in this pole.

For the orientation ($\mathcal{O}$) pole, we group studies that identify the
process of amelioration or pejoration or any perceptual change in the
”sentiment” related to words (e.g., criticism, compliment, etc.). This pole
also encompasses many forms of change in perception of opinions, as some
studies suggest that sentiments like ‘taboo’ Koch, (2016) also affect the
change in word usage.

To illustrate an example of a word that suffered a change in orientation, and
how is it natural to describe it in a single pole, we select the word ‘awful’.
In 1850s, it was used with the meaning “full of awe”, and nowadays it means
“extremely disagreeable”. As the first sense is semantically closer to good
and the second closer to bad sentiment, this word suffered a pejoration as
illustrated in Figure 2. Notice that the evolution of the orientation of this
word pass first from an amelioration before changing to a pejoration. However,
in the same figure, the word ‘awesome’, that has a close relation with ”awe”
gained a more positive sense, suffering an amelioration over time.

Figure 2: Change in meaning and orientation for the word awful. In the left
side, we reproduce the figure from Hamilton et al., (2016) that shows the
evolution of the word ‘awful’ in the embedding space. In the right side, we
present the hypothetical function ($\mathfrak{f}$) for this word over time.

In the next section, we will describe our method for reviewing the literature
based on the poles presented above.

###  2.3 Review Criteria

Querying for documents in different databases of scientific papers required an
interactive work to include terms that are relevant to our survey. There is no
consensus on the terminology used to categorize the papers. Thus, we decided
to use a semantic search approach to identify critical papers in the field and
enrich our searching query. In Table 1, we present the final set of search
terms used to collect articles related to the characterization problem. We
used Semantic Search, Scopus and Google Scholar as source of papers. The
interactive process has as “end criteria” when the search for the terms does
not find any new match. We execute this process for each term identified. The
semantic search help us to extend the set of terms. For that, we used an
auxiliary software, Research Rabbit111https://researchrabbitapp.com/, to
discover similar relevant work.

Table 1: Search terms used to discover articles related to change
characterization.

Characterization | Search terms
---|---
Dimension |  | “semantic change broadening”,“semantic change narrowing”,
---
“novel sense change”,“innovative meaning”,“new sense identification”,
“neologism identification”
Relation |  | “metonymy semantic change”,“metonymization semantic change”,
---
“metaphor semantic change”,“metaphorization semantic change”
Orientation |  | “semantic change polarity”,“semantic change feeling”,
---
“semantic change orientation”,“semantic change emotions”,
“semantic change sentiment”

We gave as input to the software a set of related papers (found in Google
Scholar and Semantic Scholar) and we performed an iterative process of adding
new papers until reaching a stable set of documents without further
improvements. In Figure 3, we present the graph of documents, where green
nodes represent the input papers, and blue nodes are papers suggested by the
tool after the first interaction.

Figure 3: Graph of selected works (green) and related articles (blue).

We manually filter the papers that characterize word changes based on context,
e.g., as a figure of language, but not a fundamental change in the word sense.
For example, those approaches that detect if a word was used metaphorically in
a sentence, but do not infer if the word suffered a metaphorization over
time/corpus.

For relation and orientation, we had a vast range of false positives as many
works are trying to capture metaphor or orientation without comparing corpora,
so this approach could not infer changes in the sense and were not included in
the selected papers.

In the next sections, we investigate in depth three elements of semantic
change characterization, (i) the corpora ($T$) analyzed, (ii) the
representation method ($\mathfrak{R}$) for capturing characteristics of
changes, and (iii) the kind of characterization
($\\{\mathcal{D},\mathcal{R},\mathcal{O}\\}$) addressed by these methods.

##  3 Corpora for Semantic Change Analysis

Corpora for semantic change could be selected based on differences in time
(diachronic studies), domain (academic fields Schlechtweg et al., (2019)),
culture (social media Carpuat et al., (2013); Shoemark et al., (2019),
countries), language Takamura et al., (2017); Uban et al., (2021), and so on,
as soon as the pair of corpora ($t_{1},t_{2}$) taken at different contexts
have in common a sufficient amount of symbols for comparison.

In diachronic studies, some studies set the problem as two different data
distributions with a meaningful gap between the time interval. Some methods
divide the corpora into discrete bins or time windows, striking a balance
between the size of the window and the quantity of data it contains. In
contrast, others create no bin and proceed with a continuous representation of
change where the timestamp associated with the text is directly provided to
the model Hu et al., (2019); Rosenfeld and Erk, (2018); Wang et al., (2021).
These corpora were designed for specific purposes. For example, the first two
types of corpus rely on a discretization of word meanings in a comparative
perspective, while the third one allows for an evolutionary view of the
changes.

With respect to characterization, the collected papers show an increasing
interest on diachronic corpora and many of them propose handcrafted rules for
automatically annotating the categories of changes Tang et al., (2013). Few
works rely on human annotation Schlechtweg et al., (2017). While these
approaches have advantages and drawbacks, from a computational perspective,
the required data to train representation models is far from being large
enough. Recent state-of-the-art models for NLP require training on billions to
trillions of tokens He et al., (2021), but diachronic corpora generally
present tokens in a scale of millions Schlechtweg et al., (2020). Given that
increasing the amount of training corpora is in most cases, infeasible, the
viable approach to learning semantics in this context is with better
algorithms.

Even with the progress in the fields of low-resource learning like rapid
adaptation Neubig and Hu, (2018), few-shot Liu et al., (2022) or in-context
learning Min et al., (2022). There is still a meaningful lack on freely
available corpora for semantic change studies Kutuzov et al., (2018).

##  4 Representation methods for Semantic Change

While interest in detecting semantic changes is growing, there is less efforts
being applied on characterizing these changes. Computational linguistics has
not yet fully explored the usefulness of having detailed information about a
type of change. For example, a change in relation would have a low impact on a
sentiment analysis algorithm, while a change in orientation is more likely to
affect it.

In this context, the methods to represent meanings are of high relevance to
understand what kind of change could be described and characterized. This
section investigates in the literature representation methods ($\mathfrak{R}$)
based on: word frequency, topics, graphs, and embeddings. We analyzed how each
method was used to characterize LSC into the three proposed poles.

###  4.1 Word frequency

Methods based on word frequency were among the first computational attempts to
tackle semantic change and are still at the heart of LSC and of many other
linguistic approaches. Analysing word frequency in a corpus contributes to a
direct interpretation of change, which in the context of characterization, is
generally executed by counting co-occurrence with other words.

####  4.1.1 Dimension

To detect changes along the dimension axis ($\mathcal{D}$), Sagi et al.,
(2009) propose a representation method ($\mathfrak{R}$) to count together
words and phonemes (e.g. ‘dog’ and ‘do’) and measure the cluster density of
the cosine distance on a decomposition of the term frequency matrix. They
assume that Latent Semantic Analysis is appropriate to examine vectors
representing the context of each occurrence of a given term in a corpus and
estimate the cohesiveness by measuring the density with which these token
context vectors are related to each other in a vector space. Tang et al.,
(2013) propose a framework for semantic change computation by assuming that
semantic change is by nature a successive and diachronic development. They
further distinguish between word level (i.e. the semantic of a given word) and
sense level (i.e. individual sense of a word) and implement their approach
based on these two notions. One essential step in their approach aims at
computing the senses of a word in a given corpus and then the diachronic
distribution of these senses in the corpus. Carpuat et al., (2013) focus on
neologism detection in corpora and they apply classification techniques for
that. Those techniques are denoted as i) Type-level Features and encompass
word frequencies change, the appearance of the word in a new context, and
word’s distribution over topics change, ii) Token-level Features that refers
to the same kind of features but generalize at the token level. Emms and
Jayapal, (2016) propose a generative conditional model to detect if a word
gained a new sense over time. This is done by modeling the word sense
distribution with Gibbs sampling (a kind of a Markov chain Monte Carlo
algorithm for obtaining a sequence of observations). Bochkarev et al., (2022)
use a neural network to predict if a word is used as a named entity. By this
method, they create a temporal view of usage of a word and they can compare
the occurrences of a word to detect if it gained a new usage in the corpora.

####  4.1.2 Relation

The work of Tang et al., (2013) transforms the problem of relation
identification into change pattern detection in time series data. They analyse
the semantic change at the word level and the individual-sense level inside a
corpora based on rules in order to infer if a word is used as metonymy or
metaphor. A different approach computes the entropy of target words (i.e., the
unpredictability of the use of these words) to identify if a word is being
used in a non-regular context Schlechtweg et al., (2017). The higher the
entropy of a word is, the more likely this word is used in a newly created
metaphoric sense. They are using a corpus of German document to validate their
approach.

####  4.1.3 Orientation

Frequency metrics to detect the co-occurrences with sets of ”good-” or ”bad-”
sentiment words has been widely used to infer the amelioration and pejoration
of a word over time, especially for neologism Cook and Stevenson, (2010);
Jatowt and Duh, (2014). Similarly, Buechel et al., (2016) use seed words in
German and English to track the emotions captured in corpora. Additionally,
the authors represent it in the Valence-Arousal-Dominance framework instead of
following a negative and positive connotation. A limitation of the later study
is that it does not track the evolution of orientation in words.

Frequency approaches are, for a long time known to be limited for the analysis
of semantic of words, given their inability to capture similarity, as stated
in the work of Mikolov et al., (2013), and compositionality, as pointed out by
Mitchell and Lapata, (2010). For instance, documents using different words
could describe the same thing. The n-gram approach, to some extent Cheng et
al., (2006), improves compositionality understanding but is hindered by the
statistical significance or Zipf distribution Zipf, (1949), which is
outperformed by complete sentence understanding as showed by Socher et al.,
2013a .

Another limitation of frequency-based methods is its inability to represent
sense and a change in word usage could not be differentiated between sense
broadening or narrowing. Therefore, an extra step is required to deal with
polysemy. Some authors tried normalizing frequency Hamilton et al., (2016),
however, as normalization developed for one corpus is hardly applicable to
another corpus Brezina, (2018), this method fell into disuse Kutuzov et al.,
(2018). Another approach that fell into disuse is the definition of general
laws for semantic change Dubossarsky et al., (2017). The use of co-occurrence
tables for a the analysis of semantic change misjudged polysemy and
prototypicality. Deep study of co-occurrence tables concluded that there is no
statistical correlation between frequency with polysemy or semantic change.

Finally, frequency-based methods has shown low performance to detect
orientation of words Socher et al., 2013b  as well as metaphor and metonymy.
Since the words are deeply related to the used context, the lack of
compositionality could mismatch irony or negation, and could not recognize the
right ”relation” between word meanings.

###  4.2 Topics

Topic modeling is a statistical method used to identify the most important
themes addressed in a collection of documents. This is done by grouping words
that are frequently used together under topics. For example, a topic model
might identify the topic of “bank” in a corpus of news articles by grouping
together words like “government,” “money,” and “insurance”.

Semantic analysis is a type of intelligent algorithm that uses topic modeling
to understand the meaning of words and phrases. It does this by tracking the
different topics that a word is associated with in a context including time,
domain, culture, etc. For example, a topic model could be used to understand
the different meanings of the word “bank” in the context of finance (i.e., a
financial establishment) and in the context of a river (i.e., the land
alongside a body of water), or the word “compute” from its early days as a
large mainframe machine to its current meaning as a small and personal device.
The different topics associated with a word are tracked and compared in order
to identify changes in their senses and, when possible, characterize these
changes.

####  4.2.1 Dimension

These approaches assume that changes in the number of topics assigned to a
word can indicate changes in dimension. Lau et al., (2012) introduced the idea
of novelty as the proportion of usages of a given word corresponding to senses
in the focus corpus and reference corpus. The goal is to use word sense
induction based on topic modeling to detect if a word is used with a new
meaning. Cook et al., (2013) extended the work of Lau et al., (2012) by
quantifying the use of a topic and estimating that a “cultural silent” topic
that becomes frequent in the focus corpus has high probability to have
neologisms. For that, they introduced the concept of relevance of a topic and
compute a rank of potential neologisms. Lau et al., (2014) improved the
definition of novelty by adding the frequency of the word in the corpus, then
they proposed to combine novelty and relevance to rank the neologisms.

####  4.2.2 Relation

To the best of our knowledge, no methodology for tackling conceptual change
with topic-based representation has been published, while some works use it in
a synchronic detection setting Heintz et al., (2013).

####  4.2.3 Orientation

To the best of our knowledge, no methodology approaching orientation change
with topic-based representation was published. Some works in sentiment
analysis literature use topic models in a workflow in order to refine the data
used by a classifier to predict the perceived sentiment. However, topic models
are not directly applied for the classification task Wankhade et al., (2022).

Topic modeling methods are powerful tools for text analysis, but it is
important to be aware of the challenges associated with interpreting the
results. The outcomes often require manual analysis to consolidate and to give
sense to the topics. This is because the aggregation of words are
statistically defined and can be difficult to be understood by humans, which
can lead to misinterpretations of word senses.

By using a variety of evaluation metrics (such as topic coherence and topic
diversity) and visualization techniques, the manual work can be reduced. In
the cited approaches, the authors automatize the selection of topics by
performing posterior filtering based on the similarity with dictionary
definitions. Although the potential of this method to detect neologisms, its
use for change characterization still need deepen investigations.

###  4.3 Graphs

Graph structures are a natural way to represent relationships between words.
They are often used in semantic change research, where they are typically
constructed manually based on dictionaries and human supervision. However,
graph structures can also be constructed automatically based on word co-
occurrence. Co-occurrence graphs are better at capturing context than count
tables because their edges represent more complex relationships, such as
cliques of words that share some context. For example, the words ”feline”,
”cat”, and ”pet” would be likely to form a clique in a co-occurrence graph,
indicating that these words are often used together and have close meanings.

####  4.3.1 Dimension

Comparing the size of a graph over time can provide rich information about the
semantic change of a word. Biemann, (2006) was the first to implement this
task in order to detect the sense broadening of words. They created a graph of
associated nouns that occur in corpora, then applied the Chinese whispers
algorithm to find clusters in the graph. By comparing these clusters over
time, they were able to infer whether a word had acquired a new meaning. Mitra
et al., (2015) analysed the Tweets and count the word co-occurrence based on
bi-grams as a way to identify potential relationships. The graph was built
based on these relationships and they run a graph-clustering algorithm and
Chinese Whispers to determine the set of senses. Each cluster is considered as
a different sense of a word. By comparing this sets of senses over time they
can infer if a word split its senses or the senses becomes broader (joined by
other words). Mathew et al., (2017) improved filtering and sense selection
over previous methods introduced by Mitra et al., (2015); McCarthy et al.,
(2004); Lau et al., (2014), to better capture sense evolution in Google Books
and newspaper datasets. Tahmasebi and Risse, (2017) consider the graph of
occurrences of noun phrases with length two words as a cluster of meaning,
then they merge sub-graphs based on the lin similarity. They track dimension
by analysing how this sub-graphs change over time. Ehmüller et al., (2020)
build a co-occurrence graph, filtering with point-wise mutual information to
remove non relevant edges, and then create an ego-network to later run graph-
clustering algorithms. This approach allowed to build a sense tree222a tree of
word usage evolution, where new senses are new branches. Similar to previous
work, the authors restricted their analysis to nouns only.

####  4.3.2 Relation

Graphs are a good way to represent word relations. Some approaches have used
graphs to detect metonymy usage Teraoka, (2016); Pedinotti and Lenci, (2020).
Their model uses the graph-based distributional model of event knowledge to
determine a contextualized representation of word semantics, which is obtained
by integrating the lexical embedding of a word with a vector representation of
the expectations activated by the context for the event role of the word.
However, to the best of our knowledge, no work proposed to use graphs in a
representation level to compare which words have changed their relations
between corpora.

####  4.3.3 Orientation

The orientation of words generally is inferred from similarity or proximity
with respect to seed words. Many word graphs present this construction and
some works Reagan et al., (2017); Barnes et al., (2021) use these graphs to
infer orientation. However, to the best of our knowledge, no work used graphs
to compare the orientation of words in different contexts.

Current graph-based approaches did not explore modern constructions of graph
structure like knowledge graphs or graph embeddings to detect and characterise
LSC. This kind of representation could help inferring changes in the proposed
poles, e.g., Wikidata has information about hierarchical relations between
entities, e.g., Washington capitalOf U.S.A., that can be useful to construct
word material relation and consequently the analysis of metonymization.

Wordnet, Babelnet and Conceptnet are examples of graph structures used
auxiliary to many methodologies for change detection Giulianelli et al.,
(2023); Tang et al., (2023). However, these structures are under-explored
(generally used only as a sense storage), as they lack temporal information,
like timestamps for first occurrence.

###  4.4 Embeddings

Word embeddings are representations of words as numeric vector. The vector is
built in such a way that words that have close values in the vector space are
expected to have similar meanings. The distance between these words can help
the interpretation of their sense.

Word embeddings have been used for a variety of tasks, including LSC
identification and characterization. This is done by comparing word vectors
between two moment in time, the distance between them and with respect to the
neighboring vectors can indicate if the sense of the word changed Hu et al.,
(2019).

####  4.4.1 Dimension

Ryskina et al., (2020) propose a statistical approach to identify neologism
based on frequency count and semantic sparsity (measured with embeddings).
They claim that words and concepts suffer from supply and demand curves, so
emerging concepts (i.e., neologism or sense broadening) would appear to
populate the sparse semantic region until the space becomes uniformly covered.
In the work of Vijayarani and Geetha, (2020), the authors demonstrated that
vector embeddings and lexical features can be used to detect sense narrowing.
They explain with the characterization of the changes in meaning of the word
“tablet”, in English. They found that the word had undergone a narrowing of
meaning because now it was commonly used to describe a portable computer than
rectangular shaped objects (as it was its initial utilisation). By analysing
the deviation of a Gaussian word embedding, Yüksel et al., (2021) identified a
way to measure the narrowing and broadening in word senses. In Moss, (2020),
the authors also performed a training per sense of these embeddings.

####  4.4.2 Relation

Computational analysis was used by Fonteyn and Manjavacas, (2021) to
investigate the process of metaphorization of the concept “to death”. They
combine methods of cluster analysis and sentiment analysis to contextualize
the occurrences of this phrasal verb and detect if a metaphorization has taken
place. Another approach is to create a training corpus and use it to train a
BERT model to predict if a word is a metaphor or literal. The work of Maudslay
and Teufel, (2022) follows this approach and the authors propose a framework
to detect words that started being used as a metaphor.

####  4.4.3 Orientation

Methods based on word embeddings have been widely used for many sentiment
analysis tasks Raffel et al., (2019) and to investigate sentiment at lexicon
level Ito et al., (2020). In the context of LSC characterization, Fonteyn and
Manjavacas, (2021) uses the distance between the word vectors of ‘good’ and
‘bad’ as a means to measure polarity in the expression ‘to death’.

In general, word embeddings have an unstable and many times incoherent
behavior, as pointed out in Hellrich and Hahn, (2016); Gladkova and Drozd,
(2016); Günther et al., (2019). The reason is that vectors representing words
with the same meaning could be sensible to minor variations in the corpora or
in the architecture utilized for language modeling, thus different vectors can
represent the same meaning in different corpora Wendlandt et al., (2018);
Leszczynski et al., (2020); Mickus et al., (2020). This undesirable effect is
often observed for low frequency words in the corpus, which corresponds to
most words in a vocabulary Zipf, (1949), and particularly for those with
polysemy.

To mitigate the stability problem, where different training routing led to
very different learned vectors, authors recurred to larger embedding sizes,
quantization May et al., (2019), or alignment between trained embeddings
Hamilton et al., (2016). However, the vector representation is already a
burden, as standard approaches to semantic change identification suffer from
meaning conflation deficiency Camacho-Collados and Pilehvar, (2018).

The conflation deficiency happens when a polysemous word is represented by a
single vector. Suppose that a word $w$ in moment $t_{1}$ has two sense, e.g.,
$w_{t_{1}}=\\{s_{1},s_{2}\\}$, where the dominant sense (the most common sense
in the corpora we are looking at) is $s_{1}$ and this configuration pushes the
embedding vector to its semantic region. At a later moment in time $t_{2}$,
the word still have both senses, e.g., $w_{t_{2}}=\\{s_{1},s_{2}\\}$, but the
dominant sense is now $s_{2}$. The result is that the embedding vector will
shift towards the semantic region of the concept $s_{2}$. There were no real
semantic changes in the word. But, the example can misled semantic change
detection algorithms to this wrong conclusion.

Some works tried to address conflation deficiency by adding a vector for each
sense Neelakantan et al., (2014); Liu et al., (2015), which generally requires
additional annotation or steps to discriminate senses (i.e., given a word,
which vector will be used).

An extension of these works includes contextualized words, and create
embeddings based on pre-trained language models, generally extracted from ELMO
Peters et al., (2018) or BERT Devlin et al., (2019). The main difference is
that instead of using the pre-trained embeddings for each word, this approach
uses the outputs of the transformer’s final (contextualized) layers to capture
meaning. These outputs are then clustered or compared using cosine similarity
to measure the sense of the word. For instance, the representation of the word
‘mouse’ in a text will be assigned to the correct semantic region, such as
‘animal’ or ‘computer device’.

Nevertheless, while they presented an improvement to sense detection, they
produce words representation that are too contextual. Thus, words with the
same meaning can have very different vectors if they are in a different
context, and the representations are anisotropic. The distribution in the
semantic space is not uniform, leading to non-interpretable distances in this
space Ethayarajh, (2019). So for diachronic studies, this methodology is not
reliable as it gives many false positives Kutuzov et al., (2022).

Recent studies try to improve the isotropic orientation and sense similarity
(instead of contextual similarity) through contrastive learning, using
entailment Gao et al., (2021); Reimers and Gurevych, (2019) or pre-training on
similarity tasks Zhou et al., (2023). With a pre-training objective of
reducing the distance between similar senses in the semantic space, these
works improved pre-trained textual and sentence similarity models.

Another set of approaches attempts to model the polysemy of words directly. In
the characterization literature, some works used density-based embeddings
(word2gauss) to track this behavior. In Moss, (2020), the author selected some
properties of this embedding to measure the dimension of change. A more in
general review of embeddings capabilities and their limitations is presented
in Apidianaki, (2022).

###  4.5 Summary

Table 2: Comparison table between studies highlighting the type of methods utilized and category of change. We mark which kind of characterization the work conducts $\mathcal{D}$ for dimension, $\mathcal{R}$ for relation, and $\mathcal{O}$ for orientation. Also, we highlight the main representation method ($\mathfrak{R}$) for the word meaning. Methodology | $\mathcal{D}$ | $\mathcal{R}$ | $\mathcal{O}$ | $\mathfrak{R}$
---|---|---|---|---
Biemann, (2006) | X | - | - | Graph
Sagi et al., (2009) | X | - | - | Frequency
Lau et al., (2012) | X | - | - | Topics
Cook et al., (2013) | X | - | - | Topics
Tang et al., (2013) | X | X | - | Frequency
Carpuat et al., (2013) | X | - | - | Frequency
Lau et al., (2014) | X | - | - | Topics
Mitra et al., (2015) | X | - | - | Graph
Emms and Jayapal, (2016) | X | - | - | Frequency
Mathew et al., (2017) | X | - | - | Graph and Topics
Tahmasebi and Risse, (2017) | X | - | - | Graph
Ryskina et al., (2020) | X | - | - | Embeddings
Vijayarani and Geetha, (2020) | X | - | - | Embeddings
Moss, (2020) | X | - | - | Embeddings
Ehmüller et al., (2020) | X | - | - | Graph
Yüksel et al., (2021) | X | - | - | Embeddings
Bochkarev et al., (2022) | X | - | - | Frequency
Schlechtweg et al., (2017) | - | X | - | Frequency
Fonteyn and Manjavacas, (2021) | - | X | X | Embeddings
Maudslay and Teufel, (2022) | - | X | - | Embeddings
Cook and Stevenson, (2010) | - | - | X | Frequency
Jatowt and Duh, (2014) | - | - | X | Frequency
Buechel et al., (2016) | - | - | X | Frequency

Table 2 summarizes the literature survey did in the previous sections, for
semantic change characterization. In the rows we add the most relevant papers,
while the columns are the main properties of LSC evaluated in this survey. The
reference to the papers is in the first column. The second, third, and forth
columns represent the three poles of change characterization, i.e. dimension
($\mathcal{D}$), relation ($\mathcal{R}$), orientation ($\mathcal{O}$).
Finally, the technology adopted to compute and/or represent the sense
($\mathfrak{R}$) is in the fifth column. Notice that an “X” in a cell
indicates that the paper applied the sense computation technology (listed in
the fifth column) to analyze the corresponding “pole”.

In the table, we can observe that no methodology was used to characterize all
three poles, showing the complexity of this problem and the preference of the
authors for some specific problem. The majority of the studies were developed
to characterize changes in the dimension pole. It seems to be the pole where
the computational linguistic approaches get the best results. Another
observation from the table that deserve our attention is the still under-
explored use of graphs and topics methods to represent sense, compared to
embeddings and frequency. It is also important to notice that the most recent
works use embeddings, showing the current trends to this kind of approach.

Although computational linguistics has been used to characterize Lexical
Semantic Change (LSC) for over ten years (the first published work was in
2006, according to this survey), there is still no standardized formalization
of these changes. Each author proposes their own solution, and many of the
proposed interpretations of change are presented in the papers in a textual or
semi-formal format. To contribute to the discussions and to the definition of
semantic change characterization, we propose a set of formal descriptions of
change characterization and provide concrete examples of the subtasks required
to cope with the LSC problem in the next section.

##  5 Formalization

In statistics (and machine learning), the problem of drift distribution is
formally defined Gama et al., (2014). However, in linguistics, the definition
of word’s meaning (from a “semasiological” perspective) still relies on ideas
such as ‘concepts’ and ‘sentiments’ without providing objective definitions
for this terms Koch, (2016); Traugott, (2017); Blank et al., (2003), making a
formal definition of drift distribution more challenging. This vague notion of
change complicates the implementation of procedural methodologies to analyse,
compare, and evaluate LSC. In this section, we based our formalisation on the
definitions coming from the statistical domain.

First, we assume that for a set of corpora of text $T$ there is a universe of
senses $S_{T}$, that holds all the senses for that corpus. Given an alphabet
$\alpha$, we define $\mathcal{S}$ as a function that takes a string of the
closure $\alpha$ where $\alpha^{*}=V$ (the vocabulary) and returns a subset of
$S_{T}$. For instance, given a word $w\in V$ and a corpus $t\in T$ , the
subset of senses $S(w,t)$ for the universe of senses ($S_{t}$) of the word $w$
in this corpus is the following:

$\begin{split}\mathcal{S}:V\times T\rightarrow\wp(S_{t}),\\\
\mathcal{S}(w,t)=\\{s_{1},s_{2},...,s_{k}\\}.\end{split}$ (1)

We define that two sets of senses are different if, and only if, the
symmetrical difference between a set of senses for the word $a$ in a corpora
$x$ and the word $b$ in a corpora $y$ is non-empty, where $a,b\in V$ and
$x,y\in T$

$\begin{split}\mathcal{S}(a,x)&\neq\mathcal{S}(b,y)\iff\\\
(\mathcal{S}(a,x)-\mathcal{S}(b,y)\neq\emptyset)&\lor(\mathcal{S}(b,y)-\mathcal{S}(a,x)\neq\emptyset)\end{split}$
(2)

###  5.1 Semantic Change

We define that $w$ had a semantic change ($\mathcal{C}$), if the set of
senses, given by the function $\mathcal{S}$, evaluated in the corpus $t_{1}$
is different from the set of senses in $t_{2}$, given that $t_{1}\neq t_{2}$,

$\mathcal{C}(x,t_{1},t_{2})=\begin{cases}\text{True}&\mathcal{S}(x,t_{1})\neq\mathcal{S}(x,t_{2})\\\
\text{False}&\text{otherwise.}\end{cases}$ (3)

The subset of words in vocabulary $\bar{W}\subset V$ that suffered a semantic
change between corpus $t_{1}$ and $t_{2}$ is given by,

$\bar{W}_{t_{1},t_{2}}=\\{w\mid\mathcal{C}(w,t_{1},t_{2}),\hskip
10.00002pt\forall w\in V\\}.$ (4)

These changes in lexical meaning between corpus are caused by many different
phenomena, like time period (1930 and 2020, gay), culture (Brazil and
Portugal, rapariga), and domain (engineering and biology, cell).

###  5.2 Change in Dimension

We define that a word $w$ had a changed in dimension if the set of meanings
increased or decreased, with respect to the cardinality of the set, when
comparing two corpus $t_{1},t_{2}$. It means that the word suffered a
broadening or a narrowing change between these corpora.

$|\mathcal{S}(w,t_{1})|\neq|\mathcal{S}(w,t_{2})|,\hskip 10.00002ptw\in V.$
(5)

This definition is compatible with Koch, (2016) concept of
innovative/reductive meaning change, in which words acquire new senses
(sememes) by being used with different meaning. However, it differs for the
more general broadening notion, where words that undergo broadening may gain
extension, meaning that they are used in a broader sense than initially, but
they may also lose intention, which is the precision of the word’s meaning,
making it more vague.

Under this setting, split, birth and join with respect to a single lexical are
particular cases of change in dimension, e.g., birth or death is considered a
change from or to an empty set. Split and join are also instances of
broadening or narrowing with respect to a word. For instance, the word ‘plane’
after the 19th century gained the sense of an aeroplane, while previously was
only used to describe the act of planing, or a flat surface. This new sense
greatly increased the dimension of the word plane, as many correlated senses,
like “travel in an aeroplane”, was also added to plane senses.

In this survey, we define broadening as the degree of polysemy in a word. This
definition is aligned with the NLP literature, given its application in data
analysis and its measurable indicator of validity, compared to the notion of
intention. We differ from the notion defined in Tang et al., (2013), where
broadening is a superclass of metaphorization and metonymization, as the
literature reviewed here does not perform this aggregation.

###  5.3 Change in Relation

We define that a word $w$ had changed in relation, if the aggregated sum of
the relational similarity ($\mathit{l}:S\times S\rightarrow\mathbf{R}$)
between senses in $\mathcal{R}(w,t_{1})$ changed for $\mathcal{R}(w,t_{2})$.
Where $\mathit{l}$ takes two senses $s_{i},s_{j}$ and returns a real value,
that correspond to the material or abstract relation between these senses
(greater the value, more related the senses are), i.e.,

$\mathcal{R}(w,t)=\left\\{\begin{aligned}
\sum^{N-1}_{i=1}\sum^{N}_{j=i+1}{\mathit{l}(s_{i},s_{j})}\hskip
30.00005pts_{i,j}\in\mathcal{S}(w,t)\\\ 0\hskip 60.00009ptN\leq
1,\end{aligned}\right.$ (6)

where $N=|\mathcal{S}(w,t)|$.

Given a word $w$, we define that it increased in relation with respect to
corpus $t_{1},t_{2}$, if $\mathcal{R}(w,t_{1})<\mathcal{R}(w,t_{2})$, and
decreased if the opposite happens. This definition relies on the notion of
material or abstract similarity between senses that we still need to clarify
for figures of language, for instance, metaphor and metonymy.

The phenomenon of metaphorization or metonymization occurs when a word takes
on a new meaning that, to some extent, inherits qualities from its original
meaning through a figurative relationship the speaker aims to convey Lakoff
and Johnson, (2008). The original meaning is invoked by establishing a
tangible or abstract connection with the initial sense, where the listener is
able to solve the non-literal source concept.

A material relation is built when we recur to some cultural association
established between the concepts (source and target) that are interleaved
under a shared context (a frame) Koch, (2016), e.g., in the sentence “The
proposition made by Washington was rejected by the other countries”, we are
replacing the location of the institution by the institution itself. Other
forms of metonymy use cause and effect, contiguity, instrumentality,
possession, etc333Littlemore and Tagg, (2018) group the forms of metonymy as
‘referential,’ ‘propositional,’ ‘illocutionary,’ and ‘situational.’. So, for a
reader be able to solve this semantics, is mandatory a cultural knowledge of
the type of link between the source and target concepts that the speaker is
trying to evoke.

On the other hand, an abstract relation is achieved by associating properties
that evoke sensory experiences, such as emotions, shapes, dimensions,
positions, and more Lakoff and Johnson, (2008). In this case, the literal
sense is elucidated by delving into the figurative properties that the
listener can grasp within the sense itself.

The theory on how metonymies and metaphors are defined is ambiguous, where
generally relying on subjective perception, such as personal experience, or
definitions that overlap with other figurative modes. For example, Choi et
al., (2021) defines metaphor as “literal meaning of a word is different from
its contextual meaning”, which is also the case for metonymy. Many authors in
linguistic try to disambiguate the relation relying on imprecise definitions
Burke, (1969), and as discussed by Koch, (2016) there’s no precise rule to
define a metonymy. In Schlechtweg et al., (2017) the authors only measure the
entropy of a word as a means to capture metaphorization, it does not provide
evidences that this measure does not vary for the case of metonymization or a
new sense not related to the previous senses 444For example, the word ‘apple’
used in a technology context increases entropy but is not a metaphorization
process, but a new sense..

While we have no methods to perfectly differentiate the material and abstract
relation, there’s no excluding factor (a word can suffer metaphorization or
metonymization at the same time), and for many applications the relevance is
tracking if a word is being used with the literal or figurative meaning. We
decided to simplify the interpretation of these relations as following:
whether it increases in material or abstract relation, we consider it as
”increasing in relation”, if a new sense a word acquires is related to the
original sense, and “decreased in relation”, if it is less used in a
figurative manner.

###  5.4 Change in Orientation

We define that a word $w$ had changed in orientation, if for some notion of
orientation $\mathfrak{f}$ (feeling), there is a mapping from a word $w$ in a
corpora $t$ to an orientation value, for example, in case of positive or
negative feeling,

$\mathfrak{f}:V\times T\rightarrow\\{-1,+1\\},$ (7)

the orientation changes between corpora $t_{1},t_{2}$,

$\mathfrak{f}(w,t_{1})\neq\mathfrak{f}(w,t_{2}).$ (8)

Analogous to the previous definition, if
$\mathfrak{f}(w,t_{1})<\mathfrak{f}(w,t_{2})$ it suffered an amelioration, and
a pejoration if $\mathfrak{f}(w,t_{1})>\mathfrak{f}(w,t_{2})$.

In this task, we rely on a subjective notion of orientation addressed by a
function ($\mathfrak{f}$) that is arbitrarily defined. Generally, studies
define ‘orientation’ as the proximity to seed words with a positive or
negative connotation (a polarity setting) manually selected to keep the same
orientation over different corpora. However, the orientation problem is
broader than just polarity, as we can classify words in more aspects than
positive and negative feelings.

###  5.5 Semantic Change Identification

Consider a function $f_{\mathcal{C}}$ that takes two representations
$\mathfrak{R}$ (e.g., embeddings, graphs, etc.) of the word $w$ in
$t_{1},t_{2}$. We define the problem of semantic change identification as
assigning a label (changed or not changed) to this word,

$f_{\mathcal{C}}(\mathfrak{R}(w,t_{1}),\mathfrak{R}(w,t_{2}))\rightarrow\\{0,1\\},$
(9)

which tries to approximate the set of words $\bar{W}_{t_{1},t_{2}}$ where we
can measure some notion of error, e.g., accuracy.

###  5.6 Semantic Change Characterization

We define semantic change characterization as a classification problem where
we need to assign identified semantic changes to categories, such as
$f_{\mathcal{D}},f_{\mathcal{R}},f_{\mathcal{O}}$,

$f_{x}(\mathfrak{R}(w,t_{1}),\mathfrak{R}(w,t_{2}))\rightarrow\\{0,1\\},\hskip
20.00003ptx\in\\{\mathcal{D},\mathcal{R},\mathcal{O}\\}.$ (10)

##  6 Discussion

In this section, we synthesize the reviewed approaches for change
characterization and discuss their limitations according to the dimension,
relation and orientation aspects previously introduced and formalized.

###  6.1 Dimension

The majority of the studies found during this survey propose methods and tools
to characterize how words change in dimension, while the other poles were less
studied. Change in dimension seems to be easier to detect than the other
categories because it does not rely on subjective interpretation, such as
relation or feeling, but mainly on countable quantity (see Section 2.2). Works
along this axis often rely on word sense induction, a research area with well
developed tooling and many data sources, such as dictionaries and synsets,
used to implement supervised learning.

A major characteristic that differentiate the approaches is the word
representation method. We identified four main types of word representation
methods, used to study dimension changes in LSC. These methods have different
trade-offs in implementation cost, representation power, and interpretability.
It is difficult to compare the effectiveness of each method for this state-of-
the-art, given the qualitative and exploratory nature of the reported results,
the use of different corpora (not always available), and the need for external
knowledge to acquire meaning (difficult to reproduce). However, the analysis
reveals that word frequency and embeddings-based methods produce better
results to identify neologism (new words), while topics and graph-based
approaches are more effective at detecting sememes (new senses) over time. A
limitation for the approaches implemented based on dictionaries is that the
meaning of new words are not always in the dictionaries because there is a
temporal delay between the first occurrences of these neologisms and their
publication in dictionaries. Another limitation comes with the presence, in
dictionaries, of metaphorical senses at the same level as the literal ones,
increasing the complexity to make the distinction between ‘dimension’ and
‘relation’. Some authors prefer to group these two poles into one, as in Tang
et al., (2013) . The limitation for topics- and graph-based approaches is that
the interpretation of the meaning of a topic (or a branch of a graph) is not
always clear and can require the intervention of an expert to, for instance,
to regroup topics/branches with the same meaning and/or to map these groups to
word’s senses.

###  6.2 Relation

Outside of this LSC survey, there is a vast body of work proposing methods and
tools to predict if a word in a sentence has a metaphorical sense Heintz et
al., (2013); Shutova and Sun, (2013); Hovy et al., (2013). These approaches
analyze texts, but they do not consider the temporal information, thus they
cannot express if a word suffers from metaphorization over time. Some works
claim to detect metaphorization of words (e.g., Tredici et al., (2016) ), but
they lay on the general hypothesis that changing sense of words implies in a
metaphorical change, which still needs to be demonstrated.

For metonymization, we could not find studies that performed metonymy
identification in a comparative setting. However, related recent studies
investigate the usage of metonymy within different methods to build the
material relation identification Nastase et al., (2012); Soares et al.,
(2019); Mathews and Strube, (2021). We expect that future investigations can
extend these approaches to allow detecting metonymization of words.

Among the four meaning representation methods analyzed, frequency-based
approaches were the first to be investigated and provided quite good outcomes.
However, the hypothesis used by the authors to define a relation is quite
similar with the one for dimension, what makes the interpretation of the ‘good
outcomes’ more difficult. Embedding is gaining the attention of the community
and in recent work, this kind of method has be widely adopted, but the correct
interpretation of the published outcomes suffers from the same difficulty
observed for the frequency-based approaches. Although the known quality of
graphs to well represent relations, this method is still under-investigated to
purpose of relation change identification.

Tang et al., (2013) mainly rely on dictionaries of heuristics to infer
metaphoricity, which served for qualitative analysis, but still require more
extensive validation. A single work used ground-truth annotations for
metaphors Schlechtweg et al., (2017), but did not reported a final precision
of the proposed methodology.

As noticed, studies about the ‘relation pole’ are few and not conclusive. The
low interest of the research community on this topic could be potentially
justified by three reasons: (I) The difficulty to formally define, from a
computational perspective, what metaphor and metonym are. (II) The rare and
insufficient quantity of annotated corpus for relation detection. (III) The
nonexistence of standard metrics to measure (and compare) approaches dealing
with the detection and characterization of material and abstract relations.

###  6.3 Orientation

The interest in orientation analysis is increasing with the wide adoption of
social networks as media for advertising products and companies. Orientation
is usually determined by using a feeling function ($\mathfrak{f}$) that relies
on seed words and on the assumption that this seed words do not vary over
time. As we illustrate in this survey, the meaning of words evolve over time
and can be different between communities. The example of the word sick shows
that it can have two opposite orientation according to the context of use.
Creating corpus that include (with statistical significance) all these
variations is a complex task. In general, we can observe that the annotation
process could suffer from many biases, both from the annotators and from the
data. The selected seed words could not generalize to be applied to all
contexts or in long moments in time. Moreover, a series of misleading results
could incur based on seed words, as presented in Antoniak and Mimno, (2021).

From the analysis of the four methods used to represent the meaning,
embeddings and frequency-based approaches are the most commonly adopted. Few
works use graphs and we could not find any approach that uses topic analysis
for detecting the orientation of words. The main reason for the preference for
embedding and frequency methods is that current approaches compute the
distances or co-occurrence of a word with seed words. It is easier and faster
to do this using embedding and frequency-based approaches. Graphs and topics
are not designed to determine this and require more complex reasoning and
assumptions.

Although there is a huge body of work proposing solutions for analysing
sentiment at sentence level, fewer of them try to infer sentiments at a word-
level Guo et al., (2021), and in the word-level, only a small fraction is
performed on a comparative manner, as we accessed in this survey.

An important outcome of this survey is that there is an harmonization on how
change in feeling are defined and applied. However, we could not observe that
there is also an harmonization on how the error analysis for the feeling
function is implemented. Some works adopt binary orientation function
(positive, negative), while other multidimensional orientations (valence,
arousal, dominance) Buechel et al., (2016). There is still further study to
analyze which function ($\mathfrak{f}$) is more suitable for characterizing
change.

From the many approaches analysed in this survey, one can observed some
converging points. This is the case for instance for the difficulty to
evaluate and compare the obtained results because, for the majority of the
tasks, no gold standard exists. This problem has been raised in many papers
like Schlechtweg et al., (2017), but some initiatives to create reference
datasets are under developing.

Another interesting point is the multilingualism aspect. In the vast majority
of cases, only English documents are used for the experiments, which makes it
difficult to generalize the proposed methods to other languages with different
roots and grammatical rules.

The recent emergence of new-generation of language models has opened up new
possibilities for the identification and characterization of semantic changes.
These models’ ability to handle large corpora of documents and to deal with
contextual information expressed in different languages is important for the
analysis of semantic drift. However, this area of research is still under-
explored.

##  7 Conclusion

In this survey we reviewed current works on semantic change characterization,
with respect to the mainstream typology, (broadening, narrowing, amelioration,
pejoration, metaphorization and metonymization). We observed four major
approaches to represent meaning (frequency, topics, embeddings, and graphs)
and respective strengths and drawbacks.

After an analysis of the literature we observed that many tasks are under-
explored or addressed without a systematic evaluation. So, we introduced a
formal definition for the task of semantic change characterization, which is a
more complex instance of the semantic change identification problem. This new
task is of extreme relevance for the field of LSC as this study and other
surveys defended.

The suggested tasks are crucial for both linguistic change research and NLP;
numerous tasks, such as sentiment analysis (orientation), semantic similarity
(relation), and word sense inference (dimension), call for a deeper
understanding of these characteristics of representation. Additionally,
improved knowledge and interpretability of these systems result from an
awareness of how changes affect the representation of words. Specially for
word embeddings, better understanding how this changes affect the geometries
could improve our knowledge in the semantics of vectors.

As a future work we expect to investigate the relation pole to better
differentiate the categories of change, and further explore the four word
representational methods, how they affect and are related to to the three
poles of change. Another important improvement is a consolidation of datasets
and metrics where semantic change characterization could be compared across
methodologies.

Finally, the emergence of new-generation language models is acknowledged as
opening up new possibilities for identifying and characterizing semantic
changes, though this area of research is still under-explored. The conclusion
suggests avenues for future research, addressing the identified challenges and
leveraging the capabilities of advanced language models.

## Acknowledgments

The research reported in this publication was supported by the Luxembourg
National Research Fund (FNR), project D4H grant number PRIDE21/16758026

## References

  * Allan, (2013) Allan, K. 2013\.  The Oxford handbook of the history of linguistics.  OUP Oxford.
  * Allan and Robinson, (2012) Allan, K. and Robinson, J. A. 2012\.  Current methods in historical semantics.  De Gruyter Mouton Berlin.
  * Antoniak and Mimno, (2021) Antoniak, M. and Mimno, D. M. 2021\.  Bad seeds: Evaluating lexical methods for bias measurement.  In Annual Meeting of the Association for Computational Linguistics.
  * Apidianaki, (2022) Apidianaki, M. 2022\.  From word types to tokens and back: A survey of approaches to word meaning representation and interpretation.  Computational Linguistics, 49:465–523.
  * Ardley, (1968) Ardley, G. 1968\.  God and the soul.  Philosophical Studies, 18:325–326.
  * Barnes et al., (2021) Barnes, J., Kurtz, R., Oepen, S., Ovrelid, L., and Velldal, E. 2021\.  Structured sentiment analysis as dependency graph parsing.  ArXiv, abs/2105.14504.
  * Biemann, (2006) Biemann, C. 2006\.  Chinese whispers - an efficient graph clustering algorithm and its application to natural language processing problems.  In Proceedings of TextGraphs: the First Workshop on Graph Based Methods for Natural Language Processing, pp. 73–80, New York City. Association for Computational Linguistics.
  * Blank et al., (2003) Blank, A. and others 2003\.  Polysemy in the lexicon and in discourse.  Polysemy: Flexible patterns of meaning in mind and language.
  * Bochkarev et al., (2022) Bochkarev, V. V., Khristoforov, S., Shevlyakova, A. V., and Solovyev, V. D. 2022\.  Neural network algorithm for detection of new word meanings denoting named entities.  IEEE Access, PP:1–1.
  * Bowern, (2019) Bowern, C. 2019\.  Semantic change and semantic stability: Variation is key.  In LChange@ACL.
  * Brezina, (2018) Brezina, V. 2018\.  Statistics in Corpus Linguistics: A Practical Guide.  Cambridge University Press.
  * Broad, (1938) Broad, C. D. 1938\.  Examination of McTaggart’s Philosophy, volume Vol. II.  Cambridge University Press.
  * Buechel et al., (2016) Buechel, S., Hellrich, J., and Hahn, U. 2016\.  Feelings from the past—adapting affective lexicons for historical emotion analysis.  In LT4DH@COLING.
  * Burke, (1969) Burke, K. 1969\.  A grammar of motives, volume 177.  Univ of California Press.
  * Camacho-Collados and Pilehvar, (2018) Camacho-Collados, J. and Pilehvar, M. T. 2018\.  From word to sense embeddings: A survey on vector representations of meaning.  J. Artif. Intell. Res., 63:743–788.
  * Carpuat et al., (2013) Carpuat, M., Daumé, H., Henry, K., Irvine, A., Jagarlamudi, J., and Rudinger, R. 2013\.  Sensespotting: Never let your parallel data tie you to an old domain.  In Annual Meeting of the Association for Computational Linguistics.
  * Cheng et al., (2006) Cheng, W., Greaves, C., and Warren, M. 2006\.  From n-gram to skipgram to concgram.  International journal of corpus linguistics, 11(4):411–433.
  * Choi et al., (2021) Choi, M., Lee, S., Choi, E., Park, H., Lee, J., Lee, D., and Lee, J. 2021\.  MelBERT: Metaphor detection via contextualized late interaction using metaphorical identification theories.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1763–1773, Online. Association for Computational Linguistics.
  * Cook et al., (2013) Cook, P., Lau, J. H., Rundell, M., McCarthy, D., and Baldwin, T. 2013\.  A lexicographic appraisal of an automatic approach for detecting new word-senses.  In Proceedings of eLex.
  * Cook and Stevenson, (2010) Cook, P. and Stevenson, S. 2010\.  Automatically identifying changes in the semantic orientation of words.  In International Conference on Language Resources and Evaluation.
  * Danilevsky et al., (2020) Danilevsky, M., Qian, K., Aharonov, R., Katsis, Y., Kawas, B., and Sen, P. 2020\.  A survey of the state of explainable ai for natural language processing.  In AACL.
  * Devlin et al., (2019) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. 2019\.  Bert: Pre-training of deep bidirectional transformers for language understanding.  ArXiv, abs/1810.04805.
  * Dubossarsky et al., (2017) Dubossarsky, H., Weinshall, D., and Grossman, E. 2017\.  Outta control: Laws of semantic change and inherent biases in word representation models.  In Conference on Empirical Methods in Natural Language Processing.
  * Ehmüller et al., (2020) Ehmüller, J., Kohlmeyer, L., McKee, H., Paeschke, D., Repke, T., Krestel, R., and Naumann, F. 2020\.  Sense tree: Discovery of new word senses with graph-based scoring.  In Lernen, Wissen, Daten, Analysen.
  * Emms and Jayapal, (2016) Emms, M. and Jayapal, A. K. 2016\.  Dynamic generative model for diachronic sense emergence detection.  In International Conference on Computational Linguistics.
  * Ethayarajh, (2019) Ethayarajh, K. 2019\.  How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings.  In Conference on Empirical Methods in Natural Language Processing.
  * Fonteyn and Manjavacas, (2021) Fonteyn, L. and Manjavacas, E. 2021\.  Adjusting scope: A computational approach to case-driven research on semantic change.  In Workshop on Computational Humanities Research.
  * Gama et al., (2014) Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., and Bouchachia, A. 2014\.  A survey on concept drift adaptation.  ACM Computing Surveys (CSUR), 46:1 – 37.
  * Gao et al., (2021) Gao, T., Yao, X., and Chen, D. 2021\.  Simcse: Simple contrastive learning of sentence embeddings.  ArXiv, abs/2104.08821.
  * Giulianelli et al., (2023) Giulianelli, M., Luden, I., Fernández, R., and Kutuzov, A. 2023\.  Interpretable word sense representations via definition generation: The case of semantic change analysis.  In Annual Meeting of the Association for Computational Linguistics.
  * Gladkova and Drozd, (2016) Gladkova, A. and Drozd, A. 2016\.  Intrinsic evaluations of word embeddings: What can we do better?  In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval@ACL 2016, Berlin, Germany, August 2016, pp. 36–42. Association for Computational Linguistics.
  * Günther et al., (2019) Günther, F., Rinaldi, L., and Marelli, M. 2019\.  Vector-space models of semantic representation from a cognitive perspective: A discussion of common misconceptions.  Perspectives on Psychological Science, 14:1006 – 1033.
  * Guo et al., (2021) Guo, X., Yu, W., and Wang, X. 2021\.  An overview on fine-grained text sentiment analysis: Survey and challenges.  Journal of Physics: Conference Series, 1757.
  * Hamilton et al., (2016) Hamilton, W. L., Leskovec, J., and Jurafsky, D. 2016\.  Diachronic word embeddings reveal statistical laws of semantic change.  ArXiv, abs/1605.09096.
  * Hammarström et al., (2019) Hammarström, H., Rönchen, P., Elgh, E., and Wiklund, T. 2019\.  On computational historical linguistics in the 21st century.  Theoretical Linguistics, 45:233 – 245.
  * He et al., (2021) He, P., Gao, J., and Chen, W. 2021\.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.  ArXiv, abs/2111.09543.
  * Heintz et al., (2013) Heintz, I., Gabbard, R., Srivastava, M., Barner, D., Black, D., Friedman, M., and Weischedel, R. M. 2013\.  Automatic extraction of linguistic metaphors with lda topic modeling.  In Proceedings of the First Workshop on Metaphor in NLP, pp. 58–66.
  * Hellrich and Hahn, (2016) Hellrich, J. and Hahn, U. 2016\.  Bad company - neighborhoods in neural embedding spaces considered harmful.  In Calzolari, N., Matsumoto, Y., and Prasad, R., editors, COLING 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan, pp. 2785–2796. ACL.
  * Hengchen et al., (2021) Hengchen, S., Tahmasebi, N., Schlechtweg, D., and Dubossarsky, H. 2021\.  Challenges for computational lexical semantic change.  ArXiv, abs/2101.07668.
  * Hock and Joseph, (2019) Hock, H. H. and Joseph, B. D. 2019\.  Chapter 7: Semantic change.  De Gruyter Mouton, Berlin, Boston.
  * Hovy et al., (2013) Hovy, D., Shrivastava, S., Jauhar, S. K., Sachan, M., Goyal, K., Li, H., Sanders, W. E., and Hovy, E. H. 2013\.  Identifying metaphorical word use with tree kernels.  In Proceedings of the First Workshop on Metaphor in NLP.
  * Hu et al., (2019) Hu, R., Li, S., and Liang, S. 2019\.  Diachronic sense modeling with deep contextualized word embeddings: An ecological view.  In Annual Meeting of the Association for Computational Linguistics.
  * Ito et al., (2020) Ito, T., Tsubouchi, K., Sakaji, H., Yamashita, T., and Izumi, K. 2020\.  Word-level contextual sentiment analysis with interpretability.  In AAAI Conference on Artificial Intelligence.
  * Jatowt and Duh, (2014) Jatowt, A. and Duh, K. 2014\.  A framework for analyzing semantic change of words across time.  IEEE/ACM Joint Conference on Digital Libraries, pp. 229–238.
  * Juvonen and Koptjevskaja-Tamm, (2016) Juvonen, P. and Koptjevskaja-Tamm, M. 2016\.  The lexical typology of semantic shifts, volume 58.  Walter de Gruyter GmbH & Co KG.
  * Koch, (2016) Koch, P. 2016\.  Meaning change and semantic shifts.  The lexical typology of semantic shifts, 58:21–66.
  * Kutuzov et al., (2018) Kutuzov, A., Øvrelid, L., Szymanski, T., and Velldal, E. 2018\.  Diachronic word embeddings and semantic shifts: a survey.  In International Conference on Computational Linguistics.
  * Kutuzov et al., (2022) Kutuzov, A., Velldal, E., and Øvrelid, L. 2022\.  Contextualized embeddings for semantic change detection: Lessons learned.  ArXiv, abs/2209.00154.
  * Lakoff and Johnson, (2008) Lakoff, G. and Johnson, M. 2008\.  Metaphors we live by.  University of Chicago press.
  * Lau et al., (2014) Lau, J. H., Cook, P., McCarthy, D., Gella, S., and Baldwin, T. 2014\.  Learning word sense distributions, detecting unattested senses and identifying novel senses using topic models.  In Annual Meeting of the Association for Computational Linguistics.
  * Lau et al., (2012) Lau, J. H., Cook, P., McCarthy, D., Newman, D., and Baldwin, T. 2012\.  Word sense induction for novel sense detection.  In Conference of the European Chapter of the Association for Computational Linguistics.
  * Leszczynski et al., (2020) Leszczynski, M., May, A., Zhang, J., Wu, S., Aberger, C. R., and Ré, C. 2020\.  Understanding the downstream instability of word embeddings.  CoRR, abs/2003.04983.
  * Littlemore and Tagg, (2018) Littlemore, J. and Tagg, C. 2018\.  Metonymy and text messaging: A framework for understanding creative uses of metonymy.  Applied Linguistics, 39(4):481–507.
  * Liu et al., (2022) Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. 2022\.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.  ArXiv, abs/2205.05638.
  * Liu et al., (2015) Liu, Y., Liu, Z., Chua, T.-S., and Sun, M. 2015\.  Topical word embeddings.  In AAAI Conference on Artificial Intelligence.
  * Lu et al., (2022) Lu, Y., Cheng, X., Liang, Z., and Rao, Y. 2022\.  Graph-based dynamic word embeddings.  In International Joint Conference on Artificial Intelligence.
  * Mathew et al., (2017) Mathew, B., Maity, S. K., Sarkar, P., Mukherjee, A., and Goyal, P. 2017\.  Adapting predominant and novel sense discovery algorithms for identifying corpus-specific sense differences.  In Proceedings of TextGraphs-11: the Workshop on Graph-based Methods for Natural Language Processing, pp. 11–20, Vancouver, Canada. Association for Computational Linguistics.
  * Mathews and Strube, (2021) Mathews, K. A. and Strube, M. 2021\.  Impact of target word and context on end-to-end metonymy detection.  ArXiv, abs/2112.03256.
  * Maudslay and Teufel, (2022) Maudslay, R. H. and Teufel, S. 2022\.  Metaphorical polysemy detection: Conventional metaphor meets word sense disambiguation.  ArXiv, abs/2212.08395.
  * May et al., (2019) May, A., Zhang, J., Dao, T., and Ré, C. 2019\.  On the downstream performance of compressed word embeddings.  CoRR, abs/1909.01264.
  * McCarthy et al., (2004) McCarthy, D., Koeling, R., Weeds, J., and Carroll, J. 2004\.  Finding predominant word senses in untagged text.  In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pp. 279–286, Barcelona, Spain.
  * McWhorter, (2011) McWhorter, J. 2011\.  The power of Babel: A natural history of language.  Random House.
  * Merriam-Webster, (2023) Merriam-Webster 2023\.  ’sick,’ entry 6.  https://www.merriam-webster.com/dictionary/sick.  Accessed: 2023-09-15.
  * Mickus et al., (2020) Mickus, T., Paperno, D., Constant, M., and van Deemter, K. 2020\.  What do you mean, BERT?  In Proceedings of the Society for Computation in Linguistics 2020, pp. 279–290, New York, New York. Association for Computational Linguistics.
  * Mikolov et al., (2013) Mikolov, T., Chen, K., Corrado, G. S., and Dean, J. 2013\.  Efficient estimation of word representations in vector space.  In International Conference on Learning Representations.
  * Min et al., (2022) Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. 2022\.  Rethinking the role of demonstrations: What makes in-context learning work?  In Conference on Empirical Methods in Natural Language Processing.
  * Mitchell and Lapata, (2010) Mitchell, J. and Lapata, M. 2010\.  Composition in distributional models of semantics.  Cognitive science, 34 8:1388–429.
  * Mitra et al., (2015) Mitra, S., Mitra, R., Maity, S. K., Riedl, M., Biemann, C., Goyal, P., and Mukherjee, A. 2015\.  An automatic approach to identify word sense changes in text media across timescales.  Natural Language Engineering, 21:773 – 798.
  * Mohammed, (2009) Mohammed, E. T. 2009\.  Polysemy as a lexical problem in translation.  Online Submission, 55:1–19.
  * Montanelli and Periti, (2023) Montanelli, S. and Periti, F. 2023\.  A survey on contextualised semantic shift detection.  ArXiv, abs/2304.01666.
  * Moss, (2020) Moss, A. 2020\.  Detecting lexical semantic change using probabilistic gaussian word embeddings.
  * Nastase et al., (2012) Nastase, V., Judea, A., Markert, K., and Strube, M. 2012\.  Local and global context for supervised and unsupervised metonymy resolution.  In Conference on Empirical Methods in Natural Language Processing.
  * Neelakantan et al., (2014) Neelakantan, A., Shankar, J., Passos, A., and McCallum, A. 2014\.  Efficient non-parametric estimation of multiple embeddings per word in vector space.  ArXiv, abs/1504.06654.
  * Neubig and Hu, (2018) Neubig, G. and Hu, J. 2018\.  Rapid adaptation of neural machine translation to new languages.  ArXiv, abs/1808.04189.
  * Nevalainen and Traugott, (2016) Nevalainen, T. and Traugott, E. C. 2016\.  The Oxford handbook of the history of English.  Oxford University Press.
  * Nivre et al., (2020) Nivre, J., de Marneffe, M.-C., Ginter, F., Hajivc, J., Manning, C. D., Pyysalo, S., Schuster, S., Tyers, F. M., and Zeman, D. 2020\.  Universal dependencies v2: An evergrowing multilingual treebank collection.  In International Conference on Language Resources and Evaluation.
  * Pedinotti and Lenci, (2020) Pedinotti, P. and Lenci, A. 2020\.  Don’t invite BERT to drink a bottle: Modeling the interpretation of metonymies using BERT and distributional representations.  In Proceedings of the 28th International Conference on Computational Linguistics, pp. 6831–6837, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Peters et al., (2018) Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. 2018\.  Deep contextualized word representations.  In North American Chapter of the Association for Computational Linguistics.
  * Pinker, (2003) Pinker, S. 2003\.  The language instinct: How the mind creates language.  Penguin UK.
  * Pinker and Bloom, (1990) Pinker, S. and Bloom, P. J. 1990\.  Natural language and natural selection.  Behavioral and Brain Sciences, 13:707 – 727.
  * Raffel et al., (2019) Raffel, C., Shazeer, N. M., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. 2019\.  Exploring the limits of transfer learning with a unified text-to-text transformer.  J. Mach. Learn. Res., 21:140:1–140:67.
  * Reagan et al., (2017) Reagan, A. J., Danforth, C. M., Tivnan, B. F., Williams, J. R., and Dodds, P. S. 2017\.  Sentiment analysis methods for understanding large-scale texts: a case for using continuum-scored words and word shift graphs.  EPJ Data Science, 6:1–21.
  * Reimers and Gurevych, (2019) Reimers, N. and Gurevych, I. 2019\.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In Conference on Empirical Methods in Natural Language Processing.
  * Rilling et al., (2012) Rilling, J. K., Glasser, M. F., Jbabdi, S., Andersson, J., and Preuss, T. M. 2012\.  Continuity, divergence, and the evolution of brain language pathways.  Frontiers in evolutionary neuroscience, 3:11.
  * Rosenfeld and Erk, (2018) Rosenfeld, A. and Erk, K. 2018\.  Deep neural models of semantic shift.  In North American Chapter of the Association for Computational Linguistics.
  * Ryskina et al., (2020) Ryskina, M., Rabinovich, E., Berg-Kirkpatrick, T., Mortensen, D. R., and Tsvetkov, Y. 2020\.  Where new words are born: Distributional semantic analysis of neologisms and their semantic neighborhoods.  ArXiv, abs/2001.07740.
  * Sagi et al., (2009) Sagi, E., Kaufmann, S., and Clark, B. 2009\.  Semantic density analysis: Comparing word meaning across time and phonetic space.  In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, GEMS ’09, 104–111, USA. Association for Computational Linguistics.
  * Schlechtweg et al., (2017) Schlechtweg, D., Eckmann, S., Santus, E., im Walde, S. S., and Hole, D. 2017\.  German in flux: Detecting metaphoric change via word entropy.  In Conference on Computational Natural Language Learning.
  * Schlechtweg et al., (2019) Schlechtweg, D., Hätty, A., Tredici, M. D., and im Walde, S. S. 2019\.  A wind of change: Detecting and evaluating lexical semantic change across times and domains.  In Annual Meeting of the Association for Computational Linguistics.
  * Schlechtweg et al., (2020) Schlechtweg, D., McGillivray, B., Hengchen, S., Dubossarsky, H., and Tahmasebi, N. 2020\.  SemEval-2020 task 1: Unsupervised lexical semantic change detection.  In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp. 1–23, Barcelona (online). International Committee for Computational Linguistics.
  * Sherwood et al., (2008) Sherwood, C. C., Subiaul, F., and Zawidzki, T. W. 2008\.  A natural history of the human mind: tracing evolutionary changes in brain and cognition.  Journal of Anatomy, 212.
  * Shoemark et al., (2019) Shoemark, P., Liza, F. F., Nguyen, D., Hale, S. A., and McGillivray, B. 2019\.  Room to glo: A systematic comparison of semantic change detection approaches with word embeddings.  In Conference on Empirical Methods in Natural Language Processing.
  * Shutova and Sun, (2013) Shutova, E. and Sun, L. 2013\.  Unsupervised metaphor identification using hierarchical graph factorization clustering.  In North American Chapter of the Association for Computational Linguistics.
  * Soares et al., (2019) Soares, L. B., FitzGerald, N., Ling, J., and Kwiatkowski, T. 2019\.  Matching the blanks: Distributional similarity for relation learning.  In Annual Meeting of the Association for Computational Linguistics.
  * (95) Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. 2013a.  Recursive deep models for semantic compositionality over a sentiment treebank.  In Conference on Empirical Methods in Natural Language Processing.
  * (96) Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. 2013b.  Recursive deep models for semantic compositionality over a sentiment treebank.  In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Tahmasebi et al., (2018) Tahmasebi, N., Borin, L., and Jatowt, A. 2018\.  Survey of computational approaches to lexical semantic change.  arXiv: Computation and Language.
  * Tahmasebi et al., (2021) Tahmasebi, N., Borin, L., Jatowt, A., Xu, Y., and Hengchen, S. 2021\.  Computational approaches to semantic change.  BoD–Books on Demand.
  * Tahmasebi and Risse, (2017) Tahmasebi, N. and Risse, T. 2017\.  Finding individual word sense changes and their delay in appearance.  In Recent Advances in Natural Language Processing.
  * Takamura et al., (2017) Takamura, H., Nagata, R., and Kawasaki, Y. 2017\.  Analyzing semantic change in japanese loanwords.  In Conference of the European Chapter of the Association for Computational Linguistics.
  * Tang, (2018) Tang, X. 2018\.  A state-of-the-art of semantic change computation.  Natural Language Engineering, 24:649 – 676.
  * Tang et al., (2013) Tang, X., Qu, W., and Chen, X. 2013\.  Semantic change computation: A successive approach.  World Wide Web, 19:375–415.
  * Tang et al., (2023) Tang, X., Zhou, Y., Aida, T., Sen, P., and Bollegala, D. 2023\.  Can word sense distribution detect semantic changes of words?  arXiv preprint arXiv:2310.10400.
  * Teraoka, (2016) Teraoka, T. 2016\.  Metonymy analysis using associative relations between words.  In International Conference on Language Resources and Evaluation.
  * Traugott, (2017) Traugott, E. C. 2017\.  Semantic change.
  * Tredici et al., (2016) Tredici, M. D., Nissim, M., and Zaninello, A. 2016\.  Tracing metaphors in time through self-distance in vector spaces.  ArXiv, abs/1611.03279.
  * Uban et al., (2021) Uban, A.-S., Ciobanu, A. M., and Dinu, L. P. 2021\.  Cross-lingual laws of semantic change.  Computational approaches to semantic change, 6:219.
  * Vijayarani and Geetha, (2020) Vijayarani, J. and Geetha, T. V. 2020\.  Knowledge-enhanced temporal word embedding for diachronic semantic change estimation.  Soft Computing, 24:12901 – 12918.
  * Vulic et al., (2020) Vulic, I., Ruder, S., and Søgaard, A. 2020\.  Are all good word vector spaces isomorphic?  ArXiv, abs/2004.04070.
  * Wang et al., (2021) Wang, B., Buccio, E. D., and Melucci, M. 2021\.  Word2fun: Modelling words as functions for diachronic word representation.  In Neural Information Processing Systems.
  * Wankhade et al., (2022) Wankhade, M., Rao, A. C. S., and Kulkarni, C. 2022\.  A survey on sentiment analysis methods, applications, and challenges.  Artificial Intelligence Review, 55:5731 – 5780.
  * Wendlandt et al., (2018) Wendlandt, L., Kummerfeld, J. K., and Mihalcea, R. 2018\.  Factors influencing the surprising instability of word embeddings.  CoRR, abs/1804.09692.
  * Yüksel et al., (2021) Yüksel, A., Ugurlu, B., and Koç, A. 2021\.  Semantic change detection with gaussian word embeddings.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3349–3361.
  * Zhou et al., (2021) Zhou, K., Ethayarajh, K., and Jurafsky, D. 2021\.  Frequency-based distortions in contextualized word embeddings.  ArXiv, abs/2104.08465.
  * Zhou et al., (2023) Zhou, W., Tahmasebi, N., and Dubossarsky, H. 2023\.  The finer they get: Combining fine-tuned models for better semantic change detection.  In Nordic Conference of Computational Linguistics.
  * Zipf, (1949) Zipf, G. K. 1949\.  Human behavior and the principle of least effort.  Addison-Wesley Press, Inc.

  *[NLP]: Natural Language Processing
  *[LSC]: Lexical Semantic Change