# Do Word Embeddings Really Understand Loughran-McDonald’s Polarities?

Mengda Li and Charles-Albert Lehalle

###### Abstract

In this paper we perform a rigorous mathematical analysis of the word2vec
model, especially when it is equipped with the Skip-gram learning scheme. Our
goal is to explain how embeddings, that are now widely used in NLP (Natural
Language Processing), are influenced by the distribution of terms in the
documents of the considered corpus. We use a mathematical formulation to
understand how this family of language models captures the joined distribution
of terms in sentences. This formulation shed light on how the decision to use
such a model makes implicit assumptions on the structure of the language, in
particular about the short range and long range dependencies between terms. We
show how Markovian assumptions, that we discuss, lead to a very clear
theoretical understanding of the formation of embeddings, and in particular
the way it captures what we call _frequentist synonyms_. These assumptions
allow to produce generative models and to conduct an explicit analysis of the
loss function commonly used by these NLP techniques, that asymptotically
reaches a cross-entropy between the language model and the underlying true
generative model.

Moreover, having in mind to assess the identifiability of embeddings, we
produce synthetic corpora with different levels of structure and show
empirically how the word2vec algorithm succeed, or not, to learn them. It
leads us to empirically assess the capability of such models to capture
structures on a corpus of around 42 millions of financial News covering 12
years. That for, we rely on the Loughran-McDonald Sentiment Word Lists largely
used on financial texts and we show that embeddings are exposed to mixing
terms with opposite polarity, because of the way they can treat antonyms as
frequentist synonyms. Beside we study the non-stationarity of such a financial
corpus, that has surprisingly not be documented in the literature. We do it
via time series of cosine similarity between groups of polarised words or
company names, and show that embedding are indeed capturing a mix of English
semantics and joined distribution of words that is difficult to disentangle.

##### Acknowledgments.

Authors would like to thank Sylvain Champonnois for deep discussions about the
nature of text polarity and biases of embeddings, Jean-Charles Nigretto for
preliminary work on biases in doc2vec models, Elise Tellier for long
discussions about the optimal size of embeddings, Laurent El Ghaoui for
challenging discussion about the representation of language models by
embeddings, and Gérard Ben Arous for discussions on the convergence of
criteria and the choice of _observables_ in the empirical analysis of word
embeddings.

##  1 Positioning of this work

Recent advances on Natural Language Processing (NLP) are largely based on the
use of embeddings. It started with performing SVD on vectorial representation
of words (see for instance [1]) before turning to less linear analysis like
the use of Self Organizing Maps [2]. More recently, the term of “embeddings”
emerged with the idea of learning a lookup table while performing downstream
tasks [3]; later on downstream tasks used LSTM (Long Short Term Memory)
networks to account for the sequential aspects of the natural languages [4].
Now BERT-like models as other attention-based models proposed to plug
embeddings in sequences of local LSTM structures, to obtain more contextual
(i.e. bi-directional) Euclidean representations of terms [5, 6].

Despite the apparent successes of this series of improvements, very few
mathematical modeling has been proposed. Some interesting papers are
descriptive, exposing the formulas behind the mechanisms of embeddings, like
[7] that investigate a SVD viewpoint on embeddings, but they do not provide a
probabilistic model corresponding to the implicit assumptions made by the
considered NLP modeling algorithm. As a comparison, the literature on topic
extraction naturally provides this kind of generative models [8].

The goal of this paper is to provide such a mathematical understanding of how
and what embeddings are learning. Since it is a complicated task, we are
limiting ourselves to the word2vec model, that is paradigmatic of this family
of language models. The sophistication of subsequent approaches (like LSTM or
BERT), is “simply” to add some sequential memory and some “locality” (since
they use an ensemble of embeddings, each of them specializing locally, i.e. in
the neighborhood of a syntactic context). Writing the formulas for such models
is probably possible, but would require a lot of sophisticated notations. We
clearly restrict our analysis to global embeddings of the word2vec family, but
cover both Skip-gram and CBOW modeling approaches.

During this theoretical analysis, we define what we call a _Reference Model_.
Qualitatively speaking, it is a very large stochastic matrix collecting for
each word of the vocabulary the probability of occurrence of all words in “its
neighborhood”. Given this Reference Model, no embedding is needed (or one can
consider that the associated embedding is the identity matrix). It allows us
to define a word2vec representation of a corpus as a compression of its
Reference Model in two components: one the one hand an embedding,
qualitatively mapping “similar words” (in the sense of the Reference Model)
together, and hence defining “frequentist synonyms”; and on the other hand a
context matrix, allowing to recover the Reference Model once it is multiplied
by the embedding (and after a slightly non linear operation, i.e. a softmax
layer).

We are here in the spirit of other papers presenting embeddings as a
compression (see for example [9, 10, 11]), along with the initial papers on
SVD, except that we can prove, under some assumptions, that the minimized
criterion is a cross-entropy between the Reference Model and the compressed
representation made of the embedding and the context matrices.

In a second stage, we explore further the concept of frequentist synonyms
(that “should be put in the same neighborhood by the compression”) in
association with the identifiability of word2vec embeddings. That for we
generate synthetic corpora with know properties thanks to the control of their
Reference Model. It allows us to test empirically that if the word2vec model
can recover part of the structure we inject in the synthetic texts. The
results are bellow standard expectations: the embeddings of synonyms do not
exhibit strong cosine similarities, despite the fact that we use generative
models satisfying strong mathematical assumptions (that is not the case of
natural language).

Last but not least, we use this understanding to train embeddings on a corpus
of 42 millions of financial news from 2008 to 2020. The literature on the use
of NLP on financial data has been pushed forward by the work of Loughran and
McDonald who analysed the polarity of 10-K fillings of listed company [12].
One of the outcome of their work is a dictionary of polarized words in
categories (Positive, Negative, Litigious, etc) that have been later used as a
supervised dataset [13], as a benchmark [14] or as an input [15] to analyze
other corpora (financial news being one of the most used). It is clear that
NLP is now part of the toolbox to predict stock returns [16, 17], but to
authors’ knowledge no systematic study analyzing the capability of language
models to capture financial semantics have been proposed. In this paper we do
not try to predict stock returns, we rather focus on providing insight about
the structure of financial news from the perspective of embeddings, with a
focus on semantic antonyms and non-stationarity.

The paper is structured as follow: Section 2 starts by defining properly
word2vec model and its Skip-gram version, then explains the relationship
between the internal representation of such an language model and the
underlying “truth” of a Reference Model. Since a Reference Model is target
that embeddings try to learn, it is pivotal in our analysis. We then make
additional Markovian assumptions to ease the process of proving how the
standard Skip-gram loss function asymptotically behaves like a cross-entropy
between the learned model and the Reference Model. This allow us to comment
the influence of synonyms on the structure of embeddings. Section 3 exhibits
experiments on synthetic corpus generated under restrictive assumptions,
allowing us to control what the embeddings should learn. It shows the
importance of the structure of the underlying model, and then it shows that
the similarity between the representation of synonyms is not as large as
expected, but significant when compared to other groups of words. Then Section
4 exploits a corpus made of 42 millions of financial News in conjunction with
the Loughran-McDonald Lexicon. The latter is providing the structure of
polarized words (mainly: Positive, Negative and Litigious) needed for our
analysis. It allows us to explore the way embeddings can mix synonyms and
antonyms, and we find and explain different results obtained on headlines only
versus on the full-text of financial News. Moreover, we study the non-
stationarity of this structure, in particular around one particular example of
a company name that is listen under Wikipedia’s _List of corporate collapses
and scandals_.

##  2 Theoretical analysis: learning embeddings and its relation with
generative models

An embedding is an vectorial representation of terms of a document, that is
learned jointly with other NLP tasks that is learned to model a language or
more recently jointly with different downstream task (like part of speech
tagging or entity recognition). The underlying idea is that the learning phase
positions the terms in the space of embeddings such a way that the downstream
tasks are easier. As a consequence, it is expected that the embeddings will
position in the same neighborhood terms having similar roles in sentences.
Academic papers, working of the topology in the space of embedding, identified
some interesting algebra properties in the space of embeddings, like: “ _King
- man + woman = Queen_ ”. In this paper we focus on the word2vec algorithm
which is proposed in [18] and [3], and especially on its Skip-gram version.

###  2.1 The word2vec model

To be compatible with the standard word2vec Skip-gram notations of [19], let
assume that we face a set of documents, or a very long document that is a
sequence of words belonging to a vocabulary ${\cal
V}:=\\{x_{1},\cdots,x_{V}\\}$. The word2vec proposed in [20] can be written as
a neural network with one hidden layer of size $N$, that is called the
_embedding size_ and that is usually far lower than $V$ (see Figure 1).
Typically English vocabulary needs at least 70,000 words and the embedding
size used by practitioners is between 150 and 400.

To feed to the neural network, the $i$-th word of the vocabulary is encoded
using a one-hot vector $x_{i}$, i.e. $x_{i}\in\mathbbm{R}^{V}$ that is a
vector with zeros everywhere but only one $1$ at its $i$-th coordinate. The
weights between the input layer and the output layer can hence be represented
by a $V\times N$ matrix ${W}$. Each row of ${W}$ is the $N$-dimension vector
representation ${e}_{i}$ of the associated word of the input layer. Formally
row $i$ of ${W}$ can be identified with the representation of $i$-th word and
noted ${x_{i}}^{T}W$, given as the context word one-hot vector ${x}_{i}$. The
activation of the hidden layer that is linear:
${h}_{x_{i}}^{T}={x_{i}}^{T}W={W}_{(i,\cdot)}$, which is essentially copying
the $i$-th row of ${W}$ to ${h}_{i}$. In this paper, we use row vector
notation. We call ${W}$ the _word-embedding matrix_ , and its $i$-th row is
commonly called the _embedding of the $i$-th word of the vocabulary_.

From the hidden layer to the output layer, there is a matrix of weight
commonly noted ${W}^{\prime}$, which is an $N\times V$ matrix, associating an
embedding to a long row of size $V$. $W^{\prime}x_{j}$ the $j$th column of the
_context matrix_ ${W}^{\prime}$. Its output layer as a softmax activation,
mapping a vector $(z_{i})_{i}$ to another vector
$(\exp(z_{i})/\sum_{j}\exp(z_{j}))_{i}$ that is positive and which coordinates
sum to one. The output of such a neural network is thus homogeneous to a
vector of probabilities. As a consequence, the word2vec allows to compute the
expected probability of occurrence of any $j$-th word of the vocabulary
conditioned on the input word $i$ that reads $\text{\sf
softmax}({x_{i}^{T}WW^{\prime}})x_{j}$.

Figure 1: The Skip-gram model.

Moreover, the word2vec Skip-gram model has an hyperparameter $C$ that is the
number of words that is explicitly considered by the model. In this paper, for
the simplicity of notation, the considered words are the $C$ words following
the input word.111It is not need in general; their positions can be chosen by
the user. As a consequence the variable

(1) $y_{c,j}^{i}=\text{\sf
softmax}({x_{i}^{T}WW^{\prime}})x_{j},\quad\quad\forall c\in\\{1,..,C\\}$

computes the estimate of the probability of occurrence of the $j$-th word of
the vocabulary amongst the $C$ words following the $i$-th word of the
vocabulary in the considered corpus of documents. Note that each output
$y_{c,j}^{i}$ is independent and only depend on the input word.

######  Definition 2.1 (Parameters of a word2vec model).

A word2vec model is defined by the triplet of parameters: embeddings, contexts
and width; i.e. $(W,W^{\prime},C)$.

###  2.2 The Skip-gram loss function

##### Assuming the existence of an “underlying” model of the text.

In the following sections, we use $(X_{k})_{k}$ to denote the sequence of
words in a corpus. To be able to define a probabilistic description of the
embeddings, one needs first to make the weak assumption that the sequence of
the $K$ words constituting the corpus are drawn from a probability
distribution ${\cal L}_{\cal T}$ that exists. You can think that this
distribution allows to generate the whole sample at once, with a lot of bi-
directional dependencies, or you can think that $(X_{k})_{k}$ is a stochastic
process (meaning that there is an “information arrow” pointing from the left
to the right in the text). It may be generated by a Markov chain if you
believe in very short term memory in the text, or even be i.i.d. realizations
of a random variable if you think the words are generated according to their
histogram of frequencies in the English language. The existence of this law
will enable us to write probabilities.
Since equality (1) defines the likelihood of the occurrence of word $j$ in the
neighborhood of word $i$, the word2vec model implies a specific structure of
this probability ${\cal L}_{\cal T}$. Or at least it assume that the loss
function it minimizes pushes this likelihood to be compatible with the
underlying distribution. This is typically what we will explore in this paper:
which assumptions on ${\cal L}_{\cal T}$ are naturally compatible with the way
the word2vec model build its likelihoods? That for we need to start with a
good understanding of the Skip-gram loss function.

Following our notations: $X_{k}$ is the $k$-th word in the document while
$x_{i}$ is the $i$-th word in the vocabulary. $x_{i}$ denotes the one hot
encoding of the $i$-th word in the vocabulary (word index). Hence the event
“$X_{k}=x_{j}$” means that the $k$-th word observed in the corpus is the
$j$-th word of the vocabulary.

##### Writing the Skip-gram loss function.

The part of the loss function associated with the $k$th word of the document
corresponds to the estimate by the word2vec model of likelihood to observe the
$C$ following words of the corpus. It reads

$\displaystyle\ell(X_{k},\ldots,X_{k+C})$ $\displaystyle=$
$\displaystyle{-\log\hat{\mathbb{P}}_{W,W^{\prime}}(X_{k+1}=x_{j_{1}},\cdots,X_{k+C}=x_{j_{C}}|X_{k}=x_{i})}$
$\displaystyle=$ $\displaystyle-\log\prod_{c=1}^{C}\text{\sf
softmax}(x_{i}^{T}WW^{\prime})x_{j_{c}}$ $\displaystyle=$
$\displaystyle-\log\prod_{c=1}^{C}\frac{\exp(x_{i}^{T}WW^{\prime}x_{j_{c}})}{\sum_{j^{\prime}=1}^{V}\exp(x_{i}^{T}WW^{\prime}x_{j^{\prime}})}$
$\displaystyle=$
$\displaystyle-\sum_{c=1}^{C}x_{i}^{T}WW^{\prime}x_{j_{c}}+C\cdot\log\sum_{j^{\prime}=1}^{V}\exp(x_{i}^{T}WW^{\prime}x_{j^{\prime}}),$

where $x_{j_{1}},\cdots,x_{j_{C}}$ are the word occurring after the $k$-th one
in the text. During the learning phase, the Skip-gram word2vec hence deals
with sequences of $C+1$ words that can be found in the text.
The full loss function is the average of $\ell(\cdot)$ over all the words of
the text, that can be expressed as
$\frac{1}{K}\sum_{k}\ell(X_{k},\ldots,X_{k+C})$, i.e. the empirical average
over the words of the text.

##### Defining a Reference Model.

First define what we will call a “ _Reference Model_ ”: qualitatively it is a
word2vec model with a trivial embedding, i.e. the size of the embedding is the
size of the vocabulary: $N=V$, and hence the embedding matrix $W$ is the
identity. Since the rows of the context matrix of a Reference Model do not
interact and the softmax layer is not needed; it is enough to arbitrary scale
its rows to sum to one. This non-linearity can thus be removed from the
word2vec of a Reference Model.

######  Definition 2.2 (Reference Model).

A _Reference Model_ RM is a triplet $(\textbf{\sf Id},W^{\prime}_{0},C)$ that
can be identified to the parameters of a word2vec neural network. With $V$ the
size of the vocabulary, $W^{\prime}_{0}$ is a $V\times V$ stochastic matrix
which element $(i,j)$ records the probability to see the $j$-th word of the
vocabulary in the $C$ words following the $i$-th word.

Different kinds of Reference Models can be built:

  * ($i$)

The Reference Model corresponding to a given word2vec model $(W,W^{\prime},C)$
that is $(\textbf{\sf Id},W^{\prime}_{0},C)$ where for any $i$,
$x_{i}^{T}W^{\prime}_{0}:=\text{\sf softmax}(x_{i}^{T}WW^{\prime})$; we use
the notation $\text{\sf RM}(W,W^{\prime})$.

  * ($ii$)

The Reference Model corresponding to a corpus of document that is
$(Id,W^{\prime}_{0},C)$ where $W^{\prime}_{0}$ is this time made of the
averaged empirical probabilities of occurrence of the $C$ words following each
possible word; we use the notation $\text{\sf RM}(X_{1},\ldots,X_{K})$.

  * ($iii$)

The Reference Model corresponding to underlying probability distribution
${\cal L}_{\cal T}$ that generated the corpus; we use the notation $\text{\sf
RM}({\cal L}_{\cal T})$.

Note that for the same corpus:

  * •

Given $(X_{1},\ldots,X_{K})$ has been generated by ${\cal L}_{\cal T}$ (and
provided that $K$ is large enough), we expect to have $\text{\sf
RM}(X_{1},\ldots,X_{K})\simeq\text{\sf RM}({\cal L}_{\cal T})$.

  * •

If a word2vec is a “good model” for this corpus, the corresponding Reference
Model $\text{\sf RM}(W,W^{\prime})$ should be “close” to the third one
$\text{\sf RM}({\cal L}_{\cal T})$.

The question of a _generative model_ can be formulated this way: _what are the
assumptions over ${\cal L}_{\cal T}$ _(i.e. the true underlying distribution)_
, such that the Reference Model of the word2vec learned on a _(very long)_
sample generated by ${\cal L}_{\cal T}$ is the same as the Reference Model of
${\cal L}_{\cal T}$?_

###  2.3 Probabilistic weaknesses of the word2vec embeddings

It is now clear that the information contained in a word2vec model on the
learned language is captured by a factorization over $V\times N+N\times V$
coefficients (i.e. the embeddings $W$ and the context $W^{\prime}$) of the
joined probability of occurrences of $C+1$ words at a given distance that
could be described by $(V\times V)^{C}$ parameters. The key quantity that is
manipulated by the word2vec is an estimate of $\mathbb{P}_{{\cal L}_{\cal
T}}(X_{k+1}=x_{j_{1}},\cdots,X_{k+C}=x_{j_{C}}|X_{k}=x_{i})$ for any word
position $k$ in the observed text. We will use the notation
$\hat{\mathbb{P}}_{W,W^{\prime}}(X_{k+1}=x_{j_{1}},\cdots,X_{k+C}=x_{j_{C}}|X_{k}=x_{i})$
to underline that this probability is estimated using matrices $W$ and
$W^{\prime}$ only.

We will list here the main weaknesses of this model and give pointers and
intuitions to how more recent models answered to these points.

##### Weakness 1: The input is made of only one word.

The first potential issue of the word2vec Skip-gram approach is that there is
only one word as input. It implies that once the model parameters $W$ and
$W^{\prime}$ are chosen, there is no difference between
$\hat{\mathbb{P}}_{W,W^{\prime}}(X_{k+1}=x_{j_{1}},\cdots,X_{k+C}=x_{j_{C}}|X_{k}=x_{i},X_{k-1}=x_{i^{\prime}})$
and
$\hat{\mathbb{P}}_{W,W^{\prime}}(X_{k+1}=x_{j_{1}},\cdots,X_{k+C}=x_{j_{C}}|X_{k}=x_{i},X_{k-1}=x_{i^{\prime\prime}})$.222If
we consider the learning process it is clear that the loss function had to
cope with the following two sequences:
$(X_{k-1}=x_{i^{\prime}},X_{k}=x_{i},X_{k+1}=x_{j_{1}},\cdots,X_{k+C-1}=x_{j_{C-1}})$
and
$(X_{k-1}=x_{i^{\prime\prime}},X_{k}=x_{i},X_{k+1}=x_{j_{1}},\cdots,X_{k+C-1}=x_{j_{C-1}})$.
The information is inside $W$ and $W^{\prime}$, but at this stage it is
difficult to know how, and it is sure that it is averaged using as weights the
relative number of occurrences of these sequences in the text. They are a lot
of different ways to explicitly fix this issue about a potential fragility of
the input:

  * •

the _CBOW_ version of the word2vec algorithm is one of them. The CBOW uses the
same weights $W$ and $W^{\prime}$ but instead of minimizing a loss that is the
likelihood of occurrence of a list of words observed “after” $X_{k}$, it takes
a list of word occurring“before” $X_{k}$ and averages their representations to
target the $k+1$ word of the text:

$\begin{array}[]{l}\hat{\mathbb{P}}^{\text{\sc
cbow}}_{W,W^{\prime}}(X_{k+C}=x_{j_{C}}|X_{k+C-1}=x_{j_{C-1}},\cdots,X_{k+1}=x_{j_{1}},X_{k}=x_{i})\\\\[5.0pt]
\phantom{\hat{\mathbb{P}}^{\text{\sc cbow}}_{W,W^{\prime}}}=\text{\sf
softmax}(\frac{1}{C}(x_{j_{C-1}}+\ldots+x_{k+1}+x_{k})^{T}WW^{\prime})x_{j_{C}}.\end{array}$

It means that the internal representations of the $C-1$ input vectors are
averaged, and that this average is used to predict the likelihood of the next
word.

  * •

Since Skip-gram is trying to estimate the averaged probability of occurrence
of words occurring after the pivotal $k$-th word, and the CBOW is doing the
reverse (estimate the probability of occurrence of the $k+1$-th word given an
average of the word that are before $X_{k}$), one could imagine a mix of these
two approaches: using a weighted average of word “before” $X_{k}$ to predict a
weighted averaged probability of words positioned after $X_{k}$.

It is not very far away of what the self-attention mechanism is doing [5]
except that weights for the average are not chosen a priori, but they are
learned and they are a function of words surrounding each weighted word: it is
the _attention_ associated to the occurrence of the sequence of words
$(X_{k-1}=x_{i^{\prime}},X_{k}=x_{i},X_{k+1}=x_{j_{1}},\cdots,X_{k+C-1}=x_{j_{C-1}})$.

  * •

Another way to have more information about the words before $X_{k}$ would be
to increase the size of the state space and to concatenate $X_{k}$ and
$X_{k-1}$. But it is considered to be far too demanding in number of
parameters. NLP methods prefer to use averages.

The take-away if this weakness is that it prevents a word2vec to capture some
subtleties of the learned corpus. One solution is to average over more words
such a way that their weights in the average (i.e. attention) are conditioned
by the joined distribution of words. It certainly produces _regularization_
and _localization_ (in the sense that the embeddings are weighted differently
according to their position in the very high dimensional space of observed
sequences of words). It means that the conclusions we will obtain on word2vec
will probably be only valid locally for attention mechanisms.

##### Weakness 2: The ordering of words is not taken into account.

From the previous Section, it is clear that for a Skip-gram word2vec (and we
know it is the same for a CBOW one) once the parameters $W$ and $W^{\prime}$
are chosen, there is no difference between the word2vec estimate of
$\hat{\mathbb{P}}_{W,W^{\prime}}(X_{k+1}=x_{j_{1}},\cdots,X_{k+C}=x_{j_{C}}|X_{k}=x_{i})$
and
$\hat{\mathbb{P}}_{W,W^{\prime}}(X_{k+1}=x_{\sigma(j_{1})},\cdots,X_{k+C}=x_{\sigma(j_{C})}|X_{k}=x_{i})$,
for any $C$-permutation $\sigma$.

Nevertheless, if we consider the learning process: only sequences that have
been seen in the text are considered in the minimization of the loss function,
hence $W$ and $W^{\prime}$ incorporate a trace of the sequences. But it would
be interesting to explicitly inject the positions of the words in the
probabilistic model.

This is what BERT is doing, by concatenating the embeddings of the words (i.e.
the $W$ matrix of the word2vec) with a positional embedding [6]. It is clear
that it adds the positional information that is missing in word2vec. As an
illustration, just have a look at the expected effect on the estimate of the
likelihood if this idea of “positional embeddings” is transposed to the
word2vec mechanism: say that we augment the two matrices $W$ and $W^{\prime}$
by a positional one $P$ that is used the standard word2vec way. Now we can
replace (with $C=2$, to keep it simple, and with the notation $p_{u}$ to
encode that a word is in position $u$)

$\hat{\mathbb{P}}_{W,W^{\prime}}(X_{k+1}=x_{j},X_{k+2}=x_{j^{\prime}}|X_{k}=x_{i})=\frac{1}{2}(\text{\sf
softmax}(x_{i}^{T}WW^{\prime})x_{j}+\text{\sf
softmax}(x_{i}^{T}WW^{\prime})x_{j^{\prime}})$

by:

$\begin{array}[]{l}\hat{\mathbb{P}}_{W,W^{\prime},P}(X_{k+1}=x_{j},X_{k+2}=x_{j^{\prime}}|X_{k}=x_{i})\\\\[3.00003pt]
\phantom{\hat{\mathbb{P}}_{W,W^{\prime},P}}=\displaystyle\frac{1}{2}\left(\text{\sf
softmax}(x_{i}^{T}WW^{\prime}+p_{1}^{T}PP^{\prime}p_{2})x_{j}+\text{\sf
softmax}(x_{i}^{T}WW^{\prime}+p_{1}^{T}PP^{\prime}p_{3})x_{j^{\prime}}\right).\end{array}$

This produces different shifts inside the softmax depending on the relative
positions of the words. It implies that using such a mechanism, a word2vec
would consider two words to be interchangeable with respect to a given
$x_{i}$, not only if they have the same $x_{i}^{T}WW^{\prime}$ but also if
they are positioned at the same distance if $i$.

###  2.4 Markovian generative models

In this section, we explore the theoretical properties of the word2vec
embeddings under the assumption that the underlying language text is generated
by a Markov model. We restrict ourselves to the Skip-gram approach: the loss
function that is minimized is derived form $\ell(k)$ defined by equality
(2.2).

######  Definition 2.3 (Markov generative model for texts).

A Markov generative model for a text over a vocabulary ${\cal V}$ of size $V$
is defined by a stochastic transition matrix $K$ of size $V\times V$ such that
$K_{i,j}$ is the probability that the $j$-th word of the vocabulary follows
the $i$-th word of the vocabulary.

Thanks to $K$, and provided that an initial distribution $m_{0}$ over words is
defined, a text of size $T$ is a stochastic process $(X_{k})_{1\leq k\leq T}$
such that $X_{0}\sim m_{0}$ and

(3) $\mathbb{P}(X_{k+1}=x_{j}|X_{k}=x_{i})=K_{i,j}.$

Markov generative models are restricted to irreducible Markov chains.

It is natural restrict Markov generative models to irreducible Markov chain,
since it is realistic to think that a language can link two arbitrary words of
the vocabulary by a finite text.

Texts generated by Markov models are of course poorer that standard English
texts. Nevertheless we could easily imagine “multiple inputs” or “local”
versions of Markov text generation that would correspond the the formerly
identified probabilistic weaknesses of the Skip-gram word2vec. Moreover, one
could think about extensions of the Markovian framework to more subtle ones
like the one chosen for the Latent Dirichlet Allocation [21] without killing
most of the theoretical mechanisms that will be used in this section; but this
is out of the scope of this paper.

Since a Markovian text satisfies
$\mathbb{P}(X_{(k+c)}=x_{j}|X_{k}=x_{i})=x_{i}^{T}K^{c}x_{j}$, it is
straightforward to see that the main quantity of interest for a Skip-gram
word2vec is linked to the Reference Model $\text{\sf RM}(K)$ corresponding to
the Markov kernel $K$ since

(4) $W^{\prime}_{0}(\text{\sf RM}(K))=\frac{1}{C}\sum_{i=1}^{C}K^{i}.$

This comes from the fact that the language model is trained on a sequence of
$C$ words without any consideration of ordering. Hence, given that ${\cal I}$
is a uniform random variable over $\\{1,\ldots,C\\}$, the Skip-gram word2vec
tries to estimate

(5) $\displaystyle\mathbb{P}(X_{k+{\cal I}}|X_{k})$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{C}\mathbb{P}(X_{k+i}|X_{k})\,\mathbb{P}({\cal I}=i)$
$\displaystyle=$
$\displaystyle\frac{1}{C}\sum_{i=1}^{C}\mathbb{P}(X_{k+i}|X_{k}).$

####  2.4.1 Markov chain properties

In this subsection, we formulate well known properties of Markov chains [22]
that we will need thereafter within our models and notations. Let ${\cal
D}(T)=(X_{1},X_{2},\ldots,X_{T})$ be a document made of $T$ words generated
using a Markov chain with a transition matrix $K$ on a vocabulary of $V$
words, and denote by $\mu$ the stationary distribution of the Markov chain.
The state space of our Markov chain is the vocabulary $\mathcal{V}$ which is
finite. Hence our Markov chain is positive recurrent; we will largely rely on
the ergodic theorem.

######  Property 2.4 (Convergence of empirical distribution).

(6)
$\lim_{T\rightarrow\infty}\frac{1}{T}\sum_{k=1}^{T}\mathbbm{1}(X_{k}=x_{1})=\mu(x_{i})$

Note that $\mu$ is a vector of size $V$, and the $i$th coordinate
$\mu_{i}=\mu(x_{i})$ is the probability of appearance of work $i$ in the
stationary distribution.

###### Proof.

Direct consequence of Ergodic theorem. See Theorem 4.16 in [23] or Corollaire
13.6.2 in [24]. ∎

Then we state a property for the convergence of empirical conditional
distribution:

######  Property 2.5 (Convergence of empirical Conditional distribution).

Let $\mathcal{I}$ be an independent uniformly random index from $1$ to $C$
(each index has equal probability $\frac{1}{C}$),

$\lim_{T\rightarrow\infty}\frac{\sum_{k=1}^{T}\mathbbm{1}(X_{k}=x_{i},X_{k+\mathcal{I}}=x_{j})}{\sum_{k=1}^{T}\mathbbm{1}(X_{k}=x_{i})}=\frac{1}{C}\sum_{l=1}^{C}K^{l}(x_{i},x_{j})$

###### Proof.

We only stage the proof in the simplest $C=1$ case to keep it short; when
$C>1$, the proof is similar. Just observe that
$(X_{k},X_{k+1})_{k\in\mathbb{N}}$ is an irreducible Markov chain too over the
set of states $\\{(x,y)|K(x,y)>0\\}$ with kernel
$Q((a,b),(c,d))=K(c,d)\mathbbm{1}(b=c)$. Its unique invariant probability
measure is $\pi(x,y)=\mu(x)K(x,y)$. Applying the ergodic theorem to
$\mathbbm{1}(X_{k}=x_{i},X_{k+1}=x_{j})$ reads

$\frac{\frac{1}{T}\sum_{k=1}^{T}\mathbbm{1}(X_{k}=x_{i},X_{k+1}=x_{j})}{\frac{1}{T}\sum_{k=1}^{T}\mathbbm{1}(X_{k}=x_{i})}\underset{T\to\infty}{\longrightarrow}\frac{\mu(x_{i})K(x_{i},x_{j})}{\mu(x_{i})}=K(x_{i},x_{j})$

∎

####  2.4.2 From a Reference Model to a Markov kernel

Given a Reference Model with $C=1$, i.e. $(\textbf{\sf Id},W_{0}^{\prime},1)$,
a Markov chain with $K:=W^{\prime}_{0}$ is the corresponding Markov generative
model: by construction the two reference Models coincide. But when $C>1$,
there does not always exist a $K$ satisfying (4) for a given $W^{\prime}_{0}$.

######  Theorem 2.6 (Representative generative model).

Given a reference model $(\textbf{\sf Id},W_{0}^{\prime},C)$ on a vocabulary
of $V$ words, there exists a Markov chain with a transition matrix $K$
verifying equality (4) if one of these two conditions is verified:

  1. (i)

$C=1$;

  2. (ii)

$W^{\prime}_{0}$ is symmetric (or diagonalizable) and all its eigenvalues are
in $[0,1]$.

If our kernel $K$ is diagonalizable (for example, if it is reversible), then
$K=P\Delta P^{-1}$ (and if it reversible, $P$ is a orthonormal basis, i.e.,
$P^{-1}=P^{T}$) where $\Delta$ is a diagonal matrix. Following the spectral
properties of transition kernel (Lemme 9 in [25]): if $K$ is ergodic then the
maximal value in the diagonal $\Delta$ is $1$ and it is unique. The other
diagonal elements have absolute value less than $1$. We can deduce the
relation linking $W^{\prime}_{0}$ and $C$ with the diagonal decomposition of
$K$:

(7) $P^{-1}W^{\prime}_{0}P=\frac{1}{C}\sum_{i=1}^{C}\Delta^{i}.$

The right side of equation can be calculated exactly by the formula of
geometric sum: it is a diagonal matrix which terms are
$\frac{1}{C}\sum_{i=1}^{C}\lambda^{i}$ for all $\lambda\neq 1$.
Now we can show the equivalence:

  * •

Given a stochastic matrix $K$, we can always find $W^{\prime}_{0}$ such that
$W^{\prime}_{0}$ is also a stochastic matrix and satisfying (4).

The matrix constructed in this way $K:=P\Delta P^{-1}$ is stochastic.
$v=(1,1,...,1)$ is the eigenvector of $W^{\prime}_{0}$ of eigenvalue $1$ such
that $W^{\prime}_{0}v=v$ and the first column of $P$ is $v$. So
$P^{-1}v=(1,0,...,0)$ hence $P\Delta P^{-1}v=P(1,0,...,0)=v$. $Kv=v$ means
that each row of $K$ sums up to $1$ which prove that $K$ is stochastic.

  * •

Inversely, given a _diagonalizable_ stochastic matrix $W^{\prime}_{0}$ with
eigenvalues in $[0,1]$, we can also find such a matrix $K$. In fact, if
$W^{\prime}_{0}=PDP^{-1}$ where $D$ is a diagonal matrix with value between
$0$ and $1$, we can find a diagonal matrix $\Delta$ such that
$\frac{1}{C}\sum_{i=1}^{C}\Delta^{i}=D$ because the function

$x\mapsto\frac{1}{C}\sum_{i=1}^{C}x^{i}=\frac{1}{C}\left(\frac{1-x^{C+1}}{1-x}-1\right)$

is bijective from $[0,1]$ to $[0,1]$.

$\square$

###  2.5 Understanding Word2Vec as a Compression of a Reference Model

####  2.5.1 How to compress a Reference Model

Assume that we start with the Reference Model $(\textbf{\sf
Id},W^{\prime}_{0},C)$ of a corpus and try to qualitatively understand what
one can expect from a word2vec model of the same corpus when $N$, the
dimension of the word embeddings, is $V-1$:

  * •

On the one hand the likelihood associated by the word2vec model to the pair of
words $(x_{i},x_{j})$ reads

$\text{\sf softmax}(x_{i}^{T}WW^{\prime})x_{j},$

  * •

on the other hand the empirical occurrences in the corpus says that this
likelihood should be

$x_{i}^{T}W_{0}^{\prime}x_{j}.$

If a pair of words $(x_{i},x_{i^{\prime}})$ is such that
$x_{i}^{T}W_{0}^{\prime}x_{j}=x_{i^{\prime}}^{T}W_{0}^{\prime}x_{j}$, then the
rows $i$ and $i^{\prime}$ of $W_{0}^{\prime}$ are identical. The best way to
compress $W^{\prime}_{0}$ is hence

  1. 1.

to use the embedding matrix to map rows $i$ and $i^{\prime}$ on the same
embedding. We will use the notation $E[i^{\prime}\mapsto i]$ for this
embedding matrix that is the $V\times V$ identity matrix with column
$i^{\prime}$ that is removed, and with $1$ in place of $0$ at its element
$(i^{\prime},i)$

  2. 2.

to remove the row $i^{\prime}$ of $W^{\prime}_{0}$. We use the notation
$W_{0}^{\prime}[i^{\prime}]^{\ominus}$ for a $(V-1)\times V$ matrix
corresponding to $W_{0}^{\prime}$ once its $i^{\prime}$-th row is deleted.

Then $E[i^{\prime}\mapsto i]\cdot
W^{\prime}_{0}[i^{\prime}]^{\ominus}=W^{\prime}_{0}$. As a consequence:

######  Property 2.7 (Trivial compression of a Reference Model by a Word2vec).

With the upper notations: the word2vec model $(E[i^{\prime}\mapsto
i],W^{\prime}_{0}[i^{\prime}]^{\ominus},C)$ is an _exact compression_ of the
Reference Model $(\textbf{\sf Id},W_{0}^{\prime},C)$, such that the $i$-th and
$i^{\prime}$-th words of $W_{0}^{\prime}$ are identical, i.e.:

(8) $\forall(k,k^{\prime}):\>\text{\sf softmax}(x_{k}^{T}\,E[i^{\prime}\mapsto
i]\,W^{\prime}_{0}[i^{\prime}]^{\ominus})x_{k^{\prime}}=x_{k}^{T}W_{0}^{\prime}x_{k^{\prime}}.$

We can have a qualitative look at $E[i^{\prime}\mapsto i]$, the matrix of
embeddings of this exact compression, and notice that it mapped $i$ and
$i^{\prime}$ on the same embedding vector. And, qualitatively once more, the
nature of words $xi$ and $x_{i^{\prime}}$ is that they have the same
“probability vector”, i.e. the same probability of occurrence of surrounding
words. See [7] and references therein to different perspectives on
compression.

##### A frequentist (and confusing) viewpoint on synonyms.

From a frequentist viewpoint, $x_{i}$ and $x_{i^{\prime}}$ are
interchangeable, this is the reason why the training algorithm will map them
on the same embeddings. One could call them “synonyms from a word2vec
viewpoint” or _frequentist synonyms_. For two words being frequentist synonyms
it is enough that in the corpus all sentences containing $x_{i}$ have a
similar sentence where it is replaced by $x_{i^{\prime}}$. This is clearly not
the definition of semantic synonyms, for instance if all sentences containing
the word “ _bad_ ” have an copy with the word “ _good_ ” in place, like:
_look at this bad guy_ $\leftrightarrow$ _look at this good guy_. _this is bad
English_ $\leftrightarrow$ _this is good English_. etc.
and these semantic antonyms will become synonyms in embeddings learned on this
corpus. Elaborating a little more on application, one can bet that on a corpus
of cars, colors will probably be frequentist synonyms, since most cars can be
seen in any color, hence such a corpus may have as many sentences like: “a
_gray_ car had an accident” than “a _blue_ car had an accident”. But on a
corpus made of cooking recipes, colors will not be frequentist synonyms
because there is no “ _red_ bananas” but “ _red_ apples” and “ _red_ peppers”.
We will empirically see in Section 4 that if frequentist synonyms provide
structure to a word2vec mode, they can confuse a task related to exploit
polarity of sentiments of financial texts.

####  2.5.2 A formal definition for compression of Markov chains

The notion of Reference Model enables to define clearly what kind of
compression can be expected from a text embedding, in particular for the
word2vec class of models. The vocabulary size being $V$, and the embedding
dimension being $N$, compressing _linearly_ a Reference Model $(\textbf{\sf
Id},W^{\prime}_{0},C)$ using a word2vec will end up with a ${V\times N}$
matrix $R$ and a ${N\times V}$ matrix $R^{\prime}W_{0}^{\prime}$, such that
the probabilities defined by $(R,R^{\prime}W_{0},C)$ are as close as possible
from the original ones. Qualitatively:

$\forall x_{i},x_{j}:\>\text{\sf
softmax}(x_{i}^{T}RR^{\prime}W_{0}^{\prime})x_{j}\mbox{ ``close to''
}x^{T}_{i}W_{0}^{\prime}x_{j}.$

The main point is to choose a quantitative criterion that has a sense to
define this desired proximity. Qualitatively, it is clear that:

  * •

the rank of $RR^{\prime}$ needs to be maximal (taking into account that the
more frequentist synonyms in the language the lower the rank of
$W^{\prime}_{0}$) to recover the space spanned by $W_{0}^{\prime}$,

  * •

without the softmax non linearity and if the SVD (Singular Value
Decomposition) of $W^{\prime}_{0}$ is $U\Sigma V^{T}$, one could expect that
$R\simeq U\Sigma$ and $R^{\prime}\simeq V^{T}$ (see [7, Section 4.2] for more
details). Nevertheless the softmax changes the setting. Moreover, the SVD
compression criterion is the minimization of the unexplained variance whereas
in word embedding, the proximity of conditional probabilities seems to be a
better criterion.

######  Definition 2.8 (Compression of a reference model $({\rm
Id},W_{0}^{\prime},C)$).

We define the compression to dimension $N$ of a reference model of size $V$
using two mappings $\phi(W^{\prime}_{0})$ and $\phi^{\prime}(W^{\prime}_{0})$.
The compressed model is the
$(\phi(W^{\prime}_{0}),\phi^{\prime}(W^{\prime}_{0}),C)$ word2vec model. For
simplicity of the notation we set

$\Phi(W^{\prime}_{0}):=\phi(W^{\prime}_{0})\phi^{\prime}(W^{\prime}_{0})$

and we define ${\mathfrak{C}}$ the class of function $\Phi$ that can be
written this way.

If we consider the sequence of words $X_{k}$ of a text of length $T$ as a
stochastic process, then we may take any distance $d(\cdot,\cdot)$ between
probability vectors (i.e. $x_{k}^{t}W^{\prime}_{0}$ or $\text{\sf
softmax}(x_{k}^{T}WW^{\prime}$) and write our optimization problem as

(9) $\min_{R,R^{\prime}}\;\frac{1}{T}\sum_{k=1}^{T}d(\text{\sf
softmax}(X_{k}^{T}\phi(W^{\prime}_{0})\phi^{\prime}(W^{\prime}_{0})),\
X_{k}^{T}W^{\prime}_{0})$

For linear compression, just note that
$RR^{\prime}W_{0}^{\prime}:=\Phi(W^{\prime}_{0})$. A last qualitatively
remark: $\text{\sf softmax}(X_{k}^{T}\Phi(W^{\prime}_{0}))$ and
$X_{k}^{T}W^{\prime}_{0}$ are both row vectors which coordinate $j^{\prime}$
represents the probability of occurrence of the $j^{\prime}$-th word of the
vocabulary (in the neighborhood of $X_{k}=x_{i}$, i.e. if the word
$x_{j^{\prime}}$ occurs at least once in the next $C$ words after
$X_{k}=x_{i}$). Hence the former will be a “good compression” of the later if
the these two probability distributions are “close”.
A natural choice is to consider the _cross entropy_ (that is a shifted version
of Kullback-Leiber divergence) between these two distributions:

$\displaystyle\mathbf{H}_{CE}(\mathbb{P}_{W^{\prime}_{0}|X_{k}},\hat{\mathbb{P}}_{\Phi(W^{\prime}_{0})|X_{k}})$
$\displaystyle:=$
$\displaystyle-\mathbb{E}_{X_{k}^{T}W^{\prime}_{0}}\log\text{\sf
softmax}(X_{k}^{T}\Phi(W^{\prime}_{0}))$ $\displaystyle=$
$\displaystyle-\sum_{x_{j^{\prime}}\in{\cal V}}\log\text{\sf
softmax}(X_{k}^{T}\Phi(W^{\prime}_{0}))x_{j^{\prime}}\cdot
X_{k}^{T}W^{\prime}_{0}x_{j^{\prime}}$

Thanks to the ergodic theorem333i.e.
$\lim_{T\rightarrow\infty}\frac{1}{T}\sum_{t}f(X_{k})=\mathbb{E}_{\mu}f(X_{i})$.
that can be used on the stochastic process made of the sequence of words when
the length of the corpus goes to infinity, it is now possible to state this
definition of a compression:

######  Definition 2.9 (Compression criterion on a Markov chain generated
text).

If a text $\mathfrak{X}=X_{1},\ldots,X_{k},\ldots$ stems from a Markov
generative model which Reference Model is $(\textbf{\sf
Id},W_{0}^{\prime},C)$, we define a compression $\Phi$, taken in the class of
functions ${\mathfrak{C}}$, of its Markov kernel $W_{0}^{\prime}$ thanks to
the expectation (according to the invariant distribution $\mu$ of
$W_{0}^{\prime}$) of the Cross Entropy between the output vectors of a
word2vec. It reads:

(11)
$\min_{\phi\in\Phi}\;\mathbb{E}_{X_{k}\sim\mu}\,\mathbf{H}_{CE}\left(\mathbb{P}_{W^{\prime}_{0}|X_{k}},\text{\sf
softmax}(X_{k}^{T}\Phi(W^{\prime}_{0}))\right).$

Note that with the already defined notation
$\hat{\mathbb{P}}_{\Phi(W^{\prime}_{0})|X_{k}}$ for $\text{\sf
softmax}(X_{k}^{T}\Phi(W^{\prime}_{0}))$, we recover a minimization that is
compatible with (2.5.2).

Moreover this definition goes beyond the Skip-gram word2vec embeddings, since
the compression $\Phi$ is quite generic at this stage. It is possible to
extend it definition using to the natural filtration $\mathfrak{X}_{k}$
associated to $\mathfrak{X}$ and replacing $\text{\sf
softmax}(X_{k}^{T}\phi(W^{\prime}_{0}))$ by $\text{\sf
softmax}(X_{k}^{T}\phi(W^{\prime}_{0},\mathfrak{X}_{k}))$ in formula (11). It
would need to change the writing of the expectation but it allows the
compression to use all the words up to the $k$-th word of the text, and hence
to embed a memory or local metrics, addressing for instance part of the
Weakness 1 exposed in Section 2.3.

####  2.5.3 Convergence of the Skip-gram word2vec loss function to a cross-
entropy

Show that the mean of Skip-gram loss function converges to the criterion when
the size of the corpus size goes to infinity, i.e. replacing it by something
link $\mathbb{E}_{t}\ell(X_{t})$ that would be close to (11).

######  Theorem 2.10 (Correspondence between Skip-gram loss function and
compression criterion).

Assume the words $X_{1},\ldots,X_{T}$ of a document of length $T$ are
generated thanks to an ergodic stochastic process according to a Reference
Model $(\textbf{\sf Id},W^{\prime}_{0},1)$ and an initial probability
distribution $m_{0}$ over the vocabulary. Then the loss function of a Skip-
gram word2vec model $(W,W^{\prime},1)$ over this corpus converges towards the
expectation of cross-entropy between $\mathbb{P}_{W^{\prime}_{0}|X_{k}}$ and
$\hat{\mathbb{P}}_{W,W^{\prime}|X_{k}}$

(12) $\lim_{T\rightarrow\infty}\frac{1}{T}\sum_{k=1}^{T}-\log\text{\sf
softmax}(X_{k}^{T}WW^{\prime})X_{k+1}=\mathbb{E}_{X_{k}\sim\mu}\mathbf{H}_{CE}(\mathbb{P}_{W^{\prime}_{0}|X_{k}},\hat{\mathbb{P}}_{W,W^{\prime}|X_{k}}).$

###### Proof.

For the sake of notations, we will restrict the theorem and its proof to
$C=1$. Note $\mu$ as the ergodic measure of the stochastic process
$(X_{k})_{k}$, i.e.
$\mu(x_{i})=\lim_{T\to\infty}\frac{}{T}\sum_{k=1}^{T}\mathbbm{1}(X_{k}=x_{i})$.
Then we can write the loss function

$\displaystyle-\frac{1}{T}$ $\displaystyle\sum_{k=1}^{T}\log\text{\sf
softmax}(X_{k}^{T}WW^{\prime})X_{k+1}$ $\displaystyle=$
$\displaystyle-\sum_{i=1}^{V}\frac{\sum_{k=1}^{T}\mathbbm{1}(X_{k}=x_{i})}{T}\sum_{j=1}^{V}\frac{\sum_{k=1}^{T}\mathbbm{1}(X_{k}=x_{i},X_{k+1}=x_{j})}{\sum_{k=1}^{T}\mathbbm{1}(X_{k}=x_{i})}\log\text{\sf
softmax}(x_{i}^{T}WW^{\prime})x_{j}$

With the notations $\mathbb{P}_{W_{0}^{\prime}|x_{i}}$ for the probability
distribution of words given the occurrence of $x_{i}$ (i.e.
$x_{i}^{T}W^{\prime}_{0}$) and $\hat{\mathbb{P}}_{WW^{\prime}|x_{i}}$ for the
probability of the same events modelled by the Skip-gram word2vec (i.e.
$\text{\sf softmax}(x_{i}^{T}WW^{\prime})$). Then limit when the number of
words goes to infinity of the Skip-gram loss reads

$\displaystyle-\lim_{T\to\infty}$
$\displaystyle\frac{1}{T}\sum_{k=1}^{T}\log\text{\sf
softmax}(X_{k}^{T}WW^{\prime})X_{k+1}$ $\displaystyle=$
$\displaystyle-\sum_{i=1}^{V}\mu(x_{i})\sum_{j=1}^{V}\frac{\mu(x_{i})x_{i}^{T}W^{\prime}_{0}x_{j}}{\mu(x_{i})}\log\text{\sf
softmax}(x_{i}^{T}WW^{\prime})x_{j}$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{V}\mu(x_{i})\;\mathbf{H}_{CE}(\mathbb{P}_{W_{0}^{\prime}|x_{i}},\text{\sf
softmax}(x_{i}^{T}WW^{\prime}))$ $\displaystyle=$
$\displaystyle\mathbb{E}_{X_{k}\sim\mu}\mathbf{H}_{CE}(\mathbb{P}_{W_{0}^{\prime}|X_{k}},\hat{\mathbb{P}}_{WW^{\prime}|X_{k}}).$

∎

When $C>1$ the deduction is similar.

##  3 Synthetic experiments: empirical study of the role of structures

In this section, we will leverage on different elements of our theoretical
analysis to perform numerical explorations around the identifiability of word
embedding models. We mainly leverage on these two elements

  * •

under restrictive assumption, we can use a Markov chain with a kernel $K$ to
build a Reference Model $(\textbf{\sf Id},W_{0}^{\prime},C)$ thanks to
equation (4);

  * •

the existence of _frequentist synonyms_ influences the capability to compress
efficiently a Reference Model.

We will hence generate different Reference Models having more or less
structure (here structure means having blocks of frequentist synonyms or not),
on vocabularies of different sizes, and observe how easy or difficult it is to
recover them from a generated corpus. If the compression performed by a
word2vec is not a principal component analysis (since it is more “low rank”
than “low variance” driven, because of the nonlinearity introduced by the
softmax and because the minimized criterion in a cross entropy with an unknown
Reference Model via a sample of text), it can nevertheless be expected to find
commonalities with the pitfalls identified long ago by Random Matrix Theory
(see [26] for an overview) that are playing an important role in the
theoretical understanding of the limits of deep learning [27].

##### Experimental conditions.

We generate different Reference Models on vocabularies of size $V$, that will
be “compressed” via embeddings of dimension $N$ (for illustration purposes we
even consider some configurations for which $N>V$). We use a Markov chain to
build the reference model so that it is “fully random” (uniformly generated),
or has a structure. This structure is made of “blocks” which rows are
identical to each other (that is the definition of frequentist synonyms).

We train a Skip-gram word2vec during a large number of epochs (enough to
stabilize the learning) via a standard SGD (Stochastic Gradient Descent,
generally Adam with a fixed rate of $10^{-4}$) implemented in pyTorch and
running on Google Colab or AWS using cuda acceleration.

##### Types of structure.

We experiment different level of structure:

  * •

_No structure_ when $K$ is a dense matrix generated randomly and uniformly in
the space of stochastic matrices each of its rows is sampled by the uniform
distribution in the space of simplex using Dirichlet distribution with
$\alpha=1$.

  * •

_Structure_ when $K$ is made of blocks of duplicated rows, when the number of
block varies, the overall size of the structure is always kept constant. For
instance: when $V=1000$ we can take either $160$ blocks of $5$ rows or $40$
block of $20$ rows, because $150\times 5=40\times 20$.

The expected role of blocks is the following: the intrinsic dimension of a
Reference Model made of $B$ blocks of size $S$ plus “noise” on $V-B\times S$
components is between $B+1$ and $B+(V-B\times S)=V-B(S-1)$. it is expected to
be closer to $B+1$ in a “low overfitting” configurations and closer to
$V-B(S-1)$ when the model considers that it is equivalent to learn one row or
$S$ similar rows. Our theoretical analysis suggests that the Skip-gram
word2vec should be more in the first configuration that in the second, since
attributing one vector of embedding to a block is far more rewarding (in terms
of the loss function) than attributing it to one isolated row of
$W_{0}^{\prime}$.

##### Criteria to monitor.

We focus on two criteria:

  * •

The loss function during the learning, to analyse the performance of the
compression. It is interesting to note that the loss function (2.2) being
compatible with the cross entropy (11), and since we are generating synthetic
dataset, in some case one can really expect a full success of the
“compression” that is in these cases a simple identification of the generating
model.

  * •

The distance between the compressed vectors and the original ones. Once again
it is a way to quantify the success of the identification of the properties of
the generative model.

That for, we focus on the (mean) cosine similarity between two words $x_{i}$
and $x_{i^{\prime}}$ defined as the scalar product between the two probability
vectors:

$\mathbf{C}_{S}^{P}(x_{i},x_{i^{\prime}}):=\sum_{j=1}^{V}\text{\sf
softmax}(x_{i}^{T}WW^{\prime})x_{j}\cdot
x_{i^{\prime}}^{T}W_{0}^{\prime}x_{j}.$

To formulate a criterion that is minimal when two groups $G_{1}$ and $G_{2}$
of words are close, we will use

(13) $d_{C}(G_{1},G_{2}):=1-\frac{1}{\\#G_{1}\cdot\\#G_{2}}\sum_{x_{i}\in
G_{1}}\sum_{x_{i^{\prime}}\in G_{2}}\mathbf{C}_{S}^{P}(x_{i},x_{i^{\prime}}).$

This metric will allow us to understand if words belonging to the same group
of frequentist synonyms are closer between themselves rather than to other
words.

###  3.1 First experimentation: low vocabulary size

Figure 2 shows that the Skip-gram word2vect has difficulties to find the
generative $W^{\prime}_{0}$ when the dimension of the word2vec is larger than
the original dimension. This exhibits a clear identifiability issue. Of course
in general the size of the embeddings is lower than the vocabulary size, so
for word2vec this configuration should never occur in practice. Nevertheless,
not that the relative size of the embeddings vs. the one of the vocabulary is
not low for embedding-driven language models like BERT.

Figure 2: The role of structure: cosine distance between $W_{0}^{\prime}$ and
$WW^{\prime}$ during the learning ($x$-axis), for a vocabulary size $V=50$ and
for 8 blocks of 5 (left) and no block (right).

When the dimension of the embeddings is low, Figure 3 suggests that the
structure can be captured by the compression. Typically one could expect that
an embedding of dimension $10$ has more chances to capture a Reference Model
made of 8 blocks, than one made of 32 blocks, and it is verified in our
experiments.

Figure 3: The role of structure: cosine distance between $W_{0}^{\prime}$ and
$WW^{\prime}$ during the learning ($x$-axis), for a vocabulary size $V=200$
and for 8 blocks of 20 (top left), 16 blocks of 10 (top right), 32 blocks of 5
(bottom left) and no block (bottom right).

In any case, Figure 4 underlines the fact that higher dimension of embeddings,
even if it is lower than the vocabulary size, is better in presence of
structure. For instance when the Reference Model is made of 160 blocks of 5
rows (i.e. 800 rows over 1000 of the Reference Model are exhibiting
structure): embedding size of $200$ and $500$ perform similarly (with more
noise for $N=500$) whereas $n=800$ performs very poorly. This underlines the
fact that the structure has to be taken into account to choose the targeted
compression dimension; in a highly structure language (with a lot of
frequentists synonyms), a low dimension can be a proper choice. We can expect
that very repetitive sentences (from a semantic perspective), like the
headlines of financial News, should exhibit identifiability issues compared to
the body of the same news, that are made of more semantically diverse
sentences.

Figure 4: The role of structure: cosine distance between $W_{0}^{\prime}$ and
$WW^{\prime}$ during the learning ($x$-axis), for a vocabulary size $V=1000$
and for 10 blocks of 80 (top left), 40 blocks of 20 (top right), 160 blocks of
5 (bottom left) and no block (bottom right).

###  3.2 Beyond the loss function: assessing the quality of the captured
structure

_8 block of size 5 for a vocabulary size of 50._

_40 blocks of size 20 for a vocabulary size of 1000._

_160 blocks of size 5 for a vocabulary size of 1000._

Figure 5: The recovery of structure: inter-block and intra-block cosine
similarities of embeddings.

Figure 5 exhibits how the compressed model $WW^{\prime}$ recovers the blocks
of the Reference Model $W^{\prime}_{0}$. The standard deviations inside a
block or between words of another block are similar, and as expected the
cosine similarity between words belonging to the same block is higher than the
cosine between words of two different blocks. Nevertheless if this cosine is
large in low dimension (0.8 in the best case of $V=50$) it is far lower in
high dimension. It is 0.02 when $V=1000$ for $N\leq 500$, but it is halves
when $N=800$, showing that overfitting is really present in such a case.

The qualitative conclusions to this empirical study for synthetic data are

  * •

the dimension of the embedding has to cope with the structure of the Reference
Model, with a highly structured Reference Model, a too large embedding
dimension is detrimental to the performances;

  * •

the skip-gram word2vec faces identifiability issues in general, and we suspect
it is not specific to this model.

  * •

Surprisingly, even if the performances of the model is not good, it succeeds
in putting words closer to frequentist synonyms (i.e. within the same block)
than to other words.

Our experiments on the dimensionality of word embeddings seems to be
compatible with [28] that is in favor of a dimension close to $N=300$.

##  4 Experiments on a corpus of financial news: What can be learned?

This section is an empirical study learning embeddings on a financial corpus.
Our goal is to explore how particular type of structure, that is usually named
_sentiment polarity_ in NLP, is preserved or not by embeddings.

The usage of NLP in finance often targets to make the difference between “good
news” and “bad news” on listed company, as a pre-requisit to build investment
strategies (see [29] and [16] for details). The “Loughran-McDonlad Lexicon”
tuned by human experts on 10-K regulatory fillings by US corporates (see [12]
for details), is commonly used to quantify the polarity of text. Since it
lists positive, negative, litigious words, we will use these lists of words as
blocks of synonyms or antonyms. They study play a similar role as the “blocks”
of the generative models studied in the previous section.

###  4.1 Sentiments as a source of structure: Does Financial News understand
better Loughran-McDonald’s polarity than Wikipedia?

| News full text | News headlines
---|---|---
| avg. LM | avg. size | nbe | avg. LM | avg. size | nbe
2008 | 0.26 | 156.18 | 415,524 | 0.24 | 10.14 | 406,315
2009 | 0.25 | 146.84 | 2,780,525 | 0.23 | 10.20 | 2,497,544
2010 | 0.22 | 173.07 | 2,911,113 | 0.21 | 10.17 | 2,651,780
2011 | 0.20 | 354.10 | 4,038,762 | 0.19 | 9.65 | 1,937,799
2012 | 0.19 | 387.70 | 4,650,571 | 0.23 | 10.23 | 132,126
2013 | 0.19 | 370.67 | 4,797,842 | 0.21 | 9.96 | 187,842
2014 | 0.19 | 360.51 | 4,785,053 | 0.18 | 9.90 | 195,455
2015 | 0.20 | 374.79 | 4,763,002 | 0.16 | 9.12 | 1,303,531
2016 | 0.24 | 242.99 | 3,716,652 | 0.18 | 9.54 | 2,763,163
2017 | 0.22 | 133.33 | 3,401,207 | 0.15 | 9.89 | 2,191,320
2018 | 0.21 | 129.60 | 3,357,355 | 0.14 | 9.75 | 1,980,199
2019 | 0.17 | 160.64 | 1,634,809 | 0.15 | 10.00 | 1,545,239
2020 | 0.17 | 175.42 | 1,388,059 | 0.15 | 9.97 | 1,449,560
Table 1: Descriptive statistics of the processed News full text and Headlines
(_avg. LM_ is the average number of words from the Longhran-McDonald lexicon
in a document of the corpus , _avg. size_ is the average number of words, and
_nbe_ is the number of news).

We use a corpus of financial News provided by a large provider of professional
financial News covering years from 2008 to 2020. We will do experiments on the
headlines only or on the whole body of the News (i.e. the headline followed by
the text). We used different ad hoc filters to prevent the repetition of News
(sometimes a News is repeated, in such a case we only keep its first
appearance, and some times a header does not have enough words, in such a case
we remove it from the headline corpus). Table 1 provides descriptive
statistics on these corpora; it is restricted to the News that will be
processed by our word2vec: the headlines contain on average 10 words where the
body of the news contains on average around 250 words. Our filtering reduced
the number of headlines years 2008, 2012, 2013 and 2014. Because of this we
will restrict some of our analysis to other years only.

As a reference, we use embedding pre-trained in Wikipedia (see Appendix A for
details).

The content of the Loughran-McDonald Sentiment Word Lists (2018) is described
in Table 2. This lexicon contains more negative words than positive words;
that is the two categories we will mainly focus on. Not all these words appear
in the considered corpora: for instance embeddings trained on Wikipedia
contains only 349 of the 353 positive words where the headlines of our
financial News from 2008 to 2012 contains 343 of them.

Here are examples of the first words of this lexicon:

  * •

Negative: _abandon, abandoned, abandoning, abandonment, abandonments,
abandons, abdicated, abdicates,_ etc.

  * •

Positive; _able, abundance, abundant, acclaimed, accomplish, accomplished,
accomplishes, accomplishing, accomplishment,_ etc.

| LM | in wiki- | in Fin. News | in Fin. News
---|---|---|---|---
| (ref) | pedia | headlines | Full text
Negative words | 2,354 | 2,097 | 2,190 | 2,335
Positive words | 353 | 349 | 343 | 354
Uncertainty words | 396 | 270 | 273 | 294
Litigious words | 903 | 590 | 637 | 820
StrongModal words | 19 | 18 | 19 | 19
WeakModal words | 27 | 26 | 26 | 27
Constraining words | 183 | 172 | 178 | 181
Table 2: Number of word in each category present in each corpus.

Figure 6 exhibits the cosine similarity between different sections of
polarized words (as it is a scalar product between two vectors of norm 1, its
maximum value is 1). The exact computation process is the following: (1) we
trained Skip-gram word2vec embedding on each year, (2) for each year we
compute the cosine similarity between any two words of the considered lists
(for instance between any two positive words, or between any positive word and
any negative word), (3) we average over all the obtained cosine similarities
(we plot the obatined time series in Figure 7), (4) we average the obtained
numbers excluding years 2008, 2012, 2013 and 2014 (Figure 7 shows that average
cosine similarity is too different over these years for embeddings learned on
headlines, probably because of the low number of headlines available).
If the learned embeddings would have been compatible with the polarities
expressed by the lexicon, we should observe

  * •

The highest average similarity for groups _Positive-Positive_ and _Negative-
Negative_ (blue and orange bars),

  * •

The lowest average similarity for the _Positive-Negative_ group (violet),

  * •

Medium similarities for groups _Positive-Random_ and _Negative-Random_ (red
and green).

Figure 6: Average cosine similarity between embeddings of different groups of
polarities trained on financial News or on Wikipedia (_Random_ means either
random words –for Financial News– either all the other words –for Wikipedia–).

_It is not what we observe_ :

  1. 1.

if Positive-Positive and Negative-Negative (blue and orange bars) are the
highest, it is not by far, especially for embeddings learned on the headlines.

  2. 2.

_Positive-Negative similarity (violet) is never the lowest_ , it is comparable
to distance to random words for embeddings

  3. 3.

Positive-Random and Negative-Random (red and green) are not very low _except
for embeddings learned on the full text_ of the News.

##### Conjectures on frequentist synonyms amongst polarized worlds in finance.

The first empirical conclusion we can make at this stage is that _it is
difficult for embeddings to separate financial polarized antonyms_. The fact
that the Positive-Negative (violet) bars are never lower than the Positive or
Negative vs. Random (red and green) bars is an evidence of this difficulty.
Nevertheless, _it is easier to separate financial polarized words with
embeddings learned on the full text of News rather than on headlines only_. We
can conjecture that it is because financial headlines are written to be
quickly understood by humans, and that for they contain similar sentences with
positive or negative words in the same environments. Indeed: when a corpus is
structure with short sentences of similar structure, the chances that semantic
antonyms becomes frequentist synonyms are high. This configuration disappear
when the corpus of full text (body) of financial News is used. Last but not
least Positive-Random and Negative-Random similarity is particularly close to
zero when embeddings are learned on the full body of the News. This similarity
is even lower for embeddings trained on the full text of financial News rather
than on the Wikipedia corpus.

Our theoretical analysis shed light on this: it is difficult for the Skip-gram
word2vec to make the difference between frequentist synonyms. From a corpus of
short and very structured sentences, antonyms have good chances to become
frequentist synonyms.

As a conclusion of this first analysis, we can conclude that using embeddings
to discriminate polarity will be more difficult on a the corpus of headlines,
that is not diverse enough, and financial enough (since it does not work is
well on Wikipedia).

###  4.2 Stationarity of embeddings associated to polarized words

####  4.2.1 The cosine similarity between polarities is stationary

Figure 7: Yearly cosine similarity between the embeddings of Loughran-McDonald
Positive and Negative lexicon, for a skip-gram word2vec model learned on the
headlines only (bottom) or on the full text (top) of financial News.

Table 1 shows the number of document in our corpus each year; we have clearly
less headlines in 2008, 2012, 2013 and 2014, and it is reflected in the
average cosine similarities of Figure 7 (top panel). There is not enough
documents, hence not enough diversity in the sentences, and as a consequence
words are not been seen in differentiating enough contexts: the global cosine
similarity is higher.

That being put aside, we observe that the averaged results of Figure 6 are in
line with their time series representation: the ordering of cosine
similarities between groups of words of same polarity are the same every year.
Despite a widening of the difference between words of the same polarity (red
and blue lines) and the cosine between polarized words and random words
(orange and violet lines) in 2017 and 2018, _the embeddings seem to be
stationary_ in the sense that there is no real change in their relative
ranking from one year to another.

####  4.2.2 Influence of the embedding size on the polarities

Figure 8 Shows the influence of embedding size on the cosine similarity
between embeddings of Positive words with other classes (Positive, Negative,
Random) words.444The effects are the same for Negative words. The empirical
results are mixed since on the one hand when the dimension decreases, the
similarity inside the class of positive words increase (that is good), but the
similarity between positive and negative words increases too. The similarity
to random words stays close to zero.

##### Conjecture on the influence of embedding size on frequentist synonyms.

This observation is compatible with the idea that the lower the dimension to
represent the language, the more difficult to memorize the differences in
contexts of words. As a consequence, words with the same polarity will become
more similar, but antonyms will become more similar too.

Figure 8: Change of the cosine similarity between positive words and random
words as a function of the size of embeddings.

####  4.2.3 When a company name becomes a frequentist synonym

Figure 9 document an interesting effect that can have an influence on using
embeddings to predict returns of listed companies. The important question to
ask is if we want that a sentence like “ _Microsoft printed booming results_
”555“booming” is part of the Loughran-McDonald list of positively polarized
words. is understood the same way as “ _Google printed booming results_ ” by
an embedding-based system. It would require that _Microsoft_ and _Google_ have
no polarity. There is no semantic reason for a company name to be positively
or negatively biased, i.e. to have a cosine similarity different from zero to
a group of polarized words.

This analysis is not systematic, we simply took the only company name of
Wikipedia’s _List of corporate collapses and scandals_ that is in our
database, and computed its cosine similarity with the five most numerous lists
of the Loughran-McDonald lexicon (Positive, Negative, Litigious and
Uncertain).

Figure 9: Yearly cosine similarity between the embeddings of Theranos and the
Loughran-McDonald lexicon (it is measured using the model.wv.n_similarity
function of gensim).

Before commenting the results, have a look at the quick summary made by
Wikipedia666On Theranos Wikipedia page as of the 10th of March 2020. of the
Theranos case:
In March 2018 the US Securities and Exchange Commission charged Theranos, its
CEO Elizabeth Holmes and former president Ramesh ”Sunny” Balwani, claiming
they had engaged in an ”elaborate, years-long fraud” wherein they ”deceived
investors into believing that its key product – a portable blood analyzer –
could conduct comprehensive blood tests from finger drops of blood”.

On Figure 9, we see the change in the polarity of the embedding of the term
_Theranos_ in the embeddings. This means that in 2018 and 2019, sentences with
the name of this company is tinted with negativity and litigation compared to
the same sentence concerning another company. With an exaggerated
anthropomorphism, we could say that the embedding now “believes” that most
sentences concerning this company are negative or litigious. In fact this
company name became a frequentist synonym of negative and litigious terms.
Probably because from the viewpoint of skip-gram Word2vec loss function, it
was “easier” to get this company name closer to negative and litigious terms
than to keep it away from them (like any company name should semantically be).

The goal of this paper is not to investigate further on this kind of
polarization of entities, that is in fact covered in a completely different
context by the literature on _fairness of NLP_ , see [30] for an example and
[31] for an overview.

##  5 Conclusion

In this paper, we presented some theoretical understanding of word embeddings,
essentially using the skip-gram word2vec model. Moreover we explain why more
sophisticated model should inherit, at least locally, of some of these
properties. It allows us the define the concept of Reference Model (that is
the uncompressed version of an embedding model), and to show that
asymptotically the loss function of such a learning algorithm is a cross
entropy between the representation of the model and the distribution of the
Reference Model. Moreover, it lead us to define frequentists synonyms, i.e.
words that have the same context in the considered corpus. It is impossible
for embeddings to make the difference between exact frequentist synonyms, and
difficult for approximate synonyms.

Then we test these concepts on synthetic corpora generated using controlled
Markovian models, so that we can focus on the identifiability of skip-gram
word2vec embeddings. We observe that if their identifiability is poor, the
cosine similarity between embeddings makes sense, even when it is low:
frequentist synonyms are closer to word from their class than to words of
another group of synonyms.

Last but not least, we provide empirical observations on a financial corpus:
we use the Loughran-McDonald lexicon to obtain semantic synonyms and antonyms:
lists of polarized words (Positive vs. negative words essentially). And we
compare the polarity of embeddings trained on headlines of News with other
trained on the full text of the same News. We observe that on News headlines,
that are short and structured sentences, semantic antonyms are often
frequentist synonyms, and hence it is difficult for embeddings learned on such
headlines to make the difference between positive and negative words. On the
opposite, embeddings learned on the full body of the News are more reflecting
the polarities of the considered lexicon. In fact they are better reflecting
financial polarities than embeddings trained on Wikipedia. It seems that the
dimension of the embeddings has an influence on the cosine similarities
between polarities: the lower the dimension, the more difficult to make the
difference, in the space of embeddings, between positive and negative words.
We moreover observe that names of companies can be tinted with polarity. The
structure of the loss function of embeddings can lead them to accept to
represent a company name close to a polarized word if it appears a lot in
negative News: this company name is thus becoming a frequentist synonym of
negative and litigious words.

## References

  * [1] TKLD Laham and Peter Foltz.  Learning human-like knowledge by singular value decomposition: A progress report.  Advances in neural information processing systems, 10:45, 1998.
  * [2] Krista Lagus, Timo Honkela, Samuel Kaski, and Teuvo Kohonen.  Websom for textual data mining.  Artificial Intelligence Review, 13(5):345–364, 1999.
  * [3] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26, pages 3111–3119. Curran Associates, Inc., 2013.
  * [4] Shalini Ghosh, Oriol Vinyals, Brian Strope, Scott Roy, Tom Dean, and Larry Heck.  Contextual lstm (clstm) models for large scale nlp tasks.  arXiv preprint arXiv:1602.06291, presented at KDD 2016, 2016.
  * [5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6000–6010, 2017.
  * [6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [7] Omer Levy and Yoav Goldberg.  Neural word embedding as implicit matrix factorization.  In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27, pages 2177–2185. Curran Associates, Inc., 2014.
  * [8] David M Blei, Andrew Y Ng, and Michael I Jordan.  Latent dirichlet allocation.  the Journal of machine Learning research, 3:993–1022, 2003.
  * [9] Avner May, Jian Zhang, Tri Dao, and Christopher Ré.  On the downstream performance of compressed word embeddings.  Advances in neural information processing systems, 32:11782, 2019\.
  * [10] Vikas Raunak, Vivek Gupta, and Florian Metze.  Effective dimensionality reduction for word embeddings.  In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 235–243, 2019.
  * [11] Anish Acharya, Rahul Goel, Angeliki Metallinou, and Inderjit Dhillon.  Online embedding compression for text classification using low rank matrix factorization.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6196–6203, 2019.
  * [12] Tim Loughran and Bill McDonald.  When is a liability not a liability? textual analysis, dictionaries, and 10-ks.  The Journal of finance, 66(1):35–65, 2011.
  * [13] Abhishek Kumar, Abhishek Sethi, Md Shad Akhtar, Asif Ekbal, Chris Biemann, and Pushpak Bhattacharyya.  Iitpb at semeval-2017 task 5: Sentiment prediction in financial text.  In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 894–898, 2017.
  * [14] Zheng Tracy Ke, Bryan T Kelly, and Dacheng Xiu.  Predicting returns with text data.  Technical report, National Bureau of Economic Research, 2019.
  * [15] Xiaodong Li, Haoran Xie, Li Chen, Jianping Wang, and Xiaotie Deng.  News impact on stock price return via sentiment analysis.  Knowledge-Based Systems, 69:14–23, 2014.
  * [16] Frank Z Xing, Erik Cambria, and Roy E Welsch.  Natural language based financial forecasting: a survey.  Artificial Intelligence Review, 50(1):49–73, 2018.
  * [17] Dogu Araci.  Finbert: Financial sentiment analysis with pre-trained language models.  arXiv preprint arXiv:1908.10063, 2019.
  * [18] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient Estimation of Word Representations in Vector Space.  arXiv e-prints, page arXiv:1301.3781, January 2013.
  * [19] Xin Rong.  word2vec Parameter Learning Explained.  arXiv e-prints, page arXiv:1411.2738, November 2014.
  * [20] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep Learning.  MIT Press, 2016.  http://www.deeplearningbook.org.
  * [21] David M. Blei, Andrew Y. Ng, and Michael I. Jordan.  Latent dirichlet allocation.  The Journal of Machine Learning Research, 3(null):993–1022, March 2003.
  * [22] Daniel Revuz.  Markov chains.  Elsevier, 2008.
  * [23] David A. Levin, Yuval Peres, and Elizabeth L. Wilmer.  Markov chains and mixing times.  American Mathematical Society, 2006.
  * [24] Jean-François Le Gall.  Intégration, probabilités et processus aléatoires, Septembre 2006.
  * [25] Justin Salez.  Temps de mélange des chaînes de markov.
  * [26] Terence Tao.  Topics in random matrix theory, volume 132.  American Mathematical Soc., 2012.
  * [27] Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun.  The loss surfaces of multilayer networks.  In Artificial intelligence and statistics, pages 192–204. PMLR, 2015.
  * [28] Zi Yin and Yuanyuan Shen.  On the dimensionality of word embedding.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31, pages 887–898. Curran Associates, Inc., 2018.
  * [29] Matthew Gentzkow, Bryan Kelly, and Matt Taddy.  Text as data.  Journal of Economic Literature, 57(3):535–74, 2019.
  * [30] Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Anna Rumshisky, and Adam Tauman Kalai.  What’s in a name? reducing bias in bios without access to protected attributes.  arXiv preprint arXiv:1904.05233, 2019.
  * [31] Deven Shah, H Andrew Schwartz, and Dirk Hovy.  Predictive biases in natural language processing models: A conceptual framework and overview.  arXiv preprint arXiv:1912.11078, 2019.
  * [32] Murhaf Fares, Andrey Kutuzov, Stephan Oepen, and Erik Velldal.  Word vectors, reuse, and replicability: Towards a community repository of large-text resources.  In Proceedings of the 21st Nordic Conference on Computational Linguistics, pages 271–276, Gothenburg, Sweden, May 2017. Association for Computational Linguistics.

##  Appendix A Technical details

We use Python 3.7.6 and Gensim 3.8.3 to train our models on AWS with financial
news. On financial news headlines, we set min_count=1 to include every word in
headlines while we let min_count=5 as the default value of Gensim on financial
news which means we only consider the words count more than 5 times. Our
epoque =1, i.e., each training sample is used only 1 time. As the model is
initialized by training sentences, our total_examples=model.corpus_count.

Training skip-gram word2vec on one year of News takes around 2h and on one
year of headlines it takes around 1h30. We use ml.p2.xlarge as of March 2021,
i.e. 1 NVIDIA K80 GPU, 1 vCPU, and 64Go of RAM.

The model trained on English Wikipedia Dump of February 2017 uses Gensim
Continuous skip-gram with no lemmatization. It is provided by Language
Technology Group at the University of Oslo in NLPL word embeddings repository
[32].