# Word Embedding for Social Sciences: An Interdisciplinary Survey

Akira Matsui 1
<EMAIL_ADDRESS>Corresponding author 1 College of Business
Administration, Yokohama National University, Yokohama, Kanagawa, Japan
2 Annenberg School of Communication, University of Southern California , Los
Angeles, CA, US
3 Information Sciences Institute, University of Southern California , Marina
Del Rey, CA, US
4 Department of Computer Science, Viterbi School of Engineering, University of
Southern California, Los Angeles, CA, US
Emilio Ferrara 2,3,4
<EMAIL_ADDRESS>1 College of Business Administration, Yokohama National
University, Yokohama, Kanagawa, Japan
2 Annenberg School of Communication, University of Southern California , Los
Angeles, CA, US
3 Information Sciences Institute, University of Southern California , Marina
Del Rey, CA, US
4 Department of Computer Science, Viterbi School of Engineering, University of
Southern California, Los Angeles, CA, US

(version: 1.1; last updated:)

###### Abstract

To extract essential information from complex data, computer scientists have
been developing machine learning models that learn low-dimensional
representation mode. From such advances in machine learning research, not only
computer scientists but also social scientists have benefited and advanced
their research because human behavior or social phenomena lies in complex
data. However, this emerging trend is not well documented because different
social science fields rarely cover each other’s work, resulting in fragmented
knowledge in the literature. To document this emerging trend, we survey recent
studies that apply word embedding techniques to human behavior mining. We
built a taxonomy to illustrate the methods and procedures used in the surveyed
papers, aiding social science researchers in contextualizing their research
within the literature on word embedding applications. This survey also
conducts a simple experiment to warn that common similarity measurements used
in the literature could yield different results even if they return consistent
results at an aggregate level.

##  1 Introduction

The advancement of machine learning technologies has revolutionized
traditional research methodologies in the social sciences, enabling the
extraction of insights from unconventional data sources. Recently, we have
been witnessing the expansion of social science research that explores textual
data, captivated by the concept of “Text as Data”[1, 2, 3]. This trend is
evident not only in fields that traditionally rely on quantitative analysis,
such as economics[4] and finance [5], but also in disciplines with a penchant
for qualitative analysis, like history [6]. The flourishing body of literature
in this domain enables social scientists to meticulously select or develop
machine learning models for text mining that meet the demands of their
research, providing a vast array of options. While rule-based models, such as
dictionary-based models, remain popular [7, 8], neural network-based embedding
models are gaining increasing traction in social science research.

Embedding models learn low-dimensional representations of entities from data
by predicting relationships between these entities [9]. The advantage of
embedding models in social science research lies in their ability to acquire
the “embedded” structure in an unsupervised manner. This inherent flexibility
allows social scientists to apply these models to their data with minimal
constraints, facilitating their analyses. These entities could be nodes in a
graph [10, 11, 12], pixels in an image [13, 14], or vertices in a network [15,
16]. Despite the multitude of embedding algorithms available for diverse
purposes, word embedding models have been recognized as some of the most
widely adopted methods [17, 18, 19, 20, 21, 22]. Among these, the most
accessible word embedding model for social science research is the word2vec
model [19, 21, 18], used in diverse fields ranging from management science
[23, 24, 25, 26, 27, 28, 29, 30, 31] to poetry [32].

The rise in the use of word2vec and other related models in social science
research has not been accompanied by adequate documentation of how these
models are utilized. However, because this trend spans multiple disciplines,
we are witnessing lacks sufficient cross-communication. First, this absence of
an integrated knowledge framework hampers discussions about the broader impact
and relevance of word embedding in social science research. Second, The
absence of comprehensive surveys spanning multiple disciplines has can led
social scientists to reference only literature within their own fields,
inadvertently leading to the reinvention of methodologies already developed in
other domains. Consequently, this siloed approach often results in deserving
scholarly works not receiving the recognition and citation they merit, thus
undermining their rightful academic evaluation. Therefore, we need to review
the broad spectrum of studies that should be referenced, many of which fall
outside their immediate expertise.

To address the knowledge gap resulting from a lack of connections between
disciplines, this paper aims to identify emerging interdisciplinary trends in
machine learning models arising across diverse fields. In this literature
review, we seek to bridge the knowledge gaps between different fields and
promote a more integrated and effective use of these models in social science
research. This literature review builts a taxonomy of these application
methodologies to illustrate how social scientists harness word embedding
models, particularly focusing on word2vec. The taxonomy emphasizes the
applications of word embedding published in social science journals, outlining
the contemporary landscape of social science research utilizing word embedding
models, rather than delving into the technical intricacies of these neural
network models. This literature review targets social scientists who are
currently using or considering the use of word embedding models like word2vec
in their research. Therefore, the presented paper does not comprehensively
review the state of the art in the technical aspects of embedding models.
Instead, we offer a review that covers various social science disciplines,
especially for those who may not specialize in machine learning or natural
language processing. To provide an overview of such different fields, we
develop a taxonomy to guide potential readers in situating their studies
within this interdisciplinary literature. This survey also benefits developers
and theorists in computational linguistics and machine learning, helping them
understand and enhance the application of their models in social science
contexts.

The paper is structured as follows. After the discussion of search
methodology, we begin this survey with a brief overview of the technical
background of word2vec in Section 2. Then, we introduce the nine labels used
in our taxonomy and review the papers with detailing the taxonomy throughout
Section 3.2.1 to Section 3.2.6. We include discuss a pitfall of similarity
measurements with a simple experiment. In addition, we employ the proposed
taxonomy to understand the literature using word2vec model for social science.

###  1.1 Search Methodology

In this survey, our primary focus has been on papers published in social
science journals, aiming to conduct a concentrated literature survey. Although
our main attention was not directed towards research in word embedding
published in computer science conference proceedings, we selectively included
some of these papers to ensure a comprehensive understanding of the field.
This approach was essential to paint a complete picture of the literature
landscape.

We primarily focused on papers that had a clear application or discussion of
word embedding models in a social science context. We gave preference to
articles published in peer-reviewed journals and conference proceedings to
guarantee the credibility and academic rigor of our sources. Conversely, we
excluded papers that did not fall within our specified subject areas. Non-
English publications were also excluded due to language constraints. This
exclusion was necessary to ensure that our sources were reproducible and
verifiable by our readers.

For this survey, we predominantly used the Scopus Database, implementing a
specific query that included terms word2vec and combinations such as word AND
embedding, refined further by limiting the search to certain subject areas
like social sciences, arts, business, environmental studies, and economics,
and to specific document types like articles, chapters, and reviews. To ensure
that our survey was as exhaustive as possible, we expanded our search to
include Google Scholar and the Social Science Research Network (SSRN) to
include working papers or preprints, adapting our query to suit their unique
search algorithms.

Through this meticulous and methodical approach, we aimed to ensure a
comprehensive and unbiased coverage of the literature. This strategy was
crucial in facilitating a robust analysis of the current state of word
embedding applications in social science research, thereby contributing
significantly to our understanding of the field.

##  2 Word embedding models

We briefly review the technical aspects of word embedding models before diving
into the main part of our survey. Since our focus is the literature that uses
word2vec models, this section discusses the word2vec algorithm. This section
ensures that the presented paper provides sufficient technical prerequisites
of word2vec, while in-depth explanations are relegated to relevant literature
sources. 111[33] summarize the history of word embedding models for social
scientists, and [17] provide an excellent description of word2vec.

Word embedding models learn the relationships between words in sentences in
the data and return embedding vectors based on their learned model. The
embedding vectors are considered to represent the semantics of the words in
the sentence, and the distributional hypothesis [34] supports this assumption.
The theory asserts that a word is determined by the surrounding words, and
experiments with human subjects have validated that word embedding models can
capture semantics similar to human cognition [35]. This prompts social
scientists to assume that studying word embedding models learned from the text
of interest can reveal how humans process information embedded in the text.

Evaluations of word embedding models roughly take two forms: intrinsic
evaluation and extrinsic evaluation. Typical tasks for intrinsic evaluations
are analogy and similarity. Many datasets for intrinsic evaluation are
publicly available and accessible (Analogy [36, 37, 38, 39, 40, 41, 42, 43,
44], Similarity [18, 21]). For extrinsic evaluation, they often use word
embedding models for translation or sentiment analysis. See [45] for recent
evaluation methods.

Although there are various word embedding models, social scientists have
favored word2vec as the most popular model due to its simplicity, high user-
friendliness, and longstanding usage [17, 46]. Thus, this survey paper
primarily focuses on the application of word2vec in social science research
while also covering other language models that describe the new trend of this
interdisciplinary research.

###  2.1 word2vec: A popular word embedding model

We begin the explanation of the technical background of word2vec by clarifying
the terminology of “word2vec,” because the term does not designate a specific
neural network model or algorithm but rather refers to a software program and
tool. Pedagogically speaking, word2vec denotes software that combines a
learning algorithm and a training method [17]. For such packages, the
continuous bag of words model (CBOW) and the skip-gram model are common
implementations within word2vec, and the negative sampling algorithm is a
widely-used technique [18, 19, 20, 21, 22]. While computer science papers
usually describe the details of their models, social science papers often use
word2vec as if it refers to a specific neural network model, and discussions
of learning algorithms and training methods are not very common in social
science papers except in those discussing research methods such as
econometrics or statistical models. Therefore, when a paper mentions only
“word2vec” for their methods, it may use the CBOW or the skip-gram model 222In
this survey, we review the papers [47, 23, 35] that uses GloV model [48], and
[49] that uses fastText model [50].

####  2.1.1 CBOW, Skip-gram Model and SGNS Model

The CBOW and skip-gram solve a “fake” problem that predicts what words appear
in a given text. The main goal of the model is not to make predictions but to
acquire the parameters obtained by solving the prediction problem as a low-
dimensional representation of words. In other words, the prediction problem
solved by word embedding is only to learn the parameters, and the prediction
problem itself is a problem that is not of central interest. Given this
background, we refer to prediction problems that word embedding solves to
learn the low-dimensional representation as“fake” problems. To explain this
fake problem in the word embedding model, we often use the terms “word” and
“context” 333While “target” is also popularly used instead of “word,” we stick
to use “word” in this survey to avoid potential confusions.. Target is the
element in the text and is often a word. Context is the set of elements around
the target and is often the set of words.

Consider the sentences, The stick to keep the bad away. The rope used to bring
the good toward us. When the target is rope, its context could be [bad, away,
the, used, to, bring] with a window size of three. The window size determines
the context size and the number of elements to capture from the target. This
example considers the three words to the left and right of the target word.
While both the CBOW and skip-gram models solve the prediction model using the
context and target, they differ in which part predicts the other. The CBOW
model predicts the target $w$ using the context $c$, $P(w\mid c)$. Conversely,
the skip-gram model predicts the context using the target, $P(c\mid w)$. The
most used word2vec is the skip-gram negative sampling (SGNS) model that
combines the skip-gram model and negative sampling algorithm because this
combination is provided in the populer ready-to-use library gensim [51, 52].

The SGNS models the distribution, $p(d\mid w,c)$, where $d$ takes 1 when a
pair of target $w$ and context $c$ is observed in the data, 0 otherwise. The
SGNS maximizes the following conditional log-likelihood [53],

$\sum_{(w,c)\in\mathcal{D}}\left\\{\log p(d=1\mid
c,w)+k\mathbb{E}_{\bar{w}\sim q}\log p(d=0\mid c,\bar{w})\right\\},$ (1)

where $q$ is the noise distribution in negative sampling, and k is the sample
size from the noise444In the literature, it is often mentioned that this log-
likelihood maximization with the noise distribution is considered as Noise
Contrastive Estimation (NCE) [54], and the SGNS is one of the variations of
NCE. ;$\mathcal{D}$ is the set of paris of target $w$ and context $c$. The
SGNS especially calculates the conditional probability $p(d=1\mid c,w)$ by
$\sigma(v_{c}\cdot v_{w})$, where $\sigma(x)$ is sigmoid function and
$v_{w},v_{c}\in R^{d}$. In other words, the SGNS seeks the parameter
$v_{w},v_{c}$ that maximizes the above conditional probability, and
$v_{w},v_{c}$ is the embedding vectors of interests.

A typical model constructs the word sets with the words around the target
word. However, the manner of constructing the context does not necessarily
have to be the surroundings of the target word; it can be arbitrary. Since
context determines the relationship between targets and words in the data, we
can define the context based on external information. [55], for example,
propose the word2vecf model, a modified version of the word2vec model, which
obtains word embedding vectors by constructing the context based on the
dependency between words. The word2vecf model can construct an arbitrary
context for the word2vec model and allows us to apply the relationships in
non-text data, not only the dependency between words in sentences. We discuss
this trend in Section 3.2.6.

##  3 Labeling and Taxonomy

We are now ready to move on to our main focus in this survey paper, which is
to provide an overview of how word embedding models are used in social science
research. This field is growing in various disciplines, making it challenging
to address concerns cohesively. To give a clear overview of the research area,
we develop a taxonomy of the analytical methods used in the surveyed social
science papers. This approach allows us to review the studies from a more
concrete viewpoint. The taxonomy introduces the labels of disciplines and
methods to discuss the literature. With the constructed label, we will survey
a wide range of papers with word2vec employs a reference-based analysis to
ensure the interpretability of results and to achieve their analytical
objectives. This literature survey also points out that a new line of
literature applies word2vec to non-textual data, such as relationships between
users in web platforms.

###  3.1 Labeling

We construct two types of labels for this survey: research topics and methods.
First, we group the papers by their research topics based on the journals or
conferences in which they were published. For working papers, we infer the
topic based on the author’s faculty affiliation and past publication history
to determine the research area, resulting in 14 research areas. Note that we
understand that defining or classifying research fields is not a trivial task.

We classify the methods that is the way the surveyed papers use word embedding
models for their research. This method-oriented classification would benefit
us when surveying the interdisciplinary topics. The nature of
interdisciplinary topics prevents us from discussing each paper according to
its research topics or questions because showcasing those high-diverse
outcomes dilutes the visibility of the survey. Therefore we, instead of
following the literature of each research field, study the way that the
surveyed papers use word embedding models for their analysis, such as what
kind of tasks they use word embedding models for and how they construct
variables for their research. To overview how the surveyed papers use word
embedding models, we build the labels of their analysis. For this survey, we
define eight labels and summarize the descriptive definitions of the
analytical method labels in Table 1. We summarizes the papers we surveyed in
Tables 2 and 3. In the following sections, we discuss each label by reviewing
several representative papers of each.

Table 1: Labels for Analysis Methods With Word Embedding  Labels |  Definition
---|---
Pre-trained |  Pre-trained model
Working variable |  Variables constructed from a word embedding model for analysis
W/theory |  Validating a theory proposed in a research field for analysis, or quoting it to justify an analysis.
W/reference |  Using references such as the words with specific semantic to define
Non-text |  Applying word embedding model to non-text data (e.g., human behavior data)
Same words |  Comparing the same words from different models
Human subj |  Employing human subjects for analysis
Clustering |  Clustering analysis using word embedding vectors
Prediction |  Prediction using word embedding vectors

###  3.2 Taxonomy

This section describes the taxonomy built upon on the labels constructed in
Section 3.1 and we discuss several representative papers for each
methodological label.

####  3.2.1 Pre-trained Models

Social science papers using word embedding models mainly focus on extracting
information from data to answer their research questions. Obtaining low-
dimensional representations from data allows us to extract the semantics or
hidden patterns in the data. When a social scientist is interested in a
specific subject, they train a word embedding model on their own data, such as
descriptions about central banks [56, 57], judges [47], organizations [28],
job openings [30], or smartphone applications [49]. To study these topics,
researchers build their own datasets and train their word embedding models on
the data.

While most studies construct their own word embedding models, pre-trained word
embedding models are also popular, with papers using pre-trained models
published in prestigious journals [58, 33, 59]. Pre-trained word embedding
models are ready-to-use models that have already been trained on large
corpora. Most pre-trained models are trained on general corpora and are open
to the public, such as the Google News pre-trained model [60] and pre-trained
models on historical corpora in several languages [61]. Models trained on
general corpora are supposed to represent general semantics, allowing studies
with these pre-trained models to answer broad research questions, such as
evaluating biases in publications [59, 35] and culture [33].

Studies have demonstrated that employing human subjects to validate the
findings of pre-trained models can aid our understanding of crucial human
perceptions [35, 27, 29]. Such studies often use models pre-trained on general
corpora to understand common human biases. For example, management scientists
combine surveys from human subjects and pre-trained models to investigate
essential concepts in their discipline, such as perceptions of risk [58] and
leadership [27]. Additionally, because of their generality, pre-trained models
help researchers validate theories proposed in their disciplines. For example,
researchers in neuroscience propose features for interpreting word embeddings
based on the method called Neurosemantic decoding [62].

####  3.2.2 Overfitting Models

Rather than focusing on general social phenomena, social science research
often targets specific subjects or classes. In such studies, researchers often
“overfit” word embedding models on corpora that represent specific subjects.
Overfitting is typically avoided by computer scientists to ensure the
generality of their trained models since overfitted models do not perform well
on computational tasks such as analogical reasoning or generating language
models, as discussed in Section 2.

However, social science researchers are interested in the results from
overfitted models because these models can capture the specificity of the
data, There are the line of research, for example, that investigates human
biases, such as stereotypes, are “embedded” in the text of interest [63, 47,
64, 59, 65, 66, 67]. Word embedding models learns from the corpora can reveal
biases where certain words are over-fitted with specific human characteristics
such as gender, and these computed biases are similar to the biases humans
hold [35]. In the context of fairness in machine learning, significant
computer science literature proposes methods for debiasing such biases [68,
69, 70, 71] and evaluates their validity [72].

####  3.2.3 Working Variable

We find that most papers we surveyed use the obtained vectors to define
variables that embody the concept or research questions to be examined in
their study, while some papers use word embedding vectors for clustering [73,
74, 29, 30, 75] or prediction [27, 76, 23, 77] tasks. To clarify this trend,
we call this procedure the working variable” using an analogy from working
hypothesis,” which is a well-used term in social science research [78]. To
answer their research questions, social scientists transform a theoretical
hypothesis into a working hypothesis that can be tested by experimental or
observational research.

In this survey, we define “working variable” as a variable that is a proxy
variable of theoretical interest. By defining working variables, researchers
investigate theoretical hypotheses in various fields and external objects such
as human perception. A working hypothesis is testable using empirical analysis
and is often transformed from a conceptual hypothesis or theoretical
hypothesis. While a theoretical hypothesis is a conceptual description of the
hypothesis to be proved in the research, a working hypothesis describes an
executable procedure or test that can prove the theoretical hypothesis. For
example, to analyze stereotypes in the documents of interest, researchers can
use word embedding to calculate some working variables that proxy stereotypes
[59]. Not only human perception, but also qualitative concepts can be defined
as working variables such as the speed of semantics in documents like research
papers [79, 24], law documents [79], and the plots of movies and TV shows [79,
24].

Because social science studies often study phenomena regarding specific events
or concepts, many define a working variable to analyze the phenomenon or
concept under analysis. Working variables are often defined as a scalar value
obtained by computing the embedding vectors obtained in the study. For
example, working variables can be proxies of important concepts, such as
opacity [80] or grammar [81]. With embedding model, authors can construct
their own indicators for their research question [82]. Calculating these
working variables often employs the reference words to be described in Section
3.2.4. Defining not only a single working variable but also multiple working
variables clarifies the concept to be studied. [24] quantified the shape of
stories by defining multiple variables such as speed, volume, and
circuitousness in the text.

There are studies that directly calculate vectors obtained from word embedding
models and some researchers use word embedding models to estimate their own
statistical model for causal inference [77] and equilibrium estimation [49],
rather than analyzing the subject represented using the embedding vectors.

####  3.2.4 Reference Words

There are two primary ways to compare the obtained word vectors with reference
words: direct and indirect comparisons. Analysis using reference words defines
working variables as the distance between the selected reference words and the
words of interest. This method determines appropriate “reference” words
representing a concept to be analyzed.

[59] analyze the gender stereotypes in text over one century by calculating
the relative similarity between the words related to gender (men or women) and
the words related to specific occupations. To calculate the relative
similarity, they first calculated the distance between the word embedding
vectors of words related to men and the words describing specific occupations.
They also calculated the same distance for women-related words. Then, they
computed the relative differences between these two distances. When the
relative distance with an occupation is large, it means that the stereotype in
the text is considerable. For example, they determined that “engineer” is
close to the men-related words in the historical corpus and argued that this
suggests the existence of stereotypes in the corpus. The paper showed that the
measured gender stereotypes are correlated with occupational employment rates
by gender. They also conducted the same analysis for ethnic stereotypes.

There are two advantages to using reference words for analysis. The first is
that introducing references improves the interpretability of the results; for
example, if we use the Subjectivity Lexicon as the reference word, we can
measure the degree to which the word of interest is subjective [56]. The
second advantage is that it avoids the problems associated with coordinate
systems when comparing different models, and we detail this point in Section
3.2.5.

####  3.2.5 Comparing the Same Words

Some papers prefer simple methods of analyzing the similarity between words.
For example, this category contains the analysis of enumerating words that are
similar to a certain word, which is often done in word embedding tutorials
because the similarity of each word is calculated to list similar words. Some
papers have used this simple analysis as seminal work and have yielded useful
findings [25].

Calculating the similarity between the same words from different models could
also answer important research questions for social science. The same word
often has different meanings depending on the speaker and the time period in
which the word was used. Examining such diachronic changes in semantics and
changes in meaning by the speaker can address research questions that arise
from natural social science ideas. [83], for example, examined the difference
in perception between experts and laypeople by investigating whether these two
groups use the same word to describe different semantics according to their
profession.

Comparing the same words from different data requires different learning
models from different corpora. When we want to compare the semantic
differences of the same word from two different documents, we need to train
two separate word embedding models on each document. It is important to note
that we cannot directly compare the two models, even if they were trained with
the same algorithm. This is because the learning algorithm uses an arbitrary
coordinate system with each training. To overcome this issue, [84] used
orthogonal Procrustes to align different models to reveal historical changes
in the semantics of words, and some other studies follow this method [83].
Orthogonal Procrustes, however, may not be stable [85]. Therefore, an improved
method using the Dynamic Word Embedding model, which explicitly incorporates
semantic changes, has been proposed [85, 86], and some papers use the Dynamic
Word Embedding model to analyze semantic changes [56]. Thanks to recent
studies on the statistical properties of word embedding models [87],
researchers have proposed a way to compare different models [88] without
aligning models, which will advance the use of word embedding models from a
more social science perspective. Additionally, this problem can be avoided if
the comparison is done within the same model and is projected onto a scalar
value. [59] calculate stereotypes as scalar values and investigate the
trajectory of the values over time. We can consider this as another advantage
of introducing reference words that allow us to avoid the problems stemming
from the issue of word embedding models using an arbitrary coordinate system
in their learning.

####  3.2.6 Non-text

Additionally, there is a recent trend that applies word2vec to non-textual
data. As discussed in Section 2.1.1, We can apply word embedding models to
non-textual data since word embedding can learn relationships between not only
words but also any entities such as metadata [89] and symbols [31], in
addition to text data. We also survey research that applies word2vec
algorithms (skip-gram or CBOW) to digital platform and geographic data [18,
19, 20, 21, 22].

Therefore, the relationships that word embedding models learn are not limited
to words in text. Word embedding models can obtain low-dimensional
representations from relationships composed of non-text data. For example,
[73] and [90] obtain embedding vectors of Points of Interest (POI) from
geological data describing the relationships between POIs. With their
clustering algorithm, they suggested it can discover the usage pattern of
urban space with reasonable accuracy. [91] noted an important theoretical
insight for analyzing geographic data with word2vec. They revealed that
word2vec is mathematically equivalent to the gravity model [92], which is
often used to analyze human mobility and conducted experiments with real
datasets to validate their findings.

Word embedding models can also learn relationships between a chemical equation
and the text around it in a research paper. [31] apply word2vec to academic
papers in chemistry to learn the relationship between text and symbols
(chemical formulae). They demonstrate that potential knowledge about future
discoveries can be obtained from past publications by showing that the
obtained embedding model can recommend discoveries several years in advance of
the actual discovery. Not only symbols but also other information can be
embedded. [89] analyze politicians’ speeches using the word2vec algorithm to
embed information about politicians’ ideologies along with the text data of
their speeches.

Adapting word2vec to non-text data may yield useful insights even when text
data is available. [93] adopt a modified version of the word2vec algorithm
555Waller and Anderson (2021) used an algorithm called word2vecf, which is a
modified version of word2vec [55, 94]. to analyze the posting behavior of
Reddit users. They obtained embedding vectors by learning the relationships
between Reddit users and subreddits.666In Waller and Anderson (2021), a
subreddit is called a community. It is worth noting that they do not analyze
the textual data of Reddit posts to investigate users’ behavior. Studying
Reddit text data might unveil the attributes of Reddit users, such as
demographic information, preferences, and age groups. However, user comments
do not always reflect such meta-information. Some supervised machine learning
models may predict such meta-information; there are successful unsupervised
methods that predict the characteristics of text, such as sentiment analysis
[95, 96, 8] or moral foundations dictionary [97, 98]. Moreover, the validity
of training data is difficult to prove due to issues regarding the annotator’s
subjectivity. To overcome this problem, the authors used a data-driven and
linguistically independent method that characterizes subreddits only from the
perspective of ”what kind of users are posting in the community.” By setting
reference “words” (subreddits), they created indicators such as age, gender,
and partisanship. For example, to calculate the working variable of
“partisanship,” they chose opposing communities in Reddit: “democrats” and
“conservatives.” Then, they calculated the relative distance for each
community. Applying our taxonomy to [93] reveals that, while they do not use
text, they share characteristics with other studies in their methodologies.
They define a working variable (partisanship) using reference words
(subreddits). In addition, they are similar to the analysis by [24] in terms
of setting multiple working variables.

Several studies apply the word embedding algorithm to learn relationships
between text and non-text data or between non-text entities. As discussed in
the introduction, this survey primarily focuses on the applications of word
embedding models and algorithms. Therefore, the Non-text label does not cover
embedding models for non-text data if they are not word embedding models, such
as graph embedding [10, 11], image embedding [13, 14], or network embedding
[15, 16].

##  4 Cosine Similarity or Euclidean Distance?

As surveyed so far, many papers examine the similarity between embedding
vectors, whether relative or absolute. Most of them use a cosine similarity to
measure the similarity between vectors, but Euclidean distance ($L_{2}$ norm)
is also popular. Because the similarity between vectors can be interpreted as
the distance between vectors, we have various options to define the distance
between vectors.777We also should note that cosine similarity is not distance
metric. A natural question we have to ask is whether the results can change
depending on the measure we use for analysis. Some papers analyze the
robustness of the results using multiple options, such as Euclidean distance
and cosine similarity [59].888Regarding the robustness check, [47] also
studied the correlations between their stereotype measurements of 100 and 300
dimension embedding vectors. In addition, they tested three sets of window
sizes in their robustness check.

[24], in their supplementary information, argues that Euclidean distance is
richer than cosine similarity that only measures the angle. Indeed, cosine
similarity is not a perfect alternative to Euclidean distance. Cosine
similarity can disregard the information of distance between two given
vectors. Figure 1 depicts five different vectors. Vectors 1 and 2 have the
same angle $\theta_{1}$, and therefore, the cosine similarity of these is one.
In contrast, the Euclidean distance between the two is $d$, which means they
are different in terms of Euclidean distance.

It is important to recognize that equal Euclidean distances do not always
correspond to equivalent cosine similarity values, and that Euclidean distance
cannot always be interchanged with cosine similarity, nor vice versa. The
Euclidean distance between Vectors 2 and 3 is the same as the Euclidean
distance between Vectors 1 and 2, but these two pairs have different angles
($\theta_{1}$ and $\theta_{2}$). Figure 1 also depicts that using the
normalized Euclidean distance eliminates the distance between Vectors 1 and 2.
The normalized Vector 1 will be on the same point as Vector 2, which is on the
unit circle. In this case, these two vectors are the same from the perspective
of both cosine similarity and Euclidean distance. In addition, we note that
small distances do not always mean small or large cosine distances. The
distance between Vectors 4 and 5 is small, but their angle is
approximately$\pi/4$; therefore, the cosine similarity between them is not
small.

Specifically, the relationship two measurement become dismissed when $\theta$
is small or around $\pi$. Let us consider the distance between two point in a
unit circle: A $(0,\ 1)$ and B $(\cos\theta,\ \sin\theta)$.We can consider two
point in a unit circle by $(x,\ y)$ and $(x\cos\theta-y\sin\theta,\
x\sin\theta+y\cos\theta)$, where the angle between the two is $\theta$. Since
we only study the distance between the two, we set $x=1$ and $y=0$ without
loss of generality. The distance between two is $(2(1-\cos\theta))^{1/2}$, and
the cosine similarity is $\cos\theta$. Therefore, the error between the
Euclidean distance of normalized vectors and cosine similarity is a function
of $\theta$, and that is

$e(\theta)=\cos\theta-(2(1-\cos\theta))^{1/2}.$

Since the first derivative of $e(\theta)$ with respect to $\theta$ is

$e^{\prime}(\theta)=-\sin(\theta)\left(1+(2(1-\cos(\theta))^{1/2})\right),$

the relationship between the two metrics become zero around ($\theta=\pi$ or
$0$). This fact implies that the two metrics are not compatible when their
angle is small or large. We also note that the fact that the angle is not
always relevant to the distance.

To further discuss the comparison between the cosine similarity and Euclidean
distance, we study the empirical relationship between them using the Google
News pre-trained word2vec model that contains the embedding vectors of
3,000,000 words [60]. We construct 1,500,000 random word pairs and plot the
relationships in Figures 2 and 2. While Figure 2 demonstrates that the two
metrics are correlated, their correlation is not strong (the Pearson
correlation coefficient $\rho=-0.357$), which implies that using cosine
similarity would lose some information and vice versa. Figure 2 also
demonstrates that the cosine similarity distributes uniformly where the
Euclidean distance is small, meaning that the two different pairs of vectors
with the same Euclidean distance can assume different values in their cosine
similarities. This deviation becomes large where the Euclidean distance is
small. This finding is consistent even when we normalize the metrics. Figure 2
demonstrates the empirical relationship between the Euclidean distance of
normalized vectors and cosine similarity. While they capture the difference of
two word embedding vectors in a similar way (the Pearson correlation
coefficient $\rho=-0.991$), they are not the same. The Euclidean distance of
normalized vectors is more sensitive than cosine similarity in which the
distance between two points are close (i.e., $\theta$ is small) or large(i.e.,
$\theta$ is around $\pi$).

The above discussion suggests that the cosine similarity and Euclidean
distance can qualitatively return in a similar manner, but they are not always
compatible. Notably, the cosine similarity can capture the difference between
two vectors when their Euclidean distance is small. However, when we study the
normalized vectors, the Euclidean distance captures the differences, but the
cosine similarity does not when the Euclidean distance is small. Given this,
we should be aware of what kind of characteristics we intended to capture when
selecting a metric.

These findings also imply that the Euclidean distance and cosine similarity
(angle) capture different characteristics. Recently, some studies in computer
science literature propose word embedding models that explicitly model this
relation, such as hierarchical relationships [99, 100, 101, 102]. For example,
[103]proposed the embedding models that use the polar coordinate system and
illustrate that their model captures the hierarchy of words in which the
radius (distance) represents generality, and the angles represent similarity .

###  4.1 Problem with Cosine Similarity for Similarity Measurement with Word
Embeddings

Cosine similarity is the most popular similarity measurement among the
surveyed papers. However, resent research points out several critical problems
of cosine similarity for word embedding models. [104] demonstrate that the
cosine similarity measurement underestimates the similarity between word
embedding vectors compared to human judgments. They attribute this
underestimation to differences in word frequency, suggesting that low-
frequency and high-frequency words have different geometric representations in
the embedding space. [105] also detect issues with cosine similarity
measurement, showing that it can return arbitrary similarity values in certain
settings using regularized linear models. [106] identify a problem where
cosine similarity underestimates the actual similarity between high-frequency
words as [104]. They propose the method to mititgete this problem that
discounts the $\ell_{2}$ norm of a word embedding by its frequency to correct
this underestimation and they show that the proposed method effectively
resolves the issue

Figure 1: Schematic comparison: Cosine-similarity vs Euclidean distance

Note: Schematic illustration of three different vectors. Vector 1 and Vector 2
have the same angle $\theta_{1}$, but Vector 1 is longer than Vector 2. The
angle between Vector 2 and Vector 3 is $\theta_{2}$ but the distance between
Vector 2 and Vector 3 is the same as the distance between Vector 1 and Vector
2. The distance between Vector 4 and Vector 5 is close, but it has a certain
angle.

(a) Euclidean distance

(b) Euclidean distance of normalized vector

Figure 2: Empirical relationship: Cosine-similarity vs Euclidean distance

Note: We built 1,500,000 pairs of two words by randomly picking words from
Google News pre-trained word2vec model [60]. Then, we plot the cosine
similarity and Euclidean distance between the pairs and relationship between
the cosine similarity and normalized Euclidean distance. We normalize word
embedding vectors such that the norm of each embedding vector is 1.

##  5 Conclusion

In this survey, we shed light on the emerging trend of interdisciplinary
studies that apply word embedding models mainly focusing on social science
research. To define and clarify this trend, we built the taxonomy of the
methods used in those application studies. Our taxonomy detected that the
surveyed social science studies use word embedding models to construct
variables of interest. We also demonstrated that the application of word
embedding models has been enlarging not limited to quantitative research but
qualitative ones.

Table 2: Summary of the Analytical Methods With Word Embedding for the Social Science Research I |  | Research
---
Area
Pre-trained |  | Define
---
variable
W/theory | W/reference |  | Non
---
text
| Same
---
words
Human subj | Clustering | Prediction
[62] | Neuro. Sci | ✓ |  | ✓ |  |  |  |  |  |
[107] | Cogni. Sci |  |  | ✓ |  |  |  |  |  |
[35] | Cogni. Sci |  | ✓ |  | ✓ |  |  | ✓ |  |
[64] | Commnication |  | ✓ |  | ✓ |  |  |  |  |
[108] | Commnication |  | ✓ |  | ✓ |  |  |  |  |
[63] | Commnication |  |  |  | ✓ |  |  |  |  |
[77] | Finance |  |  |  |  |  |  |  |  | ✓
[109] | Finance |  |  |  |  |  |  |  |  |
[23] | Humanity |  |  |  |  |  |  |  |  | ✓
[81] | Linguistic |  | ✓ | ✓ |  |  |  |  |  |
[24] | Manag. Sci |  | ✓ |  |  |  | ✓ |  |  |
[25] | Manag. Sci |  |  |  |  |  | ✓ |  |  |
[26] | Manag. Sci |  |  |  | ✓ |  |  |  |  |
[27] | Manag. Sci | ✓ |  |  |  |  |  | ✓ |  | ✓
[28] | Manag. Sci |  |  |  |  |  |  |  |  | ✓
[29] | Manag. Sci | ✓ |  |  |  |  |  |  | ✓ |
[30] | Manag. Sci |  |  |  |  |  |  |  | ✓ |
[31] | Mat. Sci |  | ✓ |  |  | ✓ |  |  |  | ✓
[32] | Cultural Studies |  | ✓ |  | ✓ |  |  |  |  |
[74] | Cultural Studies |  |  |  |  |  |  |  | ✓ |
[89] | Poli. Sci |  | ✓ |  |  | ✓ |  |  |  |
[110] | Poli. Sci |  | ✓ |  | ✓ |  |  |  |  |
[111] | Poli. Sci |  |  |  | ✓ |  |  |  |  |
[112] | Poli. Sci |  | ✓ |  | ✓ |  |  |  |  |
[113] | Poli. Sci |  |  |  |  |  |  |  |  |
[47] | Poli. Sci |  | ✓ |  |  |  |  |  |  |
[83] | Poli. Sci |  | ✓ |  |  |  |  |  |  |

Table 3: Summary of the Analytical Methods With Word Embedding for the Social Science Research I |  | Research
---
Area
Pre-trained |  | Define
---
variable
W/theory | W/reference |  | Non
---
text
| Same
---
words
Human subj | Clustering | Prediction
[114] | Psychology |  | ✓ |  | ✓ |  |  |  |  |
[115] | Psychology |  |  | ✓ |  |  |  |  |  |
[80] | Psychology |  | ✓ |  |  |  |  |  |  |
[76] | Psychology | ✓ |  |  |  |  |  | ✓ |  | ✓
[79] | Sci. metrics |  |  |  |  |  | ✓ |  |  |
[116] | Sci. metrics |  | ✓ |  |  |  |  |  |  |
[59] | General |  | ✓ |  | ✓ |  |  |  |  |
[117] | Sociology |  | ✓ |  | ✓ |  |  |  |  |
[33] | Sociology | ✓ | ✓ |  | ✓ |  |  |  |  |
[73] | Urban. Eng |  |  |  |  | ✓ |  |  | ✓ |
[91] | Urban. Eng |  |  | ✓ |  | ✓ |  |  |  |
[75] | Urban. Eng |  |  |  |  |  |  |  |  |
[90] | Urban. Eng |  |  | ✓ | ✓ | ✓ |  |  |  |
[118] | General | ✓ | ✓ |  | ✓ |  | ✓ |  |  |
[66] | Sociology | ✓ | ✓ |  | ✓ |  |  |  |  |
[65] | General | ✓ | ✓ |  | ✓ |  |  |  |  |
[82] | Psychology | ✓ | ✓ |  |  |  |  |  |  |
[67] | Soc. Psych. | ✓ | ✓ |  | ✓ |  |  |  |  |
[119] | General |  | ✓ |  |  |  |  |  |  |
[120] | General |  | ✓ |  |  | ✓ |  |  |  |

## References

  * [1] Gentzkow, M., Kelly, B. & Taddy, M.  Text as data.  _Journal of Economic Literature_ 57, 535–74 (2019).
  * [2] Benoit, K.  Text as data: An overview.  _The SAGE Handbook of Research Methods in Political Science and International Relations, SAGE Publishing, London_ (2020).
  * [3] Grimmer, J. & Stewart, B. M.  Text as data: The promise and pitfalls of automatic content analysis methods for political texts.  _Political Analysis_ 21, 267–297 (2013).
  * [4] Cong, L., Liang, T. & Zhang, X.  Textual Factors: A Scalable, Interpretable, and Data-driven Approach to Analyzing Unstructured Information.  _SSRN Electronic Journal_ (2018).  URL https://www.ssrn.com/abstract=3307057.
  * [5] Rahimikia, E., Zohren, S. & Poon, S.-H.  Realised Volatility Forecasting: Machine Learning via Financial Word Embedding.  _SSRN Electronic Journal_ (2021).  URL https://www.ssrn.com/abstract=3895272.
  * [6] Wevers, M. & Koolen, M.  Digital begriffsgeschichte: Tracing semantic change using word embeddings.  _Historical Methods: A Journal of Quantitative and Interdisciplinary History_ 53, 226–243 (2020).  URL https://www.tandfonline.com/doi/full/10.1080/01615440.2020.1760157.
  * [7] Tausczik, Y. R. & Pennebaker, J. W.  The psychological meaning of words: Liwc and computerized text analysis methods.  _Journal of language and social psychology_ 29, 24–54 (2010).
  * [8] Hutto, C. & Gilbert, E.  Vader: A parsimonious rule-based model for sentiment analysis of social media text.  In _Proceedings of the international AAAI conference on web and social media_ , vol. 8, 216–225 (2014).
  * [9] Boykis, V.  What are embeddings? (2024).  URL https://vickiboykis.com/what_are_embeddings/.  Accessed: 2024-06-12.
  * [10] Goyal, P. & Ferrara, E.  Graph embedding techniques, applications, and performance: A survey.  _Knowledge-Based Systems_ 151, 78–94 (2018).  URL https://www.sciencedirect.com/science/article/pii/S0950705118301540.
  * [11] Barros, C. D., Mendonça, M. R., Vieira, A. B. & Ziviani, A.  A survey on embedding dynamic graphs.  _ACM Computing Surveys_ 55, 1–37 (2021).
  * [12] Cai, H., Zheng, V. W. & Chang, K. C.-C.  A comprehensive survey of graph embedding: Problems, techniques, and applications.  _IEEE Transactions on Knowledge and Data Engineering_ 30, 1616–1637 (2018).
  * [13] Liu, L. _et al._ Deep learning for generic object detection: A survey.  _International Journal of Computer Vision_ 128, 261–318 (2020).
  * [14] Baltrušaitis, T., Ahuja, C. & Morency, L.-P.  Multimodal machine learning: A survey and taxonomy.  _IEEE transactions on pattern analysis and machine intelligence_ 41, 423–443 (2018).
  * [15] Cui, P., Wang, X., Pei, J. & Zhu, W.  A survey on network embedding.  _IEEE transactions on knowledge and data engineering_ 31, 833–852 (2018).
  * [16] Zhou, J., Liu, L., Wei, W. & Fan, J.  Network representation learning: From preprocessing, feature extraction to node embedding.  _ACM Computing Surveys_ 55 (2022).
  * [17] Chaubard, F., Fang, M., Genthial, G., Mundra, R. & Socher, R.  CS224n: Natural language processing with deep learning, lecture notes: Part I (2019).  https://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes01-wordvecs1.pdf, last accessed March 1, 2022.
  * [18] Mikolov, T., Chen, K., Corrado, G. & Dean, J.  Efficient estimation of word representations in vector space.  In _International Conference on Learning Representations_ (2013).
  * [19] Mikolov, T., Sutskever, I., Chen, K., Corrado, G. & Dean, J.  Distributed representations of words and phrases and their compositionality.  In _Advances in Neural Information Processing Systems_ (2013).
  * [20] Goldberg, Y. & Levy, O.  word2vec explained: deriving mikolov et al.’s negative-sampling word-embedding method.  _arXiv preprint arXiv:1402.3722_ (2014).
  * [21] Mikolov, T., Yih, W. t. & Zweig, G.  Linguistic regularities in continuous space word representations.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 746–751 (Association for Computational Linguistics, 2013).
  * [22] Mikolov, T., Chen, K., Corrado, G. S. & Dean, J.  Computing numeric representations of words in a high-dimensional space (US Patent 9, 037, 464, 2015).
  * [23] Lee, R. & Kim, J.  Developing a Social Index for Measuring the Public Opinion Regarding the Attainment of Sustainable Development Goals.  _Social Indicators Research_ 156, 201–221 (2021).  URL https://link.springer.com/10.1007/s11205-021-02643-5.
  * [24] Toubia, O., Berger, J. & Eliashberg, J.  How quantifying the shape of stories predicts their success.  _Proceedings of the National Academy of Sciences_ 118, e2011695118 (2021).  URL http://www.pnas.org/lookup/doi/10.1073/pnas.2011695118.
  * [25] Chanen, A.  Deep learning for extracting word-level meaning from safety report narratives.  In _2016 Integrated Communications Navigation and Surveillance_ , 5D2–1 (IEEE, 2016).
  * [26] Li, K., Mai, F., Shen, R. & Yan, X.  Measuring corporate culture using machine learning.  _The Review of Financial Studies_ 34, 3265–3315 (2021).
  * [27] Bhatia, S., Olivola, C. Y., Bhatia, N. & Ameen, A.  Predicting leadership perception with large-scale natural language data.  _The Leadership Quarterly_ 101535 (2021).  URL https://linkinghub.elsevier.com/retrieve/pii/S1048984321000400.
  * [28] Chen, H. & Zhang, R.  Identifying Nonprofits by Scaling Mission and Activity with Word Embedding.  _VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations_ (2021).  URL https://link.springer.com/10.1007/s11266-021-00399-7.
  * [29] Zheng, C.  Comparisons of the City Brand Influence of Global Cities: Word-Embedding Based Semantic Mining and Clustering Analysis on the Big Data of GDELT Global News Knowledge Graph.  _Sustainability_ 12, 6294 (2020).  URL https://www.mdpi.com/2071-1050/12/16/6294.
  * [30] Khaouja, I., Mezzour, G., Carley, K. M. & Kassou, I.  Building a soft skill taxonomy from job openings.  _Social Network Analysis and Mining_ 9, 43 (2019).  URL http://link.springer.com/10.1007/s13278-019-0583-9.
  * [31] Tshitoyan, V. _et al._ Unsupervised word embeddings capture latent knowledge from materials science literature.  _Nature_ 571, 95–98 (2019).
  * [32] Nelson, L. K.  Leveraging the alignment between machine learning and intersectionality: Using word embeddings to measure intersectional experiences of the nineteenth century U.S. South.  _Poetics_ 88, 101539 (2021).  URL https://linkinghub.elsevier.com/retrieve/pii/S0304422X21000115.
  * [33] Kozlowski, A. C., Taddy, M. & Evans, J. A.  The geometry of culture: Analyzing the meanings of class through word embeddings.  _American Sociological Review_ 84, 905–949 (2019).
  * [34] Harris, Z. S.  Distributional structure.  _WORD_ 10, 146–162 (1954).
  * [35] Caliskan, A., Bryson, J. J. & Narayanan, A.  Semantics derived automatically from language corpora contain human-like biases.  _Science_ 356, 183–186 (2017).
  * [36] Miller, G. A. & Charles, W. G.  Contextual correlates of semantic similarity.  _Language and cognitive processes_ 6, 1–28 (1991).
  * [37] Charles, W. G.  Contextual correlates of meaning.  _Applied Psycholinguistics_ 21, 505–524 (2000).
  * [38] Agirre, E. _et al._ A study on similarity and relatedness using distributional and WordNet-based approaches.  In _Proceedings of Human Language Technologies: Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , 19–27 (2009).
  * [39] Rubenstein, H. & Goodenough, J. B.  Contextual correlates of synonymy.  _Communications of the ACM_ 8, 627–633 (1965).
  * [40] Bruni, E., Tran, N.-K. & Baroni, M.  Multimodal distributional semantics.  _Journal of artificial intelligence research_ 49, 1–47 (2014).
  * [41] Radinsky, K., Agichtein, E., Gabrilovich, E. & Markovitch, S.  A word at a time: Computing word relatedness using temporal semantic analysis.  In _Proceedings of the International Conference on World Wide Web_ , 337–346 (2011).
  * [42] Huang, E., Socher, R., Manning, C. & Ng, A.  Improving word representations via global context and multiple word prototypes.  In _Proceedings of Annual Meeting of the Association for Computational Linguistics_ , 873–882 (2012).
  * [43] Luong, M.-T., Socher, R. & Manning, C. D.  Better word representations with recursive neural networks for morphology.  In _Proceedings of the seventeenth conference on computational natural language learning_ , 104–113 (2013).
  * [44] Hill, F., Reichart, R. & Korhonen, A.  Simlex-999: Evaluating semantic models with (genuine) similarity estimation.  _Computational Linguistics_ 41, 665–695 (2015).
  * [45] Wang, B., Wang, A., Chen, F., Wang, Y. & Kuo, C.-C. J.  Evaluating word embedding models: Methods and experimental results.  _APSIPA Transactions on Signal and Information Processing_ 8 (2019).
  * [46] Rong, X.  word2vec parameter learning explained.  _arXiv preprint arXiv:1411.2738_ (2014).
  * [47] Ash, E., Chen, D. L., Ornaghi, A. _et al._ _Stereotypes in high-stakes decisions: Evidence from us circuit courts_ (University of Warwick, Department of Economics, 2020).
  * [48] Pennington, J., Socher, R. & Manning, C. D.  Glove: Global vectors for word representation.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , 1532–1543 (2014).
  * [49] Kawaguchi, K., Kuroda, T. & Sato, S.  Merger analysis in the app economy: An empirical model of ad-sponsored media.  In _Conference paper. TPRC48: The 48th Research Conference on Communication, Information and Internet Policy, DOI: https://doi. org/10.2139/ssrn_ , vol. 3746830 (2021).
  * [50] Bojanowski, P., Grave, E., Joulin, A. & Mikolov, T.  Enriching word vectors with subword information.  _Transactions of the Association for Computational Linguistics_ 5, 135–146 (2017).
  * [51] Rehurek, R. & Sojka, P.  Software framework for topic modelling with large corpora.  In _In Proceedings of the LREC 2010 workshop on new challenges for NLP frameworks_ (Citeseer, 2010).
  * [52] Rehurek, R. & P.Sojka.  Gensim.  https://github.com/RaRe-Technologies/gensim (2010).
  * [53] Dyer, C.  Notes on noise contrastive estimation and negative sampling.  _arXiv preprint arXiv:1410.8251_ (2014).
  * [54] Gutmann, M. & Hyvärinen, A.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In _Proceedings of the Conference on Artificial Intelligence and Statistics_ (2010).
  * [55] Levy, O. & Goldberg, Y.  Dependency-based word embeddings.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics_ (2014).
  * [56] Matsui, A., Ren, X. & Ferrara, E.  Using word embedding to reveal monetary policy explanation changes.  In _Proceedings of the Third Workshop on Economics and Natural Language Processing_ , 56–61 (2021).
  * [57] Baumgärtner, M. & Zahner, J.  Whatever it takes to understand a central banker: Embedding their words using neural networks.  Tech. Rep., MAGKS Joint Discussion Paper Series in Economics (2021).
  * [58] Bhatia, S.  Predicting Risk Perception: New Insights from Data Science.  _Management Science_ 65, 3800–3823 (2019).  URL http://pubsonline.informs.org/doi/10.1287/mnsc.2018.3121.
  * [59] Garg, N., Schiebinger, L., Jurafsky, D. & Zou, J.  Word embeddings quantify 100 years of gender and ethnic stereotypes.  _Proceedings of the National Academy of Sciences_ 115, E3635–E3644 (2018).
  * [60] Google.  Google news word embedding model (2013).  URL https://code.google.com/archive/p/word2vec/.
  * [61] Hamilton, W., Leskovec, J. & Jurafsky, D.  Histwords:word embeddings for historical text (2016).  URL https://nlp.stanford.edu/projects/histwords/.
  * [62] Chersoni, E., Santus, E., Huang, C.-R. & Lenci, A.  Decoding Word Embeddings with Brain-Based Semantic Features.  _Computational Linguistics_ 47, 663–698 (2021).  URL https://direct.mit.edu/coli/article/47/3/663/102823/Decoding-Word-Embeddings-with-Brain-Based-Semantic.
  * [63] Bhatia, N. & Bhatia, S.  Changes in Gender Stereotypes Over Time: A Computational Analysis.  _Psychology of Women Quarterly_ 45, 106–125 (2021).  URL http://journals.sagepub.com/doi/10.1177/0361684320977178.
  * [64] Kroon, A. C., Trilling, D. & Raats, T.  Guilty by Association: Using Word Embeddings to Measure Ethnic Stereotypes in News Coverage.  _Journalism & Mass Communication Quarterly_ 98, 451–477 (2021).  URL http://journals.sagepub.com/doi/10.1177/1077699020932304.
  * [65] Charlesworth, T. E. S., Caliskan, A. & Banaji, M. R.  Historical representations of social groups across 200 years of word embeddings from google books.  _Proceedings of the National Academy of Sciences_ 119, e2121798119 (2022).  URL https://doi.org/10.1073/pnas.2121798119.
  * [66] Boutyline, A., Arseniev-Koehler, A. & Cornell, D. J.  School, Studying, and Smarts: Gender Stereotypes and Education Across 80 Years of American Print Media, 1930–2009†.  _Social Forces_ 102, 263–286 (2023).  URL https://doi.org/10.1093/sf/soac148.  https://academic.oup.com/sf/article-pdf/102/1/263/50880975/soac148.pdf.
  * [67] Durrheim, K., Schuld, M., Mafunda, M. & Mazibuko, S.  Using word embeddings to investigate cultural biases.  _British Journal of Social Psychology_ 62, 617–629 (2023).  URL https://doi.org/10.1111/bjso.12560.
  * [68] Bolukbasi, T., Chang, K.-W., Zou, J. Y., Saligrama, V. & Kalai, A. T.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  _Advances in Neural Information Processing Systems_ 29, 4349–4357 (2016).
  * [69] Zhao, J., Zhou, Y., Li, Z., Wang, W. & Chang, K.-W.  Learning gender-neutral word embeddings.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 4847–4853 (Association for Computational Linguistics, 2018).
  * [70] Brunet, M.-E., Alkalay-Houlihan, C., Anderson, A. & Zemel, R.  Understanding the origins of bias in word embeddings.  In _International Conference on Machine Learning_ , 803–811 (PMLR, 2019).
  * [71] Font, J. E. & Costa-Jussa, M. R.  Equalizing gender biases in neural machine translation with word embeddings techniques.  _arXiv preprint arXiv:1901.03116_ (2019).
  * [72] Gonen, H. & Goldberg, Y.  Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them.  _arXiv preprint arXiv:1903.03862_ (2019).
  * [73] Hu, S. _et al._ A framework for extracting urban functional regions based on multiprototype word embeddings using points-of-interest data.  _Computers, Environment and Urban Systems_ 80, 101442 (2020).  URL https://linkinghub.elsevier.com/retrieve/pii/S0198971519302625.
  * [74] Nanni, A. & Fallin, M.  Earth, wind, (water), and fire: Measuring epistemic boundaries in climate change research.  _Poetics_ 88, 101573 (2021).  URL https://linkinghub.elsevier.com/retrieve/pii/S0304422X21000577.
  * [75] Kim, B., Yoo, M., Park, K. C., Lee, K. R. & Kim, J. H.  A value of civic voices for smart city: A big data analysis of civic queries posed by Seoul citizens.  _Cities_ 108, 102941 (2021).  URL https://linkinghub.elsevier.com/retrieve/pii/S0264275120312890.
  * [76] Richie, R., Zou, W., Bhatia, S. & Vazire, S.  Predicting high-level human judgment across diverse behavioral domains.  _Collabra: Psychology_ 5 (2019).
  * [77] Wang, W.  Two-Stage Learning of a Systemic Risk Factor.  _SSRN Electronic Journal_ (2019).  URL https://www.ssrn.com/abstract=3017446.
  * [78] Taylor, S. D.  Concepts as a working hypothesis.  _Philosophical Psychology_ 1–26 (2021).
  * [79] Soni, S., Lerman, K. & Eisenstein, J.  Follow the leader: Documents on the leading edge of semantic change get more citations.  _Journal of the Association for Information Science and Technology_ 72, 478–492 (2021).  URL https://onlinelibrary.wiley.com/doi/10.1002/asi.24421.
  * [80] Boyce-Jacino, C. & DeDeo, S.  Opacity, obscurity, and the geometry of question-asking.  _Cognition_ 196, 104071 (2020).  URL https://linkinghub.elsevier.com/retrieve/pii/S0010027719302446.
  * [81] Čech, R., Hůla, J., Kubát, M., Chen, X. & Milička, J.  The Development of Context Specificity of Lemma. A Word Embeddings Approach.  _Journal of Quantitative Linguistics_ 26, 187–204 (2019).  URL https://www.tandfonline.com/doi/full/10.1080/09296174.2018.1491748.
  * [82] Johnson, D. R. _et al._ Divergent semantic integration (dsi): Extracting creativity from narratives with distributional semantic modeling.  _Behavior Research Methods_ 55, 3726–3759 (2023).  URL https://doi.org/10.3758/s13428-022-01986-2.
  * [83] Nyarko, J. & Sanga, S.  A Statistical Test for Legal Interpretation: Theory and Applications.  _SSRN Electronic Journal_ (2020).  URL https://www.ssrn.com/abstract=3737292.
  * [84] Hamilton, W., Leskovec, J. & Jurafsky, D.  Diachronic word embeddings reveal statistical laws of semantic change.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ (2016).
  * [85] Gillani, N. & Levy, R.  Simple dynamic word embeddings for mapping perceptions in the public sphere.  In _Proceedings of the Third Workshop on Natural Language Processing and Computational Social Science_ , 94–99 (2019).
  * [86] Yao, Z., Sun, Y., Ding, W., Rao, N. & Xiong, H.  Dynamic word embeddings for evolving semantic discovery.  In _Proceedings of the Proceedings of the International Conference on Web Search and Data Mining_ , 673–681 (2018).
  * [87] Arora, S., Li, Y., Liang, Y., Ma, T. & Risteski, A.  A latent variable model approach to PMI-based word embeddings.  _Transactions of the Association for Computational Linguistics_ 4, 385–399 (2016).  URL https://aclanthology.org/Q16-1028.
  * [88] Zhou, X., Huang, S. & Zheng, Z.  RPD: A distance function between word embeddings.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_ , 42–50 (Association for Computational Linguistics, Online, 2020).  URL https://aclanthology.org/2020.acl-srw.7.
  * [89] Rheault, L. & Cochrane, C.  Word Embeddings for the Analysis of Ideological Placement in Parliamentary Corpora.  _Political Analysis_ 28, 112–133 (2020).  URL https://www.cambridge.org/core/product/identifier/S1047198719000263/type/journal_article.
  * [90] Niu, H. & Silva, E. A.  Delineating urban functional use from points of interest data with neural network embedding: A case study in greater london.  _Computers, Environment and Urban Systems_ 88, 101651 (2021).  URL https://doi.org/10.1016/j.compenvurbsys.2021.101651.
  * [91] Murray, D. _et al._ Unsupervised embedding of trajectories captures the latent structure of mobility.  _arXiv preprint arXiv:2012.02785_ (2020).
  * [92] Zipf, G. K.  The P1 P2D hypothesis: On the intercity movement of persons.  _American Sociological Review_ 11, 677–686 (1946).  URL http://www.jstor.org/stable/2087063.
  * [93] Waller, I. & Anderson, A.  Quantifying social organization and political polarization in online platforms.  _Nature_ 600, 264–268 (2021).
  * [94] Goldberg, Y.  word2vecf.  https://github.com/BIU-NLP/word2vecf (2017).
  * [95] Pennebaker, J. W., Francis, M. E. & Booth, R. J.  Linguistic inquiry and word count: Liwc 2001.  _Mahway: Lawrence Erlbaum Associates_ 71, 2001 (2001).
  * [96] Bird, S., Klein, E. & Loper, E.  _Natural language processing with Python: analyzing text with the natural language toolkit_ (” O’Reilly Media, Inc.”, 2009).
  * [97] Graham, J. _et al._ Mapping the moral domain.  _Journal of Personality and Social Psychology_ 101, 366 (2011).
  * [98] Haidt, J. & Graham, J.  When morality opposes justice: Conservatives have moral intuitions that liberals may not recognize.  _Social Justice Research_ 20, 98–116 (2007).
  * [99] Tifrea, A., Bécigneul, G. & Ganea, O.-E.  Poincar$\backslash$’e glove: Hyperbolic word embeddings.  _Proceeding of International Conference on Learning Representations_ (2018).
  * [100] Nickel, M. & Kiela, D.  Poincaré embeddings for learning hierarchical representations.  _Advances in Neural Information Processing Systems_ 30 (2017).
  * [101] Vendrov, I., Kiros, R., Fidler, S. & Urtasun, R.  Order-embeddings of images and language.  _Proceeding of International Conference on Learning Representations_ (2016).
  * [102] Vilnis, L. & McCallum, A.  Word representations via gaussian embedding.  _Proceeding of International Conference on Learning Representations_ (2015).
  * [103] Iwamoto, R., Kohita, R. & Wachi, A.  Polar embedding.  In _Proceedings of the 25th Conference on Computational Natural Language Learning_ , 470–480 (Association for Computational Linguistics, Online, 2021).  URL https://aclanthology.org/2021.conll-1.37.
  * [104] Zhou, K., Ethayarajh, K., Card, D. & Jurafsky, D.  Problems with cosine as a measure of embedding similarity for high frequency words.  In Muresan, S., Nakov, P. & Villavicencio, A. (eds.) _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , 401–423 (Association for Computational Linguistics, Dublin, Ireland, 2022).  URL https://aclanthology.org/2022.acl-short.45.
  * [105] Steck, H., Ekanadham, C. & Kallus, N.  Is cosine-similarity of embeddings really about similarity?  In _Companion Proceedings of the ACM on Web Conference 2024_ , WWW ’24, 887–890 (Association for Computing Machinery, New York, NY, USA, 2024).  URL https://doi.org/10.1145/3589335.3651526.
  * [106] Wannasuphoprasit, S., Zhou, Y. & Bollegala, D.  Solving cosine similarity underestimation between high frequency words by $\ell_{2}$ norm discounting.  In Rogers, A., Boyd-Graber, J. & Okazaki, N. (eds.) _Findings of the Association for Computational Linguistics: ACL 2023_ , 8644–8652 (Association for Computational Linguistics, Toronto, Canada, 2023).  URL https://aclanthology.org/2023.findings-acl.550.
  * [107] Kelly, M., Ghafurian, M., West, R. L. & Reitter, D.  Indirect associations in learning semantic and syntactic lexical relationships.  _Journal of Memory and Language_ 115, 104153 (2020).  URL https://linkinghub.elsevier.com/retrieve/pii/S0749596X2030067X.
  * [108] Kroon, A. C., Trilling, D., Meer, T. G. v. d. & Jonkman, J. G.  Clouded reality: News representations of culturally close and distant ethnic outgroups.  _Communications_ 45, 744–764 (2020).  URL https://www.degruyter.com/document/doi/10.1515/commun-2019-2069/html.
  * [109] Jha, M., Liu, H. & Manela, A.  Natural disaster effects on popular sentiment toward finance.  _Journal of Financial and Quantitative Analysis_ 1–35 (2021).  URL https://www.cambridge.org/core/product/identifier/S0022109021000466/type/journal_article.
  * [110] Gennaro, G. & Ash, E.  Emotion and reason in political language.  _The Economic Journal_ (2022).
  * [111] Rodman, E.  A Timely Intervention: Tracking the Changing Meanings of Political Concepts with Word Vectors.  _Political Analysis_ 28, 87–111 (2020).  URL https://www.cambridge.org/core/product/identifier/S1047198719000238/type/journal_article.
  * [112] Rozado, D. & al Gharbi, M.  Using word embeddings to probe sentiment associations of politically loaded terms in news and opinion articles from news media outlets.  _Journal of Computational Social Science_ (2021).  URL https://link.springer.com/10.1007/s42001-021-00130-y.
  * [113] Leavy, S., Keane, M. T. & Pine, E.  Patterns in language: Text analysis of government reports on the Irish industrial school system with word embedding.  _Digital Scholarship in the Humanities_ 34, i110–i122 (2019).  URL https://academic.oup.com/dsh/article/34/Supplement_1/i110/5443277.
  * [114] Kim, Y. & Shin, H.  Finding Sentiment Dimension in Vector Space of Movie Reviews: An Unsupervised Approach.  _Journal of Cognitive Science_ 18, 85–101 (2017).  URL https://doi.org/10.17791/JCS.2017.18.1.85.
  * [115] Peterson, J. C., Chen, D. & Griffiths, T. L.  Parallelograms revisited: Exploring the limitations of vector space models for simple analogies.  _Cognition_ 205, 104440 (2020).  URL https://linkinghub.elsevier.com/retrieve/pii/S0010027720302596.
  * [116] Hu, K. _et al._ Identifying the “Ghost City” of domain topics in a keyword semantic space combining citations.  _Scientometrics_ 114, 1141–1157 (2018).  URL http://link.springer.com/10.1007/s11192-017-2604-7.
  * [117] Jones, J., Amin, M., Kim, J. & Skiena, S.  Stereotypical Gender Associations in Language Have Decreased Over Time.  _Sociological Science_ 7, 1–35 (2020).  URL https://www.sociologicalscience.com/articles-v7-1-1/.
  * [118] Grand, G., Blank, I. A., Pereira, F. & Fedorenko, E.  Semantic projection recovers rich human knowledge of multiple object features from word embeddings.  _Nature Human Behaviour_ 6, 975–987 (2022).  URL https://doi.org/10.1038/s41562-022-01316-8.
  * [119] Aceves, P. & Evans, J. A.  Human languages with greater information density have higher communication speed but lower conversation breadth.  _Nature Human Behaviour_ 8, 644–656 (2024).  URL https://doi.org/10.1038/s41562-024-01815-w.
  * [120] Lewis, M., Cahill, A., Madnani, N. & Evans, J.  Local similarity and global variability characterize the semantic space of human languages.  _Proceedings of the National Academy of Sciences_ 120, e2300986120 (2023).  URL https://doi.org/10.1073/pnas.2300986120.


# Deep Q-Learning Based Resource Allocation in Interference Systems With
Outage Constraint

Saniul Alam, Sadia Islam, Muhammad R. A. Khandaker, Risala T. Khan, Faisal
Tariq, , and Apriana Toding

###### Abstract

This correspondence considers the resource allocation problem in wireless
interference channel (IC) under link outage constraints. Since the
optimization problem is non-convex in nature, existing approaches to find the
optimal power allocation are computationally intensive and thus practically
infeasible. Recently, deep reinforcement learning has shown promising outcome
in solving non-convex optimization problems with reduced complexity. In this
correspondence, we utilize a deep $Q$-learning (DQL) approach which interacts
with the wireless environment and learns the optimal power allocation of a
wireless IC while maximizing overall sum-rate of the system and maintaining
reliability requirement of each link. We have used two separate deep
$Q$-networks to remove the inherent instability in learning process.
Simulation results demonstrate that the proposed DQL approach outperforms
existing geometric programming based solution.

###### Index Terms:

Deep reinforcement learning, $Q$-learning, deep $Q$-learning, deep
$Q$-network, wireless interference channel, resource allocation.

##  I Introduction

Resource allocation tasks such as optimal power allocation has a significant
effect on the capacity and performance of wireless communication [1].
Traditionally, resource allocation problems have been approached and tackled
by established numerical optimization approaches such as interference pricing
[2], water filling algorithms [3], fractional programming [4] and weighted
minimum mean-squared error (WMMSE) minimization [5]. Most of these algorithms
are mathematically formulated to optimize a certain system performance metrics
such as sum-rate, minimum rate or mean-squared error. These algorithms
generally adopted iterative approaches where they take certain parameters like
channel realization as input and produce the optimum resource allocation
strategy.

Even though these established resource and power allocation methods have
improved the performance to some extent, they come with their own set of
problems. These non-convex optimization problems are NP-hard and thus incur
high computational cost. As a result, many of the algorithms and solutions
become infeasible for real world deployment. Moreover, as it takes time to
compute the optimum allocation, it also has negative effect on the latency
performance of the system which is a key performance metric for current and
future communication systems [6]. Despite the high computational cost, these
algorithms have to be recomputed frequently since the channel is time varying
and the allocation may not remain usable for long.

Machine learning (ML) techniques has recently been considered in wireless
communication research as an attractive tool for providing viable solution to
major challenges such as channel decoding and estimation [7] as well as
optimal power allocation [8].

Deep reinforcement learning (DRL), a member of ML family, has been considered
as an attractive method which can learn the optimal solution of the resource
allocation problem by interacting with the wireless environment. Reinforcement
learning (RL) generally outperforms traditional approaches in most of the
cases [9]. The work in [10] uses deep $Q$-learning (DQL) for centralized
downlink power allocation scheme which maximizes the network throughput in a
multi cell network. The work in [11] also uses multi agent DQL to find optimal
power allocation in a wireless cellular network, where each transmitter
receives channel state information (CSI) from several neighbours and learns to
adjust its transmit power accordingly. A deep $Q$-network (DQN) was used to
overcome the instability problem of DL. In [12], the authors proposed faded-
experience trust-region policy optimization (FE-TRPO) base power allocation
algorithm which exploits continuous DRL. The work demonstrated that compared
to the traditional WMMSE techniques, it significantly decreases computational
complexity while maintaining similar performance.

While most of these works use DRL for maximizing the overall system
throughput, the schemes do not always guarantee reliability for each
individual user. As a consequence, users with low channel gain may not be able
to transmit at all while user with the best channel gain may be allowed to
transmit at maximum power resulting in unfairness in the system. To address
this limitation, this work exploits DQN with a reliability outage requirement
threshold for each individual user. The DQN finds the relationship between
available channel information and the solution of the power allocation
problem. A DQN agent is then used to find the collective power allocation
policy of the users that maximizes the overall system throughput while
maintaining reliability of each user. The main contribution of this paper can
be summarised as:

  * •

Firstly, the resource allocation problem of a wireless interference network
under probabilistic constraints has been solved using geometric programming
(GP) (following [13]).

  * •

Then, a DRL-based solution has been developed in order to reduce computational
complexity and make the solution practicable. Two separate deep $Q$-networks
have been used in the DRL solution to remove the inherent instability of the
learning process.

  * •

Simulation results demonstrate that the proposed DRL approach significantly
outperforms the GP approach.

##  II System Model

We consider an interference channel (IC) as shown in Fig. 1, where $K$ single-
antenna transmitters are communicating with $K$ single-antenna receivers. The
direct channel between the $k$th transmitter-receiver pair is denoted by
$h_{k,k}$ while $h_{k,j}$ denotes the IC between transmitter $j$ and receiver
$k$. Let us denote $x_{k}$ as the signal transmitted by transmitter $k$ and
$y_{k}$ as the corresponding received signal at receiver $k$. Then $y_{k}$ is
given by

$\displaystyle y_{k}=h_{kk}x_{k}+\sum_{j\neq k}h_{kj}x_{j}+n_{k}.$ (1)

Here $n_{k}\sim\mathcal{N}\left(0,\sigma_{k}^{2}\right)$ for $\sigma_{k}>0$
represents the additive channel noise. We assume that $p_{k}$ is the
transmission power of transmitter $k$. Note that the symbol transmitted by
different transmitters are independent of each other. Then the signal to
interference-plus-noise ratio (SINR) can be expressed as

$\displaystyle\gamma_{k}\triangleq\frac{\left|h_{kk}\right|^{2}p_{k}}{\sum_{j\neq
k}\left|h_{kj}\right|^{2}p_{j}+\sigma_{k}^{2}},$ (2)

where $\sigma^{2}_{k}$ denotes the noise power at receiver $k$. Thus the
achievable rate at the $k$th receiver under Gaussian channel fading is given
by

$\displaystyle
R_{k}=\log_{2}\left(1+\frac{\left|h_{kk}\right|^{2}p_{k}}{\sum_{j\neq
k}\left|h_{kj}\right|^{2}p_{j}+\sigma_{k}^{2}}\right).$ (3)

###  II-A Problem Formulation

Our aim is to maximize the overall system throughput of the communication
links by optimally allocating transmit power among the active links such that
the worst-user’s mutual-information is maximized under link outage
constraints. Accordingly, we formulate the following optimization problem:

$\displaystyle\max_{\\{p_{k}\\}}~{}$ $\displaystyle~{}\min_{k}~{}~{}R_{k}$
(4a) $\displaystyle{\rm s.t.}~{}$ $\displaystyle~{}{\rm
Pr}\\{\gamma_{k}\leq\gamma_{0}\\}\leq p_{0},\forall k,$ (4b)
$\displaystyle~{}P_{\rm min}\leq p_{k}\leq P_{\rm max},\forall k,$ (4c)

where $P_{\rm max}$ and $P_{\rm min}$ are the maximum and minimum transmit
power budgets of the users, respectively, $\gamma_{0}$ is the minimum SINR
requirement that needs to be maintained by each user to ensure reliability,
and the tolerable outage probability is denoted by $p_{0}$. Note that problem
(4) is a non-convex problem with fractional objective and the nonlinear
probabilistic constraints, and hence the exactly optimal solution is non-
trivial. In the following, we will first develop an acceptable solution to the
problem following traditional alternating approaches, and then following ML
techniques.

Figure 1: Deep reinforcement learning for IC.

For ease of exposition, let us consider an interference-limited system such
that the noise term in (2) can be ignored. Thus, the SINR in (2) reduces to

$\displaystyle\tilde{\gamma}_{k}\triangleq\frac{\left|h_{kk}\right|^{2}p_{k}}{\sum_{j\neq
k}\left|h_{kj}\right|^{2}p_{j}}.$ (5)

Then the chance SINR outage requirement in (4b) can be expressed as

$\displaystyle{\rm O}_{k}=$ $\displaystyle{\rm
Pr}\\{\gamma_{k}\leq\gamma_{0}\\},\forall k,$ $\displaystyle\approx$
$\displaystyle{\rm Pr}\\{\left|h_{kk}\right|^{2}p_{k}\leq\gamma_{0}\sum_{j\neq
k}\left|h_{kj}\right|^{2}p_{j}\\},\forall k,$ (6)

Note that the outage probability $O_{k}$ can be interpreted as the fraction of
time the $k$th transmitter/receiver pair experiences an outage due to fading.
Interestingly, it has been shown analytically in [14, 13] that the outage
probability (6) can be expressed in closed form using the following Lemma.

######  Lemma 1.

If $x_{1},x_{2}\dots,x_{N}$ are independent random variables following
exponential distribution with means
$\mu_{i}\triangleq\mathbb{E}[x_{i}]=1/\lambda_{i}$, then we have

$\displaystyle{\rm
Pr}\left(x_{1}\leq\sum_{i=2}^{N}x_{i}\right)=1-\prod_{i=2}^{N}\left({\frac{1}{1+\frac{\lambda_{1}}{\lambda_{i}}}}\right).$
(7)

###### Proof.

The proof follows similar lines as in [13, Appendix I]. Therefore, we omit the
proof here for brevity. ∎

Using Lemma 1 for an interference-limited scenario, we can express the outage-
probability in (6) as

$\displaystyle{\rm O}_{k}=1-\prod_{j\neq
k}\frac{1}{1+\frac{\gamma_{0}|{h}_{kj}|^{2}{p}_{j}}{|{h}_{kk}|^{2}{P}_{k}}}.$
(8)

Thus the outage constraint (4b) can be expressed as

$\displaystyle(1-p_{0})\prod_{j\neq
k}\left(1+\frac{\gamma_{0}|{h}_{kj}|^{2}{p}_{j}}{|{h}_{kk}|^{2}{p}_{k}}\right)\leq
1.$ (9)

It can be shown that the left-hand side of the inequality (9) is, in fact, a
posynomial function of the powers $p_{1},\,p_{2},\,\dots,\,p_{K}$. Thus we can
express the problem as

$\displaystyle\max_{\\{p_{k}\\}}~{}$ $\displaystyle~{}\min_{k}~{}~{}R_{k}$
(10a) $\displaystyle{\rm s.t.}~{}$ $\displaystyle~{}(1-p_{0})\prod_{j\neq
k}\left(1+\frac{\gamma_{0}|{h}_{kj}|^{2}{p}_{j}}{|{h}_{kk}|^{2}{p}_{k}}\right)\leq
1,\forall k,$ (10b) $\displaystyle~{}\frac{P_{\rm min}}{p_{k}}\leq 1,\forall
k,$ (10c) $\displaystyle~{}\frac{p_{k}}{P_{\rm max}}\leq 1,\forall k.$ (10d)

The problem is still non-convex due to the fractional objective function as
well as the constraint (10b), however, a solution can be obtained based on the
Perron–Frobenius theorem for the maximum eigenvalue of the matrix $\\{{\bf
H}~{}|~{}[{\bf H}]_{i,j}=|h_{i,j}|^{2}\\}$ that has non-negative elements
(channel gains in this case) [15]. Note that due to the monotonicity of the
$\log$ function, problem (10) can be equivalently solved by replacing the
objective function by the corresponding SINR values. Now, by introducing a
slack variable $\eta$, we can reformulate problem (10) as

$\displaystyle\max_{\eta,\\{p_{k}\\}}~{}$ $\displaystyle~{}\eta$ (11a)
$\displaystyle{\rm s.t.}~{}$ $\displaystyle~{}\frac{\eta\left({\sum_{j\neq
k}\left|h_{kj}\right|^{2}p_{j}}\right)}{{\left|h_{kk}\right|^{2}p_{k}}}\leq
1,\forall k,$ (11b) $\displaystyle~{}(1-p_{0})\prod_{j\neq
k}\left(1+\frac{\gamma_{0}|{h}_{kj}|^{2}{p}_{j}}{|{h}_{kk}|^{2}{p}_{k}}\right)\leq
1,\forall k,$ (11c) $\displaystyle~{}\frac{P_{\rm min}}{p_{k}}\leq 1,\forall
k,$ (11d) $\displaystyle~{}\frac{p_{k}}{P_{\rm max}}\leq 1,\forall k.$ (11e)

It can be observed that constraints (11b) and (11c) are homogeneous with
respect to powers thereby depend only on the power ratios. Accordingly,
problem (11) is a geometric program (GP) with respect to $\\{p_{k}\\}$,
therefore, it can be solved globally and efficiently using interior-point
methods for geometric programming. We solve the problem using geometric
programming and consider it as a baseline to compare the results of our DRL
method developed in the next section.

##  III Reinforcement Learning for Resource Optimisation

Fig. 1 demonstrates the basic structure of RL in the system model under
consideration. QL is a widely used model-free RL technique, which determines
the quality of action and informs the RL agent which action will be optimum in
each circumstances.

Lets $S$ denotes possible states of the environment and $A$ denotes a set of
discrete actions. Here, $s\in S$ is a specific state containing environmental
features. For each time step $t$, agent receives a state $s^{t}\in S$ from the
environment and takes a specific action $a^{t}$ from the action space $A$. The
agent follows a policy $\pi$ which is the probability of choosing an action
$a^{t}$, given the current state being $s^{t}$. After the execution of the
action, the agent receives a reward $r^{t}$ and state transition happens from
state $s^{t}$ to state $s^{(t+1)}$. Then the process forms an experience
$e^{(t+1)}=\left(s^{(t)},a^{(t)},r^{(t)},s^{(t+1)})\right.$ which summarizes
the interaction with the environment. This process is repeated until the agent
reaches the terminal state after which the process restarts again. The goal of
QL is to find an optimal policy $\pi$ which, at time t, maximizes the future
cumulative reward, denoted as [10]:

$\displaystyle R_{t}=\sum_{k=0}^{\infty}\gamma^{k}r_{t+k}.$ (12)

Here, $\gamma\in(0,1]$ is a factor which denotes the priority of future
rewards in comparison with the current reward. We define an action-value
function $Q^{\pi}(s,a)$, which is the expected return starting from state $s$,
taking an action $a$, while following policy $\pi$ [11]:

$\displaystyle Q^{\pi}(s,a)=\mathbb{E}_{\pi}\left[R^{(t)}\mid
s^{(t)}=s,a^{(t)}=a\right].$ (13)

The $Q$-function satisfies the Bellman equation given below:

$\displaystyle
Q^{*}(s,a)=R(s,a)+\gamma\sum_{s^{\prime}\in\mathcal{S}}P_{ss^{\prime}}^{a}\max_{a^{\prime}\in\mathcal{A}}Q^{*}\left(s^{\prime},a^{\prime}\right).$
(14)

The QL algorithm uses a lookup table ($Q$-table) to store the output of
$Q$-function called $Q$-values. The agent uses an $\epsilon$-greedy policy in
order to choose action. According to the $\epsilon$-greedy policy, at each
state the agent either chooses to explore the environment by choosing action
randomly or exploit $Q$-table and selects the action with the maximum
$Q$-value.

The problem with traditional QL is that it stores the $Q$-values in tabular
form. So, for high dimensional state spaces, QL becomes infeasible quickly
since its space requirement for the $Q$-table becomes impractical. In this
correspondence, we use DQN which is a combination of traditional QL and a deep
neural network (DNN). Instead of using a table to store the $Q$-values, DQN
uses a DNN which receives sates and provides $Q$-values as output for each
possible action from that state. To avoid correlation between input data, a
buffer of experience called replay memory is also created. The DQN is trained
with training data sampled randomly from the replay memory. This technique is
known as experience replay. In DQL, we approximate the optimal action-value
function by using a neural network, $Q(s,a;\theta)\approx Q^{*}(s,a).$ Here,
$Q(s,a;\theta)$ is called the DQN and $\theta$ is a parameter of the neural
network. The MSE of the Bellman equation is reduced by the iterative update,
which is used to train the $Q$-network.

###  III-A Proposed DQL Approach

In our proposed DQL approach, we exploit a single DQN agent which communicates
with the environment in order to learn the optimal power of the users. The
agent, instead of assigning more power to the user with best channel
condition, tries to maintain reliability and assign each user at least minimum
power, so that they can maintain the minimum required SINR threshold
$\gamma_{0}$. Once reliability is ensured, the agent gives more emphasis on
maximizing the global sum-rate instead of trying to maximize the rate of each
individual users. The resource sharing scheme is performed in two phases,
first, in the training or learning stage and then in the implementation or
testing phase. In the learning phase, the DQN agent access the readily
available system performance reward and adjusts its action to achieve an
optimal policy by updating the DQN. In implementation phase, the agent
receives observation from environment and selects the action according to the
$Q$-value received at the output given by the trained DQN.

####  III-A1 State Space

The state space $S$ consists of power allocation information of other users
from the previous step and channel realization of the user ($h_{kj}$), which
follows a certain distribution. We assume at the beginning of each time step
$t$ that the users’ channel information is instantaneously available at the
transmitter.

####  III-A2 Action Space

The resource allocation design using RL comes down to the transmission power
control for each user. So, the total number of actions depend on the number of
power levels we are using. Even though transmission power takes continuous
values in most cases for reducing complexity, in this paper, we limit the
power control options to a fixed number of power levels chosen from the
interval $[P_{max},P_{min}]$. We assume that the action space $A$ has $n$
power levels.

####  III-A3 Reward Design

Due to inherent flexibility in reward design, RL is an appealing tool for
solving problems that are hard to optimize using traditional methods. In the
original problem formulation in section II, we set the design objective to
maximize the overall sum-rate ensuring reliability of each user (i.e.
$SINR\geq\gamma_{0}$). Accordingly, we define the reward function for each
agent as per the achievable rate of each user, $R_{k}$. So the reward $r_{t}$
at each time step becomes:

$\displaystyle r_{t}=R_{k}.$ (15)

To ensure that the reliability constraint is satisfied, the condition in (11c)
is checked during each reward calculation. If the condition is not fulfilled,
the algorithm simply declares the power choice of that user as invalid and
sets the reward to zero. Thus, the overall reward design can be expressed as

$\displaystyle R_{t}=\left\\{\begin{array}[]{ll}\\!R_{k},\\!\\!&\\!\text{ if
}(1-p_{0})\prod_{j\neq
k}\left(1+\frac{\gamma_{0}|{h}_{kj}|^{2}{p}_{j}}{|{h}_{kk}|^{2}{p}_{k}}\right)\leq
1\\\ \\!0,\\!&\\!\text{ otherwise. }\end{array}\right.$ (18)

###  III-B Training the DNN

The objective of training design is to exploit a trained DQN to replay the
experience in learning the resource allocation policy. The DQN takes an
observation form the environment $s^{t}$ as input and outputs the possible
actions. Multiple episodes are run to train the DQN and the agent uses the
$\epsilon$-greedy policy to explore the environment. After the transition of
the environment due to the actions taken as well as the change of channel
information, the agent saves the transition tuple
$(s^{(t)},a^{(t)},r^{(t)},s^{(t+1)})$ in a replay memory. A mini batch of
transition tuple or experience is sampled randomly from the replay memory. The
mini batch is used to train the DQN and stochastic gradient-descent method is
used for updating the DQN. The goal is to minimize the sum-squared error:

$\displaystyle\sum\left[R^{(t+1)}+\gamma\max_{a^{\prime}}Q\left(s^{(t+1)},a^{\prime};\theta^{-}\right)-Q\left(s^{(t)},a_{(t)};\theta\right)\right]^{2}.$
(19)

Here $\theta^{-}$ is the parameter of target $Q$-network which is a DQN with
frozen weights. The target $Q$-network parameters are initialized by
duplicating the parameters of training DQN and after a certain number of
episodes they are updated to mirror the parameters of the training DQN. The
reason for using two DQN is that if we use same $Q$-network for generating the
target $Q$-value and the predicted $Q$-value, the learning becomes really
unstable. So, 2 DQNs are used to ensure a stable learning. The overall
training procedure is summarized in Algorithm 1.

Algorithm 1 Resource Allocation with DRL

1: Initialize the environment $E$ (users’ channel information)

2: Initialize the Q-network with action space $A$.

3: for each episode do

4: Reset environment.

5: Initialize $t=0$.

6: repeat

7: Observe state $S_{t}$.

8: Choose action $a_{t}$ from $A$ using $\epsilon$-greedy strategy.

9: Agent takes action $a_{t}$ and receives reward $r_{t}$.

10: State transition happens and agent receives next-state $S_{t+1}$.

11: Store $(s^{(t)},a^{(t)},r^{(t)},s^{(t+1)})$ in replay memory.

12: Sample mini-batch from replay memory.

13: Using the stochastic gradient descent method, optimize error between
Q-network and learning targets defined in (19).

14: t=t+1.

15: until $t>$ maximum step per episode.

16: end for

###  III-C Testing the DNN

In testing stage, at each time step $t$, the agent receives channel
realization $s^{t}$ and $\epsilon$ is set to the value from the very last
training step. The agent selects an action $a^{(t)}$, which has the maximum
$Q$-value according to the DQN. After that the agent starts transmission with
the power level according to their selected action. Since the training
procedure described in Algorithm 1 is computationally expensive, it can be
performed offline for a large number of channel conditions, over many episodes
and the computationally inexpensive, testing or implementation phases can be
executed online for the actual network deployment.

##  IV Numerical Simulations

In this section, we compare the analytical results obtained in Section II with
those of the proposed DQN algorithm. The channel coefficients are generated
using the standards of Gaussian IC [1]. The channel coefficients follow a
Rayleigh fading distribution with zero mean and unit variance. We consider
minimum SINR threshold $\gamma_{0}$ as $-10$ dB and $P_{min}$ as $0$ dB. We
use a dense DQN with three hidden layers consisting of $200$, $100$ and $40$
neurons. We also use the rectified linear unit (ReLU), $f(x)=\max(0,x)$ as
activation function and _RMSProp_ optimizer is used for updating the weights
of the network using learning rate of 0.001 and updating the DQN over $3000$
episode. The proposed method is compared with several methods namely: a GP
method discussed in Section II, a WMMSE algorithm developed in [5], a
supervised DNN [16] and a WMMSE based supervised learning method [8].

Figure 2: Sum-rate versus transmit power budget $P_{\rm max}$.

The Fig. 2 shows the sum rate performance of the DQN in a 4 user scenario with
outage probability $p_{0}=0.3$. We can see that the proposed method performs
significantly better than the existing GP algorithm [13]. The WMMSE approach
in [5] performs better since it does not maintain reliability constraint. Our
proposed approach is capable of obtaining better result than the approach in
[5], if we ignore the reliability constraint during reward design as shown in
Fig. 2.

Figure 3: Sum-rate versus outage probability threshold $p_{0}$.

Fig. 3 shows the performance of our proposed method over varying outage
probability $p_{0}$. We make the comparison considering $P_{max}$ as $4$ dB.
It can be seen from the figure that the proposed DQN method significantly
outperforms traditional GP method. It can also be observed that the sum-rate
of the proposed method increases with $p_{0}$, while in traditional approaches
performance decreases. This can be explained with the reward design (18). When
$p_{0}$ increases, due to the term $(1-p_{0})$ the overall condition term
decreases and thus offers higher probability of becoming less then or equal to
one and full-filling the condition for receiving reward resulting in increased
sum-rate performance.

Figure 4: Average sum-rate versus the number of user pairs $K$.

Fig. 4 shows average sum rate against different number of users. As expected,
the average sum-rate performance of the DQN decreases as the number of user
increases in the system since higher number of users generate additional
interference. This effect can be compensated by increasing transmit power
budget as evident from the Fig. 4.

##  V Conclusion

In this paper, we have developed a DRL based resource allocation scheme which
successfully learns the power allocation of a wireless interference channel
and guarantees reliability requirements for each user. We exploit a DQN with
experience to learn the relationship between users’ channel information and
power allocation. We have demonstrated the superiority of the proposed
approach by comparing the sum-rate performance with that of the existing GP-
based solution.

## References

  * [1] M. R. A. Khandaker and Y. Rong, “Interference MIMO relay channel: Joint power control and transceiver-relay beamforming,” _IEEE Trans. Signal Process._ , vol. 60, no. 12, pp. 6509–6518, 2012.
  * [2] F. Wang, M. Krunz, and S. Cui, “Price-based spectrum management in cognitive radio networks,” _IEEE J. Sel. Topics Signal Process._ , vol. 2, no. 1, pp. 74–87, 2008.
  * [3] Y. Hu and A. Ribeiro, “Adaptive distributed algorithms for optimal random access channels,” _IEEE Trans. Wireless Commun._ , vol. 10, no. 8, pp. 2703–2715, 2011.
  * [4] K. Shen and W. Yu, “Fractional programming for communication systems—part i: Power control and beamforming,” _IEEE Trans. Signal Process._ , vol. 66, no. 10, pp. 2616–2630, 2018.
  * [5] Q. Shi, M. Razaviyayn, Z.-Q. Luo, and C. He, “An iteratively weighted mmse approach to distributed sum-utility maximization for a mimo interfering broadcast channel,” _IEEE Trans. Signal Process._ , vol. 59, no. 9, pp. 4331–4340, 2011.
  * [6] F. Tariq, M. R. A. Khandaker, K.-K. Wong, M. A. Imran, M. Bennis, and M. Debbah, “A speculative study on 6G,” _IEEE Wireless Communications_ , vol. 27, no. 4, pp. 118–125, 2020.
  * [7] T. Li, M. R. A. Khandaker, F. Tariq, K. Wong, and R. T. Khan, “Learning the wireless V2I channels using deep neural networks,” in _IEEE 90th Vehicular Technology Conference (VTC-Fall)_ , 2019.
  * [8] J. Gao, M. R. A. Khandaker, F. Tariq, K.-K. Wong, and R. T. Khan, “Deep neural network based resource allocation for V2X communications,” in _Proc. IEEE 90th Vehicular Technol. Conf. (VTC-Fall)_ , 2019.
  * [9] Q. Mao, F. Hu, and Q. Hao, “Deep learning for intelligent wireless networks: A comprehensive survey,” _IEEE Commun. Surveys Tuts._ , vol. 20, no. 4, pp. 2595–2621, 2018.
  * [10] K. I. Ahmed and E. Hossain, “A deep Q-learning method for downlink power allocation in multi-cell networks,” _arXiv preprint arXiv:1904.13032_ , 2019\.
  * [11] Y. S. Nasir and D. Guo, “Multi-agent deep reinforcement learning for dynamic power allocation in wireless networks,” _IEEE J. Sel. Areas Commun._ , vol. 37, no. 10, pp. 2239–2250, 2019.
  * [12] M. G. Khoshkholgh and H. Yanikomeroglu, “Faded-experience trust region policy optimization for model-free power allocation in interference channel,” _IEEE Wireless Commun. Lett._ , pp. 1–1, 2020.
  * [13] S. Kandukuri and S. Boyd, “Optimal power control in interference-limited fading wireless channels with outage-probability specifications,” _IEEE Trans. Wireless Commun._ , vol. 1, pp. 46–55, 2002.
  * [14] Y.-D. Yao and A. Sheikh, “Outage probability analysis for microcell mobile radio systems with cochannel interferers in Rician/Rayleigh fading environment,” _Electronics Lett._ , vol. 26, pp. 864–866, June 1990.
  * [15] D. Mitra, _An Asynchronous Distributed Algorithm for Power Control in Cellular Radio Systems_. Boston, MA: Springer US, 1994, pp. 177–186.
  * [16] Sun et al., “Learning to optimize: Training deep neural networks for wireless resource management,” in _IEEE Int. Wksp. Signal Process. Adv. Wireless Commun. (SPAWC)_ , 2017.