# SenseFitting: Sense Level Semantic Specialization of Word Embeddings for
Word Sense Disambiguation

Manuel Stoeckel, Sajawel Ahmed, Alexander Mehler
Text Technology Lab
Goethe University Frankfurt
Frankfurt, Germany
<EMAIL_ADDRESS>
{sahmed<EMAIL_ADDRESS>

###### Abstract

We introduce a neural network-based system of Word Sense Disambiguation (WSD)
for German that is based on SenseFitting, a novel method for optimizing WSD.
We outperform knowledge-based WSD methods by up to $25\%$ F1-score and produce
a new state-of-the-art on the German sense-annotated dataset WebCAGe. Our
method uses three feature vectors consisting of a) sense, b) gloss, and c)
relational vectors to represent target senses and to compare them with the
vector centroids of sample contexts. Utilizing widely available word
embeddings and lexical resources, we are able to compensate for the lower
resource availability of German. SenseFitting builds upon the recently
introduced semantic specialization procedure Attract-Repel, and leverages
sense level semantic constraints from lexical-semantic networks (e.g.
GermaNet) or online social dictionaries (e.g. Wiktionary) to produce high-
quality sense embeddings from pre-trained word embeddings. We evaluate our
sense embeddings with a new SimLex-999 based similarity dataset, called
SimSense, that we developed for this work. We achieve results that outperform
current lemma-based specialization methods for German, making them comparable
to results achieved for English.

##  1 Introduction

Embeddings have been used successfully for a variety of Natural Language
Processing (NLP) tasks, including Word Sense Disambiguation (WSD). There is a
multitude of English evaluation datasets, sense annotated corpora, and tasks
available to measure the performance of WSD systems, some of which were
designed with embeddings in mind (Moro and Navigli, 2015), some without
(Edmonds and Cotton, 2001; Mihalcea et al., 2004). For German, however, only a
few sense annotated datasets exist and therefore possibilities for NLP
applications are limited.

In this paper, we present a system for lexical-sample WSD that is applicable
to high and low resource languages by using widely available resources such as
word embeddings trained on large unlabeled text corpora, and lexical-semantic
networks. We refine the procedure of Simple Embedding-Based Word Sense
Disambiguation introduced by Oele and van Noord (2018) for Dutch (another low-
resource language), who use sense embeddings and expanded glosses to represent
target senses. Instead of only extending glosses and contexts, as suggested by
the authors, we include a third feature vector in sense representation that is
derived from the sense relationships of the target senses in a lexical-
semantic network.

The sense embeddings used in this work are computed with the help of
SenseFitting, a novel method that extends the semantic specialization
procedure Attract-Repel (Mrkšić et al., 2017) by exploring sense level
semantic constraints. We apply SenseFitting to sense embeddings based on pre-
trained word embeddings using a technique similar to that of Chen et al.
(2014). We show that this method can significantly improve the performance of
WSD compared to the use of word embeddings alone. We also show that the
quality of semantic constraints plays a major role in this improvement.

We independently leverage both GermaNet (Hamp and Feldweg, 1997), a German
lexical-semantic network similar to WordNet (Kunze and Lemnitzer, 2002), and
the German Wiktionary111https://de.wiktionary.org, a collaborative online
social dictionary, to extract semantic constraints on the sense level and
apply SenseFitting to word embeddings trained on two large German corpora
(Leipzig (Goldhahn et al., 2012) and COW (Schäfer, 2015)). To evaluate our
sense embeddings, we create SimSense, a resource for German based on from
SimLex-999 (Hill et al., 2015) that addresses sense level semantic
similarities.

From the few open source datasets containing German sense annotations, we
chose WebCAGe (Henrich et al., 2012), which is the only dataset that contains
sense annotations from both lexical resources used in this paper: WebCAGe
mainly collects sentences sampled from the German Wiktionary and annotated
with senses from GermaNet. We align WebCAGe with a current version of the
German Wiktionary to use the resulting subset WebCAGe-aligned (WCA) for
evaluating our algorithm by means of both GermaNet and Wiktionary.

The remainder of this paper is organized as follows: sec:related-work reviews
related work. sec:methods describes SenseFitting and our WSD algorithm.
sec:setup describes the preparation of the resources (GermaNet, Wiktionary,
SimSense, WCA) used to evaluate our algorithm. sec:results presents
experimental results. Finally, sec:conclusion draws a conclusion.

##  2 Related Work

Recent years have seen multiple noteworthy approaches to WSD using sense
embeddings. Early works trained sense embeddings on automatically
disambiguated corpora. These include Chen et al. (2014), who present a unified
model for joint word sense representation and disambiguation, and Iacobacci et
al. (2015), who train sense embeddings using word2vec (Mikolov et al., 2013).
More recently, Uslu et al. (2018) used a supervised method to train sense
embeddings on manually annotated corpora.

Oele and van Noord (2018) use sense embeddings created with AutoExtend (Rothe
and Schütze, 2015), a method for fine-tuning word embeddings to include
representations of lexemes and synsets from WordNet. Tuning methods are also
proposed by Faruqui et al. (2015), who tune embeddings using synonymy-related
constraints on lemma-level extracted from WordNet, and Wieting et al. (2015),
who explore paraphrase-related constraints. Mrkšić et al. (2016, 2017)
evaluate antonymy- and synonymy-related constraints on lemma-level to refine
word embeddings, a method called semantic specialization.

In this article we combine the method of semantic specialization with the
generation of sense embeddings. That is, we examine sense-related constraints
provided by known lexical resources for the fine tuning of sense embeddings to
finally solve various tasks of WSD.

##  3 Methods

Figure 1: Flowchart showing the data flow within our work. Various data
resources (green) are used in the presented methods (red) to improve the final
WSD performance.

fig:SenseFitting-flowchart shows a flowchart of our approach. Pre-trained word
embeddings and sense level relational data are used in SenseFitting to create
sense embeddings. These are used along the lexical resources to create three-
fold vector representations which are finally applied to our WSD downstream
task. The following sec:three-fold vector representations describes our vector
representations of senses, while sec:sense-fitting describes SenseFitting and
sec:wsd describes our WSD approach. Some lexical resources, such as
Wiktionary, are ambiguous on the sense level and have to be disambiguated
first. This is briefly reported in sec:setup-sense-fitting-pre-wkt and
discussed more thoroughly in sec:supplement-sense-fitting-pre-wkt.

###  3.1 Three-fold Vector Representations of Senses

We use three feature vectors to represent any sense $S$ of a word $w$: the
sense vector $S_{s}$, the gloss vector $S_{g}$ and the sense relation vector
$S_{r}$. These three vectors are compared with lexical contexts $C$ in which
$w$ is observed to compute similarity scores between $C$ and $S$. Contexts $C$
in which $w$ has to be disambiguated are represented by the centroid $C_{w}$
of the embeddings of all textual neighbors of $w$ in $C$. Likewise, the gloss
vector $S_{g}$ corresponds to the centroid of the embeddings of all words
describing $S$ in the underlying gloss, while the sense relation vector
$S_{r}$ denotes the centroid of the lexical neighborhood of $S$ in GermaNet
and alternatively in Wiktionary. Thirdly, the sense vector $S_{s}$ is obtained
from a sense embedding which has been created using SenseFitting, a semantic
specialization method that explores sense level semantic constraints to refine
word embeddings (see sec:sense-fitting).

To compute the relational vector $S_{r}$ we exploit explicit semantic
constraints from terminological ontologies. This is inspired by Oele and van
Noord (2018), who use elements of lexical chains that reach a similarity
threshold or are similar according to a distribution thesaurus to extend the
gloss of a sense. However, we select words or senses that are related to sense
$S$ according to the given lexical-semantic network to form $S_{r}$ as an
additional feature vector. We assume that words contained in the gloss of a
sense are distributed differently than words that have an explicit semantic
relation to that sense. Thus, we distinguish $S_{g}$ and $S_{r}$ as two
different vector representations. In order to compute $S_{r}$, we exploit
synonymy and hyponymy relations from GermaNet and alternatively from
Wiktionary.

###  3.2 SenseFitting

Sense vectors $S_{s}$ are computed by means of SenseFitting, an optimization
method of semantic specialization that creates sense embeddings from word
embeddings using lexical-semantic networks. SenseFitting is based on semantic
specialization, a post-processing method that increases the quality of
embeddings by constraining their computation through semantic constraints
derived from terminological ontologies. In previous work, these constraints
are extracted on the lemma level to specialize word embeddings (e.g. Faruqui
et al., 2015; Mrkšić et al., 2016, 2017). We extend this approach by
constraining the computation of sense embeddings through sense level
relations.

Initially, we compute the sense embedding $S_{s}$ of the sense $S$ of word $w$
as the centroid of the embeddings of all words contained in the gloss
describing $S$ whose cosine similarity to the word embedding of $w$ is higher
than threshold $\delta$ (we set $\delta=0.05$), similarly to the way Chen et
al. (2014) create their initial sense embeddings. To keep distributional
information from the original embeddings and to compensate for differences in
the quality of glosses, we include the embedding of $w$ into computing this
centroid. In a second step, $S_{s}$ is post-processed with the help of
semantic specialization. This is done using the Attract-
Repel222https://github.com/nmrksic/attract-repel algorithm (Mrkšić et al.,
2017). This specialization algorithm differs from related approaches such as
retrofitting (Faruqui et al., 2015) in that it refers to synonyms and antonyms
to constrain the fine-tuning of embeddings in different ways: while training
attempts to locate the embeddings further apart from antonyms, synonyms are
brought closer together. In this way, embeddings are created that better
reflect sense relations. As a result of this constraint satisfaction process,
$S_{s}$ is computed as a hybrid vector representation to which $S_{g}$ and the
sense relations that embed $S$ into the operational lexical resource (GermaNet
or Wiktionary) contribute in different ways.

###  3.3 Word Sense Disambiguation

After $S_{g},S_{r}$ and $S_{s}$ have been calculated for each sense $S$ of a
word $w$, WSD can be performed. For this purpose, we calculate the similarity
of the context vector $C_{w}$ and the three-part vector representation
$S_{w}=\\{S_{s},S_{g},S_{r}\\}$ of the sense $S$ of $w$, choosing the sense
with the highest score as follows:

$\operatorname{sim}(C_{w},S_{w})=\dfrac{1}{|S_{w}|}\sum\limits_{X\leavevmode\nobreak\
\in\leavevmode\nobreak\ S_{w}}\cos(C_{w},X)$ (1)
$S_{\mathit{opt}}=\operatorname*{arg\,max}\limits_{S_{w}\leavevmode\nobreak\
\in\leavevmode\nobreak\ S(w)}\operatorname{sim}(C_{w},S_{w})$ (2)

Here $S(w)$ denotes the set of all senses of $w$. Note that all three
component vectors are equally weighted in eq:1. Of course, we could have
considered weighting the effects of these component vectors differently and
learning their relative weights. In our experiments, however, our simpler
approach was already very effective.

##  4 Experimental Setup

###  4.1 Embeddings

We used a set of pre-trained word
embeddings333https://texttechnologylab.org/resources2018/ of Ahmed and Mehler
(2018) as a basis for SenseFitting. This concerns wang2vec embeddings (Ling et
al., 2015) trained on two large German corpora: the COW corpus and the
LeipzigMT corpus, which is the combination of Leipzig40-2018 (Schäfer, 2015)
and WMT-2010-German corpora (Callison-Burch et al., 2010). Ahmed and Mehler
(2018) trained Structured Skip-gram wang2vec embeddings with a dimension of
100, window size of 8, minimum word count threshold of 4 and otherwise default
parameters as given in Ling et al. (2015).

As German is a highly inflected language in comparison to English, a single
word can be distributed across various morphological and spelling variants
effectively weakening the information value of their embeddings. To mitigate
the dispersion of information of an embedding over multiple variants, we also
use embeddings trained on lemmatized versions of the corpora.

The original embeddings are very large, based on a vocabulary of several
million words. Thus, we drop all entries that do not occur anywhere in
Wiktionary, GermaNet or our test data. In the case of COW, this leads to
$435\,003$ embeddings of words converted to lowercase letters and $402\,575$
lemma-based embeddings. In the case of Leipzig40MT, this leads to $347\,550$
lowercased word embeddings and $334\,625$ lemma embeddings. On top of these
embeddings, we performed SenseFitting using the default parameters for
Attract-Repel defined by Mrkšić et al. (2017).

###  4.2 Preparing SenseFitting

We utilize GermaNet and Wiktionary as sense inventories for SenseFitting.
Sections 4.2.1 and 4.2.2 describe the steps necessary to prepare them for
SenseFitting, while sec:setup-sf-resources provides a summary of the resulting
resources. sec:setup-sf-SimSense describes SimSense, a sense-annotated
semantic similarity evaluation dataset, which we specifically created for this
work.

####  4.2.1 Preprocessing GermaNet

Only a few modifications are necessary to prepare GermaNet for SenseFitting
with Attract-Repel. GermaNet does not contain own sense glosses but glosses
mapped from Wiktionary by Henrich et al. (2011), yielding $27\,903$ sense
definitions. We expand these definitions with an additional $32\,850$ mappings
from Matuschek and Gurevych (2013). In this way, we obtain a total of
$30\,352$ distinct sense glosses for our training444The entire joined dataset
holds $30\,455$ mappings, but some do not occur in our GermaNet version or are
just empty.. In order to run SenseFitting on the lemma-based wang2vec
embeddings from Ahmed and Mehler (2018), we lemmatized the glosses using spaCy
V2.0555https://spacy.io, the same tool used to lemmatize the embeddings’
training corpora.

####  4.2.2 Preprocessing Wiktionary

Senses [1] Sense 1 [2] Sense 2 [3] Sense 3  Lemma Senses [1] Sense 1 [2] Sense
2 [3] Sense 3  Lemma URL  Figure 2: An example Wiktionary sense-to-lemma
relation in solid green and a disambiguated relation as dashed line below.
Image courtesy of Mehler et al. (2018)

We used WikiDragon (Gleim et al., 2018) to obtain a Neo4j666https://neo4j.com/
graph-database from a current German Wiktionary dump (April 2018) and used
spaCy V2.0 to lemmatize $147\,363$ sense glosses, adding them to the database.
Utilizing the German Wiktionary for SenseFitting, which relies on sense-to-
sense relations, is more challenging because relations are encoded as URLs
from the source sense to the target’s page (sense-to-lemma relations, see
fig:wiktionary-example) and thus are inherently ambiguous. In a further pre-
processing step, detailed and evaluated in sec:supplement-sense-fitting-pre-
wkt, we are able to disambiguate $164\,154$ of $220\,927$ relations777Counting
only antonyms, hyponyms, hypernyms and synonyms., leaving $56\,773$ ambiguous
sense-to-lemma relations. These are still used in training, as our experiments
have shown a small positive effect on embedding quality while retaining them.

####  4.2.3 SenseFitting Training Resources

After pre-processing, the relation count in the case of GermaNet ranges from
$685\,484$ synonym relations to $3\,426$ antonym relations. In the case of
Wiktionary, the relation count ranges from $249\,751$ synonyms to $51\,173$
antonyms888These are the numbers when using the COW embeddings. With the
Leipzig embeddings, all numbers are about 10-20% lower.. tab:sense-inv-stats
in the supplemental material sec:supplement-sensefitting-training-resources
shows all resource statistics including the individual count for each relation
type. Training with this amount of data takes about 15 minutes per epoch for
Wiktionary and 30 minutes for GermaNet on a single NVIDIA 1070 GPU.

###  4.3 SenseFitting Evaluation Dataset

Dataset | Sample | Source |  Source Gloss | Target | Target Gloss | Score
---|---|---|---|---|---|---
SimLex | 1 | new |  N/A | fresh | N/A | 8.62
SimLex | 2 | wide |  N/A | fresh | N/A | 0.00
SimSense | 1 (pos) | new1 |  Recently made, or created. | fresh1 | Newly produced or obtained. | 9.00
SimSense | 2 (neg) | new1 |  Recently made, or created. | fresh2 | Not cooked, dried, frozen, or spoiled. | 3.00
SimSense | 3 (neg) | new1 |  Recently made, or created. | fresh4 | Refreshing or cool. | 0.00
SimSense | 4 (false) | wide1 |  Having a large physical extent from side to side. | fresh1 | Newly produced or obtained. | 0.00
Table 1:  SimSense annotations using English Wiktionary glosses for two word
pairs from the English SimLex. Index $n$ denotes the $n$-th sense in the
Wiktionary entry.

In their work, Mrkšić et al. (2017) measure Attract-Repel’s performance by
using the multilingual semantic similarity evaluation dataset SimLex (Hill et
al., 2015). It contains word pairs that have been annotated by experts
regarding their degree of similarity in order to clarify their status as
synonyms. Mrkšić et al. (2017) obtain state-of-the-art results using
Spearman’s rank correlation coefficient to measure the performance of their
embeddings. Although SimLex is available in German, there is, unfortunately,
no such dataset for sense similarity.

Thus, in order to evaluate our sense embeddings computed with SenseFitting, we
tagged a subset of SimLex word pairs with senses using the GermaNet sense
inventory and annotated their degree of similarity.

####  4.3.1 Dataset Description

The annotations are grouped into three categories: positive, negative and
false samples. They aim to capture four different types of relations between
senses for which we believe a semantic specialization method should be tested.
tab:simsense-example exemplifies annotations for each relation type using two
English word pairs from SimLex999In this example, we consider new1 and fresh1
to be synonyms. using sense glosses from the English Wiktionary.

Positive samples are obtained from word pairs that are connected at least by
one sense relation that exists between their senses. We look for such pairings
for each sense in our subset of SimLex using GermaNet as resource. From the
first pairing that we find for such a sense, we create a positive sample.
Sample 1 (pos) in tab:simsense-example captures two synonymous senses of new
and fresh.

Then, for any such pairing, we check all other pairings and use the first one
whose senses are unrelated to generate a negative sample. Sample 2 (neg) shows
a negative sample of two unrelated senses with a small similarity, while
sample 3 (neg) shows a negative sample of two unrelated senses with no
similarity.

Words and senses for which we did not find any positive sample are considered
to be entirely unrelated, like sample 4 (false) in the table above. In this
case, we use the first senses of such words to generate false samples. Using
synonym, antonym, hyponym and hypernym relations from GermaNet for generating
positive samples, we get 210 pairs, which were finally annotated regarding
their similarity based on their glosses (see sec:setup-sf-SimSense-
annotation). These annotations form the evaluation dataset called SimSense.

Unfortunately, GermaNet does not feature Wiktionary identifiers for its
glosses mapped by Henrich et al. (2011), complicating the use of SimSense to
evaluate SenseFitting performance with Wiktionary. Thus, only 76 of the
annotated pairs are applicable to evaluate SenseFitting with Wiktionary.

####  4.3.2 Annotation Process

The SimSense dataset was annotated by two native speakers of German. Both
annotators were given a table containing a single unannotated SimSense pair
per row. Each row only detailed the senses lemmata and definition, but did not
contain further information like the sample type. With the practical example
in tab:simsense-example, the annotators would have been given the third to
sixth column (Source, Target and respective Gloss columns).

To evaluate the resulting dataset, we computed inter-annotator agreement
$\rho$ and average response deviation $\sigma$ in the same manner as Hill et
al. (2015). The authors compute $\rho$ as the average of the Spearman rank
correlation for each annotation pairwise across the authors and $\sigma$ as
the average standard deviation, likewise.

fig:SimSense-IAA shows the inter-annotator agreement and average response
deviation for all pairs and by sample type. With $0.85$ inter-annotator
agreement $\rho$ over all annotation is significantly ($+0.177$) higher than
for SimLex-999. Further, the response deviation $\sigma$ is about half
compared with SimLex-999 (Hill et al., 2015)101010Note: The authors report a
response consistency of $0.751$ for all annotations in their work, which is
defined as $1/\sigma$.. We presume the lower ambiguity of senses with a given
definition to be responsible for our better values, possibly influenced by the
lower amount of annotators and annotation pairs.

allpositivenegativefalse$0.6$$0.7$$0.8$$0.85$$0.79$$0.67$$0.61$$0.67$$0.67$$0.64$$0.72$$\rho$$\sigma$
Figure 3: Inter-Annotator Agreement $\rho$ & Response Consistency $\sigma$ for
SimSense

###  4.4 Word Sense Disambiguation

In all experiments described below, our WSD method is based on the following
parameter setting unless otherwise stated. We set the window size to observe
context words around target words to 8, matching the window size of the
wang2vec embeddings during training. We exclude the target word from the list
of context words, as suggested by Iacobacci et al. (2016). As a measure of the
performance of WSD, we adopt the scoring system used in IMS (Zhong and Ng,
2010), taking a single best guess for every sample.

####  4.4.1 Preprocessing WebCAGe

As mentioned at the beginning, evaluation datasets such as Senseval or
SemEval, as well as training datasets of sense annotations, are missing in
German. However, there is a single sense annotated corpus that has annotations
for GermaNet and Wiktionary called WebCAGe (Henrich et al., 2012). WebCAGe is
a corpus mainly composed of examples from word sense definitions from the
German Wiktionary which were automatically annotated with GermaNet senses and
hand-corrected afterward. However, the authors do not provide a mapping to
Wiktionary; only GermaNet senses are tagged with an ID. After the pre-
processing outlined in the supplemental section sec:supplement-wca, we get a
new corpus called WebCAGe-aligned (WCA) that contains about 58% of WebCAGe’s
sentences, for a total of 4 125 sentences with 2 178 unique lemmas and thus
1.89 sentences per lemma.

##  5 Results

###  5.1 SenseFitting Performance

tab:SimSense-scores shows the Spearman’s rank correlation for all our created
sense embeddings with SimSense. The third column shows the specialization
method used for the given embedding. Here “Baseline” denotes sense embeddings
which were initialized using only the vector centroid of sense glosses as
described in sec:sense-fitting but not specialized further.

All results for GermaNet outperform the baseline by about $+0.314$, across
both COW and Leipzig embeddings and with or without lemmatization. This shows
that extending word embeddings to the sense-layer and specializing them using
sense-to-sense relations improves the representation of semantic similarity by
a large margin. For comparison to lemma-based specialization, we ran Attract-
Repel with GermaNet lemma relations on our COW embeddings. The resulting
embeddings scored $0.415$ for SimLex, slightly lower than the ones trained by
Mrkšić et al. (2017) which scored $0.43$. The underlying COW embeddings
however already score $0.352$ for Simlex, resulting in a performance increase
with Attract-Repel of $0.063$. Therefore, the increase of SenseFitting is
nearly 7 times as large as the increase of Attract-Repel using lemma-to-lemma
constraints. This improvement is also significantly higher than the
improvement with Attract-Repel on SimLex for German reported by Mrkšić et al.
(2017) of $+0.23$ using lemma-based monolingual data.

Embedding | Resource | Method | $\rho$
---|---|---|---
Leipzig-lower | GN | Baseline | $0.385$
Leipzig-lower | GN | SenseFitting | $0.700$
Leipzig-lemma | GN | Baseline | $0.359$
Leipzig-lemma | GN | SenseFitting | $0.703$
COW-lower | GN | Baseline | $0.425$
COW-lower | GN | SenseFitting | $\mathbf{0.738}$
COW-lemma | GN | Baseline | $0.420$
COW-lemma | GN | SenseFitting | $0.734$
Leipzig-lower | WKT | Baseline | $0.212$
Leipzig-lower | WKT | SenseFitting | $0.484$
Leipzig-lemma | WKT | Baseline | $0.236$
Leipzig-lemma | WKT | SenseFitting | $0.392$
COW-lower | WKT | Baseline | $0.197$
COW-lower | WKT | SenseFitting | $\mathbf{0.488}$
COW-lemma | WKT | Baseline | $0.216$
COW-lemma | WKT | SenseFitting | $0.466$
Table 2:  SimSense Spearman’s correlation $\rho$ with COW and Leipzig
embeddings. Bold results are the maxima for the corresponding resource.

It is worth noting that all COW embeddings trained with GermaNet relations
strictly outperform the Leipzig embeddings confirming Ahmed and Mehler (2018)
observation that the COW embeddings are of higher quality. The authors
attribute this to the significantly higher data size of the COW corpus which
is with over $600$ million sentences more than ten times larger than the
extended Leipzig corpus with $60$ million sentences.

The lower half of tab:SimSense-scores shows the Spearman’s correlation for the
Wiktionary sense embeddings, which perform only about half as good as
GermaNet’s did. Using GermaNet to create the SimSense dataset may introduce a
bias, as we do not control whether Wiktionary has similar relations to the
ones from which the dataset is induced. Instead we rely on the fact that the
glosses in GermaNet were automatically mapped from Wiktionary (Henrich et al.,
2011), strongly suggesting there must a certain degree of similarity to the
sense definitions and relations. Still, further work has to be done to fully
analyze the possible impact of this issue and refine the evaluation of
SenseFitting for Wiktionary. While the total performance using Wiktionary as
SenseFitting resource is lower when compared with GermaNet, the relative
increase of $+0.291$ for the COW-lower embeddings using SenseFitting is still
noteworthy.

Our results demonstrate that we can improve the semantic quality of embeddings
even more with sense level constraints. Further, our results show that
SenseFitting improves the representation of semantic relationships not only
for senses with an explicit relationship, but also for those not covered in
the lexical resource. Given the status of German as a low-resource language
and the difficulties in generating embeddings for this highly inflectional
language, these results are promising.

###  5.2 Word Sense Disambiguation

Taking all optimized resources into account, we come to the main task of WSD.
The following two tables show the results of our WSD method for the WCA
dataset. tab:wsd-wca-gn holds the results with GermaNet as sense inventory,
while tab:wsd-wca-wkt holds the results for Wiktionary: w2v stands for
wang2vec, AR for Attract-Repel and SF for SenseFitting. When running our
disambiguation with word embeddings (w2v & AR), we drop the sense feature
vector and use gloss and relational features only. We used a First Sense
baseline, as suggested by Iacobacci et al. (2016), and a Random Sense
baseline.

All results for GermaNet outperform these baselines by a significant margin.
The embeddings trained on the Leipzig corpus are outperformed by the COW
embeddings, again confirming the previous assumption that the latter are of
higher quality. However, we cannot entirely confirm our hypothesis about
lemmatization (formulated in sec:setup-embeddings). The lemmatized Leipzig
sense embeddings bring a slight improvement for verbs, beating their
lowercased variant by $+0.98$, but the COW embeddings do not profit from
lemmatization. In light of the fact that verbs usually have the most
inflections in German, followed by adjectives, we presume that lemmatizing the
many possible wordforms of a verb or adjective helps to capture their meaning
and consequently improve their embedding. Although spaCy gives decent results
for lemmatization in German, we might see an improvement with a better
lemmatizer, as our disambiguation performance depends on it.

Model |  Method | Nouns | Verbs | Adj.
---|---|---|---|---
Henrich (2015) |  Lesk | $53.17$ | $28.76$ | $29.01$
Borda c. | $55.92$ | $45.97$ | $\mathbf{56.28}$
Baseline |  First Sense | $38.68$ | $31.54$ | $33.04$
Random Sense | $34.28$ | $26.59$ | $29.17$
w2v∗ |  COW-lower | $51.49$ | $42.67$ | $50.89$
AR∗ |  COW-lower | $46.57$ | $43.01$ | $46.43$
SF |  Leipzig-lower | $50.00$ | $43.38$ | $49.70$
SF |  Leipzig-lemma | $46.88$ | $44.36$ | $47.46$
SF |  COW-lower | $\mathbf{56.03}$ | $\mathbf{47.13}$ | $53.27$
SF |  COW-lemma | $50.21$ | $45.68$ | $51.79$
Table 3:  WCA F1-Scores for GermaNet in percent. Our maximas are underlined,
absolute best in bold. ∗Only definition and relational feature vectors are
used.

We take steps towards closing the gap to WSD results for high-resource
languages such as English, and outperform the Lesk-based methods used by
Henrich (2015) on the original WebCAGe corpus with both GermaNet as dataset by
$+2.86$, $+18.37$ and $+25.81$ for nouns, verbs, and adjectives, respectively.
The work of Henrich (2015) provides the only other German WSD method for
WebCAGe, using knowledge-based and supervised machine learning (ML) methods.
The ML approaches are not comparable to our work, as, according to Henrich
(2015), they are only applicable to a very small subset of 43 out of 2 178
lemmas from the WebCAGe corpus. We outperform the author’s best knowledge-
based method Borda count, which combines all methods in their work, for nouns
($+0.11$) and verbs ($+1.16$). Further, our sense embedding performance
exceeds the Attract-Repel (AR) embeddings performance. The AR embeddings, in
turn, cannot beat the pure wang2vec embeddings, although after specialization
they perform similarly for SimLex ($0.415$) to the ones trained by Mrkšić et
al. (2017) using German monolingual data ($0.43$).

tab:wsd-wca-wkt show the results using Wiktionary as a sense inventory.
Wiktionary has a large number of sense glosses, resulting in good baseline
performance using wang2vec embeddings, however, with automatically annotated
relations, SenseFitting does not improve the performance over the original
wang2vec embeddings consistently. Still, our results beat the Wiktionary Lesk
results from Henrich (2015) by more than $5\%$. The best results throughout
our work are obtained with SenseFitting on expert-crafted lexical data from
GermaNet, which outperforms Wiktionary in all tasks.

Method |  Model | Nouns | Verbs | Adj.
---|---|---|---|---
Henrich (2015) |  Lesk | $50.38$ | $41.66$ | $42.77$
Borda c. | $55.92$ | $45.97$ | $\mathbf{56.28}$
Baseline |  First Sense | $41.68$ | $38.06$ | $42.90$
Random Sense | $29.97$ | $28.69$ | $32.63$
w2v∗ |  COW-lower | $\mathbf{55.95}$ | $46.58$ | $50.76$
AR∗ |  COW-lower | $53.96$ | $45.40$ | $49.55$
SF |  Leipzig-lower | $47.68$ | $43.94$ | $46.43$
SF |  Leipzig-lemma | $45.77$ | $39.82$ | $46.67$
SF |  COW-lower | $52.58$ | $\mathbf{47.04}$ | $48.81$
SF |  COW-lemma | $46.79$ | $40.91$ | $49.85$
Table 4:  WCA F1-Scores for Wiktionary in percent. Our maximas are underlined,
absolute best in bold. ∗Only definition and relational feature vectors are
used.

##  6 Conclusion

We introduced SenseFitting, an extension of semantic specialization algorithms
based on Attract-Repel. SenseFitting performs a semantic specialization of the
centroids of word embeddings used to represent senses by means of sense level
constraints. We evaluated our sense-specialized embeddings using a newly
created resource called SimSense and showed that SenseFitting improves sense
representation by a large margin. Then, we applied our sense embeddings to
Word Sense Disambiguation (WSD) using a simple disambiguation algorithm. Our
sense embedding-based method outperforms competitors based on word embeddings,
specialized word embeddings, the first sense baseline, as well as previous
results obtained for the WebCAGe corpus. To sum up, our results show: a)
SenseFitting generates embeddings that allow semantic similarities on the
sense level to be reconstructed in a valid manner; b) recognizing these
similarities leads to an increase in performance in downstream-tasks such as
WSD.

Future work will examine whether the results obtained for German can be
transferred to other languages. Further work will evaluate whether sense
embeddings obtained by SenseFitting can increase performance while using other
embedding based WSD methods. We believe that high-quality representations of
senses and their similarities can enhance performance in tasks other than WSD.
This will also be tested in future work. In addition, we plan to apply
SenseFitting to pre-trained sense embeddings, such as FastSense (Uslu et al.,
2018) to evaluate whether we can improve their performance as well.

## References

  * Ahmed and Mehler (2018) Sajawel Ahmed and Alexander Mehler. 2018.  Resource-Size matters: Improving Neural Named Entity Recognition with Optimized Large Corpora.  In _Proceedings of the 17th IEEE International Conference on Machine Learning and Applications (ICMLA)_.
  * Callison-Burch et al. (2010) Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar F Zaidan. 2010.  Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation.  In _Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR_ , pages 17–53. Association for Computational Linguistics.
  * Chen et al. (2014) Xinxiong Chen, Zhiyuan Liu, and Maosong Sun. 2014.  A Unified Model for Word Sense Representation and Disambiguation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1025–1035. Association for Computational Linguistics.
  * Edmonds and Cotton (2001) Philip Edmonds and Scott Cotton. 2001.  SENSEVAL-2: Overview.  In _The Proceedings of the Second International Workshop on Evaluating Word Sense Disambiguation Systems_ , SENSEVAL ’01. Association for Computational Linguistics.
  * Faruqui et al. (2015) Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015.  Retrofitting Word Vectors to Semantic Lexicons.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1606–1615. Association for Computational Linguistics.
  * Gleim et al. (2018) Rüdiger Gleim, Alexander Mehler, and Sung Y. Song. 2018.  Wikidragon: A java framework for diachronic content and network analysis of mediawikis.  In _Proceedings of the 11th edition of the Language Resources and Evaluation Conference, May 7 - 12_ , LREC 2018, Miyazaki, Japan.
  * Goldhahn et al. (2012) Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff. 2012.  Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages.  In _LREC_.
  * Hamp and Feldweg (1997) Birgit Hamp and Helmut Feldweg. 1997.  Germanet - a lexical-semantic net for german.  In _Automatic Information Extraction and Building of Lexical Semantic Resources for NLP Applications_.
  * Henrich (2015) Verena Henrich. 2015.  _Word Sense Disambiguation with GermaNet - Semi-Automatic Enhancement and Empirical Results_.  Ph.D. thesis, Eberhard Karls Universität, Tübingen.
  * Henrich et al. (2011) Verena Henrich, Erhard Hinrichs, and Tatiana Vodolazova. 2011.  Semi-automatic extension of germanet with sense definitions from wiktionary.  In _Proceedings of the 5th Language and Technology Conference (LTC 2011)_ , pages 126–130.
  * Henrich et al. (2012) Verena Henrich, Erhard Hinrichs, and Tatiana Vodolazova. 2012.  WebCAGe – A Web-Harvested Corpus Annotated with GermaNet Senses.  In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_. Association for Computational Linguistics.
  * Hill et al. (2015) Felix Hill, Roi Reichart, and Anna Korhonen. 2015.  SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation.  _Computational Linguistics_ , 41(4):665–695.
  * Iacobacci et al. (2015) Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2015.  Sensembed: Learning sense embeddings for word and relational similarity.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 95–105, Beijing, China. Association for Computational Linguistics.
  * Iacobacci et al. (2016) Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2016.  Embeddings for Word Sense Disambiguation: An Evaluation Study.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Berlin, Germany. Association for Computational Linguistics.
  * Kunze and Lemnitzer (2002) Claudia Kunze and Lothat Lemnitzer. 2002.  GermaNet – representation, visualization, application.  In _Proceedings of the 3rd International Conference on Language Resources and Evaluation (LREC 2002)_ , pages 1485–1491, Paris. European Language Resources Association.
  * Ling et al. (2015) Wang Ling, Chris Dyer, Alan W Black, and Isabel Trancoso. 2015.  Two/Too Simple Adaptations of Word2Vec for Syntax Problems.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1299–1304, Denver, Colorado. Association for Computational Linguistics.
  * Matuschek and Gurevych (2013) Michael Matuschek and Iryna Gurevych. 2013.  Dijkstra-WSA: A Graph-Based Approach to Word Sense Alignment.  _Transactions of the Association of Computational Linguistics_.
  * Mehler et al. (2018) Alexander Mehler, Rüdiger Gleim, Wahed Hemati, and Tolga Uslu. 2018.  Skalenfreie online-soziale Lexika am Beispiel von Wiktionary.  In Stefan Engelberg, Henning Lobin, Kathrin Steyer, and Sascha Wolfer, editors, _Wortschätze_ , pages 269–292. De Gruyter, Berlin, Boston.
  * Meyer and Gurevych (2012) Christian M. Meyer and Iryna Gurevych. 2012.  To Exhibit is not to Loiter: A Multilingual, Sense-Disambiguated Wiktionary for Measuring Verb Similarity.  In _Proceedings of COLING 2012_. The COLING 2012 Organizing Committee.
  * Mihalcea et al. (2004) Rada Mihalcea, Timothy Chklovski, and Adam Kilgarriff. 2004.  The Senseval-3 English lexical sample task.  In _Senseval-3: Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text_ , Barcelona, Spain. Association for Computational Linguistics.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, G.s Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  _Proceedings of Workshop at ICLR_ , 2013.
  * Moro and Navigli (2015) Andrea Moro and Roberto Navigli. 2015.  Semeval-2015 task 13: Multilingual all-words sense disambiguation and entity linking.  In _Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015)_ , pages 288–297.
  * Mrkšić et al. (2016) Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašić, Lina Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016.  Counter-fitting Word Vectors to Linguistic Constraints.  In _Proceedings of HLT-NAACL_.
  * Mrkšić et al. (2017) Nikola Mrkšić, Ivan Vulić, Diarmuid Ó Séaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, and Steve Young. 2017.  Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints.  _Transactions of the Association for Computational Linguistics_ , 5:309–324.
  * Oele and van Noord (2018) Dieke Oele and Gertjan van Noord. 2018.  Simple Embedding-Based Word Sense Disambiguation.  _Global Wordnet Conference 2018_ , page 7.
  * Pradhan et al. (2007) Sameer S. Pradhan, Edward Loper, Dmitriy Dligach, and Martha Palmer. 2007.  SemEval-2007 Task 17: English Lexical Sample, SRL and All Words.  In _Proceedings of the 4th International Workshop on Semantic Evaluations_ , SemEval ’07, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Rothe and Schütze (2015) Sascha Rothe and Hinrich Schütze. 2015.  Autoextend: Extending word embeddings to embeddings for synsets and lexemes.  In _Proceedings of the ACL_.
  * Schäfer (2015) Roland Schäfer. 2015.  Processing and querying large web corpora with the COW14 architecture.  In _Proceedings of Challenges in the Management of Large Corpora 3 (CMLC-3)_ , Lancaster. UCREL, IDS.
  * Uslu et al. (2018) Tolga Uslu, Alexander Mehler, Daniel Baumartz, Alexander Henlein, and Wahed Hemati. 2018.  fastsense: An efficient word sense disambiguation classifier.  In _Proceedings of the 11th edition of the Language Resources and Evaluation Conference, May 7 - 12_ , LREC 2018, Miyazaki, Japan.
  * Wieting et al. (2015) John Wieting, Mohit Bansal, Kevin Gimpel, Karen Livescu, and Dan Roth. 2015.  From Paraphrase Database to Compositional Paraphrase Model and Back.  _Transactions of the Association for Computational Linguistics_.
  * Zhong and Ng (2010) Zhi Zhong and Hwee Tou Ng. 2010.  It Makes Sense: A Wide-coverage Word Sense Disambiguation System for Free Text.  In _Proceedings of the ACL 2010 System Demonstrations_ , ACLDemos ’10. Association for Computational Linguistics.

##  Appendix A Supplemental Material

###  A.1 Prepraring SenseFitting

####  A.1.1 Preprocessing Wiktionary

We are able to disambiguate 164 154 of 220 927 relations, leaving 56 773
ambiguous sense-to-lemma relations. Out of the disambiguated ones, there were
$110\,584$ sense-to-lemma relations where the target lemma was monosemous,
leaving $110\,343$ ambiguous relations of $220\,927$ relations in total. To
disambiguate the remaining $110\,3435$ ambiguous relations111111Counting only
antonyms, hyponyms, hypernyms and synonyms., we utilize SenseFitting itself.
We initialize a minimalist sense embedding by creating a copy of a lemma
embedding for each of its senses and run SenseFitting for one epoch with
sense-to-lemma relations, where a single epoch means a full run of Attract-
Repel. Using the resulting embedding, we try to disambiguate the target side
of such a relation by selecting the sense with the lowest (or highest in the
case of antonyms) cosine distance to the target sense. We repeat this process
with these partially disambiguated relations for up to 10 epochs, updating the
relations each time. During our experimentation, we noticed a significant
performance decrease when disambiguating antonyms together with the other
sense relations. Thus, we run the disambiguation in two batches: one for
hyponyms, hypernyms and synonyms, and one for antonyms. Finally, we can
proceed with SenseFitting using the disambiguated relations.

####  A.1.2 Wiktionary Relation Disambiguation

To evaluate the results of this automatic disambiguation, we used the sense-
annotated Wiktionary created by Meyer and Gurevych (2012). The authors created
four datasets of different language combinations to evaluate the
disambiguation performance of Wiktionary relations. The German dataset
consists of $1\,119$ annotated relations of which $514$ are correct according
to manual annotations. Of these, we are able to use $486$ for evaluation.

To determine whether the performance of disambiguating relations during pre-
processing is influenced by the structure of the used resource, we repeated
the process described in sec:supplement-sense-fitting-pre-wkt for GermaNet. We
removed target sense information from all relations and ran SenseFitting for
10 epochs, which sums to a total of 50 Attract-Repel iterations.

Dataset | Relation type | P | R | F1
---|---|---|---|---
GermaNet | Synonyms | 99 | 90 | 94
Hypernyms | 94 | 85 | 90
Hyponyms | 96 | 90 | 93
Antonyms | 100 | 89 | 94
All | 96 | 88 | 92
Wiktionary lower | Synonyms | 71 | 34 | 46
Hypernyms | 71 | 39 | 50
Hyponyms | 71 | 54 | 62
Antonyms | 70 | 46 | 56
All | 71 | 43 | 54
Wiktionary lemma | Synonyms | 65 | 34 | 45
Hypernyms | 73 | 38 | 50
Hyponyms | 72 | 57 | 63
Antonyms | 58 | 71 | 48
All | 68 | 49 | 52
Table 5:  F1\- and precision (P) scores for relation disambiguation of
SenseFitting in percent.

tab:SenseFitting-disambig shows the F1-scores of disambiguating these
relations. The GermaNet disambiguation reaches a score of more than $90\%$ for
all relation types with a precision of $94-100\%$. We assume that this high
performance is due to GermaNet’s structure: all relations are
symmetric121212Each hyponym relation has a hypernym relation in reverse
direction, so we consider it to be symmetric, too., and relations between
synsets essentially create fully connected graphs of senses that all get moved
towards their center.

Wiktionary does not show the graph structure of GermaNet; nor is its
development controlled by a group of experts, but by a heterogeneous online
community based on the wiki principle. Mehler et al. (2018) show that this
community consists of a small number of highly active authors that contribute
to a broader set of topics and a large number of less active authors who
contribute to a narrower set of topics. This imbalance suggests that
Wiktionary’s thematic orientation may be distorted by these highly active
authors. As such, Wiktionary is likely not as consistent and stable as a
resource created by a group of experts.

The performance of Wiktionary’s relation disambiguation on the dataset from
Meyer and Gurevych (2012) confirms this hypothesis. Although precision remains
high around 70%, F1-scores drop about half, because recall is rather low.
Synonyms show the most significant difference to results obtained for
GermaNet. We assume this to be caused by the lack of symmetry in Wiktionary
relations which is present in about 20% of synonymy relations131313For
Wiktionary sense-to-lemma relations, we consider a relation to be symmetric if
$\\{(S_{a},L_{b}),(S_{a},L_{b})\\}\subseteq R_{r}$, where $S_{X}$ is a sense
of lemma $L_{X}$ and $R_{r}$ is the set of all relations of the given type
$r$..

These results show that our semantic specialization can produce viable results
for relation disambiguation if the used resource is well structured and
features dense relations.

####  A.1.3 SenseFitting Training Resources

tab:sense-inv-stats shows the relation count for each resource in all
variations. The before mentioned graph structure of GermaNet can be seen in
the count of hyponyms and hypernyms: they are equal as both relations are
symmetric. This symmetry helps drawing hypernym-hyponym pairs together, most
likely increasing SenseFittings positive effect on injecting semantic
information into sense embeddings.

Dataset | Syn | Hypo | Hyper | Ant
---|---|---|---|---
GN | $103\,554$ | $314\,058$ | $314\,058$ | $3\,486$
GN+ | $95\,740$ | $294\,872$ | $294\,872$ | $3\,426$
WKT | $93\,800$ | $182\,224$ | $124\,724$ | $82\,675$
WKT+ | $50\,335$ | $114\,762$ | $85\,349$ | $51\,173$
lower∗ | $31\,352$ | $46\,053$ | $58\,002$ | $29\,314$
lemma∗ | $29\,467$ | $43\,788$ | $53\,322$ | $28\,393$
Table 6:  The number of relations used in SenseFitting based on COW
embeddings. +: total number of applicable relations used in training; ∗:
number of automatically disambiguated relations for Wiktionary.

###  A.2 Preprocessing WebCAGe

Task | Nouns | Verbs | Adjectives | Total
---|---|---|---|---
WCA | 2 150 | 1 639 | 336 | 4 125
Senseval-2 | 1 740 | 1 806 | 375 | 3 921
Senseval-3 | 1 807 | 1 978 | 159 | 3 944
SemEval-07 | 2 559 | 2 292 | — | 4 851
Table 7:  Number of sentences per part of speech in the WCA dataset in
comparison to English datasets used for performing WSD in Iacobacci et al.
(2016).

As mentioned in sec:supplement-wca, WebCAGe is a corpus mainly composed of
examples from word sense definitions from the German Wiktionary which were
automatically annotated with GermaNet senses and hand-corrected afterward
(Henrich et al., 2011). However, the authors do not provide a mapping to
Wiktionary; only GermaNet senses are tagged with an ID. Thus we needed to map
WebCAGe back to Wiktionary to be able to evaluate the performance of our WSD
method with both Wiktionary and GermaNet. We were able to match a large
portion of all sentences from WebCAGe to a recent German Wiktionary export
from April 2018. From this portion, we excluded sentences that did not include
the target word itself (e.g. if a target word was only contained in the
sentence as part of a word formation or compound)141414For example, an
excluded instance of the target word Bank is “Bankleitzahl” (English: bank
code), a compound of the words Bank and Leitzahl.. The size of WCA is similar
to the size of datasets available for WSD tasks in English. tab:WCA-stat
compares WCA against task-related datasets used to evaluate the method of
Iacobacci et al. (2016), namely Senseval-2 (Edmonds and Cotton, 2001),
Senseval-3 (Mihalcea et al., 2004) and SemEval-07 (Pradhan et al., 2007). Note
that WCA differs from the English datasets in the number of sentences per
lemma: It has on average only 1.89 sentences per lemma, while Senseval-2 has
59.3. This might affect a WSD performance evaluation, as the methods
performance is not evaluated as consistently across many samples for the same
target word, but rather sparsely across a large variety of target words. While
WebCAGe (and WCA alike) might cover many different words, it does not cover
all possible senses for a given target word.


††thanks: Authors are listed alphabetically. Both authors contributed equally
to this work.††thanks: Authors are listed alphabetically. Both authors
contributed equally to this work.

#  Paperfetcher: A tool to automate handsearch for systematic reviews

Akash Pallath  School of Engineering and Applied Science, University of
Pennsylvania, Philadelphia, PA - 19104
<EMAIL_ADDRESS>Qiyang Zhang  School of Education, Johns Hopkins
University, Baltimore, MD - 21218
<EMAIL_ADDRESS>
###### Abstract

Handsearch is an important technique that contributes to thorough literature
search in systematic reviews. Traditional handsearch requires reviewers to
systematically browse through each issue of a curated list of field-specific
journals and conference proceedings to find articles relevant to their review.
This manual process is not only time-consuming, laborious, costly, and error-
prone, but it also lacks replicability and cross-checking mechanisms. In an
attempt to solve these problems, this paper presents a free and open-source
Python package and an accompanying web-app, Paperfetcher, to automate
handsearch for systematic reviews. With Paperfetcher’s assistance, researchers
can retrieve articles from designated journals within a specified time frame
with just a few clicks. In addition to handsearch, this tool also incorporates
snowballing in both directions. Paperfetcher allows researchers to download
retrieved studies as a list of DOIs or as an RIS database to facilitate
seamless import into citation management and systematic review screening
software. To our knowledge, Paperfetcher is the first tool that automates
handsearch with high usability and a multi-disciplinary focus.

Systematic reviews, literature search, handsearch, information retrieval,
snowballing, web-app.

##  I Introduction

Literature search is an early and crucial step for performing a comprehensive
systematic review [1, 2, 3]. The process of literature search often begins
with a field-related bibliographic database search, which retrieves studies
from electronic records that index journals and non-journal sources [4].
Limiting literature search to bibliographic search risks missing high quality
papers [3, 5] that were not indexed with identifiable terms [6], formatted as
abstracts or letters [2, 3], located in supplement editions of journals [2],
or not included in electronic databases [7]. Handsearch is an additional step
that reviewers often undertake to identify such studies, which involves
systematically browsing through the tables of contents from a curated list of
field-specific journals, abstracts, and conference proceedings in order to
gather papers relevant to the synthesis topic [4]. To ensure comprehensive
information retrieval and transparent reporting, it is important to conduct
literature search in a systematic and exhaustive manner [8]. Combining
handsearch with bibliographic database search decreases the likelihood of
missing major relevant studies and therefore underpins a solid foundation for
the systematic review [9].

Despite its importance, the prevailing practice of handsearch awaits urgent
development. Advancing handsearch strategy is also timely due to the rapid
increase in studies available online and the subsequent increase in the amount
of time required to retrieve papers. Traditionally, handsearch requires
reviewers or hired volunteers to manually read through the tables of contents
and abstracts of tens to hundreds of journal issues, supplementary materials,
and conference proceedings [4, 8]. Past literature agrees on the time and
labor-intensive nature of handsearch, reporting time spent on this task
ranging from an hour per journal volume [10] to 185 hours spent for 10
journals [11]. This procedure is not only time-consuming, laborious [4], and
costly [10], but more importantly, it is error-prone due to human fatigue
[12], lacks replicability and an easy cross-checking mechanism due to its
cumbersome manual nature.

To address these problems, we developed a freely available Python package and
an accompanying web-app – Paperfetcher – that automates handsearch to increase
efficiency and ensure replicability. Paperfetcher automatically fetches works
from a user-selected list of journals within a user-specified timeframe.
Paperfetcher not only returns article metadata, but it also generates a report
of the parameters used to perform the handsearch, which can assist with
replicability. Automation of handsearch facilitates less error-prone
systematic review, and more importantly, enables researchers to focus their
energy on the screening process rather than the search process. Paperfetcher
addresses the urgent need for a more replicable, robust, and time-efficient
method to conduct handsearch.

In addition to handsearch, Paperfetcher also includes a snowballing function
for forward and backward citation chasing [13, 14, 15, 16]. Forward citation
chasing searches for articles that cite a given article of interest while
backward citation chasing searches for articles cited by the article of
interest. Snowballing, which is an umbrella term for citation chasing in both
directions, is a very useful supplementary search strategy [5]. Researchers
have found that it identifies 51% of studies in systematic reviews [16]. We
implemented snowballing in Paperfetcher due to its important contribution to
literature search.

While there are several existing tools to automate snowballing (further
discussed in Section IV), to the best of our knowledge, there is no other tool
in the current market that provides an easy-to-use interface to automate
handsearch.

##  II How Paperfetcher works

Paperfetcher consists of a) a Python package which implements the handsearch
and snowballing algorithms to retrieve raw data and convert it to various
output formats (such as text and Research Information Systems (RIS) format),
and b) a web-app that provides an easy to use graphical interface for the
package. In the following subsections, we describe how the Python package and
the web app work.

Figure 1: Handsearch user-interface in the Paperfetcher web-app. Figure 2:
Snowballing user-interface in the Paperfetcher web-app.

###  II.1 Data sources

Paperfetcher’s handsearch algorithm queries the database of academic content
registered with Crossref. Crossref is an official digital object identifier
(DOI) registration agency of the International DOI Foundation [17]. Research
suggests that Crossref is the most robust and holistic implementation of the
DOI model [17], and covers a wide range of disciplines. Among the 11 official
DOI registration agencies, only Crossref, DataCite, and Multilingual European
DOI Registration Agency (mEDRA) cover English materials related to scholarly
and professional research content [18]. We excluded DataCite as it only
indexes research data and not research articles or reports [18]. As mutual
collaboration between mEDRA and Crossref allows DOIs registered with mEDRA to
be deposited on the Crossref platform [19], we excluded mEDRA to avoid
redundancy. Since Paperfetcher is primarily designed for English-reading
researchers, we excluded DOI agencies that contain materials of other non-
English languages, such as China National Knowledge Infrastructure [20], or
which index non-academic content, such as the Entertainment Identifier
Registry [21]. Other non-English DOI agencies can be added in future versions
of Paperfetcher once demand for multilingual research content arises.

Statistics updated on Dec 31 2021 show that Crossref covers more than 1.3
million published and unpublished content types [22], such as journals, books,
conference proceedings, dissertations, working papers, technical reports, and
data sets. It provides more than 90,000 records of journals and more than
80,000 records of conference proceedings to empower an effective handsearch
[22]. A comparison of different databases, including Scopus, Web of Science,
Dimensions, Crossref, and Microsoft Academic, concluded that Crossref is a
bibliographic data source that is of significant interest for bibliometric
analyses and is becoming increasingly valuable over the years [23]. As
compared to the widely-used Scopus database, Crossref covers a larger number
of documents that have been published in journals [23].

Paperfetcher’s snowballing algorithm queries both the Crossref database and
COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations. COCI is a
database derived from Crossref, which contains more than 1.2 billion DOI-to-
DOI citation pairs of Crossref-deposited articles with open citations [24,
25]. The COCI database is updated periodically to add more citations. As of
January 2, 2022, the last update to the database was on November 25, 2021
[24].

###  II.2 Handsearch

Each handsearch has three inputs: the International Standard Serial Number
(ISSN) of the journal to fetch articles from, a date range within which to
fetch articles, and an optional list of keywords to refine the search. We do
not recommend using keywords when performing handsearch for systematic reviews
as there is a possibility of missing relevant studies which were not indexed
in Crossref with matching keywords. However, this input can still be useful
for performing literature searches for specific purposes, or if the reviewers
are time-constrained and have to filter through a large number of matching
articles.

Paperfetcher queries the Crossref database through its Representational State
Transfer Application Programming Interface (REST API) with these three inputs
to retrieve metadata – containing fields such as article title, authors,
journal, publisher, publishing date, abstract, keywords, etc. – of all
articles from the journal with the specified ISSN, within the selected date
range, and matching the list of keywords (if specified). The Paperfetcher web
app can iterate through a list of journals, perform handsearch for each of
them, and merge the retrieved metadata into a single dataset of search
results.

###  II.3 Snowballing

Paperfetcher requires a list of DOIs as input for both forward and backward
snowballing. For backward snowballing, it queries the Crossref REST API to
retrieve DOIs of papers cited by the input DOIs. The Crossref REST API only
returns information about articles cited by a given paper, and not articles
citing it. The COCI REST API, however, can return this data, therefore
Paperfetcher uses COCI for forward snowballing. In both cases, Paperfetcher
compiles the union of the retrieved DOIs into a dataset of search results.

###  II.4 Data export

The Paperfetcher Python package can convert the search results into several
formats, such as a text file of DOIs, a CSV file or Excel spreadsheet with
rows containing article title, authors, journal, DOI, URL, etc. for each
article, RIS databases, and even pandas DataFrames [26, 27] for further data
analysis and processing in Python or R. The Paperfetcher web-app implements
two of these options – it can either export results to an RIS file or to a
text file of DOIs. Paperfetcher uses Crossref’s content negotiation service
[28] to retrieve metadata in the RIS format.

Figure 3: Search results and report in the Paperfetcher web-app.

##  III Using Paperfetcher

###  III.1 Using the web-app online

Paperfetcher’s web-app, based on the Streamlit app framework, is available
online at https://share.streamlit.io/paperfetcher/paperfetcher-web-
app/main/paperfetcher_app.py. The web-app can also be run offline, as
described in Section III.4. The highlight of this web app is that it has an
easy-to-use graphical interface and requires no programming experience. To
perform handsearch (Figure 1), users just need to fill out a few search
parameters – journal names or journal ISSNs, a date range to fetch articles
within, optional search keywords, output format – and click on the ‘Search’
button. To perform snowballing (Figure 2), users only need to enter comma-
separated DOIs of selected papers, select the type of snowballing (forward or
backward), select output format, and click on the ‘Search’ button.

###  III.2 Exporting data

After performing a search, Paperfetcher gives users the option to download
their data either as a text file of DOIs or as an RIS file, based on the
output format selection they made (Figure 3).

RIS is a standardized format that allows data exchange among different
citation management software [29]. RIS data exported from Paperfetcher can be
seamlessly imported into citation management software such as EndNote,
Mendeley, Paperpile and Zotero, and into Systematic Review Screening tools
such as ASReview [30], Covidence, DistillerSR, and EPPI-Reviewer [31] (for a
comprehensive review of screening tools, see Zhang and Neitzel [32]). DOIs
from exported text data can be bulk-imported into citation management tools
such as Zotero in order to fetch missing abstracts or other important
metadata.

###  III.3 Reporting handsearch parameters

Paperfetcher generates a report of the parameters used to perform the search
and the number of papers fetched, which is displayed to users after performing
the search. Figure 3 shows a sample report for snowballing. These reports
provide a mechanism for reviewers to document their search for cross-checking
and reproducibility.

###  III.4 Using the web-app offline

Privacy-conscious users may opt to use the Paperfetcher web-app offline. To do
so, they must first install and then run the app from source. The source code
and installation instructions for the app are available online at
https://github.com/paperfetcher/paperfetcher-web-app.

###  III.5 Using the Python package

Python programmers might prefer to directly use Paperfetcher’s Python package
as it can export search results to more data formats than the app. Users who
wish to set up data pipelines or perform additional data analysis in Python or
R will find the package’s ability to export data to pandas DataFrames
particularly useful. Such users can install the package directly from the
Python Package Index (https://pypi.org/project/paperfetcher). Developers who
are interested in modifying Paperfetcher’s source code can clone the source
repository from GitHub at https://github.com/paperfetcher/paperfetcher.
Documentation for the Python package can be found at
https://paperfetcher.github.io/paperfetcher.

##  IV Discussion

There are a few existing tools that automate snowballing. Some tools only
enable uni-directional citation chasing. For example, Sci-Finder supports
backward reference chasing, but it is not free of charge and only focuses on
literature related to chemicals, drugs, and substances [33]. Some databases,
including Web of Science, Scopus, and Google Scholar, enable forward citation
chasing. We only found two tools that support citation chasing in both
directions: SpiderCite and citationchaser. SpiderCite is part of the free
online suite of tools SR-Accelerator, and allows users to import and export
references in EndNote XML, RIS, and BibTeX formats [34]. Citationchaser is an
R package that is free of charge, open source, and provides an easy-to-use
Shiny app interface, where users can input DOIs of articles to search and
export data in RIS format [35]. However, to date, all the above tools lack
functions to automate handsearch. Paperfetcher serves to complement these
tools to help reviewers identify as many relevant studies as possible.

Although Paperfetcher has the potential to make significant contributions to
the field of systematic review, the current version is limited in a few ways.
First, there can be a lag between when a paper is first published online and
when its DOI is deposited in Crossref. In such a case, Paperfetcher will fail
to find the paper. Second, certain publishers do not make their abstracts and
references publicly available on Crossref. In the former case, reviewers can
use citation management software to retrieve missing abstracts. The latter
case, however, is more problematic as this will result in relevant studies
being missed by Paperfetcher’s snowballing function [23]. In addition, as
updates to COCI lag behind updates to Crossref, Paperfetcher’s forward
citation chasing function might miss newly published citing articles.
Citationchaser, which queries the Lens.org database (consisting of articles
from PubMed, PubMed Central, CrossRef, Microsoft Academic Graph and CORE), has
access to more reference information. As the Lens.org API requires a
subscription fee, and Paperfetcher was developed without any funding, we had
no choice but to exclude paid APIs. In the future, should we have access to
funding or resources, we can incorporate additional APIs (such as Lens.org or
Web of Science) to improve Paperfetcher’s snowballing function.

In a nutshell, Paperfetcher provides a free, easy-to-use, and efficient method
to automate handsearch. We believe that it can significantly reduce the time
reviewers spend on literature search and can improve transparency in synthesis
reporting across disciplines.

## References

  * coo [2018] Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies, 18, 85 (2018).
  * Armstrong  _et al._ [2005] R. Armstrong, N. Jackson, J. Doyle, E. Waters, and F. Howes, It’s in your hands: the value of handsearching in conducting systematic reviews of public health interventions, Journal of Public Health 27 (2005).
  * Richards [2008] D. Richards, Handsearching still a valuable element of the systematic review, Evidence-Based Dentistry 9 (2008).
  * Kugley  _et al._ [2017] S. Kugley, A. Wade, J. Thomas, Q. Mahood, A. K. Jorgensen, K. Hammerstrom, and N. Sathe, Searching for studies: a guide to information retrieval for Campbell systematic reviews, Campbell Systematic Reviews 13 (2017).
  * Papaioannou  _et al._ [2010] D. Papaioannou, A. Sutton, C. Carroll, A. Booth, and R. Wong, Literature searching for social science systematic reviews: consideration of a range of search techniques, Health Information & Libraries Journal 27 (2010).
  * Dickersin  _et al._ [1994] K. Dickersin, R. Scherer, and C. Lefebvre, Systematic Reviews: Identifying relevant studies for systematic reviews, BMJ 309 (1994).
  * Blümle and Antes [2008] A. Blümle and G. Antes, Handsearching for randomized controlled clinical trials in German medical journals, Deutsche Medizinische Wochenschrift 133 (2008).
  * Higgins and Green [2008] J. P. Higgins and S. Green,  _Cochrane handbook of systematic reviews of interventions._, edited by J. P. Higgins and S. Green (John Wiley & Sons, Ltd, 2008).
  * Hopewell  _et al._ [2007] S. Hopewell, M. J. Clarke, C. Lefebvre, and R. W. Scherer, Handsearching versus electronic searching to identify reports of randomized trials, Cochrane Database of Systematic Reviews (2007).
  * Moher and Group [1995] D. Moher and T. O. C. Group, How good are volunteers at searching for published randomized controlled trials?, Fundamental & Clinical Pharmacology 9 (1995).
  * Glanville  _et al._ [2012] J. Glanville, M. Cikalo, F. Crawford, M. Dozier, and H. McIntosh, Handsearching did not yield additional unique FDG-PET diagnostic test accuracy studies compared with electronic searches: a preliminary investigation, Research Synthesis Methods 3 (2012).
  * Adams  _et al._ [1994] C. E. Adams, A. Power, K. Frederick, and C. Lefebvre, An investigation of the adequacy of MEDLINE searches for randomized controlled trials (RCTs) of the effects of mental health care, Psychological Medicine 24 (1994).
  * Cooper  _et al._ [2017] C. Cooper, A. Booth, N. Britten, and R. Garside, A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review, Systematic Reviews 6 (2017).
  * Lefebvre  _et al._ [2019] C. Lefebvre, J. Glanville, S. Briscoe, A. Littlewood, C. Marshall, M.-I. Metzendorf, A. Noel-Storr, T. Rader, F. Shokraneh, J. Thomas, L. S. Wieland, and o. b. o. t. C. I. R. M. Group, Searching for and selecting studies, in  _Cochrane Handbook for Systematic Reviews of Interventions_ (John Wiley & Sons, Ltd, 2019) pp. 67–107.
  * Jalali and Wohlin [2012] S. Jalali and C. Wohlin, Systematic literature studies: Database searches vs. backward snowballing, in  _Proceedings of the 2012 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement_ (2012) pp. 29–38.
  * Greenhalgh and Peacock [2005] T. Greenhalgh and R. Peacock, Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources, BMJ (Clinical research ed.) 331 (2005).
  * Howells [2006] M. Howells, CrossRef: an overview, Editors’ Bulletin 2 (2006).
  * International DOI Foundation [2021] International DOI Foundation, DOI registration agencies - areas of coverage (2021), accessed 2021-12-31.
  * mEDRA [2021] mEDRA, mEDRA - who we are (2021), accessed 2021-12-31.
  * China Academic Journals Electronic Publishing House Co. Ltd [2021] China Academic Journals Electronic Publishing House Co. Ltd, CNKI (2021), accessed 2021-12-31.
  * EIDR [2021] EIDR, EIDR – a universal unique identifier for movie and television assets (2021), accessed 2021-12-31.
  * cro [2021] Crossref.org : Crossref stats (2021), accessed 2022-01-01.
  * Visser  _et al._ [2021] M. Visser, N. J. van Eck, and L. Waltman, Large-scale comparison of bibliographic data sources: Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic, Quantitative Science Studies 2 (2021).
  * OpenCitations [2021] OpenCitations, COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations (2021).
  * Heibi  _et al._ [2019] I. Heibi, S. Peroni, and D. Shotton, Software review: COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations, Scientometrics 121 (2019).
  * Wes McKinney [2010] Wes McKinney, Data Structures for Statistical Computing in Python, in  _Proceedings of the 9th Python in Science Conference_, edited by Stéfan van der Walt and Jarrod Millman (2010) pp. 56 – 61.
  * The pandas development team [2020] The pandas development team, pandas-dev/pandas: Pandas (2020).
  * [28] DOI Content Negotiation, accessed 2022-01-01.
  * USAID [2018] USAID, What is the RIS Reference Format? (2018), accessed 2022-01-02.
  * van de Schoot  _et al._ [2021] R. van de Schoot, J. de Bruin, R. Schram, P. Zahedi, J. de Boer, F. Weijdema, B. Kramer, M. Huijts, M. Hoogerwerf, G. Ferdinands, A. Harkema, J. Willemsen, Y. Ma, Q. Fang, S. Hindriks, L. Tummers, and D. L. Oberski, An open source machine learning framework for efficient and transparent systematic reviews, Nature Machine Intelligence 3 (2021).
  * Thomas  _et al._ [2020] J. Thomas, S. Graziosi, J. Brunton, Z. Ghouze, P. O’Driscoll, and M. Bond, EPPI-Reviewer: advanced software for systematic reviews, maps and evidence synthesis. (2020).
  * Zhang and Neitzel [2021] Q. Zhang and A. J. Neitzel, Methodological review: A systematic narrative review of screening tools for conducting systematic reviews in educational research (2021).
  * Vogel [2021] T. Vogel, LibGuides: SciFinder: What Is SciFinder? (2021).
  * Institute for Evidence-Based Healthcare [2021] Institute for Evidence-Based Healthcare, SR-Accelerator (2021), accessed 2022-01-01.
  * Haddaway  _et al._ [2021] N. R. Haddaway, M. J. Grainger, and C. T. Gray, citationchaser: An R package and Shiny app for forward and backward citations chasing in academic searching. (2021).