# Biographical: A Semi-Supervised Relation Extraction Dataset

Alistair Plum  0000-0003-0977-3467 RGCL,
University of Wolverhampton, UK<EMAIL_ADDRESS>,  Tharindu Ranasinghe
0000-0003-3207-3821 RGCL,
University of Wolverhampton, UK<EMAIL_ADDRESS>,  Spencer Jones
0000-0001-6241-5971 History, Politics and War Studies,
University of Wolverhampton, UK<EMAIL_ADDRESS>,  Constantin Orăsan
0000-0003-2067-8890 Centre for Translation Studies,
University of Surrey, UK<EMAIL_ADDRESS>and  Ruslan Mitkov
0000-0003-2522-066X RGCL,
University of Wolverhampton, UK<EMAIL_ADDRESS>
(2022)

###### Abstract.

Extracting biographical information from online documents is a popular
research topic among the information extraction (IE) community. Various
natural language processing (NLP) techniques such as text classification, text
summarisation and relation extraction are commonly used to achieve this. Among
these techniques, RE is the most common since it can be directly used to build
biographical knowledge graphs. RE is usually framed as a supervised machine
learning (ML) problem, where ML models are trained on annotated datasets.
However, there are few annotated datasets for RE since the annotation process
can be costly and time-consuming. To address this, we developed Biographical,
the first semi-supervised dataset for RE. The dataset, which is aimed towards
digital humanities (DH) and historical research, is automatically compiled by
aligning sentences from Wikipedia articles with matching structured data from
sources including Pantheon and Wikidata. By exploiting the structure of
Wikipedia articles and robust named entity recognition (NER), we match
information with relatively high precision in order to compile annotated
relation pairs for ten different relations that are important in the DH
domain. Furthermore, we demonstrate the effectiveness of the dataset by
training a state-of-the-art neural model to classify relation pairs, and
evaluate it on a manually annotated gold standard set. Biographical is
primarily aimed at training neural models for RE within the domain of digital
humanities and history, but as we discuss at the end of this paper, it can be
useful for other purposes as well.

Biographical Information Extraction, Relation Extraction, Transformers

††copyright: acmcopyright††journalyear: 2022††copyright:
acmcopyright††conference: Proceedings of the 45th International ACM SIGIR
Conference on Research and Development in Information Retrieval; July 11–15,
2022; Madrid, Spain††booktitle: Proceedings of the 45th International ACM
SIGIR Conference on Research and Development in Information Retrieval (SIGIR
’22), July 11–15, 2022, Madrid, Spain††price: 15.00††doi:
10.1145/3477495.3531742††isbn: 978-1-4503-8732-3/22/07††ccs: Computing
methodologies Information extraction††ccs: Computing methodologies Language
resources

##  1\. Introduction

As web technology continues to thrive, documents containing biographical
information are continuously generated and published online in large numbers
(Nasar et al., 2021). These online documents contain essential facts or events
related to the life of well-known and lesser-known individuals, which can be
used to populate structured biographical databases (Wang et al., 2021;
Smirnova and Cudré-Mauroux, 2018). These databases are capable of supporting
many interesting studies in humanities, and related areas (Zhang et al., 2017)
as we describe in Section 5. However, manually extracting information from a
massive document collection is impossible, given the amount of information
available online. Therefore, NLP methods can be used to process these
documents automatically.

Previous studies have used many NLP techniques including text classification
(Palmero Aprosio and Tonelli, 2015; Hogue et al., 2014), named entity
recognition (NER) (Jiang, 2012) and summarisation (Zhou et al., 2004) to
perform biographical information extraction, which we describe thoroughly in
Section 2. However, a major weakness in these studies is that they can not be
used directly to populate a database. Instead, they need to be combined with
other NLP techniques to extract the structured information required for
databases. A different approach, which we employ in this study, is to design
biographical information extraction as a relation extraction (RE) task.

RE is the task of extracting semantic relationships between entities from a
document, which can in turn be used to populate a database with relational
facts contained in a piece of text. Consider the following two text pieces on
two different people.

Text 1: William Shakespeare was born and raised in Warwickshire. At the age of
18, he married Anne Hathaway, with whom he had three children: Susanna Hall
and twins Hamnet Shakespeare and Judith Quiney.

Text 2: Henry Baynton (23 September 1892 in Warwickshire – 2 January 1951 in
London) was a British Shakespearean actor of the early 20th century.

For the texts shown above, the RE model can extract triples, which can be
represented as edges in a knowledge graph, such as ¡William Shakespeare,
Spouse, Anne Hathaway¿. Table 1 shows some of the relationship triples that
can be extracted from the above two text pieces. Combining such triples, a
system can produce a knowledge graph of relational facts between persons,
occupations, and locations in the text. A knowledge graph derived from the
relationships in Table 1 is shown in Figure 1.

Object | Relation | Object
---|---|---
William Shakespeare | Birth Place | Warwickshire
William Shakespeare | Spouse | Anne Hathaway
William Shakespeare | Child | Susanna Hall
William Shakespeare | Child | Hamnet Shakespeare
William Shakespeare | Child | Judith Quiney
William Shakespeare | Occupation | Actor
William Shakespeare | Occupation | PlayWright
Henry Baynton | Occupation | Actor
Henry Baynton | Birth Place | Warwickshire

Table 1. Example Biographical Relationship Triples

Knowledge graphs are commonly used by companies to provide information to end-
users and understand relationships between various types of entities. Several
machine learning models including recurrent neural networks (RNN) (Gormley et
al., 2015; Xiao and Liu, 2016), convolutional neural networks (CNN) (Zeng et
al., 2014; Shen and Huang, 2016), graph neural networks (GNN) (Baldini Soares
et al., 2019; Xue et al., 2021) and transformers (Nayak and Ng, 2020; Joshi et
al., 2020) have been proposed to automatically extract relationships from
texts. These machine learning models use a supervised paradigm where the
models require a dataset similar to Table 1 to train. Therefore, the NLP
community has a growing interest in producing datasets capable of training
machine learning models to perform RE. Several datasets in this area, such as
NYT24 (Hoffmann et al., 2011), and TACRED (Zhang et al., 2017) have been
released for this purpose. However, all of these datasets are manually
annotated, which makes it difficult to expand RE to different genres and
languages. In this paper, we propose a novel approach for producing RE
datasets that is semi-supervised and can be expanded easily to other domains
and languages. As far as we know, an approach such as this has not yet been
proposed. We develop the first dataset of this kind and evaluate its
usefulness. If the approach does prove to be useful, it will significantly
reduce the burden on the manual annotation process, as well as language and
domain-specific expertise.

The main contributions of this paper are the following:

  1. (1)

We introduce Biographical; the first and the largest dataset for biographical
RE built in a semi-supervised manner with ten relationship categories. We also
produce a manually annotated subset that can be used for evaluation111The
dataset is available at https://plumaj.github.io/biographical/.

  2. (2)

We evaluate four machine learning models to perform biographical RE, based on
state-of-the-art transformer models such as BERT (Devlin et al., 2019).

  3. (3)

We provide important resources to the community: the dataset, the code, and
the pre-trained models are made available to everyone interested in working on
biographical RE using the same methodology.

The rest of the paper is structured as follows. Section 2 presents an overview
of related work. Section 3 describes the data compilation process involved in
this study. In Section 4 we explain the experiments carried out, as well as an
evaluation of the experiments. Finally, the paper outlines an intended future
study and provides conclusions.

Figure 1. Example Knowledge Graph

##  2\. Related Work

Extracting biographical information from documents is a popular research area
in the NLP community. Most of these studies use different NLP techniques on
open and free resources such as Wikipedia.

Text classification is one of the first NLP techniques used to extract
biographical information. Biadsy et al. (2008) used an unsupervised sentence
classification framework to extract biographies from Wikipedia articles. In
more recent work, Palmero Aprosio and Tonelli (2015) have trained various
machine learning classifiers to detect biographical sections in Wikipedia
texts using a supervised approach. In a different work, Hogue et al. (2014)
use Wikipedia page traffic data to determine sentences of importance in
Wikipedia articles.

Text summarisation is another popular NLP technique that has been used to
extract biographical information. Biadsy et al. (2008) use Wikipedia articles
together with the TDT4 news corpus222https://catalog.ldc.upenn.edu/LDC2005S11
to train an unsupervised multi-document summariser for biographical
information. They used a support vector machine model and achieved state-of-
the-art performance at the time on the DUC2004 dataset (Over and Yen, 2004).
The approach is based on the one proposed by Zhou et al. (2004) who similarly
used Wikipedia data to develop a system for summarisation using a Naive Bayes
architecture. Chisholm et al. (2017) combine Wikipedia text and Wikidata
information to generate one-sentence summaries from structured biographical
information. First, the approach identifies potential biographical candidates
from Wikidata, then learns to generate the short summaries by mapping
structured information to the first sentence of the matching article in
Wikipedia. Thus follows the mostly standardised pattern of the first sentence
of a Wikipedia article containing most of the relevant information about a
person.

However, none of these approaches can be used directly to create a knowledge
graph. Therefore, more recent work in biographical information extraction has
modelled the task as a RE problem. Several ML models have been developed to
perform RE. Early approaches for RE were based on traditional machine learning
models such as support vector machines (Liu et al., 2007), and decision trees
(Singhal et al., 2016). But with the introduction of word embeddings and the
success of neural network architectures in different areas, the NLP community
has used a wide range of neural network architectures for the RE task. Zeng et
al. (2014) have used a CNN architecture and a synonym dictionary to integrate
semantic knowledge into the neural network. In a different approach, Zeng et
al. (2014) use lexical features with the word embeddings (Turian et al., 2010)
fed into a CNN to perform RE. RNNs have also been popularly used in RE. Miwa
and Bansal (2016) utilised a Tree Long Short-Term Memory (LSTM) network to
perform RE. Zhou et al. (2016) used an attention-based bi-directional LSTM
network on the SemEval-2010 relation classification task (Hendrickx et al.,
2010) and show that it provides good results. The current state-of-the-art in
RE, also used for this research, is based on neural transformers (Baldini
Soares et al., 2019). These transformer models are trained using a language
modelling task such as masked language modelling or next sentence prediction
and then have been used to perform RE as a downstream NLP task. Results on
recent RE datasets show that transformers outperform the previous
architectures based on RNNs and CNNs (Xue et al., 2019; Baldini Soares et al.,
2019).

All the ML models for RE mentioned above follow a supervised paradigm where an
annotated dataset is required to train the ML model. The most common datasets
used for this are NYT24 (Hoffmann et al., 2011), NYT29 (Riedel et al., 2010)
and TACRED (Zhang et al., 2017). All these datasets have been created using
manual annotation. As we mentioned before, since the annotation process is
expensive, these datasets are limited in size. For example, TACRED, the
largest RE dataset, has only 106,264 instances. This can prove not enough to
train data-driven methods, especially those based on neural networks.
Furthermore, the manual annotation process limits the expansion of RE research
to different domains and languages. To address this problem, we propose a
semi-supervised approach to create RE datasets using a similar approach to
Chisholm et al. (2017) which we describe in the next section.

Figure 2. System Architecture

##  3\. Data Compilation

The data compilation process is divided into two steps. The first step
involves the selection of our data sources, which are one of the most
fundamental aspects of the approach (Section 3.1). Our approach requires a
source of textual data and a source of structured information that is related
to the textual data. The second step concerns the processing of the different
data sources, as well as matching operations that allow for the automatic
labelling process (Section 3.2). These steps lead to the final dataset
consisting of sentences, marked entities and their respective relation.

###  3.1. Data Sources

Our semi-supervised approach combines data from three different sources:
Wikipedia333https://www.wikipedia.org/,
Wikidata444https://www.wikidata.org/wiki/Wikidata:Main_Page and
Pantheon555https://pantheon.world/ (Yu et al., 2016). Wikipedia serves as the
main source of textual data, in the form of sentences taken from specific
articles. Pantheon and Wikidata serve as our sources of structured
information. We also use Pantheon to select our initial set of biographical
articles from Wikipedia. We target specific biographical articles in Wikipedia
that are confirmed by the Pantheon dataset. Next, we iterate over the
sentences of each article and tag the named entities, including locations and
dates, using spaCy666https://spacy.io/ and Stanford
CoreNLP777https://stanfordnlp.github.io/CoreNLP/. Finally, we augment the
structured data from the Pantheon dataset with information from Wikidata. This
expanded dataset is matched to sentences in Wikipedia, allowing us to label
each sentence according to the type of relation. We discuss each of the data
sources in more detail in the following sections.

####  3.1.1. Wikipedia

Wikipedia is a free, online encyclopedia that contains a large amount of
information about people, and as such, serves as the backbone of our approach.
It is a vast resource of textual data, that is linked to a number of different
projects that relay the contained information in a structured way. The next
steps in our approach will focus on connecting the structured data with the
textual data.

For processing Wikipedia textual data, we follow a previously established
workflow (Plum et al., 2019) which has proved to be efficient. We work with
Wikipedia database backup dumps, which are an exact copy of all Wikipedia
articles of a given language at a specific point in time. We use the
enwiki-20190420 dump, which corresponds to the content of English Wikipedia on
20th of April 2019. Once downloaded, we extract articles corresponding to the
entries in the Pantheon dataset, which is done via the Wikipedia IDs.
Extracting the text can be a complex task in itself, since the structure of
the XML file is not uniform, as well as including certain XML parts that have
to be expanded. Since the extraction of text from Wikipedia is not our main
goal and could warrant a separate project, we use an existing tool for the
extraction process. The
wikiextractor888https://github.com/attardi/wikiextractor package for Python
converts articles to plain text. We observed some extraction problems, such as
XML-tag artefacts, mismatched quotation marks, and incomplete or illegible
sentences, which we remove at the processing stage with regular expressions.

####  3.1.2. Pantheon

In order to determine which articles in Wikipedia are biographical, i.e.
containing information that pertains to a person, we use the Pantheon dataset
(Yu et al., 2016). According to its creators, _”Pantheon [is] focused on
biographies with a presence in 15 different languages in Wikipedia”_ and
consists of roughly 85,000 entries. While it was initially created mostly by
hand, its later iterations have used a classifier to determine and extract
further entries. One particular characteristic of this dataset is that each
article has to contain unambiguous links to the respective Wikipedia and
Wikidata pages. This allows us to identify which articles from Wikipedia
contain the relevant information. While this could be done just using
Wikidata, Pantheon has been (at least partly) manually verified. Because
Pantheon only includes persons whose articles are available in 15 different
languages, this ensures that a person is somewhat well-known, in turn making a
longer Wikipedia article more likely.

In addition, each entry includes basic information, which we match to
sentences from the corresponding Wikipedia articles. This mainly includes
information such as dates of birth and death, places of birth and death, and
main occupation. The included information allows us to label the birthdate,
deathdate, birthplace, deathplace and occupation relations, while also
allowing us to confirm the name of a person. As these relations are only half
of the relations we target, we use the included Wikidata ID to obtain the
other half of the relations (introduced next).

####  3.1.3. Wikidata

Wikidata is described as a _”free, collaborative, multilingual, secondary
database” that ”provides support for Wikipedia […]”_ (Vrandečić and Krötzsch,
2014). Wikidata ties in well with the two other sources of data that we use.
Since it provides most of the information from a Wikipedia page (and often
more) in a structured format, we use it to augment the Pantheon dataset. Since
the Pantheon dataset provides distinct identifiers for Wikipedia and Wikidata,
selecting the correct entity is a straight-forward task. Using the
corresponding entries, we add the educatedAt, ofParent, sibling and hasChild
relations, as well as other. In the case of the last relation, we use this to
categorise any relation that is not explicitly targeted here and make sure
that the information matched is not part of any of the nine other relations.

###  3.2. Automatic Labelling

The next step in the approach is the automatic labelling of sentences. Once we
have extracted the text of each Wikipedia article, we begin processing the
texts, using spaCy NER to tag persons, locations, organisations, dates, as
well as Stanford CoreNLP Entity information to tag occupations in each
article. It should be noted that we run spaCy at runtime, but we carried out
one full annotation run with Stanford CoreNLP on all articles, which we store
and subsequently only access. This is because we found Stanford CoreNLP too
slow for multiple runs.

Each sentence of an article is processed in order to determine whether it is
about the main person of the article. To accomplish this, the script matches
the name with the person tags in the sentence, and also allows some substring
matches, such as first and last name excluding any other titles, or last name
only. If a match is found, the sentence is regarded as containing some
information about that person. This is ensured because the sentence is taken
from that person’s article and it includes that person’s name.

After a positive match is made within a sentence, we check the other tagged
entities in the sentence (locations, organisations, dates and occupations)
against the information provided by the Pantheon dataset and respective
Wikidata entry. Each matched pair, for instance a name and a location, is then
marked with $<$eN$>$ (begin) and $<$/eN$>$ (end) tags, where N is either 1 or
2, depending on the position of the entity (i.e. first or last). This is
followed by the respective relation tag. The following text box shows an
example of this. We estimate that this approach could be extended to all
relations where it would be possible match the information in a sentence in
this way.

Text 1: ¡e1¿William Shakespeare¡/e1¿ was born and raised in
¡e2¿Warwickshire¡/e2¿.

We hypothesise that this simple combination of named entity tagging and string
matching works because of the controlled circumstances, which were mentioned
at the beginning of this section. We only allow matches involving the person
who is the main subject of an article, ensuring that statements made in
sentences are most likely to be about this person. This may sound quite
obvious at first. However, sentences taken from articles at random, matching
random people, do not necessarily contain statements about that person. If the
subject of the Wikipedia article is a certain person, most, if not all,
statements made mentioning that person are likely to directly relate to that
person.

Another control mechanism involves the structure of Wikipedia. Often we find a
number of opening paragraphs containing the most important information about a
person (or other entity). First mentions of certain facts are likely to be the
main information, such as the first date mentioned usually being the date of
birth, first mentioned locations being the places of death and/or birth, job
titles usually the corresponding (and main) occupation of the person and so
on. It should be mentioned, however, that this structure can cause problems,
as will be elaborated on in section 4.2.

It is important to note that not every relation is always found for every
entity. We therefore tried different processing approaches for the textual
data, detailed in Section 4.1. A breakdown of the number of relations per set
is presented in Section 4.2. Each relation also requires slightly different
handling depending on the type of information. Tasks include date
normalisation, partial matching for occupations, and exact location name
matching. Exact details are presented in the following sections.

####  3.2.1. Date-based Relations

This set of relations includes birthdate (date of birth) and deathdate (date
of death). In order to match these relations, the system checks for a DATE
entity in the sentence, which is normalised to YYYY-MM-DD format. We use the
dateparser999https://dateparser.readthedocs.io/en/latest/ package and use the
date of processing as a relative date (for rare cases such as tomorrow or
today). Furthermore, we use the first match for both relations, discarding
subsequent matches. This mode of processing aligns with our restrictive
approach, which assumes most pertinent information to mentioned towards the
beginning of a Wikipedia article, rather than towards the end.

####  3.2.2. Name-based Relations

This set of relations includes ofParent, sibling and hasChild, as well as
educatedAt (the place of education). For these name-based relations, the
system checks a sentence for PER and ORG tags. It is ensured that only full
matches are accepted, even though it may seem favourable to accept partial
matches, at least for anything concerning persons. This is because with
persons, it can be reasonable to allow just the first or last name to match.
However, we found during the manual annotation process (Section 4.2) that too
many false matches occurred, caused by different persons having the same name.

####  3.2.3. Entity Information Relations

Only the occupation relation is included in this group. Since spaCy’s NER
capabilites do not include any tags such as title or job, we opted to use
Stanford CoreNLP’s entity information processing to add this relation. We
could have trained the spaCy model to include a new entity type for this step.
In the end, we used CoreNLP as we felt training a new relation could
potentially introduce another layer of errors.

The system lookup for this relation functions in a similar way to the previous
set of relations, only that instead the CoreNLP information is accessed for
matching. As mentioned, we run the initial CoreNLP processing separately due
to the increased run time. Again, we only allow the complete first match to be
annotated. Potentially, this relation set could be extended by using further
occupation information from Wikidata, which in most cases lists a number of
different occupations for a person, rather than the one main occupation listed
in Pantheon.

####  3.2.4. Other Relations

This class of relations, labelled as other in the dataset, is used for all
other relations. It is essentially the zero class, that is labelled when all
other lookups in a sentence have failed. The other label is then applied to an
entity pair that does not appear to be part of any of the other nine relations
matched. Since we obtain more sentence from this class than all the other nine
combined, we randomly select sentences and balance according to the total
number of all other sentences containing relations. We balance the other
relation class to make it equivalent in size with the remaining nine relations
combined.

If in future more relations are added to the dataset, it would be vital to
ensure that these Other labelled sentences do not contain the new relations,
since they could conceivably be anything.

##  4\. Experiments

We carried out multiple experiments to estimate the quality and usefulness of
this dataset. First, we examined the effects of different processing
approaches for the article texts. Next, we manually annotated a small sub-set
of sentences to pinpoint potential problems and to create a gold-standard set
for evaluation purposes. After re-running the compilation process, taking into
account certain observations and minor processing improvements after manual
annotation, we trained a number of state-of-the-art ML models using the
training datasets, and evaluate the performance using the gold set.

###  4.1. Labelling Approaches

For the process of automatically labelling each entity pair with a
corresponding relation, we work at the document and sentence levels of a
relevant Wikipedia article. At the document level we carry out all the NLP
processing, such as NER, and then split the article into its sentences, to
process each sentence. However, we wanted to assess the effect of two further
approaches of processing the articles: First, we wanted to see how well co-
reference resolution performs on the Wikipedia texts, and whether it would
yield more annotated sentences (Section 4.1.1). Next, we looked into
addressing sentence diversity, by implementing an approach that skips the
first sentence of an article (Section 4.1.2).

####  4.1.1. Coref Set

We hypothesise that replacing co-referential entity mentions will allow the
matching algorithm to find more matches overall. This would be due to the fact
that more names would be matched because of the increased presence. Detecting
more names could then potentially lead to more relation matches overall. For
this, we used spaCy’s built-in co-reference resolution capabilities to
automatically replace entity mentions with the most probable entity. The
matching step is carried out using the text where all the entities have been
replaced.

Table 2 shows the number of relations found across each of the sets we
compiled: normal, coref, which is described here, and skip, which is described
in the next section. The last line of the table shows the total number of
relations found per set.

| normal | coref | skip
---|---|---|---
birthdate | 52,083 | 48,004 | 45,366
birthplace | 50,396 | 46,552 | 19,746
deathdate | 17,376 | 14,505 | 87,93
deathplace | 19,055 | 20,444 | 11,202
occupation | 41,469 | 41,469 | 17,642
ofParent | 6,503 | 10,301 | 6,022
educatedAt | 5,738 | 9,430 | 5,694
hasChild | 2,343 | 4,042 | 2,215
sibling | 2,189 | 3,618 | 2,098
other | 197,952 | 199,165 | 119,578
Total | 395,104 | 397,530 | 238,356
Table 2. Number of relations in each set: normal uses the normal processing
method, coref uses automatic coreference resolution and skip skips the first
sentence of each article.

If we compare the overall counts of the relations of the normal and coref
sets, we observe a small increase. However, looking at the counts of the
different relation types, we see that it is not a simple increase across the
board. In fact, we see fewer matches in certain cases. Upon further
inspection, we found that this was mainly due to the automatic replacement
process producing illegible sentences through incorrect replacements. The two
main problems were entities that are scrambled and sentences being
unintelligible because every single entity mention was replaced with one main
entity, that was often also too long.

The main problems are demonstrated in the two examples below. In the first
sentence, an entity has been replaced many times, including an opening
bracket. Cases like these were observed frequently, and with more characters
added. These cases introduced matching errors in the set. In the second
example, we see a nested replacement, which similarly causes matching
problems.

Replaced: Born in ¡e1¿Évreux¡/e1¿, Eure, a great fan of Paris Saint-Germain
Paris Saint-Germain since ¡e2¿Bernard Mendy¡/e2¿ ( childhood,  Bernard Mendy (
achieved  Bernard Mendy ( ambitions in 2000 when Bernard Mendy ( joined PSG
from SM Caen. Original: Born in ¡e1¿Évreux¡/e1¿, Eure, a great fan of Paris
Saint-Germain since his childhood, he achieved his ambitions in 2000 when he
joined PSG from SM Caen.

Replaced: The hundreds of volumes contained Queen Victoria’s Queen
¡e1¿Victoria¡/e1¿’s’s personal views of […] Original: The hundreds of volumes
contained Queen Victoria’s personal views of […]

To understand better why this approach does not work well, we carried out a
manual annotation of 100 randomly selected sentences per relation from this
set, which is described in Section 4.2. We also trained a neural model using
this dataset, the evaluation of this is detailed in Section 4.3.

####  4.1.2. Skip Set

The skip set was compiled to study the effects of leaving out the first
sentence of an article from Wikipedia. One problem with using Wikipedia texts
stems from the first sentence of an article, or rather the structure of the
first sentence of an article, is seen in the following example.

William Shakespeare (bapt. 26 April 1564 – 23 April 1616) was an English
playwright, poet and actor, widely regarded as the greatest writer in the
English language and the world’s greatest dramatist.

We see that the date of birth (and death) occur within parentheses after the
name, in addition to the fact that the sentence usually contains a large
amount of summarised information. This type of sentence structure (and
content) is not only extremely frequent, but also quite specific to Wikipedia,
suggesting that unnatural behaviour could be learned by a machine learning
model. This was observed by Chisholm et al. (2017) who exploited this for
their benefit. However, for this approach, we wanted to achieve as many
natural matches as we could. Therefore, we compiled a dataset that follows the
previously described methodology, but skips the first sentence of each
article. The hypothesis is that this forces more matches elsewhere in the
article, where more natural sentences occur.

Table 2 shows the total and individual counts for each relation, as referred
to previously. We see that overall, the skip set has much fewer matches than
the other two sets, and it never has the highest number of individual counts
in any category, although the numbers are comparable in some categories to the
normal set. Regardless, some of the generally larger categories, such as
birthplace and birthdate are significantly smaller than the other two sets,
generally pointing towards the fact that the identification is successful, as
this information is extremely common in the first sentence. It is not always
certain that this information will appear later on in an article, therefore
leading to a smaller number of matches.

As with the previous set, we present a manual evaluation of 100 randomly
selected sentences per relation from this set in Section 4.2, and the results
of a trained neural model using this dataset in Section 4.3.

###  4.2. Manual Annotation

We assessed the quality of our semi-supervised datasets before using it to
train machine learning models by means of manual annotation. This was
important in order to find areas where the approach fails to match data
accurately, where processing methods do not work, and any other similar
problems. In addition, we needed a gold standard test set for benchmarking our
neural models.

As pointed out in previous sections, we extracted 100 sentences per relation
across the three datasets, equalling 3000 sentences in total that we manually
annotated and refer to as the gold set. The data was annotated by two persons,
one native English speaker and one non-native but fluent English speaker, both
postgraduate students. For each sentence, the task was to look at the relation
assigned by our matching algorithm and add the correct relation if it had been
labelled incorrectly. We used one of the nine indicative labels where
appropriate, and the other label if a different relation was expressed. Our
annotation guideline was that a human should understand by reading the
sentence which relation is expressed, regardless of prior knowledge. This is
demonstrated by the following examples.

The first example shows a sentence that clearly mentions the occupation E2 of
the entity E1. The second example shows an implicit relation. Although it is
not directly stated, the word orphaned in relation to entity E2 with the
statement that E1 died, implies that E1 is the parent of E2. In the final
example, the algorithm labels the sentence as expressing the parent relation
between the two entities. Although this may indeed be the case, and the
annotator may have prior knowledge of this, or it has been expressed in a
different sentence, it is not clearly stated in this sentence.

Explicit: ¡e1¿Renate Künast¡/e1¿ (born 15 December 1955) is a German
¡e2¿politician¡/e2¿ of Bündnis 90/Die Grünen.

Implicit: A few months later ¡e1¿Apollo Korzeniowski¡/e1¿ died, leaving
¡e2¿Conrad¡/e2¿ orphaned at the age of eleven.

Unclear: Thus, ¡e1¿Janaka¡/e1¿ tries to find the best husband for
¡e2¿Sita¡/e2¿.

The Cohen’s Kappa for the inter-annotator agreement is 0.908 which indicates a
very high agreement between our annotators. The annotations allowed us to make
a number of observations. First, we notice that two very similar relations
work very differently. While birthplace works extremely well across sets,
deathplace does not. Upon further examination, we found that the first mention
of the place where someone died often was also the place where a person lived.
In future, cases like these may warrant a different approach to processing by
our algorithm, but for now we leave it unchanged. Second, we observed that
many relations in the coref set were incoherent and probably incorrect, due to
imprecise replacements by the coreference resolution algorithm.

While Wikidata as a source does work quite well, categories can sometimes be
ambiguous, such as the educatedAt and parent relations. Here, we observed that
the Wikidata entries contained information at odds with our interpretation of
the type of entry, such as educatedAt containing a University that is the
place of work, or parent containing a person that the target entry is a parent
of rather has. Since this did not occur often in our manual evaluation, we did
not implement a strategy to solve this problem.

Finally, we found a number of simple processing errors that we solved by
improving our regular expressions for text cleaning. We also adjusted the
matching procedure for the occupation relation, to avoid matches where the
occupation mentioned belonged to a different entity. This leads to a slightly
smaller number of relations overall, with a detailed overview shown in Table
3.

| normal | coref | skip
---|---|---|---
birthdate | 51,524 | 47,977 | 45,211
birthplace | 50,226 | 46,551 | 17,537
deathdate | 17,197 | 14,500 | 5,925
deathplace | 18,944 | 20,430 | 10,790
occupation | 18,114 | 18,111 | 8,716
ofParent | 6,352 | 10,291 | 5,596
educatedAt | 5,639 | 9,415 | 3,858
hasChild | 2,209 | 4,053 | 2,123
sibling | 2,083 | 3,601 | 1,997
Other | 173,969 | 175,916 | 103,248
Total | 346,257 | 350,845 | 205,001
Table 3. Relations per Set after Processing Improvements

Overall, we have formed the following impressions for each set. The normal
approach works well, while not offering a very diverse set of sentences. As
alluded to earlier, it is clear that this approach matches mainly the standard
Wikipedia first sentence, as described in previous sections. The coref set,
while seemingly the largest set, must also include the most unusable sentences
and bad examples. During the course of evaluating the sentences we found this
set to be imprecise, not explicit and difficult to understand due to bad
replacements. Finally, we found the skip set to be very mixed in terms of
success. While for some relations it seems that none of the matching has
returned usable results, other relations seem to have worked very well,
offering in addition a wide variety of different sentences demonstrating the
desired effects.

In order to determine the performance of the matching algorithm, we present
the evaluation metrics for the gold set. For this, we compared the labels
produced by the automatic matching algorithm to our manually produced labels.
We removed 100 sentences from the gold set that contained processing errors
caused by conversion to plain text, automatic replacement of coreferences and
spaCy tagging errors. Since these would all have been annotated as ”Other”, we
decided to remove these sentences since they could have caused an imbalanced
test set. Table 4 shows the results of the evaluation for each set. We observe
that most of the matches found are correct, indicated by high precision and
recall scores. However, the problem with deathplace we observed during the
evaluation is confirmed here. In addition, recall drops significantly for the
Other class, mainly due to the fact that this was increased because of
incorrect classifications by the matching algorithm.

| normal | coref | skip
---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1
birthdate | 1.0 | 1.0 | 1.0 | 0.99 | 1.0 | 0.99 | 1.0 | 1.0 | 1.0
birthplace | 0.84 | 0.9 | 0.87 | 0.86 | 0.88 | 0.87 | 0.79 | 0.83 | 0.81
deathdate | 1.0 | 0.99 | 1.0 | 0.98 | 1.0 | 0.99 | 0.94 | 0.99 | 0.96
deathplace | 0.37 | 0.95 | 0.53 | 0.31 | 1.0 | 0.48 | 0.36 | 0.97 | 0.53
occupation | 0.8 | 1.0 | 0.89 | 0.9 | 1.0 | 0.85 | 0.68 | 1.0 | 0.81
educatedAt | 0.88 | 1.0 | 0.94 | 0.92 | 0.99 | 0.95 | 0.96 | 0.99 | 0.97
ofParent | 0.77 | 0.99 | 0.87 | 0.73 | 1.0 | 0.85 | 0.8 | 1.0 | 0.89
hasChild | 0.8 | 0.99 | 0.88 | 0.64 | 1.0 | 0.78 | 0.63 | 1.0 | 0.77
sibling | 0.75 | 0.95 | 0.84 | 0.62 | 1.0 | 0.77 | 0.7 | 0.92 | 0.8
other | 0.97 | 0.37 | 0.54 | 0.98 | 0.36 | 0.53 | 0.96 | 0.33 | 0.49
macro avg. | 0.82 | 0.91 | 0.83 | 0.79 | 0.92 | 0.81 | 0.78 | 0.9 | 0.8
Table 4. Evaluation of Manual Annotations on Gold Set

###  4.3. Neural Models

The machine learning model we used to perform relationship classification is
based on transformers. Since their introduction, transformer models have shown
excellent results in various NLP tasks (Devlin et al., 2019) such as text
classification (Ranasinghe and Zampieri, 2020), NER (Jia et al., 2020) and
question answering (Yang et al., 2019) including RE (Yamada et al., 2020;
Joshi et al., 2020; Wu and He, 2019; Alt et al., 2019). In this research, we
utilised the architecture introduced by Baldini Soares et al. (2019).

Figure 3. Neural Network Architecture

The input to the transformer models is the sentence with “[E1]” and “[E2]”
markers marking the positions of their respective entities. Then the output
hidden states of transformer at the “[E1]” and “[E2]” token positions are
concatenated as the final output representation of the relationship. Finally,
a linear classifier is stacked on top of the output representation. The
architecture diagram is visualised in Figure 3.

We fine-tune all the parameters from the transformer as well as the linear
classifier jointly by maximising the log-probability of the correct label. For
all the experiments we optimised parameters (with AdamW) using a learning rate
of $7e-5$, a maximum sequence length of $512$, and a batch size of $32$
samples. The models were trained using a $24$ GB RTX 3090 GPU over five
epochs. As the pre-trained transformer model, we used the bert-base-uncased
model available in HuggingFace (Wolf et al., 2020).

For training the BERT-based classifier, we used each of the three sets
separately, as well as a combination of the three sets we refer to as all,
where we remove any duplicates that might be caused by the combination. We did
not focus on producing the best possible results, and rather on indicating
whether the produced dataset is even suitable for training a model. Table 5
shows the evaluation results of the models trained on the four different sets.
While largely the results of the matching algorithm are echoed, we observe
that some other relations, including hasChild, ofParent and sibling seem to
score quite low in terms of recall. When comparing to the counts per set (See
Table 3) these relations are quite low in number compared to the others,
possibly explaining the results.

| normal | coref | skip | all
---|---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
birthdate | 1.0 | 0.99 | 1.0 | 1.0 | 0.99 | 0.99 | 0.87 | 0.92 | 0.89 | 1 | 0.99 | 0.99
birthplace | 0.85 | 0.77 | 0.81 | 0.8 | 0.8 | 0.8 | 0.81 | 0.74 | 0.8 | 0.88 | 0.8 | 0.83
deathdate | 1.0 | 0.95 | 0.97 | 1.0 | 0.98 | 0.99 | 0.98 | 0.86 | 0.91 | 1.0 | 0.98 | 0.99
deathplace | 0.73 | 0.53 | 0.62 | 0.79 | 0.52 | 0.62 | 0.82 | 0.42 | 0.55 | 0.81 | 0.48 | 0.6
occupation | 1.0 | 0.99 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.98 | 0.99 | 1.0 | 0.99 | 1.00
educatedAt | 0.98 | 0.87 | 0.92 | 0.97 | 0.91 | 0.94 | 0.98 | 0.83 | 0.9 | 1.0 | 0.87 | 0.93
ofParent | 0.92 | 0.54 | 0.66 | 0.82 | 0.57 | 0.67 | 0.78 | 0.51 | 0.61 | 0.88 | 0.6 | 0.7
hasChild | 0.96 | 0.36 | 0.42 | 0.92 | 0.43 | 0.55 | 0.97 | 0.39 | 0.5 | 0.98 | 0.39 | 0.49
sibling | 0.92 | 0.45 | 0.57 | 0.94 | 0.43 | 0.55 | 0.87 | 0.45 | 0.55 | 0.93 | 0.46 | 0.57
other | 0.38 | 0.95 | 0.54 | 0.41 | 0.94 | 0.57 | 0.38 | 0.93 | 0.54 | 0.39 | 0.95 | 0.56
macro avg. | 0.9 | 0.73 | 0.76 | 0.89 | 0.75 | 0.78 | 0.87 | 0.7 | 0.74 | 0.92 | 0.74 | 0.78
Table 5. Evaluation Metrics for Relations in each Set

##  5\. Proposed Application

The availability of compiled datasets for historical research is more
important than ever. While NLP methods in domains such as biomedical and news
continue to be expanded greatly, smaller areas of research such as specific
historical (biographical) research inherently lack these opportunities. Being
able to compile datasets to train neural extraction models with relative ease,
as described here, is crucial for future research. In this section, we
highlight this by example of the Army List in the United Kingdom, a study that
we plan to embark on in the coming months.

The Army List (191, 1913) is a biographical compendium of officers serving in
the British Army. It was first published in 1840 and volumes were subsequently
published annually, although this varied during wartime. Each volume lists the
name and rank of every serving officer in the British Army, along with
important biographical details including length of service, past roles, and
current position held. The Army List is an essential starting point for any
research about the careers of military officers in the period.

Despite its importance to historical research, the Army List can prove
difficult to access. Copies are held by a handful of specialist archives in
the United Kingdom and there has been no systematic attempt to digitise them
or apply data processing to them. Each Army List contains a wealth of
information that invites cross-referencing and comparison to learn more about
professional and social links amongst the officer class. However, the sheer
number of biographical entries, amounting to several thousand per volume, made
this an impossible task for historians in the pre-digital age. Digital
processing offers a solution to this problem and opens the possibility of
being able to map connections in new and illuminating ways. For example, it
would allow the identification of professional networks based on age, shared
roles, unit associations, and overseas service. A dataset based upon it would
be of enormous value to historians, and it would open exciting new avenues for
research and would contribute to ongoing historiographical debates on the
professional bonds of the officer class.

To enable the kind of research described above, there is clearly a need for
datasets like Biographical so that systems can be trained to extract large
amounts of information quickly and efficiently. Not only could the dataset we
present here be used itself, but also new datasets, compiled with the method
we present here. Both the dataset and method therefore present significant
opportunities for application, enabling research in under-resourced areas.

##  6\. Conclusion

We have presented Biographical, a relation extraction dataset that is semi-
supervised, and described its compilation process in detail. Furthermore, we
carried out a number of experiments to understand the dataset better. This
included different processing approaches, a manual annotation task and the
training of different neural models. Not only have these experiments
investigated different ways of optimising the compilation of the dataset for
different goals, they have also validated the results in terms of machine
learning.

In more general terms, this work marks an exciting first step at applying data
processing to historical documentation. Archival digitisation in the United
Kingdom and other countries remains hesitant and inconsistent, and there has
been very little data processing of that which is available. The application
of more computational resources to mine the data would be of immense value to
historians and those working in related fields.

In the future, we would like to address a number of different aspects
concerning this dataset. First, the optimisation of the compilation process
for even more precise results will be focused on. Next, we would like to
extend the number of relations, and demonstrate how simple this could be. As
mentioned in the previous section, we also intend to test this approach on
real-world texts in collaboration with historians.

## References

  * (1)
  * 191 (1913) 1913\.  The quarterly army list for the quarter ending April 1914\.
  * Alt et al. (2019) Christoph Alt, Marc Hübner, and Leonhard Hennig. 2019\.  Improving Relation Extraction by Pre-trained Language Representations. In _Automated Knowledge Base Construction (AKBC)_.  https://openreview.net/forum?id=BJgrxbqp67
  * Baldini Soares et al. (2019) Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019.  Matching the Blanks: Distributional Similarity for Relation Learning. In _Proceedings of ACL 2019_. ACL, Florence, Italy, 2895–2905.  https://aclanthology.org/P19-1279
  * Biadsy et al. (2008) Fadi Biadsy, Julia Hirschberg, and Elena Filatova. 2008\.  An Unsupervised Approach to Biography Production Using Wikipedia. In _Proceedings of ACL 2008: HLT_. 807–815.
  * Chisholm et al. (2017) Andrew Chisholm, Will Radford, and Ben Hachey. 2017\.  Learning to generate one-sentence biographies from Wikidata.  _CoRR_ abs/1702.0 (2017).  http://arxiv.org/abs/1702.06235 _eprint: 1702.06235.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of NAACL 2019: HLT_. ACL, Minneapolis, Minnesota, 4171–4186.  https://www.aclweb.org/anthology/N19-1423
  * Gormley et al. (2015) Matthew R. Gormley, Mo Yu, and Mark Dredze. 2015.  Improved Relation Extraction with Feature-Rich Compositional Embedding Models. In _Proceedings of EMNLP 2015_. ACL, Lisbon, Portugal, 1774–1784.  https://aclanthology.org/D15-1205
  * Hendrickx et al. (2010) Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010\.  SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of Nominals. In _Proceedings of the 5th International Workshop on Semantic Evaluation_. ACL, Uppsala, Sweden, 33–38.  https://aclanthology.org/S10-1006
  * Hoffmann et al. (2011) Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S. Weld. 2011.  Knowledge-Based Weak Supervision for Information Extraction of Overlapping Relations. In _Proceedings of ACL 2011: HLT_. ACL, Portland, Oregon, USA, 541–550.  https://aclanthology.org/P11-1055
  * Hogue et al. (2014) Alexander Hogue, Joel Nothman, and James R Curran. 2014\.  Unsupervised Biographical Event Extraction Using Wikipedia Traffic. In _Proceedings of the Australasian Language Technology Association Workshop 2014_. 41–49.
  * Jia et al. (2020) Chen Jia, Yuefeng Shi, Qinrong Yang, and Yue Zhang. 2020\.  Entity Enhanced BERT Pre-training for Chinese NER. In _Proceedings of EMNLP 2020_. ACL, Online, 6384–6396.  https://doi.org/10.18653/v1/2020.emnlp-main.518
  * Jiang (2012) Jing Jiang. 2012\.  _Information Extraction from Text_.  Springer US, Boston, MA, 11–41.  https://doi.org/10.1007/978-1-4614-3223-4_2
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020\.  SpanBERT: Improving Pre-training by Representing and Predicting Spans.  _Transactions of ACL_ 8 (2020), 64–77.  https://aclanthology.org/2020.tacl-1.5
  * Liu et al. (2007) Yudong Liu, Zhongmin Shi, and Anoop Sarkar. 2007.  Exploiting Rich Syntactic Information for Relation Extraction from Biomedical Articles. In _Proceedings of NAACL 2007: HLT_ (Rochester, New York) _(NAACL-Short ’07)_. Association for Computational Linguistics, USA, 97–100.
  * Miwa and Bansal (2016) Makoto Miwa and Mohit Bansal. 2016.  End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures. In _Proceedings of ACL 2016_. Association for Computational Linguistics, Berlin, Germany, 1105–1116.  https://doi.org/10.18653/v1/P16-1105
  * Nasar et al. (2021) Zara Nasar, Syed Waqar Jaffry, and Muhammad Kamran Malik. 2021\.  Named Entity Recognition and Relation Extraction: State-of-the-Art.  _ACM Comput. Surv._ 54, 1, Article 20 (feb 2021), 39 pages.  https://doi.org/10.1145/3445965
  * Nayak and Ng (2020) Tapas Nayak and Hwee Tou Ng. 2020.  Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extraction.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 34, 05 (Apr. 2020), 8528–8535.  https://ojs.aaai.org/index.php/AAAI/article/view/6374
  * Over and Yen (2004) Paul Over and James Yen. 2004.  An introduction to DUC-2004.  _National Institute of Standards and Technology_ (2004).
  * Palmero Aprosio and Tonelli (2015) Alessio Palmero Aprosio and Sara Tonelli. 2015.  Recognizing Biographical Sections in Wikipedia. In _Proceedings of EMNLP 2015_. ACL, Lisbon, Portugal, 811–816.  https://aclanthology.org/D15-1095
  * Plum et al. (2019) Alistair Plum, Marcos Zampieri, Constantin Orăsan, Eveline Wandl-Vogt, and Ruslan Mitkov. 2019\.  Large-scale Data Harvesting for Biographical Data. In _Proceedings of (BD-2019)_.
  * Ranasinghe and Zampieri (2020) Tharindu Ranasinghe and Marcos Zampieri. 2020.  Multilingual Offensive Language Identification with Cross-lingual Embeddings. In _Proceedings of EMNLP 2020_. ACL, Online, 5838–5844.  https://doi.org/10.18653/v1/2020.emnlp-main.470
  * Riedel et al. (2010) Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010\.  Modeling Relations and Their Mentions without Labeled Text. In _Machine Learning and Knowledge Discovery in Databases_ , José Luis Balcázar, Francesco Bonchi, Aristides Gionis, and Michèle Sebag (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 148–163.
  * Shen and Huang (2016) Yatian Shen and Xuanjing Huang. 2016.  Attention-Based Convolutional Neural Network for Semantic Relation Extraction. In _Proceedings of COLING 2016: Technical Papers_. Osaka, Japan, 2526–2536.  https://aclanthology.org/C16-1238
  * Singhal et al. (2016) Ayush Singhal, Michael Simmons, and Zhiyong Lu. 2016\.  Text mining for precision medicine: automating disease-mutation relationship extraction from biomedical literature.  _J Am Med Inform Assoc_ 23, 4 (April 2016), 766–772.
  * Smirnova and Cudré-Mauroux (2018) Alisa Smirnova and Philippe Cudré-Mauroux. 2018\.  Relation Extraction Using Distant Supervision: A Survey.  _ACM Comput. Surv._ 51, 5, Article 106 (nov 2018), 35 pages.  https://doi.org/10.1145/3241741
  * Turian et al. (2010) Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010\.  Word Representations: A Simple and General Method for Semi-Supervised Learning. In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_. ACL, Uppsala, Sweden, 384–394.  https://aclanthology.org/P10-1040
  * Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch. 2014.  Wikidata: a free collaborative knowledgebase.  _Commun. ACM_ 57, 10 (2014), 78–85.
  * Wang et al. (2021) Hailin Wang, Guoming Lu, Jin Yin, and Ke Qin. 2021\.  Relation Extraction: A Brief Survey on Deep Neural Network Based Methods. In _2021 The 4th International Conference on Software Engineering and Information Management_ (Yokohama, Japan) _(ICSIM 2021)_. ACM, New York, NY, USA, 220–228.  https://doi.org/10.1145/3451471.3451506
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020\.  Transformers: State-of-the-Art Natural Language Processing. In _Proceedings of EMNLP 2020: System Demonstrations_. ACL, Online, 38–45.  https://aclanthology.org/2020.emnlp-demos.6
  * Wu and He (2019) Shanchan Wu and Yifan He. 2019.  Enriching Pre-Trained Language Model with Entity Information for Relation Classification. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ (Beijing, China) _(CIKM ’19)_. Association for Computing Machinery, New York, NY, USA, 2361–2364.  https://doi.org/10.1145/3357384.3358119
  * Xiao and Liu (2016) Minguang Xiao and Cong Liu. 2016.  Semantic Relation Classification via Hierarchical Recurrent Neural Network with Attention. In _Proceedings of COLING 2016: Technical Papers_. Osaka, Japan, 1254–1263.  https://aclanthology.org/C16-1119
  * Xue et al. (2021) Fuzhao Xue, Aixin Sun, Hao Zhang, and Eng Siong Chng. 2021\.  GDPNet: Refining Latent Multi-View Graph for Relation Extraction.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 35, 16 (May 2021), 14194–14202.  https://ojs.aaai.org/index.php/AAAI/article/view/17670
  * Xue et al. (2019) Kui Xue, Yangming Zhou, Zhiyuan Ma, Tong Ruan, Huanhuan Zhang, and Ping He. 2019\.  Fine-tuning BERT for Joint Entity and Relation Extraction in Chinese Medical Text. In _2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_. 892–897.  https://doi.org/10.1109/BIBM47256.2019.8983370
  * Yamada et al. (2020) Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020.  LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention. In _Proceedings of EMNLP 2020_. ACL, Online, 6442–6454.  https://doi.org/10.18653/v1/2020.emnlp-main.523
  * Yang et al. (2019) Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019\.  End-to-End Open-Domain Question Answering with BERTserini. In _Proceedings of NAACL 2019_. ACL, Minneapolis, Minnesota, 72–77.  https://doi.org/10.18653/v1/N19-4013
  * Yu et al. (2016) Amy Zhao Yu, Shahar Ronen, Kevin Hu, Tiffany Lu, and César A Hidalgo. 2016.  Pantheon 1.0, a Manually Verified Dataset of Globally Famous Biographies.  _Scientific data_ 3, 1 (2016), 1–16.
  * Zeng et al. (2014) Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014.  Relation Classification via Convolutional Deep Neural Network. In _Proceedings of COLING 2014: Technical Papers_. Dublin City University and ACL, Dublin, Ireland, 2335–2344.  https://aclanthology.org/C14-1220
  * Zhang et al. (2017) Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017.  Position-aware Attention and Supervised Data Improve Slot Filling. In _Proceedings of EMNLP 2017_. ACL, Copenhagen, Denmark, 35–45.  https://aclanthology.org/D17-1004
  * Zhou et al. (2004) Liang Zhou, Miruna Ticrea, and Eduard Hovy. 2004\.  Multi-Document Biography Summarization. In _Proceedings of EMNLP 2004_. ACL, Barcelona, Spain, 434–441.  https://aclanthology.org/W04-3256
  * Zhou et al. (2016) Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016.  Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification. In _Proceedings of ACL 2016_. ACL, Berlin, Germany, 207–212.  https://doi.org/10.18653/v1/P16-2034


# Auditing Source Diversity Bias in Video Search Results Using Virtual Agents

Aleksandra Urman<EMAIL_ADDRESS>University of BernUniversity
of ZurichSwitzerland ,  Mykola Makhortykh<EMAIL_ADDRESS>University of BernSwitzerland and  Roberto Ulloa<EMAIL_ADDRESS>GESIS – Leibniz Institute for the Social SciencesGermany

(2021)

###### Abstract.

We audit the presence of domain-level source diversity bias in video search
results. Using a virtual agent-based approach, we compare outputs of four
Western and one non-Western search engines for English and Russian queries.
Our findings highlight that source diversity varies substantially depending on
the language with English queries returning more diverse outputs. We also find
disproportionately high presence of a single platform, YouTube, in top search
outputs for all Western search engines except Google. At the same time, we
observe that Youtube’s major competitors such as Vimeo or Dailymotion do not
appear in the sampled Google’s video search results. This finding suggests
that Google might be downgrading the results from the main competitors of
Google-owned Youtube and highlights the necessity for further studies focusing
on the presence of own-content bias in Google’s search results.

source diversity bias, algorithmic auditing, web search

††journalyear: 2021††copyright: iw3c2w3††conference: Companion Proceedings of
the Web Conference 2021; April 19–23, 2021; Ljubljana, Slovenia††booktitle:
Companion Proceedings of the Web Conference 2021 (WWW ’21 Companion), April
19–23, 2021, Ljubljana, Slovenia††doi: 10.1145/3442442.3452306††isbn:
978-1-4503-8313-4/21/04

##  1\. Introduction

The public tends to put high trust in the information retrieved via search
engines (noa, [n.d.]a). However, search engine outputs are prone to different
forms of bias (Kay et al., 2015; Goldman, 2008) which can result in distorted
representation of subjects which users are searching for (Grimmelmann, 2010;
Mowshowitz and Kawaguchi, 2002). One way of uncovering bias in web search
outputs is to engage in algorithmic auditing (Mittelstadt, 2016). Though a few
studies have audited bias in text and image search (e.g., (Kay et al., 2015;
Otterbacher et al., 2017; Kulshrestha et al., 2019)), to the best of our
knowledge no audits of video search results were conducted. However, video
search is important in the societal context since people increasingly consume
news via online videos (Kalogeropoulos, 2018) and treat video hosting
platforms as a preferred environment for news finding (Stocking et al., 2020).
The fact that video information can have powerful influence on users and even
affect their behaviors (Izawa, [n.d.]; Erickson et al., 2018) prompts the need
for auditing whether video search is subjected to bias.

To address this gap, we investigate the presence of source diversity bias -
that is a systematic prevalence of specific information sources (Goldman,
2008) \- in video search outputs. By consistently prioritizing the same set of
sources independently of search queries, search algorithms can diminish the
quality of outputs by making them less representative (Mowshowitz and
Kawaguchi, 2002) and negatively affecting the user experience (Ciampaglia et
al., 2018). Source diversity bias is also related to the phenomenon of search
concentration, namely the tendency of search engines to prioritize few well-
established domains over other sources (Jiang, 2014b) that, according to
media, often results in search companies promoting their own services (e.g.,
YouTube in the case of Google (West, 2020)). By diminishing the diversity in
the composition of source domains, companies can consolidate global media
monopolies (Jiang, 2014b) through gaining unfair advantage over their
competitors.

To examine whether video search outputs are subjected to source diversity
bias, we audit search results coming from five search engines - four Western
(Bing, DuckDuckGo, Google, and Yahoo) and one non-Western (Yandex) - in
response to English and Russian queries. Including Yandex along with queries
in Russian - a language dominating the main markets of Yandex - allowed us to
test whether (some of) our observations can be attributed to structural
differences between Western and non-Western markets (e.g., almost monopolistic
status of Google in the former and its competition with Yandex in post-Soviet
countries). We use virtual agent-based auditing approach to prevent search
outputs from being affected by search personalization and search
randomization. Then, using a selection of metrics, we assess the level of
source domain diversity in search outputs and investigate whether there is
evidence of certain engines prioritizing specific information sources.
Specifically, we examine whether search engines tend to promote platforms
associated with their parent companies (e.g., Alphabet for Google or Microsoft
for Bing) or downgrade the competitors as was claimed by earlier research
(Jiang, 2014b).

##  2\. Related work

The problem of auditing systematic skewness of web search outputs is
increasingly recognized in the field of information retrieval (IR) (Mowshowitz
and Kawaguchi, 2002; Grimmelmann, 2010; Goldman, 2008; Introna and Nissenbaum,
2000). Existing studies primarily look at it from one of the two perspectives:
user bias and retrieval bias. User bias concerns skews in user perceptions of
search outputs (Ieong et al., 2012). Retrieval bias relates to a skewed
selection of search results (Introna and Nissenbaum, 2000; Grimmelmann, 2010).

One form of retrieval bias, which the current paper focuses on, is source
diversity bias. Originally discussed in the context of search engines’
tendency to prioritize web pages with the highest number of visitors (Introna
and Nissenbaum, 2000), source diversity bias is currently investigated in the
context of prioritization of certain categories of sources in response to
particular types of queries (e.g., (Makhortykh et al., 2020)). A
disproportionate visibility of specific types of web resources can diminish
overall quality of search results (Ciampaglia et al., 2018) and provide unfair
advantage to companies and individuals that own specific search engines
(Introna and Nissenbaum, 2000) \- e.g., through own-content bias, - or direct
most of the traffic to a handful of well-established sources, a phenomenon
also known as search concentration (Jiang, 2014b).

To date, source diversity bias has been primarily investigated in the context
of text search results (Diakopoulos et al., [n.d.]; Unkel and Haim, 2019;
Trielli and Diakopoulos, 2019) with the focus exclusively on one engine -
Google. At the same time, few comparative studies that were conducted
highlight substantial cross-engine differences in search diversity bias levels
(Jiang, 2014a; Zavadski and Toepfl, 2019; Makhortykh et al., 2020). In the
case of video search results, there is no systematic comparative assessment of
source diversity bias. In 2020, the Wall Street Journal (West, 2020) and,
subsequently, Mozilla (noa, [n.d.]c) have found that Youtube appears among the
top-3 featured "carousel" results in text search 94% of the time (noa,
[n.d.]c). These findings highlight search concentration (Jiang, 2014b) around
Youtube in Google’s "carousel" results. However, it is unclear, first, whether
the distribution of domains is similar in dedicated video search results. And
second, how Google’s results compare to those of its competitors - a
comparison that is necessary to establish whether Youtube’s dominance in video
search results from the way Google’s algorithm works exhibiting own-content
bias. We aim to address these gaps with the present study.

##  3\. Methods

In this study, we have opted for a combination of methods for bias detection
in web search outlined by Edelman (Edelman, [n.d.]a): 1) comparative analysis
of the results provided by multiple search engines across a variety of search
queries in two languages; 2) identification of skewness of results towards
specific domains and whether the skewness, if observed, can be explained by
market structure incentives. We have used video search results from the 4
biggest Western search engines by market share - Google, Yahoo, Bing,
DuckDuckGo, - and one major non-Western search engine - Yandex (noa, [n.d.]b).
Since Yandex has the largest presence in post-Soviet states where large shares
of populations speak Russian as a first language, we utilized queries in both,
English and Russian, to conduct the search and estimate whether there are
differences in the observations. There were 62 queries in total - 31 English
queries with translations into Russian, - that concerned contemporary events
(i.e., the US presidential elections and coronavirus), conspiracy theories
(i.e., Flat Earth), and historical events (i.e., Holocaust)111We used the
following queries: coronavirus, bernie sanders, joe biden, pete buttigieg,
elizabeth warren, michael bloomberg, donald trump, us elections, syria
conflict, ukraine conflict, yemen conflict, holocaust, holodomor, armenian
genocide, second world war, first world war, artificial intelligence, big
data, virtual reality, vaccination, vaccination benefits, vaccination dangers,
george soros, illuminati, new world order, Flat Earth, UFO, Aliens,
misinformation, disinformation, fake news. All queries were entered into
search engines in lower case. For the searches in Russian, we used the exact
translations of the English queries listed below into Russian verified by two
native Russian speakers. . Below we outline the details on the data collection
and analysis.

###  3.1. Data collection

To collect the data, we utilized a set of virtual agents - that is software
simulating user browsing behavior and recording its outputs. The benefits of
this approach, which extends algorithmic auditing methodology introduced by
Haim et al. (Haim et al., 2017), is that it allows controlling for
personalization (Hannak et al., 2013) and randomization (Makhortykh et al.,
2020) factors.

For the current study, we built a network of 84 CentOS virtual machines based
in the Frankfurt region of Amazon Elastic Compute Cloud (EC2). On each
machine, we deployed 2 virtual agents (one in Chrome browser and one in
Mozilla Firefox browser), thus providing us with 188 agents overall. Each
agent was made of two browser extensions: a tracker and a bot. The tracker
collected the HTML and the metadata of all pages visited in the browser and
immediately sent it to a storage server. The bot emulated a sequence of
browsing actions that consisted of (1) visiting a video search engine page,
(2) entering one of the 62 queries, and (3) scrolling down the search result
page to load at least 50 images. Before searching for a new query, the
browsers were cleaned to prevent the search history affecting the search
outputs, and there was a 7-minute break between searches to mitigate potential
effects of previous searches on the results.

The study was conducted on February 26, 2020. We equally distributed the
agents between five search engines: Google, Bing, Yahoo, Yandex, and
DuckDuckGo (DDG)222For all engines, the ”.com” version of the image search
engine was used (e.g., google.com).. Because of technical issues (e.g., bot
detection mechanisms), some agents did not manage to complete their routine.
The overall number of agents per engine which completed the full simulation
routine and returned the search results differed by query - sometimes the
search engine would detect automation and temporarily ban the agent. This was
particularly often the case with Yandex, where for some queries all 34
deployed agents successfully finished the routine while for others (a minority
of queries) Yandex only returned the results for 10 agents and banned the
rest. The mean number of agents who completed the full routine by engine
across all queries is the following: Bing (29), DDG (34), Google (33), Yahoo
(31), and Yandex (17).

After the data was collected, we have extracted top-10 individual video links
obtained by each agent for each search query and proceeded with the analysis
using this data. Our decision to rely on the top-10 results only is motivated
by the fact that users tend to pay the most attention to the first few results
- i.e., those on the first results page (Pan et al., 2007). A comparison of
search results by browser has demonstrated that there are no major between-
browser differences - a finding in contrast with those observed for text
search results (Makhortykh et al., 2020) \- thus for the analysis we have
proceeded aggregating the results for both browsers.

###  3.2. Data analysis

####  3.2.1. Source diversity

To assess whether there is evidence suggesting that diversity bias - that is,
lack of source diversity, - is present in the sampled results on domain level,
we have calculated how many distinct source domains are, on average, present
in the results for each query. To account for the potential randomization due
to so-called "Google Dance" (Battelle, 2005) in the results, for each query we
aggregated the calculation over the results obtained from each individual
autonomous agent. Afterwards, we have calculated mean numbers of distinct
sources separately for English and Russian queries to establish whether there
are differences in the observations depending on the language of the search.
We have also qualitatively examined the results per query to find out whether
there are distinct patterns with regard to query categories.

####  3.2.2. Search concentration

To establish whether there is evidence of search concentration in the
collected video results, we have calculated, first, the share of times
different domains appear as the top result for each search query, and second,
the proportion of times different domains appear among the top-10 search
results per query at all. We suggest that the consistent appearance of a
specific domain or few specific domains at the very top of search results and,
in general, among top-10 results more frequently than the others would
indicate search concentration.

In addition, we have scrutinized the results with regard to own-content bias
exhibited by Google according to the media (West, 2020). We aimed to establish
whether Google’s results lend evidence of own-content bias either through the
promotion of Youtube in the results or the demotion of the results provided by
its main competitors. One way to assess that is to compare Google’s results
with those obtained through other search engines (Edelman, [n.d.]a) and see
whether there are major differences in the observed frequencies of appearance
of different domains between them. Thus, we compared the proportion of times
each domain - Youtube and its competitors such as Vimeo, Dailymotion and
Rutube, - appears in the results provided by different engines.

##  4\. Findings

###  4.1. Source diversity

Figure 1. Mean number of distinct domains in top-10 video search results
(Y-axis) per query, grouped by query languages (legend) and search engines
(X-axis)

As Fig.1 shows, there are differences in the level of source diversity
exhibited by the examined search engines. Google has consistently presented
more diverse video results in terms of source domains than its competitors.
Yandex has taken a second place in terms of domain diversity. This domain
diversity hierarchy is similar for both English and Russian queries, albeit
for the Russian queries the observed domain diversity is a bit lower on all
five search engines than for the English queries.

Qualitative analysis of the domain diversity by query has demonstrated that
there are no consistent patterns with regard to the proportion of distinct
sources by query category in our dataset. Hence, we suggest that domain
diversity is affected more by the algorithms used by each search engine
examined and, probably, the data they are trained on - the latter might
explain the observed differences between Russian and English queries, - rather
than by the specific topics of the search queries.

###  4.2. Search concentration

As shown in Fig.2, Youtube has been featured as the top result most frequently
in all cases but one - namely, the results for English queries on Yandex where
Vimeo surfaced as the first result most often. Remarkably, on Google itself
Youtube appeared as the top result less frequently than on other platforms, a
finding in contrast with those made by The Wall Street Journal (West, 2020)
and Mozilla (noa, [n.d.]c) in the context of featured video "carousel" on the
first page of Google’s text results. DuckDuckGo, Yahoo, Yandex are the three
engines exhibiting sizeable differences in the prominence of Youtube as the
top result between English and Russian queries with Youtube being featured as
the top result more frequently in response to English queries. We suggest that
the findings reported in Fig.2 lend evidence to search concentration bias in
video search results on the examined search engines, with the effect in our
sample being stronger for English than for Russian queries.

Figure 2. Domain most frequently appearing as the top result by query group
and search engine; % of time it appears as the top result.

Figure 3. Domains most frequently appearing among top-10 results by query
group and search engine and % of time a domain appears among top-10 results.

In Fig.3 we list the domains most frequently featured among top-10 search
results on each of the examined search engines for English and Russian
queries. As with the top-1 result, the domain most frequently featured in
top-10 is Youtube on all engines but Yandex in response to English queries
where the most frequent domain is Vimeo. The share of other domains in search
results is comparatively marginal - less than 10% - in almost all cases with
the exception of Ok.ru for Russian queries on Yandex. Youtube thus emerges as
the most prominent domain in search results. Apart from it, there is no other
domain that is among the top 5 domains most frequently appearing among the
first 10 results on all search engines.

The findings reported in Fig.3 suggest that search engines feature different
arrays of domains - with the exception of Youtube - in search results. Google
tends to retrieve results from legacy media in both Russian and English more
frequently than other search engines, a finding in line with the previous
research (Trielli and Diakopoulos, 2019; Diakopoulos et al., [n.d.]; Unkel and
Haim, 2019). Other search engines also include some legacy media, though to a
lower extent than Google, as well as social media (e.g., Ok.ru, Facebook.com),
and several video portals that are Youtube’s competitors - Dailymotion, Vimeo
and Rutube. None of these potential Youtube competitors, however, appear in
the top-10 results on Google in our dataset at all despite their presence on
other search engines. Vimeo, Dailymotion and Rutube all appear at least once
among the top-10 results on all search engines except Google. This finding
suggests that Google might downgrade Youtube’s direct competitors, however an
analysis based on a broader spectrum of queries is necessary to estimate the
scope and persistence of this result.

##  5\. Conclusions and future work

Top 10 outputs of video search for most of search engines except Google show
limited source diversity. By relying on average on 2-3 unique sources to
retrieve top results for English queries, search engines create a situation in
which users’ information choices are shaped by a few content providers. This
raises concerns about search engines facilitating consolidation of power on
the information markets.

The only exception among the five search engines examined is Google, where the
degree of source diversity is almost twice as high. This effect can be
attributed to Google putting substantial effort into diversifying search
results in response to earlier criticisms. With 6 unique domains per 10 top
results, Google follows its declared principle of having no more than 2
results coming from the same domain in the top results (noa, 2019). The
finding also suggests that low diversity on other search engines is likely
attributed to the absence of diversification mechanisms which Google
implements.

The importance of integrating diversification measurements is highlighted by
high degree of source diversity bias. The top results for all search engines
(except Yandex in English and Google) are dominated by one platform, namely
YouTube. Its systematic prevalence reinforces the platform’s almost
monopolistic status. It is problematic considering that the platform is
already used as a major news source among certain shares of the population
(Kalogeropoulos, 2018), despite earlier audits demonstrating that its
algorithms might lead to user radicalization (Ribeiro et al., 2020) and,
affected by users’ viewing history, aggressively promote pseudoscientific
content to users who have watched pseudoscientific videos before (Papadamou et
al., 2020). Search concentration around Youtube can only help cement its
monopoly and the associated effects.

Google fairs better than other search engines in terms of domain-level source
diversity and, at a first glance, does not exhibit own-content bias since
Youtube is less prominent in its results than on other search engines.
However, in our sample, Google is the only search engine that did not provide
any results from Youtube’s major competitors. It is thus plausible that Google
indeed, as media reports suggested (West, 2020), might be downgrading the
results coming from the major competitors of Youtube thus exhibiting own-
content bias manifested not in promoting Youtube but in lowering the
prominence of its competitors in search results. However, it could also be
that the obtained results with regards to Google and the absence of Youtube’s
competitors in its outputs are specific to the topics addressed by our sample
of queries and is absent in other contexts. Hence, to make a robust conclusion
about the presence or absence of own-content bias in Google’s video search
results, further studies encompassing broader sets of queries are necessary.
We suggest that our observations highlight the need for such studies.

Further, the present analysis is based on a snapshot experiment on a limited
selection of queries. We believe that our findings warrant subsequent
longitudinal audits of video search results to assess the persistence of our
observations and, potentially, the changes that occur overtime. Such audits
are crucial to inform the decisions of policy-makers and regulators. This is
especially timely and pressing given the recent anti-trust cases against
Google (Edelman, [n.d.]b) and calls for putting tech giants under scrutiny,
among other, in the context of their market power.

## References

  * (1)
  * noa ([n.d.]a) [n.d.]a.  2020 Edelman Trust Barometer.  https://www.edelman.com/trustbarometer
  * noa ([n.d.]b) [n.d.]b.  Search Engine Market Share Worldwide.  https://gs.statcounter.com/search-engine-market-share
  * noa ([n.d.]c) [n.d.]c.  YouTube Dominates Google Video in 2020.  https://moz.com/blog/youtube-dominates-google-video-results-in-2020
  * noa (2019) 2019\.  Google search update aims to show more diverse results from different domain names.  https://searchengineland.com/google-search-update-aims-to-show-more-diverse-results-from-different-domain-names-317934
  * Battelle (2005) John Battelle. 2005\.  _The Search: How Google and Its Rivals Rewrote the Rules of Business and Transformed Our Culture_.  Portfolio.  Google-Books-ID: FR9PAAAAMAAJ.
  * Ciampaglia et al. (2018) Giovanni Luca Ciampaglia, Azadeh Nematzadeh, Filippo Menczer, and Alessandro Flammini. 2018.  How algorithmic popularity bias hinders or promotes quality.  _Scientific Reports_ 8, 1 (Oct. 2018), 15951\.  https://doi.org/10.1038/s41598-018-34203-2 Number: 1 Publisher: Nature Publishing Group.
  * Diakopoulos et al. ([n.d.]) Nicholas Diakopoulos, Daniel Trielli, Jennifer Stark, and Sean Mussenden. [n.d.].  I Vote For—How Search Informs Our Choice of Candidate.  ([n. d.]), 22.
  * Edelman ([n.d.]a) Benjamin Edelman. [n.d.]a.  BIAS IN SEARCH RESULTS?: DIAGNOSIS AND RESPONSE.  7 ([n. d.]), 17\.
  * Edelman ([n.d.]b) Gilad Edelman. [n.d.]b.  Google’s Antitrust Cases: A Guide for the Perplexed.  _Wired_ ([n. d.]).  https://www.wired.com/story/google-antitrust-lawsuits-explainer/
  * Erickson et al. (2018) Thane M. Erickson, Adam P. McGuire, Gina M. Scarsella, Tara A. Crouch, Jamie A. Lewis, Ashley P. Eisenlohr, and Tasha J. Muresan. 2018\.  Viral videos and virtue: Moral elevation inductions shift affect and interpersonal goals in daily life.  _The Journal of Positive Psychology_ 13, 6 (Nov. 2018), 643–654.  https://doi.org/10.1080/17439760.2017.1365163 Publisher: Routledge _eprint: https://doi.org/10.1080/17439760.2017.1365163.
  * Goldman (2008) E. Goldman. 2008\.  Search Engine Bias and the Demise of Search Engine Utopianism.  In _Web Search: Multidisciplinary Perspectives_ , Amanda Spink and Michael Zimmer (Eds.). Springer, Berlin, Heidelberg, 121–133.  https://doi.org/10.1007/978-3-540-75829-7_8
  * Grimmelmann (2010) James Grimmelmann. 2010\.  Some Skepticism About Search Neutrality.  _Faculty Scholarship_ (Jan. 2010).  https://digitalcommons.law.umaryland.edu/fac_pubs/1417
  * Haim et al. (2017) Mario Haim, Florian Arendt, and Sebastian Scherr. 2017\.  Abyss or Shelter? On the Relevance of Web Search Engines’ Search Results When People Google for Suicide.  _Health Communication_ 32, 2 (Feb. 2017), 253–258.  https://doi.org/10.1080/10410236.2015.1113484
  * Hannak et al. (2013) Aniko Hannak, Piotr Sapiezynski, Arash Molavi Kakhki, Balachander Krishnamurthy, David Lazer, Alan Mislove, and Christo Wilson. 2013\.  Measuring personalization of web search. In _Proceedings of the 22nd international conference on World Wide Web - WWW ’13_. ACM Press, Rio de Janeiro, Brazil, 527–538.  https://doi.org/10.1145/2488388.2488435
  * Ieong et al. (2012) Samuel Ieong, Nina Mishra, Eldar Sadikov, and Li Zhang. 2012\.  Domain bias in web search. In _Proceedings of the fifth ACM international conference on Web search and data mining_ _(WSDM ’12)_. Association for Computing Machinery, New York, NY, USA, 413–422.  https://doi.org/10.1145/2124295.2124345
  * Introna and Nissenbaum (2000) Lukas Introna and Helen Nissenbaum. 2000.  Shaping the Web: Why the Politics of Search Engines Matters.  _The Information Society_ 16, 3 (July 2000), 169–185.  https://doi.org/10.1080/01972240050133634 Publisher: Routledge _eprint: https://doi.org/10.1080/01972240050133634.
  * Izawa ([n.d.]) Michiko Izawa. [n.d.].  WHAT MAKES VIRAL VIDEOS VIRAL?: ROLES OF EMOTION, IMPRESSION, UTILITY, AND SOCIAL TIES IN ONLINE SHARING BEHAVIOR.  ([n. d.]), 56.
  * Jiang (2014a) Min Jiang. 2014a.  The business and politics of search engines: A comparative study of Baidu and Google’s search results of Internet events in China.  _New Media & Society_ 16, 2 (March 2014), 212–233.  https://doi.org/10.1177/1461444813481196 Publisher: SAGE Publications.
  * Jiang (2014b) Min Jiang. 2014b.  Search Concentration, Bias, and Parochialism: A Comparative Study of Google, Baidu, and Jike’s Search Results From China.  _Journal of Communication_ 64, 6 (2014), 1088–1110.  https://doi.org/10.1111/jcom.12126 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/jcom.12126.
  * Kalogeropoulos (2018) Antonis Kalogeropoulos. 2018\.  Online News Video Consumption.  _Digital Journalism_ 6, 5 (May 2018), 651–665.  https://doi.org/10.1080/21670811.2017.1320197 Publisher: Routledge _eprint: https://doi.org/10.1080/21670811.2017.1320197.
  * Kay et al. (2015) Matthew Kay, Cynthia Matuszek, and Sean A. Munson. 2015\.  Unequal Representation and Gender Stereotypes in Image Search Results for Occupations. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ _(CHI ’15)_. Association for Computing Machinery, New York, NY, USA, 3819–3828.  https://doi.org/10.1145/2702123.2702520
  * Kulshrestha et al. (2019) Juhi Kulshrestha, Motahhare Eslami, Johnnatan Messias, Muhammad Bilal Zafar, Saptarshi Ghosh, Krishna P. Gummadi, and Karrie Karahalios. 2019.  Search bias quantification: investigating political bias in social media and web search.  _Information Retrieval Journal_ 22, 1 (April 2019), 188–227.  https://doi.org/10.1007/s10791-018-9341-2
  * Makhortykh et al. (2020) Mykola Makhortykh, Aleksandra Urman, and Roberto Ulloa. 2020\.  How search engines disseminate information about COVID-19 and why they should do better.  _Harvard Kennedy School Misinformation Review_ 1, COVID-19 and Misinformation (May 2020).  https://doi.org/10.37016/mr-2020-017
  * Mittelstadt (2016) Brent Mittelstadt. 2016\.  Automation, Algorithms, and Politics| Auditing for Transparency in Content Personalization Systems.  _International Journal of Communication_ 10, 0 (Oct. 2016), 12.  https://ijoc.org/index.php/ijoc/article/view/6267 Number: 0.
  * Mowshowitz and Kawaguchi (2002) Abbe Mowshowitz and Akira Kawaguchi. 2002.  Assessing bias in search engines.  _Information Processing & Management_ 38, 1 (Jan. 2002), 141–156.  https://doi.org/10.1016/S0306-4573(01)00020-6
  * Otterbacher et al. (2017) Jahna Otterbacher, Jo Bates, and Paul Clough. 2017\.  Competent Men and Warm Women: Gender Stereotypes and Backlash in Image Search Results. In _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_ _(CHI ’17)_. Association for Computing Machinery, New York, NY, USA, 6620–6631.  https://doi.org/10.1145/3025453.3025727
  * Pan et al. (2007) Bing Pan, Helene Hembrooke, Thorsten Joachims, Lori Lorigo, Geri Gay, and Laura Granka. 2007\.  In Google We Trust: Users’ Decisions on Rank, Position, and Relevance.  _Journal of Computer-Mediated Communication_ 12, 3 (2007), 801–823.  https://doi.org/10.1111/j.1083-6101.2007.00351.x _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1083-6101.2007.00351.x.
  * Papadamou et al. (2020) Kostantinos Papadamou, Savvas Zannettou, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, and Michael Sirivianos. 2020\.  "It is just a flu": Assessing the Effect of Watch History on YouTube’s Pseudoscientific Video Recommendations.  _arXiv:2010.11638 [cs]_ (Nov. 2020).  http://arxiv.org/abs/2010.11638 arXiv: 2010.11638.
  * Ribeiro et al. (2020) Manoel Horta Ribeiro, Raphael Ottoni, Robert West, Virgílio A. F. Almeida, and Wagner Meira. 2020.  Auditing radicalization pathways on YouTube. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ _(FAT* ’20)_. Association for Computing Machinery, New York, NY, USA, 131–141.  https://doi.org/10.1145/3351095.3372879
  * Stocking et al. (2020) Galen Stocking, Patrick van Kessel, Michael Barthel, Katerina Eva Matsa, and Maya Khuzam. 2020\.  Many Americans Get News on YouTube, Where News Organizations and Independent Producers Thrive Side by Side.  https://www.journalism.org/2020/09/28/many-americans-get-news-on-youtube-where-news-organizations-and-independent-producers-thrive-side-by-side/
  * Trielli and Diakopoulos (2019) Daniel Trielli and Nicholas Diakopoulos. 2019.  Search as News Curator: The Role of Google in Shaping Attention to News Information. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ _(CHI ’19)_. Association for Computing Machinery, Glasgow, Scotland Uk, 1–15.  https://doi.org/10.1145/3290605.3300683
  * Unkel and Haim (2019) Julian Unkel and Mario Haim. 2019.  Googling Politics: Parties, Sources, and Issue Ownerships on Google in the 2017 German Federal Election Campaign.  _Social Science Computer Review_ (Dec. 2019), 0894439319881634\.  https://doi.org/10.1177/0894439319881634
  * West (2020) Sam Schechner West, Kirsten Grind and John. 2020.  Searching for Video? Google Pushes YouTube Over Rivals.  _Wall Street Journal_ (July 2020).  https://www.wsj.com/articles/google-steers-users-to-youtube-over-rivals-11594745232
  * Zavadski and Toepfl (2019) Andrei Zavadski and Florian Toepfl. 2019.  Querying the Internet as a mnemonic practice: how search engines mediate four types of past events in Russia.  _Media, Culture & Society_ 41, 1 (Jan. 2019), 21–37.  https://doi.org/10.1177/0163443718764565 Publisher: SAGE Publications Ltd.