# Leveraging Wikidata’s edit history in knowledge graph refinement tasks

Alejandro Gonzalez-Hevia  [<EMAIL_ADDRESS>Daniel Gayo-Avello  [
<EMAIL_ADDRESS>Department of Computer Science, University of Oviedo, Spain

###### Abstract

Knowledge graphs have been adopted in many diverse fields for a variety of
purposes. Most of those applications rely on valid and complete data to
deliver their results, pressing the need to improve the quality of knowledge
graphs. A number of solutions have been proposed to that end, ranging from
rule-based approaches to the use of probabilistic methods, but there is an
element that has not been considered yet: the edit history of the graph. In
the case of collaborative knowledge graphs (e.g., Wikidata), those edits
represent the process in which the community reaches some kind of fuzzy and
distributed consensus over the information that best represents each entity,
and can hold potentially interesting information to be used by knowledge graph
refinement methods. In this paper, we explore the use of edit history
information from Wikidata to improve the performance of type prediction
methods. To do that, we have first built a JSON dataset containing the edit
history of every instance from the 100 most important classes in Wikidata.
This edit history information is then explored and analyzed, with a focus on
its potential applicability in knowledge graph refinement tasks. Finally, we
propose and evaluate two new methods to leverage this edit history information
in knowledge graph embedding models for type prediction tasks. Our results
show an improvement in one of the proposed methods against current approaches,
showing the potential of using edit information in knowledge graph refinement
tasks and opening new promising research lines within the field.

###### keywords:

Semantic Web, Wikidata , Edit History , Knowledge Graph Refinement , Type
Prediction , Knowledge Graph Embeddings

††journal: Journal of Web Semantics

url]www.alejgh.com

url]www.danigayo.info

##  1 Introduction

Different fields have incorporated the use of domain-specific knowledge graphs
during recent years to solve their tasks. Some concrete examples of such
domain-specific tasks include performing investment analysis [1], managing
diseases and symptoms from medical records [2], or automatically generating
test cases for software projects [3], among many others. Furthermore, the
emergence of several open and general-purpose knowledge graphs, such as
DBPedia [4] and Wikidata [5], has also attracted new communities closer to the
Semantic Web by allowing them to exploit this structured information for many
different applications. It goes without saying that most of those applications
rely on the correctness and completeness of the data in the knowledge graph to
deliver their results.

It is therefore crucial to ensure a high level of quality for those knowledge
graphs. This has led to works that define quality metrics and dimensions to
better analyze and understand data quality [6]. Those works reveal the
existence of constraint violations and missing information in modern knowledge
graphs, among other quality problems [7, 8].

Therefore, a number of different approaches have been proposed to improve the
quality of knowledge graphs. Some of them follow a deductive approach, where a
set of rules or constraints that each triple must follow are defined to
enforce data quality [9, 10]. Other proposals follow an inductive approach,
using predictive models or alternative probabilistic methods to try to fill
incomplete information or fix errors in the knowledge graph [11].

However, in the specific case of collaborative knowledge graphs like Wikidata,
there is an element that has not been fully explored yet: its edit history
information. One of the main features of Wikidata, telling it apart from other
open general-purpose knowledge graphs, is its collaborative approach: anyone
can start from scratch editing entities in Wikidata. At the time of this
writing, there have been 1,640,933,943 edits made to Wikidata. In those edits,
the community has progressively built a consensus –fuzzy and somewhat
distributed among the editors– over the information that best represents each
entity within the knowledge graph, while also capturing the natural evolution
of those entities across time.

In this paper we explore the possibilities of leveraging edit information to
refine the contents of a knowledge graph, laying the foundations for future
work in this area. Our main contributions are:

  1. 1.

The creation of a JSON dataset containing the complete edit history of every
entity of the 100 most important classes in Wikidata, following Wikidata’s
data model (Section 3).

  2. 2.

An analysis of the main editing patterns from contributors, edits made by
class, and divisiveness in Wikidata based on the edit information (Section 4).
This information is analyzed with a focus on its possible applications to
knowledge graph refinement tasks.

  3. 3.

The proposal of two approaches to leverage edit history data in type
prediction tasks: the use of edits in the negative sampling process of
knowledge graph embeddings models, and using the edit information as labeled
data fed to a classifier (Section 5.1). We perform an evaluation of these
approaches against a set of baselines and analyze the impact of using edit
history information in both approaches.

  4. 4.

An RDF dataset containing edit history information about Wikidata, following a
custom data format where each operation and revision is serialized to the
graph. This RDF dataset is also available without edit history information,
and can serve as a baseline to measure the impact of using edit history data
in knowledge graph refinement models (Section 5.2).

The rest of this paper is structured as follows. In the next section, we go
over Wikidata’s data model and provide a formal definition of a knowledge
graph and its edits. These concepts are needed to better understand the
successive aspects of our work. Section 3 goes over the process of acquiring
edit information from Wikidata. This edit information is explored in section
4. In Section 5, we propose two new methods to leverage edit information to
improve existing knowledge graph embedding models, and we evaluate their
performance with respect to current approaches. Related work is reviewed in
Section 6. And, finally, in Section 7 we present the conclusions of this work
and future research directions.

##  2 Background

###  2.1 Wikidata data model

Entities are the basic building block of Wikidata’s data model. There are two
different types of entities: items and properties. Each entity is given a
unique incremental numeric id, with items being prefixed by a ‘Q’ and
properties by a ‘P’.

A statement is composed of a property and a value assigned to that property,
optionally having $1$ to $n$ qualifiers and $1$ to $n$ references. In the rest
of this paper we will use the term simple statement to refer to statements
that are just composed of a property and a value. A statement group is the set
of statements that an item has of a given property. Each entity in Wikidata is
composed of $0$ to $n$ statement groups.

Qualifiers are used to give further information about a given statement (e.g.,
the point in time when the statement holds true). Each qualifier is also
composed of a property and a value assigned to that property. The combination
of a property, value, and qualifier is called a claim. References are also
property-value pairs, and they provide the source that validates a statement.

Aliases, descriptions, and labels constitute the fingerprint of an entity.
These elements are mapped internally to skos:altLabel, schema:description, and
rdfs:label URIs in the RDF representation of an entity. The combination of
description and label of an entity in a given language must be unique. An
entity can have multiple aliases but only a single description and label for a
given language.

Snaks are the most basic information structure in Wikidata, and provide
information about the value of a property. There are three types of snaks:
value, somevalue, and novalue. Value snaks indicate that the property has a
known value, which is then represented using Wikidata’s available
datatypes111More information available at
https://www.wikidata.org/wiki/Special:ListDatatypes. Somevalue snaks indicate
that the property has a value but its value is unknown222Somevalue snaks are
represented with blank nodes in the RDF serialization of the data model..
Finally, novalue snaks indicate that the property does not have a value.

Wikidata also introduces three ranks which can be assigned to each statement:
preferred, deprecated, and normal. These ranks are generally used to decide
which statements must be returned when querying Wikidata, and also to clean up
its user interface when exploring an entity. Statements can also have an order
within each statement group. Although the order of statements within each rank
is not relevant, it can be changed by users.

All these elements are internally serialized in Wikidata to JSON and different
RDF serialization formats. It must be noted that in this section we have
covered all the elements that are mentioned in the rest of this paper, but the
list is not exhaustive. Additional information about Wikidata’s data model and
its serialization is available
online333https://www.mediawiki.org/wiki/Wikibase/DataModel?tableofcontents=0.

###  2.2 Revisions

Wikidata allows any user to edit entities. Therefore, each entity is composed
of a revision history that holds all the changes made to the entity by
Wikidata contributors. Any of the elements described in the previous section
can be changed, and a single revision may hold changes to any number and
combination of elements of an entity. Throughout this paper we will use the
terms edition and revision interchangeably, following Wikidata’s terminology.

A revision also contains additional metadata, including its timestamp, author
of the revision, tags, and a description. Tags are usually used to indicate
the device from which the revision was made (or the tool that made the edit,
if it was an automated process) and also to indicate the cause of the edit444A
list of the most common tags can be accessed at
https://www.wikidata.org/wiki/Special:Tags (e.g., to revert vandalism).

In the context of this paper we will use the term operation to refer to a
single modification made to an entity’s element in a revision. One revision
may be composed of $1$ to $n$ operations. An operation may represent the
addition or removal of a single element from the entity. For the sake of
simplicity, we will also consider replacements as operations, which are a
combination of an addition and removal operation to an entity.

Wikidata allows restoring and undoing revisions made to an entity. Restoring
allows users to undo all the edits made to an entity up to the selected
restoration state. Undoing is more versatile, since it undoes from $1$ to $n$
edits selected by the user, which do not need to be consecutive edits like in
the restoration process. In none of those cases the revisions being undone are
removed from the revision history of the entity. A new revision is made
instead, which includes the necessary operations to undo the selected
revisions.

Edits can be manually removed by Wikidata administrators under specific
circumstances. These include revisions that contain private information, a
violation of copyright, or personal attacks of a serious nature.

###  2.3 Formal definitions

We will now formally introduce the main elements used throughout this paper.
Let $\mathcal{I}$, $\mathcal{L}$ and $\mathcal{B}$ be disjoint countably
infinite sets of IRIs, literals and blank nodes respectively. A knowledge
graph can be formally defined from a static point of view as a set of triples
$(s,p,o)\in(\mathcal{I}\cup\mathcal{B})\times\mathcal{I}\times(\mathcal{I}\cup\mathcal{L}\cup\mathcal{B})$.

From a dynamic point of view, a knowledge graph is built from a sequence of
operations $Op=\\{op_{j}:1\leq j\leq\infty\\}$. Each operation $op_{j}$ is
composed of a triple
$t=(s,p,o):s\in(\mathcal{I}\cup\mathcal{B})\times\mathcal{I}\times(\mathcal{I}\cup\mathcal{L}\cup\mathcal{B})$.
$Op^{+}=\\{t_{1},t_{2},...,t_{n}\\}$ represents the set of addition operations
of the graph, while
$Op^{-}=\\{t^{\prime}_{1},t^{\prime}_{2},...,t^{\prime}_{m}\\}$ represents the
set of removal operations. The set of all operations is therefore defined as
$Op=Op^{+}\cup Op^{-}$. A knowledge graph is built out of $n$ operations, with
$K_{i}$ representing the state of the graph after applying all operations up
to operation $i$. Applying an addition operation $op^{+}_{i+1}=(s,p,o)$ to a
graph $K_{i}$ results in graph $K_{i+1}=K_{i}\cup(s,p,o)$. On the other hand,
applying a removal operation $op^{-}_{i+1}=(s,p,o)$ to a graph $K_{i}$ results
in graph $K_{i+1}=K_{i}\setminus(s,p,o)$. The final state of a knowledge graph
can be obtained by a successive application of all its operations.

##  3 Extracting edit history data from Wikidata

We now present the approach followed to extract the edit history information
from Wikidata to conduct our experiments.

###  3.1 Subset selection

Wikidata was composed of 97,795,169 entities at the time of this writing, with
more than 1,640,933,943 revisions in total made by users555Source:
https://www.wikidata.org/wiki/Wikidata:Statistics. Given that working with the
entire Wikidata revision history could be too computationally expensive to
validate our proposal, we extracted a subset to conduct our experiments.

This subset is composed of instances from the most important Wikidata classes.
To that end, we have computed the ClassRank [12] score of every class in
Wikidata, choosing the top 100 classes with the highest score. Then, we
extracted the edit history information of every entity that is an instance of
any of those classes. We preferred choosing the most important classes for our
experiments over producing a random sample since –in general– entities
belonging to central classes receive more attention from the community, and
are therefore more promising for exploiting their edit history information.

To run ClassRank we defined the P31 property (instance of) of Wikidata as a
class-pointer666The class-pointer is used by ClassRank to fetch those entities
from Wikidata that are classes.. The ClassRank score of a class is computed by
aggregating the PageRank [13] scores of all its instances. However, since
computing the PageRank score of every entity in Wikidata was too
computationally expensive, we used a set of pre-computed PageRank scores.
These scores were obtained using the
Danker777https://github.com/athalhammer/danker tool, which periodically
computes the PageRank score of every existing entity in Wikipedia [14]. The
scores are then mapped from Wikipedia pages to their respective Wikidata
entities, getting an approximation of their PageRank value888These dumps can
be accessed at https://danker.s3.amazonaws.com/index.html.

The 20 most important classes based on their ClassRank score can be seen in
table 1. These results were then filtered manually, since some of those
entities could be considered Wikidata classes at the ontological level, but
not at a conceptual one. To do so, we have removed those classes that
contained the term “Wikimedia” in their labels, since they are used to
organize Wikimedia content but do not represent classes at a conceptual level.

Table 1: Top 20 most important classes based on their ClassRank score Name | ClassRank score | Number of instances
---|---|---
human | 2,167,439 | 3,873,812
Wikimedia category | 1,057,559 | 2,207,283
sovereign state | 837,883 | 203
taxon | 755,681 | 1,962,491
country | 746,208 | 193
point in time with respect to recurrent timeframe | 635,401 | 3,273
calendar year | 499,551 | 666
big city | 321,893 | 3,238
human settlement | 321,637 | 512,417
Wikimedia disambiguation page | 317,631 | 1,294,218
Wikimedia administration category | 302,196 | 12,146
Wikimedia list article | 287,605 | 301,370
language | 284,433 | 8,894
modern language | 257,324 | 6,875
city | 247,022 | 8,650
academic discipline | 204,426 | 1,603
time zone named for a UTC offset | 170,254 | 72
metacategory in Wikimedia projects | 167,871 | 2,545
republic | 165,881 | 78
capital | 161,384 | 388
taxonomic rank | 160,822 | 67

The final subset is composed of 89 classes and 9.3 million instances –around
10% of the total number of entities in Wikidata. Although this subset is
composed of just a 10% of the entities in Wikidata, its size is around a 35%
of the total size of Wikidata. This can be explained due to the fact that the
most important entities have, in general, more content introduced by the
community with respect to lesser important classes.

###  3.2 Data extraction

Wikidata periodically releases public dumps of its contents999Available at
https://dumps.wikimedia.org/wikidatawiki/. We have selected the pages-meta-
history dumps to extract the edit history of every entity from our subset,
since these dumps are the only ones containing every revision of each entity
and not just their final content. This dataset is composed of several XML
files containing metadata of every revision made to each entity, and also a
JSON blob with the complete content of the entity after each revision. Our
final dataset is built from the pages-meta-history dumps from 2021-11-01.

Since working with the complete content of every entity after each revision
leads to a lot of redundant entity information, we made some preprocessing
steps to reduce the dataset size. Instead of storing the complete JSON content
of each entity after every revision $r$, we computed the diff between the JSON
content in the previous revision ($r_{t-1}$) and the current one ($r_{t}$).
These diffs are stored in the JSON Patch
format101010https://datatracker.ietf.org/doc/html/rfc6902, therefore allowing
the reconstruction of the entity contents after any revision. To obtain the
complete JSON content of an entity at revision $r_{t}$ we just need to apply
the patches of every previous revision up to $r_{t}$. A simplified example of
the decomposition of an entity in diffs is illustrated in figure 1.

Figure 1: Example of revision decompositions into JSON Patch. At the top of
the figure we show the simplified content of a Wikidata entity at each
revision $r$. At the bottom of the figure we show the decomposition of this
entity into diffs in JSON format.

In this process, the following revision data was extracted:

  * 1.

id: ID of the Wikidata revision.

  * 2.

parent_id: ID of the previous revision affecting the entity.

  * 3.

entity_id: ID of the entity altered by the revision.

  * 4.

timestamp: Date of the revision, following the ISO 8601 format (e.g.,
+2019-05-27T09:31:10Z).

  * 5.

username: Name of the user that made the revision.

  * 6.

comment: Comments of the revision, if they exist.

  * 7.

entity_diff: Diff of the revision, following the JSON Patch format previously
explained.

###  3.3 Indexing of data

The revision data previously fetched was then indexed into a MongoDB database
for further exploration. Table 2 shows the main statistics of the revisions
dataset that was indexed.

Table 2: Statistics of the edits dataset used in the experiments Dataset size: | 401GB
---|---
Number of entities: | 9,709,099
Number of revisions: | 443,116,607
Number of operations: | 986,678,861

A custom database architecture was defined in order to optimize the
performance of queries needed in the following experiments. The database was
split into two main collections: one for entities and another one for
revisions. The former contains the final complete JSON content of each entity,
while the latter contains the JSON diffs of each entity after each revision.
Although the final content of an entity can be obtained by applying all of its
revision diffs, this architecture allows for a performant exploration of the
revision data and also the current content of each entity.

This dataset has been released in order to promote its exploration by other
researchers [15].

##  4 Wikidata edits analysis

In this section we explore the edits dataset to exploit the dynamic aspects of
Wikidata. While Wikidata is usually explored from a static point of view
(i.e., having into account just the state of the knowledge graph at a point in
time $t$), having information about its changes lets us understand the
community’s approach when building the knowledge graph in fine-grained detail.

###  4.1 Life cycle of an entity

Figure 2 shows a graph of transitions from one edit operation applied to an
entity to the next one. Edge weights represent the number of transitions from
one operation to another, while node size represents the total for each state
–with higher weights representing a larger count. These transition and state
counts were accumulated across every entity in the dataset. In order to better
illustrate the most common transitions and reduce noise from the graph, those
transitions with a count lower than 10% of the total count of outgoing
transitions from a state were removed.

Figure 2: Most common transitions from one edit operation to another in the
entities lifetime. The purple circle represents the start in the life cycle of
an entity, and blue circles represent different operations applied to the
entity during its life cycle.

We can see how the central operation in this graph is adding a statement group
to an entity, followed by adding references to a claim. These are the most
prominent operations applied to entities in Wikidata. There is also a group of
three operations in the top right corner of the graph, which represents adding
single values to the entity: either a single statement, reference, or
qualifier. Both behaviors are expected, since a common practice in Wikidata is
to add appropriate references and qualifiers to a statement that has been
added.

In general, replacements or removals are not as common as addition operations.
However, these operations are usually connected between them (i.e., a
replacement or removal operation is usually followed by another replacement or
removal). While adding a statement may not imply adding a reference or
qualifier to it, removing a statement implies the removal of its qualifiers
and references. Therefore, some connections between a removal state and the
next one are expected.

Finally, changing the rank or order of statements, references, or qualifiers
is also an infrequent operation within Wikidata.

###  4.2 Edit information

##### Operations applied to the most important classes

Figure 3 shows the top 10 classes with the highest average number of
operations performed on its instances. These frequencies are split into
additions, removals, and replacements. We can see how in the classes that have
a higher operation count replacements are the most prominent operation (over
additions and removals). As the number of operations keeps decreasing,
replacements are less common and additions become the most common operation.
The ratio between additions and removals can be higher or smaller depending on
the class, which could reveal of the difficulty of reaching a consensus on the
content of its instances.

Another interesting insight from this chart is that some of the most important
classes based on their ClassRank score do not have a high number of average
operations per instance. In particular, the 2 most important classes (human
and taxon) do not appear in the top 10 classes with the most operations. One
possible explanation for this could be that instances of these classes do not
have as much information to be added as instances of other classes. Another
possibility could be that these classes contain a high number of instances
that are not edited as often, so the average number of operations is not as
high as in other classes.

Figure 3: Top 10 classes with the most number of operations performed on its
instances.

##### Most frequently removed properties of a class

Figure 4 shows the most removed properties of 4 classes from the dataset:
human, taxon, sovereign state, and big city. Properties with the name
‘[deleted property](PXX)’ refer to properties that were removed from Wikidata,
for which the original label cannot be retrieved111111The process of creating
and removing properties from Wikidata is more complex than creating entities
or adding statements to them. It can only be done by users who have the
‘Property Creator’ role, and involves creating a formal proposal and reaching
a consensus over the proposal..

We can see in the figure some differences in the removal behavior for
different classes. For some classes, like ‘taxon’ and ‘big city’, the most
removed properties are properties that were directly removed from Wikidata.
These properties do not reflect the level of consensus in the class, since
they are removed completely from Wikidata at an ontological level. The
remaining properties have a lower deletion rate, indicating that the
properties used in these classes are overall stable and rarely change.

Other classes, like ‘sovereign state’, have a high number of removals across
many properties –which are still existing in Wikidata. This means that, in
general, instances of that class have a set of properties that either lead to
controversy or are evolving naturally along the lifetime of the instance.

Finally, the ‘human’ class only has a property with more than $0.3$ deletions
per instance, which was removed from Wikidata. The rest of the properties have
a really low deletion rate. This follows the same trend that we have seen when
analyzing the number of operations per class, showing that although the human
class is the most important one in Wikidata –based on ClassRank– it does not
have a high average number of operations per instance.

A deep understanding of these results across every class could be beneficial
to analyze which properties are more controversial and which ones do not
experience as many deletions. This information could be used by recommendation
systems to aid editors by signaling those properties that are more
controversial and need special consideration when used.

(a) Class: Human

(b) Class: Taxon

(c) Class: Sovereign State

(d) Class: Big City

Figure 4: Most deleted properties of each class.

###  4.3 Conflict in Wikidata

Our last set of analyses of the edit history dataset revolves around the idea
of divisiveness in Wikidata. We have analyzed the edits dataset to detect
cases where an edit war between users occurs when editing an entity121212The
interested reader should consult [16] for more information on edit wars,
albeit in Wikipedia, not in Wikidata.. In our case we have defined an edit war
as a succession of edits in which a value for a property is added, the value
is removed or replaced by another different value in a later revision, and
finally the original value is added again to the entity. Certainly, such a
definition does not only cover the case of a conflict within the community on
the value that an entity should have for a given property, but also acts of
vandalism that are eventually reverted.

Figure 5 shows the properties with the most amount of edit wars in the
dataset. The property with the highest edit war count was ‘sex or gender’,
with a notable difference when compared to other properties. Typing entities
with the ‘instance of’ property is the next case with the highest amount of
edit wars. The remaining properties are more domain-specific and have a
considerably lower number of edit wars when compared to the top 2.

Figure 5: Top 15 properties with the highest amount of edit wars.

In figure 6 we can see the classes with the highest mean amount of edit wars
across its instances. Most edit wars occur in classes related to territories,
with only some exceptions not belonging to that domain (e.g., chemical
element, natural language…). This follows the same pattern we have seen
previously, with the ‘sovereign state’ class having a high number of deletions
and replacements of property values.

Figure 6: Top 15 classes with the highest amount of edit wars.

##  5 Type prediction leveraging edit data

In this section, we explore a potential sub-field within knowledge graph
refinement where this edit history data could be exploited: type prediction in
knowledge graphs. More specifically, our objective is predicting possible
values of the instance of (P31) property of a given entity. In the following
sections, we will explore our approach and the results obtained.

###  5.1 Proposed approaches

In this paper we analyze the impact of leveraging edit history information in
knowledge graph embedding models. We have decided to employ these models since
they are currently used in type prediction and other knowledge graph
refinement tasks.

Since these models usually employ a ‘static’ version of a graph (i.e., without
including any information about the editions that the graph underwent until
reaching its final version), our objective is to include information from the
edit history of the graph to obtain an improvement in their performance. To do
so, we propose the following two techniques in our experiments: using the
removed triples from the graph to enrich negative sampling methods, and
labeling each triple that was added or removed from the graph to train a
supervised model.

####  5.1.1 Negative sampling enriched by edit history information

Knowledge graph embedding models perform a process called negative sampling
when generating the embeddings, where negative triples –corruptions– are
generated. Those corruptions are used during the training process of the
embeddings as examples of negative triples. This generation of corruptions is
usually carried out randomly, under the premise that the total space of
possible triples ($|\mathcal{S}\times\mathcal{P}\times\mathcal{O}|$) is much
greater than the one of true triples, and therefore the possibility of
introducing false negatives is small.

However, in the revision history of an entity we have potentially useful
information to determine which corruptions could be generated: triples that
the community had considered invalid sometime along the life cycle of a given
entity. We can draw an analogy of this idea to the task of training an
automatic spell checker. One option could be to artificially generate typos on
the set of training words by randomly replacing, removing, or adding one
character to the word (basic negative sampling approach). However, a more
optimal approach could be to also use as a training sample real typos made by
humans on these words when writing on their devices (edit history based
negative sampling approach).

Therefore, we propose the following set of negative sampling methods based on
the edit history of a knowledge graph:

  * 1.

Edit history negative sampling. We start by selecting the set of negative
triples of each triple that needs to be corrupted. This set of negative
triples is obtained from all the removal operations in the knowledge graph
that share the same subject and predicate of the given positive triple. For
example, if we want to generate corruptions for the triple ‘DouglasAdams
occupation writer’, we would look into the edit history of the graph for
triples that were removed having ‘DouglasAdams’ as subject and ‘occupation’ as
predicate131313In these examples the corruption of objects of each triple is
shown. The corruption of subjects is analogous, where we instead select as
negative triples those that share the same predicate and object of the
positive triple.. Afterward, the set of negative triples is shuffled, and the
first $n$ elements are selected as the generated corruptions. If the set of
negative triples contains less than $n$ elements, the remaining corruptions
are generated randomly, following the traditional process. A variation of this
technique consists of generating the set of negative triples without including
those triples that were the subject of edit wars. In the following
experiments, we will refer to models using this type of negative sampling
techniques with the terms ‘edit hist’ y ‘edit hist (no edit wars)’, with the
former including edit wars in the set of negative triples while the later
omits them.

  * 2.

Inverse edit history negative sampling. This method is based on the opposite
hypothesis with respect to the previous one: triples that were removed by the
community represent a conflict of opinion about the value of a property, and
they do not entail that its value is necessarily invalid. Therefore, these
removed triples should not be added to the corruptions of the model since they
are triples that could be considered valid or, at least, partially related to
the entity that is being corrupted. For this reason, although we start from
the same set of negative triples explained previously, in this case the
objective is to avoid the generation of any corruption that belongs to that
set. Corruptions are generated randomly, following the traditional approach,
but if the generated corruption belongs to the set of negative triples the
corruption is generated again randomly until it does not belong to it.

The pseudo-code of the corruption fetching process and both negative sampling
approaches is shown in algorithm 1 and algorithm 2 respectively.

Input:

$t=(s,p,o)$ Triple that needs to be corrupted.

$Op^{-}=\\{t_{1},t_{2}...t_{m}\\}$ Set of removal operations of the knowledge
graph.

$omit\\_edit\\_wars\in\\{True,False\\}$ Whether edit wars should be omitted or
not.

Variables :

$F\subset Op^{-}$ Set of all filtered corruptions.

Output:

$T_{neg}\subset F$ Set of negative samples of $t$.

begin

$F\longleftarrow\emptyset$

for  _$t^{\prime}=(s^{\prime},p^{\prime},o^{\prime})\in Op^{-}$_ do

if  _$omit\\_edit\\_wars=False$ or not $HasEditWars(t^{\prime})$_ then

$F\longleftarrow F\cup\\{t^{\prime}\\}$

$T_{neg}\longleftarrow\\{(s^{\prime},p^{\prime},o^{\prime})\in
F:s^{\prime}=s,p^{\prime}=p\\}$

return $T_{neg}$

Algorithm 1 Fetching of corruptions from the edit history pseudo-code

Input:

$n\in\mathbb{N}$: Number of negative samples to be generated per triple.

$Op^{-}$ Set of removal operations of the knowledge graph.

$T=\\{t_{1},t_{2},...,t_{m}\\}$ Set of triples to be corrupted.

$omit\\_edit\\_wars\in\\{True,False\\}$ Whether edit wars should be omitted or
not.

Variables :

$T_{neg}$ Set of negative samples of triple $t$.

Output:

$C_{out}\subset(\mathcal{I}\cup\mathcal{B})\times\mathcal{I}\times(\mathcal{I}\cup\mathcal{L}\cup\mathcal{B})$
Generated corruptions.

Function  _InverseNegativeSampling( $T$, $Op^{-}$, $omit\\_edit\\_wars$,
$n$)_:

$C_{out}\longleftarrow\emptyset$

for  _$t=(s,p,o)\in T$_ do

$T_{neg}\longleftarrow FetchCorruptions(Op^{-},t,omit\\_edit\\_wars)$

for  _$i=1:n$_ do

do

$t^{\prime}\longleftarrow RandomCorruption(t)$

while  _$t^{\prime}\notin T_{neg}$_

$C_{out}\longleftarrow C_{out}\cup{t^{\prime}}$

return $C_{out}$

Function  _EditHistoryNegativeSampling( $T$, $Op^{-}$, $omit\\_edit\\_wars$,
$n$)_:

$C_{out}\longleftarrow\emptyset$

for  _$t=(s,p,o)\in T$_ do

$T_{neg}\longleftarrow FetchCorruptions(Op^{-},t,omit\\_edit\\_wars)$

while  _$|T_{neg}| <n$_ do

$T_{neg}\longleftarrow T_{neg}\cup RandomCorruption(t)$

$T_{neg}\longleftarrow Shuffle(T_{neg})$

$C_{out}\longleftarrow C_{out}\cup\\{t_{1},t_{2}...t_{n}:t_{i}\in T_{neg}\\}$

return $C_{out}$

Algorithm 2 Negative sampling methods pseudo-code

####  5.1.2 Transformation of non-supervised task to a supervised one

While the generation of knowledge graph embeddings is a non-supervised
technique where it is not necessary to label any triple from the knowledge
graph, having information about the edit history of the graph allows us to
convert this task into a supervised one.

To be more precise, from the set of addition operations $Op^{+}$ and the set
of removal operations $Op^{-}$, we can assign a negative label $y=0$ to each
triple $t\in Op^{-}$, while assigning a positive label $y=1$ to each triple
$t\in Op^{+}$. With this approach, we can train a predictive model from the
embeddings representation of each triple and their corresponding label
obtained from the edit history of the knowledge graph. In our experiments the
input of the classifiers will be the concatenation of the subject, predicate,
and object embeddings. In order to obtain a ranking of triples for evaluation
purposes, we have ordered the results by the log probability given by the
classifier of the results being positive.

###  5.2 Dataset

Knowledge graph embedding models work on a graph version of a dataset, while
our dataset indexed on MongoDB that has been previously explored is stored in
JSON format. Therefore, to run these models it was necessary to transform that
data into RDF.

The first step of that process is to transform the ‘static’ JSON content of
each entity to RDF. To do so, we transformed just the simple statements of
every entity, ignoring complex values like qualifiers and references. Since
the JSON dataset indexed in MongoDB is composed of millions of triples, a
subset of this data was selected to be converted to RDF. This subset has 300
randomly selected instances from each class indexed in MongoDB, resulting in
390.000 triples in the final graph.

We also created an additional RDF dataset containing the ‘dynamic’ information
of each entity (i.e., the edit history information). This dataset represents
each revision made to an entity, plus the atomic operations that compose the
revision. It is possible to build the static RDF graph by applying each
operation on the dynamic version, following the same principles as with the
JSON dataset.

Finally, both datasets (dynamic and static) were split into training,
validation, and test sets. Every entity is maintained on each set when making
the divisions, but in the training dataset we include the state of every
entity until a time step $t_{i}$, in the validation set their new state from a
time step $t_{i}$ until $t_{i+n}$, and in the test set its new state since
time step $t_{i+n}$ until now. In our particular case, the training set
includes the first 70% revisions of each entity (in chronological order), the
validation set includes an additional 15% (between the 70% and 85% revisions),
and the test set contains the last 15% of revisions.

###  5.3 Training

##### Unsupervised models

We have selected 3 knowledge graph embedding models to evaluate the impact of
our proposed negative sampling techniques: RotatE [17], TransE [18], and MuRE
[19]. Those models were selected based on the large-scale benchmarks done by
Ali et al. under the PyKEEN framework [20]. We have chosen those with a good
performance under the Yago dataset141414Due to it being the dataset more
closely related to the one used in this paper. and that can scale to large
dataset sizes without requiring large computational resources.

Hyperparameters were optimized for each model using the training and
validation sets. Since the total space of parameter combinations was too big
to be completely covered, a random search was made to find the best
parameters. Each model parameters were optimized for 80h, for a total of 240h
(10 days). Table 3 shows the final set of optimized parameters found for each
model. Regarding parameters specific to each model, TransE had an optimized
parameter $scoring\\_fct\\_norm=2$ and MuRE’s best value for the $l_{p}$ norm
parameter was $2$.

Table 3: Optimized hyperparameters of each unsupervised model Model | Embedding dim. | Num. epochs | Batch size | Learning rate | Num. negatives
---|---|---|---|---|---
RotatE | 768 | 13 | 64 | 0.009 | 5
TransE | 64 | 10 | 521 | 0.024 | 7
MuRE | 150 | 21 | 256 | 0.088 | 28

To make the most out of the available data in the dataset, each combination of
model and negative sampler was then trained on the union of both the training
and validation sets. That is, the final training set of each model contains
the 85% first revisions of each entity, while the last 15% of revisions are
reserved for the test set.

##### Supervised model

For the supervised version of this task we have first trained RDF2Vec
embeddings [21] on the dataset, and fed the embeddings to a Random Forest
classifier with the labeled data.

The same optimization and training process done with the unsupervised models
was followed with the supervised approach. In this case, we have also
optimized the parameters of both the RDF2Vec and the Random Forest models,
performing a random search over a space of parameters for each model. The
final set of parameters found for the RDF2Vec model was a vector size of $50$,
with $50$ training epochs, a maximum depth of $5$, and $50$ walks generated.
Regarding the random forest classifier, the set of parameters found was $75$
for the number of trees, a minimum of $2$ samples to perform a split, and the
$gini$ criterion to perform splits.

###  5.4 Evaluation

To measure the performance of each model in the test set we have used the
following metrics: mean rank (MR), mean reciprocal rank (MRR), and hits@1, 5,
and 10. We selected those entities which had a new class added to the test set
(i.e., a new value for the instance of property), and run each model to get
the ranked list of class suggestions for the entity. We have followed the
‘filtered setting’ proposed by Bordes et al. in the evaluation process by
removing those triples suggested by the model which had already appeared in
the train set (true triples) [18]. The final test sample is made of 4193
triples, which are the triples from the test dataset containing the instance
of (P31) property.

###  5.5 Results

####  5.5.1 Negative sampling techniques

The results obtained for each combination of unsupervised model and negative
sampling technique are shown in table 4. Results for the hits@5 measure across
every model combination are graphically shown in figure 7.

Table 4: Evaluation results of unsupervised models. ‘basic’ represents the basic negative sampler currently used in knowledge graph embedding models; ‘edits’ and ‘edits (no w.)’ refer to the proposed negative sampling techniques that generate corrupt triples from removals of the edit history, considering or omitting edit wars respectively; and ‘inv.’ refers to the proposed inverse edit history negative sampling, where triples that were removed from the knowledge graph cannot be generated as corruptions. | RotatE | TransE | MuRE
---|---|---|---
| basic | edits |  edits (no w.) | inv. | basic | edits |  edits (no w.) | inv. | basic | edits |  edits (no w.) | inv.
MR | 10819 | 21268 |  23577 | 7197 | 3204 | 2385 |  2756 | 2595 | 527 | 4650 |  3385 | 447
MRR | 0.177 | 0.175 |  0.211 | 0.260 | 0.091 | 0.043 |  0.055 | 0.046 | 0.204 | 0.063 |  0.085 | 0.237
hits@1 | 0.086 | 0.109 |  0.146 | 0.163 | 0.050 | 0.015 |  0.028 | 0.020 | 0.115 | 0.018 |  0.031 | 0.144
hits@5 | 0.300 | 0.241 |  0.280 | 0.382 | 0.121 | 0.055 |  0.072 | 0.054 | 0.294 | 0.090 |  0.122 | 0.330
hits@10 | 0.382 | 0.308 |  0.339 | 0.445 | 0.164 | 0.091 |  0.097 | 0.083 | 0.385 | 0.148 |  0.210 | 0.422
Figure 7: Hits@5 results of each unsupervised model.

We performed McNemar tests over each combination of models to prove whether
there are statistically significant differences ($pvalue\leq 0.01$) between
the results obtained [22]. To build the contingency table we measured the
accuracy (hits@1) of each model being compared, storing those cases when both
models guessed correctly the new class of an entity, those cases when both
models failed to guess the class, and the cases when one of the models guessed
the class correctly while the other did not. Therefore, each comparison will
measure whether both models have a similar proportion of errors in the test
set. Since we are interested in the differences between each negative sampler,
we only performed comparisons between the different negative samplers of each
model type.

Results of running the McNemar test over each pair of negative samplers show
the following statistically significant differences for the RotatE model:
basic against edits $[\chi^{2}(1)=15.00,\;pvalue=1\cdot 10^{-4}]$; basic
against edits (no wars) $[\chi^{2}(1)=79.27,\;pvalue=5.4\cdot 10^{-19}]$;
basic against inverse $[\chi^{2}(1)=144.27,\;pvalue=3.1\cdot 10^{-33}]$; edits
against edits (no wars) $[\chi^{2}(1)=39.75,\;pvalue=2.8\cdot 10^{-10}]$;
edits against inverse $[\chi^{2}(1)=72.20,\;pvalue=1.9\cdot 10^{-17}]$.

With regards to the TransE model, we obtain the following statistically
significant differences: basic against edits
$[\chi^{2}(1)=72.90,\;pvalue=1.3\cdot 10^{-17}]$; basic against edits (no
wars) $[\chi^{2}(1)=25.44,\;pvalue=4.5\cdot 10^{-7}]$; basic against inverse
$[\chi^{2}(1)=54.37,\;pvalue=1.6\cdot 10^{-13}]$; edits against edits (no
wars) $[\chi^{2}(1)=16.66,\;pvalue=4.4\cdot 10^{-5}]$;.

Regarding the MuRE model, there is a statistically significant difference
between every negative sampler, with the following detailed results: basic
against edits $[\chi^{2}(1)=274.69,\;pvalue=1\cdot 10^{-61}]$; basic against
edits (no wars) $[\chi^{2}(1)=206.86,\;pvalue=6.6\cdot 10^{-47}]$; basic
against inverse $[\chi^{2}(1)=20.48,\;pvalue=6.0\cdot 10^{-6}]$; edits against
edits (no wars) $[\chi^{2}(1)=12.69,\;pvalue=3.6\cdot 10^{-4}]$; edits against
inverse $[\chi^{2}(1)=393.26,\;pvalue=1.6\cdot 10^{-87}]$; edits (no wars)
against inverse $[\chi^{2}(1)=307.65,\;pvalue=7.1\cdot 10^{-69}]$.

The remaining combinations of negative samplers do not show statistically
significant differences ($pvalue>0.01$), with the following detailed results:
RotatE edits (no wars) and RotatE inverse $[\chi^{2}(1)=6.01,\;pvalue=0.014]$;
TransE edits and TransE inverse $[\chi^{2}(1)=2.58,\;pvalue=0.10]$; TransE
edits (no wars) and TransE inverse $[\chi^{2}(1)=5.80,$ $\;pvalue=0.016]$.

In order to report the impact of our proposed negative samplers we also follow
the evaluation reporting approach proposed by Sparck-Jones [23]: performance
differences are regarded as non-relevant if they are lower than $5\%$; as
noticeable if they are between $5-10\%$; and as material if they are greater
than $10\%$.

We can observe from the results that the use of the inverse edit history
negative sampling technique has a positive effect on both the RotatE and MuRE
models across every measure, having a material and noticeable impact
respectively. However, with the TransE model this is not the case, with the
performance of the basic negative sampler improving that of the proposed
negative samplers across almost every measure. The only exception is the mean
rank measure, where the basic negative sampler has on average a $24.68\%$
higher mean rank score than the proposed approaches. It is important to note
that the TransE model results are in general considerably worse than the
RotatE and MuRE ones, regardless of the negative sampler being used. This
could be due to not finding an optimal set of parameters for the model, or to
the model having a worse performance in this dataset with respect to the other
alternatives. In general, using the proposed inverse edit history negative
sampler seems to influence positively in the results, improving the
performance of 2 out of the 3 models being evaluated and obtaining the 2 best
results in terms of MRR and hits@k.

On the other hand, an improvement over the basic approach is not as clear in
the case of the edit history negative samplers. If we consider the RotatE
model, both edit history negative samplers still perform better than the basic
approach in terms of hits@1, but the improvements in performance are not as
high as in the case of the inverse negative sampler. However, in the remaining
models the performance of the edit history negative samplers is worse than the
basic sampling approach across almost every metric. This could be attributed
to the fact that, in some cases, a value that was removed from an entity may
actually be true –or at least, partially related to the domain of the entity
being affected.

We are going to illustrate this with a real example from the dataset: the
entity representing MAN Truck & Bus151515https://www.wikidata.org/wiki/Q708667
was initially an instance of
‘company’161616https://www.wikidata.org/wiki/Q783794 in Wikidata, but this was
later removed and replaced by the
‘business’171717https://www.wikidata.org/wiki/Q4830453 class. Although there
are some legal details that differentiate both terms, they are still really
close from a conceptual point of view, especially for Wikidata contributors
without expertise in business law. In cases like the one mentioned above,
generating the removed triple (MAN instanceof company) as a corruption could
add noise to the model. This could also be a reason for the good performance
of the inverse negative sampling approach, since it avoids those cases.

It is also crucial to understand that there is a random component to the edit
history negative sampling methods. Since we are fetching $n$ random negative
triples for each triple that needs to be corrupted, in some cases we may be
fetching true negative triples (e.g., an edit that removed an act of
vandalism) while in other cases we may be getting negative triples with a
close relation to the entity conceptually (like the MAN Truck & Bus example
mentioned above) or even a removal of a completely valid triple. This can
explain some of the variability in the performance of the edit history
negative samplers and leads to the need of further analysis which is addressed
in the future work section.

Another interesting insight from these results is that omitting edit wars is
beneficial for the performance of the edit history negative samplers across
every model. This is intuitively expected, since values that cause conflict
between the community should not be added –in general– as false triples to the
model. These values are more commonly associated with information that is not
clear whether an entity should have or not, and therefore are not usually
related to incorrect information. Furthermore, although these values were
removed in the training set, since they were part of edit wars it is more
probable that they will be part of the added triples in the test set.
Establishing these triples as false in our negative sampling approach will
decrease their suggested ranking in the testing phase, lowering the
performance of the model.

####  5.5.2 Supervised task

Table 5 shows the results obtained by the supervised approach against each
unsupervised model with its best negative sampler.

The same process was followed to determine whether there are statistically
significant differences between the error rates of each model. Results of the
McNemar test showed a statistically significant difference across every
combination of models, with the following detailed results: RotatE against
TransE $[\chi^{2}(1)=15.00,\;pvalue=1\cdot 10^{-4}]$; RotatE and MuRE
$[\chi^{2}(1)=79.27,\;pvalue=5.4\cdot 10^{-19}]$; RotatE and RDF2Vec + Random
Forest $[\chi^{2}(1)=15.00,\;pvalue=1\cdot 10^{-4}]$; TransE and MuRE
$[\chi^{2}(1)=79.27,\;pvalue=5.4\cdot 10^{-19}]$; TransE and RDF2Vec + Random
Forest $[\chi^{2}(1)=79.27,\;pvalue=5.4\cdot 10^{-19}]$; MuRE and RDF2Vec +
Random Forest $[\chi^{2}(1)=79.27,\;pvalue=5.4\cdot 10^{-19}]$.

As we can see in the results, the supervised approach (RDF2Vec + Random
Forest) is the worst approach in terms of mean rank (MR) by a considerable
amount, having a 34 times larger mean rank than the best approach (MuRE). This
larger decrease in mean rank score with respect to other metrics is expected,
since the supervised approach converts the type prediction task to a binary
classification problem. That is, this model tries to maximize the probability
of guessing correctly whether an entity is true or false: the ranking of true
over false triples does not matter to the model.

With respect to the other metrics, its performance is better than TransE but
worse than both RotatE and MuRE. This follows the same trends that were
outlined when analyzing the performance of each negative sampler. Using
removals from the edit history as false triples to the classifier adds a lot
of noise in some cases, decreasing the overall performance of the model.

Table 5: Evaluation results of each best unsupervised model and the supervised model. | RotatE (inv.) | TransE (basic) | MuRE (inv.) | RDF2Vec + Random Forest
---|---|---|---|---
MR | 7191 | 2385 | 447 | 15752
MRR | 0.260 | 0.091 | 0.237 | 0.124
hits@1 | 0.163 | 0.050 | 0.144 | 0.062
hits@5 | 0.382 | 0.121 | 0.330 | 0.170
hits@10 | 0.445 | 0.164 | 0.422 | 0.249

###  5.6 Reproducibility

Experiments were executed using Python 3.8. Knowledge graph embedding models
were created and trained using the PyKEEN 1.8.0 [24] and PyTorch 1.11.0 [25]
libraries. The supervised model was built with pyRDF2Vec 0.2.3 [26], and
Scikit-learn 1.0.2 [27]. A random seed of 42 was established to create the RDF
dataset and train each model.

The complete code where all the experiments of this work were conducted is
also available at https://github.com/alejgh/wikidata-edithist-refinement. The
RDF datasets with static and dynamic serializations used in the type
prediction task are also publicly available [28]. We think that new research
opportunities can arise by exploiting the dynamic information of an RDF
dataset beyond the negative sampling techniques proposed in this paper, and
both datasets could be used to measure the performance of models leveraging
the edit history of a knowledge graph.

##  6 Related Work

###  6.1 Collaboration dynamics of Wikidata

Several works have tried to better understand the collaboration and edition
dynamics of Wikidata. T. Steiner was one of the first authors exploring this
field, by implementing a tool that provided real-time statistics about edits
made in both Wikipedia and Wikidata [29]. That author performed a quantitative
analysis of the editions made to Wikidata across 3 days in 2013, focusing on
the impact that bots had on the knowledge graph. Their analysis shows that
–during that time frame– most edits in Wikidata were made by bots (around
$88\%$).

While the quantitative analysis of Steiner provided a better understanding of
the editor distribution in Wikidata, a qualitative analysis was still needed
to understand which aspects of Wikidata were edited by each contributor, and
therefore get a better view of their behavior within the Wikidata ecosystem.
This is one of the main contributions of the study conducted by Müller-Birn et
al. in 2015 [30]. Those authors performed clustering on edit statistics
computed across 1 month time frames, and those clusters were used to detect
different editing roles in Wikidata (e.g., property editor, domain expert,
instance creator…). Those editing roles correspond to the different
collaboration patterns that exist in Wikidata.

Another point of interest within the field is understanding the path that a
new editor follows to become an established contributor within Wikidata.
Piscopo et al. detected that Wikidata has a low entry barrier, allowing
newcomers to start making simple statement contributions right away [31].
However, as contributors become more expert around the Wikidata ecosystem they
transition from performing editions through the interface to using more
advanced automatic tools. Understanding at an early stage which editors may be
more engaged with editing in Wikidata was the goal of Sarasua et al. [32].
They monitored the way users participated in Wikidata for 4 years, and
developed predictive models to estimate the edit lifetime of a contributor
reaching an F1-score of $0.9$.

More recent works have shifted to understanding how these collaboration
dynamics affect the overall quality of the knowledge graph. Piscopo et al.,
analyzed several edit metrics of 5,000 Wikidata items to understand these
relationships [33]. They detected that the interaction between human and bot
users is crucial to ensure the quality of Wikidata items, and that anonymous
edits are instead detrimental to the quality (mainly due to vandalism). In
their follow-up work, they tried to detect a correlation between a set of
defined quality metrics and editing roles detected with a clustering approach
[34]. Results showed that Wikidata is unevenly distributed at an ontological
level, with some domains having a lot of sub-classes automatically created by
bots.

###  6.2 Type prediction in knowledge graphs

Regarding works on type prediction tasks, SDType was one of the first tools
made to add missing type statements in knowledge graphs [11] –specifically, in
DBPedia. SDType works by calculating the statistical distribution of types in
the object and subject position of a property and using this data to fill
missing types of an entity, outperforming other type prediction approaches at
the time. Melo et al. then proposed a new system that exploited the
hierarchical nature of type systems in knowledge graphs by using a
hierarchical classifier [35]. This system was tested on several knowledge
graphs, including Wikidata, outperforming SDType and other alternatives while
also scaling better than other multilabel classifiers.

The use of knowledge graph embeddings for the task of type prediction has been
explored recently. Kejriwal et al. proposed a method to create class
embeddings based on a series of pre-computed entity embeddings [36]. Class
embeddings are closer to related instances and similar classes in the vector
space. Those authors then proposed the use of cosine similarity between the
embedding of an entity and the class embeddings to recommend the typing of the
entity. Other approaches try to generate fine-grained type predictions using
entity embeddings in both supervised and unsupervised approaches, obtaining
better results than SDType in two DBPedia datasets [37].

Ridle is a system that leverages neural networks and a custom knowledge graph
representation to improve existing type prediction approaches [38]. The
proposed representation of each entity is built from its relations, under the
idea that some sets of properties are more likely to appear as relations in
the instances of a class than in others. Ridle outperforms SDType and other
embedding-based approaches across multiple datasets, including Wikidata.

###  6.3 Negative sampling methods

Negative sampling methods have been shown to be an important step in the
performance of knowledge graph embedding models. Kotnis et al. proposed two
new embedding based negative sampling methods, comparing their performance to
random approaches [39]. Their results indicated that the performance of each
negative sampling method depended on the characteristics of the dataset being
used, with no approach being the best across all datasets.

Using generative adversarial networks (GANs) to produce negative samplers has
also been introduced recently, showing promising results by improving the
performance of knowledge graph embeddings under different settings [40, 41].
NSCaching is a method that improves the efficiency of GAN based methods by
incorporating a cache of high-quality negative triples, outperforming state-
of-the-art negative sampling methods [42, 43].

###  6.4 Leveraging the edit history of knowledge graphs

Exploiting edit information of knowledge graphs for several tasks has been
increasingly studied recently. Initial work tried to offer better ways to
access and query edits from Wikidata. The Wikidata edit history query service
was one of the first available tools, providing a SPARQL endpoint to query
edit information made to Wikidata until July 1st of 2018 [44]. During the time
of experimentation of this paper (November 2021 to March 2022) that service
was down, but at the time of this writing it is back online.

Wikidated is a dataset that contains the entire edit history of Wikidata
encoded as a set of triples additions and deletions [45]. It provides a Python
API for exploring and interacting with the dataset without needing to know the
underlying data model being used. This dataset could not be used in our
research since it is still undergoing active development.

Regarding the application of edit history information to knowledge graph
refinement tasks, AlGhamdi et al. proposed a Wikidata recommendation system
that leverages this information [46]. Their system summarizes the editing
activities of the user and combines this with entity representations to
recommend items that the user could edit next. Results are promising and lay
the ground for future work in Wikidata recommendation systems.

Finally, Tanon et al. proposed a system that automatically mines knowledge
graph correction rules from its edit history [47]. Their system starts from a
set of constraints violations from the knowledge graph, and mines the edit
history to find edits that solved those violations in the past, which are then
returned as a set of possible correction rules for the violation. They run a
set of experiments in Wikidata, showing significant improvements over
baselines. In their follow up work, they proposed a refinement of their
previous system by using neural networks to predict the correction rules of a
given constraint [48]. That neural network receives as input a vector
representation of the constraint, the triples that have been violated, and
facts about the entities. Results over the Wikidata dataset showed significant
improvements over both the previous rule-based approach and the baselines.

##  7 Conclusions and future work

This paper provides the first exploration of the use of edit history
information in knowledge graph refinement and, specifically, type prediction
tasks. We have built and provided a JSON dataset containing the edit history
information of every Wikidata instance from the top 100 most important
Wikidata classes based on their ClassRank score. This edit history information
has been explored, making a special emphasis on its potential influence in
knowledge graph refinement tasks. Finally, we have proposed two new approaches
to improve knowledge graph embeddings in type prediction tasks: the use of
negative samplers that generate corruptions based on the edit history of the
graph and training a supervised model by labeling triples based on the edit
history information. These approaches have been evaluated on a RDF dataset
against currently used approaches in the field.

The main findings of our work are:

  * 1.

Instances in Wikidata have different edit behaviors based on the class they
belong to. Some classes lead to more controversial statements that are the
subject of edit wars, while some other classes are more stable and experience
almost no removals or value replacements.

  * 2.

Using removed triples from the graph as negative samples can improve the
performance of a basic negative sampling approach in some cases, but in
general it results in a worse performance of the models. On average, a basic
negative sampling approach leads to a 123% higher hits@1 score across every
model compared to performing negative sampling from removed triples of the
graph. Avoiding removed triples that experienced edit wars improves the
performance of our proposed edit history based negative sampling approach
across every knowledge graph embeddings model. On average, avoiding these
triples improves the hits@5 of our proposed approach by 27.18%.

  * 3.

On the contrary, avoiding the generation of corruptions that belong to the set
of removed triples from the graph improves the performance of 2 out of the 3
models, getting the 2 best results across every model and negative sampler
combination being evaluated. This approach improves the hits@1 and hits@5 of
the basic negative sampler approach by a 89.53% and a 27.33% respectively.

New promising research lines have emerged from this work, among them:

  * 1.

Reaching a deeper understanding on the differences between edits related to
vandalism, a conflict of opinion within the community, or the natural
evolution of an entity. Having a method to distinguish between such kinds of
edits could be crucial to have a better understanding of the editing behavior
in Wikidata and also to improve and create new methods that use this edit
information.

  * 2.

To perform an analysis of the corruptions generated by the negative samplers
proposed in this work. This could provide a better understanding of the
strengths and weaknesses of each model, while also providing an explanation of
which types of edits should be avoided or considered by the negative samplers.

  * 3.

There are also several optimizations that could be applied to the negative
samplers proposed in this work. One of them could be to assign a score to each
one of the corruptions generated by the negative samplers, and choosing the
top $n$ corruptions based on this score instead of choosing them randomly.

  * 4.

Performing an incremental evaluation of the performance of each negative
sampler approach depending on the size of the dataset, and the number of
negative triples from the edit history avoided or added. This evaluation could
also include a comparison to more advanced negative sampling approaches (e.g.,
GAN-based negative samplers).

  * 5.

Finally, the use of edit history information in other knowledge graph
refinement tasks and models could also be explored. For example, one
interesting research direction could be the creation of models that work
directly with the dynamic version of the RDF dataset provided in this work.

## CRediT author statement

Alejandro Gonzalez-Hevia: Methodology, Conceptualization, Software,
Validation, Formal Analysis, Investigation, Data Curation, Writing - Original
Draft, Visualization, Project Administration. Daniel Gayo-Avello: Methodology,
Resources, Writing - Review & Editing, Supervision.

## Declaration of Competing Interests

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## References

  * [1] T. Ruan, L. Xue, H. Wang, F. Hu, L. Zhao, J. Ding, Building and exploring an enterprise knowledge graph for investment analysis, in: International semantic web conference, Springer, 2016, pp. 418–436.
  * [2] M. Rotmensch, Y. Halpern, A. Tlimat, S. Horng, D. Sontag, Learning a health knowledge graph from electronic medical records, Scientific reports 7 (1) (2017) 1–11.
  * [3] A. Nayak, V. Kesri, R. K. Dubey, Knowledge graph based automated generation of test cases in software engineering, in: Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, 2020, pp. 289–295.
  * [4] C. Bizer, J. Lehmann, G. Kobilarov, S. Auer, C. Becker, R. Cyganiak, S. Hellmann, Dbpedia-a crystallization point for the web of data, Journal of web semantics 7 (3) (2009) 154–165.
  * [5] D. Vrandečić, Wikidata: A new platform for collaborative data collection, in: Proceedings of the 21st international conference on world wide web, 2012, pp. 1063–1064.
  * [6] A. Piscopo, E. Simperl, What we talk about when we talk about wikidata quality: a literature survey, in: Proceedings of the 15th International Symposium on Open Collaboration, 2019, pp. 1–11.
  * [7] M. Färber, F. Bartscherer, C. Menne, A. Rettinger, Linked data quality of dbpedia, freebase, opencyc, wikidata, and yago, Semantic Web 9 (1) (2018) 77–129.
  * [8] K. Shenoy, F. Ilievski, D. Garijo, D. Schwabe, P. Szekely, A study of the quality of wikidata, Journal of Web Semantics 72 (2022) 100679.
  * [9] P. F. Patel-Schneider, Using description logics for rdf constraint checking and closed-world recognition, in: Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
  * [10] E. Prud’hommeaux, J. E. Labra Gayo, H. Solbrig, Shape expressions: an rdf validation and transformation language, in: Proceedings of the 10th International Conference on Semantic Systems, 2014, pp. 32–40.
  * [11] H. Paulheim, C. Bizer, Improving the quality of linked data using statistical distributions, International Journal on Semantic Web and Information Systems (IJSWIS) 10 (2) (2014) 63–86.
  * [12] D. Fernández-Álvarez, J. Frey, J. E. Labra Gayo, D. Gayo-Avello, S. Hellmann, Approaches to measure class importance in knowledge graphs, Plos one 16 (6) (2021) e0252862.
  * [13] L. Page, S. Brin, R. Motwani, T. Winograd, The pagerank citation ranking: Bringing order to the web., Tech. rep., Stanford InfoLab (1999).
  * [14] A. Thalhammer, A. Rettinger, PageRank on Wikipedia: Towards General Importance Scores for Entities, in: The Semantic Web: ESWC 2016 Satellite Events, Heraklion, Crete, Greece, May 29 – June 2, 2016, Revised Selected Papers, Springer International Publishing, Cham, 2016, pp. 227–240.  doi:10.1007/978-3-319-47602-5_41.
URL https://dx.doi.org/10.1007/978-3-319-47602-5_41

  * [15] [dataset] A. Gonzalez-Hevia, Wikidata subset with revision history information [JSON] (Jun. 2022).  doi:10.5281/zenodo.6614264.
URL https://doi.org/10.5281/zenodo.6614264

  * [16] T. Yasseri, R. Sumi, A. Rung, A. Kornai, J. Kertész, Dynamics of conflicts in wikipedia, PloS one 7 (6) (2012) e38869.
  * [17] Z. Sun, Z.-H. Deng, J.-Y. Nie, J. Tang, Rotate: Knowledge graph embedding by relational rotation in complex space, arXiv preprint arXiv:1902.10197.
  * [18] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, O. Yakhnenko, Translating embeddings for modeling multi-relational data, Advances in neural information processing systems 26.
  * [19] I. Balazevic, C. Allen, T. Hospedales, Multi-relational poincaré graph embeddings, Advances in Neural Information Processing Systems 32.
  * [20] M. Ali, M. Berrendorf, C. T. Hoyt, L. Vermue, M. Galkin, S. Sharifzadeh, A. Fischer, V. Tresp, J. Lehmann, Bringing light into the dark: A large-scale evaluation of knowledge graph embedding models under a unified framework, IEEE Transactions on Pattern Analysis and Machine Intelligence.
  * [21] P. Ristoski, H. Paulheim, Rdf2vec: Rdf graph embeddings for data mining, in: International Semantic Web Conference, Springer, 2016, pp. 498–514.
  * [22] T. G. Dietterich, Approximate statistical tests for comparing supervised classification learning algorithms, Neural computation 10 (7) (1998) 1895–1923.
  * [23] K. S. Jones, Automatic indexing, Journal of documentation.
  * [24] M. Ali, M. Berrendorf, C. T. Hoyt, L. Vermue, S. Sharifzadeh, V. Tresp, J. Lehmann, PyKEEN 1.0: A Python Library for Training and Evaluating Knowledge Graph Embeddings, Journal of Machine Learning Research 22 (82) (2021) 1–6.
URL http://jmlr.org/papers/v22/20-825.html

  * [25] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, Pytorch: An imperative style, high-performance deep learning library, in: H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, R. Garnett (Eds.), Advances in Neural Information Processing Systems 32, Curran Associates, Inc., 2019, pp. 8024–8035.
  * [26] G. Vandewiele, B. Steenwinckel, T. Agozzino, F. Ongenae, pyrdf2vec: A python implementation and extension of rdf2vecdoi:10.48550/ARXIV.2205.02283.
URL https://arxiv.org/abs/2205.02283

  * [27] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830.
  * [28] [dataset] A. Gonzalez-Hevia, Wikidata subset with revision history information [RDF] (Jun. 2022).  doi:10.5281/zenodo.6613875.
URL https://doi.org/10.5281/zenodo.6613875

  * [29] T. Steiner, Bots vs. wikipedians, anons vs. logged-ins (redux) a global study of edit activity on wikipedia and wikidata, in: Proceedings of The International Symposium on Open Collaboration, 2014, pp. 1–7.
  * [30] C. Müller-Birn, B. Karran, J. Lehmann, M. Luczak-Rösch, Peer-production system or collaborative ontology engineering effort: What is wikidata?, in: Proceedings of the 11th International Symposium on Open Collaboration, 2015, pp. 1–10.
  * [31] A. Piscopo, C. Phethean, E. Simperl, Wikidatians are born: Paths to full participation in a collaborative structured knowledge base.
  * [32] C. Sarasua, A. Checco, G. Demartini, D. Difallah, M. Feldman, L. Pintscher, The evolution of power and standard wikidata editors: comparing editing behavior over time to predict lifespan and volume of edits, Computer Supported Cooperative Work (CSCW) 28 (5) (2019) 843–882.
  * [33] A. Piscopo, C. Phethean, E. Simperl, What makes a good collaborative knowledge graph: group composition and quality in wikidata, in: International Conference on Social Informatics, Springer, 2017, pp. 305–322.
  * [34] A. Piscopo, E. Simperl, Who models the world? collaborative ontology creation and user roles in wikidata, Proceedings of the ACM on Human-Computer Interaction 2 (CSCW) (2018) 1–18.
  * [35] A. Melo, J. Völker, H. Paulheim, Type prediction in noisy rdf knowledge bases using hierarchical multilabel classification with graph and latent features, International Journal on Artificial Intelligence Tools 26 (02) (2017) 1760011.
  * [36] M. Kejriwal, P. Szekely, Supervised typing of big graphs using semantic embeddings, in: Proceedings of The International Workshop on Semantic Big Data, 2017, pp. 1–6.
  * [37] R. Sofronova, R. Biswas, M. Alam, H. Sack, Entity typing based on rdf2vec using supervised and unsupervised methods, in: European Semantic Web Conference, Springer, 2020, pp. 203–207.
  * [38] T. Weller, M. Acosta, Predicting instance type assertions in knowledge graphs using stochastic neural networks, in: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, 2021, pp. 2111–2118.
  * [39] B. Kotnis, V. Nastase, Analysis of the impact of negative sampling on link prediction in knowledge graphs, arXiv preprint arXiv:1708.06816.
  * [40] L. Cai, W. Y. Wang, Kbgan: Adversarial learning for knowledge graph embeddings, arXiv preprint arXiv:1711.04071.
  * [41] P. Wang, S. Li, R. Pan, Incorporating gan for negative sampling in knowledge representation learning, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, 2018.
  * [42] Y. Zhang, Q. Yao, Y. Shao, L. Chen, Nscaching: simple and efficient negative sampling for knowledge graph embedding, in: 2019 IEEE 35th International Conference on Data Engineering (ICDE), IEEE, 2019, pp. 614–625.
  * [43] Y. Zhang, Q. Yao, L. Chen, Simple and automated negative sampling for knowledge graph embedding, The VLDB Journal 30 (2) (2021) 259–285.
  * [44] T. Pellissier Tanon, F. Suchanek, Querying the edit history of wikidata, in: European Semantic Web Conference, Springer, 2019, pp. 161–166.
  * [45] L. Schmelzeisen, C. Dima, S. Staab, Wikidated 1.0: An evolving knowledge graph dataset of wikidata’s revision history, arXiv preprint arXiv:2112.05003.
  * [46] K. AlGhamdi, M. Shi, E. Simperl, Learning to recommend items to wikidata editors, in: International Semantic Web Conference, Springer, 2021, pp. 163–181.
  * [47] T. Pellissier Tanon, C. Bourgaux, F. Suchanek, Learning how to correct a knowledge base from the edit history, in: The World Wide Web Conference, 2019, pp. 1465–1475.
  * [48] T. Pellissier Tanon, F. Suchanek, Neural knowledge base repairs, in: European Semantic Web Conference, Springer, 2021, pp. 287–303.