# Agreeing and Disagreeing in Collaborative Knowledge Graph Construction: An
Analysis of Wikidata

Elisavet Koutsiana<EMAIL_ADDRESS>King’s College London,
Informatics DepartmentLondonUK ,  Tushita Yadav<EMAIL_ADDRESS>King’s College London, Informatics DepartmentLondonUK ,  Nitisha Jain
<EMAIL_ADDRESS>King’s College London, Informatics DepartmentLondonUK ,
Albert Meroño-Peñuela<EMAIL_ADDRESS>King’s College London,
Informatics DepartmentLondonUK and  Elena Simperl<EMAIL_ADDRESS>King’s College London, Informatics DepartmentLondonUK

###### Abstract.

In this work, we study disagreement in discussions around Wikidata, an online
knowledge community that builds the data backend of Wikipedia. Discussions are
important in collaborative work as they can increase contributor performance
and encourage the emergence of shared norms and practices. While disagreements
can play a productive role in discussions, they can also lead to conflicts and
controversies, which impact contributor well-being and their motivation to
engage. We want to understand if and when such phenomena arise in Wikidata,
using a mix of quantitative and qualitative analyses to identify the types of
topics people disagree about, the most common patterns of interaction, and
roles people play when arguing for or against an issue. We find that decisions
to create Wikidata properties are much faster than those to delete properties
and that more than half of controversial discussions do not lead to consensus.
Our analysis suggests that Wikidata is an inclusive community, considering
different opinions when making decisions, and that conflict and vandalism are
rare in discussions. At the same time, while one-fourth of the editors
participating in controversial discussions contribute with legit and
insightful opinions about Wikidata’s emerging issues, they do not remain
engaged in the discussions. We hope our findings will help Wikidata support
community decision making, and improve discussion tools and practices.

collaborative knowledge graph, collaborative knowledge base, knowledge
community, discussion analysis, community analysis, argumentation

##  1\. Introduction

Discussions play an important role in collaborative work as they help increase
motivation and enhance emerging norms and practices. Previous studies on
Computer-Supported Cooperative Work (CSCW) have emphasized the importance of
disagreement in collaborative work (Easterbrook, 1993; Jehn, 1995; Franco et
al., 1995; De Dreu and Weingart, 2003). Disagreements in discussions can
improve performance, lead to better project practices, and offer different
perspectives on a subject. However, intense disagreements can cause
controversies and conflicts, which could in turn negatively influence
contributors’ motivation and performance, subsequently reducing engagement and
leading to dropouts (Jehn, 1995; Franco et al., 1995; De Dreu and Weingart,
2003). In peer-production, self-organised communities that collaborate to
produce a knowledge artefact (e.g., articles, software, or knowledge bases),
disagreements can lead to higher quality results as well as the bonding of the
community (Franco et al., 1995; Arazy et al., 2011; De Kock and Vlachos, 2021;
Kittur and Kraut, 2010). More research is needed to understand how peer-
production communities manage disagreements in decision making on platforms
with asynchronous discussions, particularly self-organised online knowledge
communities.

In peer-production communities, construction can be explicit through community
discussions or implicit through co-editing of the content. Disagreements occur
in discussions when expressing different opinions over a subject or in co-
editing when objecting and reverting the work of others. A continuing
disagreement can lead to a controversy or conflict. Previous studies on
disagreements in online peer-production systems have focused on controversies
or conflicts, and how communities argue when expressing opinions. Studies have
investigated disagreements in Q&A forums (Ren et al., 2019), open source
software development (Filippova and Cho, 2015, 2016; Elliott and Scacchi,
2002), and knowledge projects (Wang and Cardie, 2016; Borra et al., 2015;
Zielinski et al., 2018; Kittur et al., 2007; Vuong et al., 2008; Sumi et al.,
2011; Jankowski-Lorek et al., 2014; Yasseri et al., 2014). Identifying
controversies or conflicts is essential for their resolution; however, it is
important to understand the manner in which the community manages disagreement
and takes decisions (Borra et al., 2015; Vuong et al., 2008), and the way of
cooperation among the community members (Zielinski et al., 2018). This
understanding can ensure high quality content, balanced viewpoints, and
prevent misinformation (Zielinski et al., 2018). In addition, argument
detection is essential to help participants handle the vast amount of
information in a discussion and include all opinions in a decision (Wang et
al., 2020), and get an understanding of positions (Sepehri Rad and Barbosa,
2011).

In this paper, we perform a comprehensive analysis on disagreements in
Wikidata (Vrandečić and Krötzsch, 2014), the world’s largest open-source
collaborative knowledge graph. Knowledge graphs are collections of entities
and their relations structured as graphs describing a specific domain.
Collaborative knowledge graphs are created by humans in an enterprise
environment or using crowd-sourcing or co-editing platforms (Hogan et al.,
2021). With a large number of entities (more than $100M$) and a community of
approximately $24K$ active
members,111https://www.wikidata.org/wiki/Wikidata:Statistics Wikidata serves
as the backend of the online encyclopedia Wikipedia and supports many
intelligent systems like search engines and virtual assistance.

Our analysis broadly focuses on the following research questions:

  * •

$RQ1$ \- Where do we find disagreements in Wikidata?

  * •

$RQ2$ \- What are the main issues in controversial discussions in Wikidata?

  * •

$RQ3$ \- What are the characteristics of controversial discussions in
Wikidata?

  * •

$RQ4$ \- How do editors argue when disagreeing in Wikidata?

  * •

$RQ5$ \- What are the roles of editors in argumentation in Wikidata?

To answer these research questions, we analyse the discussions recorded as
textual conversations between Wikidata editors through a mixed-methods
approach, including: (i) descriptive statistical analysis on the whole corpus
of discussions in Wikidata; (ii) thematic analysis on a sample of discussions
with disagreements, where we identify controversies; (iii) measurements of the
radial tree structure of discussions (Herke and Mynhardt, 2009) as well as
statistical tests for the sample of discussions used in the thematic analysis;
and (iv) content analysis for the discussions identified as controversial in
the thematic analysis.

This work contributes to the analysis of collaborative knowledge graph
construction with: (i) an overview of the complete corpus of Wikidata
discussions; (ii) a framework to identify discussions with intense
disagreements; and (iii) a coding schema for content analysis to identify
argumentation patterns and the role of discussion participants. Our study
shows that:

  * •

Compare to Wikipedia, there is little conflict in Wikidata ($7\%$ of the
codes), and a few disruptions or vandals in discussions ($1\%$ of participants
in controversies). However, despite the lack of intense conflicts, over half
of the controversial discussions are closed without consensus ($62\%$). This
can indicate that controversial discussions (including many participants and
long content) make it difficult for the community to make consensus and they
need extra support.

  * •

The most frequent controversial issue is Wikidata processes, i.e.,
disagreement on how to perform a task ($52\%$ of controversial discussions),
with most arguments using examples of similar cases, policies and practices as
counter examples ($33\%$ of arguments). This means that participants in
argumentation need to be very well informed about policies to participate and
engage in discussions. A good understanding of policies and values has been
shown to be essential in other peer-production projects like Wikipedia
(Schneider et al., 2013). This suggests that managers could support the
community with clear and frequently updated policies and guidelines in easy to
access areas of the projects. This can enhance cooperation and decision
making.

  * •

While the majority of participants in controversies ($47\%$) engage in the
discussion and aim to influence others with their arguments, a high number of
participants ($25\%$) do not engage in the discussion, despite the legitimate
and insightful arguments. The lack of engagement combined with the lack of
consensus may suggest that controversial discussions require attention,
flagging the need for new tools to support when needed, control discussions,
further instructions, or practice changes.

##  2\. Background: The Wikidata collaborative knowledge graph

###  2.1. The Wikidata knowledge graph

Wikidata is an online peer-production system (Benkler et al., 2015) where a
community of volunteers from all over the world create and maintain a large
knowledge graph (Vrandečić and Krötzsch, 2014). The term knowledge graph was
coined around ten years ago by Google to refer to the data backend used by
their search engine to answer factual queries (Hogan et al., 2021). More
generally, a knowledge graph can be thought of as a technology to structure
and organise factual data. In this graph, the nodes represent the entities of
interest in a domain such as people or places. The edges describe the entities
in form of their attributes, e.g. the date of birth of a person, or in form of
relations to other entities, e.g. the relation between a person and the place
they were born in. Wikidata knowledge graph currently adds more than $100M$
nodes (called “items”) and $10K$ edges (called “properties”).

Furthermore, Figure 1 presents a node and some of its edges as an example. The
figure shows statements about the item Ada Lovelace in the form of tuples
consisting of a subject (the item that is discussed, in this case Ada
Lovelace), a property (for instance, instance of) and the value of that
property (human). Some statements have more than three fields called
qualifiers (for instance start time and end time) to add additional
information to the item. Items and properties are accompanied by human-
readable labels, descriptions, and aliases in $358$ languages (Vandenbussche
et al., 2017).222https://www.wikidata.org/wiki/Wikidata:Introduction Both
items and properties in Wikidata are organised in taxonomies, i.e. classes.
Wikidata assigns classes for instances with common features using the property
‘instance of’. In the figure, the item Ada Lovelace is an instance of the
class human.

Figure 1. An example of the Wikidata knowledge graph for the item Ada
Lovelace. We use red annotations to highlight the main features.

###  2.2. Wikidata community and their contributions

People contribute to Wikidata in multiple ways. They can add, remove, or alter
content, at various levels of granularity, ask or answer questions, discuss
and decide upon different options to structure a certain piece of information,
or help others do so. Contributors are referred to as editors, which can be
either humans or bots (i.e., software executing simple, repetitive edits).
Human editors may choose to register themselves or contribute anonymously. All
editors are allowed to work on items and properties, but registered editors
have additional editing rights, depending on their experience with the
project, extent of their contributions, and the duration of their
membership.333https://www.wikidata.org/wiki/Wikidata:User_access_levels For
instance, one group of editors with higher administrative editing rights are
the Administrators. They are elected by a community vote and are responsible
for tasks such as deleting items, blocking editors or unauthorised bots, and
protecting pages to prevent vandalism. They can also grant higher rights to
other editors. An example of editors with higher technical editing rights are
the Property creators, who respond to community requests to create new
properties, initiate community discussions about the request, and ensure that
a decision is made.

Figure 2. An example of editing a statement for the Wikidata item humanity

Wikidata has approximately $24K$ active registered editors, $20K$ anonymous
editors, and $100$ active bots every
month.444https://stats.wikimedia.org/#/metrics/wikidata.org However, for these
groups, different participation patterns have been observed over the years.
According to the statistics related to the editing activities of Wikidata,
published by Wikimedia Foundation (i.e., the non-profit organization hosting
Wikidata), bots are responsible for $52\%$, while human editors perform $48\%$
of all edits. Previous studies on participation patterns show that almost
$60\%$ of the editing activities by bots are related to item statements, i.e.,
adding context within a domain of interest, while $30\%$ are related to item
descriptions, labels and aliases (Müller-Birn et al., 2015). Similarly, for
human editors, $13\%$ are related to item descriptions and almost $30\%$ of
item statements. In addition, $30\%$ of editing activities by humans
contribute to the core of the graph by describing and creating new properties
and their talk pages. However, these activities come from $2\%$ of human
editors (Müller-Birn et al., 2015). Regarding human editors, it is interesting
to note that only a small percentage of editing activities, $0.5\%$, comes
from anonymous editors (Piscopo and Simperl, 2018). For the registered
editors, only $2\%$ of editors are responsible for $95\%$ of the edits
(Piscopo et al., 2017).

From a technical point of view, Wikidata is built using a wiki, a
collaborative web platform that allows users to add and edit items and
properties using web pages in a browser (Wagner, 2004). Each item or property
has its own wiki page, which can be changed on the fly. Each wiki page has a
discussion page, referred to as a talk page, which is discussed in greater
detail in Section 2.3. Figure 2 shows an example of how editors can edit a
statement in the page of the item humanity, and Figure 3 shows the
corresponding talk page.

Figure 3. An example of a talk page for the Wikidata item humanity

Wikimedia Foundation provides public access to comprehensive logs, including
the history of edits for all content and discussion pages. The editing
activities for each page, referred to as revisions, include metadata such as
the page name, the name of the editor responsible for the revision, a
timestamp, a comment, and a text description. The comment classifies the edit
into a pre-defined set of actions e.g. addition, deletion, revert, undo,
restore etc. The textual description adds context to the edit action. The data
can be used for various research purposes to help and improve Wikidata
practices.

Table 1. Lists of the discussion channels in Wikidata and their descriptions. Discussion Channel |  Description
---|---
Talk pages |
Items |  Each item page can have a corresponding page for discussions related to their construction.
Properties |  Each property page can have a corresponding page for discussions related to their construction.
Users |  Each editor can have a corresponding page for discussions related to their editing activities.
Wikiprojects |  Wikiprojects are groups of editors collaborating to improve Wikidata on particular topics (e.g, sports or biology). Each Wikiproject has a corresponding page for discussions to coordinate the editing activities relates to the specific topic
Others |  Every content page (e.g. Wikidata main page, help pages) can have a corresponding page for discussions related to their content.
Communication pages |
Project chat |  General questions about how to do or what are the common practices for tasks in the project, e.g, “ How we should represent external identifiers where there is both a permanent ID and a human-friendly ID?”.
Requests for comment |  Requests for changes in practices or guidelines, e.g, “Gender neutral labels for occupations and positions in French”.
Report a technical problem |  Discussion about platform complication, e.g, “User contributions search is borken”.
Request a query |  Requests for Wikidata SPARQL queries, e.g, “Query for items that are both subclass/instance of something and it’s opposite”.
Interwiki conflicts |  Report problems with content on other wikis, e.g, when two or more items exist that seemingly correspond to the same Wikipedia article or other Wikimedia site page.
Bot requests |  Requests for tasks to be done by a bot, e.g, “request to add identifiers from FB”.
Property proposal |  Propose the creation of a property, e.g, a suggestion to create the property “number of penalty kicks scored” because “Differentiate regular goals from penalty kicks, people use twice the quantifier in last editions of World Cup, rendering strange result”
Administrators’ noticeboard |  Reporting vandalism, requesting page protections, etc, e.g, “Would like to draw attention to the edits of user Turktimex3”.
Translators’ noticeboard |  Report a translation problem, ask to mark a page for translation, e.g, “Please, mark for translation Template:AdminsChart/Title and Template:AdminsChart/Elected”.
Bureaucrats’ noticeboard |  Requesting for flood flag, etc, e.g. editors should use the flood flag to avoid confusion when they do repetitive edits.
Requests for deletions |  Deletion requests of items and pages, e.g, in case of duplication or not following Wikidata policies like notability.
Properties for deletion |  Deletion requests of properties, e.g, “P9099 (P9099) seems to be redundant to Property:P7900”
Requests for permission |  Permissions requests for permission to create a bot ot to get higher rights for editors, e.g, “ request for administrator: Trusted user who previously was overwhelmingly made a administrator. Rights were taken away due to inactivity but looking at xtools they are much more active this year”.

###  2.3. Discussions in Wikidata

Similar to other peer-production systems, including those creating knowledge
bases and knowledge graphs (Simperl and Luczak-Rösch, 2014), discussions are a
primary tool for collaboration in Wikidata. Wikidata has two types of
discussion channels: the talk pages, and the communication pages. Table 1)
contains lists of the discussion channels and their functionality. As noted
earlier, each item and property has its own talk page for the purpose of
raising questions, facilitating coordination between editors, and flagging any
errors.

Figure 4. An example of a Wikidata thread from the Property proposal
discussion channel. We use annotations to highlight the thread’s main
features.

In addition, Wikidata has various communication pages to support discussions
on general concerns that are not specific to an item or property. Broadly,
some communication pages are about technical support e.g. with the formulation
of a query to retrieve certain data from the graph, or for automating a task;
data structure and organisation e.g. introducing a new type of property (i.e.,
edge) on the graph, deleting content from the graph; as well as general and
administrative concerns around user management, suspected vandalism,
translations of pages etc. It is worth noting that the community do not
proceed to changes unless reaching a consensus on the discussed issues. In our
analysis, we will focus primarily on three specific communication pages,
Properties for deletion, Property proposal and Requests for comment as we have
identified these pages to be associated with the most disagreements. The
Properties for deletion and Property proposal pages are concerned with the
properties (i.e., edges) of the graph. In the former, editors suggest the
deletion of properties that are not considered useful anymore or that are
leading to confusion, while in the latter, editors request the creation of
properties in order to complete semantic gaps and extend the graph. The
Requests for comment is a general purpose page that is concerned with a wide
range of requests about changes in policies, regulations or practices, e.g.,
gender for labels in French, changes in the way editors write in the Request
for comment page.

All discussions on Wikidata follow the same format. Figure 4 presents an
example of discussion from the channel Property proposal. We refer to each
discussion as thread. A thread is a set of posts under a subject title. At the
end of each post, the name of the corresponding editor and a timestamp should
be included.

To help the flow of discussion on communication pages, Wikidata recommends the
use of specific templates when raising a new issue. The templates include a
rationale for the proposal or request, and examples of the issue raised. Once
an issue is raised, Wikidata gives a time frame (e.g. $7$ days for Properties
for deletion and $30$ days for Requests for comment) for the community to
respond. Participants in discussions can vote to agree or disagree with a
suggestion. Finally, any editor who is not involved in a discussion can close
the request if participants have reached a consensus or if there is no
consensus, but the discussion is no longer active.

##  3\. Related work

In this section, we provide an overview of prior work that has researched
disagreement and helped in shaping the research questions of this study.
Previous studies have investigated subgroups of opinions (Hassan et al.,
2012), power relationships (Danescu-Niculescu-Mizil et al., 2012; Biran et
al., 2012), interactions (Brandes et al., 2009; Garimella et al., 2018;
Leskovec and Horvitz, 2008), and the relevances of participants (Alsinet et
al., 2020). A big part of the literature in peer-production focused on the
study of disagreements, investigating (i) controversies and conflicts, and
(ii) argumentation.

Controversies and conflicts. The Oxford dictionary defines controversy as a
“public discussion and argument about something that many people strongly
disagree about, think is bad, or are shocked
by”.555https://www.oxfordlearnersdictionaries.com/definition/english/controversy?q=controversy
Studying scientific controversies, McMullin states that controversy is a
“publicly and persistently maintained dispute” making the argument that “to
count as controversy a disagreement must be a continuing one” (McMullin,
1987). In addition, according to the Oxford dictionary, a conflict is “a
situation in which people, groups or countries disagree strongly or are
involved in a serious
argument”.666https://www.oxfordlearnersdictionaries.com/definition/english/conflict_1?q=conflict
We find controversies and conflicts on the web in, e.g., debate forums (Beck
et al., 2018), social media (Garimella et al., 2018), knowledge sharing forums
(Hara and Sanfilippo, 2016), and collaborative settings (Filippova and Cho,
2015; Elliott and Scacchi, 2002; Wang and Cardie, 2016; Zielinski et al.,
2018; Kittur et al., 2007; Yasseri et al., 2014). In online collaboration,
which is our object of study, researchers investigated the detection of
controversies and conflict (Wang and Cardie, 2016; Borra et al., 2015;
Zielinski et al., 2018; Kittur et al., 2007; Ren et al., 2019), and their
characteristics (Suh et al., 2007; Beck et al., 2018; Filippova and Cho, 2015,
2016; Elliott and Scacchi, 2002).

Plenty of previous works have focused on Wikipedia for the detection of
controversial or conflictive articles (Zielinski et al., 2018; Vuong et al.,
2008; Sumi et al., 2011; Jankowski-Lorek et al., 2014; Yasseri et al., 2014),
the content of the articles themselves (Bykau et al., 2015; Kittur et al.,
2007; Jankowski-Lorek et al., 2014), or discussions related to edits on the
articles (Wang and Cardie, 2016; Ho-Dac et al., 2016). These studies have
investigated controversies using the history of edits i.e., the sequence of
revisions during their write-up (e.g., number of revisions in the article,
content changed by editors, article length, article age) (Viégas et al., 2004;
Vuong et al., 2008; Sepehri Rad et al., 2012; Yasseri et al., 2014). Others
combined the history of edits with the discussions taking place in parallel
(e.g. number of revisions in the discussion page, number of editors, number of
anonymous or administrative edits on the article or discussion page) (Kittur
et al., 2007; Jankowski-Lorek et al., 2014). To detect controversies in
Wikipedia, researchers have used various methods, such as support vector
machine (Kittur et al., 2007; Wang and Cardie, 2016), sentiment analysis (Wang
and Cardie, 2016; Jankowski-Lorek et al., 2014) and network analysis based on
edit reverts (Vuong et al., 2008). Their results showed that patterns of
controversy and conflict were too complex to be captured by one single
characteristic, and combinations of various factors such as number and type of
edits, discussion length and content must be considered (Rad and Barbosa,
2012). However, other works have shown that some features were more valuable
than others for identifying controversial characteristics. Kittur et al.
(Kittur et al., 2007) proved that the most important feature in detecting
conflicts was the number of revisions in the talk pages, implying that a
higher number of posts on the talk pages increased the probability of
conflict. In addition, Wang and Cardie (Wang and Cardie, 2016) used sentence
level sentiment analysis and showed that their best models combined the
article’s topic, discussion length, number of participants, and
positive/negative sentiment ranks, rather than just sentiment analysis scores.
This was also suggested by Jankowski-Lorek and Nielek (Jankowski-Lorek et al.,
2014), who argued that sentiment analysis alone could not predict a
controversial article but could indicate its controversial parts.

In other peer-production projects such as Q&A forums, where members discuss,
ask, and answer questions related to a variety of topics such as
Quora777https://www.quora.com/ and Reddit888https://www.reddit.com/ or
specialised topics like Stack Overflow,999https://stackoverflow.com/ (which is
for software developers), studies also aimed to detect and explore
controversies in order to support community operation. Prior work on
StackOverflow (Ren et al., 2019) created a tool to detect and summarise
controversies. The authors aimed to reduce the negative impact of
controversial answers to developers, and helped them find controversial
discussions and take decisions. Their experiments showed that when members
used controversial summaries to solve a task they had a high rate of correct
answers. In addition, previous work on the Reddit community (Zhang et al.,
2017) aimed to categorise posts in discourse act such as question, answer, and
disagreement. Zhang et al. (Zhang et al., 2017) found that disagreements
lasted from $1$ to $7$ posts and that often ($60\%$ of the cases) did not
receive answers. The study also found that the topics with the highest
proportion of disagreements were “change my view” and “political discussion”.
However, a study by Jasser et al. (Jasser et al., 2022) on the Reddit forum
showed that discussions staged as controversial were more likely to attract
higher activity. While many studies in online communities focused on how to
help members perform better, there were also studies interested in what made
members to participate in discussions. In the general interest forum Qora,
previous work studied what influenced members to follow and discuss with other
members (Nwadiugwu and Nwadiugwu, 2021). Nwadiugwu et al. (Nwadiugwu and
Nwadiugwu, 2021) showed that memebrs use controversies as a means to attract
followers. Quora members explained that they use controversial posts to
attract others’ attention and participate in discussions.

In addition, previous studies about conflicts in peer-production projects like
open source software development (Filippova and Cho, 2016) and Wikipedia
(Arazy et al., 2013) categorised conflicts into task, affective, process and
normative. Task conflicts had a cognitive dimension related to disagreements
on the content to be added. Affective conflicts, by contrast, involved
emotional disagreement. Process conflicts occurred when people disagree on how
to perform a task rather than the task itself. Finally, normative conflicts
involved group function disagreements. Fillippova et al. (Filippova and Cho,
2015, 2016) studied the impact of conflict categories in open source software
development. Normative conflicts were positively correlated with dropouts;
while task conflicts did not significantly affect editors’ performance
(Filippova and Cho, 2016). Furthermore, participation in discussions about the
core tasks of the project, where editors needed to take decisions, reduced
normative and process conflicts (Filippova and Cho, 2016). In extensive
discussions over issues, it was possible for different types of conflicts to
transform into others. A common case was when task conflicts transformed into
affective or process conflicts (Filippova and Cho, 2015). Prior work in peer-
production had also studied how conflicts arose and got resolved (Elliott and
Scacchi, 2002). Elliot et al. (Elliott and Scacchi, 2002) showed that archives
of past editing and discussion actions helped editors to resolve their
disagreements by giving examples. Furthermore, occasional editors, people who
contribute for short periods, were able to start but also mitigate conflicts.

Argumentation. Looking again in the Oxford dictionary, an argument is “a
conversation or discussion in which two or more people disagree, often
angrily”, and argumentation consists of “logical arguments used to support a
theory, an action or an
idea”.101010https://www.oxfordlearnersdictionaries.com/definition/english/argument?q=argument
111111https://www.oxfordlearnersdictionaries.com/definition/english/argumentation?q=argumentation
Argumentation mining is the study of automatically identifying arguments
(Lippi and Torroni, 2016). Schneider (Schneider, 2014) emphasised the
importance of argumentation mining, suggesting that its automation in debates
could give personalised feedback to post convincing arguments, helped in
identifying the weaknesses in each others’ arguments, and offered a summary of
the arguments in a debate. Furthermore, Wang et al. (Wang et al., 2020) argued
that argument detection in peer-production helped in handling the vast amount
of opinions and avoiding overlooking arguments in the middle of the
conversation.

Previous studies have analysed argumentation in online debates from different
perspectives, with quantitative and qualitative means, in terms of classifying
the arguments, identifying participants roles, and ranking their persuasive
power. Prior work, to classify arguments used Walton’s (Walton et al., 2008)
argumentation on social media (Schneider et al., 2014) and Wikipedia
(Schneider et al., 2013) with codes like argument from example, practical
reasoning, argument from rules etc.. Other studies in Wikipedia, focused on
knowledge sharing in arguments and looked, for example, background knowledge,
facts about the topic, citation, etc. (Hara and Sanfilippo, 2016), or used a
dialogue scheme, such as inform, manage, evaluate, etc. (Fréard et al., 2010).
Furthermore, similar studies in open source software development projects used
an adapted method of the general field argumentation model by Toulman (Wang et
al., 2020) using codes like claim, warrant, rebuttal, etc. For knowledge
engineering projects, the Toulman scheme was also used with variation because
did not fulfil the projects’ needs (Stranieri et al., 2001). However, previous
studies in collaborative knowledge engineering identified the value of
argumentation in building ontologies and introduced Diligent, an
argumentation-based ontology engineering methodology (Vrandečić et al., 2005;
Pinto et al., 2004) using codes such as issue, elaboration, counter example,
etc.. Based on the idea that during ontology building, participants exchange
arguments which may support or object to certain ontology engineering
decisions, the authors suggested a model for tracking arguments (Tempich et
al., 2005). Tempich et al. (Tempich et al., 2007) tested the theory of
argumentation to ontology engineering and showed that the Diligent
argumentation process ensured agreement, helped non-experts to quickly follow
processes, and made it easier to detect conflicted arguments.

In addition, prior work explored participants’ roles in argumentation. Similar
to the classification of arguments, several studies in Wikipedia focused on
the participants’ roles based on knowledge sharing like knowledge shaper,
reflective reframing [reframe],Reflective reinforcing [reinforcer], etc. (Hara
and Sanfilippo, 2016, 2017). Jain et al. suggested a scheme of social roles in
order to identify stubbornness, sensibility, influence, and ignorance in
Wikipedia (Jain et al., 2014) with roles like leader, follower, rebell, etc..
The authors argued that different participants’ roles in discussions may have
an influence on the outcome of decisions and identifying the roles helps in
monitoring and managing the communities.

Another view of studying argumentation is the persuasion of arguments. Under
this perspective, studies have shown that convincing arguments had specific
linguistic characteristics (Tan et al., 2016; Wei et al., 2016; Durmus and
Cardie, 2019; Habernal and Gurevych, 2016). Studies have investigated
automated methods such as machine learning techniques, to distinguish
persuasive from non-persuasive arguments. While patterns of interaction gave
good results, language was essential for the classification of persuasive
arguments (Tan et al., 2016). However, in successful arguments, peoples’
characteristics such as debating experience, prior success in persuasion and
social network features outperformed well structure in terms of language
characteristics (Durmus and Cardie, 2019). In addition, argumentation
characteristics such as connecting words, modal verbs, argument relevance and
originality performed well in the early stages of the discussion. As the
discussion progresses, responses deviated from the main subject with
participants responding with less well-structured arguments (Wei et al.,
2016).

The different structure of peer-production projects. Previous studies on
disagreements in peer-production projects have suggested that the different
structures of the projects, for example, an encyclopedia, a software, and a
knowledge graph, may raise different challenges and indicate alternative
results (Filippova and Cho, 2016; Kittur and Kraut, 2010). Filippova and Cho
(Filippova and Cho, 2016) have highlighted that the different nature of peer-
production projects may affect how disagreements emerged and were managed,
while Kittur and Kraut (Kittur and Kraut, 2010) found that conflict management
was related to the communities’ size. In addition, a study from Wikipedia
Foundation by Trumper (Sáez Trumper, 2021) suggested that when comparing
Wikipedia and Wikidata projects the difference in editing process needs to be
considered. While many studies explored disagreements in peer-production
projects, no prior work has investigated how disagreements arose, evolved, and
impacted decision making in peer-production projects that build knowledge
graphs. Knowledge graphs have become an essential infrastructure in the field
of AI as they provide semantically structured data. Collaborative knowledge
graphs like Wikidata improve tasks like question answering, recommendation
systems, and information retrieval (Ehrlinger and Wöß, 2016). Our study fills
a gap in collaborative knowledge graphs by analysing discussions and
particularly disagreements to explore collaboration and decision making.

##  4\. Exploratory data analysis

In this work, we considered three sources of data to study disagreements: the
revisions of item content pages; the item and property talk pages; and the
communication pages, as shown in Table 1. In this section, we present a
preliminary analysis of the revisions and the talk pages to understand how to
identify disagreements in Wikidata.

We first discuss our analysis concerned with the revisions of item content
pages. In Wikipedia, intense disagreements have been previously identified
using revisions (Borra et al., 2015; Kittur et al., 2007; Rad and Barbosa,
2012), specifically reverting actions like revert or remove. We randomly
sampled $0.5\%$ of the total number of items to explore similar patterns in
Wikidata. In Wikidata, we observed that the revision types revert, remove,
restore, and undo, presented $0.05\%$, $5\%$, $0.02\%$, and $0.06\%$
respectively of the total number of revisions in the item pages we examined
($11,153,126$ revisions). This means that each item had a very low frequency
of an average number of $0.01$ revert, $1$ remove, $0.01$ restore, and $0.02$
undo revisions. Moreover, the heat map of the four revision types, the number
of editors, and the total number of edits per item in Figure 5 showed that
none of these features presents a negative or positive correlation with each
other. Hence, the type of revision was not correlated with either the amount
of work or the number of people involved in the same item. Similar results
were presented in a study by Wikipedia Foundation (Sáez Trumper, 2021), where
authors also noticed that reverting actions were more associated with users’
characteristics than the items’ content. However, it is worth noting that
controversy metrics like reverting action in edit history could be used as
signal to other knowledge graph tasks. A study on knowledge graph refinement
argued that using the Wikidata edit history improved the performance of
knowledge graph embedding models (Gonzalez-Hevia and Gayo-Avello, 2022). On
the contrary, in our case, from the analysis of items’ revisions, it was clear
that disagreements cannot easily be identified using revisions in Wikidata. It
was necessary to explore further the disagreements in text-based asynchronous
discussions, threads, taking place on talk pages and communication pages (see
Table 1).

Figure 5. Heatmap with the number of different types of revisions, the number
of editors, and edits per item.

In Wikidata, only $0.02\%$ of items had a corresponding talk page; this
contrasts with Wikipedia where $27\%$ of the articles had one (Laniado et al.,
2012). Similar findings have been introduced by the Wikimedia Foundation study
(Sáez Trumper, 2021). For the property talk pages, we found that despite
$94\%$ of properties having a corresponding talk page, most of them included
tables with information about the use of the property and only $24\%$ included
discussions. To understand the discussions in item and property talk pages, we
randomly selected a sample of $70$ items (out of $23,271$) and $70$ properties
(out of $8,406$), with various numbers of topics and discussion sizes in their
talk pages. We applied thematic analysis and identified a low frequency of
disagreement ($7\%$ for items and $6\%$ for properties of the codes used in
the thematic analysis) and intensified conflict ($0.3\%$ for items and $0.1\%$
for properties) between the editors. Considering that $95\%$ of items in
Wikidata were edited by only one editor (Sáez Trumper, 2021), we concluded
that disagreements are very rare in the item and property talk pages.

To study the communication pages, we collected all Wikidata discussion pages
publicly available until November $2021$. We classified the threads based on
their discussion channel, e.g. item, property, user etc. talk pages, Project
chat, Requests for comment, etc. communication pages (see Table 1). After
splitting into threads, for each thread, we retained the information on the
number of posts, the unique editors participating in the discussion, and the
timestamps, to be able to measure the size, duration and participation levels
of each discussion. Table 2 presents the number of threads, number of posts,
and number of unique editors for each discussion channel. Our complete dataset
of Wikidata discussions includes more than $480K$ threads and $1M$ posts. The
size of the dataset and the different discussion channels led us to adopt a
new approach. This involved identifying where we can find disagreements and
then analysing controversies and argumentation. Our methodology and results
are presented in detail in Section 5 and Section 6 respectively.

Table 2. Number of threads, number of posts, and number of unique editors writing in each discussion channel from Table 1. Discussion Channel |  # threads |  # posts |  # unique editors
---|---|---|---
Item talk pages |  16,510 |  33,266 |  6,101
Property talk pages |  5,833 |  20,869 |  1,942
Users talk pages |  84,510 |  218,994 |  11,387
Wikiproject talk pages |  5,324 |  21,967 |  1,620
Other talk pages |  7,549 |  27,070 |  2,493
Project chat |  14,924 |  80,211 |  4,136
Requests for comment |  278 |  10,604 |  888
Report a technical problem |  1,967 |  8,583 |  696
Request a query |  2,414 |  9,304 |  698
Interwiki conflicts |  3,090 |  6,602 |  2,005
Bot requests |  1,115 |  5,705 |  552
Property proposal |  10,009 |  76,573 |  3,058
Administrators’ noticeboard |  6,522 |  27,939 |  2,258
Translators’ noticeboard |  241 |  903 |  135
Bureacrats’ noticeboard |  301 |  1,191 |  242
Requests for deletions |  316,469 |  681,842 |  9,935
Properties for deletion |  538 |  6,775 |  739
Requests for permission |  3,494 |  19,358 |  1,800

##  5\. Methodology

Table 3. The methods and data we use to answer our research questions. Research Question |  Method | # threads | # posts |  discussion channel used
---|---|---|---|---
RQ1 |  Descriptive Statistics | 481,088 | 1,257,756 |  all (see Table 1)
RQ1 & RQ2 |  Thematic Analysis | 69 | 1,163 |  Property proposal, Properties for deletion and Requests for a comment
RQ3 |  Measurements of Radial Trees & Statistical Tests | 69 | 1,163 |  Property proposal, Properties for deletion and Requests for a comment
RQ4 & RQ5 |  Content Analysis | 21 | 766 |  Property proposal, Properties for deletion and Requests for a comment

In the previous section, we presented our preliminary results on analysing the
Wikidata revisions and talk pages. Our findings led us to investigate
disagreements in discussions taking place on communication pages. In this
section, we present our methodology. Table 3 shows the methods and data we
used to analyse controversies in Wikidata discussions and how editors argue
within controversies.

###  5.1. $RQ1$ \- Where do we find disagreements in Wikidata?

We used descriptive statistics for the complete set of discussions in all
discussion channels ($481,088$ threads), and thematic analysis for a sample of
$69$ threads. To identify possible discussions channels including threads with
intense disagreements, we used the notion of controversies (see Section 3),
that thread including disagreements can have a high number of participants who
continuously exchange a high number of posts with opinions. We counted the
number of posts, the number of unique editors participating, and the duration
of threads in all discussion channels and identified the ones including the
highest average number of participants and length per thread. We identified
three discussion channels that could possibly include disagreements - the
Property proposal, Properties for deletion and Requests for a comment
discussion channels, that amounted to $10,825$ threads in total (see Section
6.1). Furthermore, to classify controversial and non-controversial threads we
used a standard qualitative method for text analysis to a sample of threads
from the three discussion channels. In this study, we used two qualitative
methods for text analysis: thematic and content analysis. In our analysis, one
coder read the discussions and assigned themes to annotate text as summary
markets. In thematic analysis, the goal was to qualify the data in order to
extract a thematic map (Marks and Yardley, 2004), while in content analysis,
the goal was to quantify the data based on targeted themes (Vaismoradi et al.,
2013). We chose thematic analysis for $RQ1$ and $RQ2$ to identify all possible
themes discussed in the randomly selected threads, and the content analysis
for $RQ4$ and $RQ5$ to investigate predefined patterns of argumentation.

The size of this sample was calculated to be $69$ using the Cochran’s formula
($90\%$ confidence level and $10\%$ margin error) (Israel, 1992). As such, we
sampled $23$ threads from each of the three channels. The size of our samples
is quite similar to previous qualitative studies for Wikipedia that sampled
approximately $25$ talk pages from article categories related to the language
(Hara et al., 2010), page characteristics (number of contributors, number of
views etc.)(Schneider et al., 2010), topic (controversial or not) (Viegas et
al., 2007).

After identifying controversial and non-controversial threads we continued by
investigating the main issues discussed.

###  5.2. $RQ2$ \- What are the main issues in controversial discussions in
Wikidata?

We used thematic analysis for the threads identified as controversial in
$RQ1$. In collaborative knowledge graphs, the community discusses conceptual
topics and knowledge engineering (KE) practices, i.e., the processes, methods,
languages and tools to create, maintain, and use the knowledge (Studer et al.,
1998). For this reason, we started the analysis with two main themes, “fact
accuracy” and “knowledge engineering practices”, expecting to identify the
various controversial topics. We first identified whether the thread included
a fact accuracy or a KE controversy. Next, we determined the specific cases,
e.g. disagreement among editors on what should be done with references that
did not have valid websites. Finally, we classified the specific cases into
themes e.g., sources, policies, quality, etc. to present the main issues of
controversies in Wikidata discussions. After investigating the main issues in
controversial threads we continued by studying their characteristics.

###  5.3. $RQ3$ \- What are the characteristics of controversial discussions
in Wikidata?

We used the sample of $69$ threads from $RQ1$ to study whether there was a
correlation between various features of controversial and non-controversial
threads. We used Pearson’s coefficient (Pearson, 1931) to calculate
correlation for three types of features: thread measurements, i.e., the number
of posts, the number of editors, and the duration of the discussion; the label
of controversial and non-controversial identified in $RQ1$; and measurements
of radial trees. A radial tree (Herke and Mynhardt, 2009) represents each
thread in a circular tree structure. We considered the first post, stating the
main issue of the thread, as the central node. Any post replying directly to
the first post was considered as the first nested level. Respectively, any
post replying to other posts expanded the tree by increasing the nested
levels. After including all the posts in a thread, we obtained a circular
structure that represents the direction of editors’ replies in a discussion
radially (Figure 6 shows examples of a radial tree). Using this tree
representation, we were able to study the hierarchy of discussions and the
nested patterns.

Figure 6. A radial tree for a thread discussion.

We used two measurements of radial trees for our analysis: the h-index
(Hirsch, 2005), and the depth of the tree. We measured an adapted version of
h-index (commonly used to measure researchers’ scientific output) proposed by
Gomez et al. (Gómez et al., 2008). With h-index the idea is to quantify the
impact of researchers’ work based on the number of citations their research
papers receive. The papers are listed based on their number of citations in
descending order. The h-index is defined as the maximum rank number when the
number of citations was greater or equal. In this paper we used an adapted
metric, where given a radial tree for a thread, we counted the number of radii
per nested level, starting with root node i.e. level one. The h-index was then
determined to be the maximum level for which the number of radii was greater
than or equal to the level number. For example, for the radial tree in Figure
6, the first level is the root node, the second level has $12$ radii, the
third $5$, the fourth $2$, and so on and so forth, giving an h-index of $2$.
Furthermore, we measured the depth, i.e., the maximum number of levels for the
radial trees to investigate the sub-threads in the discussions. This turned
out to be $6$ for the previous example in Figure 6. After the study of
controversial thread we continued our analysis by exploring argumentation.

###  5.4. $RQ4$ \- How do editors argue when they disagree in Wikidata?

We investigated the argumentation techniques used in controversial discussions
to express opinions. We used content analysis (Vaismoradi et al., 2013) for
the threads identified as controversial in $RQ1$.

Table 4. Coding scheme used for the content analysis to study the arguments based on the Diligent argumentation framework (Tempich et al., 2007). Code/Argument |  Description
---|---
Issue |  It is the main subject starting the discussion.
Idea |  It is a new subject for discussion during the argumentation process.
Elaboration |  It is the extension of an issue presenting additional details.
Justification - Example |  Justification is an argument supporting an idea or issue. Example is used to provide evidence to increase the belief on a matter.
Justification - Evaluation |  Justification is an argument supporting an idea or issue. Evaluation gives a measurable mean to increase belief on a matter.
Challenge - Counter Example |  Challenge is an argument against an idea or issue. Counter Example provides counter evidence for a matter.
Challenge - Alternative |  Challenge is argument against an idea or issue. Alternative is used to suggest an alternative solution on a matter.
Position - Agree |  This clarifies the participant’s position on an issue, idea, or argument, either by voting support or by agreeing within the sequence of arguments.
Position - Disagree |  This clarifies the participant’s position on an issue, idea, or argument, either by voting oppose or by disagreeing within the sequence of arguments.
Decision - Under discussion |  The issue is still open and anyone can argue.
Decision - Postponed |  There was no consensus on the issue, but it is frozen and no one can express a new opinion on it.
Decision - Discarded |  The initial issue or an idea was rejected as not relevant.
Decision - Agreed |  The final decision has been taken regarding the initial issue or an idea that was formalised over the sequence of arguments.

Table 4 presents the coding scheme we used to identify patterns of
augmentations and answer $RQ4$. Previous coding schemes in knowledge
collaboration projects like Wikipedia (Hara and Sanfilippo, 2016; Fréard et
al., 2010) have focused on knowledge sharing. Furthermore, similar studies in
open source software development projects used a general field argumentation
model by Toulman (Wang et al., 2020). However, in order to focus on the
argumentation patterns in collaborative knowledge engineering, we chose to use
the Diligent argumentation framework (Tempich et al., 2007). The Diligent
argumentation has been designed for collaborative knowledge engineering
projects with main concepts being issues, ideas and arguments. The issue
introduces the initial subject for discussion. An idea is a response to an
issue. Moreover, arguments, positive or negative, could justify or challenge
an issue or ideas. Furthermore, positions could agree or disagree with issues,
ideas, or arguments. Finally, the discussion state is shown by a decision,
which could be still open and under discussion, or postponed, discarded or
agreed. With this coding scheme, we identified types of arguments in
controversial threads and we continued our analysis with the role of editors
in argumentation.

###  5.5. $RQ5$ \- What are the roles of editors in argumentation in Wikidata?

Similar to $RQ4$, we used content analysis for the threads identified as
controversial in $RQ1$. The coding scheme used to study the roles of the
different participants is presented in Table 5. This scheme was based on a
study by Jain et al. (Jain et al., 2014) that identified social roles in
contentious online discussions. Other similar studies about the content of
arguments were focused on the roles based on knowledge sharing and modifying,
and facilitating collaboration, such as knowledge shaper, organizer, giver
etc. (Faraj et al., 2011; Kane et al., 2014; Hara and Sanfilippo, 2017).
However, our aim was to understand the roles of social interactions. We used
the roles suggested by Jain et al. (Jain et al., 2014) with small variations
to the names as leader, follower, rebel, ignored rebel, outsider, loner,
disruptor. This coding scheme helped us to investigate types of editors in
argumentation in the sense of whether we could find editors that influence
others, editors whose opinions are ignored, editors who have minimum
participation, or spammers and trolls.

Table 5. Coding scheme used for the content analysis to study the roles of participants in argumentation. Participant role |  Description
---|---
Leader |  An editor who manages to influence another editor to change her/his opinion or influence her/him to support her/his arguments.
Follower |  An editor who is influenced by another editor and changes her/his opinion or supports the other editor’s argument usually follows the “Leader”. She/he can be an editor who does not have her/his own arguments.
Rebel |  An editor who drives the discussion in some direction. She/he is devoted to the discussion and engaged with other editors. A “Rebel” posts sensible arguments which cannot be ignored by others.
Ignored Rebel |  An editor with similar behaviour as a “Rebel”. The main difference is that hers//his arguments are ignored by others.
Outsider |  An editor who does not contribute to the main purpose of the discussion. She/he is engaged in the discussion but the content is either emotional, illogical, or out of topic.
Loner |  An editor who makes minimal but legitimate contributions to the discussion.
Disruptor |  An editor who makes minimal and unuseful contributions to the discussion. She/he can be a potential spammer.
Others |  For editor who do not follow any of the previous behavioural patterns.

##  6\. Results

In this section, we present the insights from our analysis on discussions
towards the different research questions.

###  6.1. $RQ1$: Where do we find disagreements Wikidata?

We suggest that Properties for deletion, Property proposal, and Requests for
comments discussion channels are the ones that are most likely to include
threads with disagreements. For a sample of threads randomly selected from the
tree discussion channels, we identified $30\%$ as controversial.

Our analysis shows that the discussion channel with the highest traffic (i.e.,
the most visited channel with the highest number of threads) is Requests for
deletion, presenting $66\%$ of the total threads ($316,469$ threads, and
$681,842$ posts). This is followed by the User talk page channel with $18\%$
of the total threads ($84,510$ threads, and $218,994$ posts). Figure 7 shows
(a) the number of threads and (b) the number of posts for the different
discussion channels.

Figure 7. (a) the number of threads and (b) the number of posts for the
different discussion channels.

Based on our notion of intense disagreements (see Section 3 and Section 5)
potential threads with disagreements can include a high number of posts and a
high number of participants. While Requests for deletion and User talk page
are the most visited channels, they host very short threads with an average
number of posts per thread of $2$ and $3$ respectively (see Figure 8 (a)) and
very short duration (Figure 8 (b) shows one day for Requests for deletion).
This makes the evolution into a disagreement unlikely. Figure 8 (a) shows that
Properties for deletion, Property proposal, and Requests for comments present
the highest number of posts and editors per thread in Wikidata indicating that
they may include controversial discussions.

Figure 8. (a) the mean number of posts and editors per thread and (b) the mean
duration of threads for the different discussion channels.

As mentioned in Section 5, to investigate this hypothesis we first sampled the
threads from the three channels based on Cochran’s Formula which resulted in
$69$ threads in total ($23$ per discussion channel). With thematic analysis,
$30\%$ ($21$ out of $69$) of threads were assigned to be controversial. To be
specific, we identified $30\%$ of the threads at Properties for deletion ($7$
out of $23$), $9\%$ of threads at Property proposal ($2$ out of $23$), and
$52\%$ of the threads at Requests for comments ($12$ out of $23$) as
controversial. We found that Requests for comments, with a mean number of $38$
posts and $11$ editors per thread, is the most controversial among the three
discussion channels that were analysed. However, for Property proposal, with a
mean number of $8$ posts and $5$ editors per thread, we found only $2$
controversial threads.

The results suggest that a high number of posts combined with a high number of
editors in threads could indicate controversial discussions. However, we
conjecture that despite the length of the thread, the content of the
discussion and the policy of use for each discussion channel have an essential
role in controversies and their evolution. For this reason, we further
analysed the controversial threads to identify the main issues of controversy
in Wikidata.

###  6.2. $RQ2$: What are the main issues in controversial discussions in
Wikidata?

Figure 9. An example discussion from Wikidata Request for comments discussion
channel. The first component of the example includes the initial request and
the current status of the issue. The second component of the example shows
part of the discussion with an intense disagreement and two counter example
arguments. The last component of the example shows part of the discussions
with a conflicting interaction.

We found that the main issues in controversial discussions are related to KE
and particularly concerns process controversies (i.e., policies or practices).
Furthermore, controversies do not often lead to conflicts. We found that more
than half of controversial discussions do not reach a consensus, while in the
Property proposal channel, a couple of votes are enough to take decisions on
the creation of a property contrasting the decision on the deletion of a
property.

As described in Section 5, in collaborative knowledge engineering projects we
expected to find two main themes for disagreements, one related to fact
accuracy, and another related to KE practices. We found that $20\%$ ($4$ out
of $21$) of the controversial threads are related to fact accuracy with $3$,
including both fact accuracy and KE practices issues. The majority of threads,
$95\%$ ($20$ out of $21$) of threads, are related to KE practices issues.

For fact accuracy, we found three cases regarding the use of specific
properties based on the meaning of their name. For example, this happens for
the properties “given name”, “surname”, and “birth name”, where editors
disagreed on the appropriate property to be used in different cases. The
properties “developer” and “programmer” also have a similar case of
disagreement. Further, a controversy evolved about the creation of a new
property named “at” since its generic naming could cause semantic
misunderstandings. Finally, we found a discussion about the evaluation of data
quality in Wikidata, where editors disagreed on the definitions of data and
information quality.

Figure 10. Part of a discussion about how Wikidata should order the list of
properties.

Figure 11. (a) Part of a discussion for a suggestion to delete a property. The
participants often refer to discussions on other pages, especially in the
property proposal of this property. (b) Part of a discussion between two
editors, who are suggesting that this discussion should be held in one place.

For KE practices, we found that Requests for comments includes issues related
to interwiki links (i.e., links that connect the different Wikimedia projects
like Wikipedia, Wikibooks, Wikisource, etc.), the references linked to the
pages as justification of facts, the quality of the project, the policies and
practices of Wikidata, and the use of properties. However, for the other two
discussion channels, as their name and purpose indicate, we found issues
related to the use of properties. The references linked to sources to verify
the accuracy of facts connected to items. Editors disagreed on the type of
sources they used to justify the facts, and the changes made over time to the
source websites. Figure 9 shows an example of a controversy about references.
The reference issue is the use of the source “Find a Grave” for the date of
death. The discussion was active for $6$ months, including $73$ posts and $17$
unique editors. The editors disagreed on the reliability of this source, since
it is a community-sourced project, and on its use as a reference. Furthermore,
changes in guidelines, policies and practices attract a variety of opinions in
any peer-collaboration. In Wikidata, we found controversies about changes in
guidelines regarding the creation of a property or the writing instructions in
the discussion channel Request for comment. Figure 10 presents another case
where editors disagreed on the manner in which Wikidata should present the
list of properties, for example, in alphanumeric order, in groups based on the
theme, or by listing the most important properties at the top. The figure
shows that editors vote support on this issue but they disagree on the way it
should be done. In addition, controversies about the use of properties are
related to constraints (i.e., rules for the use of the properties), qualifiers
(i.e., properties used in combination with other properties to extend their
semantic connections), and taxonomic use (i.e., properties connect items to
classes and subclasses). We found that often in threads about property issues,
editors point to discussions about the same issues that occur in other
discussion channels. Figure 11 (a) shows an example of a discussion related to
the deletion of a property, and editors refer to the discussion related to the
proposal of this property. Furthermore, Figure 11 (b) points out the confusion
of two participants regarding discussions about the same topic in multiple
places. This may suggest that discussions about properties, proposal or
deletion, need attention.

Figure 12. Examples of (a) task controversy related to property taxonomy, and
(b) process controversy related to qualifier guidelines

Based on previous studies in peer-production about conflict categorisation
(Filippova and Cho, 2015; Arazy et al., 2013) (see Section 5), we identified
two types of controversies in the analysed threads, namely task and process.
In Wikidata one can think of task controversy as disagreements about the
editing of an item or properties and process controversy as disagreements on
how to perform a task rather than the task itself. The latter is related to
policies or practices set by the community in order to preserve knowledge
consistency, e.g., the use of the instance of (P31) property when one cannot
specify the subclass of (P279) property in an item.

Figure 12 (a) shows an example of task controversy and Figure 12 (b) for
process controversy from our analysis. For task controversy ($38\%$ of
threads), the issues were related mostly to fact accuracy and particularly to
properties e.g., disagreement on decisions for the use of “given name”,
“surname”, and “birth name” properties. Furthermore, for KE practices, task
controversy could be seen in the taxonomy of properties, e.g., a disagreement
for the deletion of a property due to the discontinuity between Finland and
Swedish wikis. For process controversy ($52\%$ of threads), the issues were
related to KE practices. We found issues about guidelines, policies and
practices, e.g., guidelines in links connecting the different Wikimedia
projects, and policies on use of property constraints and qualifiers.

Figure 13. Wordcloud for the posts identified as conflicts.

While we identified controversial threads in Wikidata, we found that
controversies do not lead to intense conflicts with use of strong language or
threats. Conflicting posts were either ironic, indicated anger, or used
expressions like “nonsense” and “this is irrelevant” (see Figure 9). Figure 13
presents a wordcloud for the posts we identified as a conflict. We can see
that the main topics of conflicts are “sources” and “quality”. $30\%$ of the
threads identified as controversial included minor conflicts which lasted from
$2$ to $4$ posts and arose between two editors. The other editors
participating in the thread continued with their arguments and did not get
involved in the conflict. In most case, controversies is Wikidata evolve with
peaceful arguments without any intense conflict. The reason behind this could
be the fact that editors do not revert or change the work of others
repeatedly. This is different from co-editing activities in Wikipedia, such as
writing an article, which could lead to edit wars since in Wikidata each
suggestion needs to reach a consensus to proceed to changes.

Studying the decisions in our analysed threads we found that often, there is
no consensus in controversial discussions, $62\%$ ($13$ out of $21$) of
threads did not lead to consensus. On the contrary, in non-controversial
threads, $38\%$ ($18$ out of $48$) did not lead to consensus. We found many
cases where the discussion was open for months without being active, and
administrators closed them without consensus.

Regarding the non-controversial threads, we identified two types: short
threads and the threads that include many vote posts with no further
justification. The Property proposal was the least controversial between the
three channels. This could be because the editors reach a consensus with $2$
to $3$ votes and thus, disagreement is more unlikely. This contrasts with the
other two discussion channels where we found extended voting and elaboration
around the discussed issue.

###  6.3. $RQ3$: What are the characteristics of controversial discussions in
Wikidata?

To find the characteristics of controversial threads, we used the sample of
$69$ threads and the labels of controversial and non-controversial threads
from $RQ1$. Based on descriptive statistics and the radial trees built for
each thread, we measured the number of posts, the number of unique editors
participating in the discussions, the duration of the discussion, the h-index,
and the depth of the thread (i.e., the number of levels in a thread). We found
that controversies are correlated with the number of participants and the
depth of the thread, but not with the duration of the thread. Furthermore, in
controversial discussions it is likely to find many participants exchanging
fewer posts than few participants exchanging many posts.

Figure 14 presents the correlation matrix of these measurements (Pearson’s
correlation (Pearson, 1931)). The results indicate that a controversy is
correlated with the number of editors and the depth of the thread and less to
the h-index. It is worth noting that the number of participants, i.e., the
number of editors, is correlated to controversies more than the length of the
discussion, i.e., the number of posts. Furthermore, the depth of the thread is
highly correlated to the number of posts and editors. This was expected since
the number of posts increases with increasing levels. In addition, there is a
high correlation between the depth of the thread and the number of editors,
and between editors and posts. This indicates that in controversial threads it
is likely to find many participants exchanging fewer posts each than few
participants exchanging many posts. Finally, the duration of the discussion
has no correlation with any feature, indicating that the time response has no
influence on the course of the discussion (see also Figure 14). This may come
from the editors’ individual habits on editing Wikidata, which can vary in
days or weeks. Therefore, the frequency of participation in discussions can
vary, suggesting that every issue can have an unpredictable duration.

While depth seemed like a good characteristic for controversial threads, it
might be biased due to the structure templates that editors use to raise an
issue. Editors can use specific templates to raise an issue, explain their
proposal, give examples, and specify further information depending on the
channel. Specifically, for Requests for comment channel, the issues are
generally related to the project’s practices. For the Request for comment we
found that using the template editors describe the main issue at the beginning
of the request and create sub-issues with suggested solutions or ideas for the
community to discuss. This creates multiple sub-discussions under the main
issue and increases the depth of the thread.

Figure 14. Correlation matrix for features related to thread characteristics

###  6.4. $RQ4$: How do editors argue when they disagree in Wikidata?

To analyse the way in which editors argue, we used the coding scheme from
Table 4. In a few cases where the posts were long including several arguments,
we assigned more than one code. We found that editors most frequently disagree
using a counter example. Counter example can be an example of a similar case
or of policies and rules.

Table 6 shows the percentage of posts assigned with the argumentation scheme
to the sample of $21$ threads that were identified as controversial in $RQ1$.
Results showed that editors frequently argue by disagreeing with the main
issue or other opinions using Challenge - Counter example ($33\%$), followed
by Challenge - Alternative, ($14\%$). For Wikidata, counter example could
serve as an example of a similar case, but also as an example of practices and
rules (see Figure 9 for counter examples). Consequently, the next most used
argument was for agreement with other opinions. Editors use Justification -
Example, $10\%$. Similar to the case of Challenge - Counter example,
Justification - Example could serve as an example of similar cases, but also
as an example of policies.

The codes Issue and Idea were the ones that point respectively to the main
issues that were suggested for discussion in the three channels and any new
ideas introduced during the discussion for taking a decision over the issue.
Results showed that a new idea is introduced often($10\%$). This code usually
was assigned in combination with the opposing code Challenge-Alternative.
Figure 15 shows an example for the use of both codes, where an editor
disagrees with the proposal to create a property naming at.

Figure 15. An example for the combined use of code Challenge-Alternative and Idea. Table 6. This table presents the percentage of the codes assigned to the posts of 21 controversial discussions in order to analyse the type of arguments. Code |  %
---|---
Challenge - Counter Example |  33
Challenge - Alternative |  14
Justification - Example |  10
Justification - Evaluation |  5
Issue |  3
Idea |  10
Elaboration |  9
Position - Agree |  5
Position - Disagree |  8
Decision - Under discussion |  0.1
Decision - Postponed |  1
Decision - Discarded |  0.1
Decision - Agreed |  1

We used the Position codes to find the agreements and disagreements in the
threads for the cases where editors chose sides, mainly by voting, but did not
express opinions to justify their choice. In this case, similar to codes for
Challenge where editors disagree with an opinion and Justification where
editors agree, editors more often Disagree ($8\%$) than Agree ($5\%$).

Finally, our coding scheme included codes relating to Decision to capture the
ratio of decision making in controversial threads. We found cases where
editors reached an agreement or the discussion was postponed, and there were
other different cases where the discussion was discarded or it is still open.
Figure 9 shows he status of this discussion at the top, which is “consensus”.

###  6.5. $RQ5$: What are the roles of editors in an argumentation?

We found that the majority of participants are Rebels, i.e., editors with
strong ideas, followed by Loners, i.e., editors who express their opinions
with legit arguments but do not engage in the discussion. In addition, we
found a low frequency of Ignored Rebel, suggesting that most of the opinions
are taken into account, and a low frequency of Disruptors indicating low
levels of vandalism.

We used the coding scheme from Table 5 to assign one code for each editor.
Table 7 shows the results of the content analysis about the roles of editors
in the argumentation.

All analyzed threads included Rebels and Loners. We found that the majority of
the posts (46%) were posted by a Rebel. This indicated that most frequently,
editors had strong opinions and aimed to change others’ opinions in their
direction. We also found cases of controversy where a Rebel, very early in a
discussion, stated an opinion and then responded to every editor with an
opposing opinion. In addition, many posts (25%) were posted by a Loner. These
editors had minimal participation in the discussion but posted valid and
useful arguments for the discussion. This often happened in the format of a
voting process where editors voted and justified their opinion but did not
argue or interact further with other editors in the discussion.

The high number of posts by Rebels, combined with the low number of posts by
Leaders, showed that editors did not often simply follow the ideas of others,
but rather expressed and defended their own opinions. We also identified a low
number of posts by Followers (8%), who did not necessarily change their
opinions but often voted using other editors’ opinions as justification.

We identified a low number of posts by Outsiders (9%). These editors
participated in the voting process without justifying their votes or
participating in the rest of the discussion. Furthermore, we found a low
number of posts by Ignored Rebel or Disruptors. This means that at the core of
Wikidata, all editors who were willing to participate in a discussion received
a response from the community. Additionally, there were no vandals among the
editors. We did find one case of possible vandalism in the proposition of a
Property for deletion. In the early stages of this thread, there was some
confusion when some editors suggested that the request may be fake based on
the proposer’s editing activity and the lack of description in the
proposition. However, others expressed arguments as usual. Very soon, the
disruption stopped, and the discussion continued.

Table 7. This table presents the percentage of the codes assigned to the posts of 21 controversial discussions in order to analyse the role of participants in argumentation. Participant role |  %
---|---
Rebel |  46
Loner |  25
Leader |  8
Follower |  8
Irrelevant |  9
Ignored |  2
Disruptor |  1
Others |  1

Issues are usually started by a Loner or a Rebel in the Property Proposal and
Properties for Deletion discussion channels. However, in Requests for
Comments, the main issues were quite often not signed by editors to identify
their roles. In controversial discussions, Rebels and Loners, who are the most
usual roles, most often used the argument Counter example, with 39% and 37%
respectively, followed by Alternative with 17% and 14%.

##  7\. Discussion

In this work, we have studied Wikidata discussions to understand how editors
disagree in collaborative knowledge graphs. We found disagreements, identified
controversial threads, investigated the main themes, and explored their
characteristics through a variety of measurements. Additionally, we studied
the argumentation structure and editors’ roles in argumentation.

$RQ1$ showed that $30\%$ of the analysed threads identified as controversial.
For the two channels related to properties, Properties for deletion attract
more disagreements and include longer discussions, than Property proposal. In
addition, in $RQ2$ we found that for Property proposal a few votes are enough
to take decisions. These results suggest that the community takes fast
decisions on creating properties, and it is worth further analysis of whether
these fast decisions in creating properties impact the quality of Wikidata.

$RQ2$ indicated that the most frequently identified theme of controversy in
Wikidata is the process controversy ($52\%$). This contrasts the results
related to Wikipedia and open-source software development projects where the
most frequent type is the task controversy (Arazy et al., 2013; Filippova and
Cho, 2016). The difference in these results between the peer-production
projects may come from the nature of the projects and the purpose of the
analysed discussion channels. In collaborative knowledge graphs, the goal is
to create a structural artifact using cognitive knowledge but also procedural
rules, like editing item pages and following similar taxonomies with other
cases in the project (similar to process controversy), while in Wikipedia, for
example, writing an article is more related to cognitive knowledge (similar to
task controversy). This particular difference in knowledge graphs compared to
other peer-production projects may imply the need for clear policies or extra
guidelines related to the process of edit and the hierarchy of data, i.e.
classes, subclasses, etc.

$RQ3$ showed that measurements of radial trees like h-index was not enough to
indicate individually if a discussion includes controversies. Similar results
can be seen in online discussions in Wikipedia (Laniado et al., 2011) and
Slashdot (Gómez et al., 2008). In Wikipedia, similar studies suggested that
h-index is a good indicator for the depth of the discussion but the chains of
direct replies were a good indicator for contentious topics. Furthermore, in
Slashdot, the authors found that many discussions shared the same h-index and
suggested another measurement, the sum of h-index and the inverse of the
number of posts, to break the ties.

In addition, $RQ3$ suggests that the number of participants presented a higher
positive correlation with controversial threads than the length of the
discussion. In contrast, studies for the detection of conflicts in Wikipedia
show the opposite, proving that the most valuable feature in conflict
prediction was the number of edits in the article talk pages, while the number
of unique participants on the talk page is negatively correlated with conflict
(Kittur et al., 2007). The results point to one fundamental difference between
the two projects and similar to other studies (Sáez Trumper, 2021; Filippova
and Cho, 2016; Kittur and Kraut, 2010) suggest considering their different
editing processes and their structures.

$RQ4$ indicated that counter example using similar cases, and examples of
policies and practices was the most used argument. This is similar to results
in Wikipedia where the study of suggestions for the deletion of Wikipedia
articles, using the Walton argumentation scheme, showed that among others
rules and evidence comprise $36\%$ of the arguments (Schneider et al., 2013).
These results highlight again the importance of clear and detailed policies.
Our findings suggest that project designers and managers need to support
members with easy-to-access and clear instructions regarding policies and
guidelines in order to help them participate in discussions with legitimate
and evident arguments.

Our argumentation results also identified that editors most often disagree
when arguing (the sum of Challenge codes was $47\%$), as compared to agreeing
(the sum of Justification codes was $15\%$). This contrasts with results from
argumentation analysis in an open source software development project in which
the authors found $13\%$ disagreement codes and $46\%$ agreement codes using
data from GitHub using Toulmin’s argumentation model (Wang et al., 2020). This
difference may come from the difference in the datasets, where our analysis in
Wikidata is focused on controversial threads, while the GitHub analysis chose
five threads with a common topic based on the application used, heterogeneity
and a high number of participants, and the variety of issues over this topic.
Furthermore, we found differences between the two projects in the number of
non-argumentative posts in the discussions. Looking the Position codes, we
found that in Wikidata, $25\%$ of codes were not argumentative, while in
GitHub it was almost twice the size, i.e. $41\%$ of codes (Wang et al., 2020).
The high number of posts with no argument in a long discussion with many
participants may confuse the editors and make the final decision harder. In
this case, a tool to summarise legitimate opinions and hide unrelated comments
may support decision making.

$RQ5$ revealed that a high number of participants, loners ($25\%$), did not
engage in the discussion. The number of loners may indicate the reasons behind
the lack of decision making (we found $62\%$ of controversial discussion with
no consensus). Participants who do not engage in discussions do not support
their arguments or do not accept other valid arguments to conclude in
decisions, thus leading the discussion to be closed because it remains
inactive for a long, but with no consensus. These results, combined with our
findings above that editors mostly disagree and that a high number of posts do
not include arguments, may form explanations for the lack of decision making.
All results indicate the need to change strategies in making decision when
there is a long discussion with a high number of participants. A tool for
flagging long discussions with controversies and maybe a list of arguments
included in the discussion could support the community to make decisions.

The limitations of our study were the lack of information regarding
participants’ characteristics and the investigation of three specific
discussion channels in Wikidata. Including information regarding the age of
editors in Wikidata, their rights, and the rhythm of participation in editing
activities could reveal more insights regarding how decisions are made in
Wikidata and by whom. Furthermore, we analysed three discussion channels with
topics related to the construction of the graph and suggestions regarding
general policies and practices. Other discussion channels might include
disagreements at smaller scale, but based on the topic of the channel the
controversial themes might differ.

##  8\. Conclusion and Future work

Community interactions, particularly disagreements, form the means to study
how community functions and take decisions. In this work, we used descriptive
statistics, thematic analysis, tree construction, statistical tests and
content analysis to analyse disagreements in the collaborative KG Wikidata. We
identified and explored three discussion channels as candidates that include
disagreements, Request for comment, Properties for deletion and Property
proposal. Our main findings are that quick decisions are taken in the creation
of properties, the majority of controversies are related to policies and
practices, and that more than half of controversial discussions do not lead to
consensus. Therefore, we recommend to the Wikidata community to pay attention
to the quick decisions regarding properties which still need to be explored in
order to understand the impact on the quality; enable clear and easy access of
policies and rules to help editors participate efficiently in argumentation;
and improve strategies for making consensus in long and controversial
discussion so as to avoid unresolved issues.

In future work, we aim to extend our analysis to understand editors and their
interaction in collaboration. Specifically, we are interested in exploring
questions such as: Who participates more in discussions, newcomers or
experienced editors? How do newcomers and experienced editors argue?
Furthermore, it would be worthwhile to investigate how quickly decisions,
particularly those related to the creation of core entities like properties,
impact the quality of the knowledge graph over time.

## References

  * (1)
  * Alsinet et al. (2020) Teresa Alsinet, Josep Argelich, Ramón Béjar, and Santi Martínez. 2020.  Measuring user relevance in online debates through an argumentative model.  _Pattern Recognition Letters_ 133 (2020), 41–47.
  * Arazy et al. (2011) Ofer Arazy, Oded Nov, Raymond Patterson, and Lisa Yeo. 2011\.  Information quality in Wikipedia: The effects of group composition and task conflict.  _Journal of management information systems_ 27, 4 (2011), 71–98.
  * Arazy et al. (2013) Ofer Arazy, Lisa Yeo, and Oded Nov. 2013.  Stay on the Wikipedia task: When task-related disagreements slip into personal and procedural conflicts.  _Journal of the American Society for Information Science and Technology_ 64, 8 (2013), 1634–1648.
  * Beck et al. (2018) Jordan Beck, Bikalpa Neupane, and John M Carroll. 2018\.  Managing Conflict in Online Debate Communities: Foregrounding Moderators’ Beliefs and Values on Kialo.  (2018).
  * Benkler et al. (2015) Yochai Benkler, Aaron Shaw, and Benjamin Mako Hill. 2015\.  Peer production: A form of collective intelligence.  _Handbook of collective intelligence_ 175 (2015).
  * Biran et al. (2012) Or Biran, Sara Rosenthal, Jacob Andreas, Kathleen McKeown, and Owen Rambow. 2012.  Detecting influencers in written online conversations.  (2012).
  * Borra et al. (2015) Erik Borra, Esther Weltevrede, Paolo Ciuccarelli, Andreas Kaltenbrunner, David Laniado, Giovanni Magni, Michele Mauri, Richard Rogers, and Tommaso Venturini. 2015.  Societal controversies in Wikipedia articles. In _Proceedings of the 33rd annual ACM conference on human factors in computing systems_. 193–196.
  * Brandes et al. (2009) Ulrik Brandes, Patrick Kenis, Jürgen Lerner, and Denise Van Raaij. 2009.  Network analysis of collaboration structure in Wikipedia. In _Proceedings of the 18th international conference on World wide web_. 731–740.
  * Bykau et al. (2015) Siarhei Bykau, Flip Korn, Divesh Srivastava, and Yannis Velegrakis. 2015.  Fine-grained controversy detection in Wikipedia. In _2015 IEEE 31st International Conference on Data Engineering_. IEEE, 1573–1584.
  * Danescu-Niculescu-Mizil et al. (2012) Cristian Danescu-Niculescu-Mizil, Lillian Lee, Bo Pang, and Jon Kleinberg. 2012.  Echoes of power: Language effects and power differences in social interaction. In _Proceedings of the 21st international conference on World Wide Web_. 699–708.
  * De Dreu and Weingart (2003) Carsten KW De Dreu and Laurie R Weingart. 2003.  Task Versus Relationship Conflict, Team Performance, and Team Member Satisfaction: A Meta-Analysis.  _Journal of Applied Psychology_ 88, 4 (2003), 741–749.
  * De Kock and Vlachos (2021) Christine De Kock and Andreas Vlachos. 2021.  I Beg to Differ: A study of constructive disagreement in online conversations. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_. 2017–2027.
  * Durmus and Cardie (2019) Esin Durmus and Claire Cardie. 2019.  A corpus for modeling user and language effects in argumentation on online debating.  _arXiv preprint arXiv:1906.11310_ (2019).
  * Easterbrook (1993) Steve M Easterbrook. 1993\.  Cooperation or Conflict?
  * Ehrlinger and Wöß (2016) Lisa Ehrlinger and Wolfram Wöß. 2016.  Towards a Definition of Knowledge Graphs.  _SEMANTiCS (Posters, Demos, SuCCESS)_ 48 (2016), 1–4.
  * Elliott and Scacchi (2002) M Elliott and Walt Scacchi. 2002.  Communicating and mitigating conflict in open source software development projects.  _Projects & Profits_ (2002), 25–41.
  * Faraj et al. (2011) Samer Faraj, Sirkka L Jarvenpaa, and Ann Majchrzak. 2011\.  Knowledge collaboration in online communities.  _Organization science_ 22, 5 (2011), 1224–1239.
  * Filippova and Cho (2015) Anna Filippova and Hichang Cho. 2015.  Mudslinging and manners: Unpacking conflict in free and open source software. In _Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing_. 1393–1403.
  * Filippova and Cho (2016) Anna Filippova and Hichang Cho. 2016.  The effects and antecedents of conflict in free and open source software development. In _Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing_. 705–716.
  * Franco et al. (1995) Vivian Franco, R Piirto, H-Y Hu, Bruce V Lewenstein, Russ Underwood, and Noni Korf Vidal. 1995\.  Anatomy of a flame: conflict and community building on the Internet.  _IEEE Technology and Society Magazine_ 14, 2 (1995), 12–21.
  * Fréard et al. (2010) Dominique Fréard, Alexandre Denis, Françoise Détienne, Michael Baker, Matthieu Quignard, and Flore Barcellini. 2010\.  The role of argumentation in online epistemic communities: the anatomy of a conflict in Wikipedia. In _Proceedings of the 28th Annual European conference on cognitive ergonomics_. 91–98.
  * Garimella et al. (2018) Kiran Garimella, Gianmarco De Francisci Morales, Aristides Gionis, and Michael Mathioudakis. 2018.  Quantifying controversy on social media.  _ACM Transactions on Social Computing_ 1, 1 (2018), 1–27.
  * Gómez et al. (2008) Vicenç Gómez, Andreas Kaltenbrunner, and Vicente López. 2008.  Statistical analysis of the social network and discussion threads in slashdot. In _Proceedings of the 17th international conference on World Wide Web_. 645–654.
  * Gonzalez-Hevia and Gayo-Avello (2022) Alejandro Gonzalez-Hevia and Daniel Gayo-Avello. 2022.  Leveraging Wikidata’s edit history in knowledge graph refinement tasks.  _arXiv preprint arXiv:2210.15495_ (2022).
  * Habernal and Gurevych (2016) Ivan Habernal and Iryna Gurevych. 2016.  What makes a convincing argument? empirical analysis and detecting attributes of convincingness in web argumentation. In _Proceedings of the 2016 conference on empirical methods in natural language processing_. 1214–1223.
  * Hara and Sanfilippo (2016) Noriko Hara and Madelyn Rose Sanfilippo. 2016.  Co-constructing controversy: Content analysis of collaborative knowledge negotiation in online communities.  _Information, Communication & Society_ 19, 11 (2016), 1587–1604.
  * Hara and Sanfilippo (2017) Noriko Hara and Madelyn Rose Sanfilippo. 2017.  Analysis of roles in engaging contentious online discussions in science.  _Journal of the Association for Information Science and Technology_ 68, 8 (2017), 1953–1966.
  * Hara et al. (2010) Noriko Hara, Pnina Shachaf, and Khe Foon Hew. 2010\.  Cross-cultural analysis of the Wikipedia community.  _Journal of the American Society for Information Science and Technology_ 61, 10 (2010), 2097–2108.
  * Hassan et al. (2012) Ahmed Hassan, Amjad Abu-Jbara, and Dragomir Radev. 2012\.  Detecting subgroups in online discussions by modeling positive and negative relations among participants. In _Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning_. 59–70.
  * Herke and Mynhardt (2009) Sarada Herke and Christina M Mynhardt. 2009.  Radial trees.  _Discrete mathematics_ 309, 20 (2009), 5950–5962.
  * Hirsch (2005) Jorge E Hirsch. 2005\.  An index to quantify an individual’s scientific research output.  _Proceedings of the National academy of Sciences_ 102, 46 (2005), 16569–16572.
  * Ho-Dac et al. (2016) Lydia-Mai Ho-Dac, Veronika Laippala, Céline Poudat, and Ludovic Tanguy. 2016.  French Wikipedia talk pages: Profiling and conflict detection. In _4th Conference on CMC and Social Media Corpora for the Humanities_.
  * Hogan et al. (2021) Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d’Amato, Gerard de Melo, Claudio Gutierrez, Sabrina Kirrane, José Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, et al. 2021\.  Knowledge graphs.  _Synthesis Lectures on Data, Semantics, and Knowledge_ 12, 2 (2021), 1–257.
  * Israel (1992) Glenn D Israel. 1992\.  _Determining sample size_.  Technical Report. University of Florida, Cooperative Extension Service, Institute of Food and Agricultural Sciences.
  * Jain et al. (2014) Siddharth Jain, Archna Bhatia, Angelique Rein, and Eduard Hovy. 2014.  A corpus of participant roles in contentious discussions. In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_. 1751–1756.
  * Jankowski-Lorek et al. (2014) Michał Jankowski-Lorek, Radosław Nielek, Adam Wierzbicki, and Kazimierz Zieliński. 2014\.  Predicting controversy of Wikipedia articles using the article feedback tool. In _Proceedings of the 2014 international conference on social computing_. 1–7.
  * Jasser et al. (2022) Jasser Jasser, Ivan Garibay, Steve Scheinert, and Alexander V Mantzaris. 2022.  Controversial information spreads faster and further than non-controversial information in Reddit.  _Journal of Computational Social Science_ 5, 1 (2022), 111–122.
  * Jehn (1995) Karen A Jehn. 1995\.  A multimethod examination of the benefits and detriments of intragroup conflict.  _Administrative science quarterly_ (1995), 256–282.
  * Kane et al. (2014) Gerald C Kane, Jeremiah Johnson, and Ann Majchrzak. 2014\.  Emergent life cycle: The tension between knowledge change and knowledge retention in open online coproduction communities.  _Management Science_ 60, 12 (2014), 3026–3048.
  * Kittur and Kraut (2010) Aniket Kittur and Robert E Kraut. 2010.  Beyond Wikipedia: coordination and conflict in online production groups. In _Proceedings of the 2010 ACM conference on Computer supported cooperative work_. 215–224.
  * Kittur et al. (2007) Aniket Kittur, Bongwon Suh, Bryan A Pendleton, and Ed H Chi. 2007\.  He says, she says: conflict and coordination in Wikipedia. In _Proceedings of the SIGCHI conference on Human factors in computing systems_. 453–462.
  * Laniado et al. (2012) David Laniado, Andreas Kaltenbrunner, Carlos Castillo, and Mayo Fuster Morell. 2012.  Emotions and dialogue in a peer-production community: the case of Wikipedia. In _proceedings of the eighth annual international symposium on wikis and open collaboration_. 1–10.
  * Laniado et al. (2011) David Laniado, Riccardo Tasso, Yana Volkovich, and Andreas Kaltenbrunner. 2011.  When the Wikipedians talk: Network and tree structure of Wikipedia discussion pages. In _Fifth international AAAI conference on weblogs and social media_.
  * Leskovec and Horvitz (2008) Jure Leskovec and Eric Horvitz. 2008.  Planetary-scale views on a large instant-messaging network. In _Proceedings of the 17th international conference on World Wide Web_. 915–924.
  * Lippi and Torroni (2016) Marco Lippi and Paolo Torroni. 2016.  Argumentation mining: State of the art and emerging trends.  _ACM Transactions on Internet Technology (TOIT)_ 16, 2 (2016), 1–25.
  * Marks and Yardley (2004) David F Marks and Lucy Yardley. 2004.  _Research methods for clinical and health psychology_.  Sage.
  * McMullin (1987) Ernan McMullin. 1987\.  Scientific controversy and its termination.  _Scientific Controversies, Cambridge University Press, Cambridge_ 1987 (1987), 49–91.
  * Müller-Birn et al. (2015) Claudia Müller-Birn, Benjamin Karran, Janette Lehmann, and Markus Luczak-Rösch. 2015.  Peer-production system or collaborative ontology engineering effort: What is Wikidata?. In _Proceedings of the 11th International Symposium on Open Collaboration_. 1–10.
  * Nwadiugwu and Nwadiugwu (2021) Martin C Nwadiugwu and Cynthia C Nwadiugwu. 2021.  Influencing Followership: Understanding the Perspective of Those Leading Active Discussions on Quora.  _Frontiers in Computer Science_ 3 (2021), 582242.
  * Pearson (1931) Egon S Pearson. 1931\.  The test of significance for the correlation coefficient.  _J. Amer. Statist. Assoc._ 26, 174 (1931), 128–134.
  * Pinto et al. (2004) Helena Sofia Pinto, Steffen Staab, and Christoph Tempich. 2004\.  DILIGENT: Towards a fine-grained methodology for DIstributed, Loosely-controlled and evolvInG Engineering of oNTologies. In _ECAI_ , Vol. 16. 393\.
  * Piscopo et al. (2017) Alessandro Piscopo, Christopher Phethean, and Elena Simperl. 2017.  Wikidatians Are Born: Paths to Full Participation in a Collaborative Structured Knowledge Base. In _50th Hawaii International Conference on System Sciences_. University of Hawaii, 4354–4363.
  * Piscopo and Simperl (2018) Alessandro Piscopo and Elena Simperl. 2018.  Who Models the World? Collaborative Ontology Creation and User Roles in Wikidata.  _Proceedings of the ACM on Human-Computer Interaction_ 2, CSCW (2018), 1–18.
  * Rad and Barbosa (2012) Hoda Sepehri Rad and Denilson Barbosa. 2012.  Identifying controversial articles in Wikipedia: A comparative study. In _Proceedings of the eighth annual international symposium on wikis and open collaboration_. 1–10.
  * Ren et al. (2019) Xiaoxue Ren, Zhenchang Xing, Xin Xia, Guoqiang Li, and Jianling Sun. 2019.  Discovering, explaining and summarizing controversial discussions in community q&a sites. In _2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 151–162.
  * Schneider (2014) Jodi Schneider. 2014\.  Automated argumentation mining to the rescue? Envisioning argumentation and decision-making support for debates in open online collaboration communities. In _Proceedings of the First Workshop on Argumentation Mining_.
  * Schneider et al. (2010) Jodi Schneider, Alexandre Passant, and John G Breslin. 2010\.  A content analysis: How Wikipedia talk pages are used. In _The Web Science Conference 2010 (WebSci ’10). Raleigh, North Carolina, USA_.
  * Schneider et al. (2013) Jodi Schneider, Krystian Samp, Alexandre Passant, and Stefan Decker. 2013.  Arguments about deletion: How experience improves the acceptability of arguments in ad-hoc online task groups. In _Proceedings of the 2013 conference on Computer supported cooperative work_. 1069–1080.
  * Schneider et al. (2014) Jodi Schneider, Serena Villata, and Elena Cabrio. 2014\.  Why did they post that argument? Communicative Intentions of Web 2.0 Arguments.  _Arguing on the Web_ 2 (2014).
  * Sepehri Rad and Barbosa (2011) Hoda Sepehri Rad and Denilson Barbosa. 2011.  Towards identifying arguments in Wikipedia pages. In _Proceedings of the 20th international conference companion on World wide web_. 117–118.
  * Sepehri Rad et al. (2012) Hoda Sepehri Rad, Aibek Makazhanov, Davood Rafiei, and Denilson Barbosa. 2012.  Leveraging editor collaboration patterns in Wikipedia. In _Proceedings of the 23rd ACM conference on Hypertext and social media_. 13–22.
  * Simperl and Luczak-Rösch (2014) Elena Simperl and Markus Luczak-Rösch. 2014.  Collaborative ontology engineering: a survey.  _The Knowledge Engineering Review_ 29, 1 (2014), 101–131.
  * Stranieri et al. (2001) Andrew Stranieri, John Zeleznikow, and John Yearwood. 2001\.  Argumentation structures that integrate dialectical and non-dialectical reasoning.  _The Knowledge Engineering Review_ 16, 4 (2001), 331–348.
  * Studer et al. (1998) Rudi Studer, V Richard Benjamins, and Dieter Fensel. 1998\.  Knowledge engineering: Principles and methods.  _Data & knowledge engineering_ 25, 1-2 (1998), 161–197.
  * Suh et al. (2007) Bongwon Suh, Ed H Chi, Bryan A Pendleton, and Aniket Kittur. 2007\.  Us vs. them: Understanding social dynamics in Wikipedia with revert graph visualizations. In _2007 IEEE symposium on visual analytics science and technology_. IEEE, 163–170.
  * Sumi et al. (2011) Róbert Sumi, Taha Yasseri, et al. 2011\.  Edit wars in Wikipedia. In _2011 IEEE Third International Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third International Conference on Social Computing_. IEEE, 724–727.
  * Sáez Trumper (2021) Diego Sáez Trumper. 2021\.  Research:Identifying Controversial Content in Wikidata.  https://meta.wikimedia.org/wiki/Research:Identifying_Controversial_Content_in_Wikidata.  [Online; accessed 10-January-2023].
  * Tan et al. (2016) Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016.  Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions. In _Proceedings of the 25th international conference on world wide web_. 613–624.
  * Tempich et al. (2005) Christoph Tempich, H Sofia Pinto, York Sure, and Steffen Staab. 2005\.  An argumentation ontology for distributed, loosely-controlled and evolving engineering processes of ontologies (DILIGENT). In _The Semantic Web: Research and Applications: Second European Semantic Web Conference, ESWC 2005, Heraklion, Crete, Greece, May 29–June 1, 2005. Proceedings 2_. Springer, 241–256.
  * Tempich et al. (2007) Christoph Tempich, Elena Simperl, Markus Luczak, Rudi Studer, and H Sofia Pinto. 2007.  Argumentation-based ontology engineering.  _IEEE Intelligent Systems_ 22, 6 (2007), 52–59.
  * Vaismoradi et al. (2013) Mojtaba Vaismoradi, Hannele Turunen, and Terese Bondas. 2013\.  Content analysis and thematic analysis: Implications for conducting a qualitative descriptive study.  _Nursing & health sciences_ 15, 3 (2013), 398–405.
  * Vandenbussche et al. (2017) Pierre-Yves Vandenbussche, Ghislain A Atemezing, María Poveda-Villalón, and Bernard Vatant. 2017\.  Linked Open Vocabularies (LOV): a gateway to reusable semantic vocabularies on the Web.  _Semantic Web_ 8, 3 (2017), 437–452.
  * Viégas et al. (2004) Fernanda B Viégas, Martin Wattenberg, and Kushal Dave. 2004\.  Studying cooperation and conflict between authors with history flow visualizations. In _Proceedings of the SIGCHI conference on Human factors in computing systems_. 575–582.
  * Viegas et al. (2007) Fernanda B Viegas, Martin Wattenberg, Jesse Kriss, and Frank Van Ham. 2007.  Talk before you type: Coordination in Wikipedia. In _2007 40th Annual Hawaii International Conference on System Sciences (HICSS’07)_. IEEE, 78–78.
  * Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch. 2014.  Wikidata: A Free Collaborative Knowledge Base.  _Commun. ACM_ 57, 10 (2014), 78–85.
  * Vrandečić et al. (2005) Denny Vrandečić, Sofia Pinto, Christoph Tempich, and York Sure. 2005.  The DILIGENT knowledge processes.  _Journal of Knowledge Management_ 9, 5 (2005), 85–96.
  * Vuong et al. (2008) Ba-Quy Vuong, Ee-Peng Lim, Aixin Sun, Minh-Tam Le, Hady Wirawan Lauw, and Kuiyu Chang. 2008\.  On ranking controversies in wikipedia: models and evaluation. In _Proceedings of the 2008 international conference on Web search and data mining_. 171–182.
  * Wagner (2004) Christian Wagner. 2004\.  Wiki: A technology for conversational knowledge management and group collaboration.  _Communications of the association for information systems_ 13, 1 (2004), 19.
  * Walton et al. (2008) Douglas Walton, Christopher Reed, and Fabrizio Macagno. 2008.  _Argumentation schemes_.  Cambridge University Press.
  * Wang and Cardie (2016) Lu Wang and Claire Cardie. 2016.  A piece of my mind: A sentiment analysis approach for online dispute detection.  _arXiv preprint arXiv:1606.05704_ (2016).
  * Wang et al. (2020) Wenting Wang, Deeksha Arya, Nicole Novielli, Jinghui Cheng, and Jin LC Guo. 2020.  Argulens: Anatomy of community opinions on usability issues using argumentation models. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–14.
  * Wei et al. (2016) Zhongyu Wei, Yang Liu, and Yi Li. 2016.  Is this post persuasive? Ranking argumentative comments in online forum. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. 195–200.
  * Yasseri et al. (2014) Taha Yasseri, Anselm Spoerri, Mark Graham, and János Kertész. 2014.  The most controversial topics in Wikipedia.  _Global Wikipedia: International and cross-cultural issues in online collaboration_ 25 (2014), 25–48.
  * Zhang et al. (2017) Amy Zhang, Bryan Culbertson, and Praveen Paritosh. 2017\.  Characterizing online discussion using coarse discourse sequences. In _Proceedings of the International AAAI Conference on Web and Social Media_ , Vol. 11. 357–366.
  * Zielinski et al. (2018) Kazimierz Zielinski, Radoslaw Nielek, Adam Wierzbicki, and Adam Jatowt. 2018.  Computing controversy: Formal model and algorithms for detecting controversy on Wikipedia and in search queries.  _Information Processing & Management_ 54, 1 (2018), 14–36.