# Measuring Wikipedia Article Quality in One Dimension by Extending ORES with
Ordinal Regression

Nathan TeBlunthuis<EMAIL_ADDRESS>0000-0002-3333-5013 University of
WashingtonBox 353740SeattleWashingtonUSA98195

(2021)

###### Abstract.

Organizing complex peer production projects and advancing scientific knowledge
of open collaboration each depend on the ability to measure quality. Article
quality ratings on English language Wikipedia have been widely used by both
Wikipedia community members and academic researchers for purposes like
tracking knowledge gaps and studying how political polarization shapes
collaboration. Even so, measuring quality presents many methodological
challenges. The most widely used systems use labels on discrete ordinal scales
when assessing quality, but such labels can be inconvenient for statistics and
machine learning. Prior work handles this by assuming that different levels of
quality are “evenly spaced” from one another. This assumption runs counter to
intuitions about the relative degrees of effort needed to raise Wikipedia
encyclopedia articles to different quality levels. Furthermore, models from
prior work are fit to datasets that oversample high-quality articles. This
limits their accuracy for representative samples of articles or revisions. I
describe a technique extending the Wikimedia Foundations’ ORES article quality
model to address these limitations. My method uses weighted ordinal regression
models to construct one-dimensional continuous measures of quality. While
scores from my technique and from prior approaches are correlated, my approach
improves accuracy for research datasets and provides evidence that the “evenly
spaced” assumption is unfounded in practice on English Wikipedia. I conclude
with recommendations for using quality scores in future research and include
the full code, data, and models.

sociotechnical systems, measurement, statistics, quality, machine learning,
peer production, Wikipedia, online communities, methods, datasets

††journalyear: 2021††copyright: acmlicensed††conference: 17th International
Symposium on Open Collaboration; September 15–17, 2021; Online,
Spain††booktitle: 17th International Symposium on Open Collaboration (OpenSym
2021), September 15–17, 2021, Online, Spain††price: 15.00††doi:
10.1145/3479986.3479991††isbn: 978-1-4503-8500-8/21/09††ccs: Human-centered
computing Collaborative and social computing theory, concepts and
paradigms††ccs: Human-centered computing Social content sharing††ccs: Human-
centered computing Computer supported cooperative work

##  1\. Introduction

Measuring content quality in peer production projects like Wikipedia is
important so projects can learn about themselves and track progress. Measuring
quality also helps build confidence that information is accurate and supports
monitoring how well an encyclopedia includes diverse subject areas to identify
gaps needing attention (Redi et al., 2021). Measuring quality enables tracking
and evaluating the progress of subprojects and initiatives organized to fill
the gaps (Halfaker, 2017; Warncke-Wang et al., 2015). Raising an article to a
high standard of quality is a recognized achievement among contributors, so
assessing quality can help motivate contributions (Ayers et al., 2008; Forte
and Bruckman, 2005). In these ways, measuring quality can be of key importance
to advancing the priorities of the Wikimedia movement and is also important to
other kinds of open collaboration (Champion and Hill, 2021).

Measuring quality also presents methodological and ontological challenges. How
can “quality” be conceptualized so that measurement of the goals of a project
and the value it produces can be precise and accurate? Language editions of
Wikipedia, including English, peer produce quality labels that have been
useful both for motivating and coordinating project work and for enabling
research. Epistemic virtues of this approach stem from the community-
constructed criteria for assessment and from formalized procedures for third-
party evaluation organized by WikiProjects. These systems also have two
important limitations: (1) ratings are likely to lag behind changes in article
quality, and (2) quality is assessed on a discrete ordinal scale, which
violates typical assumptions in statistical analysis. Both limitations are
surmountable.

The machine learning framework introduced by Warncke-Wang et al. (Warncke-Wang
et al., 2013), further developed by Halfaker (Halfaker, 2017), implemented by
the Objective Revision Evaluation
Service111https://www.mediawiki.org/wiki/ORES (https://perma.cc/TH6L-KFT6)
(ORES) article quality models and adopted by several research studies of
Wikipedia article quality (e.g. Halfaker and Geiger, 2020; Kocielnik et al.,
2018; Shi et al., 2019; Warncke-Wang et al., 2015) was designed to address the
first limitation by using article assessments at the time they were made as
“ground truth.” Article quality might drift in the periods between
assessments, but it seems safe to assume that new quality assessments are
accurate at the time they are made. A model trained on recent assessments can
predict what quality label an article would receive if assessed in its current
state.

This paper introduces a method for constructing interpretable one-dimensional
measures of article quality from Wikipedia quality assessments and the ORES
article quality model. The method improves upon prior approaches in two
important ways. First, by using inverse probability weighting to calibrate the
model, it is more accurate for typical research applications, and second, it
does not depend on the assumption that quality levels are “evenly spaced,”
which threatens the validity of prior research (Halfaker, 2017; Arazy et al.,
2019). In addition, this paper helps us understand the validity of previous
work by analyzing the performance of the ORES quality model and testing the
“evenly spaced” assumption.

In §2, I provide a brief overview of quality measurement in peer production
research, in which I foreground the importance of the assumptions needed to
use machine learning predictions in downstream analysis—particularly the
“evenly spaced” assumption used by Halfaker (Halfaker, 2017) to justify the
use of a handpicked weighted sum to combine article class probabilities. Next,
in §3, I describe how to build accurate ordinal quality models that are
appropriately calibrated for analyses of representative samples of Wikipedia
articles or revisions. I also briefly explain how ordinal regression provides
an interpretable one-dimensional measure of quality and how it relaxes the
“evenly spaced” assumption. Finally, in §4 I present the results of my
analysis to (1) show how the precision of the measurement depends on proper
calibration and (2) demonstrate that the “evenly spaced” assumption is
violated. Despite this, I find that scores from the ordinal models are highly
correlated with those from prior work so the “evenly spaced” assumption may be
acceptable in some applications. I conclude in §5 with recommendations for
measuring article quality in future research.

##  2\. Background

Measurement is important to science as available knowledge often constrains
the development of improved tools for advancing knowledge. For example, in the
book Inventing Temperature, Hasok Chang (Chang, 2004), the philosopher and
historian of science, documents how extending theories of heat beyond the
range of human sense perception required scientists to develop new types of
thermometers. This in turn required better knowledge of heat and of
thermometric materials such as the freezing point of mercury. Part of the
challenge of scientific advancement is that measurement devices developed
under certain conditions may give unexpected results outside of the range in
which they are calibrated: a thermometer will give impossibly low temperature
readings when its mercury unexpectedly freezes. Today, machine learning models
are used to extend the range of quality measurements in peer production
research, but state of the art machine learning can be quite sensitive to the
nuances of how their training data are selected (Recht et al., 2019).

###  2.1. Measuring Quality in Peer Production

As described in §1, measuring quality has been of great importance to peer
production projects like Wikipedia and in the construction of knowledge about
how such projects work. The foundation of article quality measurement in
Wikipedia has been the peer production of article quality assessment organized
by WikiProjects who develop criteria for articles in their domain (Phoebe
Ayers et al., 2008). This enables quality assessment to be consistent across
different subject areas, but the procedures for assessing quality are tailored
to the values of each WikiProject. Yet, like human sense perception of
temperature, these quality assessments are limited in that they require human
time and attention. In addition, humans’ limited ability to discriminate
between levels on a scale limits the sensitivity of quality assessments.
Articles are assessed irregularly and infrequently at the discretion of
volunteer editors. Therefore, for most article revisions, it is not known what
quality class the article would be assigned if it were newly assessed.

Researchers have proposed many ideas to extend the range of quality
measurement beyond the direct perception of Wikipedians, such as page length
(Blumenstock, 2008), persistent word revisions (Adler and de Alfaro, 2007;
Biancani, 2014), collaboration network structures (Raman et al., 2020), and
template-based flaw detection (Anderka et al., 2012). Carefully constructed
indexes benchmarked against English language Wikipedia quality assessments
might allow quality measurement of articles that have not been assessed or in
projects that have underproduced article assessments (Lewoniewski et al.,
2017). However, such indexes may lack emic validity if they fail to capture
important aspects of quality or if notions of quality vary between linguistic
communities and might even shape the editing activity in unexpected ways that
could ultimately defeat their purpose (Goodhart, 1984; Strathern, 1997). Peer-
produced quality labels depend on the limited capacity of volunteer
communities to coordinate quality assessment, but also provide impressive
validity for evaluating projects on their own terms.

###  2.2. Article Quality Models Extend Measurement to Unassessed Articles

Perhaps the most successful approaches to extending the range of quality
measurements use machine learning models trained on available article quality
assessments to predict the quality of revisions that have not been assessed.
The ORES article quality model (henceforth ORES) implements this approach, but
other similar article quality predictors have been developed (Anderka and
Stein, 2012; Dang and Ignat, 2016; Zhang et al., 2018; Druck et al., 2008;
Sarkar et al., 2019; Raman et al., 2020), and additional features including
those based on language models can substantially improve classification
performance compared to ORES (Schmidt and Zangerle, 2019). The ORES model is a
tree-based classifier that predicts the quality class of a Wikipedia article
at the time it is assessed.222The system uses cross-validation to select among
candidates that include random-forest and boosted decision tree models. These
tree-based models are reasonable for practical purposes with the reported
ability to predict within one level of the true quality class with 90%
accuracy (although in §4.2 I find a decline in accuracy in a more recent
dataset). Yet, since these models do not account for the ordering of quality
labels, the use of these predictions in downstream analysis introduces
complicated methodological challenges.

The ORES classifiers are fit using scikit-learn333https://scikit-
learn.org/stable/(https://perma.cc/5Y8B-W8T5) through minimization of the
multinomial deviance as shown (Pedregosa et al., 2011; Hastie et al., 2018):

(1)
$L(y_{i},p(x_{i}))=-\sum_{k=1}^{K}{I(y_{i}=\mathcal{G}_{i,k})\mathrm{log}\leavevmode\nobreak\
p_{k}(x_{i})}$

For each article $i$ with predictors $x_{i}$ that has been labeled with a
quality class $y_{i}$, the ORES model outputs an estimated probability
$p_{k}(x_{i})$ that the article belongs to each quality class
$k\in\\{\mathrm{\textit{stub}},\mathrm{\textit{start}},\mathrm{\textit{C-class}},\mathrm{\textit{B-class}},\mathrm{\textit{Good
article (GA)}},\mathrm{\textit{Featured article (FA)}}\\}$. The predicted
probabilities $p(x_{i})$ sum to one so the ORES model outputs a unit vector
for each article. If $\mathcal{G}_{i,k}$, the most probable quality class
(MPQC) according to the model, is the true label, then
$I(y_{i}=\mathcal{G}_{i,k})$ equals $1$ ($I$ is the indicator function) and
the log predicted probability $p_{k}(x_{i})$ of the correct class is
subtracted from the loss $L(y_{i},p(x_{i}))$. Note that this model does not
use the fact that article quality classes are ordered. If it did, then it
would have to penalize an incorrect classification of a Good article as
C-class more than a classification of a Good article as B-class. In this
model, different quality classes have no intrinsic rank or ordering and thus
are akin to different categories of article subjects like animals, vegetables,
or minerals.

The MPQC is perhaps the most natural way to use the ORES output to measure
quality. It has been used in several studies including to provide evidence
that politically polarized collaboration on Wikipedia leads to high quality
articles (Shi et al., 2019) and to understand the relationship between article
quality and donation (Kocielnik et al., 2018). However, the MPQC is limited in
that it does not measure quality differences between articles that have the
same MPQC. Consider two hypothetical articles; the first has the multinomial
prediction $(0.1,0.3,0.4,0.075,0.075,0)$ and the second has the prediction
$(0.075,0.075,0.4,0.3,0.1,0)$. The MPQC will assign both the C-class label
even though the first article has an even chance at being a Stub or Start-
class while the second article has an even chance at being a B-class or even a
Good article. At best, the MPQC has limited sensitivity to subtle variations
or gradual changes in quality (Halfaker, 2017).

###  2.3. Combining Scores for Granular Measurement

To further extend the range of article quality measurement within article
quality classes, Halfaker (Halfaker, 2017) constructed a numerical quality
score using a linear combination (a weighted sum) of the elements of the
multinomial prediction $p(x_{i})$. This is advantageous from a statistical
perspective as it naturally provides a continuous measure of quality which can
typically justify a normal or log-normal statistical model. It can also
support higher-order aggregations for measuring the quality of a set of
articles (Halfaker, 2017). Halfaker handpicks the coefficients $[0,1,2,3,4,5]$
to make a linear combination of the predictions under the assumption “that the
ordinal quality scale developed by Wikipedia editors is roughly cardinal and
evenly spaced,” which I refer to the “evenly spaced” assumption. It
essentially says that a Start-class article has one more unit quality of a
Stub-class article, and that a C-class article has one more unit of quality
than a Start-class article and so on. This approach is being adopted by other
researchers including Arazy et al. (Arazy et al., 2019).

The considerable degree of effort and expertise required to raise articles to
higher levels of quality raises doubt in the assumption (Jemielniak, 2014).
Higher quality levels correspond to increasing completeness, encyclopedic
character, usefulness to wider audiences, incorporation of multimedia,
polished citations, and adherence to Wikipedia’s policies. The English
language Wikipedia editing guideline on content
assessment444https://en.wikipedia.org/w/index.php?title=Wikipedia:Content_assessment&oldid=1023695750
(https://perma.cc/2JUV-6SD) defines a Good article as “useful to nearly all
readers, with no obvious problems” and a Featured article article as
“professional, outstanding and thorough.” According to Wikipedians, it can
take “three to six months of full time work” to write a _Featured
article_.555Public statement by Stuart Yeates, an expert Wikipedian; quoted
with permission. https://lists.wikimedia.org/hyperkitty/list/wiki-
<EMAIL_ADDRESS>(https://perma.cc/9V4P-WRXR) Are we to assume that the difference in quality
between a Good article and a Featured article is measurably the same as that
between a Stub defined as as “little more than a dictionary definition” and a
Start-class that is “a very basic description of the topic?” How could we even
answer this question?

If the “evenly spaced” assumption is reasonable, then Halfaker’s (Halfaker,
2017) weighted sum approach is too. But if increasing Wikipedia article
classes do not represent roughly equal improvements in quality, this may
threaten the accuracy of analysis dependent on the assumption. Suppose that a
B-class has not 1, but 2 units of quality greater than a C-class article, then
Halfaker could have underestimated the improvement in the knowledge gap of
women scientists, which was considerably driven by improvement in B-class
articles. In the next section, I provide a straightforward extension of the
ORES article quality model based on ordinal regression can both relax the
“evenly spaced” assumption and provide a better calibrated and more accurate
one-dimensional measure of quality.

##  3\. Data, Methods and Measures

I use Bayesian ordinal regression models that use the ORES predicted
probabilities to predict the quality class labels and quantify the distance
between quality classes. I now provide a brief overview of ordinal regression
as needed to explain my approach to measuring quality. Understanding ordinal
regression depends on background knowledge of odds and generalized linear
models. I recommend McElreath and Safari (McElreath and Safari, 2018) for
reference.

###  3.1. Bayesian Ordinal Regression

Ordinal regression predicts quality class membership using a single linear
model for all classes and identifies boundaries between classes using the log
cumulative odds link function shown below in Eq. 2. The log cumulative odds is
not the only possible choice of link function, but it is the most common, is
the easiest to interpret, and is appropriate here.

(2) $\displaystyle\mathrm{log}$ $\displaystyle\leavevmode\nobreak\
\frac{\mathrm{Pr}(y_{i}\leq k)}{1-\mathrm{Pr}(y_{i}\leq
k)}=\alpha_{k}-\phi_{i}$ $\displaystyle\phi_{i}$ $\displaystyle=Bx_{i}$

As in Eq. 1, $y_{i}$ is the quality label for article $i$. The left hand side
of Eq. 2 gives the log odds that $y_{i}$ is less than or equal to quality
level $k$. The ordinal quality measure is given by a linear model
$\phi_{i}=Bx_{i}$ ($x_{i}$ is a vector of transformed ORES scores for article
$i$). Key to interpreting $\phi_{i}$ as a quality measure are the intercept
parameters $a_{k}$ for each quality level $k$. The log cumulative odds (the
log odds that the article $y_{i}$ has quality less than or equal to $k$) are
given by the difference between the intercept and the linear model $a_{k}$ \-
$\phi_{i}$. Therefore, if $\phi_{i}=\alpha_{k}$ then the chances that $i<=k$
equal the chances that $i>k$. When $\phi_{i}$ is less than $\alpha_{k}$, the
quality of article $i$ is probably less than or equal to quality level $k$. As
$\phi_{i}-\alpha_{k}$ increases so do the chances that article $i$ is of
quality better than $k$. In this way, the threshold parameters $a_{k}$ define
quantitative article quality levels on the scale of the ordinal quality
measure $\phi_{i}$.

Informally, an ordinal regression model maps a linear regression model to the
ordinal scale using the log cumulative odds link function. It does this by
inferring thresholds that partition the range of linear predictions. When the
linear predictor for an article crosses a threshold, the probability that the
article has quality greater than that corresponding to the threshold begins to
increase.

Bayesian inference allows interpreting model parameters like $\phi_{i}$ and
$\alpha_{k}$ as random variables and provides accurate quantification of
uncertainty in thresholds and predictions. I fit models using the R package
Bayesian regression modeling using Stan (brms) (Bürkner, 2017) version 2.15.0.
I use the default priors for ordinal regression, which are weakly informative.
Due to the large sample size, the data overwhelm the priors and the priors
have little influence over results. I confirmed this by fitting equivalent
frequentist models using the polr function in the MASS R package (Venables et
al., 2002) and found that the estimates of intercepts and coefficients were
very close.

The six quality scores output by the ORES article quality classifier are
perfectly collinear by construction because they sum to one. This means they
cannot all be included in the same regression model. Since interpreting the
coefficients is not important, I take the linear transformation of the ORES
scores using appropriately weighted principle component analysis and use the
first five principle components as the independent variables. This is simpler
and more statistically efficient than a model selection procedure.

###  3.2. Dataset and Model Calibration

I draw a new random sample of 5,000 articles from each quality class to
develop my models. I first reuse code from the
articlequality666https://pypi.org/project/articlequality
(https://perma.cc/8R4H-MAZ9) Python package to process the March 2020 XML
dumps for English Wikipedia and extract up-to-date article quality labels. I
then select pages that have been assessed by a member of at least one
WikiProject. Following prior work, if an article is assessed at different
levels according to more than one WikiProject, I assign it to the highest such
level and I drop articles having the rarely used _A-class_ quality level
(Halfaker, 2017; Warncke-Wang et al., 2015, 2013). Next, I use the
revscoring777https://pypi.org/project/revscoring (https://perma.cc/3HFN-V23Z)
Python package to obtain the ORES scores of the labeled article versions. Some
of these versions have been deleted leading to missing observations at each
quality level. Table 1 shows the number of articles sampled in each quality
class. I reserve a random sample of 2000 articles which I use in reporting my
results and fit my ordinal regression models on the remainder.

The ORES article quality classifiers are fit on a “balanced” dataset having an
equal number of articles in each quality class. Thus, an ORES score is the
probability that an article is a member of a quality class under the
assumption that the article was drawn from a population where each quality
class contains an equal number of articles. Simply put, the model has learned
from its training data that each quality class is about the same size.

Figure 1. Calibration of each predictive quality model on datasets
representative of each unit of analysis (article, revision, quality class).
Each chart shows, for each quality class, the miscalibration of a model
(columns) with respect to a dataset weighted to represent a unit of analysis
(rows). The y-axis shows difference between the true probability of the
quality class and the average predicted probability of that class, given a
chosen unit of analysis. Points close to zero indicate good calibration. For
example, the top-left chart shows that the article model is well-calibrated to
the dataset on which it was fit and the middle-left chart shows that the
article model predicts that articles are Stubs with probability greater than
the frequency of Stubs in a random sample of revisions. Error bars show 95%
confidence intervals.

This is not representative of the overall article quality on Wikipedia, which
is highly skewed with over 3 million Stubs but only around 7,000 Featured
articles as shown in Table 1. Although using a balanced dataset likely
improves the accuracy of the ORES models, for the ordinal regression models,
the choice of unit of analysis presents a trade-off between accuracy in a
representative sample of articles or revisions and accuracy within each
quality class. Constructing a balanced dataset by oversampling is a common
practice in machine learning because it can improve predictive performance.
However, oversampling can also lead to badly calibrated predictive
probabilities as shown in Fig. 1. Calibration means that, on average, the
predicted probability of a quality class equals the average true probability
of that class for the unit of analysis.

The “balanced” dataset on which ORES is trained has the quality class unit of
analysis because each quality class has equal representation. However,
researchers are more interested in analyzing representative samples of
articles or revisions. For example, the article unit of analysis would be used
to estimate the average quality of a random sample of articles and the
revision unit of analysis might be used to model the change in the quality of
an encyclopedia over time. Weighting allows the use of the balanced dataset to
estimate a model as if the dataset were a uniform random sample of a different
unit of analysis. My method uses a balanced dataset to fit ordinal regression
models with inverse probability weighting to calibrate each model to the unit
of analysis of a research project. For example, each article in the model
calibrated to the article unit of analysis is weighted by the probability of
its quality class in the population of articles divided by the probability of
its quality class in the sample. The size of the sample and the weights for
the article and revision levels of analysis are also shown in Table 1.

Table 1. Number of articles sampled at each quality level Label | No. of articles | No. of revisions | Sample size | Article weights | Revision weights
---|---|---|---|---|---
Stub | 3,359,351 | 12,005,611 | 4,969 | 4.23 | 2.52
Start | 1,019,038 | 7,828,335 | 4,979 | 1.28 | 1.64
C | 235,655 | 3,889,639 | 4,988 | 0.30 | 0.81
B | 128,875 | 3,640,591 | 4,990 | 0.16 | 0.76
GA | 31,808 | 924,468 | 4,999 | 0.04 | 0.19
FA | 7,438 | 365,255 | 4,995 | 0.01 | 0.08

##  4\. Results

I first report my findings about the spacing of the quality classes in each of
the models in §4.1. Quality classes are not evenly spaced, especially when
articles or revisions are the unit of analysis. Next, in §4.2, I report the
accuracy of each of the models and the uncertainty of the ordinal quality
scale. All models perform similarly to or better than the MPQC within the
pertinent unit of analysis. The unweighted model provides the best accuracy
and lowest uncertainty across the entire range of quality levels, but is
poorly calibrated for other units of analysis. Finally, in §4.3, I show that
all quality measures are highly correlated, but the ordinal quality measures
agree with one another more than with the “evenly spaced” measure.

Figure 2. Quality scores and predictions of the ordinal regression models.
Columns in the grid of charts correspond to the ordinal quality model
calibrated to the indicated unit of analysis and rows correspond to sampled
articles having the indicated level of quality as assessed by Wikipedians.
Each chart shows the histogram of scores, thresholds inferred by the ordinal
model with 95% credible intervals colored in gray, and colors indicating when
the model makes correct or incorrect predictions. The thresholds are not
evenly spaced, especially in revision model and article model that has more
weight on lower quality classes. These two models infer that the gaps between
Stub and Start and between Start and C-class articles are considerably wider
than the gap between C-class and B-class articles.

###  4.1. Spacing of Quality Classes

The grid of charts in Fig. 2 shows quality scores and thresholds for each
model (columns) and article quality level (rows). Each chart shows the
histogram of quality scores $\phi_{i}$ given to articles having the true
quality label corresponding to the row of the grid. The histograms are colored
to indicate regions where the model correctly predicts that articles belong to
their true class. Vertical dashed lines show the thresholds inferred by the
model with 95% credible intervals colored in gray. Different models have
different ranges of scores, so Fig. 2 shows results normalized between 0 and
1.

No matter the unit of analysis, article quality classes are not evenly spaced.
The quality class model provides a quality scale in which Featured articles
take up $27\%$ of the scale and are expected to score in the range of
$[0.73,1]$, but probable C-class articles only span $14\%$ of the scale in the
range $[0.31,0.45]$. Researchers are likely to be interested in models
calibrated to the article or revision units of analysis, and in these cases,
the quality classes are far from evenly spaced. The revision model assigns
$28\%$ of the scale to Stubs, from $0$ to $0.28$. It assigns C-class articles
the smallest part of the scale, only $4\%$ of it, from $0.54$ to $0.58$. The
article model is even more extreme. It assigns Stubs to the interval
$[0,0.39]$, $39\%$ of the scale, and the space between thresholds defining the
range of C-class articles is so narrow that it virtually never predicts that
an article will be C-class. In general terms, the quality class model gives
relatively equal amounts of space to each quality class compared to the other
models, while reserving nearly the top half of the scale for the top 2 quality
classes. The revision model and article model do the opposite and use the
bottom half of the scale to account for differences within the bottom two
quality classes, leave some room for B-class articles, but squeeze the top end
of the scale and C-class articles into relatively small intervals.

###  4.2. Accuracy and Uncertainty

I evaluate predictive performance in terms of accuracy, the proportion of
predictions of article quality that are correct. To allow comparison with the
reported accuracy of the ORES quality models, I also report off-by-one
accuracy, which includes predictions within one level of the true quality
class among correct predictions.

Table 2. Accuracy of quality prediction models depends on the unit of analysis. The greatest accuracy and off-by-one accuracy scores are highlighted. Models are more accurate when calibrated on the same unit of analysis on which they are evaluated. Compared to the MPQC, the ordinal quality models have better accuracy when revisions or articles are the unit of analysis. When the quality class is the unit of analysis, the ordinal quality model has worse accuracy, but predicts within one quality class with slightly better accuracy.  Unit of analysis | Model | Ordinal model? | Accuracy | Off-by-one accuracy
---|---|---|---|---
Quality class | Article | Yes | 0.33 | 0.75
Quality class | Revision | Yes | 0.44 | 0.84
Quality class | Quality class | Yes | 0.52 | 0.87
Quality class | ORES MPQC | No | 0.55 | 0.86
Revision | Article | Yes | 0.57 | 0.87
Revision | Revision | Yes | 0.61 | 0.92
Revision | Quality class | Yes | 0.54 | 0.88
Revision | ORES MPQC | No | 0.58 | 0.9
Article | Article | Yes | 0.76 | 0.97
Article | Revision | Yes | 0.73 | 0.96
Article | Quality class | Yes | 0.63 | 0.92
Article | ORES MPQC | No | 0.65 | 0.94

As shown in Table 2, the ordinal regression models have better predictive
ability than the MPQC except when the unit of analysis is the quality class.
In this case, the best ordinal quality model has worse accuracy than the MPQC
but slightly better off-by-one accuracy. Table 2 shows accuracy and off-by-one
accuracy weighted for each unit of analysis. Accuracy for a given unit of
analysis depends on having a model fit to data representative of that unit of
analysis. Accuracy scores are higher when greater weight is placed on lower
article quality classes, suggesting that it is easier to discriminate between
these classes.

The ORES article quality model has been quickly adopted by researchers, but
its accuracy is limited. While off-by-one accuracy is above 90% when the
article is the unit of analysis, the MPQC only predicts the correct quality
class 55% of the time when the quality class is the unit of analysis.

The trade-offs in selecting a unit of analysis on which to calibrate the
models are further illustrated by Fig. 3, which plots the size of the 95%
credible intervals as a function of the quality scores for each model. As in
Fig. 2, quality scores in this plot are rescaled between 0 and 1. The models
calibrated to articles or revisions have more certainty in the lower range of
the quality scale compared to the model that places equal weight in all
quality classes. This comes with a trade-off for the higher range of quality.
While the quality class model has relatively low uncertainty across the entire
range of quality, the revision model and article model have greater
uncertainty at higher levels of quality.

Figure 3. Uncertainty in ordinal quality scores for models calibrated at each
unit of analysis. Points show the size of the 95% credible interval for the
ordinal quality score for each article in the dataset. The quality class model
has low uncertainty across the range of quality. Models calibrated to the
revision and article levels of analysis have less uncertainty at the low end
of the quality scale, but greater uncertainty at the higher end of the scale.

###  4.3. Correlation Between Scores

Although the models have different predictive performances and uncertainties,
as measures of quality, they are nearly perfectly correlated with one another
as shown in Fig. 4. For each quality score, including the “evenly spaced”
weighted sum, Fig. 4 shows a scatter plot and two correlation statistics:
Kendall’s $\tau$ and Pearson’s $r$. Pearson’s $r$ is the standard linear
correlation coefficient and Kendall’s $\tau$ is a nonparametric rank-based
correlation defined as the probability that the quality scores will agree
about which of any two articles has higher quality minus the probability that
they will disagree.

According to Pearson’s $r$ all the quality scores are highly correlated with
correlation coefficients of about $0.98$ or higher. Kendall’s $\tau$ measures
nonlinear correlation and reveals discrepancies between the ordinal models and
the “evenly spaced” measures. The Pearson correlation between scores from the
revision model and the scores from the quality class model are about the same
as the correlation between the revision model scores and the “evenly spaced”
scores ($r=0.98$). However, according to Kendall’s $\tau$, scores from the
revision model are more similar to those from the quality class model
($r=0.98$) than to the scores from the “evenly spaced” approach ($r=0.9$).

The evenly spaced model is more likely to disagree with the model-based scores
than any of the model-based scores are to disagree with one another as
visualized in the scatter plots in Fig. 4. Disagreement between the “evenly
spaced” method and the ordinal models is greatest among articles in the middle
of the quality range.

Figure 4. Correlations between quality measures show that the different
approaches to measuring quality are quite similar. “Evenly spaced” uses a
weighted sum of the ORES scores with handpicked coefficients (Halfaker, 2017).
Lower values of Kendall’s $\tau$, a nonparametric rank correlation statistic,
compared to Pearson’s $r$ suggest nonlinear differences between the weighted
sum and the other measures.

##  5\. Discussion

Past efforts to extend the measurement of Wikipedia article quality from peer-
produced article quality assessments to unassessed versions of articles and
from the discrete to the continuous domain have relied upon machine learning
and expedient but untested assumptions like that quality levels are “evenly
spaced.” While I suggest technical improvements for statistical models for
measuring quality, I also find that scores from my models are highly
correlated to those obtained under the “evenly spaced” assumption.

I set out to provide a better way to convert the probability vector output by
the ORES article quality model into a continuous scale and to test the
assumption that the quality levels are evenly spaced. I used ordinal
regression models to infer spacing between quality levels and used the linear
predictor of these models as a continuous measure of quality. While I found in
§4.1 that the quality levels are not evenly spaced and that the spacing
depends on the unit of analysis to which the models are calibrated, I also
showed in §4.3 that the model-based quality measures are highly, although not
perfectly, correlated with the “evenly spaced” measure. This provides some
assurance that past results built on this measure are unlikely to mislead.
That said, I recommend that future work adopt appropriately calibrated model-
based quality measures instead of the “evenly spaced” approach, and I argue
that it is important to improve the accuracy of article quality predictors to
enable more precise article quality measurement.

###  5.1. Recommendations for Measuring Article Quality

How should future researchers approach the question of how to measure
Wikipedia article quality? While I cannot provide a final or complete answer
to the question, I believe the exercise reported in this paper provides some
insights on which to base recommendations. It is important to note that I
consider here only approaches to measuring quality that assume the use of a
good predictor of article quality assessment, such as the ORES quality model.
I do not consider other based approaches such as those based on indexes
(Lewoniewski et al., 2017) described in §2.

####  5.1.1. Use the principle components of ORES scores for statistical
control of article quality

In many statistical analyses, the only purpose of measuring quality will be as
a statistical control or adjustment. For example, Zhang et al. (Zhang et al.,
2017) used the MPQC as a control variable in a propensity score matching
analysis of promotion to Featured article status, but as argued in §3, the
MPQC provides less information than the vector of ORES scores. Using the
principle components is simpler than using an ordinal quality model. I
recommend obtaining ORES scores for your dataset, taking the principle
components, and dropping the least significant one to remove collinearity.

####  5.1.2. Use ordinal quality scores when article quality is an independent
variable

In other cases, research questions will ask how article quality is related to
an outcome of interest, like how Kocielnik et al. (Kocielnik et al., 2018) set
out to explore factors associated with donations to the Wikimedia Foundation.
They use the MPQC as an independent variable, which complicates their
analysis. Although they conclude that “pages with higher quality attract more
donations,” this is not strictly true. They actually found a nonlinear
relationship where readers of B-class articles were more likely to donate than
readers of Featured articles. Using a continuous measure of quality is more
convenient when the average linear relationship is the target of inference.

I recommend using an ordinal regression model appropriate to the downstream
unit of analysis because this will justify the interpretation of the measure.
If the downstream unit of analysis differs substantively from those used here,
such as if different selection criteria are applied, I recommend reusing my
code to calibrate a new ordinal regression model to a new dataset. Otherwise,
reusing one of my models should be adequate. Finally, in the Bayesian
framework, the scores are interpretable as random variables. This provides a
justification for incorporating the variance of these scores as measurement
errors to improve estimation in downstream analysis (McElreath and Safari,
2018).

####  5.1.3. Use the MPQC or ordinal quality scores when article quality is
the dependent variable

Using the MPQC as the outcome in an ordinal regression model, as is done by
Shi et al. (Shi et al., 2019) in their analysis of Wikipedia articles with
politically polarized editors, is a reasonable choice as long as it provides
sufficient variation and a more granular quality measure is not needed.
Although it is theoretically possible that using the MPQC might introduce
statistical bias because it less accurate than ordinal quality scores for
units of analysis other than the quality class and omits variation within
quality classes, such threats to validity do not seem more significant than
the threat introduced by inaccurate predictions. If the MPQC does not provide
sufficient granularity and a continuous measure is desired as in Halfaker
(Halfaker, 2017) or Arazy et al. (Arazy et al., 2019), I recommend using a
measure based on ordinal regression as described in §5.1.2.

###  5.2. Limitations

Although intuitions about the varying degrees of effort required to develop
articles with different levels of quality led me to question the “evenly
spaced” assumption, my findings that quality classes are not evenly spaced do
not necessarily reflect relative degrees of effort. Rather, spaces between
levels are chosen to link a linear model to ordinal data. The spacing of
intervals depends on the ability of the ORES scores to predict quality
classes. The ORES article quality model has relative difficulty classifying
C-class and B-class articles (Halfaker, 2017). Perhaps, the differences
between these quality classes are minor compared to the other classes. Maybe
ORES lacks the features or ability to model these differences and the space
between these classes will grow if its predictive performance improves.

The usefulness of article quality scores depends on the accuracy of the model.
The ORES quality models are accurate enough to be useful for researchers, but
they still only predict the correct quality class 55% of the time on a
balanced dataset. Of course, this limits the accuracy of the ordinal
regression models reported here. Furthermore, while the ORES quality models
were designed with carefully chosen features intended to limit biases
(Halfaker and Geiger, 2020), it is still quite plausible that the accuracy of
predictive quality models may vary depending on characteristics of the article
(Kleinberg et al., 2016). Such inaccuracies may introduce bias, threaten
downstream analysis or lead to unanticipated consequences of collaboration
tools built upon the models (TeBlunthuis et al., 2021). Therefore, improving
the accuracy of article quality prediction models is important to the validity
of future article quality research. Adopting machine learning models that can
incorporate ordinal loss functions is a promising direction and can reduce the
need for auxiliary ordinal regression models (Cardoso et al., 2007).

This paper only considers measuring article quality for English language
Wikipedia, but expanding knowledge of collaborative encyclopedia production
depends on studying other languages as audiences and collaborative dynamics
can greatly vary between projects (Hecht and Gergle, 2010; Lemmerich et al.,
2019; TeBlunthuis et al., 2019). Other languages carry out quality assessments
(Lewoniewski et al., 2017), and some of these have been used to build ORES
article quality models. Future work should extend this project to provide
multilingual article quality measures in one continuous dimension.

An additional limitation stems from the likelihood that peer-produced quality
labels are biased. For instance, the English Wikipedia community has a well-
documented pattern of discrimination against content associated with
marginalized groups such as biographies of women (Tripodi, 2021; Menking et
al., 2019) and indigenous knowledge (van der Velden, 2013). Although
demonstrating biases in article quality assessment is a task for future
research, if Wikipedians’ assessments of article quality are biased then model
predictions of quality will almost certainly be as well.

##  6\. Conclusion

Measuring article quality in one continuous dimension is a valuable tool for
studying the peer production of information goods because it provides
granularity and is amenable to statistical analysis. Prior approaches extended
ORES article quality prediction into a continuous measure under the “evenly
spaced” assumption. I showed how to use ordinal regression models to transform
the ORES predictions into a continuous measure of quality that is
interpretable as a probability distribution over article quality levels,
provides an account of its own uncertainty and does not assume that quality
levels are “evenly spaced.” Calibrating the models to the chosen unit of
analysis improves accuracy for research applications. I recommend that future
work adopt this approach when article quality is an independent variable in a
statistical analysis.

##  7\. Code and Data Availability

Code, data and instructions for replicating or reusing this analysis are
available in the Harvard Dataverse at https://doi.org/10.7910/DVN/U5V0G1.

###### Acknowledgements.

I am grateful to the members of the Community Data Science Collective for
their feedback on early drafts of this work including Kaylea Champion, Sneha
Narayan, Jeremy Foote, and Benjamin Mako Hill. I would also like to thank
Aaron Halfaker for encouraging me to write this after seeing a preliminary
version. Thanks to Stuart Yeates and other participants in the wiki-research-l
mailing list<EMAIL_ADDRESS>for answering my questions
about measuring article quality and effort. Finally, thank you to the
anonymous OpenSym reviewers whose careful and constructive feedback improved
the paper.

## References

  * (1)
  * Adler and de Alfaro (2007) B. Thomas Adler and Luca de Alfaro. 2007.  A Content-Driven Reputation System for the Wikipedia. In _Proceedings of the 16th International Conference on World Wide Web_ _(WWW ’07)_. ACM, New York, NY, USA, 261–270.
  * Anderka and Stein (2012) Maik Anderka and Benno Stein. 2012.  A Breakdown of Quality Flaws in Wikipedia. In _Proceedings of the 2Nd Joint WICOW/AIRWeb Workshop on Web Quality_ _(WebQuality ’12)_. ACM, New York, NY, 11–18.
  * Anderka et al. (2012) Maik Anderka, Benno Stein, and Nedim Lipka. 2012\.  Predicting Quality Flaws in User-Generated Content: The Case of Wikipedia. In _Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’12)_. ACM, New York, NY, USA, 981–990.
  * Arazy et al. (2019) Ofer Arazy, Aron Lindberg, Mostafa Rezaei, and Michele Samorani. 2019.  The Evolutionary Trajectories of Peer-Produced Artifacts: Group Composition, the Trajectories’ Exploration, and the Quality of Artifacts.  _MIS Quarterly_ (Dec. 2019).
  * Ayers et al. (2008) Phoebe Ayers, Charles Matthews, and Ben Yates. 2008\.  _How Wikipedia Works and How You Can Be a Part of It_.  No Starch Press, San Francisco, CA.
  * Biancani (2014) Susan Biancani. 2014\.  Measuring the Quality of Edits to Wikipedia. In _Proceedings of The International Symposium on Open Collaboration_ _(OpenSym ’14)_. ACM, New York, NY, USA, 33:1–33:3.
  * Blumenstock (2008) Joshua E. Blumenstock. 2008\.  Size Matters: Word Count as a Measure of Quality on Wikipedia. In _Proceeding of the 17th International Conference on World Wide Web - WWW ’08_. ACM Press, Beijing, China, 1095.
  * Bürkner (2017) Paul-Christian Bürkner. 2017\.  Brms: An R Package for Bayesian Multilevel Models Using Stan.  _Journal of Statistical Software_ 80, 1 (Aug. 2017), 1–28.
  * Cardoso et al. (2007) Jaime S Cardoso, Jaime Cardoso, and Inescporto Pt. 2007\.  Learning to Classify Ordinal Data: The Data Replication Method.  _Journal of Machine Learning Research_ 8 (2007), 37.
  * Champion and Hill (2021) Kaylea Champion and Benjamin Mako Hill. 2021.  Underproduction: An Approach for Measuring Risk in Open Source Software.  _IEEE International Conference on Software Analysis, Evolution and Reengineering_ (Feb. 2021).  arXiv:2103.00352 [cs.SE]
  * Chang (2004) Hasok Chang. 2004\.  _Inventing Temperature._ OUP, Oxford.
  * Dang and Ignat (2016) Quang Vinh Dang and Claudia-Lavinia Ignat. 2016.  Quality Assessment of Wikipedia Articles Without Feature Engineering. In _Proceedings of the 16th ACM/IEEE-CS on Joint Conference on Digital Libraries_ _(JCDL ’16)_. ACM, New York, NY, USA, 27–30.
  * Druck et al. (2008) Gregory Druck, Gerome Miklau, and Andrew McCallum. 2008\.  Learning to Predict the Quality of Contributions to Wikipedia. In _WikiAI_. 6.
  * Forte and Bruckman (2005) Andrea Forte and Amy Bruckman. 2005.  Why Do People Write for Wikipedia? Incentives to Contribute to Open-Content Publishing. In _Proceedings of GROUP_. 6.
  * Goodhart (1984) C. A. E. Goodhart. 1984\.  Problems of Monetary Management: The UK Experience.  In _Monetary Theory and Practice: The UK Experience_ , C. A. E. Goodhart (Ed.). Macmillan Education UK, London, 91–121.
  * Halfaker (2017) Aaron Halfaker. 2017\.  Interpolating Quality Dynamics in Wikipedia and Demonstrating the Keilana Effect. In _Proceedings of the 13th International Symposium on Open Collaboration_ _(OpenSym ’17)_. Association for Computing Machinery, New York, NY, USA, 1–9.
  * Halfaker and Geiger (2020) Aaron Halfaker and R Stuart Geiger. 2020.  ORES: Lowering Barriers with Participatory Machine Learning in Wikipedia.  4, 148 (Oct. 2020), 37.
  * Hastie et al. (2018) Trevor Hastie, Jerome Friedman, and Robert Tisbshirani. 2018\.  _The Elements of Statistical Learning: Data Mining, Inference, and Prediction_.  Springer, New York.
  * Hecht and Gergle (2010) Brent Hecht and Darren Gergle. 2010.  The Tower of Babel Meets Web 2.0: User-Generated Content and Its Applications in a Multilingual Context. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ _(CHI ’10)_. Association for Computing Machinery, Atlanta, Georgia, USA, 291–300.
  * Jemielniak (2014) Dariusz Jemielniak. 2014\.  _Common Knowledge?: An Ethnography of Wikipedia_.  Stanford University Press.
  * Kleinberg et al. (2016) Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2016\.  Inherent Trade-Offs in the Fair Determination of Risk Scores.  _arXiv:1609.05807 [cs, stat]_ (Sept. 2016).  arXiv:1609.05807 [cs, stat]
  * Kocielnik et al. (2018) Rafal Kocielnik, Os Keyes, Jonathan T. Morgan, Dario Taraborelli, David W. McDonald, and Gary Hsieh. 2018.  Reciprocity and Donation: How Article Topic, Quality and Dwell Time Predict Banner Donation on Wikipedia.  _Proceedings of the ACM on Human-Computer Interaction_ 2, CSCW (Nov. 2018), 1–20.
  * Lemmerich et al. (2019) Florian Lemmerich, Diego Sáez-Trumper, Robert West, and Leila Zia. 2019.  Why the World Reads Wikipedia: Beyond English Speakers. In _Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining_ _(WSDM ’19)_. ACM, New York, NY, USA, 618–626.
  * Lewoniewski et al. (2017) Włodzimierz Lewoniewski, Krzysztof Wecel, and Witold Abramowicz. 2017.  Relative Quality and Popularity Evaluation of Multilingual Wikipedia Articles.  _Informatics_ 4, 4 (Dec. 2017), 43\.
  * McElreath and Safari (2018) Richard McElreath and an O’Reilly Media Company Safari. 2018\.  _Statistical Rethinking_.
  * Menking et al. (2019) Amanda Menking, Ingrid Erickson, and Wanda Pratt. 2019\.  People Who Can Take It: How Women Wikipedians Negotiate and Navigate Safety.  In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. Association for Computing Machinery, New York, NY, USA, 1–14.
  * Pedregosa et al. (2011) Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011.  Scikit-Learn: Machine Learning in Python.  _Journal of Machine Learning Research_ 12, 85 (2011), 2825–2830.
  * Phoebe Ayers et al. (2008) Phoebe Ayers, Charles Matthews, and Ben Yates. 2008\.  _How Wikipedia Works_.  No Starch Press.
  * Raman et al. (2020) Narun Raman, Nathaniel Sauerberg, Jonah Fisher, and Sneha Narayan. 2020.  Classifying Wikipedia Article Quality With Revision History Networks. In _Proceedings of the 16th International Symposium on Open Collaboration_ _(OpenSym 2020)_. Association for Computing Machinery, New York, NY, USA, 1–7.
  * Recht et al. (2019) Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019.  Do ImageNet Classifiers Generalize to ImageNet?. In _International Conference on Machine Learning_. PMLR, 5389–5400.
  * Redi et al. (2021) Miriam Redi, Martin Gerlach, Isaac Johnson, Jonathan Morgan, and Leila Zia. 2021.  A Taxonomy of Knowledge Gaps for Wikimedia Projects (Second Draft).  _arXiv:2008.12314 [cs]_ (Jan. 2021).  arXiv:2008.12314 [cs]
  * Sarkar et al. (2019) Soumya Sarkar, Bhanu Prakash Reddy, Sandipan Sikdar, and Animesh Mukherjee. 2019.  StRE: Self Attentive Edit Quality Prediction in Wikipedia.  _arXiv:1906.04678 [cs]_ (June 2019).  arXiv:1906.04678 [cs]
  * Schmidt and Zangerle (2019) Manuel Schmidt and Eva Zangerle. 2019.  Article Quality Classification on Wikipedia: Introducing Document Embeddings and Content Features. In _Proceedings of the 15th International Symposium on Open Collaboration_ _(OpenSym ’19)_. Association for Computing Machinery, New York, NY, USA, 1–8.
  * Shi et al. (2019) Feng Shi, Misha Teplitskiy, Eamon Duede, and James A. Evans. 2019.  The Wisdom of Polarized Crowds.  _Nature Human Behaviour_ 3, 4 (April 2019), 329–336.
  * Strathern (1997) Marilyn Strathern. 1997\.  ‘Improving Ratings’: Audit in the British University System.  _European Review_ 5, 3 (July 1997), 305–321.
  * TeBlunthuis et al. (2019) Nathan TeBlunthuis, Tilman Bayer, and Olga Vasileva. 2019\.  Dwelling on Wikipedia: Investigating Time Spent by Global Encyclopedia Readers. In _OpenSym ’19, The 15th International Symposium on Open Collaboration_. Skövde, Sweden, 14.
  * TeBlunthuis et al. (2021) Nathan TeBlunthuis, Benjamin Mako Hill, and Aaron Halfaker. 2021.  Effects of Algorithmic Flagging on Fairness: Quasi-Experimental Evidence from Wikipedia.  _Proceedings of the ACM on Human-Computer Interaction_ 5, CSCW1 (April 2021), 56:1–56:27.  arXiv:2006.03121
  * Tripodi (2021) Francesca Tripodi. 2021\.  Ms. Categorized: Gender, Notability, and Inequality on Wikipedia.  _New Media & Society_ (June 2021), 14614448211023772.
  * van der Velden (2013) Maja van der Velden. 2013\.  Decentering Design: Wikipedia and Indigenous Knowledge.  _International Journal of Human–Computer Interaction_ 29, 4 (March 2013), 308–316.
  * Venables et al. (2002) W. N Venables, Brian D Ripley, and W. N Venables. 2002\.  _Modern Applied Statistics with S_.  Springer, New York.
  * Warncke-Wang et al. (2015) Morten Warncke-Wang, Vladislav R. Ayukaev, Brent Hecht, and Loren G. Terveen. 2015.  The Success and Failure of Quality Improvement Projects in Peer Production Communities. In _Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing_ _(CSCW ’15)_. Association for Computing Machinery, New York, NY, USA, 743–756.
  * Warncke-Wang et al. (2013) Morten Warncke-Wang, Dan Cosley, and John Riedl. 2013\.  Tell Me More: An Actionable Quality Model for Wikipedia. In _Proceedings of the 9th International Symposium on Open Collaboration_ _(WikiSym ’13)_. Association for Computing Machinery, New York, NY, USA, 1–10.
  * Zhang et al. (2017) Ark Fangzhou Zhang, Danielle Livneh, Ceren Budak, Lionel P. Robert, and Daniel M. Romero. 2017.  Crowd Development: The Interplay between Crowd Evaluation and Collaborative Dynamics in Wikipedia.  _Proceedings of the ACM on Human-Computer Interaction_ 1, CSCW (Dec. 2017), 1–21.
  * Zhang et al. (2018) Shiyue Zhang, Zheng Hu, Chunhong Zhang, and Ke Yu. 2018\.  History-Based Article Quality Assessment on Wikipedia. In _2018 IEEE International Conference on Big Data and Smart Computing (BigComp)_. 1–8.