2014 1 2 245-264 10.15346/hc.v1i2.12 © 2014, Ponciano & Brasileiro. CC-BY-3.0

# Finding Volunteers’ Engagement Profiles in
Human Computation for Citizen Science Projects

Lesandro Ponciano  Universidade Federal de Campina Grande  Francisco
Brasileiro  Universidade Federal de Campina Grande

###### Abstract

Human computation is a computing approach that draws upon human cognitive
abilities to solve computational tasks for which there are so far no
satisfactory fully automated solutions even when using the most advanced
computing technologies available. Human computation for citizen science
projects consists in designing systems that allow large crowds of volunteers
to contribute to scientific research by executing human computation tasks.
Examples of successful projects are Galaxy Zoo and FoldIt. A key feature of
this kind of project is its capacity to engage volunteers. An important
requirement for the proposal and evaluation of new engagement strategies is
having a clear understanding of the typical engagement of the volunteers;
however, even though several projects of this kind have already been
completed, little is known about this issue. In this paper, we investigate the
engagement pattern of the volunteers in their interactions in human
computation for citizen science projects, how they differ among themselves in
terms of engagement, and how those volunteer engagement features should be
taken into account for establishing the engagement encouragement strategies
that should be brought into play in a given project. To this end, we define
four quantitative engagement metrics to measure different aspects of volunteer
engagement, and use data mining algorithms to identify the different volunteer
profiles in terms of the engagement metrics. Our study is based on data
collected from two projects: Galaxy Zoo and The Milky Way Project. The results
show that the volunteers in such projects can be grouped into five distinct
engagement profiles that we label as follows: hardworking, spasmodic,
persistent, lasting, and moderate. The analysis of these profiles provides a
deeper understanding of the nature of volunteers’ engagement in human
computation for citizen science projects.

keywords: citizen science, human computation, engagement, participation,
retention

##  1 Introduction

Human computation is a computing approach based on harnessing human cognitive
abilities to solve computational tasks for which there are so far no
satisfactory fully automated solutions even when using the most advanced
computing technologies currently available Quinn and Bederson (2011). Examples
of such tasks may be found in the areas of natural language processing, image
understanding, and creativity. They have been shown to be often in scientific
applications related to disciplines such as biology, linguistics, and
astronomy Wiggins and Crowston (2012); Lintott and Reed (2013). As a result,
it has become common among scientists to start projects to recruit ordinary
people for executing human computation tasks, which we call human computation
for citizen science projects. Citizen science can be broadly defined as a
partnership between scientists and ordinary people willing to contribute to an
authentic scientific research effort Cohn (2008); Dickinson et al. (2012);
Lintott and Reed (2013). A large range of activities can be carried out by
ordinary people in citizen science Goodchild (2007); Cohn (2008); Wiggins and
Crowston (2012). Those activities may require only some simple abilities, such
as data collecting and reporting, or more complex cognitive abilities such as
data aggregation and classification. In human computation for citizen science
projects, participants contribute by executing tasks that require cognitive
abilities. Examples of projects with such feature are Galaxy Zoo Lintott et
al. (2008) and FoldIt Cooper et al. (2010).

The contribution behaviour of people taking part in this type of project can
be examined in the light of two different research approaches centered on the
notions of voluntarism Clary et al. (1998); Wilson (2000) and human engagement
O’Brien and Toms (2008); Simpson (2009); Lehmann et al. (2012). Voluntarism
literature usually distinguishes between two different types of contribution
behaviour: helping activity behaviour and volunteerism behaviour Clary et al.
(1998); Wilson (2000). Helping activity behaviour designates a form of
_sporadic_ participation in which the individual is faced with an unexpected
request to help someone to do something. Volunteerism behaviour, on the other
hand, concerns to a kind of _planned_ behaviour. Volunteers are usually
actively seeking out opportunities to help others. They typically commit
themselves to an ongoing relationship at considerable personal cost in terms
of dedicated time or cognitive effort. Drawing this distinction between
helping activity and voluntarism seems to us to be important also in the
context of human computation for citizen science projects. A recent
characterization of the behaviour of volunteers in such projects brings to
light the existence of two main groups of participants: transient and regular
Ponciano et al. (2014b). Transient participants exhibit a helping behaviour,
whereas the behaviour of regular participants fits into the definition of
volunteerism. Not surprisingly, volunteers typically constitute a minority
among the participants, and execute the largest part of tasks in the project.
Thus, a key feature for the success of a human computation for citizen science
project is the capacity to foster such kind of sustained contribution
behaviour.

Fostering sustained contribution behaviour is an issue that has been widely
addressed in human engagement studies. Current literature on human engagement
focuses on the human behaviour when individuals are self-investing personal
resources such as time, physical energy, and cognitive power Bakker and
Demerouti (2008); O’Brien and Toms (2008); Simpson (2009); Lehmann et al.
(2012); McCay-Peet et al. (2012). Studies in this area usually focus on both
qualitative and quantitative dimensions of engagement by (i) analysing the
psychological factors behind engagement/disengagement such as motivation,
satisfaction, and frustration; and (ii) measuring the level of engagement
quantitatively in terms of the degree of contribution and the duration of the
contribution.

Several studies have been devoted to the understanding of psychological
factors of volunteer engagement in human computation for citizen science
projects Raddick et al. (2010); Rotman et al. (2012); Jennett et al. (2014);
Nov et al. (2014), while few studies have focused on quantitatively estimation
of the level of engagement of the volunteers Ponciano et al. (2014b). The lack
of studies with this perspective is an important constraint because a
fundamental requirement for proposing and evaluating new engagement strategies
is having a clear understanding of how volunteers typically behave in such
situations. This study aims at filling this gap by providing a quantitative
analysis of the nature of engagement of volunteers by using log data related
to their execution of tasks. Three research questions are addressed in this
study: $1)$ how engaged the volunteers are during their interaction with the
project; $2)$ what similarities and differences they exhibit among themselves
in terms of engagement; and $3)$ how the engagement characteristics of the
volunteers can be exploited for establishing the engagement strategies to be
implemented in a given project.

In order to answer these questions, we go through existing human engagement
studies and, based on the concepts and theories put forward, we propose the
following four metrics to measure the level of engagement of each volunteer:
activity ratio, relative activity duration, daily devoted time, and variation
in periodicity. Activity ratio is a measure of the return rate of the
volunteer to the project during the period that he/she stays contributing to
it. Daily devoted time is a measure of the length of the daily engagement.
Relative activity duration, in turn, is a measure of the duration of the
volunteer’s long-term engagement. Finally, variation in periodicity informs us
about the deviation in the periodicity with which the volunteer executes tasks
in the project. By using hierarchical and k-means algorithms, we cluster the
volunteers according to the values of their engagement metrics in order to
find out the different engagement profiles that arise from their natural
behaviour within the project.

We analyse volunteer engagement profiles according to the data collected from
two popular projects hosted at the Zooniverse platform: Galaxy Zoo and The
Milky Way Project. These projects ran for almost 2 years between 2010 and 2012
and involved more than one billion executed tasks and thousands of
participants, which turns them into valuable sources for the analysis of a
wide range of engagement aspects of the volunteers. In both projects, we found
5 different clusters of volunteers based on visual inspection and statistical
measures. Each cluster stands for a distinct engagement profile brought for by
the behaviour shown by the volunteers during their participation in the
projects. The distinct engagement profiles brought to light in this way are
labelled as: hardworking, spasmodic, persistent, lasting, and moderate.

Hardworking engagement is characterised by larger activity ratio, low
variation in periodicity and shorter relative activity duration. Volunteers
who exhibit this type of engagement profile typically work hard and regularly
when arriving at the project, but may leave the project quickly. Spasmodic
engagement is distinguished by a relatively high activity ratio and moderate
variation in periodicity. Volunteers who exhibit this engagement profile
provide an intense contribution, at a short period of time and with irregular
periodicity within this period. Persistent engagement, in turn, is
characterised by a larger activity duration and low activity ratio. Volunteers
who exhibit a persistent engagement profile remain in the project for a long
period of time but contribute only a few days within this time period. Lasting
engagement, in turn, is characterised by an engagement pattern similar to
persistent engagement, with the difference that volunteers exhibit here a much
shorter activity duration. Finally, moderate volunteers have intermediate
scores in all categories of engagement metrics.

Regarding the distribution of the volunteers per profile, the highest
percentage of volunteers ($30\%$ in The Milky Way Project and $31\%$ in Galaxy
Zoo) exhibits a moderate engagement profile, while few volunteers ($13\%$ in
The Milky Way Project and $16\%$ in Galaxy Zoo) show persistent engagement.
Given the total amount of human effort time required to execute all the tasks
in the project, the aggregate time devoted by volunteers who exhibit a
persistent engagement profile accounts for $40\%$ of total time in The Milky
Way Project and $46\%$ in Galaxy Zoo; this is the volunteer profile that
stands for the largest contribution.

The method we propose to measure the engagement of volunteers and set up
engagement profiles has been shown to be satisfactory in bringing to light the
main similarities and differences among the volunteers. The fact that the
results thus obtained are consistent throughout different projects strengthens
the thesis that engagement profiles can arise in various other projects.
Several other discussions can be drawn from our analysis. For example, the
engagement profiles enable the development of new recruitment strategies to
attract volunteers with a desired engagement profile as well as the design of
personalised engagement strategies that focuses on improving specific
engagement metrics. Finally, our results call for further theoretical and
qualitative studies that investigate the motivation of volunteers in the light
of the distinct engagement profiles they may exhibit. The combination of a
quantitative analysis of volunteer engagement and the psychological factors
established in qualitative studies will advance our comprehension about the
engagement patterns of volunteers in human computation and citizen science.

In this study we put forward three main contributions. First, we propose four
metrics to measure the level of engagement of volunteers with regard to both
the duration of the period of engagement with the project and the degree of
engagement during this period. Furthermore, we provide a deeper quantitative
assessment of volunteer engagement profiles derived from two popular human
computation for citizen science projects. To the best of our knowledge, this
is the first study assessing natural engagement profiles in volunteer task
execution behaviour in this type of project. Finally, this study allows us to
go beyond previous studies by covering a larger number of volunteers and
bringing forth engagement aspects which have so far not been identified in
studies focusing on qualitative methodologies.

The rest of this work is organised as follows. We provide first a background
of human engagement studies and discuss relevant previous work. Next we
describe our method to measure the volunteer engagement and identify
engagement profiles. Finally, we present an analysis of volunteer engagement
in Galaxy Zoo and The Milky Way Project.

##  2 Background and Related Work

This study builds on a broad set of studies covering volunteer engagement,
human computation and citizen science projects. In this section, we first
provide a background to the subject of human engagement. Thereafter, we
discuss the related work.

###  2.1 What is engagement and how to approach it

The subject of human engagement has been studied within a variety of
disciplines, such as education Meece et al. (1988), management science Simpson
(2009) and computer science O’Brien and Toms (2008). Some studies make an
attempt to conceptualize the term engagement in an interdisciplinary
perspective González-Romá et al. (2006); Bakker and Demerouti (2008); O’Brien
and Toms (2008); Simpson (2009); Lehmann et al. (2012); McCay-Peet et al.
(2012). A consensus that emerges from these studies is that engagement means
to participate in any enterprise by self-investing personal resources, such as
time, physical energy, and cognitive power.

O’Brien and Toms (2008) provide a conceptual framework to study human
engagement with technology. This framework establishes that the entire process
of engagement is comprised of four stages: point of engagement, period of
sustained engagement, disengagement and reengagement. The point of engagement
is the time at which the human perform the first action in the system. The
period of sustained engagement is the continuous period of time in which
he/she keeps on performing actions in the system. Disengagement occurs when
the period of sustained engagement ends. Finally, reengagement denotes new
engagement cycles composed of point the three first stages. Studies of such
process involve at least four dimensions: type of engagement, psychological
factors of engagement, duration of engagement, and degree of engagement.

The type of engagement is defined by the kind of personal resources and skills
that humans invest in performing an activity. Examples of types of engagement
are social engagement Porges (2003) and cognitive engagement Corno and
Mandinach (1983). Social engagement refers to actions that require humans to
interact with others. It is widely studied in areas such as online social
networks and communities Preece (2000); Millen and Patterson (2002). Cognitive
engagement refers to actions that require mainly human cognitive effort. It
has been widely addressed in educational psychology and work engagement Meece
et al. (1988); Simpson (2009).

The psychological factors of engagement are related to the motives leading to
a point of engagement, disengagement and reengagement, such as motivation,
satisfaction, perceived control, and frustration. Studies have proposed and/or
instantiated various theories in order to construct a framework of theories
that explain the psychological factors behind human engagement González-Romá
et al. (2006); O’Brien and Toms (2008). These theories include the self-
determination theory Deci and Ryan (2000) and the self-efficacy theory Bandura
(1977). The self-determination theory establishes that human motivation can be
broadly divided into intrinsic motivations, associated with inner personal
reward, and extrinsic motivations, associated with earning an external reward
or avoiding a punishment. The self-efficacy theory, in turn, advances the idea
that perceived human efficacy determines if an individual will initiate an
activity, how much effort will be expended, and how long the activity will be
sustained.

The duration of engagement measures the duration of the period of sustained
engagement, sometimes called retention. It expresses how long a human keeps on
to the system. It is short-term engagement when it occurs during a relatively
short period of time (e.g. minutes or hours), and long-term engagement when it
lasts a long period of time (e.g. months or years). In short-term engagement,
the point of engagement is the point in time at which the individual performs
the first action within the system, the period of engagement is the time span
under which he/she keeps interacting with the system in a continuous working
session, and the point of disengagement is the point in time at which the
working session ends. In long-term engagement, the point of engagement is the
point in time at which the individual performs the first action within the
system, the period of engagement refers to the number of days under which
she/he keeps on interacting with the system, and the point of disengagement
refers to the day when he/she leaves the system. Thus, long-term engagement
may consist of several short-term engagement cycles.

Finally, the degree of engagement is a quantitative measure of the degree of
participation during the period of sustained engagement. It can also be viewed
as a measure of the amount of resources invested by humans in participating in
the system. Measuring the degree of engagement has proven a challenging task.
Some studies use surveys to collect information about how humans perceive
their level of engagement and hence estimate their degree of engagement (e.g.,
O’Brien and Toms (2010); McCay-Peet et al. (2012)). Other studies use
behavioural data stored in logs of the system to measure the degree of
engagement (e.g. Lehmann et al. (2012)).

###  2.2 Related work

The dimensions of engagement presented in the last section are helpful to
framing the previous studies in engagement. There is an extensive body of work
dealing with engagement in technology-mediated social participation systems
Kraut et al. (2010) such as wiki-based systems Butler et al. (2002); Bryant et
al. (2005); Butler et al. (2008); Schroer and Hertel (2009); Preece and
Shneiderman (2009); Niederer and Van Dijck (2010); Liu and Ram (2011); Welser
et al. (2011); Zhu et al. (2012), open source software projects Hertel et al.
(2003); Niederer and Van Dijck (2010), and human computation for citizen
science projects Raddick et al. (2010); Rotman et al. (2012); López et al.
(2012); Mao et al. (2013); Jennett et al. (2014).

Wiki-based systems such as Wikipedia provide means that allow participants to
engage in a broad range of activities, such as the insertion of a sentence in
an article, modification of an existing reference, reverting an article to a
former version etc Butler et al. (2008); Liu and Ram (2011); Welser et al.
(2011). Participants assume different roles in the system when some of them
focus on performing a single type of activity, and others focus on performing
other types of activities Butler et al. (2008); Niederer and Van Dijck (2010);
Liu and Ram (2011). Such roles characterise different types of engagement in
the system. The motivation of the participants and their perception of their
own roles usually change as they become more active in the system Bryant et
al. (2005); Burke and Kraut (2008); Schroer and Hertel (2009); Preece and
Shneiderman (2009). Since such systems provide a collaborative environment,
the behaviour of some of the participants may also affect the behaviour of
others Butler et al. (2002); Zhu et al. (2012).

Studies on open source software (OSS) projects, in turn, have focused on
understanding the psychological factors that lead participants to engage in
OSS projects, and the kind of rewards they expect Hertel et al. (2003);
Roberts et al. (2006). For example, Hertel et al. (2003) show that
psychological factors appeared to be similar to those behind voluntary action
within social movements such as the civil rights, labour, and peace movements.
Studies on Apache projects suggest that there are also interrelationships
between motivation and degree of engagement Roberts et al. (2006). Extrinsic
motivation, such as monetary and status within the system, leads to above
average contribution levels, while intrinsic motivations do not significantly
impact average contribution levels.

Differently from Wiki-based systems, in which there is a diversity of types of
engagement, the role played by volunteers in human computation for citizen
science projects is mainly the execution of well defined human computation
tasks, although some projects allow volunteers to carry out social engagement
activities, for instance interacting in forums Fortson et al. (2012); Luczak-
Roesch et al. (2014). In such projects, as in the case of studies in wiki-
based systems and OSS projects, the psychological factor is the dimension of
engagement that has received most attention Raddick et al. (2010); Rotman et
al. (2012); Jennett et al. (2014); Nov et al. (2014).

Raddick et al. (2010) analyse the motivations of volunteers in the Galaxy Zoo
project. It is shown that, among $12$ categories of motivations mentioned by
the volunteers, the most mentioned category is interest in astronomy, which is
the theme of the project. Rotman et al. (2012) and Rotman et al. (2014) show
that the motivation of volunteers changes dynamically throughout the period of
their contribution to the projects. Jennett et al. (2014) analyse factors that
led volunteers to dabble and/or drop-out in the Old Weather project. The
analysis shows that this kind of volunteers are less motivated, though they
care about the project and the quality of the work they perform. Thus,
projects should be designed to encourage both dabbling and commitment. Nov et
al. (2014) analyses motivation factors that affect the quality and the
quantity of contributions to citizen science projects.

In general, these studies clarify several aspects of why volunteer engages in
human computation for citizen science projects. However, little progress has
been made in terms of understanding how to measure volunteer engagement and to
uncover natural patterns in which the engagement occurs. This fact constitutes
an important shortcoming because a key feature of this kind of project is its
capacity to engage volunteers. A clear understanding of how volunteers
typically engage with such kinds of projects is fundamental for proposing and
evaluating new strategies to encourage engagement.

##  3 Finding Engagement Profiles

In this section, we first present the metrics proposed to measure the degree
of engagement and the duration of engagement of volunteers. Then, we present a
strategy to cluster volunteer based on the values of these metrics for the
volunteers. This clustering allows the identification of profiles of
volunteers exhibiting similar engagement patterns.

###  3.1 Measuring engagement

We characterise volunteers according to how they score in different engagement
metrics. Engagement metrics are measures of volunteer interaction and
involvement with the project. The engagement metrics proposed in this section
are based on the conceptual framework proposed by O’Brien and Toms (2008). By
using this framework, we analyse the engagement over time of volunteers taking
into account their points of engagement, periods of sustained engagement,
disengagements and reengagements.

Figure 1 shows the structure of the time line of a volunteer during
participation in a project. This figure shows five concepts used in the
calculations of our metrics: the time the volunteer could potentially remain
linked to the project, days the volunteer remain linked to the project, the
active days, the time devoted on an active day, and the number of days elapsed
between two active days. Our metrics are designed to measure the engagement of
participants that exhibit an ongoing contribution and have contributed in at
least two different days. By doing so, we focus on participants that are more
likely to fit into the voluntarism definition Clary et al. (1998); Wilson
(2000).

Figure 1: Structure of the time line of a volunteer in a project, highlighting
the active days and working sessions on the active days.

The time a volunteer $i$ can potentially remain linked to the project is the
number of days elapsed between the day in which the volunteer joined the
project and the day in which the project is concluded. It is denoted by
$w_{i}$ days. An active day of a volunteer $i$ is a day on which this
volunteer is active in the project. We consider that a volunteer is active on
a particular day if he/she executes at least one task during that day. We
define $A_{i}$ as the sequence of dates in which the volunteer $i$ is active.
The time devoted on a specific active day is the sum of the time duration of
the contribution sessions of the volunteer on that active day. Contribution
sessions are continuous short periods of time during which the volunteer keeps
executing tasks. We define $D_{i}$ as the multiset of the amount of time the
volunteer $i$ devotes to the project on each active day. The time elapsed
between two active days is the number of days it took to the volunteer to
return to the project since the latest active day. We define $B_{i}$ as the
multiset of the number of days elapsed between every two sequential active
days. Considering $w_{i}$, $A_{i}$, $D_{i}$ and $B_{i}$, we can derive metrics
to measure the degree and the duration of engagement of each volunteer.

We define two metrics of degree of engagement: activity ratio and daily
devoted time. Activity ratio ($a_{i}$) is the proportion of days on which the
volunteer was active in relation to the total of days he/she remained linked
to the project. It can be computed as
$a_{i}=\frac{|A_{i}|}{(Max(A_{i})-Min(A_{i}))+1}$, $a\in(0,1]$. The closer to
1, the more assiduous the volunteer is during the time he/she remained linked
to the project. Daily devoted time ($d_{i}$) is the averaged hours the
volunteer remain executing tasks on each day he/she is active. It can be
computed as $d_{i}=avg(D_{i})$, $d\in(0,24]$. The higher the average, the
longer the time the volunteer devotes to the project executing tasks on the
days he/she is active. Note that, because the human computation projects
usually consist of different time-consuming tasks, the time devoted by the
volunteers executing tasks is a better measure of their degree of engagement
than the number of tasks they execute Geiger and Halfaker (2013); Ponciano et
al. (2014b).

We also define two metrics to assess the duration of engagement: relative
activity duration and variation in periodicity. Relative activity duration
($r_{i}$) is the ratio of days during which a volunteer $i$ remains linked to
the project in relation to the total number of days elapsed since the
volunteer joined the project until the project is over ($w_{i}$). It is
defined as $r_{i}=\frac{(Max(A_{i})-Min(A_{i}))+1}{w_{i}}$, $r\in(0,1]$. When
$r_{i}=1$, the volunteer remains linked to project since she/he came to the
project until the project is completed. The closer to $1$, the more persistent
is the participation of the volunteer in the project. Variation in periodicity
($v_{i}$) is the standard deviation of the times elapsed between each pair of
sequential active days. It is computed as $v_{i}=sd(B_{i})$. When $v_{i}=0$,
the volunteer exhibits a constant elapsed time between each pair of sequential
active days; this indicates that he/she comes back to the project with perfect
periodicity. On the contrary, the larger $v_{i}$, the larger the deviation in
the periodicity in which the volunteer comes back to the project to perform
more tasks.

The above engagement metrics fit well into our objective of analysing the
degree of engagement and the duration of engagement of the volunteers.
Activity ratio allows us to analyse the return rate of each volunteer to the
project during the period that he/she stays contributing. Daily devoted time
gives us a view of the length of the daily engagement, which is related to the
duration of the short-term engagement. Relative activity duration allows us to
analyse the duration of long-term engagement weighted by the duration of the
period in which the volunteer can potentially remain linked to the project.
Finally, variation in periodicity informs us about the periodicity of return
during the long-term engagement.

###  3.2 Clustering volunteers according to engagement metrics

We use clustering algorithms to find out groups of volunteers who exhibit
similar values for the engagement metrics. The input to clustering algorithms
is a matrix $|I|\times 4$ in which each row stands for a volunteer $i\in I$
and each column is an engagement metric, i.e. $a$, $d$, $r$, and $v$. As the
results of clustering depend on the relative values of the parameters being
clustered, a normalisation of the parameters prior to clustering would be
desirable Jain (2008). We use range normalisation to scale the values of the
engagement metrics in the interval $[0,1]$. The scaling formula is
$x_{i}=\frac{x_{i}-x_{min}}{x_{max}-x_{min}}$, where $x$ denotes the
engagement metric and $i$ the volunteer.

To identify the suitable number of clusters, we first run a hierarchical
clustering algorithm and observe its dendrogram, which yields a suitable
interval to test the number of clusters. Next we run k-means, varying the
number of clusters ($k$) in the suggested interval and using as initial
centroids the centres identified in the hierarchical clustering, which usually
reduces the impact of noise and requires less iteration time Lu et al. (2008).
We select thereafter a suitable $k$ and evaluate the quality of the clustering
by computing the within-group sum of squares Anderberg (1973) and Average
Silhouette width Rousseeuw (1987).

Within-group sum of squares measures the differences between the volunteers
and the centre of the group to which they belong. The lower the within-group
sum of squares, the better the clustering. It indicates that volunteers
clustered in the same group exhibit similar values for the engagement metrics
and that the centre of the group represents the group adequately. Average
Silhouette width, in turn, measures how well separated and cohesive the groups
are. This statistics ranges from $-1$, indicating a very poor clustering, to
$1$, indicating an excellent clustering. Struyf et al. (1997) propose the
following subjective interpretation of the silhouette statistics: between
$0.71$ and $1.00$, a strong structure has been found; between $0.51$ and
$0.70$, a reasonable structure has been found; between $0.26$ and $0.50$, the
structure is weak and could be artificial, and hence it is recommended that
additional methods of analysis are tried out; less than or equal to $0.25$, no
substantial structure has been found. In this study, a silhouette statistics
larger than or equal to $0.51$ indicates a reasonable partition of the
different patterns of engagement exhibited by the volunteers.

##  4 Engagement Profiles in Galaxy Zoo and The Milky Way Project

In this section we use the proposed method to analyse the engagement of
volunteers in two projects: Galaxy Zoo and The Milky Way Project. We first
introduce these projects and detail the data set collected from them. Then, we
present the results on the quality of clustering in these data sets and the
discovered engagement profiles. Finally, we discuss the results and their
implications.

###  4.1 Datasets

The data used in this study was collected from two human computation for
citizen science projects: Galaxy Zoo Hubble and The Milky Way Project. Both
projects were developed and deployed in the Zooniverse (zooniverse.org)
citizen science platform.

The original Galaxy Zoo Lintott et al. (2008) was launched in July 2007, but
has been thereafter redesigned and relaunched several times. In this project,
participants were asked to answer a series of simple questions about the
morphology of galaxies. Each classifying volunteer on Galaxy Zoo is presented
with a galaxy image captured by either the Sloan Digital Sky Survey (SDSS) or
the Hubble Space Telescope. A decision tree of questions is presented with the
answer to each question being represented by a fairly simple icon. The task is
straightforward and no specialist knowledge is required. In this paper, we
used data of the third iteration of Galaxy Zoo: Galaxy Zoo Hubble. It was
launched in April 2010 and ran until September 2012. It consisted of
$9,667,586$ tasks executed by $86,413$ participants. In The Milky Way Project
Simpson et al. (2012), participants are asked to draw ellipses onto the image
to mark the locations of bubbles. A short online tutorial shows how to use the
tool, and examples of prominent bubbles are given. As a secondary task, users
can also mark rectangular areas of interest, which can be labelled as small
bubbles, green knots, dark nebulae, star clusters, galaxies, fuzzy red objects
or “other”. Users can add as many annotations as they wish before submitting
the image, at which point they are given another image for annotation. We used
data of The Milky Way Project launched in December 2010 and ran until
September 2012. It consisted of $643,468$ tasks executed by $23,889$
participants.

Each entry in the data set refers to one task execution. Each task execution
is described by project_id, task_id, user_id, datetime. The project_id field
is the name of the project. The task_id field is a unique task identifier in
the project. The user_id field is a unique volunteer identifier in the
project. Finally, the datetime field indicates the date and time when the task
was executed. To form volunteers’ working sessions, we use the threshold-based
methodology Geiger and Halfaker (2013); Mehrzadi and Feitelson (2012);
Ponciano et al. (2014b). Following this methodology, we compute the interval
of time elapsed between every two sequential task executions for each
volunteer. Given these intervals, we use the method proposed by Mehrzadi and
Feitelson (2012) to identify for each volunteer a threshold that distinguishes
short intervals from long intervals. Hence, whenever the interval between the
execution of two tasks is not larger than the threshold, the two tasks are
assumed to have been executed in the same working session; otherwise, the
tasks are assumed to have been executed in two different and consecutive
working sessions. For more details about this methodology, see Mehrzadi and
Feitelson (2012).

In both projects, participants are considered volunteers only if they have
been engaged in at least two days of activity. Only volunteers who arrived
before the last quarter of the total duration time of the project were
considered in the analyses, i.e. the first 502 days of The Milky Way Project
and the first 630 days of the Galaxy Zoo project. As Table 1 shows, the final
dataset consists of $23,547$ volunteers for the Galaxy Zoo and $6,093$
volunteers for The Milky Way Project, whereas 2485 volunteers contributed to
both projects. As shown by the descriptive statistics in this table, in both
projects the volunteers differ among themselves significantly in terms of all
the engagement metrics, all of which are significantly non-normal (Kolmogorov-
Smirnov normality tests showing p-value $<0.05$). The variations in the
engagement metrics of the volunteers do not point out at any form of anomalous
behaviour among the volunteers, which can thus be considered as natural
throughout.

Table 1: Descriptive statistics of engagement metrics of volunteers in the studied datasets | The Milky Way Project | Galaxy Zoo
---|---|---
#Volunteers | 6,093 | 23,547
Activity ratio |  $mean=0.40$, $sd=0.40$ |  $mean=0.33$, $sd=0.38$
Daily devoted time |  $mean=0.44$, $sd=0.54$ |  $mean=0.32$, $sd=0.40$
Relative activity duration |  $mean=0.20$, $sd=0.30$ |  $mean=0.23$, $sd=0.29$
Variation in periodicity |  $mean=18.27$, $sd=43.31$ |  $mean=25.23$, $sd=49.16$

###  4.2 Clustering

The result of the quality of the clustering when the number of clusters varies
between $2$ and $10$ is shown in Figure 2 for The Milky Way Project and in
Figure 3 for Galaxy Zoo. These figures show that $5$ is the number of groups
that best optimise the trade-off between the number of groups and the within-
group sum of squares (Fig 2(a) and 3(a)). This number of groups also yields an
Averaged Silhouette statistic of $0.53$ in The Milky Way Project (Fig.2(b))
and $0.51$ in the Galaxy Zoo project (Fig. 3(b)). These values indicate that a
reasonable clustering structure has been found for both projects.

(a) Within-groups sum of squares

(b) Average Silhouette statistic

Figure 2: Analysis of k-means clustering in The Milky Way Project. Within-
groups sum of squares and average Silhouette statistic as the number of groups
(k) is varied.

(a) Within-groups sum of squares

(b) Average Silhouette statistic

Figure 3: Analysis of k-means clustering in the Galaxy Zoo project. Within-
groups sum of squares and average Silhouette statistic as the number of groups
(k) is varied.

###  4.3 Profiles

In order to understand the different groups uncovered by the clustering
algorithm, we analyse: (i) the centroids that represent the groups; (ii) the
correlation between each pair of volunteer engagement metrics for each group;
and (iii) how the groups differ in terms of the number of volunteers and
aggregate contribution. In this analysis, we established labels to the groups
in order to put into pespective their main engagement characteristics. Thus,
the groups represent different engagement profiles labelled as follows:
hardworking engagement; spasmodic engagement, persistent engagement; lasting
engagement; and moderate engagement. The general characteristics of these
profiles are shown in Figure 4, Table 2 and Table 3.

Figure 4 shows the centroids that represent each profile and how they differ
in terms of engagement metrics. In each image, the horizontal axis stands for
the engagement profiles, each bar representing one engagement metric, and the
vertical axis indicates how the profiles score in the particular engagement
metrics. Table 2, in turn, shows how the profiles differ in terms of
correlation between their engagement metrics. Finally, Table 3 shows how the
profiles differ in terms of the number of volunteers and how their aggregate
contributions differ in terms of total working time devoted to the project. In
the following paragraphs, we elaborate on these results by analysing each
engagement profile in turn.

(a) The Milky Way Project

(b) Galaxy Zoo

Figure 4: Score of each engagement profile in each engagement metric.
Engagement profiles are represented by the centroids of groups of volunteers
identified by the k-means algorithm in (a) The Milky Way Project and (b)
Galaxy Zoo project. Table 2: Spearman $\rho$ correlation between each pair of
engagement metrics of volunteers within each engagement profile

The Milky Way Project
---
Pair | Hardworking | Spasmodic | Persistent | Lasting | Moderate
$N=1,535$ | $N=1,060$ | $N=817$ | $N=844$ | $N=1,837$
$\rho(a,r)$ | -0.24* | -0.38* | -0.14* | -0.26* |  -0.74*
$\rho(a,v)$ |  -0.99* | -0.22* | 0.06 | 0.39* | -0.13*
$\rho(a,d)$ | -0.07* | -0.05 |  0.43* | 0.37* | 0.14*
$\rho(r,v)$ | 0.24* |  0.59* | -0.13* | -0.04 |  0.44*
$\rho(r,d)$ | 0.14* | 0.23* | -0.09* | 0.02 | 0.01
$\rho(v,d)$ | 0.07* | 0.29* | 0.19* | 0.31* | 0.21*
Galaxy Zoo
Pair | Hardworking | Spasmodic | Persistent | Lasting | Moderate
$N=4,572$ | $N=3,611$ | $N=3,783$ | $N=4,250$ | $N=7,331$
$\rho(a,r)$ | -0.30* |  -0.45* | 0.15* | -0.23* |  -0.76*
$\rho(a,v)$ |  -0.99* | -0.31* | -0.26 | 0.27* | -0.12*
$\rho(a,d)$ | -0.10* | 0.03 | 0.33* | 0.30* | 0.19*
$\rho(r,v)$ | 0.30* |  0.66* | -0.12* | 0.00 |  0.43*
$\rho(r,d)$ | 0.07* | 0.17* | 0.08* | 0.02 | -0.05*
$\rho(v,d)$ | 0.10* | 0.26* | -0.01 | 0.16* | 0.16*

  * •

Note 1: *Spearman’ $\rho$ significant coefficient of correlation (p-value
$<0.05$).

  * •

Note 2: Moderate and strong correlations are highlighted in boldface.

Table 3: Profiles importance in terms of the number of volunteers and their
devoted time

Profiles | The Milky Way Project | Galaxy Zoo
---|---|---
#Volunteers | Devoted time | #Volunteers | Devoted time
Hardworking | 1,535 (25.19%) | 2,030.26 (13.86%) | 4,572 (19.42%) | 4,857.49 (9.44%)
Spasmodic | 1,060 (17.40%) | 1,912.05 (13.05%) | 3,611 (15.34%) | 6,061.40 (11.78%)
Persistent | 817 (13.41%) | 5,846.58 (39.91%) | 3,783 (16.07%) | 23,757.64 (46.16%)
Lasting | 844 (13.85%) | 2,273.10 (15.52%) | 4,250 (18.05%) | 8,168.95 (15.87%)
Moderate | 1,837 (30.15%) | 2,588.28 (17.67%) | 7,331 (31.13%) | 8,621.64 (16.75%)
sum | 6,093 (100%) | 14,650.27 (100%) | 23,547 (100%) | 51,467.12 (100%)

  * •

Note: The highest number of volunteers and the longest devoted time for each
project are highlighted in boldface.

Hardworking engagement. Volunteers who exhibit a hardworking engagement
profile have larger activity ratio and shorter relative activity duration
compared to others profiles (Fig 4). Such metrics indicate that volunteers in
this profile work hard when they come into the project, but may leave the
project soon. This engagement profile also exhibits low variation in
periodicity. This means that volunteers who exhibit this engagement profile
return to the project to perform more tasks in nearly equal intervals of time,
which makes the time of return of these volunteers fairly predictable. Other
intrinsic feature of this group of volunteers is a very strong negative
correlation between activity ratio and variation in periodicity
($\rho(a,v)=-0.99$, in both projects). This correlation indicates that the
more days the volunteers return to the project to perform tasks, the less
variable are the time intervals between their active days.

Spasmodic engagement. This engagement profile is distinguished by a relatively
high activity ratio and low activity duration (Fig 4). This group of
volunteers exhibits a positive correlation between relative activity duration
and variation in periodicity. This correlation is moderate ($\rho(r,v)=0.59$)
in the Milky Way Project and strong ($\rho(r,v)=0.66$) in the Galaxy Zoo
project (Table 2). These correlations indicate that the longer the period of
time the volunteers remain linked to the project, the more erratic is the
periodicity of their return to the project within this period. All these
characteristics indicate that contributions of volunteers exhibiting this
profile typically takes place during a short period of time and with irregular
periodicity within this period.

Persistent engagement. Persistent engagement is characterised by the largest
relative activity duration, the highest variation in period, and a short
activity ratio (Fig 4). Thus, volunteers with a persistent engagement profile
remain linked to the project for a long interval of time but are active only a
few days within this interval. Considering these engagement metrics,
persistent engagement may be seen as the opposite of hardworking engagement.
In both projects, a small percentage of all the volunteers fall in this
engagement profile: $13.41\%$ in The Milky Way Project and $16.07\%$ in the
Galaxy Zoo project. Together, these volunteer stands for the largest
percentage of the total working time devoted to each project, $39.91\%$ in The
Milky Way Project and $46.16\%$ in the Galaxy Zoo project (Table 3). It is the
most important profile in terms of devoted working time.

Lasting engagement. This is the engagement profile of volunteers exhibiting
comparatively high relative activity duration and variation in periodicity
(Fig 4). This kind of volunteers show an activity ratio similar to that
exhibited by the volunteers who stay longer in the project (persistent
engagement) but remain in the project during a shorter period of time.
Finally, this is the only engagement profile showing very weak or weak
correlation between all pairs of metrics in both projects (Table 2).

Moderate engagement. As shown in Figure 4, this engagement profile has no
particularly distinguishable engagement metrics. Compared to the other
profiles, moderate volunteers exhibit intermediate values in all engagement
metrics. One important characteristic of moderate engagement is a strong
negative correlation between activity ratio and relative activity duration.
This correlation is $\rho(a,r)=-0.74$ in The Milky Way Project and
$\rho(a,r)=-0.76$ in Galaxy Zoo (Table 2). These values indicate that the
degree of volunteer engagement in this profile falls with increased engagement
duration. Hence, the more days the volunteers return to the project to perform
tasks, the shorter is the total period of time that they remain linked to the
project. This engagement profile is exhibited by most volunteers in both
studied projects: nearly $30\%$ of the volunteers in The Milky Way Project and
$31\%$ in Galaxy Zoo fall into this engagement profile (Table 3).

###  4.4 Discussion

Our results show that volunteers in the studied projects share several
similarities and differences in terms of engagement. The identified profiles
of engagement put into perspective such similarities and differences.
Furthermore, they help us to better understand how the different engagement
patterns result in different levels of aggregated contribution to the
projects. Several practical and research discussions can be done from this
analysis. We focus on four of them, which are: profile-oriented volunteers’
recruitment, personalised engagement strategies, psychological factors behind
the engagement profiles, and external validity of the results.

Profile-oriented volunteers’ recruitment. It is natural that scientists
running citizen science projects that require human computation want to devote
more effort in recruiting volunteers who exhibits a desired engagement
profile. It is still the most important aspect when they want to optimise the
tradeoff between the costs of recruiting volunteers and the benefit of having
all tasks of the project performed as soon as possible Ponciano et al.
(2014a). Studies have been devoted to understanding how different disclosure
campaigns (e.g. traditional media and online media Robson et al. (2013))
differ in terms of the type of volunteers they attract. In a similar
direction, it is also important to know how different disclosure campaigns
differ in terms of the engagement profile of the volunteers they attract. For
example, could a disclosure campaign based on sending e-mails to people
interested in the theme of the project (e.g., astronomy, biology) attract more
persistent volunteers than advertising campaigns in traditional media? Other
important aspects that can be taken into account in optimising volunteer
recruitment is human homophily McPherson et al. (2001), which is the principle
that humans tend to be similar to their friends in several aspects. Perhaps
taking homophily into account one could motivate volunteers with a desired
engagement profile to recruit volunteers among his/her relatives, friends, and
colleagues with a similar profile? Hence, new and more effective recruitment
procedures might be brought forth with an increased knowledge on volunteer
engagement profiles.

Personalised engagement strategies. Besides recruiting more suitable
volunteers, it is also important to keep existing volunteers engaged. The
impact of management practices on volunteer engagement is a widely discussed
issue in volunteerism literature Clary et al. (1992); Cravens (2000). Such
practices are implemented by volunteer supervisors in a way that takes into
account the specific behaviour of each volunteer, aiming thereby at enriching
the volunteer experience and satisfying organizational needs. By showing that
volunteers in human computation for citizen science projects behave very
differently from each other, this study encourages the development of a
component to manage the engagement of volunteers in such projects. This
component would incorporate personalised engagement strategies Fischer (2001);
López et al. (2012); Mao et al. (2013) derived from the volunteer engagement
profiles uncovered in the present work. The component could also both monitor
the contribution behaviour of each volunteer and, when necessary,
automatically trigger a suitable engagement strategy. Prospective volunteers
with different behaviour profiles should be approached with different
engagement strategies, which could focus on e.g. encouraging a reduction or an
improvement of their engagement.

Strategies can focus on encouraging a reduction of volunteer engagement when,
for example, some volunteers start to compromise too much of their time to the
project, which could perhaps have a negative impact on the rest of his/her
social life, in the worst case leading to a state of burnout González-Romá et
al. (2006); Simpson (2009). Fortunately, this is not the typical situation in
the two projects we have studied; even volunteers with a hardworking
engagement profile devote typically less than $21$ minutes per day to the
project, which is not alarming. It is important that this kind of behavior can
be monitored, and, if necessary, strategies are put in place to deal with the
potential harm that this can bring to volunteers. When volunteers exhibit a
suitable engagement profile, it is very important to recognize their
contributions in order to keep them engaged Wilson (2000); Rotman et al.
(2012). Strategies can also focus on encouraging the improvement of volunteer
engagement when volunteers exhibit a level of engagement below project
average. This occurred frequently in the projects we have studied. Each
volunteer engagement profile shows a lower level of engagement than the
moderate engagement profile in at least one engagement metric.

There is a large body of work on strategies for encouraging contribution to
online projects. Many of those strategies are discussed by Kraut et al.
(2012). Example of strategies are (i) sending a message to the volunteers
asking them for more contribution; or (ii) providing volunteers online in the
project with specific and highly challenging goals, e.g. executing a number
$n$ of tasks before logoff. One non trivial question that must be answered
before putting a strategy to work is which engagement metrics one wishes to
improve. Discovering the engagement profiles of the volunteers enables finding
out in which engagement metric each profile falls short, and to decide which
strategy to develop focusing on each volunteer profile. The correlations
between the engagement metrics in each engagement profile tell us how other
engagement metrics are affected when strategies are put into practice to
improve one specific engagement metric. They also allow one to assess, for
example, the additional gains that could be obtained from the multiplicative
effects resulting from relationships between various metrics.

Psychological factors behind the engagement profiles. As we discussed early,
some studies have sought to understand the motivation of volunteers to
participate in human computation for citizen science projects Raddick et al.
(2010); Rotman et al. (2012); Jennett et al. (2014). Our results open a new
perspective for such studies. Given that we have shown that volunteers exhibit
different engagement profiles, new studies on the motivation factors can be
conducted considering the engagement peculiarities of each profile. One major
question to be answered in such studies is which motivations may lay behind
each engagement profile. This calls for a more theoretical perspective, for
example: (i) considering self-determination theory Deci and Ryan (2000), are
persistent volunteers more extrinsically motivated than the volunteers who
exhibit other engagement profiles? or (ii) considering self-efficacy theory
Bandura (1977), why do hardworking volunteers expend much effort in the short
term, but do not sustain their engagement in the long term. Besides
complementing our understanding of volunteer engagement, such studies may
provide information about volunteer motivation and experience in the projects.

In the profiles’ analysis, we observe an opposition between degree of
engagement and duration of engagement. Such opposition is clear in two main
points: $1)$ very strong negative correlation between activity ratio and
activity duration in the moderate engagement profile; $2)$ the opposition
between the characteristics of hardworking engagement and persistent
engagement. The negative correlation between activity ratio and activity
duration in the moderate engagement profile indicates that participating in
the project with a high frequency rate and remaining a long time in the
project are contradictory characteristics. It can also be observed in the
opposition between hardworking volunteers and persistent volunteers.
Hardworking volunteers show a higher degree of engagement, but with a shorter
duration. Persistent volunteers, on the contrary, show a lower degree of
engagement but during a longer time period. It is important to understand the
factors behind this opposition and to ask if there are situations in which the
volunteers would present both a high degree and a long duration of engagement.

External validity. Here we discuss about the generality of our study
considering two main aspects: (i) whether the methodology we have proposed to
measure the engagement of volunteers and identify their engagement profiles
can be applied in other projects; and (ii) whether the results obtained in the
case study with data collected from Galaxy Zoo and The Milky Way Project can
be generalised to other human computation for citizen science projects.

The methodology we have proposed is based on theoretical frameworks that
support the study of voluntarism Clary et al. (1998); Wilson (2000) and human
engagement Bandura (1977); O’Brien and Toms (2008). We draw on such frameworks
to derive metrics for measuring the engagement of volunteers and to uncover
engagement profiles from grouping them. In the case study conducted with data
collected from Galaxy Zoo and The Milky Way Project, this methodology shown to
be satisfactory in uncovering groups of volunteers that bring to light the
main similarities and differences among them. Thus, studies seeking such
quantitative analysis of the engagement can take advantage of this
methodology.

Regarding the generality of the engagement profiles, there are two aspects
that reinforce the idea that these types of profiles are more generic and thus
can arise also in other types of projects. First, the same set of profiles
have arisen in projects significantly different in terms of the tasks and the
number of volunteers involved. Tasks in Galaxy Zoo are less time consuming
than tasks in The Milky Way Project Ponciano et al. (2014b). Galaxy Zoo has
almost four times more volunteers than The Milky Way Project (Table 1),
considering as volunteers those participants who have been active in at least
two different days. As most of our results and conclusions are equivalent in
both projects, the differences in the design of the tasks and in the number of
volunteers have been shown not to affect the engagement profiles. Second, some
profiles describe behaviours that are common in Web systems. For example, the
observed fact that a small group of volunteers (persistent engagement) are
responsible for the largest amount of contribution to the project has been
shown to be valid also elsewhere Hargittai and Walejko (2008); van Mierlo
(2014).

##  5 Conclusions and Future Work

In this study we answer three research questions: $1)$ how we can measure the
level of engagement of volunteers during their interaction with a citizen
science project that uses human computation; $2)$ which different patterns of
volunteer engagement behaviour can be identified and specified as typical
volunteer profiles; and $3)$ how the identified volunteer engagement profiles
can be exploited for designing strategies for increasing the engagement of
volunteers in a project. We go through existing human engagement studies and,
based on the concepts and theories put forward, we propose quantitative
engagement metrics to measure different aspects of volunteer engagement, and
use data mining algorithms to identify the different volunteer profiles in
terms of the engagement metrics. We use this method to analyse the engagement
of volunteers in two projects: Galaxy Zoo and The Milky Way Project.

Our results show that volunteers in the studied projects share several
similarities and differences in terms of engagement. We identify five distinct
engagement profiles that put into perspective such similarities and
differences. They are labelled as follows: hardworking, spasmodic, persistent,
lasting, and moderate. These profiles differ among themselves according to a
set of metrics that we have defined for measuring the degree and duration of
volunteer engagement. Regarding the distribution of the volunteers along the
profiles, the highest percentage of volunteers falls into the moderate
engagement profile, while only a few volunteers exhibit a persistent
engagement profile. On the other hand, persistent volunteers account for the
highest percentage of the total human effort dedicated to execute all the
tasks in the project. Several discussions are drawn from our analysis, such as
profile-oriented volunteers’ recruitment, personalised engagement strategies,
and psychological factors behind the engagement profiles.

Our analysis of volunteer engagement, based on log data, yielded a powerful
framework for identifying the relevant patterns of volunteer engagement in
human computation for citizen science projects. However, the current framework
still presents some shortcomings that will be addressed in future work. We
have focused on cognitive engagement of volunteers executing human computation
tasks, but it is known that volunteers also contribute by creating additional
content such as posts in project forums, which can be regarded as a form of
social engagement. Assessing the behaviour of volunteers with regard to this
type of engagement is also important. Finally, future work may be dedicated to
analysing volunteer engagement in the context of other citizen science
projects that use human computation. This analysis may give an answer to the
question whether the set of engagement profiles we have identified on the
basis of the two described projects is generic enough to be applied to the use
of human computation for citizen science projects in general. Thus, we hope
this study motivates further research on volunteer engagement in this type of
projects.

##  6 Acknowledgements

We are indebted to Arfon Smith and Robert Simpson for providing us the dataset
used in this study. We are also grateful to Herman Martins, Jussara Almeida,
Nazareno Andrade, Jose Luis Vivas Frontana and the anonymous reviewers for
their suggestions to improve several aspects of the manuscript. The authors
would like to acknowledge the financial support received from CNPq/Brazil,
CAPES/Brazil, and the European Union Seventh Framework Programme through the
SOCIENTIZE project (contract RI-312902).

## References

  * (1)
  * Anderberg (1973) Anderberg, M. (1973).  Cluster analysis for applications.  Academic Press, Waltham, Massachusetts,United States.
  * Bakker and Demerouti (2008) Bakker, A. B and Demerouti, E. (2008).  Towards a model of work engagement.  Career development international 13, 3 (2008), 209–223.  DOI:http://dx.doi.org/10.1108/13620430810870476
  * Bandura (1977) Bandura, A. (1977).  Self-efficacy: toward a unifying theory of behavioral change.  Psychological review 84, 2 (1977), 191.  DOI:http://dx.doi.org/10.1037/0033-295X.84.2.191
  * Bryant et al. (2005) Bryant, S. L, Forte, A, and Bruckman, A. (2005).  Becoming Wikipedian: Transformation of Participation in a Collaborative Online Encyclopedia. In Proceedings of the 2005 International ACM SIGGROUP Conference on Supporting Group Work. ACM, New York, NY, USA, 1–10.  DOI:http://dx.doi.org/10.1145/1099203.1099205
  * Burke and Kraut (2008) Burke, M and Kraut, R. (2008).  Mopping Up: Modeling Wikipedia Promotion Decisions. In Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work. ACM, New York, NY, USA, 27–36.  DOI:http://dx.doi.org/10.1145/1460563.1460571
  * Butler et al. (2008) Butler, B, Joyce, E, and Pike, J. (2008).  Don’T Look Now, but We’Ve Created a Bureaucracy: The Nature and Roles of Policies and Rules in Wikipedia. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1101–1110.  DOI:http://dx.doi.org/10.1145/1357054.1357227
  * Butler et al. (2002) Butler, B, Sproull, L, Kiesler, S, and Kraut, R. (2002).  Community effort in online groups: Who does the work and why?  In Leadership at a distance: Research in technologically supported work. Taylor & Francis Group, UK, 171–194.
  * Clary et al. (1992) Clary, E. G, Snyder, M, and Ridge, R. (1992).  Volunteers’ motivations: A functional strategy for the recruitment, placement, and retention of volunteers.  Nonprofit Management and Leadership 2, 4 (1992), 333–350.  DOI:http://dx.doi.org/10.1002/nml.4130020403
  * Clary et al. (1998) Clary, E. G, Snyder, M, Ridge, R. D, Copeland, J, Stukas, A. A, Haugen, J, and Miene, P. (1998).  Understanding and assessing the motivations of volunteers: a functional approach.  Journal of personality and social psychology 74, 6 (1998), 1516\.  DOI:http://dx.doi.org/10.1037/0022-3514.74.6.1516
  * Cohn (2008) Cohn, J. P. (2008).  Citizen Science: Can Volunteers Do Real Research?  BioScience 58, 3 (2008), 192–197.  DOI:http://dx.doi.org/10.1641/B580303
  * Cooper et al. (2010) Cooper, S, Khatib, F, Treuille, A, Barbero, J, Lee, J, Beenen, M, Leaver-Fay, A, Baker, D, Popović, Z, and others, . (2010).  Predicting protein structures with a multiplayer online game.  Nature 466, 7307 (2010), 756–760.  DOI:http://dx.doi.org/10.1038/nature09304
  * Corno and Mandinach (1983) Corno, L and Mandinach, E. B. (1983).  The role of cognitive engagement in classroom learning and motivation.  Educational psychologist 18, 2 (1983), 88–108.  DOI:http://dx.doi.org/10.1080/00461528309529266
  * Cravens (2000) Cravens, J. (2000).  Virtual volunteering: Online volunteers providing assistance to human service agencies.  Journal of Technology in Human Services 17, 2-3 (2000), 119–136.  DOI:http://dx.doi.org/10.1300/J017v17n02_02
  * Deci and Ryan (2000) Deci, E. L and Ryan, R. M. (2000).  The "What" and "Why" of Goal Pursuits: Human Needs and the Self-Determination of Behavior.  Psychological Inquiry 11, 4 (2000), 227–268.  DOI:http://dx.doi.org/10.1207/S15327965PLI1104_01
  * Dickinson et al. (2012) Dickinson, J. L, Shirk, J, Bonter, D, Bonney, R, Crain, R. L, Martin, J, Phillips, T, and Purcell, K. (2012).  The current state of citizen science as a tool for ecological research and public engagement.  Frontiers in Ecology and the Environment 10, 6 (2012), 291–297.  DOI:http://dx.doi.org/10.1890/110236
  * Fischer (2001) Fischer, G. (2001).  User Modeling in Human-Computer Interaction.  User Modeling and User-Adapted Interaction 11, 1-2 (2001), 65–86.  DOI:http://dx.doi.org/10.1023/A:1011145532042
  * Fortson et al. (2012) Fortson, L, Masters, K, Nichol, R, Borne, K, Edmondson, E, Lintott, C, Raddick, J, Schawinski, K, and Wallin, J. (2012).  Galaxy Zoo: Morphological Classification and Citizen Science.  In Advances in Machine Learning and Data Mining for Astronomy. CRC Press, Boca Raton, Florida, United States, 213–236.
  * Geiger and Halfaker (2013) Geiger, R. S and Halfaker, A. (2013).  Using edit sessions to measure participation in Wikipedia. In Proceedings of the 2013 conference on Computer supported cooperative work. ACM, New York, NY, USA, 861–870.  DOI:http://dx.doi.org/10.1145/2441776.2441873
  * González-Romá et al. (2006) González-Romá, V, Schaufeli, W. B, Bakker, A. B, and Lloret, S. (2006).  Burnout and work engagement: Independent factors or opposite poles?  Journal of Vocational Behavior 68, 1 (2006), 165–174.  DOI:http://dx.doi.org/10.1016/j.jvb.2005.01.003
  * Goodchild (2007) Goodchild, M. F. (2007).  Citizens as sensors: the world of volunteered geography.  GeoJournal 69, 4 (2007), 211–221.  DOI:http://dx.doi.org/10.1007/s10708-007-9111-y
  * Hargittai and Walejko (2008) Hargittai, E and Walejko, G. (2008).  The participation divide: Content creation and sharing in the digital age.  Information, Communication & Society 11, 2 (2008), 239–256.  DOI:http://dx.doi.org/10.1080/13691180801946150
  * Hertel et al. (2003) Hertel, G, Niedner, S, and Herrmann, S. (2003).  Motivation of software developers in Open Source projects: an Internet-based survey of contributors to the Linux kernel.  Research Policy 32, 7 (2003), 1159 – 1177.  DOI:http://dx.doi.org/10.1016/S0048-7333(03)00047-7
  * Jain (2008) Jain, R. (2008).  The art of computer systems performance analysis.  John Wiley & Sons, Hoboken, New Jersey, US.
  * Jennett et al. (2014) Jennett, C, Blandford, A, Brohan, P, and Cox, A. (2014).  Designing for Dabblers and Deterring Drop-Outs in Citizen Science. In Proceedings of the ACM 2014 Conference on Human Factors in Computing System. ACM, New York, NY, USA, 2985–2994.  DOI:http://dx.doi.org/10.1145/2556288.2557262
  * Kraut et al. (2010) Kraut, R, Maher, M, Olson, J, Malone, T, Pirolli, P, and Thomas, J. (2010).  Scientific Foundations: A Case for Technology- Mediated Social- Participation Theory.  Computer 43, 11 (Nov 2010), 22–28.  DOI:http://dx.doi.org/10.1109/MC.2010.324
  * Kraut et al. (2012) Kraut, R. E, Resnick, P, Kiesler, S, Burke, M, Chen, Y, Kittur, N, Konstan, J, Ren, Y, and Riedl, J. (2012).  Building successful online communities: Evidence-based social design.  Mit Press, Cambridge, Massachusetts, US.
  * Lehmann et al. (2012) Lehmann, J, Lalmas, M, Yom-Tov, E, and Dupret, G. (2012).  Models of User Engagement. In Proceedings of the 20th International Conference on User Modeling, Adaptation, and Personalization. Springer-Verlag, Berlin, Heidelberg, 164–175.  DOI:http://dx.doi.org/10.1007/978-3-642-31454-4_14
  * Lintott and Reed (2013) Lintott, C and Reed, J. (2013).  Human Computation in Citizen Science.  In Handbook of Human Computation. Springer, New York, United States, 153–162.  DOI:http://dx.doi.org/10.1007/978-1-4614-8806-4_14
  * Lintott et al. (2008) Lintott, C. J, Schawinski, K, Slosar, A, Land, K, Bamford, S, Thomas, D, Raddick, M. J, Nichol, R. C, Szalay, A, Andreescu, D, Murray, P, and Vandenberg, J. (2008).  Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey.  Monthly Notices of the Royal Astronomical Society 389, 3 (2008), 1179–1189.  DOI:http://dx.doi.org/10.1111/j.1365-2966.2008.13689.x
  * Liu and Ram (2011) Liu, J and Ram, S. (2011).  Who Does What: Collaboration Patterns in the Wikipedia and Their Impact on Article Quality.  ACM Trans. Manage. Inf. Syst. 2, 2 (2011), 11:1–11:23.  DOI:http://dx.doi.org/10.1145/1985347.1985352
  * López et al. (2012) López, C, Farzan, R, and Brusilovsky, P. (2012).  Personalized incremental users’ engagement: driving contributions one step forward. In Proceedings of the 17th ACM international conference on Supporting group work. ACM, New York, NY, USA, 189–198.  DOI:http://dx.doi.org/10.1145/2389176.2389206
  * Lu et al. (2008) Lu, J, Tang, J, Tang, Z, and Yang, J. (2008).  Hierarchical initialization approach for K-Means clustering.  Pattern Recognition Letters 29, 6 (2008), 787 – 795.  DOI:http://dx.doi.org/10.1016/j.patrec.2007.12.009
  * Luczak-Roesch et al. (2014) Luczak-Roesch, M, Tinati, R, Simperl, E, Van Kleek, M, Shadbolt, N, and Simpson, R. (2014).  Why Won’t Aliens Talk to Us? Content and Community Dynamics in Online Citizen Science. In Eighth International AAAI Conference on Weblogs and Social Media. AAAI, Palo Alto, CA, US, 315–324.  http://www.aaai.org/ocs/index.php/ICWSM/ICWSM14/paper/view/8092
  * Mao et al. (2013) Mao, A, Kamar, E, and Horvitz, E. (2013).  Why Stop Now? Predicting Worker Engagement in Online Crowdsourcing. In Proceedings of the First AAAI Conference on Human Computation and Crowdsourcing. AAAI, Palo Alto, CA, USA, 103–111.
  * McCay-Peet et al. (2012) McCay-Peet, L, Lalmas, M, and Navalpakkam, V. (2012).  On Saliency, Affect and Focused Attention. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 541–550.  DOI:http://dx.doi.org/10.1145/2207676.2207751
  * McPherson et al. (2001) McPherson, M, Smith-Lovin, L, and Cook, J. M. (2001).  Birds of a Feather: Homophily in Social Networks.  Annual Review of Sociology 27, 1 (2001), 415–444.  DOI:http://dx.doi.org/10.1146/annurev.soc.27.1.415
  * Meece et al. (1988) Meece, J. L, Blumenfeld, P. C, and Hoyle, R. H. (1988).  Students’ goal orientations and cognitive engagement in classroom activities.  Journal of educational psychology 80, 4 (1988), 514.  DOI:http://dx.doi.org/10.1037/0022-0663.80.4.514
  * Mehrzadi and Feitelson (2012) Mehrzadi, D and Feitelson, D. G. (2012).  On extracting session data from activity logs. In Proceedings of the 5th Annual International Systems and Storage Conference. ACM, New York, NY, USA, 3:1–3:7.  DOI:http://dx.doi.org/10.1145/2367589.2367592
  * Millen and Patterson (2002) Millen, D. R and Patterson, J. F. (2002).  Stimulating social engagement in a community network. In Proceedings of the 2002 ACM conference on Computer supported cooperative work. ACM, New York, NY, USA, 306–313.  DOI:http://dx.doi.org/10.1145/587078.587121
  * Niederer and Van Dijck (2010) Niederer, S and Van Dijck, J. (2010).  Wisdom of the crowd or technicity of content? Wikipedia as a sociotechnical system.  New Media & Society 12, 8 (2010), 1368–1387.  DOI:http://dx.doi.org/10.1177/1461444810365297
  * Nov et al. (2014) Nov, O, Arazy, O, and Anderson, D. (2014).  Scientists@ Home: what drives the quantity and quality of online citizen science participation?  PloS one 9, 4 (2014), e90375.  DOI:http://dx.doi.org/10.1371/journal.pone.0090375
  * O’Brien and Toms (2008) O’Brien, H. L and Toms, E. G. (2008).  What is user engagement? A conceptual framework for defining user engagement with technology.  Journal of the American Society for Information Science and Technology 59, 6 (2008), 938–955.  DOI:http://dx.doi.org/10.1002/asi.20801
  * O’Brien and Toms (2010) O’Brien, H. L and Toms, E. G. (2010).  The development and evaluation of a survey to measure user engagement.  Journal of the American Society for Information Science and Technology 61, 1 (2010), 50–69.  DOI:http://dx.doi.org/10.1002/asi.21229
  * Ponciano et al. (2014b) Ponciano, L, Brasileiro, F, Simpson, R, and Smith, A. (2014)b.  Volunteers’ Engagement in Human Computation for Astronomy Projects.  Computing in Science and Engineering 1, 1 (2014).  DOI:http://dx.doi.org/10.1109/MCSE.2014.4
  * Ponciano et al. (2014a) Ponciano, L, Brasileiro, F. V, Andrade, N, and Sampaio, L. M. R. (2014)a.  Considering human aspects on strategies for designing and managing distributed human computation.  J. Internet Services and Applications 5, 1 (2014).  DOI:http://dx.doi.org/10.1186/s13174-014-0010-4
  * Porges (2003) Porges, S. W. (2003).  Social Engagement and Attachment.  Annals of the New York Academy of Sciences 1008, 1 (2003), 31–47.  DOI:http://dx.doi.org/10.1196/annals.1301.004
  * Preece (2000) Preece, J. (2000).  Online Communities: Designing Usability and Supporting Socialbilty (1st ed.).  John Wiley & Sons, Inc., New York, NY, USA.
  * Preece and Shneiderman (2009) Preece, J and Shneiderman, B. (2009).  The reader-to-leader framework: Motivating technology-mediated social participation.  Transactions on Human-Computer Interaction 1, 1 (2009), 13–32.  http://aisel.aisnet.org/thci/vol1/iss1/5
  * Quinn and Bederson (2011) Quinn, A. J and Bederson, B. B. (2011).  Human computation: a survey and taxonomy of a growing field. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1403 – 1412.  DOI:http://dx.doi.org/10.1145/1978942.1979148
  * Raddick et al. (2010) Raddick, J, Bracey, G, Gay, P. L, Lintott, C. J, Murray, P, Schawinski, K, Szalay, A. S, and Vandenberg, J. (2010).  Galaxy zoo: Exploring the motivations of citizen science volunteers.  Astronomy Education Review 9, 1 (2010), 010103.  DOI:http://dx.doi.org/10.3847/AER2009036
  * Roberts et al. (2006) Roberts, J. A, Hann, I.-H, and Slaughter, S. A. (2006).  Understanding the Motivations, Participation, and Performance of Open Source Software Developers: A Longitudinal Study of the Apache Projects.  Manage. Sci. 52, 7 (July 2006), 984–999.  DOI:http://dx.doi.org/10.1287/mnsc.1060.0554
  * Robson et al. (2013) Robson, C, Hearst, M, Kau, C, and Pierce, J. (2013).  Comparing the Use of Social Networking and Traditional Media Channels for Promoting Citizen Science. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work. ACM, New York, NY, USA, 1463–1468.  DOI:http://dx.doi.org/10.1145/2441776.2441941
  * Rotman et al. (2014) Rotman, D, Hammock, J, Preece, J, Hansen, D, Boston, C, Bowser, A, and He, Y. (2014).  Motivations Affecting Initial and Long-Term Participation in Citizen Science Projects in Three Countries. In iConference. iSchools, Illinois, US, 110–124.  DOI:http://dx.doi.org/10.9776/14054
  * Rotman et al. (2012) Rotman, D, Preece, J, Hammock, J, Procita, K, Hansen, D, Parr, C, Lewis, D, and Jacobs, D. (2012).  Dynamic changes in motivation in collaborative citizen-science projects. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, New York, NY, USA, 217–226.  DOI:http://dx.doi.org/10.1145/2145204.2145238
  * Rousseeuw (1987) Rousseeuw, P. J. (1987).  Silhouettes: a graphical aid to the interpretation and validation of cluster analysis.  Journal of computational and applied mathematics 20 (1987), 53–65.  DOI:http://dx.doi.org/10.1016/0377-0427(87)90125-7
  * Schroer and Hertel (2009) Schroer, J and Hertel, G. (2009).  Voluntary Engagement in an Open Web-Based Encyclopedia: Wikipedians and Why They Do It.  Media Psychology 12, 1 (2009), 96–120.  DOI:http://dx.doi.org/10.1080/15213260802669466
  * Simpson (2009) Simpson, M. R. (2009).  Engagement at work: A review of the literature.  International Journal of Nursing Studies 46, 7 (2009), 1012–1024.  DOI:http://dx.doi.org/10.1016/j.ijnurstu.2008.05.003
  * Simpson et al. (2012) Simpson, R, Povich, M, Kendrew, S, Lintott, C, Bressert, E, Arvidsson, K, Cyganowski, C, Maddison, S, Schawinski, K, Sherman, R, and others, . (2012).  The milky way project first data release: a bubblier galactic disc.  Monthly Notices of the Royal Astronomical Society 424, 4 (2012), 2442–2460.  DOI:http://dx.doi.org/10.1111/j.1365-2966.2012.20770.x
  * Struyf et al. (1997) Struyf, A, Hubert, M, and Rousseeuw, P. (1997).  Clustering in an Object-Oriented Environment.  Journal of Statistical Software 1, 4 (10 2 1997), 1–30.  http://www.jstatsoft.org/v01/i04
  * van Mierlo (2014) van Mierlo, T. (2014).  The 1% Rule in Four Digital Health Social Networks: An Observational Study.  J Med Internet Res 16, 2 (04 Feb 2014), e33.  DOI:http://dx.doi.org/10.2196/jmir.2966
  * Welser et al. (2011) Welser, H. T, Cosley, D, Kossinets, G, Lin, A, Dokshin, F, Gay, G, and Smith, M. (2011).  Finding Social Roles in Wikipedia. In Proceedings of the 2011 iConference. ACM, New York, NY, USA, 122–129.  DOI:http://dx.doi.org/10.1145/1940761.1940778
  * Wiggins and Crowston (2012) Wiggins, A and Crowston, K. (2012).  Goals and Tasks: Two Typologies of Citizen Science Projects. In Proceedings of the 45th Hawaii International Conference on System Sciences. IEEE Computer Society, Los Alamitos, CA, USA, 3426–3435.  DOI:http://dx.doi.org/10.1109/HICSS.2012.295
  * Wilson (2000) Wilson, J. (2000).  Volunteering.  Annual review of sociology 26, 1 (2000), 215–240.  DOI:http://dx.doi.org/10.1146/annurev.soc.26.1.215
  * Zhu et al. (2012) Zhu, H, Kraut, R, and Kittur, A. (2012).  Effectiveness of Shared Leadership in Online Communities. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work. ACM, New York, NY, USA, 407–416.  DOI:http://dx.doi.org/10.1145/2145204.2145269