# A Needle in a Haystack:
An Analysis of High-Agreement Workers on MTurk for Summarization

Lining Zhang,1* Simon Mille,2 Yufang Hou,3 Daniel Deutsch,4 Elizabeth Clark,5
Yixin Liu,6 Saad Mahamood,7 Sebastian Gehrmann,5 Miruna Clinciu,8 Khyathi
Chandu,9 João Sedoc1 1New York University, 2ADAPT Centre, DCU, 3IBM Research,
4Google, 5Google Research, 6Yale University, 7trivago N.V., 8University of
Edinburgh, 9Allen Institute for AI

###### Abstract

To prevent the costly and inefficient use of resources on low-quality
annotations, we want a method for creating a pool of dependable annotators who
can effectively complete difficult tasks, such as evaluating automatic
summarization. Thus, we investigate the recruitment of high-quality Amazon
Mechanical Turk workers via a two-step pipeline. We show that we can
successfully filter out subpar workers before they carry out the evaluations
and obtain high-agreement annotations with similar constraints on resources.
Although our workers demonstrate a strong consensus among themselves and
CloudResearch workers, their alignment with expert judgments on a subset of
the data is not as expected and needs further training in correctness. This
paper still serves as a best practice for the recruitment of qualified
annotators in other challenging annotation tasks.

††* Correspondence to<EMAIL_ADDRESS>
##  1 Introduction

Natural language generation (NLG) tasks like text summarization are
challenging to evaluate both in terms of automatic metrics and human
evaluations (Gehrmann et al., 2022). Although automatic metrics are
inexpensive proxies for human annotations for tasks like dialog evaluation
(Mehri et al., 2022), they may have problems dealing with paraphrases,
capturing distant dependencies, or identifying nuances in human languages
(Banerjee and Lavie, 2005; Isozaki et al., 2010; Manning et al., 2020). Thus,
it is still crucial to obtain high-quality human annotations as gold labels
for evaluation. Amazon Mechanical Turk (MTurk)111https://www.mturk.com/ is a
commonly used crowdsourcing platform for collecting human annotations on
designed tasks, known as Human Intelligence Tasks (HITs). However, finding
qualified workers for high-quality annotations with a better inter-annotator
agreement (IAA) is challenging, especially for difficult tasks such as text
summarization. Best practices for recruiting high-quality workers are also
poorly understood, and the relationship between high quality and high
agreement needs further investigation.

Figure 1: Two-step pipeline for finding high-agreement MTurk workers:
participants who satisfy basic qualification settings and answer designed
questions correctly (Qualification) are subsequently filtered in a longer task
(Endurance). The maintained worker list is tested for the true annotation task
later (Reference-based).

To tackle the above issues, we design a recruitment pipeline to identify
workers who are able to produce high-agreement annotations for the evaluation
of text summarization on MTurk. It comprises a qualification task and an
endurance task, followed by a reference-based task (see Figure 1). In the
qualification task, workers who meet pre-defined qualification settings
receive instructions and qualification questions, including an attention check
(Oppenheimer et al., 2009). The qualification questions are designed to assess
the annotator’s ability to evaluate multiple dimensions of a summary
correctly. Performance on this task determines whether they are categorized
into gold, silver, bronze, or block. Only the best workers (gold and silver)
move on to the endurance task, which consists of 10 HITs with 4 summaries in
each to evaluate. This task only tests the summary’s saliency, which is the
most subjective dimension (Howcroft et al., 2020), but it challenges the
annotator’s capacity for handling a heavy annotation workload. gold and silver
workers who complete all HITs are added to a maintained worker list as high-
agreement annotators for future tasks. To ensure their general performance for
the true annotation task, a reference-based task to evaluate information
coverage between summaries is conducted with these workers later.

While serving as a best practice beyond its scope, our study has the following
contributions:

  * •

establish a cost-effective recruitment pipeline on MTurk to consistently build
a pool of annotators for high-agreement annotations.

  * •

successfully recruit 12 out of 200 (6%) superior annotators for text
summarization evaluation, while reducing costs and guaranteeing high
agreement.

  * •

rigorously demonstrate that the annotators identified through our pipeline can
match or surpass the IAA of expert annotators and standard statistical
techniques, though further calibration may be required for correctness.

##  2 Related Work

Challenges of Human Evaluation Compared to automatic evaluation metrics for
NLG tasks like BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), human
annotations from non-expert annotators on MTurk can reach an agreement with
gold standards or expert judgments (Callison-Burch, 2009). Although recent
works leverage language models like BERT (Devlin et al., 2019) to get better
automatic evaluations (Zhang et al., 2020), human judgments are still
indispensable in identifying nuances in specific language tasks (Manning et
al., 2020). Finding qualified workers to carry out the evaluations is crucial.
This is especially true for tasks like text summarization, which lacks
consensus on evaluation protocols (Fabbri et al., 2021) and is often
inconsistent with previous human evaluations (Hardy et al., 2019). However,
human evaluation from non-expert crowdsourcing platforms have low quality
(Gillick and Liu, 2010) and a simple qualification filter is not sufficient to
identify qualified workers (Berinsky et al., 2012; Robinson et al., 2019).
Some studies applied quality control mechanisms to filter out poor quality
annotations, resulting in a relatively low pass rate for a variety of tasks
(Graham et al., 2017, 2018; Mille et al., 2019). The fact that up to 70% of
the HITs are eventually discarded indicates a huge resource waste.

Even with qualified workers, human annotations might still be adversely
affected by factors like incomplete instructions or unfair wages paid to
annotators (Huynh et al., 2021), and workers need clear references, schemes,
or standards to follow (Howcroft et al., 2020; Karpinska et al., 2021). Thus,
our study serves as a detailed reference for finding qualified MTurk workers
for a summarization evaluation task and further identifying those who can
assist in a large number of annotations.

Inter-Annotator Agreement For annotations without true labels or those
evaluated with a qualitative scale such as Likert scale (Likert, 1932), the
inter-annotator agreement (IAA) among MTurk workers measures the reliability
of the annotations. For example, Cohen’s Kappa (Cohen, 1960) measures IAA
between a pair of results of the same length from two annotators, while
Krippendorff’s Alpha (Hayes and Krippendorff, 2007) measures the agreement of
a set of results from any number of annotators, even with unequal sample
sizes. Both range from $-1$ to $1$, with $1$ indicating complete agreement.
Further studies also continue to mitigate annotator bias through complementary
methods to IAA (Amidei et al., 2020), aimed at high-quality annotations. In
our study, we utilize both Cohen’s Kappa and Krippendorff’s Alpha as the
measurement of annotation reliability.

##  3 Methods

In this section, we detail how the workers were recruited and which tasks were
carried out.222Appendix A.9 shows instructions given during the tasks.

###  3.1 MTurk Qualification Settings

To narrow down the pool of our target workers, we set a few pre-defined
qualifications for workers on MTurk before publishing the qualification task:
(i) the Location is set to UNITED STATES (US); (ii) the Number of HITs
Approved is set to be greater than 1000 to target workers who are already
experienced on MTurk; (iii) the HIT Approval Rate (%) is set to be greater
than or equal to 99 to target workers who are able to finish tasks with high
quality and have stable performance. We also set the task visibility as
Private, which means our tasks are visible to any worker, but only workers who
meet all qualification requirements can preview and accept.

Paolacci et al. (2010) show that the annotations collected with the Location
setting on MTurk are representative of the population of our target country in
terms of demographic data. This helps mitigate biases introduced by samples
from traditional recruitment methods like college undergraduate samples
(Buhrmester et al., 2011). We set qualification settings (ii) and (iii) based
on previous work (Whiting et al., 2019; Oppenlaender et al., 2020; Kummerfeld,
2021) and our own experience on MTurk. Workers who meet all qualification
requirements are eligible to participate in the qualification task.

###  3.2 Qualification Task

Summarization task In summarization, the input is the text of a document and
the output is a short summary. We evaluate a summary S according to 6
dimensions based on the criteria taxonomy presented in Howcroft et al. (2020),
and workers are asked for a binary answer as to whether a dimension is
satisfied in a summary or not:

  * •

Understandability: can the worker understand S and is S worth being annotated.

  * •

Compactness: S does not contain duplicated information.

  * •

Grammaticality: S is free from grammatical & spelling errors.

  * •

Coherence: S is presented in a clear, well-structured, logical, and meaningful
way.

  * •

Faithfulness: all of the information in S can be found in the article; S
accurately reflects the contents of the article.

  * •

Saliency: S captures the most important information of the article and does
not include parts of the article that are less important.

Training and qualification There are two main parts of the qualification task.
The _training part_ guides the workers through the above evaluation dimensions
and instructs them on how to annotate. The definition of each dimension is
illustrated with positive and negative examples, and full annotation examples
are shown (summary and binary rating for each dimension). Then, workers are
required to write an instruction summary in their own words to make sure they
have understood the task and are ready to annotate. The _qualification part_
tests the worker’s understanding of the task. Three documents are provided,
each with one summary. The worker reads the document and annotates the
corresponding summary according to each dimension. The ratings are then
compared to expert ratings provided by the authors of this paper. The last
document comes with an attention check to test whether a worker is just
randomly assigning scores without reading: a highlighted instruction asks the
worker to ignore the task and select specific answers. Finally, an optional
field is provided to collect feedback.

Worker categorization Upon finishing their task, workers are categorized into
four types:

  * •

GOLD. The gold workers pass the attention check and annotate every dimension
of every document in the qualification part correctly.

  * •

SILVER. The silver workers pass the attention check and make only one mistake
when annotating each dimension of the documents in the qualification part.

  * •

BRONZE. The bronze workers pass the attention check and make more than one
mistake when annotating each dimension of the documents in the qualification
part.

  * •

BLOCK. The block workers fail to pass the attention check.

The gold and silver workers are assigned a qualification score and proceed
with the endurance task. Besides, we conducted multiple rounds of the
qualification task to avoid influence from the time or day when the task was
conducted and randomly sampled workers (Arechar et al., 2017; Berinsky et al.,
2012).

Round Number | 1 | 2 | 3 | 4 | Total
---|---|---|---|---|---
Total participants at the beginning | 50 | 50 | 50 | 50 | 200
# gold workers passed qualification task | 1 | 3 | 2 | 2 | 8
# silver workers passed qualification task | 4 | 5 | 3 | 6 | 18
# workers entered endurance task | 5 | 8 | 5 | 8 | 26
# gold workers passed endurance task | 1 | 1 | 1 | 1 | 4
# silver workers passed endurance task | 0 | 3 | 2 | 3 | 8
# workers passed both tasks | 1 | 4 | 3 | 4 | 12

Table 1: Number of MTurk workers qualified after each task.

###  3.3 Endurance Task

The endurance task is designed to test whether a worker can reliably perform a
large number of annotations. The workers who finish all HITs of this task are
assigned the highest qualification score and are added to a maintained worker
list.

The endurance task comprises 10 HITs. For each HIT, a document and 4
corresponding summaries generated by different models are provided; each HIT
takes around 5 minutes to finish (approximately an hour for all HITs). To keep
the task simple we only evaluate each summary on one dimension, but to ensure
that the task is challenging enough we (i) use the most subjective of the 6
dimensions, Saliency, and (ii) use a more fine-grained 10-point Likert scale
(from 1 to 10).

Rationale for choosing 10 HITs Our motivation is two-fold: to find workers who
were able to complete many tasks and whose annotations are better than random.
As the number of HITs increases, the number of remaining workers drops from 26
to 12. The survival rate defined by the Kaplan–Meier estimator (Kaplan and
Meier, 1958) is 38.59% when the number of HITs is set to 10 which is an
estimate of a worker’s capacity to be able to complete many tasks. We
empirically found that we need a minimum of 8 HITs completed by a worker in
order to validate that their annotations are statistically significantly
different from random noise (see Table 2).

Num. of HITs finished |  Num. of workers remaining |  Survival rate % (Kaplan–Meier estimator) |  Confidence interval of Cohen’s Kappa
---|---|---|---
Lower bound |  Upper bound
- | 26[1] | 100 | -
1 | 19 | 63.16 | -
2 | 18 | 59.65 | -
3 | 17 | 56.14 | -
4 | 16 | 52.63 | -
5 | 15 | 49.12 | -0.18 | 0.44
6 | 15 | 49.12 | -0.18 | 0.44
7 | 15 | 49.12 | -0.18 | 0.44
8 | 14 | 45.61 | 0.06 | 0.44
9 | 13 | 42.10 | 0.08 | 0.42
10 | 12 | 38.59 | 0.09 | 0.42

  * •

$[1]$ This (26) is the number of workers who entered the endurance task (GOLD
and SILVER workers passed the qualification task).

Table 2: Statistical results as number of HITs grows.

###  3.4 Reference-based Task

Finally, to test whether the selected MTurk workers actually perform better at
annotating summaries in general, we conduct a reference-based task that
comprises 30 HITs. In each HIT, a reference summary and 4 candidate summaries
are provided. The worker is asked to assign each candidate summary two scores
(can2ref score and ref2can score) on a scale from 1 to 5. The can2ref score
indicates whether all of the information in the candidate summary can also be
found in the reference summary, while the ref2can score checks the converse
coverage direction. A score of 1 means that almost no information in one
summary can be found in the other, while a score of 5 indicates complete
information coverage. The worker is provided with instructions and examples of
the rating at the beginning of the task.

##  4 Results

###  4.1 Annotation Data and Cost

The collected experimental data not only contained annotation results but also
metadata reflecting annotator behaviors.333The data and code used for the
analysis of all tasks are available at https://github.com/GEM-
benchmark/MTurkRequirementPipeline. The cost of annotation on MTurk included
both the wages paid to MTurk Workers and the fees paid to MTurk (which may
vary according to the task). A worker who participated in the qualification
and the endurance tasks earned $8.5 ($1 for the qualification task plus $7.5
for the endurance task) on average, while a worker who participated only in
the qualification task (i.e. who did not qualify) earned $1 on average. Given
the total cost of $514 for the entire pipeline which yielded 12 workers, the
cost of identifying a qualified worker is $42.8. For details, the breakdown of
the cost is shown in Table 3.

Annotation Task |  Reward per Assignment |  Num. of Assignment per Task |  Total Reward |  Fees to MTurk |  Total Cost |  Hourly Wage
---|---|---|---|---|---|---
Qualification Task (Each of 4 rounds) | $1.00 | 50 | $50 | $20 | $70 | $2
Endurance Task | Round 1 | $0.75 | 5 | $37.5 | $7.5 | $45 | $7.5
Round 2 | $0.75 | 8 | $60 | $12 | $72 | $7.5
Round 3 | $0.75 | 5 | $37.5 | $7.5 | $45 | $7.5
Round 4 | $0.75 | 8 | $60 | $12 | $72 | $7.5
Table 3: Wage Paid to MTurk Workers and total amount spent on annotation. The
number of assignment per task indicates the number of workers who entered the
task, which is not equal to the number of workers who passed the task. The
hourly wage is calculated for one MTurk worker given a task.

###  4.2 Qualification Task Results

We conducted four rounds of the qualification task, each round included 50
MTurk workers (see Table 1). This choice of multiple rounds aimed to guarantee
the stability of the annotation results (Berinsky et al., 2012; Arechar et
al., 2017). The overall pass rate of the attention check was 0.69; thus, 62
workers in total did not pass the attention check and were categorized as
block. Out of 200 MTurk workers, there were only 8 gold workers and 18 silver
after the qualification task. Thus, only 26 MTurk workers (13% of all
participants) qualified for the endurance task.

For each round, we calculated Krippendorff’s
Alpha444https://pypi.org/project/krippendorff/ to measure the agreement among
annotators. The highest Krippendorff’s Alpha was 0.33 reached by the first
round, and the average Krippendorff’s Alpha of all four rounds was 0.25. In
addition, the exclusion of block workers led to an increase in Krippendorff’s
Alpha, compared to the value calculated on all workers. The highest
Krippendorff’s Alpha without block workers was 0.44 (second round), and the
average Krippendorff’s Alpha of all four rounds increased to 0.41. These
results showed that, as expected, block workers seemed to lack good-faith
effort in the task and likely yielded low quality annotations.

###  4.3 Endurance Task Results

We published the same endurance task for gold and silver workers separately,
and reported IAA using Cohen’s Kappa and Krippendorff’s Alpha among each type
of worker; we also reported similar IAA results from combined gold and silver
workers. We additionally collected endurance task results from volunteer
researchers unrelated to this paper for a comparison between MTurk workers and
NLG experts.

SILVER Workers There were 18 silver workers after the qualification task, 13
of whom accepted the endurance task. However, only 8 silver workers finished
all 10 HITs–a yield rate of around 44% given the number of silver workers
entering this task. To calculate the IAA, we considered the annotation scores
of all summaries (40 ratings) for each of the 8 workers and calculated Cohen’s
Kappa for each worker pair; the highest Cohen’s Kappa was 0.451 between
workers $S_{22}$ and $S_{43}$. To avoid influence from a possible unstable
performance at the beginning of the task, we also tried to omit the first two
HITs, that is, we only used 32 ratings when calculating Cohen’s Kappa; the
resulting improvement for Cohen’s Kappa was very low. In addition, we
calculated Krippendorff’s Alpha on the entire annotation results for all
summaries and workers, and it reached 0.358.

GOLD Workers There were 8 gold workers after the qualification task and 6 of
them accepted the endurance task. However, only 4 gold workers finished all 10
HITs, for a yield rate of around 67% given the number of gold workers entering
this task. This rate was higher than that of silver workers. We calculated
pairwise Cohen’s Kappa using all the scores, and the highest IAA score
increased to 0.48, compared to 0.45 for silver workers. There was no
significant improvement after omitting the first two HITs. Krippendorff’s
Alpha for the gold workers reached 0.443, which is higher than with silver
workers (0.358).

GOLD and SILVER Workers To investigate IAA of worker pairs across gold and
silver workers, we combined the results of these two categories of workers and
calculated pairwise Cohen’s Kappa. The highest pairwise Cohen’s Kappa on the
40 ratings per worker was 0.55; see the matrix in Figure 2. Again, omitting
the first two HITs also did not change the scores much. For Krippendorff’s
Alpha, the value was 0.396, which fell in the range between the silver
worker’s (0.358) and gold worker’s (0.443) values.555Note that the relatively
low Krippendorff’s Alpha scores may in part be due to the large size of the
scale (10 points).

In Appendix A.2, we show a breakdown of the results per text position in each
HIT (correlations for all first texts, for all second texts, etc.) for each of
the three subgroups (silver, gold, gold and silver); the possibly sightly
darker heat maps could indicate higher correlations for the second text of
each HIT.

Figure 2: Cohen’s Kappa for endurance task (grey frame: silver workers;
yellow: gold workers).

Figure 3: Comparison of online behaviors between the abnormal worker
($S_{42}$, right) and the regular worker (left).

Comparison to Expert Ratings To get an idea of the quality of qualified MTurk
workers according to our approach, we compared their IAA with the IAA obtained
by conducting the same endurance task with three researchers as NLG experts.
The pairwise Cohen’s Kappa for all 40 ratings only reached 0.268 (see Table 10
in Appendix A.3). The IAA among the experts was comparatively lower than the
gold and silver workers, indicating that qualified workers identified by our
tasks reached a better agreement at least for the endurance task. Thus, it
seems possible to recruit high-quality workers using our pipeline.

Detection of Abnormal Workers From Cohen’s Kappa scores shown in Figure 2, the
worker $S_{42}$666$S_{42}$ stands for the second silver worker from Round 4
had much lower agreement scores (heatmap in the yellow colors on the row and
column corresponding to the worker). Recent studies have uncovered the
presence of bots on MTurk (Webb and Tangney, 2022). To understand the reason
for this worker’s lower agreement with other workers, we analyzed their online
behavior using the metadata extracted from their annotation results.

Figure 3 shows the timeline of each of the 10 HITs as a horizontal gray line.
The timelines are plotted from top to bottom, corresponding to the first to
the last HIT in the endurance task. The X-axis represents the duration between
the time of acceptance and submission, which is normalized by the duration for
each HIT (ranging from 0 to 1). Different marks present each annotator
behavior, as shown in the legend. Among these behaviors, blue points represent
the time when the MTurk worker assigned a score for one of the four summaries,
and the corresponding number on top represents the summary index (valued from
0 to 3). Orange crosses denote the suggested reading time of the article in
each HIT, given the average human reading speed of 130 words per
minute.777https://wordstotime.com/ If the suggested reading time after
normalization was longer than the duration, we marked the orange cross as 1 at
the time of submission which is at the end of the gray line.

Most of the orange crosses were marked at the end of the timelines in Figure 3
(right), indicating this worker assigned scores and submitted the HIT in less
time than it usually takes for a human to even finish reading the article.
This result demonstrates that this worker may not have put in good faith in
the endurance task, which possibly explains the low IAA with other workers. By
removing this worker and calculating Krippendorff’s Alpha again within gold
and silver workers, the IAA increased to 0.454 (compared to 0.396 when
including the worker).

###  4.4 Reference-based Task Results

To test the reliability of our qualified workers and compare them to workers
who do not undergo our selection process, we launched the reference-based task
(see Section 3.4), which is open to our qualified workers as well as to any
other workers satisfying basic qualification settings.

Qualified Workers after Pipeline We published the reference-based task to the
12 MTurk workers from four rounds who have passed both the qualification and
the endurance task. All 12 workers accepted this task but only 8 workers
finished 30 HITs within a week.

There are two scores to evaluate the information coverage between each
candidate summary and the reference summary. We use the can2ref score to
represent whether all information in the candidate summary can be found in the
reference summary, and the ref2can score to represent the converse coverage.
For both types of scores, we calculated Cohen’s Kappa for every worker pair
(given 4 candidate summaries per HIT, 30 HITS per worker). Cohen’s Kappa for
can2ref score ranges from 0.15 to 0.71, with a relatively high IAA between the
first gold workers from the first two rounds ($G_{11}$ and $G_{21}$).
Similarly, Cohen’s Kappa for ref2can score ranges from 0.14 to 0.66. Finally,
Cohen’s Kappa for the combined scores ranges from 0.15 to 0.68 (see Figure 4),
demonstrating that the agreement numbers are stable across multiple measures.
Krippendorff’s Alpha for the above scenarios (can2ref score, ref2can score,
and combined) are 0.558, 0.508, and 0.534.

Figure 4: Cohen’s Kappa for reference-based task (grey frame: silver workers;
yellow: gold workers).

Baseline MTurk Workers For comparison, we published the same reference-based
task to MTurk workers who did not participate in our previous experiments. 276
MTurk workers participated and each worker finished on average 2 HITs (In
total 30 HITs $\times$ 20 Assignments/HIT). Krippendorff’s Alpha for can2ref,
ref2can, and the two combined were extremely low, at 0.087, 0.077, and 0.080
respectively, demonstrating the necessity of a high-quality recruitment
pipeline. We experimented with the following approaches to investigate whether
we could increase the agreement between random MTurk workers to a level
comparable to qualified workers from our pipeline.

IAA with Median

Among the 20 assignments of each HIT, we randomly divided the workers into 4
groups of 5 workers and took the median of each group representing a new
worker (Lau et al., 2014). Then, we concatenated the results of 20 HITs for
the 4 new workers to calculate IAA. Krippendorff’s Alpha scores increased to
0.191, 0.185, and 0.188 respectively.

Filter on Timing and Number of Finished HITs

To exclude unqualified workers whose annotations may decrease IAA, only
workers who (i) spent more than the suggested reading time888We performed the
same timing analysis as in Section 4.3. and (ii) finished 3 or more HITs were
selected for calculation of IAA. This resulted in 25 workers remaining, but
Krippendorff’s Alpha remained almost the same as calculated without the
filter.

Statistical Filter (MACE)

We applied the Multi-Annotator Competence Estimation (MACE) (Hovy et al.,
2013; Paun et al., 2018) to identify reliable workers based on competence
scores calculated on annotations. The workers with competence scores above a
threshold were kept. We additionally calculated Spearman’s coefficient
(Spearman, 1904) within the groups of our pipeline and MACE (see Table 4). We
report the results of additional failed attempts to improve Spearman’s
coefficient across these two groups, in Table 12 in the Appendix.

Threshold | 0.5 | 0.6 | 0.7
---|---|---|---
% of workers kept | 19.2% | 15.9% | 7.6%
HIT coverage | 30/30 | 27/30 | 18/30
Avg. num. workers per HIT | 2.4 | 1.9 | 1.2
Krippendorff’s Alpha (all scores) | 0.380 | 0.472 | 0.754
Spearman’s coefficient (MACE workers) | 0.351 | 0.414 | 0.770
Spearman’s coefficient (pipeline workers) | 0.558 | 0.565 | 0.577

Table 4: IAA for different thresholds of MACE.

In summary, the most effective methods to improve agreement numbers among
random workers were median grouping and MACE. IAA on median scores can raise
Krippendorff’s Alpha to almost 0.2. MACE increases Krippendorff’s Alpha as the
threshold increases, but at the cost of an incomplete HIT coverage (27/30 and
18/30 respectively for the threshold of 0.6 and 0.7 in Table 4) and fewer
workers per HIT (1.9 and 1.2, respectively, for the threshold of 0.6 and 0.7
in Table 4). Similarly, Spearman’s coefficient of MACE workers can be
increased above our pipeline workers’ only at the same expense as above.

CloudResearch MTurk Workers To further test our pipeline, we conducted the
same reference-based task on the CloudResearch platform (cloudresearch.com),
which helps researchers recruit high-quality annotators. We recruited the same
number (eight) of CloudResearch workers as our pipeline. The Krippendorff’s
Alpha and Cohen’s Kappa999The range of Cohen’s Kappa is slightly smaller for
CloudResearch workers. for CloudResearch workers is slightly lower than our
pipeline workers (see Table 5 and Figure 9). Additionally, we found that our
pipeline workers have a higher task acceptance rate. This results in a shorter
experimental period compared to the task conducted on CloudResearch.

Worker Source |  IAA Metric | can2ref | ref2can |  combined score
---|---|---|---|---
Pipeline | CK | 0.15-0.71 | 0.14-0.66 | 0.15-0.68
KA | 0.558 | 0.508 | 0.534
Cloud Research | CK | 0.18-0.60 | 0.19-0.61 | 0.18-0.60
KA | 0.527 | 0.498 | 0.513

Table 5: The range of Cohen’s Kappa (CK) and Krippendorff’s Alpha (KA) of
pipeline and CloudResearch workers for reference-based task.

Analysis of Correctness Across Annotation Sources We randomly sampled 50
annotation questions from the reference-based task to test correctness, which
is defined as the alignment with expert judgments.101010Fifty random samples
were chosen in order to differentiate between MACE and pipeline assuming 20%
superiority in terms of correctness. In addition, we also compared the expert
judgment with scores generated by GPT models: GPT-3.5 (text-davinci-003) and
ChatGPT which are built on InstructGPT (Ouyang et al., 2022), and GPT-4
(OpenAI, 2023). Scores are aggregated by taking the median within groups of
pipeline, MACE, and CloudResearch workers, as well as experts.111111We use the
median of a group of experts as the expert judgment, which has Krippendorff’s
Alpha of 0.52. For ChatGPT we ran inference 5 times with default parameters
(temperature=1, top_p=1) and took the median. To obtain GPT-3.5 and GPT-4
scores temperature was set to 0 with a single run.

We did not find that pipeline workers were superior to MACE workers in terms
of correctness. Pipeline and CloudResearch workers had a significant
Spearman’s correlation with each other (see Figure 5), which indicates a
reproduction of the recruitment procedure on CloudResearch at a lower cost.
However, the confidence intervals are too wide to draw any conclusion about
the correlation between crowd annotators and expert judgments (see Table 6).
This indicates that the pipeline may not guarantee the training of the
correctness of annotations. However, we found that GPT models correlated well
with expert judgments. Further details can be found in Appendix A.7 and A.8.

Class | Group Type |  Spearman’s Coefficient |  95% Confidence Interval
---|---|---|---
Crowd Annotators | Pipeline | 0.03 | (-0.61, 0.65)
MACE | 0.10 | (-0.56, 0.69)
CloudResearch | 0.08 | (-0.58, 0.67)
GPT models | GPT-3.5 | 0.73 | (0.18, 0.93)
ChatGPT | 0.73 | (0.20, 0.93)
GPT-4 | 0.83 | (0.41, 0.96)

Table 6: Spearman’s coefficient of the expert judgment and groups for crowd
annotators and GPT models. Figure 5: Spearman’s coefficient for scores of 50
random samples in reference-based task among groups. 95% confidence interval
is shown below the coefficient.

###  4.5 Discussion

In Section 4.4, we published the same reference-based task as a test to
different crowd annotators (pipeline, MACE, and CloudResearch). It showed that
filtering workers before the actual evaluation task (pipeline) can avoid the
waste of time and resources and achieve high agreement at a lower cost and a
full coverage of HITs, compared to discarding annotations after the task
(MACE) (see Table 7). Our pipeline also recruited workers of similar quality
to CloudResearch at a lower cost; however, based on further analysis, the
correctness of annotations was not guaranteed (see Section 7 for details).
Besides, details about the estimated cost of GPT models for the reference-
based task can be found in Table 15 in Appendix A.8.2.

| Pipeline | MACE (0.5) | CloudResearch
---|---|---|---
Num. of initial workers | 200 | 276 | 45
% of workers kept | 4% | 19.2% | 17.8%
HIT coverage | 30/30 | 30/30 | 30/30
Avg. num. workers per HIT | 8 | 2.4 | 8
Krippendorff’s Alpha | 0.534 | 0.380 | 0.513
Cost per worker (for Avg. num. workers per HIT) | $27 | $175 | $31

Table 7: Comparison between approaches of crowd annotators (pipeline, MACE,
and CloudResearch) for the reference-based task.

##  5 Statistical Test for Stability of Pipeline

We next examined whether there was a difference in the probability of passing
the qualification and endurance task among MTurk workers. Thus, we started by
assuming the probability of passing each task for each round came from the
same distribution, and we performed a statistical test as follows.

Annotation Task | Qual. Task | End. Task
---|---|---
Pass Rate | 0.13 | 0.06
Mean of Pass Rate (Bootstrap) | 0.1302 | 0.0602
Standard Dev. of Pass Rate (Bootstrap) | 0.0236 | 0.0168

Table 8: Statistical test results for stability of pipeline.

Let $\mathcal{X}$ denote the random variable representing the MTurk worker.
For the qualification task, let $q_{x\in\mathcal{X}}(x)$ denote the binary
random variable which has the value of 1 if the worker can pass the task, and
0 otherwise. Similarly, let $e_{x\in\mathcal{X}}(x)$ denote the binary random
variable indicating whether the worker can pass the endurance task. Given 50
MTurk workers in each round, we use $Q$ to denote the binary random variables
in a round as (1). It can also be regarded as examples sampled from
$q_{x\in\mathcal{X}}(x)$. Among the samples, the probability of a worker who
can pass the qualification task is equal to the expectation of
$q_{x\in\mathcal{X}}(x)=1$ as (2). Since only workers who passed the
qualification task are eligible for the endurance task, the probability of a
worker passing the endurance task is equal to the expectation of
$e_{x\in\mathcal{X},q(x)=1}(x)=1$ as (5), which is a joint distribution of
$q_{x\in\mathcal{X}}(x)$ and $e_{x\in\mathcal{X}}(x)$.

$Q=\\{q_{x_{1}\in\mathcal{X}}(x_{1}),...,q_{x_{50}\in\mathcal{X}}(x_{50})\\}$
(1)

$P(q_{x\in\mathcal{X}}(x)=1)=\mathbb{E}(q_{x\in\mathcal{X}}(x)=1)$ (2)

$\displaystyle P(e_{x\in\mathcal{X},q(x)=1}(x)=1)$ $\displaystyle=$
$\displaystyle\mathbb{E}(e_{x\in\mathcal{X},q(x)=1}(x)=1)$ (3)
$\displaystyle=$ $\displaystyle P(e_{x\in\mathcal{X}}(x)=1|q(x)=1)P(q(x)=1)$

Thus, we used the Bootstrap method (Efron, 1992) with 10,000 iterations to
estimate the mean and standard deviation of the probability of passing the
qualification and endurance task. Table 8 shows the results of all rounds with
breakdowns of each round. We can see some variance that might come from MTurk
workers given each round. To test whether there is a difference in the
probability of passing each task among different rounds, we conducted the
permutation test (Fisher, 1936; Pitman, 1937) for every two rounds. The
results show that we cannot reject the null hypothesis that the underlying
distributions of every two rounds are the same (see Appendix A.4).

##  6 Conclusion

In this paper, we present a two-step recruitment pipeline that yields 12
qualified workers (4 gold and 8 silver workers) out of 200 MTurk workers with
basic qualification settings in our experiments. We show that workers
identified by our pipeline can (i) achieve a higher inter-annotator agreement
than expert annotators in the endurance task, (ii) outperform the statistical
filter (MACE) that discards annotation after the reference-based task, and
(iii) replicate a proxy of CloudResearch annotations in the correctness
analysis. Though the 6% yield rate is not as expected, our pipeline serves as
the best practice to deliver high-agreement annotations and addresses the
widespread waste of resources on low-quality annotations through filtering out
subpar workers before they embark on large-scale tasks. In the future, we plan
to build up a pool of reliable annotators who can deliver high-quality (both
high agreement and correctness) evaluations on a large scale and in multiple
tasks, languages, and platforms.

##  7 Limitations

This research creates a relatively complete pipeline to identify qualified
MTurk workers for high-quality human evaluations based on existing techniques,
and thoroughly tests the effectiveness of this pipeline both qualitatively and
quantitatively compared to other methods. However, there are several
limitations of this work:

  * •

The experiments are only conducted for summarization tasks in English on MTurk
platform. Thus, this pipeline can also be tested on other NLG tasks, in other
languages, and on other platforms to see whether our three-step concept
generalizes broadly to all human evaluations.

  * •

The specific questions designed for each task are not panacea solutions. A
better HIT design may exist for different experimental purposes, as long as it
follows the ideas behind each task. For example, the endurance task aims to
ensure the worker’s reliable performance on a large number of annotations, so
modifications based on this idea might work better in case-by-case
scenarios121212We encourage starting the design from the reference-based task
(which performs as the test of true annotation task) and thinking about what
specific training the annotators are expected to have through the
qualification and endurance task..

  * •

There is no guarantee for the training of correctness in the pipeline though a
high agreement is achieved. An additional correctness check might need to be
included along with the endurance task to achieve both high agreement and
correctness through the filtering of the pipeline.

##  8 Ethical Considerations

Considering that crowd workers are often underpaid, experiments in this work
all followed fair working wage
standards131313https://livingwage.mit.edu/counties/27053 when using MTurk for
recruitment purposes (details for each task are in Table 3). In addition, we
have not rejected the work from any unqualified workers so far, though we
reserve the right to do so when conducting the experiments.

In our experiments, personal data (any information relating to an identifiable
natural person) was collected, processed, and stored based on certain data
protection regulations,141414https://gdpr.eu/article-4-definitions/ given
relevant privacy concerns. Special category information (i.e. personal data
revealing racial or ethnic origin, etc.) was not included in this work. More
information about the details of human evaluation experiments in this work can
be found in the Human Evaluation Datasheet (HEDS) (Shimorina and Belz, 2022)
in the Appendix.

## Acknowledgements

We would like to thank the anonymous reviewers for their helpful feedback on
our paper. We would like to thank Claire Daniele for her editorial support.
Mille’s contribution was funded by the European Union under the Marie
Skłodowska-Curie grant agreement No 101062572 (M-FleNS).

## References

  * Amidei et al. (2020) Jacopo Amidei, Paul Piwek, and Alistair Willis. 2020.  Identifying annotator bias: A new IRT-based method for bias identification.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4787–4797, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Arechar et al. (2017) Antonio A Arechar, Gordon T Kraft-Todd, and David G Rand. 2017.  Turking overtime: How participant characteristics and behavior vary over time and day on amazon mechanical turk.  _Journal of the Economic Science Association_ , 3(1):1–11.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Berinsky et al. (2012) Adam J. Berinsky, Gregory A. Huber, and Gabriel S. Lenz. 2012.  Evaluating online labor markets for experimental research: Amazon.com’s mechanical turk.  _Political Analysis_ , 20(3):351–368.
  * Buhrmester et al. (2011) Michael Buhrmester, Tracy Kwang, and Samuel D Gosling. 2011.  Amazon’s mechanical turk: A new source of inexpensive, yet high-quality data?  _Perspectives on Psychological Science_ , 6(1):3–5.
  * Callison-Burch (2009) Chris Callison-Burch. 2009.  Fast, cheap, and creative: Evaluating translation quality using Amazon’s Mechanical Turk.  In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing_ , pages 286–295, Singapore. Association for Computational Linguistics.
  * Cohen (1960) Jacob Cohen. 1960.  A coefficient of agreement for nominal scales.  _Educational and Psychological Measurement_ , 20:37 – 46.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Efron (1992) Bradley Efron. 1992.  Bootstrap methods: another look at the jackknife.  In _Breakthroughs in statistics_ , pages 569–593. Springer.
  * Fabbri et al. (2021) Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021.  SummEval: Re-evaluating summarization evaluation.  _Transactions of the Association for Computational Linguistics_ , 9:391–409.
  * Fisher (1936) Ronald Aylmer Fisher. 1936.  Design of experiments.  _British Medical Journal_ , 1(3923):554.
  * Gehrmann et al. (2022) Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2022.  Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text.  _arXiv preprint arXiv:2202.06935_.
  * Gillick and Liu (2010) Dan Gillick and Yang Liu. 2010.  Non-expert evaluation of summarization systems is risky.  In _Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk_ , pages 148–151, Los Angeles. Association for Computational Linguistics.
  * Graham et al. (2018) Yvette Graham, George Awad, and Alan Smeaton. 2018.  Evaluation of automatic video captioning using direct assessment.  _PLOS ONE_ , 13(9):1–20.
  * Graham et al. (2017) Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2017.  Can machine translation systems be evaluated by the crowd alone.  _Natural Language Engineering_ , 23(1):3–30.
  * Hardy et al. (2019) Hardy Hardy, Shashi Narayan, and Andreas Vlachos. 2019.  HighRES: Highlight-based reference-less evaluation of summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3381–3392, Florence, Italy. Association for Computational Linguistics.
  * Hayes and Krippendorff (2007) Andrew F Hayes and Klaus Krippendorff. 2007.  Answering the call for a standard reliability measure for coding data.  _Communication methods and measures_ , 1(1):77–89.
  * Hovy et al. (2013) Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013.  Learning whom to trust with MACE.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1120–1130, Atlanta, Georgia. Association for Computational Linguistics.
  * Howcroft et al. (2020) David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020.  Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 169–182, Dublin, Ireland. Association for Computational Linguistics.
  * Huynh et al. (2021) Jessica Huynh, Jeffrey Bigham, and Maxine Eskenazi. 2021.  A survey of nlp-related crowdsourcing hits: what works and what does not.  _arXiv preprint arXiv:2111.05241_.
  * Isozaki et al. (2010) Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. 2010\.  Automatic evaluation of translation quality for distant language pairs.  In _Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing_ , pages 944–952, Cambridge, MA. Association for Computational Linguistics.
  * Kaplan and Meier (1958) E. L. Kaplan and Paul Meier. 1958.  Nonparametric estimation from incomplete observations.  _Journal of the American Statistical Association_ , 53(282):457–481.
  * Karpinska et al. (2021) Marzena Karpinska, Nader Akoury, and Mohit Iyyer. 2021.  The perils of using Mechanical Turk to evaluate open-ended text generation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1265–1285, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kummerfeld (2021) Jonathan K. Kummerfeld. 2021.  Quantifying and avoiding unfair qualification labour in crowdsourcing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 343–349, Online. Association for Computational Linguistics.
  * Lau et al. (2014) Jey Han Lau, David Newman, and Timothy Baldwin. 2014.  Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality.  In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 530–539, Gothenburg, Sweden. Association for Computational Linguistics.
  * Likert (1932) Rensis Likert. 1932.  A technique for the measurement of attitudes.  _Archives of psychology_.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Manning et al. (2020) Emma Manning, Shira Wein, and Nathan Schneider. 2020.  A human evaluation of AMR-to-English generation systems.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4773–4786, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Mehri et al. (2022) Shikib Mehri, Jinho Choi, L. F. D’Haro, Jan Deriu, Maxine Eskénazi, Milica Gasic, Kallirroi Georgila, Dilek Z. Hakkani-Tür, Zekang Li, Verena Rieser, Samira Shaikh, David R. Traum, Yi-Ting Yeh, Zhou Yu, Yizhe Zhang, and Chen Zhang. 2022.  Report from the nsf future directions workshop on automatic evaluation of dialog: Research directions and challenges.  _ArXiv_ , abs/2203.10012.
  * Mille et al. (2019) Simon Mille, Anja Belz, Bernd Bohnet, Yvette Graham, and Leo Wanner. 2019.  The second multilingual surface realisation shared task (SR’19): Overview and evaluation results.  In _Proceedings of the 2nd Workshop on Multilingual Surface Realisation (MSR 2019)_ , pages 1–17, Hong Kong, China. Association for Computational Linguistics.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774.
  * Oppenheimer et al. (2009) Daniel M. Oppenheimer, Tom Meyvis, and Nicolas Davidenko. 2009.  Instructional manipulation checks: Detecting satisficing to increase statistical power.  _Journal of Experimental Social Psychology_ , 45(4):867–872.
  * Oppenlaender et al. (2020) Jonas Oppenlaender, Kristy Milland, Aku Visuri, Panos Ipeirotis, and Simo Hosio. 2020.  Creativity on paid crowdsourcing platforms.  In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ , CHI ’20, page 1–14, New York, NY, USA. Association for Computing Machinery.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Paolacci et al. (2010) Gabriele Paolacci, Jesse Chandler, and Panagiotis G Ipeirotis. 2010.  Running experiments on amazon mechanical turk.  _Judgment and Decision making_ , 5(5):411–419.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: A method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_ , ACL ’02, page 311–318, USA. Association for Computational Linguistics.
  * Paun et al. (2018) Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018.  Comparing Bayesian models of annotation.  _Transactions of the Association for Computational Linguistics_ , 6:571–585.
  * Pitman (1937) E. J. G. Pitman. 1937.  Significance tests which may be applied to samples from any populations.  _Supplement to the Journal of the Royal Statistical Society_ , 4(1):119–130.
  * Robinson et al. (2019) Jonathan Robinson, Cheskie Rosenzweig, Aaron J. Moss, and Leib Litman. 2019.  Tapped out or barely tapped? recommendations for how to harness the vast and largely unused potential of the mechanical turk participant pool.  _PLOS ONE_ , 14(12):1–29.
  * Shimorina and Belz (2022) Anastasia Shimorina and Anya Belz. 2022.  The human evaluation datasheet: A template for recording details of human evaluation experiments in NLP.  In _Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval)_ , pages 54–75, Dublin, Ireland. Association for Computational Linguistics.
  * Spearman (1904) C. Spearman. 1904.  The proof and measurement of association between two things.  _The American Journal of Psychology_ , 15(1):72–101.
  * Webb and Tangney (2022) Margaret A Webb and June P Tangney. 2022.  Too good to be true: Bots and bad data from mechanical turk.  _Perspectives on Psychological Science_ , page 17456916221120027.
  * Whiting et al. (2019) Mark E. Whiting, Grant Hugh, and Michael S. Bernstein. 2019.  Fair work: Crowd work minimum wage with one line of code.  _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ , 7(1):197–206.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.

##  Appendix A Appendix

###  A.1 Proportion of Worker Categories in Qualification Task for Each Round

Annotation Task |  Total Number of Workers |  gold Workers |  silver Workers |  broze Workers |  block Workers
---|---|---|---|---|---
Qualification Task | Round 1 | 50 | 1 (2%) | 4 (8%) | 32 (64%) | 13 (26%)
Round 2 | 50 | 3 (6%) | 5 (10%) | 29 (58%) | 13 (26%)
Round 3 | 50 | 2 (4%) | 3 (6%) | 24 (48%) | 21 (42%)
Round 4 | 50 | 2 (4%) | 6 (12%) | 27 (54%) | 15 (30%)
Table 9: Proportion of worker categories for each round.

###  A.2 Cohen’s Kappa for Each Summary in Endurance Task

For the figures below, Answer.score_0 to Answer.score_3 correspond to the
scores aggregated from the 1st to 4th summary separately for each HIT. The
dark color indicates a high IAA in terms of Cohen’s Kappa score. $S42$ stands
for the second silver worker from Round 4.

Figure 6: Cohen’s Kappa for each summary among silver workers (Pairwise).
Figure 7: Cohen’s Kappa for each summary among gold workers (Pairwise). Figure
8: Cohen’s Kappa for each summary across silver and gold workers (Pairwise).

###  A.3 Endurance Task Result of Lab Members

Worker Combination | A and B | B and C | C and A
---|---|---|---
Cohen’s Kappa (Each Summary) | Answer.score_0 | -0.261 | -0.083 | 0.246
Answer.score_1 | 0.285 | 0.13 | 0.285
Answer.score_2 | 0.206 | -0.006 | -0.049
Answer.score_3 | 0.066 | 0.006 | 0.387
Cohen’s Kappa (Concatenation) | 0.1 | 0.055 | 0.268
Cohen’s Kappa (Omit first 2 HITs) | 0.2 | 0.091 | 0.196
Krippendorff’s Alpha | 0.201
Table 10: Endurance task result of lab members.

###  A.4 Statistical Test Results of Qualification and Endurance Tasks for
Each Round

Annotation Task |  Pass Rate |  Mean of Pass Rate (Bootstrap) |  Standard Dev. of Pass Rate (Bootstrap)
---|---|---|---
Round 1 | Qua. Task | 0.1 | 0.0997 | 0.0424
End. Task | 0.02 | 0.0199 | 0.0198
Round 2 | Qua. Task | 0.16 | 0.1611 | 0.0521
End. Task | 0.08 | 0.0805 | 0.0384
Round 3 | Qua. Task | 0.1 | 0.1000 | 0.0482
End. Task | 0.06 | 0.0599 | 0.0339
Round 4 | Qua. Task | 0.16 | 0.1595 | 0.0511
End. Task | 0.08 | 0.0800 | 0.0380
All Rounds | Qua. Task | 0.13 | 0.1302 | 0.0236
End. Task | 0.06 | 0.0602 | 0.0168
Table 11: Statistical test results of qualification and endurance task.

###  A.5 Cohen’s Kappa (combined scores) for CloudResearch Workers in
Reference-based Task

Figure 9: Cohen’s Kappa (combined scores) among CloudResearch workers.

###  A.6 Spearman’s Coefficient for Inter-groups (Pipeline & MACE) in
Reference-based Task

For the reference-based task, we used 4 methods to calculate Spearman’s
coefficient:

  * •

Method 1: Given the different numbers of remaining MACE workers for each HIT,
we calculate Spearman’s coefficient between our pipeline and MACE workers in
each HIT. Then we take the average of these coefficients as the inter-group
Spearman’s coefficient shown in Table 12 151515We also calculate Spearman’s
coefficient within the group of our pipeline and MACE workers separately for
comparison, as shown in Table 12..

  * •

Method 2: The only difference between this method and Method 1 is that we take
the absolute value when calculating Spearman’s coefficient for each HIT.

  * •

Method 3: We take the average of each annotation question in each HIT within
the group of our pipeline and MACE workers separately, then concatenate these
average scores of all HITs together for each group and calculate Spearman’s
coefficient.

  * •

Method 4: The only difference between this method and Method 3 is that we
calculate Spearman’s coefficient for each HIT and then take the average of all
coefficients instead of concatenating first and then calculating the
coefficient.

Threshold | 0.5 | 0.6 | 0.7
---|---|---|---
% of workers kept | 19.2% | 15.9% | 7.6%
HIT coverage | 30/30 | 27/30 | 18/30
Avg. num. workers per HIT | 2.4 | 1.9 | 1.2
Krippendorff’s Alpha (all scores) | 0.380 | 0.472 | 0.754
Method 1 |  Spearman’s coefficient (MACE workers) | 0.351 | 0.414 | 0.770
Spearman’s coefficient (pipeline workers) | 0.558 | 0.565 | 0.577
Spearman’s coefficient (inter-group) | -0.081 | -0.063 | -0.234
Method 2 |  Spearman’s coefficient (MACE workers) | 0.396 | 0.418 | 0.770
Spearman’s coefficient (pipeline workers) | 0.575 | 0.580 | 0.591
Spearman’s coefficient (inter-group) | 0.307 | 0.299 | 0.308
Method 3 |  Spearman’s coefficient (inter-group) | -0.107 | -0.067 | -0.355
Method 4 |  Spearman’s coefficient (inter-group) | -0.102 | -0.113 | -0.194

Table 12: Methods for calculation of Spearman’s coefficient within and across
groups of pipeline and MACE workers in reference-based task.

###  A.7 Qualitative Analysis of Correctness Across Annotation Sources in
Reference-based Task

For the reference-based task, we first randomly select 50 HITs out of 30 HITs
(HIT index ranges from 0 to 29), and then 1 annotation question out of 8
questions (annotation index ranges from 0 to 7) for each of these HITs
selected in the above step.

For each randomly selected annotation question, we calculate the median within
the groups of our pipeline, MACE, and CloudResearch workers separately, as
well as the scores generated by GPT models (GPT-3.5 (text-davinci-003),
ChatGPT, and GPT-4161616For the ChatGPT score, we ran 5 times with default
parameters (temperature=1, top_p=1) to take the median, but set the
temperature as 0 with a single run for GPT-3.5 and GPT-4 scores.). The expert
judgment (aggregated by the median) and details for 50 randomly selected
annotation questions can be found in Table 13 and Table 14.

Figure 10 shows Spearman’s coefficient among different groups aggregated by
the median before (left) and after (right) the removal of controversial HITs
(HIT with index 15, 16, and 28). We also perform a similar analysis aggregated
by the mean shown in Figure 11.

Figure 10: Spearman’s coefficient for scores of 50 random samples aggregated
by median among groups before (left) and after (right) the removal of
controversial HITs (95% confidence interval is shown below the coefficient).

Figure 11: Spearman’s coefficient for scores of 50 random samples aggregated
by mean among groups before (left) and after (right) the removal of
controversial HITs (95% confidence interval is shown below the coefficient).

Sample Index | Two Types of Summaries |  Inclusion Direction | Human Annotators (Median) | GPT series scores |  Expert Judgment
---|---|---|---|---|---
|  | Pipeline | MACE | CloudResearch | GPT-3.5 | ChatGPT | GPT-4
1 | Reference |  The government has given regulators more time to investigate the proposed takeover of broadcaster Sky by 21st Century Fox. | can2ref | 5.0 | 4.0 | 4.0 | 4.0 | 5.0 | 5.0 | 5.0
Candidate |  The government has extended the deadline for an inquiry into the takeover of Sky by 21st Century Fox.
2 | Reference |  A Chinese woman has been found guilty of trespassing at President Donald Trump’s Mar-a-Lago club in Florida and of lying to a federal agent. | can2ref | 3.0 | 5.0 | 3.5 | 4 | 5 | 4.5 | 4.0
Candidate |  A Chinese woman who sparked alarm when she walked into US President Donald Trump’s Mar-a-Lago resort has been found guilty of trespassing.
3 | Reference |  A unique garden is helping Canadians to break a taboo that exists in many societies. It is allowing parents to talk openly about miscarriage. | ref2can | 4.0 | 4.0 | 4.0 | 4.0 | 5.0 | 4.0 | 3.0
Candidate |  A Canadian cemetery has created a garden dedicated to the memory of babies lost during pregnancy. It’s a place that’s especially for those who have had multiple miscarriages.
4 | Reference |  Gadgets that track your steps, sleeping and heart rate could help us live longer and cut national healthcare costs by billions - or so we are told. | can2ref | 3.0 | 4.0 | 2.5 | 1.0 | 1.0 | 1.0 | 1.0
Candidate |  It is a huge amount of us have a smartphone, a smartphone and a gadget that feeds data from a smartphone.
5 | Reference |  A unique garden is helping Canadians to break a taboo that exists in many societies. It is allowing parents to talk openly about miscarriage. | can2ref | 2.0 | 4.0 | 2.0 | 4.0 | 5.0 | 4.0 | 3.0
Candidate |  A Canadian garden dedicated to the memory of children lost during pregnancy is helping to heal the pain of grief.
6 | Reference |  The 2017 Oscar nominations are out, with La La Land the frontrunner. Here’s a round-up of the surprises and talking points from this year’s list. | can2ref | 4.0 | 3.0 | 3.5 | 3.0 | 4.0 | 4.0 | 4.0
Candidate |  The full list of Oscar nominations has been announced. Here are 10 talking points from the shortlists.
7 | Reference |  Welsh victims of the contaminated blood scandal have said it is not fair they get less financial help than people affected in England and Scotland. | ref2can | 2.0 | 4.0 | 1.5 | 4.0 | 4.0 | 4.0 | 2.0
Candidate |  A man who contracted hepatitis C from the contaminated blood scandal has said Welsh support payments are not fair.
8 | Reference |  An anonymous letter sent to a council outlining an alleged plan to oust head teachers is "defamatory", the leader of Birmingham City Council has said. | can2ref | 4.0 | 4.0 | 4.0 | 3.0 | 1.0 | 3.0 | 2.0
Candidate |  A letter written by a council officer calling for schools to be taken over by a council has been defamatory.
9 | Reference |  Graduates from ethnic minorities in Britain are less likely to be in work than their white peers, a study says. | ref2can | 4.0 | 4.0 | 3.5 | 2.0 | 2.0 | 1.0 | 2.0
Candidate |  The number of ethnic minority graduates in the UK has fallen by almost 5% in the last year, according to a think tank.
10 | Reference |  Two endangered red panda cubs have been born at a wildlife park on the Isle of Man. | ref2can | 2.0 | 4.0 | 4.0 | 5.0 | 5.0 | 5.0 | 5.0
Candidate |  Two endangered red panda cubs have been born at a wildlife park in the Isle of Man.
11 | Reference |  Two endangered red panda cubs have been born at a wildlife park on the Isle of Man. | ref2can | 5.0 | 3.0 | 5.0 | 4.0 | 4.0 | 5.0 | 5.0
Candidate |  Two endangered red panda cubs have been born at a wildlife park on the Isle of Man, a year after a giant themed elephant calf escaped from his enclosure.
12 | Reference |  Welsh Water has announced pre-tax profits of £7m for the last financial year. | can2ref | 5.0 | 4.0 | 5.0 | 5.0 | 5.0 | 5.0 | 4.0
Candidate |  Welsh Water has announced pre-tax profits of £7m for the year to April.
13 | Reference |  A "poo-powered" VW Beetle has taken to the streets of Bristol in an attempt to encourage sustainable motoring. | ref2can | 4.0 | 4.0 | 2.5 | 4.0 | 4.0 | 4.0 | 3.0
Candidate |  A car powered by biogas has been seen on the streets of Bristol.
14 | Reference |  An anonymous letter sent to a council outlining an alleged plan to oust head teachers is "defamatory", the leader of Birmingham City Council has said. | can2ref | 5.0 | 3.0 | 4.5 | 4.0 | 5.0 | 4.0 | 3.0
Candidate |  A letter sent to Birmingham City Council by a whistle-blower has been described as "defamatory" by the city council’s chief inspector of schools.
15 | Reference |  In our media-saturated age, it’s rare to have a chief executive who doesn’t speak to the press or, indeed, very often publicly. | can2ref | 5.0 | 4.0 | 5.0 | 1.0 | 1.0 | 1.0 | 1.0
Candidate |  Chinese entrepreneurs are a familiar sight.
16 | Reference |  Parliament has been dissolved and the official election campaign has begun. BBC Reality Check listened in to Prime Minister Boris Johnson’s campaign speeches in Downing Street and in Birmingham to check the facts and figures. | ref2can | 5.0 | 4.0 | 5.0 | 3.0 | 3.0 | 3.0 | 1.0
Candidate |  Boris Johnson made a series of claims about his government’s plans for the next few years. Here are six of the key pledges he made.
17 | Reference |  Naturalist Sir David Attenborough and the Queen are the greatest living British man and woman, according to readers of Best of British magazine. | can2ref | 3.0 | 4.0 | 3.5 | 4.0 | 5.0 | 4.0 | 4.0
Candidate |  David Attenborough has been voted the best of British by the magazine.
18 | Reference |  An Edinburgh adventurer has become the youngest woman to ski solo to the South Pole. | can2ref | 4.0 | 4.0 | 4.0 | 5.0 | 5.0 | 4.5 | 4.0
Candidate |  A woman from Edinburgh has become the youngest person to reach the South Pole solo.
19 | Reference |  Resurfacing work on a newly-repaired canal towpath that washed away after vandals left a lock gate open has begun. | can2ref | 4.0 | 3.0 | 3.5 | 4.0 | 4.0 | 4.0 | 5.0
Candidate |  Work has begun to resurface a canal towpath which was damaged by flooding.
20 | Reference |  The Brexit vote is already having a negative impact on business, a survey of bosses from some of the UK’s biggest companies has suggested. | ref2can | 4.0 | 5.0 | 4.0 | 4.0 | 5.0 | 4.0 | 5.0
Candidate |  The majority of business leaders believe the Brexit vote has already had a negative impact on their company, a survey suggests.
21 | Reference |  A campaign has begun to stop the spread of norovirus in Cornwall. | can2ref | 5.0 | 5.0 | 3.5 | 4.0 | 5.0 | 5.0 | 5.0
Candidate |  A campaign has been launched to prevent the spread of norovirus in Cornwall.
22 | Reference |  Welsh victims of the contaminated blood scandal have said it is not fair they get less financial help than people affected in England and Scotland. | can2ref | 3.0 | 3.5 | 2.5 | 3.0 | 2.0 | 2.0 | 3.0
Candidate |  The Welsh Government has said it is not fair to pay for patients who have contaminated blood in the 1970s and 1980s.
23 | Reference |  People on Jersey’s Ecrehous islands are concerned travellers are arriving from France by boat and not being tested for coronavirus. | ref2can | 1.0 | 4.0 | 1.0 | 3.0 | 3.0 | 3.0 | 2.0
Candidate |  People living on Jersey’s Ecrehous islands have said they are worried about the number of people arriving ashore.
24 | Reference |  The government has given regulators more time to investigate the proposed takeover of broadcaster Sky by 21st Century Fox. | can2ref | 2.5 | 3.0 | 3.0 | 4.0 | 4.0 | 4.0 | 3.0
Candidate |  The government has extended its takeover inquiry into Sky’s takeover deal with regulator Ofcom.
25 | Reference |  Graduates from ethnic minorities in Britain are less likely to be in work than their white peers, a study says. | can2ref | 3.0 | 3.5 | 3.0 | 2.0 | 2.0 | 1.0 | 2.0
Candidate |  The number of ethnic minority graduates in the UK has fallen by almost 5% in the last year, according to a think tank.

Table 13: Qualitative analysis of correctness with 50 random samples (Part 1).

Sample Index | Two Types of Summaries |  Inclusion Direction | Human Annotators (Median) | GPT series scores |  Expert Judgment
---|---|---|---|---|---
|  | Pipeline | MACE | CloudResearch | GPT-3.5 | ChatGPT | GPT-4
26 | Reference |  Joan Miro’s 1927 work Peinture (Etoile Bleue) has sold for more than £23.5 million in London, setting a new auction record for the Spanish painter. | can2ref | 4.0 | 5.0 | 4.0 | 3.0 | 1.0 | 2.0 | 3.0
Candidate |  Joan Miro’s painting, which inspired the famous Joan Miro, has smashed its auction record for £15m.
27 | Reference |  One of Oxford’s main routes remains closed because of flooding for the second time in a month. | ref2can | 2.5 | 5.0 | 2.0 | 5.0 | 5.0 | 5.0 | 5.0
Candidate |  A major route through Oxford has been closed for the second time in a month due to flooding.
28 | Reference |  Holidaymakers say they have been left thousands of pounds out of pocket after a letting company ceased trading without notice. | ref2can | 5.0 | 4.0 | 5.0 | 4.0 | 4.0 | 4.0 | 2.0
Candidate |  Brighton Holiday Homes has gone bust with bookings cancelled after a third of its customers claimed their money was lost.
29 | Reference |  A £4.4m revamped Denbighshire leisure centre will open on Saturday. | cand2ref | 4.0 | 5.0 | 3.0 | 5.0 | 4.0 | 4.0 | 4.0
Candidate |  A Denbighshire leisure centre is reopening on Thursday after a £4.4m revamp.
30 | Reference |  Gadgets that track your steps, sleeping and heart rate could help us live longer and cut national healthcare costs by billions - or so we are told. | ref2cand | 1.0 | 3.5 | 3.0 | 4.0 | 1.0 | 1.0 | 1.0
Candidate |  Every step we take is going to be tracked by a device that cannot simply put our fingers on our wrists.
31 | Reference |  Gadgets that track your steps, sleeping and heart rate could help us live longer and cut national healthcare costs by billions - or so we are told. | cand2ref | 2.0 | 3.0 | 4.0 | 4.0 | 1.0 | 1.0 | 1.0
Candidate |  Every step we take is going to be tracked by a device that cannot simply put our fingers on our wrists.
32 | Reference |  Joan Miro’s 1927 work Peinture (Etoile Bleue) has sold for more than £23.5 million in London, setting a new auction record for the Spanish painter. | ref2cand | 2.0 | 5.0 | 4.0 | 4.5 | 4.0 | 4.0 | 4.0
Candidate |  A painting by Joan Miro has sold for £18.8m at auction, breaking the previous record for a work by the artist.
33 | Reference |  A unique garden is helping Canadians to break a taboo that exists in many societies. It is allowing parents to talk openly about miscarriage. | cand2ref | 3.0 | 3.0 | 4.0 | 3.0 | 4.0 | 5.0 | 5.0
Candidate |  A Canadian memorial garden is helping parents come to terms with the pain of losing a child during pregnancy.
34 | Reference |  Holidaymakers say they have been left thousands of pounds out of pocket after a letting company ceased trading without notice. | cand2ref | 3.0 | 4.0 | 4.0 | 4.0 | 4.0 | 3.0 | 4.0
Candidate |  A holiday home firm has gone bust after customers were told they had been left "heartbroken" after bookings were cancelled.
35 | Reference |  A woman rescued after falling from a North Sea ferry has told how she thought she was going to die. | ref2cand | 4.0 | 4.5 | 5.0 | 1.5 | 5.0 | 5.0 | 5.0
Candidate |  A woman who fell from a ferry into the North Sea has described how she thought she was going to die.
36 | Reference |  The Brexit vote is already having a negative impact on business, a survey of bosses from some of the UK’s biggest companies has suggested. | ref2cand | 4.0 | 3.0 | 3.0 | 2.0 | 4.0 | 5.0 | 5.0
Candidate |  The UK’s vote to leave the European Union is already having a negative impact on businesses, a survey suggests.
37 | Reference |  Welsh Water has announced pre-tax profits of £7m for the last financial year. | ref2cand | 4.0 | 4.5 | 4.0 | 4.0 | 5.0 | 5.0 | 5.0
Candidate |  Welsh Water has announced pre-tax profits of £7m for the year to April.
38 | Reference |  One of Oxford’s main routes remains closed because of flooding for the second time in a month. | ref2cand | 5.0 | 3.0 | 5.0 | 3.5 | 5.0 | 5.0 | 5.0
Candidate |  A major route through Oxford has been closed for the second time in a month because of flooding.
39 | Reference |  A 10-year-old boy died after he hit his head on a wall while playing football at school, an inquest heard. | ref2cand | 4.0 | 4.0 | 3.0 | 3.5 | 4.0 | 5.0 | 5.0
Candidate |  A 10-year-old boy who hit his head while playing football at school died from traumatic brain injury, an inquest heard.
40 | Reference |  A video artist who uses YouTube clips, a print-maker and an artist who pairs spoken word with photography are among this year’s Turner Prize nominees. | ref2cand | 3.0 | 4.0 | 3.5 | 3.5 | 4.0 | 4.0 | 4.0
Candidate |  A YouTube artist who splices together clips of horror films and a print-maker who works with women’s groups are among the nominees for this year’s Turner Prize.
41 | Reference |  Parliament has been dissolved and the official election campaign has begun. BBC Reality Check listened in to Prime Minister Boris Johnson’s campaign speeches in Downing Street and in Birmingham to check the facts and figures. | ref2cand | 1.0 | 1.0 | 3.0 | 1.0 | 3.0 | 4.0 | 2.0
Candidate |  Boris Johnson has been making his pitch to Conservative voters in the final week of the election campaign. What did he get right and wrong?
42 | Reference |  Film director Roman Polanski has been released after being questioned by prosecutors in Poland over sex offences in the US. | cand2ref | 3.0 | 4.0 | 3.5 | 4.0 | 4.0 | 4.0 | 4.0
Candidate |  Polish film director Roman Polanski has been freed after prosecutors said they had not made an extradition bid for him.
43 | Reference |  A video artist who uses YouTube clips, a print-maker and an artist who pairs spoken word with photography are among this year’s Turner Prize nominees. | cand2ref | 3.0 | 3.5 | 4.0 | 4.0 | 3.0 | 4.0 | 3.0
Candidate |  A video artist who uses YouTube and a storyteller who uses storytelling techniques are among the nominees for the 2014 Turner Prize.
44 | Reference |  DJ Dave Lee Travis has told a court he does not have a "predatory nature". | ref2cand | 4.0 | 3.5 | 4.0 | 4.0 | 4.0 | 4.0 | 4.0
Candidate |  Former radio DJ Dave Lee Travis has told a court he is "cuddly" not "predatory".
45 | Reference |  Naturalist Sir David Attenborough and the Queen are the greatest living British man and woman, according to readers of Best of British magazine. | cand2ref | 3.0 | 3.0 | 3.0 | 3.0 | 4.0 | 4.0 | 3.0
Candidate |  Sir David Attenborough has been named the best living British celebrity in a poll by the Magazine of British History.
46 | Reference |  A Chinese woman has been found guilty of trespassing at President Donald Trump’s Mar-a-Lago club in Florida and of lying to a federal agent. | ref2cand | 2.0 | 5.0 | 3.0 | 4.5 | 1.0 | 1.0 | 1.0
Candidate |  A woman who sparked alarm at Mar-a-Lago has been found guilty of killing herself.
47 | Reference |  Graduates from ethnic minorities in Britain are less likely to be in work than their white peers, a study says. | ref2cand | 5.0 | 3.0 | 3.0 | 3.0 | 4.0 | 5.0 | 5.0
Candidate |  Black and ethnic minority graduates are less likely to be employed than white British counterparts, a report suggests.
48 | Reference |  An anonymous letter sent to a council outlining an alleged plan to oust head teachers is "defamatory", the leader of Birmingham City Council has said. | ref2cand | 2.0 | 4.5 | 4.0 | 3.5 | 3.0 | 2.0 | 3.0
Candidate |  A letter written by a council officer calling for schools to be taken over by a council has been defamatory.
49 | Reference |  The 2017 Oscar nominations are out, with La La Land the frontrunner . Here’s a round-up of the surprises and talking points from this year’s list. | ref2cand | 3.0 | 4.0 | 3.0 | 4.0 | 3.0 | 4.0 | 4.0
Candidate |  The full list of Oscar nominations has been announced. Here are 10 talking points from the shortlists.
50 | Reference |  People on Jersey’s Ecrehous islands are concerned travellers are arriving from France by boat and not being tested for coronavirus. | ref2cand | 3.0 | 5.0 | 4.0 | 5.0 | 4.0 | 3.0 | 4.0
Candidate |  People living on Jersey’s Ecrehous islands have said they fear they are "playing Russian roulette" with coronavirus restrictions after a rise in arrivals.

Table 14: Qualitative analysis of correctness with 50 random samples (Part 2).

###  A.8 Interaction with GPT models in Reference-based Task

####  A.8.1 Prompt Design

In Figure 12, we show an example of the interaction with ChatGPT and the exact
prompt design we use to acquire scores generated by GPT models through
API171717https://platform.openai.com/docs/api-reference for the analysis of
correctness in the reference-based task.

This prompt design follows the instructions we provide to the crowd annotators
in the reference-based task (see Figure 16 for details) with minor
modifications for the score generation from GPT models. Details about running
experiments through API can be found in Section 4.4.

Figure 12: Example of interaction with ChatGPT in the reference-based task.

####  A.8.2 Estimated Cost of GPT Models

We estimate the cost of running GPT models for the score generation in the
reference-based task (240 annotation questions in total) based on the cost of
50 random annotation questions. Details of pricing can be found on OpenAI’s
website181818https://openai.com/pricing. We assume the GPT model only returns
the score without explanations.

GPT Models | Cost per 1K Token | Estimated Cost
---|---|---
GPT-3.5 | $0.02 | $0.21
ChatGPT | $0.002 | $0.02
GPT-4 |  $0.03 (prompt) $0.06 (completion) | $0.32

Table 15: Estimated cost of GPT models for the reference-based task.

###  A.9 Instruction and Annotation Question Examples of HIT

Here we provide some examples of instructions and annotation questions for all
three tasks as screenshots.

####  A.9.1 Qualification Task

  * •

Figure 13 shows the definition of an evaluation dimension illustrated with
examples in the training part.

  * •

Figure 14 shows the example of the qualification question in the qualification
part.

Figure 13: Example from training part of qualification task. Figure 14:
Example from qualification part of qualification task.

####  A.9.2 Endurance Task

Figure 15 shows the example of the annotation question on a Likert scale of 1
to 10 in the endurance task.

Figure 15: Example of the annotation question in endurance task.

####  A.9.3 Reference-based Task

  * •

Figure 16 shows the instructions for the reference-based task.

  * •

Figure 17 shows the example of the annotation question in the reference-based
task.

Figure 16: Instructions for the reference-based task. Figure 17: Example of
the annotation question in the reference-based task.