# Cascading Biases: Investigating the Effect of Heuristic Annotation
Strategies on Data and Models

Chaitanya Malaviya  Sudeep Bhatia  Mark Yatskar
University of Pennsylvania
{cmalaviy, bhatiasu<EMAIL_ADDRESS>

###### Abstract

Cognitive psychologists have documented that humans use cognitive heuristics,
or mental shortcuts, to make quick decisions while expending less effort.
While performing annotation work on crowdsourcing platforms, we hypothesize
that such heuristic use among annotators cascades on to data quality and model
robustness. In this work, we study cognitive heuristic use in the context of
annotating multiple-choice reading comprehension datasets. We propose tracking
annotator heuristic traces, where we tangibly measure low-effort annotation
strategies that could indicate usage of various cognitive heuristics. We find
evidence that annotators might be using multiple such heuristics, based on
correlations with a battery of psychological tests. Importantly, heuristic use
among annotators determines data quality along several dimensions: (1) known
biased models, such as partial input models, more easily solve examples
authored by annotators that rate highly on heuristic use, (2) models trained
on annotators scoring highly on heuristic use don’t generalize as well, and
(3) heuristic-seeking annotators tend to create qualitatively less challenging
examples. Our findings suggest that tracking heuristic usage among annotators
can potentially help with collecting challenging datasets and diagnosing model
biases.

##  1 Introduction

While crowdsourcing is an effective and widely-used data collection method in
NLP, it comes with caveats. Crowdsourced datasets have been found to contain
artifacts from the annotation process, and models trained on such data, can be
brittle and fail to generalize to distribution shifts Gururangan et al.
(2018); Kaushik and Lipton (2018); McCoy et al. (2019). In this work, we ask
whether systematic patterns in annotator behavior influence the quality of
collected data.

We hypothesize that usage of cognitive heuristics, which are mental shortcuts
that humans employ in everyday life, can cascade on to data quality and model
robustness. For example, an annotator asked to write a question based on a
passage might not read the entire passage or might use just one sentence to
frame a question. Annotators may seek shortcuts to economize on the amount of
time and effort they put into a task. This behavior in annotators,
characterized by examples that are acceptable but not high-quality, can be
problematic.

We analyze the extent to which annotators engage in various low-effort
strategies, akin to cognitive heuristics, by tracking indicative features from
their annotation data in the form of annotator heuristic traces. First, we
crowdsource reading comprehension questions where we instruct workers to write
hard questions. Inspired by research on human cognition Simon (1956); Tversky
and Kahneman (1974), we identify several heuristics that could be employed by
annotators for our task, such as satisficing Simon (1956), availability
Tversky and Kahneman (1973) and representativeness Kahneman and Tversky
(1972). We measure their potential usage by featurizing the collected data and
annotation metadata (e.g., time spent and keystrokes entered) (§4). Further,
we identify instantiations of these heuristics that correlate well with
psychological tests measuring heuristic thinking tendencies in humans, such as
the cognitive reflection test Frederick (2005); Toplak et al. (2014); Sirota
et al. (2021). Our psychologically plausible measures of heuristic use during
annotation can be aggregated per annotator, forming a holistic summary of the
data they produce.

Based on these statistics, we analyze differences between examples created by
annotators who engage in different levels of heuristics use. Our first finding
is that examples created by strongly heuristic-seeking annotators are also
easier for models to solve using heuristics (§5). We evaluate models that
exploit a few known biases and find that examples from annotators who use
cognitive heuristics are more easily solvable by biased models. We also
examine what impact heuristics have on trained models. Previous work Geva et
al. (2019) shows that models generalize poorly when datasets are split
randomly by annotators, likely due to the existence of artifacts. We replicate
this result and find that models generalize even worse when trained on
examples from heuristic-seeking annotators.

To understand which parts of the annotation pipeline contribute to heuristic-
seeking behavior in annotators, we also tease apart the effect of components
inherent to the task (e.g., passage difficulty) as opposed to the annotators
themselves (e.g., annotator fatigue) (§6). Unfortunately, we don’t discover
simple predictors (i.e., passage difficulty) of when annotators are likely to
use heuristics.

A qualitative analysis of the collected data reveals that heuristic-seeking
annotators are more likely to create examples that are not valid, and require
simpler word-matching on explicitly stated information (§7). Crucially, this
suggests that measurements of heuristic usage, such as those examined in this
paper, can provide a general method to find unreliable examples in
crowdsourced data, and direct our search for discovering artifacts in the
data. Because we implicate heuristic use in terms of robustness and data
quality, we suggest future dataset creators track similar features and
evaluate model sensitivity to annotator heuristic use.111Our code and
collected data is available at https://github.com/chaitanyamalaviya/annotator-
heuristics.

##  2 Background and Related Work

#### Cognitive Heuristics.

The study of heuristics in human judgment, decision making, and reasoning is a
popular and influential topic of research Simon (1956); Tversky and Kahneman
(1974). Heuristics can be defined as mental shortcuts, that we use in everyday
tasks for fast decision-making. For example, Tversky and Kahneman (1974) asked
participants whether more English words begin with the letter K or contain K
as the $3^{rd}$ letter, and more than 70% participants chose the former
because words that begin with K are easier to recall, although that is
incorrect. This is an example of the availability heuristic. Systematic use of
such heuristics can lead to cognitive biases, which are irrational patterns in
our thinking.

At first glance, it may seem that heuristics are always suboptimal, but
previous work has argued that heuristics can lead to accurate inferences under
uncertainty, compared to optimization Gigerenzer and Gaissmaier (2011). We
hypothesize that heuristics can play a considerable role in determining data
quality and their impact depends on the exact nature of the heuristic.
Previous work has shown that crowdworkers are susceptible to cognitive biases
in a relevance judgement task Eickhoff (2018), and has provided a checklist to
combat these biases Draws et al. (2021). In contrast, our work focuses on how
potential use of such heuristics can be measured in a writing task, and
provides evidence that heuristic use is linked to model brittleness.

Features of annotator behavior have previously been useful in estimating
annotator task accuracies Rzeszotarski and Kittur (2011); Goyal et al. (2018).
Annotator identities have also been found to influence their annotations Hube
et al. (2019); Sap et al. (2022). Our work builds on these results and
estimates heuristic use with features to capture implicit clues about data
quality.

#### Mitigating and discovering biases.

The presence of artifacts or biases in datasets is well-documented in NLP, in
tasks such as natural language inference, question answering and argument
comprehension (Gururangan et al., 2018; McCoy et al., 2019; Niven and Kao,
2019, inter alia). These artifacts allow models to solve NLP problems using
unreliable shortcuts Geirhos et al. (2020). Several researchers have proposed
approaches to achieve robustness against known biases. We refer the reader to
Wang et al. (2022) for a comprehensive review of these methods. Targeting
biases that are unknown continues to be a challenge, and our work can help
find examples which are likely to contain artifacts, by identifying heuristic-
seeking annotators.

Prior work has proposed methods to discover shortcuts using explanations of
model predictions Lertvittayakumjorn and Toni (2021), including sample-based
explanations Han et al. (2020) and input feature attributions Bastings et al.
(2021); Pezeshkpour et al. (2022). Other techniques that can be helpful in
diagnosing model biases include building a checklist of test cases Ribeiro et
al. (2020); Ribeiro and Lundberg (2022), constructing contrastive Gardner et
al. (2020) or counterfactual Wu et al. (2021) examples and statistical tests
Gururangan et al. (2018); Gardner et al. (2021). Our work is complementary to
these approaches, as we provide an alternative approach to bias discovery that
is tied to annotators.

#### Improved crowdsourcing.

A related line of work has studied modifications to crowdsourcing protocols to
improve data quality Bowman et al. (2020); Nangia et al. (2021). In addition,
model-in-the-loop crowdsourcing methods such as adversarial data collection
Nie et al. (2020) and the use of generative models Bartolo et al. (2022); Liu
et al. (2022) have been shown to be helpful in creating more challenging
examples. We believe that tracking annotator heuristics use can help make
informed adjustments to crowdsourcing protocols.

##  3 Annotation Protocol

We consider multiple-choice reading comprehension as our crowdsourcing task,
because of the richness of responses and interaction we can get from
annotators, which allows us to explore a range of hypothetical heuristics. We
describe here the methodology for our data collection.

We provided annotators on Amazon Mechanical Turk with passages and ask them to
write a multiple-choice question with four options. We used the first
paragraphs of ‘vital articles’ from the English Wikipedia222Wikipedia Level 4
vital articles:
https://en.wikipedia.org/wiki/Wikipedia:Vital_articles/Level/4, and ensured
that passages are at least 50 words long and at most 250 words long. Passages
spanned 11 genres including arts, history, physical sciences, and others, and
passages were randomly sampled from the 10K passages. Annotators were asked to
write challenging questions that cannot be answered by reading just the
question or passage alone, and have a single correct answer. Further, they
were asked to ensure that passages provided sufficient information to answer
the question while allowing questions to require basic inferences using
commonsense or causality.

Annotators were first qualified to avoid spamming behavior. This qualification
checked for spamming behavior in the form of invalid questions, and not
example quality. Annotators were then asked to write a multiple-choice
question to 4 passages in a single HIT on MTurk. Annotators were asked to not
work on more than 8 HITs. We collected 1225 multiple-choice question-answer
pairs from 73 annotators. In addition, we also logged their keystrokes and the
time taken to complete an example (ensuring that time away from the screen was
not counted). Our annotation interface was built upon Nangia et al. (2021).
For other details about our annotation protocol, please refer to Appendix A.

##  4 Cognitive Heuristics in Crowdsourcing

Consequence of cognitive heuristic |  Featurization
---|---
Satisficing (lowtime) |  (1) time, (2) log (time), (3) time / doc length, (4) log (time / doc length)
Satisficing (loweffort) |  (1) question length, (2) keystroke length, (3) question+ops length,
|  (4) question+ops length / keystroke length
Availability (first option bias) |  First option is marked as correct answer
Availability (serial position) |  Correct answer matches span in first or last sentence of passage
Representativeness (word overlap) |  Average word overlap in all pairs of examples by annotator
Representativeness (copying) |  (1) Length of longest common subsequence (lcs) b/w doc & question,
|  (2) Max of normalized length of lcs between doc & {question, options},
|  (3) Normalized avg of length of lcs between doc & {question, options}
Table 1: Consequences of cognitive heuristics and featurizations for multiple-
choice reading comprehension data.

Cognitive heuristics are mental shortcuts, that humans employ in problem-
solving tasks to make quick judgments Simon (1956); Tversky and Kahneman
(1974). Annotators, tasked with authoring natural language examples, are not
infallible to using such heuristics. We hypothesize that, in writing tasks,
reliance on heuristics is a traceable indicator of poor data quality. In this
section, we identify several heuristics, their consequences in annotator
behavior, and features to track them. Later, we also show they are predictors
of qualitatively important dimensions of data.

###  4.1 Methodology

To test the above hypothesis, we consider several known cognitive heuristics
which could be relevant for our task. This list is not comprehensive, and we
refer the readers to prior work for a thorough overview of cognitive biases
Shah and Oppenheimer (2008); Draws et al. (2021). To tangibly measure the
potential usage of a heuristic, we featurize each heuristic into a measurable
quantity that can be computed automatically for an example (see Table 1).
While we do not conclusively determine that an annotator is using a heuristic,
we explore various featurizations that align with the intuition behind each
heuristic. These featurizations can sometimes be mapped to multiple heuristics
that interact together, but for ease of presentation, we list them under the
most related cognitive heuristic. These help us create annotator heuristic
traces, which contain average heuristic values across all of an annotator’s
examples.

To verify if our instantiation of a heuristic aligns with heuristic-seeking
tendencies in annotators, we measure correlations of heuristic values with
annotator performances on a battery of psychological tests Frederick (2005);
Toplak et al. (2014); Sirota et al. (2021), described in §4.4.

###  4.2 Heuristics Studied

#### Satisficing:

Satisficing is a cognitive heuristic that involves making a satisfactory
choice, rather than an optimal one Simon (1956). In terms of mental process,
strong satisficing can involve inattention to information and lack of
information synthesis. In social cognition, Krosnick (1991) described how
satisficing can manifest in various patterns in survey responses. For example,
survey-takers might pick the same response to several questions in sequence,
pick a random response, or use the acquiescence bias (where they always choose
to agree with the given statement). A potential outcome of satisficing in our
task is low time spent on the task and low effort put into forming a question.

Assuming the working time is $t$ and number of tokens in a passage $d$ is
$l_{d}$, we consider the following lowtime featurizations: (1) $t$, (2)
$\log{t}$ , (3) $t/l_{d}$, (4) $\log(t/l_{d})$.333Previous work shows that
taking the logarithm normalizes the response time distribution Whelan (2008).

We estimate an annotator’s amount of effort through their responses. An
annotator who is consistently editing their work or writing long questions
might be attempting to thoughtfully draft their question. While this may not
always be true (for eg, a worker might spend time thinking about their
question and only start writing later), we hypothesize that often, short
responses can be indicators of satisficing. Given the number of words found in
a stream of keystrokes, $k$, the question $q$, and all options $o_{i}$ is
$l_{k}$, $l_{q}$ and $l_{o}$, we consider these loweffort featurizations: (1)
$l_{q}$, (2) $l_{k}$, (3) $l_{q}+l_{o}$, (4) $(l_{q}+l_{o})/l_{k}$.

#### Availability heuristic:

The tendency to rely upon information that is more readily retrievable from
our memory is the availability heuristic Tversky and Kahneman (1973). For
example, after hearing about a plane crash on the news, people may overstate
the dangers of flying. For our task, once an annotator has read a passage and
formulated a question, the question and the correct answer are likely to be
readily available in their mind. This could cause them to write that
information before any of the distractor options. Therefore, we check whether
the first option specified for an example is also the correct answer (first
option bias).

Another consequence of this heuristic is the serial-position effect. When
presented with a series of items like a list of words or items in a grocery
list, people recall the first and last few items from the series better than
the middle ones Murdock Jr (1962); Ebbinghaus (1964) because of their easier
availability. This effect can also be explained as a combination of the
primacy effect and recency effect. To test if an annotator anchors their
questions on the first or last sentence of the passage due to this heuristic,
we check if the correct answer marked for an example matches a span in the
first or last sentence of the passage (serial position).

#### Representativeness heuristic:

The representativeness heuristic is our tendency to use the similarity of
items to make decisions Kahneman and Tversky (1972). For example, if a person
is picking a movie to watch, they might think of movies they previously liked
and look for those attributes in a new movie. Similarly, an annotator may
repeat the same construction in their questions to ease decision-making (e.g.,
"which of the following is true?" or "what year did [event] happen?"). This
could either mean that they are not fully engaged, or, they found a writing
strategy that works well and they choose to stick to it. We measure this
tendency by computing the average word overlap across all pairs of questions
from an annotator.

A different manner in which this heuristic can manifest is using similarity
with the provided context, i.e., through copying. Copying, or imitation, is a
common building block that guides human behavior and decision making. In
deciding what clothes to buy or which book to read, humans use imitation-of-
the-majority to make quicker inferences with lesser cognitive effort Garcia-
Retamero et al. (2009); Gigerenzer and Gaissmaier (2011). Similarly,
annotators can have tendencies to copy text word-for-word from the context
they are primed with, to reduce their cognitive load. Assuming lcs is a
function that computes the length of the longest common subsequence between
two sequences, we consider these featurizations for copying: (1) lcs$(d,q)$,
(2) $\textnormal{max}(\textsc{lcs}(d,q),\textsc{lcs}(d,o))$ and (3)
$\textnormal{avg}(\textsc{lcs}(d,q),\textsc{lcs}(d,o))$.

###  4.3 Annotator Heuristic Traces

The consequences of heuristics we compute, as summarized in Table 1, may not
in themselves be problematic per example. However, we claim that annotators
who consistently rely on such heuristics may impart larger, harder-to-detect,
undesirable regularities in data.

Annotator heuristic traces capture global behavioral trends per annotator. For
each annotator and heuristic, we average the heuristic values across all of
the annotator’s examples, forming a matrix of annotators and their average
heuristic values.

#### Principal components of heuristics:

We also evaluate if a low-dimensional representation of an annotator’s
heuristic trace is useful for predicting data quality. We compute the first
principal component of this matrix to simultaneously consider multiple
heuristic indicators.

###  4.4 Cognitive Reflection Test

Figure 1: Correlations of annotator scores on the CRT and their average
features values for each heuristic. Feature names, left, correspond to feature
names from Table 1. The CRT3 includes the original questions from Frederick
(2005) and CRT7 includes 4 more questions from Toplak et al. (2014). The black
boxes indicate the featurization with the highest average correlations for the
heuristic. $*$ indicates $p<0.01$ and ^ indicates $p<0.1$.

Although we cannot determine whether an annotator is definitively using a
heuristic, we can probe if our features correlate with heuristic-seeking
tendencies in annotators. Previous work in cognitive psychology has designed
tests measuring such tendencies. These help us validate the psychological
plausibility of our features, ensuring they are generally applicable.

Perhaps the best known test of heuristic-seeking tendencies is the Cognitive
Reflection Test (CRT) Frederick (2005). The test has 3 questions, but we
instead use the 7-item CRT from Toplak et al. (2014) to find more variance
among annotators. The numerical CRT requires mathematical reasoning and
previous work has highlighted that its results might be conflated with
mathematical reasoning capabilities. Further, since our task requires writing,
we also perform the verbal CRT Sirota et al. (2021). This test has 9
items444We exclude a question that requires cultural knowledge., and is known
to correlate well with the numerical CRT, and other indicators of cognitive
capabilities. The questions in these tests are provided in Appendix B. 555The
use of the CRT has issues due to repeated exposure Stieger and Reips (2016);
Haigh (2016), so we ensured that the names/quantities are different from the
ones used in the original questions. We also emphasize that the CRT does not
provide interpretability into the mechanism of the heuristics, whereas our
individual heuristic features do.

We asked annotators who completed at least 5 question writing examples to do
two surveys asking logical questions (CRT-7 and Verbal CRT). 49 of 59
annotators completed the surveys. We then compute Pearson correlations between
annotator accuracies on the three versions of the CRT, and values in their
heuristic traces, shown in Figure 1. The results indicate that our
featurizations have significant, medium correlations with the CRTs, and the
PCA projection, which captures multiple heuristics, has the highest
correlations. For the sake of further analysis, for each feature group, we use
the feature that has the highest average correlations with the CRT tests
(enclosed in black boxes in Figure 1).

Figure 2: Precision of labeling heuristic examples $H_{k}$ as solvable by
biased models, when the set $H_{k}$ is formed by examples from the $k^{th}$
percentile of heuristic-seeking annotators. Figure 3: Precision of labeling
heuristic examples $H_{k}$ as solvable by biased models on data from Sugawara
et al. (2022).

##  5 Biased Model Solvability

Annotator heuristic traces are cognitively plausible measures that we
hypothesize are indicators of large, potentially undesirable patterns
annotators impart on data. To verify this, we test if examples created by
heuristic-seeking annotators are more easily solvable by biased models.

We consider heuristic examples as examples from those annotators who score
highly on our heuristic indicator features. Given the initial set of examples
$D$, we distinguish a subset as heuristic, $H_{k}$, formed by all examples
from annotators in the top $k$% of average heuristic use across all
annotators. We form such a subset independently for all heuristic indicator
features we consider.666We exclude those annotators who wrote less than 5
examples and exclude all invalid examples. When $H_{k}$ is formed from the top
quartile (k=25), 68% of annotators have examples included in at least 1
heuristic set, and 14% in for at least 4/6 heuristic sets. We find that few
annotators never use heuristics.

Next, we evaluate how well biased models perform on heuristic subsets
($H_{k}$) compared to the remaining examples, $D\setminus H_{k}$. We evaluate
a few biased models, trained to use unreliable heuristics, on examples created
with or without heuristics. Below we describe the biased models we use. In all
cases, we train or finetune models on QA data from Nangia et al. (2021) and
evaluate them on our data. For hyperparameter settings, please see Appendix C.

#### Lexical Overlap Model (overlap).

We train a logistic regression classifier by building upon features from the
bias-only model from Clark et al. (2019). Assuming the concatenated passage
and question are the context for each option, we use the following features:
1) is the option a subsequence of context, 2) do all words in the option exist
in context, 3) the fraction of words in the option that exist in context, 4)
the log of length difference between the context and the option, 5) the
average and maximum of minimum distance between each context word with each
option word using 300-dimensional fastText embeddings Joulin et al. (2017). We
then pick the option with the highest probability as the model prediction. The
model achieves an accuracy of 42.27% on $D$.

#### Partial Input Models.

As a benchmark for diagnosing the collected data, we consider several partial
input models. These include no passage (no_passage), no question (no_ques),
first & last sentence of passage only (fl_passage). We use a RoBERTa-Large Liu
et al. (2019) model initially finetuned on RACE Lai et al. (2017) and further
trained on the baseline data from Nangia et al. (2021). These models achieve
accuracies of 42.44%, 59.49%, and 55.98% on $D$, respectively, demonstrating
better than random performance.

#### Human heuristic solvability (human_biased).

In addition to biased models, we also consider an implicit notion of example
difficulty from a biased human. Specifically, we evaluate whether a human can
answer an example just by skimming the passage. We use an interface where a
passage is only visible for 30 seconds, after which, a human needs to answer
the question. One of the authors conducted this annotation for the collected
examples and achieved an accuracy of 79.79%.

#### Results.

Figure 2 shows the precision of $H_{k}$ being solvable by biased models, as
$k$ is varied. As we can see from the plots, there is a downward trend as the
percentile is increased for all heuristics. The features for the availability
and representative heuristic, and the PCA projection are particularly
effective. This suggests that strongly heuristic-seeking annotators are more
likely to create examples solvable by biased models.

#### Other non-Wikipedia domains.

To test if the heuristics we considered are indicative of solvability by
biased models in domains other than Wikipedia, we repeated our analysis on
1,982 examples from the standard data collection setting in Sugawara et al.
(2022), who collected questions for passages from many different sources. The
precision plot is shown in Figure 3. With the exception of serial-position,
heuristic-seeking features identify annotators that create examples more
easily solvable by biased models in these domains too.777The serial-position
feature might be more effective for Wikipedia because they tend to be more
factual, which makes it easier to form questions using the first or last
sentence.

Figure 4: Pearson correlations of annotators’ average heuristic values and
accuracies with biased models on their annotated examples. $*$ and ^ indicate
$p<0.01$ and $p<0.1$.

#### As a predictor of bias.

In addition to evaluating the predictiveness of annotator heuristic features
at the extreme, we also evaluated whether heuristic features are predictive of
solvability by biased models across annotators. Specifically, we calculated
Pearson correlations between annotators’ average heuristic values, and the
accuracies of biased models on their examples, in Figure 4. These correlations
are not strong for the satisficing heuristics, but we do notice some
significant, medium correlations for the other heuristics we studied.
Importantly, we contrast this with the same correlations measured over the
entire pool of data (without averaging per annotator). Those correlations,
shown in Figure 6 in the Appendix, are much weaker showing the value of our
annotator-level measures.

#### Model generalization across annotators.

Previous work showed that models do not generalize well to annotator-based
random splits of crowdsourced datasets, suggesting models might be learning
annotator-specific biases Geva et al. (2019). We suspect that generalization
might deteriorate when models are trained on heuristic-seeking annotators, as
models could more easily specialize to their examples. Hence, we ask whether
heuristic-based splits (heuristic) lead to worse performance than random
annotator splits (random).

While controlling the number of training examples, we trained models on
examples from heuristic-seeking annotators or a random set of annotators, and
test on the remaining examples. For heuristic-based splits, we train on
examples in $H_{33}$, the top 33% of heuristic-seeking annotators for a
heuristic indicator. For random annotator splits, we resampled splits with 3
random seeds and report means. In addition, we trained models on random splits
of the same training size (random-pooled), where data is not split by
annotator. The accuracies on these splits are shown in Table 2. We find
generalization is poorer for almost all of the heuristic-based annotator
splits compared to random annotator splits. This suggests that heuristic-based
splits can serve as natural challenge sets and inadvertently sampling
heuristic-seeking annotators for training may not generalize well.

Heuristic | heuristic | random | random-pooled | n
---|---|---|---|---
lowtime | $84.27$ | $87.42_{\pm 1.17}$ | $87.14_{\pm 0.64}$ | 394
loweffort | $85.06$ | $88.56_{\pm 1.01}$ | $86.27_{\pm 1.75}$ | 415
first_option | $87.33$ | $87.92_{\pm 0.85}$ | $88.12_{\pm 1.63}$ | 341
serial_position | $84.78$ | $88.07_{\pm 0.84}$ | $88.43_{\pm 0.6}$ | 389
word_overlap | $86.86$ | $86.64_{\pm 2.16}$ | $87.50_{\pm 0.67}$ | 362
copying | $83.75$ | $86.94_{\pm 1.78}$ | $87.47_{\pm 1.02}$ | 332
pca | $83.40$ | $86.94_{\pm 0.77}$ | $87.52_{\pm 0.81}$ | 385
Table 2: Performance on heuristic-based, random annotator splits and random
splits with the same training set size. We performed 3 runs on the randomly
sampled splits, and report means and standard deviations. Figure 5: Percentage
difference of examples in heuristic set, $H_{25}$, and the remaining examples,
$D\setminus H_{25}$, labeled as having a qualitative property. Examples in the
heuristic set are less valid & require more word matching based on explicitly
stated information.

##  6 Influencers of heuristic behavior

Next, we aim to understand what role the annotation pipeline plays in
influencing heuristic use among annotators. Various factors have been shown to
determine example quality in crowdsourcing. These include task difficulty,
incentives, annotator ability, motivation and fatigue Krosnick (1991); Yan et
al. (2010). We looked at how such markers influence heuristic use among
annotators.

We considered two types of measures that could indicate difficulty: passage
length (number of tokens) and inverse entity count (doc length / number of
named entities) in the passage. Longer documents, with fewer named entities,
might provide context that is harder to form questions about. Further, having
completed more examples could make an annotator fatigued and/or gain expertise
at the task. Hence, we also used the sequence index of each example for an
annotator. We computed Pearson correlations between these indicators and the
heuristic values for each annotator, and averaged the correlations across
annotators. Our results are summarized in Table 3. We find that neither of
these factors show significant correlations with heuristic features among
annotators.

Heuristic | passage length | entity | index
---|---|---|---
lowtime | 0.12 | 0.16 | -0.09
loweffort | 0.09 | -0.03 | -0.01
first_option | 0.08 | 0.00 | 0.02
serial_position | -0.09 | -0.05 | -0.03
copying | 0.13 | 0.06 | -0.05
Table 3: Correlations between heuristic values and factors, averaged across
annotators.

##  7 Qualitative Analysis

To better understand the differences in data produced by heuristic-seeking
annotators, and otherwise, we conducted a comprehensive qualitative analysis
of all our data. We annotate questions with properties inspired from previous
work Lai et al. (2017); Trischler et al. (2017); Sugawara et al. (2018, 2022)
along the following dimensions: validity (is the question answerable given the
context in the passage), context (how much context from the passage is needed
to answer the question), and comprehension type (what kinds of comprehension
are needed to answer the question). Each question can have multiple labels.
For a detailed description of these labels, please refer to Appendix E.

#### Results.

Figure 5 presents the results of our annotation. We show the differences in
the percentage of examples in the heuristic set, $H_{25}$, and the remaining
examples, $D\setminus H_{25}$. First, examples in the non-heuristic set are
more likely to be valid, and less likely to be unsolvable compared to the
heuristic set. Further, we find that examples in the heuristic set often
require simple word matching and paraphrasing, while the ones in the non-
heuristic set, are more likely to require multi-sentence reasoning. In terms
of comprehension type, we find that heuristic examples are more likely to be
answerable using information explicitly stated in the passage. On the other
hand, non-heuristic examples are more likely to require implicit inference.
These results suggest there are significant qualitative differences in
examples from heuristic-seeking annotators.

##  8 Discussion

Our work measures the implications of annotators’ potential use of cognitive
heuristics in data quality. The analyses we present suggest that models are
indirectly influenced by heuristic use and that previous observations, such as
the success of partial input models, is a consequence. While many such
consequences of heuristic use appear to be negative, we believe that this
judgement should be left up to practical applications that use the data. We
propose a fruitful direction for characterizing what models learn from data by
considering annotator behaviors.

Practically, it is an open question as to how we can control downstream data
using annotator heuristic traces. Instead, we propose that future annotation
efforts minimally track indicators of heuristic usage, using task-specific
features, in an effort to document how they are reflected in the collected
data and trained models.

##  9 Limitations

One limitation of our study is that we analyze the implications of heuristic-
seeking behavior in annotators for one task. Future work could consider
extending this methodology to many annotation tasks. For example, in sentence-
pair classification tasks such as textual entailment, or in annotation of
machine translation or summarization datasets, annotator heuristics could be
useful in determining the quality of data and the biases embedded in them. To
find stronger signals in the annotator heuristic traces, future work could
consider training models to featurize heuristics.

## Acknowledgements

This work is supported in part by an Allen Institute for AI Young Investigator
award and National Science Foundation grant SES-1847794. We thank the
anonymous reviewers, and the annotators who participated in our study. We also
thank the University of Pennsylvania NLP group for helpful discussions.

## References

  * Bartolo et al. (2022) Max Bartolo, Tristan Thrush, Sebastian Riedel, Pontus Stenetorp, Robin Jia, and Douwe Kiela. 2022.  Models in the loop: Aiding crowdworkers with generative annotation assistants.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3754–3767, Seattle, United States. Association for Computational Linguistics.
  * Bastings et al. (2021) Jasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm, and Katja Filippova. 2021.  " will you find these shortcuts?" a protocol for evaluating the faithfulness of input salience methods for text classification.  _arXiv preprint arXiv:2111.07367_.
  * Bowman et al. (2020) Samuel R. Bowman, Jennimaria Palomaki, Livio Baldini Soares, and Emily Pitler. 2020\.  New protocols and negative results for textual entailment data collection.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8203–8214, Online. Association for Computational Linguistics.
  * Clark et al. (2019) Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019.  Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4069–4082, Hong Kong, China. Association for Computational Linguistics.
  * Draws et al. (2021) Tim Draws, Alisa Rieger, Oana Inel, Ujwal Gadiraju, and Nava Tintarev. 2021.  A checklist to combat cognitive biases in crowdsourcing.  In _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ , volume 9, pages 48–59.
  * Ebbinghaus (1964) Hermann Ebbinghaus. 1964.  Memory: A contribution to experimental psychology.  _New York, Dover Publications_.
  * Eickhoff (2018) Carsten Eickhoff. 2018.  Cognitive biases in crowdsourcing.  In _Proceedings of the eleventh ACM international conference on web search and data mining_ , pages 162–170.
  * Frederick (2005) Shane Frederick. 2005.  Cognitive reflection and decision making.  _Journal of Economic perspectives_ , 19(4):25–42.
  * Garcia-Retamero et al. (2009) Rocio Garcia-Retamero, Masanori Takezawa, and Gerd Gigerenzer. 2009.  Does imitation benefit cue order learning? _Experimental Psychology_ , 56(5):307–320.
  * Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020.  Evaluating models’ local decision boundaries via contrast sets.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1307–1323, Online. Association for Computational Linguistics.
  * Gardner et al. (2021) Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, and Noah A. Smith. 2021.  Competency problems: On finding and removing artifacts in language data.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1801–1813, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Geirhos et al. (2020) Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020.  Shortcut learning in deep neural networks.  _Nature Machine Intelligence_ , 2(11):665–673.
  * Geva et al. (2019) Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019.  Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1161–1166, Hong Kong, China. Association for Computational Linguistics.
  * Gigerenzer and Gaissmaier (2011) Gerd Gigerenzer and Wolfgang Gaissmaier. 2011.  Heuristic decision making.  _Annual review of psychology_ , 62:451–482.
  * Goyal et al. (2018) Tanya Goyal, Tyler McDonnell, Mucahid Kutlu, Tamer Elsayed, and Matthew Lease. 2018\.  Your behavior signals your reliability: Modeling crowd behavioral traces to ensure quality relevance annotations.  In _Sixth AAAI Conference on Human Computation and Crowdsourcing_.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018.  Annotation artifacts in natural language inference data.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.
  * Haigh (2016) Matthew Haigh. 2016.  Has the standard cognitive reflection test become a victim of its own success? _Advances in cognitive psychology_ , 12(3):145.
  * Han et al. (2020) Xiaochuang Han, Byron C. Wallace, and Yulia Tsvetkov. 2020.  Explaining black box predictions and unveiling data artifacts through influence functions.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5553–5563, Online. Association for Computational Linguistics.
  * Hube et al. (2019) Christoph Hube, Besnik Fetahu, and Ujwal Gadiraju. 2019.  Understanding and mitigating worker biases in the crowdsourced collection of subjective judgments.  In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ , pages 1–12.
  * Joulin et al. (2017) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017.  Bag of tricks for efficient text classification.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 427–431, Valencia, Spain. Association for Computational Linguistics.
  * Kahneman and Tversky (1972) Daniel Kahneman and Amos Tversky. 1972.  Subjective probability: A judgment of representativeness.  _Cognitive psychology_ , 3(3):430–454.
  * Kaushik and Lipton (2018) Divyansh Kaushik and Zachary C. Lipton. 2018.  How much reading does reading comprehension require? a critical investigation of popular benchmarks.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5010–5015, Brussels, Belgium. Association for Computational Linguistics.
  * Krosnick (1991) Jon A Krosnick. 1991.  Response strategies for coping with the cognitive demands of attitude measures in surveys.  _Applied cognitive psychology_ , 5(3):213–236.
  * Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017.  RACE: Large-scale ReAding comprehension dataset from examinations.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 785–794, Copenhagen, Denmark. Association for Computational Linguistics.
  * Lertvittayakumjorn and Toni (2021) Piyawat Lertvittayakumjorn and Francesca Toni. 2021.  Explanation-based human debugging of nlp models: A survey.  _Transactions of the Association for Computational Linguistics_ , 9:1508–1528.
  * Liu et al. (2022) Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2022.  Wanli: Worker and ai collaboration for natural language inference dataset creation.  _arXiv preprint arXiv:2201.05955_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448, Florence, Italy. Association for Computational Linguistics.
  * Murdock Jr (1962) Bennet B Murdock Jr. 1962.  The serial position effect of free recall. _Journal of experimental psychology_ , 64(5):482.
  * Nangia et al. (2021) Nikita Nangia, Saku Sugawara, Harsh Trivedi, Alex Warstadt, Clara Vania, and Samuel R. Bowman. 2021.  What ingredients make for an effective crowdsourcing protocol for difficult NLU data collection tasks? In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1221–1235, Online. Association for Computational Linguistics.
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.  Adversarial NLI: A new benchmark for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4885–4901, Online. Association for Computational Linguistics.
  * Niven and Kao (2019) Timothy Niven and Hung-Yu Kao. 2019.  Probing neural network comprehension of natural language arguments.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4658–4664, Florence, Italy. Association for Computational Linguistics.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Pezeshkpour et al. (2022) Pouya Pezeshkpour, Sarthak Jain, Sameer Singh, and Byron Wallace. 2022.  Combining feature and instance attribution to detect artifacts.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1934–1946, Dublin, Ireland. Association for Computational Linguistics.
  * Ribeiro and Lundberg (2022) Marco Tulio Ribeiro and Scott Lundberg. 2022.  Adaptive testing and debugging of NLP models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3253–3267, Dublin, Ireland. Association for Computational Linguistics.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4902–4912, Online. Association for Computational Linguistics.
  * Rzeszotarski and Kittur (2011) Jeffrey M Rzeszotarski and Aniket Kittur. 2011.  Instrumenting the crowd: using implicit behavioral measures to predict task performance.  In _Proceedings of the 24th annual ACM symposium on User interface software and technology_ , pages 13–22.
  * Sap et al. (2022) Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022.  Annotators with attitudes: How annotator beliefs and identities bias toxic language detection.  In _Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL)_ , Seattle, USA.
  * Shah and Oppenheimer (2008) Anuj K Shah and Daniel M Oppenheimer. 2008.  Heuristics made easy: an effort-reduction framework. _Psychological bulletin_ , 134(2):207.
  * Simon (1956) Herbert A Simon. 1956.  Rational choice and the structure of the environment. _Psychological review_ , 63(2):129.
  * Sirota et al. (2021) Miroslav Sirota, Chris Dewberry, Marie Juanchich, Lenka Valuš, and Amanda C Marshall. 2021.  Measuring cognitive reflection without maths: Development and validation of the verbal cognitive reflection test.  _Journal of Behavioral Decision Making_ , 34(3):322–343.
  * Stieger and Reips (2016) Stefan Stieger and Ulf-Dietrich Reips. 2016.  A limitation of the cognitive reflection test: familiarity.  _PeerJ_ , 4:e2395.
  * Sugawara et al. (2018) Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018.  What makes reading comprehension questions easier? In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4208–4219, Brussels, Belgium. Association for Computational Linguistics.
  * Sugawara et al. (2022) Saku Sugawara, Nikita Nangia, Alex Warstadt, and Samuel Bowman. 2022.  What makes reading comprehension questions difficult? In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6951–6971, Dublin, Ireland. Association for Computational Linguistics.
  * Toplak et al. (2014) Maggie E Toplak, Richard F West, and Keith E Stanovich. 2014.  Assessing miserly information processing: An expansion of the cognitive reflection test.  _Thinking & Reasoning_, 20(2):147–168.
  * Trischler et al. (2017) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017.  NewsQA: A machine comprehension dataset.  In _Proceedings of the 2nd Workshop on Representation Learning for NLP_ , pages 191–200, Vancouver, Canada. Association for Computational Linguistics.
  * Tversky and Kahneman (1973) Amos Tversky and Daniel Kahneman. 1973.  Availability: A heuristic for judging frequency and probability.  _Cognitive psychology_ , 5(2):207–232.
  * Tversky and Kahneman (1974) Amos Tversky and Daniel Kahneman. 1974.  Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty. _science_ , 185(4157):1124–1131.
  * Wang et al. (2022) Xuezhi Wang, Haohan Wang, and Diyi Yang. 2022.  Measure and improve robustness in nlp models: A survey.  _Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
  * Whelan (2008) Robert Whelan. 2008.  Effective analysis of reaction time data.  _The Psychological Record_ , 58(3):475–482.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu et al. (2021) Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2021.  Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6707–6723, Online. Association for Computational Linguistics.
  * Yan et al. (2010) Yan Yan, Rómer Rosales, Glenn Fung, Mark Schmidt, Gerardo Hermosillo, Luca Bogoni, Linda Moy, and Jennifer Dy. 2010.  Modeling annotator expertise: Learning when everybody knows a bit of something.  In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ , pages 932–939. JMLR Workshop and Conference Proceedings.

##  Appendix A Crowdsourcing setup

For annotators to participate in our task, they needed to have an acceptance
rate greater than or equal to 98% and have at least 1000 approved HITs. In
addition, we required annotators to be located in US, UK or Canada. We
estimated each HIT to take approximately 15 minutes, and paid $4 per HIT ($15
/ hr). Figure 7 shows the interface presented to the annotators for data
collection.

##  Appendix B Cognitive Reflection Tests

We list the questions used in the numerical CRT and the verbal CRT in Table 4
and Table 5 respectively. The first 3 questions in Table 4 correspond to the
original CRT from Frederick (2005).

Question |  Intuitive Answer |  Correct Answer
---|---|---
1) A carpet and a lamp cost $450 in total. The carpet costs $400 more than the lamp. How much does the lamp cost? |  $50 |  $25
2) It takes 10 computers 10 minutes to run 10 programs. How many minutes does it take 500 computers to run 500 programs? |  500 |  10
3) There is a patch of lily pads in a pond. The patch doubles in size every day. If it takes 100 days for the patch to cover the entire pond, how many days would it take to cover half the pond? |  50 |  99
4) If Jason can drink one barrel of water in 6 days, and Jen can drink one barrel of water in 12 days, how long would it take them to drink one barrel of water together? |  9 |  4
5) Aidan received both the 25th highest and the 25th lowest mark in the class. How many students are in the class? |  50 |  49
6) A farmer buys a sheep for $500, sells it for $600, buys it back for $700, and sells it finally for $800. How much has he made? |  $100 |  $200
7) Ramona decided to invest $5,000 in the stock market early in 2008. Six months after she invested, on July 17, the stocks she had purchased were down 50%. Fortunately for Ramona, from July 17 to October 17, the stocks she had purchased went up 75%. At this point, does Ramona have a) the same amount of money as when she invested, b) more money, c) less money ? (Respond with a, b or c) |  b |  c
Table 4: Questions in the numerical CRT. Question |  Intuitive Answer |  Correct Answer
---|---|---
1) Angie’s father has 5 daughters but no sons—Nana, Nene, Nini, Nono. What is the fifth daughter’s name probably? |  Nunu |  Angie
2) If you were running a race, and you passed the person in 5th place, what place would you be in now? |  4th |  5th
3) It is a stormy night and a plane takes off from JFK airport in New York. The storm worsens, and the plane crashes - half lands in the United States, the other half lands in Canada. In which country do you bury the survivors? |  USA |  we do not bury survivors
4) A monkey, a squirrel, and a bird are racing to the top of a coconut tree. Who will get the banana first, the monkey, the squirrel, or the bird? |  bird |  there is no banana on a coconut tree
5) In a one-storey pink house, there was a pink person, a pink cat, a pink fish, a pink computer, a pink chair, a pink table, a pink telephone, a pink shower—everything was pink! What colour were the stairs probably? |  pink |  no stairs in a one-storey house
6) The wind blows west. An electric train runs east. In which cardinal direction does the smoke from the locomotive blow? |  west |  no smoke from an electric train
7) If you have only one match and you walk into a dark room where there is an oil lamp, a newspaper and wood— which thing would you light first? |  oil lamp |  match
8) Would it be ethical for a man to marry the sister of his widow? |  no |  not possible
9) Which sentence is correct: (a) ‘the yolk of the egg are white’ or (b) ‘the yolk of the egg is white’? |  b |  the yolk is yellow
Table 5: Questions in the verbal CRT.

##  Appendix C Hyperparameter Settings

#### Lexical overlap model.

The logistic regression was trained with C=100 and a maximum of 100 iterations
for convergence with the scikit-learn library Pedregosa et al. (2011).

#### Partial input models.

The partial input models were trained with a learning rate of 1e-4 and batch
size of 1, for 4 epochs and all the default hyperparameters in the multiple-
choice QA example in the Transformers library Wolf et al. (2020). These
experiments took approximately a week of compute time on a single Quadro RTX
6000 GPU.

##  Appendix D Correlations with pooled data

Figure 6: Pearson correlations of average heuristic values and biased model
solvability, pooled across all examples. We exclude word overlap since it is
computed across all examples of an annotator and is not a sample-level
measure. $*$ indicates $p<0.01$ and ^ indicates $p<0.1$.

In Figure 6, we show correlations between heuristic features and biased model
accuracies when all examples are pooled together. We contrast this with the
annotator-wise plots shown in Figure 4.

Figure 7: Annotation interface used for data collection.

##  Appendix E Question Annotation Scheme

We describe the annotation scheme used to label examples for the analysis in
section 7. In addition, we show a breakdown of those results across all
heuristic features in Figure 8.

#### Validity.

We annotate whether examples are answerable or not using the following labels:

  1. 1.

Unsolvable: It is not possible to answer the question given the context in the
passage and question, or the question is underspecified or incoherent.

  2. 2.

Incorrect: The answer is marked incorrectly.

  3. 3.

Ambiguous: The question does not have a unique correct answer.

  4. 4.

Valid: The question can be reasonably answered from the passage.

#### Context.

To understand how much context from the passage is needed to answer the
question, we label questions using the following labels:

  1. 1.

Word matching: The question matches a span in the passage, and the answer is
easily extractable by matching spans.

  2. 2.

Paraphrasing: The question paraphrases information in exactly one sentence in
the passage, and the answer can be retrieved from it.

  3. 3.

Single-sentence reasoning: The question can be answered by exactly one
sentence in the passage, but requires a conceptual overlap, or performing some
other form of inference.

  4. 4.

Multi-sentence reasoning: The question can only be answered by synthesizing
information from multiple sentences in the passage. This excludes just
performing coreference.

  5. 5.

Coreferential Reasoning: The question requires performing coreference.

#### Comprehension Type.

To determine what kinds of comprehension are required to answer the question,
we label examples with the following labels:

  1. 1.

Math/numerical: Questions that require mathematical or numerical reasoning.

  2. 2.

Whole: Questions that require a complete understanding of the passage or ask
about the author’s opinion on the passage.

  3. 3.

Factuality: Questions asking about truthfulness of the statements presented in
the question or the options (e.g., questions of the form "which of the
following is true / false?", or "is it true/false that ..")

  4. 4.

Spatial/temporal: Requires understanding of location and temporal order of
events.

  5. 5.

Explicit: Asks about information (facts, events or entities) stated in the
passage explicitly or in a paraphrased manner. These shouldn’t require much of
a concept jump.

  6. 6.

Implicit: Asks about information not directly stated, but which can be
inferred through commonsense, causality, numerical or other types of
inference.

  7. 7.

Negation: Questions which are phrased in the form of a negation (for e.g.
using keywords like "not" and "without").

Figure 8: Breakdown of the question annotation results from section 7 when
percentile k=25.


∎

e1e-mail<EMAIL_ADDRESS>e2e-mail<EMAIL_ADDRESS>_(corresponding
author)_ e3e-mail<EMAIL_ADDRESS>
11institutetext: Department of Physics, Liaoning Normal University, Dalian
116029, China

# Production of axion-like particles via vector boson fusion at future
electron-positron colliders

Chong-Xing Yue e1,addrA  Hua-Ying Zhang e2,addrA  Han Wang e3,addrA

(Received: date / Revised version: date)

###### Abstract

One kind of particularly interesting pseudoscalar particles, called axion-like
particles (ALPs), have rich physical phenomenology at high- and low-energy
collider experiments. After discussing most of single production channels of
ALP at electron-positron colliders, we investigate the possibility of
detecting this kind of new particles through the W+W- fusion process e+e-
$\rightarrow$ $\overline{\nu}_{e}$$\nu_{e}a$ $(\rightarrow\gamma\gamma)$ at
the CLIC. The 3$\sigma$ and 5$\sigma$ bounds on the ALP parameter space at the
three energy stages of the CLIC are obtained. We find that the bounds given by
the CLIC are complementary to the existing experiments exclusion regions.

###### Keywords:

axion-like particles vector boson fusion electron-positron collider

###### pacs:

††journal: Eur. Phys. J. C

##  1 Introduction

After discovery of the Higgs boson in the CMS and ATLAS experiments
Aad:2012tfa ; Chatrchyan:2012ufa , experiments at the LHC have made great
achievements and demonstrated that the standard model (SM) is a correct model
to explain most of observed phenomena at the electroweak scale. However,
deviations of experimental measurements from the SM predictions at or below
the percent level are beyond the achievable precision at the LHC and is
expected to be performed at the planned e+e- colliders with high luminosity
and energy. It is well known that, compared to the LHC, the future e+e-
colliders, such as the ILC Djouadi:2007ik ; Baer:2013cma ; Moortgat-
Picka:2015yla ; Aihara:2019gcq ; Bambade:2019fyw , CLIC Battaglia:2004mw ;
Roloff:2018dqu , FCC-ee Gomez-Ceballos:2013zzn ; Abada:2019zxq , and CEPC
CEPCStudyGroup:2018rmc ; CEPCStudyGroup:2018ghi ; An:2018dwb  have higher
luminosity and more clean experimental environment, which can not only study
the SM observables at unprecedented accuracy, but would be very useful to
discover the evidence of new physics beyond the SM.

The discovery of the Higgs boson has stimulated interest in searching for
additional (pseudo) scalar particles in high- and low-energy experiments.
Axion-like particles (ALPs) are pseudoscalar bosons that naturally appear in
many extensions of the SM as pseudo Nambu-Goldstone bosons arising in explicit
global symmetry breaking Peccei:1977hh ; Weinberg:1977ma ; Wilczek:1977pj ;
Kim:1979if ; Shifman:1979if ; Dine:1981rt ; Zhitnitsky:1980tq . The properties
of ALPs, such as their masses and coupling strengths to the SM particles, are
model-dependent and have been extensively investigated. High-energy collider
experiments provide possibilities to explore ALPs with masses about $M_{a}$
$\geq$ 1 GeV Bauer:2017ris ; Bauer:2018uxu ; Fortin:2021cog ;
dEnterria:2021ljz ; Agrawal:2021dbo ; Brivio:2017ije ; Knapen:2016moh ;
Baldenegro:2019whq ; Coelho:2020saz ; Coelho:2020syp ; Goncalves:2021pdc . The
constraints on a wide range of the ALP parameter space are obtained using the
available data, such as the LEP, Tevatron and LHC data, the future prospects
of discovering ALPs at running or future collider experiments are studied. For
instance, searches for ALPs via photon fusion are investigated at the LHC and
e+e- colliders Jaeckel:2012yz ; Baldenegro:2018hng ; Florez:2021zoo ;
Marciano:2016yhf ; Inan:2020aal ; Inan:2020kif ; Zhang:2021sio ;
Buttazzo:2018qqp ; Gavela:2019cmq . The capabilities of these colliders to
probe the ALP parameter space are discussed and the expected bounds on the
ALP-photon coupling for a wide range of masses are obtained.

ALPs can also be produced via massive vector boson fusion (VBF) processes
(i.e. W+W--fusion and ZZ-fusion processes) at high-energy e+e- colliders
Georgi:1986df . Since the production cross sections of these processes depend
logarithmically on the center-of-mass (c.m.) energy, and are generally smaller
than those of other processes at low-energy e+e- colliders, there are very few
studies about ALP productions via VBF processes with V = W±, Z. In this work,
we will consider the possibilities of detecting ALPs via VBF processes at
high-energy e+e- colliders like the CLIC. We will focus on ALPs that only
couple to the electroweak gauge bosons and perform a careful investigation of
the ALP signals at high-energy e+e- colliders from VBF processes. The relevant
Feynman diagrams are shown in Fig. 1. Our numerical results show that VBF
processes can extend the e+e- sensitivity to a region of the ALP parameter
space that is not covered by other experiments.

The rest of this paper is organized as follows. After summarizing the
effective description of ALP interactions with the electroweak gauge bosons,
we calculate and compare the production cross sections of ALPs via e+e-
annihilation and VBF processes at e+e- colliders in Sec. 2. Based on the
details of the analysis of the ALP signals and the relevant SM backgrounds
from W+W- fusion process, sensitivity projections of the CLIC to the ALP
parameter space are presented in Sec. 3. We report our conclusions in Sec. 4.

##  2 Single production of ALP at e+e- colliders

The ALP interactions with the SM fermions and gauge bosons arise through five-
dimensional operators, and their masses can be treated independently of their
couplings Georgi:1986df , which can be described via an effective Lagrangian.
The effective interactions of ALP with electroweak gauge bosons, which are
related to our calculation, are given by the following dimension-5 effective
Lagrangian Brivio:2017ije :

$\displaystyle\begin{aligned} \mathcal{L}_{\text{eff }}^{D\leq
5}&=\frac{1}{2}\left(\partial_{\mu}a\right)\left(\partial^{\mu}a\right)-\frac{M_{a}^{2}}{2}a^{2}+C_{WW}\frac{a}{f_{a}}W_{\mu\nu}^{A}\tilde{W}^{\mu\nu,A}\\\
&+C_{BB}\frac{a}{f_{a}}B_{\mu\nu}\tilde{B}^{\mu\nu},\end{aligned}$ (1)

where ${X}_{\mu\nu}$ denotes the field strength tensor for $SU(2)_{L}$ or
$U(1)_{Y}$,
${\tilde{X}}^{\mu\nu}=\frac{1}{2}\varepsilon^{\mu\nu\alpha\beta}X_{\alpha\beta}$
with $\varepsilon^{0123}=1$ and $X\in\\{B,W\\}$. The ALP field and mass are
denoted by $a$ and $M_{a}$ , respectively. The interaction of the ALP with two
photons is contributed by the last two terms of Eq. 1. The dimensionful
couplings $g_{a\gamma\gamma}$, $g_{aWW}$, $g_{a\gamma Z}$ and $g_{aZZ}$,
control the coupling strength of the interactions between ALP and electroweak
gauge bosons ($W^{\pm}$, Z, $\gamma$), which are given by

$\displaystyle\begin{aligned}
\@add@centering\centering\quad\quad&g_{a\gamma\gamma}=\frac{4}{f_{a}}(c_{\theta_{W}}^{2}C_{BB}+s_{\theta_{W}}^{2}C_{WW}),\\\
&g_{aWW}=\frac{4}{f_{a}}C_{WW},\quad~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\\\
&g_{aZZ}=\frac{4}{f_{a}}(s_{\theta_{W}}^{2}C_{BB}+c_{\theta_{W}}^{2}C_{WW}),\\\
&g_{a\gamma Z}=\frac{4}{f_{a}}s_{2\theta_{W}}(C_{WW}-C_{BB}),\end{aligned}$
(2)

where $s_{\theta_{W}}=\sin\theta_{W}$, $c_{\theta_{W}}=\cos\theta_{W}$ with
$\theta_{W}$ being the weak mixing angle. To simplify our analysis, we will
assume CWW = CBB in this work and there are $g_{a\gamma Z}$ = 0,
$g_{a\gamma\gamma}$ = $g_{aWW}$ = $g_{aZZ}$111In this case, Ref. Wang:2021uyb
has investigated searching for ALPs via ALP-strahlung process at the LHC.. In
this case, ALPs can be singly produced via e+e- annihilation and VBF processes
at e+e- colliders. We use FeynRules Alloul:2013bka  to generate the UFO model
file corresponding to the effective Lagrangian Eq. 1. The cross sections of
e+e- annihilation and VBF processes at e+e- colliders are calculated, which
are shown in Fig. 2. One can see from Fig. 2 that the ALP-strahlung processes
($\gamma a$ and $Za$) are the main production channels. However, their cross
sections become independent of $s$ in the high-energy limit $M^{2}_{a}\ll s$
and thus saturate quite soon and flatten out. The cross sections of VBF
processes depend only logarithmically on $s$. Therefore, the cross section of
W+W- fusion process grows with $s$ increasing and can reach sizable values at
high energies. In principle, the cross section of the $ZZ$ fusion process
follows the same trend as that of the W+W- fusion process, while its value is
about an order of magnitude smaller than that of the W+W- fusion process, due
to the fact that the neutral current couplings are smaller than the charged
current couplings. The $\gamma\gamma$ fusion process has been extensively
discussed in Ref. Zhang:2021sio . The cross section of the ZZ fusion process
e+e- $\rightarrow$ e+e-$a$ is the smallest. In Ref. Azzurri:2021nmy , the
similar conclusion is given by exploring the mass and cross section of Higgs
with ultimate precision at the FCC-ee.

ALPs have been investigated through the ALP-strahlung processes e+e-
$\rightarrow$ $V$$a$ $(a\rightarrow\gamma\gamma)$, where $V$ = $\gamma,Z$
Bauer:2018uxu ; Jaeckel:2015jla ; Mimasu:2014nea . In Ref. Bauer:2018uxu , the
ALP-strahlung processes have also been studied at the CLIC, but the author
assumes $C_{WW}$ = 0, considering $\gamma Za$ coupling, which is opposite to
our work. There are previous works about the ALP single production induced by
two-photon fusion in future electron-positron colliders Inan:2020aal ;
Inan:2020kif ; Zhang:2021sio . Thus, in this paper, we only consider the
production of ALP via W+W- fusion process e+e- $\rightarrow$
$\overline{\nu}$e$\nu$e$a$ with $a$ subsequently decaying to photon pairs (see
Fig. 3).

##  3 Search for ALP at the CLIC

As we know, the high-energy e+e- collider can provide clean experimental
environment to explore new physics beyond the Standard Model (BSM). The future
e+e- colliders are contemplated to study the properties of the SM observables
with an unprecedented precision, and the CLIC is one of them. The CLIC is
planned to be built and operated in three energy stages. The first stage is
planned to accurately measure the properties of the Higgs boson at $\sqrt{s}$
= 380 GeV with $\mathcal{L}$ = 0.5 ab-1. The second and ultimate stages
running at $\sqrt{s}$ = 1500 GeV with $\mathcal{L}$ = 1.5 ab-1 and $\sqrt{s}$
= 3000 GeV with $\mathcal{L}$ = 3 ab-1 are not only able to measure the top
Yukawa coupling, the Higgs trilinear self-coupling and the Higgs rare decays,
but also can give indirect sensitivity to many new physics scenarios through
precision measurements CLIC:2016zwp . They will be able to discover new
particles through direct detection. We will do calculations and analysis of
the signal and background for the process
$e^{+}e^{-}\rightarrow\overline{\nu}_{e}\nu_{e}a(\to\gamma\gamma$) at the CLIC
with three energy stages.

In this section, we perform the Monte Carlo (MC) simulation to analyze the
possibility of exploring ALPs at the CLIC through the W+W- fusion process e+e-
$\rightarrow$ $\overline{\nu}_{e}$$\nu_{e}a$ $(\rightarrow\gamma\gamma)$,
where the gauge bosons W± are emitted from the incoming electron/positron
beams. Its signal is characterised by two photons, missing transverse energy
from the escaping undetected two neutrinos ($\overline{\nu}$e$\nu$e). In our
numerical calculations, we take into account the process e+e- $\rightarrow$
$\overline{\nu}$$\nu$$\gamma\gamma$ as the dominant SM background, which is
mainly induced through electroweak interaction. For the signal, the final two
photons mainly come from the decay of pseudo-scalar ALPs (spin-0), while there
is no particle with spin-0 in the background that decays to two photons at
tree level. So the interference is very small. We have estimated the cross
section of pure ALP signal to be 17.25 fb and the cross section of the
interference to be -0.1549 fb at $\sqrt{s}=1500$ GeV, $M_{a}=10$ GeV and
$g_{a\gamma\gamma}=10^{-3}$ GeV-1 without any cut applied. Apparently the
cross section of pure ALP signal is two orders of magnitude larger than that
of the interference. We have estimated the cross sections of pure ALP signal
and interference for multiple ALP benchmark points, and the conclusions are
similar. For simplicity, we have neglected the influence of the interference
between ALP production and SM background in
$e^{+}e^{-}\to\overline{\nu}\nu\gamma\gamma$.

Table 1: The improved cuts on the signal and background events.  Cuts |  $\sqrt{s}=$ 380 GeV |  $\sqrt{s}=$ 1500 GeV |  $\sqrt{s}=$ 3000 GeV
---|---|---|---
Cut-1: Angle between the ALP and the beam axis |  0.6 $\textless~{}\theta(\gamma\gamma)~{}\textless$ 2.6 |  0.7 $\textless~{}\theta(\gamma\gamma)~{}\textless$ 2.5 |  0.5 $\textless~{}\theta(\gamma\gamma)~{}\textless$ 2.6
Cut-2: The missing transverse energy | $\not{E}_{T}~{}\textgreater~{}70~{}\mathrm{GeV}$ | $\not{E}_{T}~{}\textgreater~{}150~{}\mathrm{GeV}$ | $\not{E}_{T}~{}\textgreater~{}200~{}\mathrm{GeV}$
Cut-3: The transverse energy | $E_{T}~{}\textgreater~{}85~{}\mathrm{GeV}$ | $E_{T}~{}\textgreater~{}160~{}\mathrm{GeV}$ | $E_{T}~{}\textgreater~{}220~{}\mathrm{GeV}$
Table 2: After different cuts applied, the cross sections for the signal and
SM background at the $\sqrt{s}=$ 380 GeV CLIC for $M_{a}$ = 8, 10, 20, 100,
200 GeV and $g_{a\gamma\gamma}=10^{-3}~{}$ GeV-1.  CLIC @ $\sqrt{s}=380$ GeV
---
Cuts | Signal (fb) | Background (fb)
|  $M_{a}$ = 8 GeV |  $M_{a}$ = 10 GeV |  $M_{a}$ = 20 GeV |  $M_{a}$ = 100 GeV |  $M_{a}$ = 200 GeV |  ${\gamma\gamma\overline{\nu}{\nu}}$
Basic cuts |  0.3572 |  0.3542 |  0.3506 |  0.3279 |  0.0731 |  12.66
Cut 1 |  0.3436 |  0.3416 |  0.3364 |  0.2762 |  0.0612 |  7.4612
Cut 2 |  0.3404 |  0.3384 |  0.332 |  0.2464 |  0.0459 |  5.5602
Cut 3 |  0.3198 |  0.318 |  0.3118 |  0.2442 |  0.0457 |  5.3916
Table 3: Same as Table 2 but for the $\sqrt{s}=$ 1500 GeV CLIC.  CLIC @
$\sqrt{s}=1500$ GeV
---
Cuts | Signal (fb) | Background (fb)
|  $M_{a}$ = 8 GeV |  $M_{a}$ = 10 GeV |  $M_{a}$ = 20 GeV |  $M_{a}$ = 100 GeV |  $M_{a}$ = 500 GeV |  $M_{a}$ = 1000 GeV |  ${\gamma\gamma\overline{\nu}{\nu}}$
Basic cuts |  6.6575 |  8.8253 |  12.176 |  11.9803 |  1.7862 |  0.1573 |  21.322
Cut 1 |  3.7949 |  5.4839 |  8.6115 |  8.3208 |  1.0745 |  0.0913 |  8.1313
Cut 2 |  3.3201 |  4.9969 |  8.12 |  7.8021 |  1.0991 |  0.0756 |  3.2893
Cut 3 |  3.1913 |  4.8613 |  8.0028 |  7.7769 |  1.0089 |  0.0756 |  3.2211
Table 4: Same as Table 2 but for the $\sqrt{s}=$ 3000 GeV CLIC.  CLIC @
$\sqrt{s}=3000$ GeV
---
Cuts | Signal (fb) | Background (fb)
|  $M_{a}$ = 8 GeV |  $M_{a}$ = 10 GeV |  $M_{a}$ = 20 GeV |  $M_{a}$ = 100 GeV |  $M_{a}$ = 1000 GeV |  $M_{a}$ = 1500 GeV |  $M_{a}$ = 2000 GeV |  ${\gamma\gamma\overline{\nu}{\nu}}$
Basic cuts |  9.6983 |  14.4563 |  27.993 |  32.5997 |  3.5915 |  1.3519 |  0.3323 |  29.1063
Cut 1 |  5.557 |  8.7537 |  20.1513 |  24.8866 |  2.2756 |  0.8178 |  0.2003 |  14.56
Cut 2 |  3.8733 |  7.0693 |  18.4306 |  23.1416 |  2.1683 |  0.7647 |  0.1769 |  5.0493
Cut 3 |  3.3481 |  6.552 |  17.903 |  22.946 |  2.1682 |  0.7647 |  0.1769 |  4.8927

We look for photons reconstructed in the barrel region $|\eta_{\gamma}|<2.5$.
A light ALP tends to decay a pair of collimated photons, which appear
essentially as one photon of the combined energy in a detector and thus can
not be correctly reconstructed as two individual photons Sheff:2020jyw . In
Ref. Jaeckel:2015jla , the light ALP is studied via the decay
$Z\rightarrow\gamma a~{}(\to\gamma\gamma)$ with $\Delta R(\gamma,\gamma)\sim
4\frac{m_{a}}{m_{Z}}$. $\Delta
R(\gamma,\gamma)=\sqrt{(\Delta\phi)^{2}+(\Delta\eta)^{2}}$ is the particle
separation in the rapidity-azimuth plane, $\Delta\phi$ and $\Delta\eta$ are
the azimuth difference and pseudo-rapidity difference of the photon pair,
respectively. To reconstruct collimated pair of photons from the light ALP
decay, a peak separation of $\Delta
R(\gamma,\gamma)=4\frac{m_{\pi}}{E_{\pi}}=0.035$ is taken in Ref.
Steinberg:2021iay , where $m_{\pi}$ and $E_{\pi}$ represent the mass and
energy of $\pi^{0}$, and we can see that if $\Delta R(\gamma,\gamma)$ is
smaller than 0.035, making photon pair reconstruction challenging. In our
work, the peak of the normalized distribution of $\Delta R(\gamma,\gamma)$ is
near 0.06 which allows for sufficient separation for both photons in the pair
at $M_{a}=5$ GeV. To better isolate produced photon pairs, we ask for the
isolation between the two photons $\Delta R(\gamma,\gamma)>0.06$ and the
photon having a minimum transverse momentum of 30 GeV. Based on the above
analysis, the basic cuts are chosen as follows:

$\begin{array}[]{l}\left|\eta_{\gamma}\right|<2.5,~{}~{}\quad\Delta
R(\gamma,\gamma)>0.06,~{}~{}\quad p_{T}^{\gamma}>30~{}\mathrm{GeV}.\\\
\end{array}$

In the following, the signal and background events are generated at tree-level
by Madgraph5-aMC@NLO Alwall:2014hca  with the above basic cuts. The fast
detector simulation are performed with Delphes deFavereau:2013fsa  using the
CLIC detector card. Finally, we use MadAnalysis5 Conte:2012fm  package to do
the kinematic and cut-based analysis with the reconstructed-level events of
signal and background. The final states of the signal and background contain
invisible neutrino and two photons. For the signal, two photons in the final
state from ALP decay could be a powerful trigger, and the angular separation
of two photons depends largely on the ALP mass. Therefore, we choose the
invariant mass of the photon pair $M(\gamma~{}\gamma)$, the angle between
reconstructed ALP and beam axis $\theta(\gamma~{}\gamma)$, the missing
transverse energy $\not{E}_{T}$ associated to the invisible neutrino system,
as well as the transverse energy $E_{T}$ associated to the visible particles
as observables in the lab frame. The missing transverse energy and transverse
energy are defined by $\not{E}_{T}=\arrowvert\sum\limits_{\rm
visible~{}particles}\limits\overrightarrow{p}_{T}\arrowvert$ and
$E_{T}=\sum\limits_{\rm
visible~{}particles}\limits\arrowvert\overrightarrow{p}_{T}\arrowvert$, where
$\overrightarrow{p}_{T}$ stands for the visible particles transverse momentum.

In Fig. 4, we plot the normalized distributions of $M(\gamma~{}\gamma)$,
$\theta(\gamma~{}\gamma)$, $\not{E}_{T}$ and $E_{T}$ for the signal and
background with the signal benchmark points $M_{a}$ = 8, 10, 20, 100, 200 GeV
and the fixed parameter $g_{a\gamma\gamma}$ = $10^{-3}$ GeV-1 for the c.m.
energy $\sqrt{s}$ $=$380 GeV. In Fig. 5 we show the same normalized kinematic
distributions for $M_{a}$ = 8, 10, 20, 100, 500, 1000 GeV and
$g_{a\gamma\gamma}=10^{-3}$ GeV-1 at $\sqrt{s}=$ 1500 GeV, and the same is
shown in Fig. 6 for $M_{a}$ = 8, 10, 20, 100, 1000, 1500, 2000 GeV and
$g_{a\gamma\gamma}=10^{-3}$ GeV-1 at $\sqrt{s}=$ 3000 GeV. In Figs. 4, 5 and
6, all the signal and background samples are obtained by after applying the
above basic cuts. From Figs. 4, 5 and 6, we can see that, in order to optimize
the signal significance, some additional cuts of kinematic distributions are
needed. From these figures, we can see that it is not suitable to impose
restrictions on the normalized $M(\gamma~{}\gamma)$ distribution, hence we do
not put any constraint. According to the behavioral characteristics of the
other three distributions, we can further impose the following improved cuts
on the signal and background events as shown in Table 1. Of course, we have
applied the basic cuts before applying the cuts in Table 1, and the cuts 1, 2
and 3 are applied in sequence.

After imposing the cuts for a few representative ALP mass benchmark points, we
summarize the cross sections of the signal and background in Tables 2, 3 and 4
for $\sqrt{s}=$ 380 GeV, 1500 GeV and 3000 GeV, respectively. From Tables 2, 3
and 4, after applying the basic cuts we can find that Tables 3 and 4 show very
large differences in the cross sections between $M_{a}=8,10$ and $20$ GeV,
while the variations appear to be minimal in Table 2. For the ALP-photon
coupling $g_{a\gamma\gamma}$ = $10^{-3}$ GeV-1, the variations in the cross
section of signal are minimal in the ALP mass interval 5 GeV $\sim$ 20 GeV
without cut applied at $\sqrt{s}=$ 380, 1500 and 3000 GeV. The peak of the
distribution shifts towards smaller $\Delta R(\gamma,\gamma)$ for the larger
c.m. energy at same ALP mass, while the range of $|\eta_{\gamma}|$ for the
high c.m. energy is larger than that for the low c.m. energy at same ALP mass.
That is to say, for $\sqrt{s}=$ 3 TeV, the signal for $M_{a}=$ 8, 10 and 20
GeV lie very close to the signal region boundaries, so the lighter the ALP the
more the signal is cut out. Therefore, the effects of the basic cuts on the
cross sections of $M_{a}=8,10$ and $20$ GeV at $\sqrt{s}=1500$ and $3000$ GeV
are greater than those at $\sqrt{s}=380$ GeV. From these tables, one can see
that the background is suppressed very efficiently, while the signal still has
good efficiency after imposing all cuts.

We use the following Poisson formula Cowan:2010js  to estimate the statistical
significance (SS) at the CLIC with the luminosities of 0.5 ab-1, 1.5 ab-1 and
3 ab-1, respectively,

$\quad\quad SS=\sqrt{2\mathcal{L}[(S+B)\ln(1+\frac{S}{B})-S]},$ (3)

where $\mathcal{L}$ represents the integrated luminosity, $S$ and $B$
respectively represent the effective cross sections of the signal and
background after imposing all cuts.

In Fig. 7, we present the 3$\sigma$ and 5$\sigma$ bounds on the ALP parameter
space from the W+W- fusion process e+e- $\rightarrow$
$\overline{\nu}$e$\nu$${}_{e}a$ $(\rightarrow\gamma\gamma)$ at the CLIC. The
blue, red and green lines depict the bounds coming from the analysis for the
three energy stages of the CLIC. This figure shows that the sensitivity bounds
on the ALP-photon coupling $g_{a\gamma\gamma}$ can be improved to 0.8 TeV-1
(1.1 TeV-1) with the first energy stage, 0.13 TeV-1 (0.15 TeV-1) with the
second energy stage and 0.073 TeV-1 (0.091 TeV-1) with the third energy stage
for the ALP mass interval 5 GeV $\sim$ 340 GeV, 5 GeV $\sim$ 1300 GeV and 5
GeV $\sim$ 2600 GeV at 3$\sigma$ (5$\sigma$) levels, respectively. Comparing
our results of the three energy stages at the CLIC with each other, it is
obvious that the sensitivity bound increases with the c.m. energy increases.

Our obtained bounds and other current existing constraints on the ALP-photon
coupling are presented in Fig. 8. The green region comes from different beam
dump experiments
Riordan:1987aw ; Bjorken:1988as ; Dobrich:2015jyk . The dark blue and light
blue regions depict the results given by Mimasu:2014nea ; Jaeckel:2015jla  via
the diphoton and triphoton final states
(e+e${}^{-}~{}\to~{}2\gamma,~{}3\gamma$) at the LEP. The dark gray and black
regions denote the constraints from radiative $\Upsilon~{}\to~{}\gamma a$ at
the Babar BaBar:2011kau  and the diphoton resonance searches at the LHCb
CidVidal:2018blh , respectively. The dark green region labelled as "L3"
represents looking for isolated and energetic photons from hadronic decay of Z
boson at the L3 collaboration L3:1992kcg . The exclusion regions also include
the results from the LHC resonant $\gamma\gamma$ searches in red
Jaeckel:2012yz ; ATLAS:2012fgo ; Mariotti:2017vtv . Our projected CLIC
sensitivity for $\sqrt{s}=$ 380, 1500 and 3000 GeV are enclosed by the cyan,
blue and wine lines, respectively. Comparing our exclusion limits with the
results of the LEP and LHC, we can conjecture that the CLIC has better
potential to search for ALPs in the lower ALP mass interval 5 GeV $\sim$ 50
GeV and high ALP mass interval 2 TeV $\sim$ 2.7 TeV.

##  4 Conclusions

As a class of attractive particles, ALPs may influence the structure
Electroweak phase transition, play an important role in solving the hierarchy
problem and serve as cold dark matter (DM). There are excellent motivations to
study ALPs. In this paper, we study the CLIC potential to search for ALP via
the W+W- fusion process e+e- $\rightarrow$ $\overline{\nu}$e$\nu$${}_{e}a$
$(\rightarrow\gamma\gamma)$. We have performed the numerical calculations and
the phenomenological analysis for the signal and relevant SM background and
obtained the 3$\sigma$ and 5$\sigma$ bounds on the ALP parameter space at the
three energy stages of CLIC. Comparing our numerical results with other
existing bounds, we find that, for the low mass region 5 GeV $\sim$ 50 GeV,
the constraint on the ALP-photon coupling $g_{a\gamma\gamma}$ can be improved
to 5$\times$10-2 TeV-1. Meanwhile, our result complements the research of ALP-
photon coupling by the ATLAS and CMS in the ALP mass range of 2 TeV $\sim$ 2.7
TeV.

In conclusion, we have found that searching for ALPs at the CLIC via the W+W-
fusion process e+e- $\rightarrow$ $\overline{\nu}$e$\nu$${}_{e}a$
$(\rightarrow\gamma\gamma)$ can not only cover low mass ALPs, but also have
the greater potential to study the high mass ALPs. Thus, studying ALPs through
the W+W- fusion process e+e- $\rightarrow$ $\overline{\nu}$e$\nu$${}_{e}a$
$(\rightarrow\gamma\gamma)$ at the CLIC provides a complementary window to the
opportunities offered by other high energy collider experiments.

## Acknowledgments

This work was supported in part by the National Natural Science Foundation of
China under Grants No. 11875157 and 12147214.

## References

  * (1) Georges Aad et al.  Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC.  Phys. Lett. B, 716:1–29, 2012.
  * (2) Serguei Chatrchyan et al.  Observation of a New Boson at a Mass of 125 GeV with the CMS Experiment at the LHC.  Phys. Lett. B, 716:30–61, 2012.
  * (3) Gerald Aarons et al.  International Linear Collider Reference Design Report Volume 2: Physics at the ILC.  9 2007.
  * (4) The International Linear Collider Technical Design Report - Volume 2: Physics.  6 2013.
  * (5) A. Arbey et al.  Physics at the e+ e- Linear Collider.  Eur. Phys. J. C, 75(8):371, 2015.
  * (6) Hiroaki Aihara et al.  The International Linear Collider. A Global Project.  1 2019.
  * (7) Philip Bambade et al.  The International Linear Collider: A Global Project.  3 2019.
  * (8) E. Accomando et al.  Physics at the CLIC multi-TeV linear collider.  In 11th International Conference on Hadron Spectroscopy, CERN Yellow Reports: Monographs, 6 2004.
  * (9) The Compact Linear e+e- Collider (CLIC): Physics Potential.  12 2018.
  * (10) M. Bicer et al.  First Look at the Physics Case of TLEP.  JHEP, 01:164, 2014.
  * (11) A. Abada et al.  FCC-ee: The Lepton Collider: Future Circular Collider Conceptual Design Report Volume 2.  Eur. Phys. J. ST, 228(2):261–623, 2019.
  * (12) CEPC Conceptual Design Report: Volume 1 - Accelerator.  9 2018.
  * (13) Mingyi Dong et al.  CEPC Conceptual Design Report: Volume 2 - Physics & Detector.  11 2018.
  * (14) Fenfen An et al.  Precision Higgs physics at the CEPC.  Chin. Phys. C, 43(4):043002, 2019.
  * (15) R. D. Peccei and Helen R. Quinn.  CP Conservation in the Presence of Instantons.  Phys. Rev. Lett., 38:1440–1443, 1977.
  * (16) Steven Weinberg.  A New Light Boson?  Phys. Rev. Lett., 40:223–226, 1978.
  * (17) Frank Wilczek.  Problem of Strong $P$ and $T$ Invariance in the Presence of Instantons.  Phys. Rev. Lett., 40:279–282, 1978.
  * (18) Jihn E. Kim.  Weak Interaction Singlet and Strong CP Invariance.  Phys. Rev. Lett., 43:103, 1979.
  * (19) Mikhail A. Shifman, A. I. Vainshtein, and Valentin I. Zakharov.  Can Confinement Ensure Natural CP Invariance of Strong Interactions?  Nucl. Phys. B, 166:493–506, 1980.
  * (20) Michael Dine, Willy Fischler, and Mark Srednicki.  A Simple Solution to the Strong CP Problem with a Harmless Axion.  Phys. Lett. B, 104:199–202, 1981.
  * (21) A. R. Zhitnitsky.  On Possible Suppression of the Axion Hadron Interactions. (In Russian).  Sov. J. Nucl. Phys., 31:260, 1980.
  * (22) Martin Bauer, Matthias Neubert, and Andrea Thamm.  Collider Probes of Axion-Like Particles.  JHEP, 12:044, 2017.
  * (23) Martin Bauer, Mathias Heiles, Matthias Neubert, and Andrea Thamm.  Axion-Like Particles at Future Colliders.  Eur. Phys. J. C, 79(1):74, 2019.
  * (24) Jean-François Fortin, Huai-Ke Guo, Steven P. Harris, Doojin Kim, Kuver Sinha, and Chen Sun.  Axions: From magnetars and neutron star mergers to beam dumps and BECs.  Int. J. Mod. Phys. D, 30(07):2130002, 2021.
  * (25) David d’Enterria.  Collider constraints on axion-like particles.  In Workshop on Feebly Interacting Particles, 2 2021.
  * (26) Prateek Agrawal et al.  Feebly-interacting particles: FIPs 2020 workshop report.  Eur. Phys. J. C, 81(11):1015, 2021.
  * (27) I. Brivio, M. B. Gavela, L. Merlo, K. Mimasu, J. M. No, R. del Rey, and V. Sanz.  ALPs Effective Field Theory and Collider Signatures.  Eur. Phys. J. C, 77(8):572, 2017.
  * (28) Simon Knapen, Tongyan Lin, Hou Keong Lou, and Tom Melia.  Searching for Axionlike Particles with Ultraperipheral Heavy-Ion Collisions.  Phys. Rev. Lett., 118(17):171801, 2017.
  * (29) C. Baldenegro, S. Hassani, C. Royon, and L. Schoeffel.  Extending the constraint for axion-like particles as resonances at the LHC and laser beam experiments.  Phys. Lett. B, 795:339–345, 2019.
  * (30) R. O. Coelho, V. P. Goncalves, D. E. Martins, and M. S. Rangel.  Production of axionlike particles in $PbPb$ collisions at the LHC, HE–LHC and FCC: A phenomenological analysis.  Phys. Lett. B, 806:135512, 2020.
  * (31) R. O. Coelho, V. P. Gonçalves, D. E. Martins, and M. Rangel.  Exclusive and diffractive $\gamma\gamma$ production in $PbPb$ collisions at the LHC, HE - LHC and FCC.  Eur. Phys. J. C, 80(5):488, 2020.
  * (32) V. P. Goncalves, D. E. Martins, and M. S. Rangel.  Searching for axionlike particles with low masses in pPb and PbPb collisions.  Eur. Phys. J. C, 81(6):522, 2021.
  * (33) Joerg Jaeckel, Martin Jankowiak, and Michael Spannowsky.  LHC probes the hidden sector.  Phys. Dark Univ., 2:111–117, 2013.
  * (34) Cristian Baldenegro, Sylvain Fichet, Gero von Gersdorff, and Christophe Royon.  Searching for axion-like particles with proton tagging at the LHC.  JHEP, 06:131, 2018.
  * (35) Andrés Flórez, Alfredo Gurrola, Will Johns, Paul Sheldon, Elijah Sheridan, Kuver Sinha, and Brandon Soubasis.  Probing axionlike particles with $\gamma\gamma$ final states from vector boson fusion processes at the LHC.  Phys. Rev. D, 103(9):095001, 2021.
  * (36) W. J. Marciano, A. Masiero, P. Paradisi, and M. Passera.  Contributions of axionlike particles to lepton dipole moments.  Phys. Rev. D, 94(11):115033, 2016.
  * (37) S. C. İnan and A. V. Kisselev.  A search for axion-like particles in light-by-light scattering at the CLIC.  JHEP, 06:183, 2020.
  * (38) S. C. İnan and A. V. Kisselev.  Polarized light-by-light scattering at the CLIC induced by axion-like particles.  Chin. Phys. C, 45(4):043109, 2021.
  * (39) Hua-Ying Zhang, Chong-Xing Yue, Yu-Chen Guo, and Shuo Yang.  Searching for axionlike particles at future electron-positron colliders.  Phys. Rev. D, 104(9):096008, 2021.
  * (40) Dario Buttazzo, Diego Redigolo, Filippo Sala, and Andrea Tesi.  Fusing Vectors into Scalars at High Energy Lepton Colliders.  JHEP, 11:144, 2018.
  * (41) M. B. Gavela, J. M. No, V. Sanz, and J. F. de Trocóniz.  Nonresonant Searches for Axionlike Particles at the LHC.  Phys. Rev. Lett., 124(5):051802, 2020.
  * (42) Howard Georgi, David B. Kaplan, and Lisa Randall.  Manifesting the Invisible Axion at Low-energies.  Phys. Lett. B, 169:73–78, 1986.
  * (43) Daohan Wang, Lei Wu, Jin Min Yang, and Mengchao Zhang.  Photon-jet events as a probe of axionlike particles at the LHC.  Phys. Rev. D, 104(9):095016, 2021.
  * (44) Adam Alloul, Neil D. Christensen, Céline Degrande, Claude Duhr, and Benjamin Fuks.  FeynRules 2.0 - A complete toolbox for tree-level phenomenology.  Comput. Phys. Commun., 185:2250–2300, 2014.
  * (45) Paolo Azzurri, Gregorio Bernardi, Sylvie Braibant, David d’Enterria, Jan Eysermans, Patrick Janot, Ang Li, and Emmanuel Perez.  A special Higgs challenge: measuring the mass and production cross section with ultimate precision at FCC-ee.  Eur. Phys. J. Plus, 137(1):23, 2022.
  * (46) Joerg Jaeckel and Michael Spannowsky.  Probing MeV to 90 GeV axion-like particles with LEP and LHC.  Phys. Lett. B, 753:482–487, 2016.
  * (47) Ken Mimasu and Verónica Sanz.  ALPs at Colliders.  JHEP, 06:173, 2015.
  * (48) M J Boland et al.  Updated baseline for a staged Compact Linear Collider.  8 2016.
  * (49) Benjamin Sheff, Noah Steinberg, and James D. Wells.  Higgs boson decays into narrow diphoton jets and their search strategies at the Large Hadron Collider.  Phys. Rev. D, 104(3):036009, 2021.
  * (50) Noah Steinberg and James D. Wells.  Axion-Like Particles at the ILC Giga-Z.  JHEP, 08:120, 2021.
  * (51) J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H. S. Shao, T. Stelzer, P. Torrielli, and M. Zaro.  The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations.  JHEP, 07:079, 2014.
  * (52) J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lemaître, A. Mertens, and M. Selvaggi.  DELPHES 3, A modular framework for fast simulation of a generic collider experiment.  JHEP, 02:057, 2014.
  * (53) Eric Conte, Benjamin Fuks, and Guillaume Serret.  MadAnalysis 5, A User-Friendly Framework for Collider Phenomenology.  Comput. Phys. Commun., 184:222–256, 2013.
  * (54) Glen Cowan, Kyle Cranmer, Eilam Gross, and Ofer Vitells.  Asymptotic formulae for likelihood-based tests of new physics.  Eur. Phys. J. C, 71:1554, 2011.  [Erratum: Eur.Phys.J.C 73, 2501 (2013)].
  * (55) E. M. Riordan et al.  A Search for Short Lived Axions in an Electron Beam Dump Experiment.  Phys. Rev. Lett., 59:755, 1987.
  * (56) J. D. Bjorken, S. Ecklund, W. R. Nelson, A. Abashian, C. Church, B. Lu, L. W. Mo, T. A. Nunamaker, and P. Rassmann.  Search for Neutral Metastable Penetrating Particles Produced in the SLAC Beam Dump.  Phys. Rev. D, 38:3375, 1988.
  * (57) Babette Döbrich, Joerg Jaeckel, Felix Kahlhoefer, Andreas Ringwald, and Kai Schmidt-Hoberg.  ALPtraum: ALP production in proton beam dump experiments.  JHEP, 02:018, 2016.
  * (58) J. P. Lees et al.  Search for hadronic decays of a light Higgs boson in the radiative decay $\Upsilon\to\gamma A^{0}$.  Phys. Rev. Lett., 107:221803, 2011.
  * (59) Xabier Cid Vidal, Alberto Mariotti, Diego Redigolo, Filippo Sala, and Kohsaku Tobioka.  New Axion Searches at Flavor Factories.  JHEP, 01:113, 2019.  [Erratum: JHEP 06, 141 (2020)].
  * (60) O. Adriani et al.  Isolated hard photon emission in hadronic Z0 decays.  Phys. Lett. B, 292:472–484, 1992.
  * (61) Georges Aad et al.  Measurement of isolated-photon pair production in $pp$ collisions at $\sqrt{s}=7$ TeV with the ATLAS detector.  JHEP, 01:086, 2013.
  * (62) Alberto Mariotti, Diego Redigolo, Filippo Sala, and Kohsaku Tobioka.  New LHC bound on low-mass diphoton resonances.  Phys. Lett. B, 783:13–18, 2018.

## Appendix

Figure 1:  The Feynman diagrams of the single production of ALP with V =
$\gamma$, Z at e+e- colliders.

Figure 2: The ratio $\sigma$/$(g_{a\gamma\gamma})^{2}$ as a function of the
c.m. energy $\sqrt{s}$ for $M_{a}$ = 100, 200, 300 and 500 GeV.

Figure 3: Feynman diagram of the W+W- fusion process e+e- $\rightarrow$
$\overline{\nu}_{e}$$\nu_{e}$$a$ ($\to\gamma\gamma$).

Figure 4:  Normalized distributions of $M(\gamma~{}\gamma)$,
$\theta(\gamma~{}\gamma)$, $\not{E}_{T}$, $E_{T}$ for the signal of selected
ALP-mass benchmark points and SM background at the $\sqrt{s}$ = 380 GeV CLIC
with designed luminosity.

Figure 5: Same as Fig. 4 but for $\sqrt{s}$ = 1500 GeV CLIC with designed
luminosity.

Figure 6: Same as Fig. 4 but for $\sqrt{s}=$ 3000 GeV CLIC with designed
luminosity.

Figure 7: The 3$\sigma$ and 5$\sigma$ curves in the $M_{a}-g_{a\gamma\gamma}$
plane from the W+W- fusion process e+e- $\rightarrow$
$\overline{\nu}$e$\nu$${}_{e}a$ $(\rightarrow\gamma\gamma)$ at the 380, 1500
and 3000 GeV CLIC with the designed luminosities.  Figure 8: The 2$\sigma$
exclusion limits on the ALP couplings $g_{a\gamma\gamma}$ as a function of
$M_{a}$ from the W+W- fusion process e+e- $\rightarrow$
$\overline{\nu}$e$\nu$${}_{e}a$ $(\rightarrow\gamma\gamma)$ and other current
exclusion regions.