# Interpreting Answers to Yes-No Questions in User-Generated Content

Shivam Mathur,1 Keun Hee Park,2 Dhivya Chinnappa,3
Saketh Kotamraju,4 Work done while at Arizona State University. Eduardo
Blanco5
1Walmart Global Tech, 2Arizona State University, 3JP Morgan Chase & Co.,
4University of Texas at Austin, 5University of Arizona
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Interpreting answers to yes-no questions in social media is difficult. _Yes_
and _no_ keywords are uncommon, and the few answers that include them are
rarely to be interpreted what the keywords suggest. In this paper, we present
a new corpus of 4,442 yes-no question-answer pairs from Twitter. We discuss
linguistic characteristics of answers whose interpretation is _yes_ or _no_ ,
as well as answers whose interpretation is _unknown_. We show that large
language models are far from solving this problem, even after fine-tuning and
blending other corpora for the same problem but outside social media.

##  1 Introduction

Social media has become a global town hall where people discuss whatever they
care about in real time. Despite its challenges and potential misuse, the
language of social media can be used to approach many problems, including
mental health Harrigian et al. (2020), rumor detection Ma and Gao (2020), and
fake news Mehta et al. (2022).

There is extensive literature on question answering, but few previous efforts
work with social media. In particular, yes-no questions are unexplored in
social media. Figuring out the correct interpretation of answers to yes-no
questions (_yes_ , _no_ , or somewhere in the middle) from formal texts Clark
et al. (2019), short texts Louis et al. (2020), and conversations Choi et al.
(2018); Reddy et al. (2019) is challenging. This is the case in any domain,
but especially in social media, where informal language and irony are common.
Additionally, unlike previous work on yes-no questions outside social media,
we find that _yes_ and _no_ keywords are not only rare in answers, but also
that they are poor indicators of the correct interpretation of the answer.

Figure 1:  Yes-No question from Twitter and three answers. Interpreting
answers cannot be reduced to finding keywords (e.g., _yes_ , _of course_).
Answers ought to be interpret as _yes_ (despite the negation), _no_
(commonsense and reasoning are needed), and _unknown_ respectively.

Consider the Twitter thread in Figure 1. The question is mundane: whether
people preheat the oven before using it. The answers, however, are complicated
to decipher. The first answer states that it is _very strange_ that somebody
else (the author’s girlfriend) does not preheat the oven. Note that the
correct interpretations is _yes_ (or _probably yes_) despite the negation. The
second answer explains why it is unnecessary to preheat the oven—without
explicitly saying so—thus the correct interpretation is _no_. The third answer
includes both _yes_ and _no_ yet the correct interpretation is _unknown_ , as
the author does not commit to either option.

In this paper, we tackle the problem of interpreting _natural_ answers to yes-
no questions posted on Twitter by real users, as exemplified above. The main
contributions are:111Twitter-YN and implementation available at
https://github.com/shivammathur33/twitter-yn. (a) Twitter-YN, a new corpus of
4,442 yes-no questions and answers from Twitter along with their
interpretations; (b) analysis discussing the language used in the answers, and
in particular, showing that keyword presence in answers is a bad predictor of
their interpretations, (c) experimental results showing that the problem is
challenging—even for large language models, including GPT3—and blending with
related corpora is useful; and (d) qualitative analysis of the most common
errors made by our best model.

#### Motivation

Traditionally, question answering assumes that there are (a) correct answers
to questions or (b) no answers Kwiatkowski et al. (2019). In the work
presented in this paper, answers are available—the problem is to interpret
what they mean. As Figure 1 shows, answers capture personal preferences rather
than correct answers supported by commonsense or common practice (e.g., oven
manuals state that preheating is recommended). Interpreting answers to yes-no
questions opens the door to multiple applications. For example, dialogue
systems for social media Sheng et al. (2021) must understand answers to yes-no
question to avoid inconsistencies (e.g., praising the author of the second
reply in Figure 1 for preheating the oven is nonsensical). Aggregating
interpretations could also assist in user polling Lampos et al. (2013). As the
popularity of chat-based customer support grows Cui et al. (2017), automatic
interpretation of answers to yes-no questions could enhance the customer
experience. Lastly, interactive question answering and search, in which
systems ask clarification questions to better address users’ information
needs, would benefit from this work Li et al. (2017). Clarification questions
are often yes-no questions, and as we shall see, people rarely use _yes_ ,
_no_ or similar keywords in their answers. Rather, people answer with
intricate justifications that are challenging to interpret.

##  2 Previous Work

#### Question Answering Outside Social Media

has a long tradition. There are several corpora, including some that require
commonsense Talmor et al. (2019) or understanding tables Cheng et al. (2022),
simulate open-book exams Mihaylov et al. (2018), and include questions
submitted by “real” users Yang et al. (2015); Kwiatkowski et al. (2019).
Models include those specialized to infer answers from large documents Liu et
al. (2020), data augmentation Khashabi et al. (2020), multilingual models
Yasunaga et al. (2021), and efforts to build multilingual models Lewis et al.
(2020). None of them target social media or yes-no questions.

#### Yes-No Questions

and interpreting their answers have been studied for decades. Several works
target exclusive answers not containing _yes_ or _no_ Green and Carberry
(1999); Hockey et al. (1997). Rossen-Knill et al. (1997) work with yes-no
questions in the context of navigating maps Carletta et al. (1997) and find
that answers correlate with actions taken. More recent works de Marneffe et
al. (2009) target questions from SWDA Stolcke et al. (2000). Clark et al.
(2019) present yes-no questions submitted to a search engine, and Louis et al.
(2020) present a corpus of crowdsourced yes-no questions and answers. Several
efforts work with yes-no questions in dialogues, a domain in which they are
common. Annotation efforts include crowdsourced dialogues Reddy et al. (2019);
Choi et al. (2018) and transcripts of phone conversations Sanagavarapu et al.
(2022) and _Friends_ Damgaard et al. (2021).

#### Question Answering in the Social Media Domain

is mostly unexplored. TweetQA is the only corpus available Xiong et al.
(2019). It consists of over 13,000 questions and answers about a tweet. Both
the questions and answers were written by crowdworkers, and the creation
process ensured that it does _not include any yes-no question_. In this paper,
we target yes-no questions in social media for the first time. We do not
tackle the problem of extracting answers to questions. Rather, we tackle the
problem of _interpreting_ answers to yes-no questions, where both questions
and answers are posted by genuine social media users rather than paid
crowdworkers.

##  3 Twitter-YN: A Collection of Yes-No Questions and Answers from Twitter

Since previous research has not targeted yes-no questions in social media, our
first step it to create a new corpus, Twitter-YN. Retrieving yes-no questions
and their answers takes into account the intricacies of Twitter, while the
annotation guidelines are adapted from previous work.

Question | Answer | Intpn.
---|---|---
|  | _yes_
|  | _no_
|  | _prob. yes_
|  | _prob. no_
|  | _unknown_
Table 1:  Examples of yes-no questions and answers from our corpus. Some
answers include negative keywords but their interpretation is _yes_ (first
example), whereas others include positive keywords but their interpretation is
_no_ (second example). Answers imposing conditions are interpreted _probably
yes_ or _probably no_ (third and fourth examples). Negation does not
necessarily indicate that the author leans to either _yes_ or _no_ (fifth
example).

###  3.1 Retrieving Yes-No Questions and Answers

We define a battery of rules to identify yes-no questions. These rules were
defined after exploring existing corpora with yes-no questions (Section 2),
but we adapted them to Twitter. Our desideratum is to collect a variety of
yes-no questions, so we prioritize precision over recall. This way, the
annotation process is centered around interpreting answers rather than
selecting yes-no questions from the Twitter fire hose. The process is as
follows:

  1. 1.

Select tweets that (a) contain a question mark (‘?’) and the bigram _do you_ ,
_is it_ , _does it_ or _can it_ ; and (b) do not contain wh-words (where, who,
why, how, whom, which, when, and whose). Let us name the text between the
bigram and ‘?’ a _candidate_ yes-no question.

  2. 2.

Exclude candidate questions unless they (a) are between 3 and 100 tokens long;
(b) do not span more than one paragraph; and (c) do not contain named entities
or numbers.

  3. 3.

Exclude candidate questions with links, hashtags, or from unverified accounts.

The first step identifies likely yes-no questions. We found that avoiding
other questions with wh-words is critical to increase precision. The second
step disregards short and long questions. The reason to discard questions with
named entities is to minimize subjective opinions and biases during the
annotation process—most yes-no questions involving named entities ask for
opinions about the entity (usually a celebrity). We discard questions with
numbers because the majority are about cryptocurrencies or stocks and we want
to avoid having tweets about the financial domain (out of a random sample of
100 questions, 68% of questions with numbers were about cryptocurrencies or
stocks). The third step increases precision by considering only verified
accounts, which are rarely spam. Avoiding hashtags and links allows us to
avoid answers whose interpretation may require external information including
trending topics.

Once yes-no questions are identified, we consider as answers all replies
except those that contain links, more than one user mention (@user), no verb,
less than 6 or more than 30 tokens, or question marks (‘?’). The rationale
behind these filters is to avoid answers whose interpretation require external
information (links) or are unresponsive. We do not discard answers with _yes_
or _no_ keywords because as we shall see (and perhaps unintuitively) they are
not straightforward to interpret (Section 4).

#### Temporal Spans: Old and New

We collect questions and their answers from two temporal spans: older (years
2014–2021) and newer (first half of 2022). Our rationale is to conduct
experiments in a realistic setting, i.e., training and testing with questions
posted in non-overlapping temporal spans.

The retrieval process resulted in many yes-no question-answer pairs. We set to
annotate 4,500 pairs. Because we wanted to annotate all answers to the
selected questions across both temporal spans, Twitter-YN consists of 4,442
yes-no question-answer pairs (old: 2,200, new: 2,242). The total number of
questions are 1,200 (3.7 answers per question on average).

###  3.2 Interpreting Answers to Yes-No Questions

After collecting yes-no questions and answers, we manually annotate their
interpretations. We work with the five interpretations exemplified in Table 1.
Our definitions are adapted from previous work de Marneffe et al. (2009);
Louis et al. (2020); Sanagavarapu et al. (2022) and summarized as follows:

  * •

_Yes_ : The answer ought to be interpreted as _yes without reservations_. In
the example, the answer doubles down on the comment that it is rude to suggest
that somebody would look better if they lost weight (despite also using _not_)
by characterizing the comment as lethal.

  * •

_No_ : The answer ought to be interpreted as _no without reservations_. In the
example, the author uses irony to effectively state that one should not read
books in bars.

  * •

_Probably yes_ : The answer is _yes_ under certain condition(s) (including
time constraints) or it shows arguments for _yes_ from a third party. In the
example, the answer relies on the weather report to mean that it will be big
coat weather.

  * •

_Probably no_ : The answer is a _no_ under certain condition(s) or shows
arguments for _no_ held by a third party, or the author shows hesitancy. In
the example, the author indicates that it is _not bad_ to brake with your left
foot while driving if you are a racecar driver.

  * •

_Unknown_ : The answer disregards the question (e.g., changes topics) or
addresses the question without an inclination toward _yes_ or _no_. In the
example, the author states that the question is irrelevant without answering
it.

Appendix A presents the detailed guidelines.

Figure 2:  Differences between adjudicated interpretations and interpretations
by each annotator. There are few disagreements; the diagonals (i.e., the
adjudicated interpretation and the one by each annotator) account for 84.72%
and 84.37%. Additionally, most disagreements are minor: they are between (a)
_yes_ (_no_) and _probably yes_ (_probably no_) or (b)  _unknown_ and another
label.

Annotation Process and Quality We recruited four graduate students to
participate in the definition of the guidelines and conduct the annotations.
The 4,442 yes-no question-answers pairs were divided into batches of 100. Each
batch was annotated by two annotators, and disagreements were resolved by an
adjudicator. In order to ensure quality, we calculated inter-annotator
agreement using linearly weighted Cohen’s $\kappa$. Overall agreement was
$\kappa=0.68$. Note that $\kappa$ coefficients above 0.6 are considered
_substantial_ Artstein and Poesio (2008) (above 0.8 are considered _nearly_
perfect). Figure 2 shows the disagreements between the interpretations by
annotators and the adjudicator. Few disagreements raise concerns. The
percentage of disagreements between _yes_ and _no_ is small: 1.40, 1.44, 1.28,
and 0.88. We refer the reader to the Data Statement in Appendix B for further
details about Twitter-YN.

##  4 Analyzing Twitter-YN

|  | % Contains Keyword
---|---|---
Interpretation | % | positive | negative
_yes_ | 40.79 | 13.42 | 11.98
_no_ | 30.23 | 2.23 | 18.91
_probably yes_ | 8.04 | 1.33 | 2.43
_probably no_ | 3.85 | 0.33 | 2.39
_unknown_ | 17.09 | 0.92 | 5.70
Table 2:  Distribution of interpretations in our corpus (Columns 1 and 2) and
keyword-based analysis (Column 3 and 4). Only 13.42% (18.91%) of answers with
_yes_ (_no_) interpretations include a positive (negative) keyword.
Interpreting answers cannot be reduced to checking for positive or negative
keywords.

Twitter-YN is an unbalanced dataset: the interpretation of 70% of answers is
either _yes_ or _no_ (Table 2). More interestingly, few answers contain a
_yes_ or _no_ keyword, and those that do often are not interpreted what the
keyword suggests. For example, 11.98% of answers labeled _yes_ contain a
negative keyword, while only 13.42% contain a positive keyword.

Are Keywords Enough to Interpret Answers? No, they are not. We tried simple
keyword-based rules to attempt to predict the interpretations of answers. The
rules we consider first check for _yes_ (yes, yea, yeah, sure, right, you bet,
of course, certainly, definitely, or uh uh), _no_ (no, nope, no way, never, or
n’t), and _uncertainty_ (maybe, may, sometimes, can, perhaps, or if) keywords.
Then they return (a)  _yes_ (or _no_) if we only found keywords for _yes_ (or
_no_), (b)  _probably yes_ (or _probably no_) if we found keywords for
_uncertainty_ and _yes_ (or _no_), (c)  _unknown_ otherwise. As Figure 3
shows, such a strategy does not work. Many instances are wrongly predicted
_unknown_ because no keyword is present, and keywords also fail to distinguish
between _no_ and _probably no_ interpretations.

Figure 3:  Heatmap comparing the interpretations predicted with keyword-based rules (Section 4) and the ground truth. Rules are insufficient for this problem. Bigram |  Verb (base form, top-10)
---|---
_do you_ (66.4%) |  _think_ (9.5), _have_ (7.8), _believe_ (5.4), _want_ (3.7), _use_ (2.6), _get_ (2.4), _do_ (2.3), _like_ (2.1), _go_ (1.8), and _feel_ (1.7)
_is it_ (30.6%) |  _have_ (5.3), _go_ (4.9), _drive_ (3.6), _ask_ (3.3), _look_ (3.0), _put_ (2.5), _want_ (2.3), _use_ (2.2), _wear_ (2.0), and _eat_ (1.8)
_does it_ (2.5%) |  _make_ (19.1), _have_ (6.0), _hear_ (5.5), _matter_ (5.5), _get_ (5.1), _take_ (4.7), _tear_ (4.7), _count_ (4.3), _work_ (4.3), and _exist_ (3.0)
_can it_ (0.5%) |  _offer_ (32.0), _ask_ (32.0), _understand_ (32.0), _reply_ (2.0), and _wait_ (2.0)
Table 3:  Most common bigrams and verbs in the questions. Most questions
include the bigrams _do you_ or _is it_ (97%). The verb percentages show that
the questions are diverse. Indeed, the top-10 most frequent verbs with _do
you_ only account for 39.3% of questions with _do you_.

###  4.1 Linguistic Analysis

Despite all yes-no questions include one of four bigrams (_do you_ , _is it_ ,
_does it_ or _can it_ , Section 3.1), and 97% include _do you_ or _is it_ ,
Twitter-YN has a wide variety of questions. Indeed, the distribution of verbs
has a long tail (Table 3). For example, less than 10% of questions containing
_do you_ have the verb _think_ , Further, the the top-10 most frequent verbs
account for 39.3% of questions starting with _do you_. Thus, the questions in
Twitter-YN are diverse from a lexical point of view.

We also conduct an analysis to shed light on the language used in answers to
yes-no questions (Table 4). Verbs and pronouns are identified with spaCy
Honnibal and Montani (2017). We use WordNet lexical files Miller (1995) as
verb classes, the negation cue detector by Hossain et al. (2020), and the
sentiment lexicon by Crossley et al. (2017).

First, we compare answers interpreted as _yes_ and _no_. We observe that
cardinal numbers are more common in _yes_ rather than _no_ interpretations.
Object pronoun _me_ is also more common with _yes_ interpretations; most of
the time it is used to describe personal experiences. Despite the low
performance of the keyword-based rules, negation cues and negative sentiment
are more frequent with _no_ interpretations. Finally, emojis are more frequent
in humorous answers meaning _yes_ rather than _no_.

Second, we compare _unknown_ and other interpretations (i.e., when the author
leans towards neither _yes_ nor _no_). We observe that longer answers are less
likely to be _unknown_ , while more verbs indicate _unknown_. All verb classes
are indicators of _unknown_. For example, communication (say, tell, etc.),
creation (bake, perform etc.), motion (walk, fly etc.), and perception (see,
hear etc.) verbs appear more often with _unknown_ answers. Finally, _unknown_
answers rarely have negation cues.

Linguistic Feature |  _yes_ vs. _no_ |  _unk_ vs other
---|---|---
Tokens |  | $\downarrow$
Cardinal numbers | $\uparrow$ |
Verbs |  |  $\uparrow$$\uparrow$$\uparrow$
verb.communication |  |  $\uparrow$$\uparrow$$\uparrow$
verb.creation |  | $\uparrow$
verb.motion |  | $\uparrow$
verb.perception |  |  $\uparrow$$\uparrow$
_would_ |  $\downarrow$$\downarrow$ |
_must_ |  | $\uparrow$
_shall_ |  |  $\uparrow$$\uparrow$
1st person pronoun |  |  $\downarrow$$\downarrow$
_I_ |  |  $\downarrow$$\downarrow$$\downarrow$
_me_ |  $\uparrow$$\uparrow$ |
2nd person pronoun |  |  $\uparrow$$\uparrow$$\uparrow$
_he_ |  | $\uparrow$
_you_ |  |  $\uparrow$$\uparrow$$\uparrow$
Negation cues |  $\downarrow$$\downarrow$$\downarrow$ |  $\downarrow$$\downarrow$$\downarrow$
Negative sentiment |  $\downarrow$$\downarrow$$\downarrow$ |
Emojis |  $\uparrow$$\uparrow$$\uparrow$ |
Table 4:  Linguistic analysis comparing answers interpreted as (a)  _yes_ and
_no_ and (b)  _unknown_ and the other labels. Number of arrows indicate the
p-value (t-test; one: p<0.05, two: p<0.01, and three: p<0.001). Arrow
direction indicates whether higher counts correlate with the first
interpretation (_yes_ or _unknown_) or the second.

##  5 Automatically Interpretation of Answers to Yes-No Questions

We experiment with simple baselines, RoBERTa-based classifiers, and GPT3
(zero- and few-shot).

#### Experimental settings

Randomly splitting Twitter-YN into train, validation, and test splits would
lead to valid but unrealistic results. Indeed, in the real world a model to
interpret answers to yes-no questions ought to be trained with (a) answers to
_different questions_ than the ones used for evaluation, and (b) answers to
questions from _non-overlapping temporal spans_.

In the main paper, we experiment with the most realistic setting: _unmatched
question and unmatched temporal span_ (train: 70%, validation: 15%, test:
15%). We refer the reader to Appendix C for the other two settings: _matched
question and matched temporal span_ and _unmatched question and matched
temporal span_.

#### Baselines

We present three baselines. The _majority class_ always chooses the most
frequent label in the training split (i.e., _yes_). The _keyword-based rules_
are the rules presented in Section 4. We also experiment with a version of the
rules in which the lexicon of negation cues is replaced with a _negation cue
detector_ Hossain et al. (2020).

###  5.1 RoBERTa-based Classifiers

We experiment with RoBERTa Liu et al. (2019) as released by TensorFlow Abadi
et al. (2015) Hub. Hyperparameters were tuned using the train and validation
splits, and we report results with the test split. We refer readers to
Appendix D for details about hyperparameters and the tuning process.

#### Pretraining and Blending

In addition to training with our corpus, Twitter-YN, we also explore combining
related corpora. The most straightforward approach is pretraining with other
corpora and then fine-tuning with Twitter-YN. Doing so is sound but obtains
worse results (Appendix E) than blending, so we will focus here on the latter.

Pretraining could be seen as a two-step fine-tuning process: first with
related corpora and then with Twitter-YN. On the other hand, blending Shnarch
et al. (2018) combines both during the same training step. Blending starts
fine-tuning with the combination of both and decreases the portion of
instances from related corpora after each epoch by an $\alpha$ ratio. In other
words, blending starts tuning with many yes-no questions (from the related
corpus and Twitter-YN) and finishes only with the ones in Twitter-YN. The
$\alpha$ hyperparameter is tuned like any other hyperparameter; see Appendix
D.

We experiment with pretraining and blending with the following corpora. All of
them include yes-no questions and manual annotations indicating the
interpretation of their answers. However, none of them are in the Twitter
domain:

  * •

BOOLQ Clark et al. (2019), 16,000 natural yes-no questions submitted to a
search engine and (potential) answers from Wikipedia;

  * •

Circa Louis et al. (2020), 34,000 yes-no questions and answers falling into 10
predefined scenarios and written by crowdworkers;

  * •

SWDA-IA Sanagavarapu et al. (2022), $\approx$2,500 yes-no questions from
transcriptions of telephone conversations; and

  * •

FRIENDS-QIA Damgaard et al. (2021), $\approx$6,000 yes-no questions from
scripts of the popular Friends TV show.

#### Prompt-Derived Knowledge.

Liu et al., 2022 have shown that integrating knowledge derived from GPT3 in
question-answering models is beneficial. We follow a similar approach by
prompting GPT3 to generate a _disambiguation text_ given a question-answer
pair from Twitter-YN. Then we complement the input to the RoBERTa-based
classifier with the disambiguation text. For example, given Q: _Do you still
trust anything the media says?_ A: _Modern media can make a collective lie to
become the truth_ , GPT3 generates the following disambiguation text:
_Mentions how media can manipulate the truth, they do not trust anything the
media says_. We refer the reader to Appendix G for the specific prompt and
examples of correct and nonsensical disambiguation texts.

###  5.2 GPT3: Zero- and Few-Shot

Given the success of of large language models and prompt engineering in many
tasks Brown et al. (2020), we are keen to see whether they are successful at
interpreting answers to yes-no questions from Twitter. We experiment with
various prompt-based approaches and GPT3. Our prompts do not exactly follow
previous work, but they are inspired by previous work Liu et al. (2021):

Zero-Shot. We use prefix prompts Li and Liang (2021); Lester et al. (2021)
since such prompts work well with generative language models such as GPT3 Liu
et al. (2021). As shown by Efrat and Levy, 2020, we also experiment with the
same zero-shot prompt including the annotation guidelines.

Few-Shot. We experiment with longer versions of the zero-shot prompts that
include two examples from the training split per label (10 examples total; two
versions: with and without the guidelines).

We refer the reader to Appendix G for (a) additional details about the prompts
and examples and (b) specifics about the GPT3 experimental setup.

| All | yes | no | pyes | pno | unk
---|---|---|---|---|---|---
| Accuracy | P | R | F1 |  | F1 | F1 | F1 | F1 | F1
Baseline, Majority Class | 0.39 | 0.15 | 0.39 | 0.22 |  | 0.56 | 0.00 | 0.00 | 0.00 | 0.00
Baseline, Keyword-Based Rules | 0.37 | 0.51 | 0.37 | 0.37 |  | 0.40 | 0.48 | 0.13 | 0.15 | 0.34
with negation cue detector | 0.30 | 0.48 | 0.30 | 0.27 |  | 0.22 | 0.44 | 0.08 | 0.18 | 0.33
RoBERTa trained with yes-no questions corpora |
BoolQ | 0.43 | 0.29 | 0.43 | 0.32 |  | 0.58 | 0.35 | 0.00 | 0.00 | 0.00
Circa | 0.56 | 0.56 | 0.56 | 0.53 |  | 0.70 | 0.59 | 0.32 | 0.06 | 0.35
SwDA-IA | 0.50 | 0.53 | 0.50 | 0.47 |  | 0.67 | 0.52 | 0.26 | 0.02 | 0.15
Friends-QIA | 0.53 | 0.53 | 0.53 | 0.51 |  | 0.67 | 0.55 | 0.22 | 0.00 | 0.44
Twitter-YN (our corpus) |  |  |  |  |  |  |  |  |  |
Question only | 0.38 | 0.41 | 0.38 | 0.29 |  | 0.53 | 0.28 | 0.00 | 0.00 | 0.02
Answer only | 0.53 | 0.50 | 0.53 | 0.48 |  | 0.65 | 0.56 | 0.19 | 0.00 | 0.32
Question and Answer | 0.58 | 0.58 | 0.58 | 0.56 |  | 0.68 | 0.61 | 0.39 | 0.22 | 0.40
RoBERTa blending Twitter-YN |
BoolQ ($\alpha$ = 0.5) | 0.57 | 0.45 | 0.57 | 0.55 |  | 0.70 | 0.60 | 0.31 | 0.06 | 0.44
Circa ($\alpha$ = 0.5) | 0.60 | 0.59 | 0.60 | 0.58 | † | 0.72 | 0.65 | 0.37 | 0.25 | 0.44
+disambiguation text (GPT3) | 0.62 | 0.62 | 0.62 | 0.61 | *† | 0.75 | 0.67 | 0.35 | 0.22 | 0.57
SWDA-IA ($\alpha$ = 0.5) | 0.56 | 0.57 | 0.56 | 0.55 |  | 0.67 | 0.59 | 0.36 | 0.22 | 0.44
Friends-QIA ($\alpha$ = 0.8) | 0.60 | 0.59 | 0.60 | 0.58 | † | 0.71 | 0.63 | 0.38 | 0.20 | 0.45
+disambiguation text (GPT3) | 0.56 | 0.56 | 0.56 | 0.56 |  | 0.70 | 0.63 | 0.36 | 0.24 | 0.37
GPT3, Zero-Shot | 0.44 | 0.55 | 0.44 | 0.46 |  | 0.56 | 0.51 | 0.16 | 0.13 | 0.43
GPT3, Few-Shot | 0.54 | 0.55 | 0.54 | 0.53 |  | 0.67 | 0.55 | 0.14 | 0.30 | 0.49
Table 5: Results in the _unmatched question and unmatched temporal span_ setting. Only training with other corpora obtains modest results, and taking into account the question in addition to the answer is beneficial (Twitter-YN, F1: 0.56 vs. 0.48). Blending Circa and Friends-QIA is beneficial (F1: 0.58 vs. 0.56). Adding the disambiguation text from GPT3 with Circa is also beneficial (F1: 0.61). Few-Shot prompt with GPT3 obtains results close to a supervised model trained with Twitter-YN (F1: 0.53 vs. 0.56), but underperforms blending Circa using the disambiguation text from GPT3 (0.61). Statistical significance with respect to training with Twitter-YN (Q and A) is indicated with ‘*’ and with respect to few-shot GPT3 with ‘†’ (McNemar’s Test McNemar (1947), p<0.05).  Error Type | Question | Answer | P, | G
---|---|---|---|---
_Yes_ distractor |  |  | y, | n
(4.4%) |  |
_No_ distractor |  |  | n, | y
(28.6%) |  |
Social media |  |  | u, | y
lexicon (20.7%) |  |
Humor |  |  | pn, | py
(13.2%) |  |
Condition or |  |  | u, | py
contrast (29.1%) |  |
External entities |  |  | u, | y
(14.9%) |  |
Unresponsive |  |  | y, | u
(11.4%) |  |
Table 6:  Most frequent errors made by the best performing system in the
_unmatched question and unmatched temporal span_ setting (Table 5). Error
types may overlap, and P and G stand for _p_ rediction and _g_ round truth
respectively. In addition to _yes_ and _no_ distractors, social media lexicon
(emojis, abbreviation, etc.), humor (including irony and sarcasm), presence of
(a) conditions or contrasts or (b) external entities are common errors.

##  6 Results and Analysis

We present results in the _unmatched questions and unmatched temporal span_
setting in Table 5. Appendix C presents the results in the other two settings.
We will focus the discussion on average F1 scores as Twitter-YN is unbalanced
(Section 4).

Regarding the baselines, using a negation cue detector to identify _no_
keywords rather than our predefined list is detrimental (F1: 0.27 vs. 0.37),
especially with _yes_ label. This result leads to the conclusion that many
answers with ground truth interpretation _yes_ include negation cues such as
affixes _im-_ and _-less_ , which are identified by the cue detector but are
not in our list. Zero- and Few-Shot GPT3 outperforms the baselines, and
obtains moderately lower results than the supervised model trained on Twitter-
YN (Question and Answer; F1: 0.53 vs. 0.56). Including the annotation
guidelines in the zero and few-shot prompts obtains worse results (Appendix G,
Table 18).

The supervised RoBERTa-based systems trained with other corpora yield mixed
results. Circa, SWDA-IA, and Friends-QIA are the ones which outperform the
keyword-based baseline (0.53, 0.47 and 0.51 vs. 0.37). We hypothesize that
this is due to the fact that BoolQ includes yes-no questions from formal texts
(Wikipedia). Unsurprisingly, training with Twitter-YN yields the best results.
The question alone is insufficient (F1: 0.29), but combining it with the
answer yields better results than the answer alone (0.56 vs. 0.48).

It is observed that Friends-QIA and Circa are the only corpora worth blending
with Twitter-YN. Both obtain better results than training with only Twitter-YN
(F1: 0.58 in both vs. 0.56). Finally, including in the input the
disambiguation texts automatically generated by GPT3 further improves the
results (F1: 0.61) blending with Circa. On the other hand, disambiguation
texts are detrimental when blending with Friends-QIA. These results were
surprising to us as disambiguation texts are often nonsensical. Appendix G
(Table 17) provides several examples of disambiguation texts generated by
GPT3, including sensical and nonsensical texts.

Regarding statistical significance, it is important to note that blending
Circa and using disambiguation texts yields statistically significantly better
results than few-shot GPT3. GPT3, however, is by no means useless: it provides
a solid few-shot baseline and the disambiguation texts can be used to
strengthen the supervised models.

###  6.1 Error Analysis

Table 6 illustrates the most common errors made by our best performing model
(RoBERTa blending Twitter-YN (with disambiguation text generated by GPT3) and
Circa, Table 5). note that error types may overlap. For example, a wrongly
predicted instance may include humor and a _no_ distractor).

_Yes_ and _no_ keywords are not good indicators of answer interpretations
(Section 4), and they act as distractors. In the first and second examples,
the answers include _sure_ and _n’t_ , but their interpretations are _no_ and
_yes_ respectively. Indeed, we observe them in 33% of errors, especially _no_
distractors (28.6%). Many errors (20.7%) contain social media lexicon items
such as emojis and hashtags, making their interpretation more difficult. Humor
including irony is also a challenge (13.2% of errors). In the example, which
also includes a contrast, the answer implies that the author likes coffee as
the fruit fly has a positive contribution.

Answers with conditions and contrasts are rarely to be interpreted _unknown_ ,
yet the model often does so. Mentioning external entities also poses a
challenge as the model does not have any explicit knowledge about them.
Finally, unresponsive answers are always to be interpreted _unknown_ , yet the
model sometimes (11.4% of errors) fails to properly interpret unresponsive
answers and mislabels them as either _yes_ or _no_.

##  7 Conclusions

We have presented Twitter-YN, the first corpus of yes-no questions and answers
from social media annotated with their interpretations. Importantly, both
questions and answer were posted by real users rather than written by
crowdworkers on demand. As opposed to traditional question answering, the
problem is not to find answers to questions, but rather to _interpret_ answers
to yes-no questions.

Our analysis shows that answers to yes-no questions vary widely and they
rarely include a _yes_ or _no_ keyword. Even if they do, the underlying
interpretation is rarely what one may think the keyword suggests. Answers to
yes-no questions usually are long explanations without stating the underlying
interpretation (_yes_ , _probably yes_ , _unknown_ , _probably no_ or _no_).
Experimental results show that the problem is challenging for state-of-the-art
models, including large language models such as GPT3.

Our future plans include exploring combination of neural- and knowledge-based
approaches. In particular, we believe that commonsense resources such as
ConceptNet Speer et al. (2017), ATOMIC Sap et al. (2019), and CommonsenseQA
Talmor et al. (2019) may be helpful. The challenge is to differentiate between
common or recommended behavior (or commonly held commonsense) and what
somebody answered in social media. Going back to the example in Figure 1, oven
manuals and recipes will always instruct people to preheat the oven. The
problem is not to obtain a universal ground truth—it does not exist. Instead,
the problem is to leverage commonsense and reasoning to reveal nuances about
how indirect answers to yes-no questions ought to be interpreted.

## Limitations

Twitter-YN, like any other annotated corpus, is not necessarily representative
(and it certainly is not a complete picture) of the problem at hand—in our
case, interpreting answers to yes-no questions in user generated content.
Twitter-YN may not transfer to other social media platforms. More importantly,
our process to select yes-no questions and answers disregards many yes-no
questions. In addition to working only with English, we disregard yes-no
questions without question marks (e.g., _I wonder what y’all think about
preheating the oven_); the questions we work with are limited to those
containing four bigrams. We exclude unverified accounts to avoid spam and
maximize the chances of finding answers (verified accounts are more popular),
but this choice ignores many yes-no questions. Despite the limitations, we
believe that the variety of verbs and including two temporal spans alleviates
this issue. Regardless of size, no corpus will be complete.

Another potential limitation is the choice of labels. We are inspired by
previous work (Section 2 and 3), some of which work with five labels and
others with as many as nine. We found no issues using five labels during the
annotation effort, but we certainly have no proof that five is the best
choice.

Lastly, we report negative results using several prompts and GPT3—although
prompting GPT3 is the best approach if zero or few training examples are
available. Previous work on prompt engineering and recent advances in large
language models makes us believe that it is possible that researchers
specializing in prompts may come up with better prompts (and results) than we
did. Our effort on prompting and GPT3, however, is an honest and thorough
effort to the best of our ability with the objective of empirically finding
the optimal prompts. Since these large models learn from random and
maliciously designed prompts Webson and Pavlick (2022), we acknowledge that
other prompts may yield better results.

## Ethics Statement

While tweets are publicly available, users may not be aware that their public
tweets can be used for virtually any purpose. Twitter-YN will comply with the
Twitter Terms of Use. Twitter-YN will not include any user information,
although we acknowledge that it is easily accessible given the tweet id and
basic programming skills. The broad temporal spans we work with minimize the
opportunities for monitoring users, but it is still possible.

While it is far from our purposes, models to interpret answers to yes-no
questions could be used for monitoring individuals and groups of people.
Malicious users could post questions about sensitive topics, including
politics, health, and deeply held believes. In turn, unsuspecting users may
reply with their true opinion, thereby exposing themselves to whatever the
malicious users may want to do (e.g., deciding health insurance quotes based
on whether somebody enjoys exercising). That said, this hypothetical scenario
also opens the door to users to answer whatever they wish, truthful or not—and
in the process, mislead the hypothetical monitoring system. The work presented
here is about interpreting answers to yes-no questions in Twitter, not
figuring the “true” answer. If somebody makes a Twitter persona, we make no
attempt to reveal the real person underneath.

## Acknowledgments

This material is based upon work supported by the National Science Foundation
under Grant No. 1845757. Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the authors and do not
necessarily reflect the views of the NSF. We would like to acknowledge the
Chameleon platform Keahey et al. (2020) for providing computational resources.

## References

  * Abadi et al. (2015) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015.  TensorFlow: Large-scale machine learning on heterogeneous systems.  Software available from tensorflow.org.
  * Artstein and Poesio (2008) Ron Artstein and Massimo Poesio. 2008.  Survey article: Inter-coder agreement for computational linguistics.  _Computational Linguistics_ , 34(4):555–596.
  * Bender and Friedman (2018) Emily M. Bender and Batya Friedman. 2018.  Data statements for natural language processing: Toward mitigating system bias and enabling better science.  _Transactions of the Association for Computational Linguistics_ , 6:587–604.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  _CoRR_ , abs/2005.14165.
  * Carletta et al. (1997) Jean Carletta, Amy Isard, Stephen Isard, Jacqueline C. Kowtko, Gwyneth Doherty-Sneddon, and Anne H. Anderson. 1997.  The reliability of a dialogue structure coding scheme.  _Computational Linguistics_ , 23(1):13–31.
  * Cheng et al. (2022) Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. 2022.  HiTab: A hierarchical table dataset for question answering and natural language generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1094–1110, Dublin, Ireland. Association for Computational Linguistics.
  * Choi et al. (2018) Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018.  QuAC: Question answering in context.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2174–2184, Brussels, Belgium. Association for Computational Linguistics.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  BoolQ: Exploring the surprising difficulty of natural yes/no questions.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Crossley et al. (2017) Scott A Crossley, Kristopher Kyle, and Danielle S McNamara. 2017.  Sentiment analysis and social cognition engine (seance): An automatic tool for sentiment, social cognition, and social-order analysis.  _Behavior research methods_ , 49(3):803–821.
  * Cui et al. (2017) Lei Cui, Shaohan Huang, Furu Wei, Chuanqi Tan, Chaoqun Duan, and Ming Zhou. 2017\.  SuperAgent: A customer service chatbot for E-commerce websites.  In _Proceedings of ACL 2017, System Demonstrations_ , pages 97–102, Vancouver, Canada. Association for Computational Linguistics.
  * Damgaard et al. (2021) Cathrine Damgaard, Paulina Toborek, Trine Eriksen, and Barbara Plank. 2021.  “I’ll be there for you”: The one with understanding indirect answers.  In _Proceedings of the 2nd Workshop on Computational Approaches to Discourse_ , pages 1–11, Punta Cana, Dominican Republic and Online. Association for Computational Linguistics.
  * de Marneffe et al. (2009) Marie-Catherine de Marneffe, Scott Grimm, and Christopher Potts. 2009.  Not a simple yes or no: Uncertainty in indirect answers.  In _Proceedings of the SIGDIAL 2009 Conference_ , pages 136–143, London, UK. Association for Computational Linguistics.
  * Efrat and Levy (2020) Avia Efrat and Omer Levy. 2020.  The turking test: Can language models understand instructions? _CoRR_ , abs/2010.11982.
  * Green and Carberry (1999) Nancy Green and Sandra Carberry. 1999.  Interpreting and generating indirect answers.  _Computational Linguistics_ , 25(3):389–435.
  * Harrigian et al. (2020) Keith Harrigian, Carlos Aguirre, and Mark Dredze. 2020.  Do models of mental health based on social media data generalize? In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3774–3788, Online. Association for Computational Linguistics.
  * Hockey et al. (1997) Beth Ann Hockey, Deborah Rossen-Knill, Beverly Spejewski, Matthew Stone, and Stephen Isard. 1997.  Can you predict responses to yes/no questions? yes, no, and stuff.  pages 2267–2270.  Publisher Copyright: © 1997 5th European Conference on Speech Communication and Technology, EUROSPEECH 1997. All rights reserved.; 5th European Conference on Speech Communication and Technology, EUROSPEECH 1997 ; Conference date: 22-09-1997 Through 25-09-1997.
  * Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.  To appear.
  * Hossain et al. (2020) Md Mosharaf Hossain, Kathleen Hamilton, Alexis Palmer, and Eduardo Blanco. 2020\.  Predicting the focus of negation: Model and error analysis.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8389–8401, Online. Association for Computational Linguistics.
  * Keahey et al. (2020) Kate Keahey, Jason Anderson, Zhuo Zhen, Pierre Riteau, Paul Ruth, Dan Stanzione, Mert Cevik, Jacob Colleran, Haryadi S. Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, François Halbach, Alex Rocha, and Joe Stubbs. 2020.  Lessons learned from the chameleon testbed.  In _Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC ’20)_. USENIX Association.
  * Khashabi et al. (2020) Daniel Khashabi, Tushar Khot, and Ashish Sabharwal. 2020.  More bang for your buck: Natural perturbation for robust question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 163–170, Online. Association for Computational Linguistics.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lampos et al. (2013) Vasileios Lampos, Daniel Preoţiuc-Pietro, and Trevor Cohn. 2013.  A user-centric model of voting intention from social media.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 993–1003, Sofia, Bulgaria. Association for Computational Linguistics.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020\.  MLQA: Evaluating cross-lingual extractive question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7315–7330, Online. Association for Computational Linguistics.
  * Li et al. (2017) Huayu Li, Martin Renqiang Min, Yong Ge, and Asim Kadav. 2017.  A context-aware attention network for interactive question answering.  In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’17, page 927–935, New York, NY, USA. Association for Computing Machinery.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Liu et al. (2020) Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, and Nan Duan. 2020.  RikiNet: Reading Wikipedia pages for natural question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6762–6771, Online. Association for Computational Linguistics.
  * Liu et al. (2022) Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi. 2022.  Generated knowledge prompting for commonsense reasoning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3154–3169, Dublin, Ireland. Association for Computational Linguistics.
  * Liu et al. (2021) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _CoRR_ , abs/2107.13586.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Louis et al. (2020) Annie Louis, Dan Roth, and Filip Radlinski. 2020.  “I’d rather just go to bed”: Understanding indirect answers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7411–7425, Online. Association for Computational Linguistics.
  * Ma and Gao (2020) Jing Ma and Wei Gao. 2020.  Debunking rumors on Twitter with tree transformer.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 5455–5466, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * McNemar (1947) Quinn McNemar. 1947.  Note on the sampling error of the difference between correlated proportions or percentages.  _Psychometrika_ , 12(2):153–157.
  * Mehta et al. (2022) Nikhil Mehta, Maria Leonor Pacheco, and Dan Goldwasser. 2022.  Tackling fake news detection by continually improving social context representations using graph neural networks.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1363–1380, Dublin, Ireland. Association for Computational Linguistics.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics.
  * Miller (1995) George A Miller. 1995.  Wordnet: a lexical database for english.  _Communications of the ACM_ , 38(11):39–41.
  * Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019.  CoQA: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ , 7:249–266.
  * Rossen-Knill et al. (1997) Deborah Rossen-Knill, Beverly Spejewski, Beth Hockey, Stephen Isard, and Matthew Stone. 1997.  Yes/no questions and answers in the map task corpus.  _IRCS Technical Reports Series_.
  * Sanagavarapu et al. (2022) Krishna Sanagavarapu, Jathin Singaraju, Anusha Kakileti, Anirudh Kaza, Aaron Mathews, Helen Li, Nathan Brito, and Eduardo Blanco. 2022.  Disentangling indirect answers to yes-no questions in real conversations.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4677–4695, Seattle, United States. Association for Computational Linguistics.
  * Sap et al. (2019) Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019.  Atomic: An atlas of machine commonsense for if-then reasoning.  In _Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence_ , AAAI’19/IAAI’19/EAAI’19. AAAI Press.
  * Sheng et al. (2021) Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021.  “nice try, kiddo”: Investigating ad hominems in dialogue responses.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 750–767, Online. Association for Computational Linguistics.
  * Shnarch et al. (2018) Eyal Shnarch, Carlos Alzate, Lena Dankin, Martin Gleize, Yufang Hou, Leshem Choshen, Ranit Aharonov, and Noam Slonim. 2018.  Will it blend? blending weak and strong labeled data in a neural network for argumentation mining.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 599–605, Melbourne, Australia. Association for Computational Linguistics.
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _Thirty-first AAAI conference on artificial intelligence_.
  * Stolcke et al. (2000) Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000.  Dialogue act modeling for automatic tagging and recognition of conversational speech.  _Computational Linguistics_ , 26(3):339–374.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Webson and Pavlick (2022) Albert Webson and Ellie Pavlick. 2022.  Do prompt-based models really understand the meaning of their prompts? In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2300–2344, Seattle, United States. Association for Computational Linguistics.
  * Xiong et al. (2019) Wenhan Xiong, Jiawei Wu, Hong Wang, Vivek Kulkarni, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019.  TWEETQA: A social media focused question answering dataset.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5020–5031, Florence, Italy. Association for Computational Linguistics.
  * Yang et al. (2015) Yi Yang, Wen-tau Yih, and Christopher Meek. 2015.  WikiQA: A challenge dataset for open-domain question answering.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 2013–2018, Lisbon, Portugal. Association for Computational Linguistics.
  * Yasunaga et al. (2021) Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021.  QA-GNN: Reasoning with language models and knowledge graphs for question answering.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 535–546, Online. Association for Computational Linguistics.

##  Appendix A Annotation Guidelines

We give four annotators the following annotation guidelines for each of the
five labels. These guidelines are also used in some of the zero-shot and few-
shot prompts.

  1. 1.

_yes_ (y)
The reply is an affirmative reply to the question without reservations.
Or, the reply states the question in affirmative terms. For example, _Mexican
food is delicious._ states _Do you like Mexican food?_ in positive terms.

  2. 2.

_probably yes_ (py)
The reply is an affirmative reply under certain conditions.
Or, the reply states the question in hopeful terms. For example, _I am keen on
eating tacos this weekend_ states _Do you like Mexican food?_ in hopeful
terms.
Or, the reply is affirmative but refers to a point of time in the future.
Or, the reply states that an entity (person, group of people, place,
organization, etc.) leans towards _yes_. For example, _Q: Do you like Mexican
food? A: My whole family loves it._

  3. 3.

_no_ (n)
The reply is a negative reply to the question without reservations.
Or, the reply states the question in negative terms. For example, _Mexican
food is bland_ states _Do you like Mexican food?_ in negative terms.

  4. 4.

_probably no_ (pn)
The reply is a negative reply under certain conditions.
Or, the reply states that an entity (person, group of people, place,
organization, etc.) leans towards no. For example, _Q: Do you like Mexican
food? A: My friends avoid Mexican restaurants._

  5. 5.

_unknown_ (uk)
A reply that is responsive to the question but does not lean towards _yes_ or
_no_.
Or, a reply that does not address the question at all.

##  Appendix B Data Statement

As per the recommendations by Bender and Friedman (2018), we provide here a
Data Statement for better understanding of Twitter-YN.

###  B.1 Curation Rationale

We develop Twitter-YN to build and evaluate models to interpret answers to
yes-no questions in user-generated content. Twitter-YN includes yes-no
questions and answers from Twitter along with annotations indicating the
interpretation of the answers.

Questions come from main tweets (i.e., any tweet except replies and retweets).
Answers are reply tweets to the question tweets (retweets do not count as
replies). Questions and answers are selected based on manually defined rules.
These rules are developed and refined through an iterative process. We scrape
Twitter for question-answer pairs and assign 100 instances to four annotators
(each annotator gets the same instances). Based on (a) feedback by each of
these four annotators on the questions and answers and (b) agreement scores
(linearly weighted Cohen’s $\kappa$), the rules are refined. A new set of 100
random instances is then collected based on the refined rules. This is once
again given for annotation to the four annotators. This process continues
until annotators find that the questions and answers identified with the rules
are actual yes-no questions and answers to the questions.

The final version of the rules used to scrape yes-no questions and answers
from Twitter are detailed in Section 3.1. Section 3.2 provides descriptions of
the five interpretations annotators choose from (i.e., annotation guidelines),
and Table 1 presents examples. All question-answer pairs (4,442) were
annotated by two annotators independently. Four annotators participated in the
annotations. Inter-annotator agreement (linearly weighted Cohen’s $\kappa$) is
0.68, indicating substantial agreement Artstein and Poesio (2008); over 0.8
would be nearly perfect. After the independent double annotation process, the
adjudicator resolves the disagreements to create the ground truth.

###  B.2 Language Variety

The data collection process was carried out from May to July 2022. Question
tweets are in English (‘en’ as per the Twitter API). We also use SpaCy to
confirm that the reply is in English. Information on the specific type of
English (American, British, Australian etc.) is not available.

###  B.3 Speaker Demographic

The questions come from 1,200 unique question tweets posted by verified
twitter accounts (as of July 2022). We look specifically at verified users
only, since such tweets are more popular and are therefore more likely to have
replies. We do not require replies to come from verified accounts. As per
Twitter age restrictions, the minimum user age is 13 years for authors of both
questions and answers. Speakers are not reachable in our scenario, thus
demographic information is limited.

###  B.4 Annotator Demographic

Four annotators (including one adjudicator) are part of the annotation process
and development of annotation guidelines. All of the individuals are graduate
students who are fluent in English. Annotators are three men and one woman,
and their ages range from 18 to 40 years old. Ethnic backgrounds are as
follows: one Asian and three South Asian. Annotators reported that they are
from a middle class economic background.

The final version of the annotation guidelines is developed by considering the
annotator feedback during pilot annotations. There is no overlap between the
questions and answers used in the pilot annotations and the ones that are in
Twitter-YN.

###  B.5 Speaker Situation

Text in Twitter-YN is retrieved from Twitter between May and July of 2022.
Modality of text is written (typed by speaker). Twitter allows users
(speakers) to edit what they tweet. We use the version of the tweet available
as of July 2022 regardless of the original posting date. Twitter-YN includes
asynchronous interactions since reply tweets are posted only after the tweet
with the question is posted. Questions and answers cannot be appear in Twitter
simultaneously. The intended audience for the text could be anyone on the
Internet.

###  B.6 Text Characteristics

Genre of text is in the social media domain. Question-answer instances are
generally based on topics such as general advice, day-to-day chores, health,
food, finance, music, policies, and politics. The text is informal. Social
media cues such as shorter forms of phrases (e.g., lol: laughing out loud),
slang, and emojis are common. Most tweets are text only (including Unicode
symbols such as emojis), although a few also contain images. Twitter-YN does
not include these images.

###  B.7 Recording Quality

N/A

###  B.8 Other

N/A

###  B.9 Provenance Appendix

N/A

##  Appendix C Results with the other Two Experimental Settings

Tables 7 and 8 present results with the _unmatched question and matched
temporal span_ and _matched question and matched temporal span_ settings.
These tables complement the results discussed in Section 5. While valid, these
settings are unrealistic in terms of what the model is trained and evaluated
with. In the first setting, models are trained with question-answer pairs
posted during the same temporal span than those in the test split. In the
second setting, models are trained with answers to the same questions seen
during training.

Since the test splits are not the same, the results in Tables 7 and 8 are not
comparable. Because of the same reason, the results in Table 5 and either
Table 7 or 8 cannot be compared either.

| All | yes | no | pyes | pno | unk
---|---|---|---|---|---|---
| Accuracy | P | R | F1 |  | F1 | F1 | F1 | F1 | F1
Baseline, Majority Class | 0.41 | 0.17 | 0.41 | 0.24 |  | 0.58 | 0.00 | 0.00 | 0.00 | 0.00
Baseline, Keyword-Based Rules | 0.38 | 0.55 | 0.38 | 0.39 |  | 0.38 | 0.49 | 0.19 | 0.25 | 0.35
with negation cue detector | 0.32 | 0.50 | 0.32 | 0.30 |  | 0.23 | 0.44 | 0.12 | 0.29 | 0.31
RoBERTa trained with yes-no questions corpora |
BoolQ | 0.44 | 0.36 | 0.44 | 0.32 |  | 0.59 | 0.29 | 0.00 | 0.00 | 0.00
Circa | 0.58 | 0.56 | 0.58 | 0.54 |  | 0.70 | 0.61 | 0.38 | 0.10 | 0.26
SwDA-IA | 0.53 | 0.45 | 0.53 | 0.44 |  | 0.65 | 0.56 | 0.00 | 0.00 | 0.09
Friends-QIA | 0.56 | 0.53 | 0.56 | 0.53 |  | 0.71 | 0.58 | 0.30 | 0.00 | 0.28
Twitter-YN (our corpus) |  |  |  |  |  |  |  |  |  |
Question only | 0.41 | 0.27 | 0.41 | 0.30 |  | 0.57 | 0.24 | 0.00 | 0.00 | 0.00
Answer only | 0.53 | 0.47 | 0.53 | 0.47 |  | 0.64 | 0.59 | 0.00 | 0.00 | 0.25
Question and Answer | 0.58 | 0.54 | 0.58 | 0.56 |  | 0.70 | 0.60 | 0.23 | 0.00 | 0.45
RoBERTa blending Twitter-YN (question and answer) and other yes-no questions corpora |
BoolQ ($\alpha$ = 0.5) | 0.60 | 0.61 | 0.60 | 0.60 | * | 0.71 | 0.62 | 0.37 | 0.25 | 0.52
Circa ($\alpha$ = 0.2) | 0.61 | 0.58 | 0.61 | 0.58 |  | 0.72 | 0.67 | 0.30 | 0.14 | 0.38
SWDA-IA ($\alpha$ = 0.5) | 0.58 | 0.56 | 0.58 | 0.57 |  | 0.68 | 0.63 | 0.26 | 0.22 | 0.45
Friends-QIA ($\alpha$ = 0.8) | 0.57 | 0.57 | 0.57 | 0.57 |  | 0.70 | 0.62 | 0.31 | 0.18 | 0.40
Table 7: Results with the test split of Twitter-YN (our corpus) in the _unmatched question and matched temporal span_ setting. We present results with (a) baselines, (b) training with yes-no questions corpora (other corpora and the training split of Twitter-YN), and (c) blending the training split of Twitter-YN and other corpora. While direct comparison is unsound because the training and test splits differ, we observe either the same results or just slightly higher than the ones in the _unmatched question and unmatched temporal span_ setting (Table 5), especially when blending (last block). In other words, training with questions and answers of the new temporal span does not present an advantage. Statistical significance with respect to training RoBERTa trained with Twitter-YN (Question and Answer, second block) is indicated with and asterisk (McNemar’s Test McNemar (1947), $p<0.05$)  | All | yes | no | pyes | pno | unk
---|---|---|---|---|---|---
| Accuracy | P | R | F1 |  | F1 | F1 | F1 | F1 | F1
Baseline, Majority Class | 0.40 | 0.18 | 0.40 | 0.23 |  | 0.57 | 0.00 | 0.00 | 0.00 | 0.00
Baseline, Keyword-Based Rules | 0.34 | 0.49 | 0.34 | 0.34 |  | 0.35 | 0.45 | 0.19 | 0.16 | 0.29
with negation cue detector | 0.29 | 0.52 | 0.29 | 0.29 |  | 0.23 | 0.45 | 0.15 | 0.17 | 0.27
RoBERTa trained with yes-no questions corpora |
BoolQ | 0.50 | 0.36 | 0.50 | 0.40 |  | 0.63 | 0.49 | 0.00 | 0.00 | 0.00
Circa | 0.55 | 0.52 | 0.55 | 0.51 |  | 0.67 | 0.58 | 0.31 | 0.00 | 0.25
SwDA-IA | 0.52 | 0.60 | 0.52 | 0.52 |  | 0.68 | 0.59 | 0.20 | 0.13 | 0.26
Friends-QIA | 0.56 | 0.55 | 0.56 | 0.54 |  | 0.65 | 0.62 | 0.34 | 0.00 | 0.42
Twitter-YN (our corpus) |  |  |  |  |  |  |  |  |  |
Question only | 0.50 | 0.40 | 0.50 | 0.43 |  | 0.63 | 0.50 | 0.00 | 0.00 | 0.18
Answer only | 0.56 | 0.57 | 0.56 | 0.53 |  | 0.67 | 0.58 | 0.33 | 0.18 | 0.32
Question and Answer | 0.63 | 0.74 | 0.63 | 0.60 |  | 0.71 | 0.66 | 0.41 | 0.13 | 0.47
RoBERTa blending Twitter-YN (question and answer) and other yes-no questions corpora |
BoolQ ($\alpha$ = 0.5) | 0.66 | 0.66 | 0.66 | 0.64 |  | 0.75 | 0.69 | 0.40 | 0.23 | 0.56
Circa ($\alpha$ = 0.5) | 0.64 | 0.61 | 0.64 | 0.62 |  | 0.76 | 0.70 | 0.32 | 0.14 | 0.45
SWDA-IA ($\alpha$ = 0.5) | 0.64 | 0.62 | 0.64 | 0.62 |  | 0.76 | 0.67 | 0.38 | 0.13 | 0.46
Friends-QIA ($\alpha$ = 0.8) | 0.65 | 0.62 | 0.65 | 0.63 |  | 0.74 | 0.71 | 0.36 | 0.06 | 0.56
Table 8: Results with the test split of Twitter-YN (our corpus) in the
_matched question and matched temporal span_ setting. We present results with
(a) baselines, (b) training with yes-no questions corpora (other corpora and
the training split of Twitter-YN), and (c) blending the training split of
Twitter-YN and other corpora. While direct comparison is unsound because the
training and test splits differ, we generally observe higher or the same
results than the ones in the _unmatched question and unmatched temporal span_
setting (Table 5). Unsurprisingly, using (different) answers to the same
questions in training and testing is beneficial. Note, however, that these
results are unrealistic.

##  Appendix D Hyperparameters

All RoBERTa-based classifiers were tuned using the train and validation
splits. Results (Tables 5 and 7–8, 12–14), were obtained with the test split
after the tuning process. We experimented with the following ranges for
hyperparameters, and chose the optimal values based on the loss calculated
with the validation split:

  * •

Learning rate: 1e-5, 2e-5, 3e-5

  * •

Epochs: Up to 200 with early stopping (patience = 3, min_delta = 0.01)

  * •

Batch size: 16, 32

  * •

Blending factor ($\alpha$): 0.2–0.8

Tables 9–11 present the tuned hyperparameters in the three experimental
settings. These tables complement the results presented in Tables 5, 7, and 8.
Hyperparameters were tuned with the train and validation splits.

Experiment | Learning Rate | Epochs | Batch Size
---|---|---|---
RoBERTa trained with yes-no questions corpora
BoolQ | 3e-5 | 1 | 16
Circa | 3e-5 | 3 | 16
SWDA-IA | 2e-5 | 3 | 16
Friends-QIA | 2e-5 | 2 | 16
Twitter-YN (our corpus) |  |  |
Question | 1e-5 | 2 | 16
Answer | 3e-5 | 2 | 16
Questions and Answer | 2e-5 | 2 | 16
RoBERTa pretraining with yes-no question corpora + finetuning with Twitter-YN
BoolQ | 3e-5 | 1+1 | 16
Circa | 3e-5 | 3+1 | 16
SWDA-IA | 2e-5 | 3+1 | 16
Friends-QIA | 2e-5 | 2+1 | 16
RoBERTa blending Twitter-YN (question and answer) and other yes-no questions
corpora
BoolQ | 3e-5 | 1 | 16
Circa | 2e-5 | 1 | 16
+disambiguation text (GPT3) | 2e-5 | 2 | 16
SWDA-IA | 1e-5 | 2 | 16
Friends-QIA | 2e-5 | 1 | 16
+disambiguation text (GPT3) | 2e-5 | 2 | 16
Table 9:  Hyperparameters with the _unmatched question and unmatched temporal span_ setting found after tuning with the development set. This table complements Table 5.  Experiment | Learning Rate | Epochs | Batch Size
---|---|---|---
RoBERTa trained with yes-no questions corpora
BoolQ | 1e-5 | 1 | 16
Circa | 3e-5 | 1 | 16
SWDA-IA | 3e-5 | 2 | 16
Friends-QIA | 3e-5 | 3 | 16
Twitter-YN (our corpus) |  |  |
Question | 1e-5 | 1 | 16
Answer | 3e-5 | 2 | 16
Question and Answer | 2e-5 | 2 | 16
RoBERTa pretraining with yes-no question corpora + finetuning with Twitter-YN
BoolQ | 1e-5 | 1+2 | 16
Circa | 3e-5 | 1+1 | 16
SWDA-IA | 3e-5 | 2+2 | 16
Friends-QIA | 3e-5 | 3+1 | 16
RoBERTa blending Twitter-YN (question and answer) and other yes-no questions
corpora
BoolQ | 2e-5 | 3 | 16
Circa | 3e-5 | 3 | 16
SWDA-IA | 1e-5 | 3 | 16
Friends-QIA | 2e-5 | 2 | 16
Table 10:  Hyperparameters with the _unmatched question and matched temporal span_ setting found after tuning with the development set. This table complements Table 7.  Experiment | Learning Rate | Epochs | Batch Size
---|---|---|---
RoBERTa trained with yes-no questions corpora
BoolQ | 1e-5 | 3 | 16
Circa | 3e-5 | 1 | 16
SWDA-IA | 2e-5 | 3 | 16
Friends-QIA | 3e-5 | 3 | 16
Twitter-YN (our corpus) |  |  |
Question | 2e-5 | 3 | 16
Answer | 3e-5 | 2 | 16
Question and Answer | 2e-5 | 2 | 16
RoBERTa pretraining with yes-no question corpora + finetuning with Twitter-YN
BoolQ | 1e-5 | 3+3 | 16
Circa | 3e-5 | 1+2 | 16
SWDA-IA | 2e-5 | 3+2 | 16
Friends-QIA | 3e-5 | 3+1 | 16
RoBERTa blending Twitter-YN (question and answer) and other yes-no questions
corpora
BoolQ | 1e-5 | 3 | 16
Circa | 2e-5 | 2 | 16
SWDA-IA | 1e-5 | 2 | 16
Friends-QIA | 2e-5 | 2 | 16
Table 11:  Hyperparameters with the _matched question and matched temporal
span_ setting found after tuning with the development set. This table
complements Table 8.

##  Appendix E Results Pretraining with Related Corpora

Tables 12–14 present results pretraining with yes-no questions and then fine-
tuning with Twitter-YN rather than blending. These tables complement Tables 5,
7, and 8. Although pretraining sometimes outperforms blending in two settings,
blending always outperforms pretraining in the most important setting:
_unmatched question and unmatched temporal span_.

| Overall |  |  | yes | no | pyes | pno | unk
---|---|---|---|---|---|---|---|---
| Accuracy | P | R | F1 |  |  | F1 | F1 | F1 | F1 | F1
BoolQ | 0.51 | 0.54 | 0.51 | 0.41 |  |  | 0.64 | 0.57 | 0.03 | 0.00 | 0.02
Circa | 0.60 | 0.60 | 0.60 | 0.58 |  |  | 0.71 | 0.62 | 0.38 | 0.22 | 0.45
SwDA-IA | 0.55 | 0.55 | 0.55 | 0.51 |  |  | 0.69 | 0.55 | 0.36 | 0.00 | 0.33
Friends-QIA | 0.58 | 0.55 | 0.58 | 0.54 |  |  | 0.70 | 0.60 | 0.26 | 0.00 | 0.42
Table 12:  Results with RoBERTa pretrained on relevant yes-no questions corpora and fine-tuned on Twitter-YN in the _unmatched question and unmatched temporal span_ setting. Pretraining obtains worse results than blending (Table 5).  | Overall |  |  | yes | no | pyes | pno | unk
---|---|---|---|---|---|---|---|---
| Accuracy | P | R | F1 |  |  | F1 | F1 | F1 | F1 | F1
BoolQ | 0.59 | 0.51 | 0.59 | 0.54 |  |  | 0.72 | 0.62 | 0.00 | 0.00 | 0.42
Circa | 0.61 | 0.63 | 0.61 | 0.59 |  |  | 0.74 | 0.63 | 0.25 | 0.06 | 0.47
SwDA-IA | 0.57 | 0.49 | 0.57 | 0.51 |  |  | 0.71 | 0.59 | 0.00 | 0.00 | 0.33
Friends-QIA | 0.57 | 0.55 | 0.57 | 0.51 |  |  | 0.69 | 0.59 | 0.23 | 0.00 | 0.25
Table 13:  Results with RoBERTa pretrained on relevant yes-no questions corpora and fine-tuned on Twitter-YN in the _unmatched question and matched temporal span_ setting.  | Overall |  |  | yes | no | pyes | pno | unk
---|---|---|---|---|---|---|---|---
| Accuracy | P | R | F1 |  |  | F1 | F1 | F1 | F1 | F1
BoolQ | 0.59 | 0.56 | 0.59 | 0.56 |  |  | 0.70 | 0.62 | 0.19 | 0.00 | 0.48
Circa | 0.64 | 0.62 | 0.64 | 0.62 |  |  | 0.72 | 0.71 | 0.36 | 0.18 | 0.47
SwDA-IA | 0.62 | 0.64 | 0.62 | 0.61 |  |  | 0.73 | 0.69 | 0.39 | 0.00 | 0.50
Friends-QIA | 0.61 | 0.64 | 0.61 | 0.60 |  |  | 0.71 | 0.67 | 0.40 | 0.06 | 0.50
Table 14:  Results with RoBERTa pretrained on relevant yes-no questions
corpora and fine-tuned on Twitter-YN in the _matched question and matched
temporal span_ setting.

##  Appendix F Detailed Error Analysis

Tables 15 and 16 detail the error types exemplified in Table 6. Specifically,
we provide the percentages of errors for each combination of ground truth
(gold) and predicted interpretations.

Error Type | Ground Truth | Predicted | Percentage
---|---|---|---
_Yes_ distractor | no | yes | 0.88
| yes | no | 0.88
| probably yes | yes | 0.44
| yes | probably yes | 0.44
| unknown | no | 0.44
| no | unknown | 0.44
| no | probably yes | 0.44
| yes | unknown | 0.44
_No_ distractor | unknown | no | 5.29
| yes | unknown | 5.29
| yes | no | 3.52
| probably yes | no | 3.08
| probably no | no | 2.20
| probably yes | yes | 1.76
| no | unknown | 0.88
| yes | probably yes | 0.88
| yes | probably no | 0.88
| no | yes | 0.88
| unknown | yes | 0.88
| probably yes | probably no | 0.88
| probably no | unknown | 0.44
| probably no | probably yes | 0.44
| unknown | probably no | 0.44
| probably yes | unknown | 0.44
| unknown | probably yes | 0.44
External Entities | yes | unknown | 3.52
| probably yes | unknown | 2.64
| probably yes | yes | 2.20
| probably yes | no | 1.32
| no | unknown | 0.88
| unknown | yes | 0.88
| unknown | no | 0.88
| yes | no | 0.88
| no | yes | 0.44
| probably no | no | 0.44
| probably no | yes | 0.44
| yes | probably yes | 0.44
Table 15: Detailed error analysis showing the ground truth and predicted labels of the best model (Part 1). This table complements Table 6.  Error Type | Ground Truth | Predicted | Percentage
---|---|---|---
Social media lexicon | yes | unknown | 5.29
| probably yes | unknown | 3.08
| no | yes | 2.20
| unknown | no | 1.32
| unknown | probably yes | 1.32
| probably no | unknown | 1.32
| probably yes | yes | 1.32
| probably no | no | 0.88
| no | unknown | 0.88
| probably yes | no | 0.88
| probably no | probably yes | 0.44
| yes | probably no | 0.44
| unknown | yes | 0.44
| no | probably no | 0.44
| yes | probably yes | 0.44
Humor | no | unknown | 2.64
| yes | no | 2.20
| yes | unknown | 1.76
| no | yes | 1.32
| probably yes | yes | 0.88
| probably no | no | 0.88
| probably no | unknown | 0.88
| unknown | probably yes | 0.44
| no | probably yes | 0.44
| yes | probably yes | 0.44
| probably no | yes | 0.44
| probably yes | unknown | 0.44
| probably no | probably yes | 0.44
Condition or contrast | no | unknown | 3.52
| probably yes | yes | 3.52
| no | yes | 3.08
| probably no | no | 2.64
| yes | probably yes | 2.64
| yes | unknown | 2.64
| no | probably no | 1.76
| unknown | no | 1.76
| probably yes | unknown | 1.32
| yes | no | 1.32
| unknown | probably yes | 0.88
| probably no | yes | 0.88
| no | probably yes | 0.88
| probably no | probably yes | 0.44
| probably yes | no | 0.44
| unknown | yes | 0.44
| probably no | unknown | 0.44
| probably yes | probably no | 0.44
Unresponsive | unknown | no | 6.16
| unknown | yes | 3.52
| unknown | probably yes | 1.76
Table 16: Detailed error analysis showing the ground truth and predicted
labels of the best model (Part 2). This table complements Table 6.

##  Appendix G Experiments Using Prompts and GPT3

Given the success of of large language models and prompt engineering in many
tasks Brown et al. (2020), we are keen to see whether they are successful at
interpreting answers to yes-no questions from Twitter. We experiment with
various prompt-based approaches and GPT3. Specifically, we experiment with
GPT3’s _text-davinci-002_ engine, _temperature=0.7_ , _frequency_penalty=0_ ,
_presence_penalty=0_ and _top_p=1_. Our prompts do not exactly follow previous
work, but they are inspired by previous work Liu et al. (2021):

Figure 4: Template to generate zero-shot prompts for GPT3 Figure 5: Template
to generate zero-shot prompts with annotation guidelines for GPT3. We refer
the reader to Appendix A for the annotation guidelines for each class.

#### Zero-Shot.

We use prefix prompts Li and Liang (2021); Lester et al. (2021) since such
prompts work well with generative language models such as GPT3 Liu et al.
(2021). As shown by Efrat and Levy, 2020, we also experiment with the same
zero-shot prompt including the annotation guidelines. Specifically, we
experiment with two zero-shot prompts:

  * •

The first zero-shot prompt includes (a) the yes-no question and answer, and
(b) a plain English question asking GPT3 to interpret the answer: _What does
the given answer mean? Choose only from the following options: yes, probably
yes, no, probably no, unknown_. Figure 4 illustrates this prompt.

  * •

The second zero-shot prompt includes the annotation guidelines (Appendix A)
for each label, and then the same content as the first prompt (i.e., yes-no
question and answer followed by the plain English question asking GPT3 to
interpret the answer). Figure 5 illustrates this prompt.

Figure 6: Template to generate few-shot prompts for GPT3 Figure 7: Template to
generate few-shot prompts with annotation guidelines for GPT3. We refer the
reader to Appendix A for the annotation guidelines for each class, and Figure
6 for examples.

#### Few-Shot.

We experiment with longer versions of the zero-shot prompts that include two
examples from the training split per label. Specifically, we add two examples
per label (total: 10 examples) to create few-shot versions of the zero-shot
prompts detailed above. Figures 6 and 7 illustrate the few-shot prompts (with
and without guidelines respectively).

Figure 8: Template to generate the prompt to elicit knowledge (or disambiguation text) regarding the interpretation of an answer with GPT3 (i.e., a justification for the interpretation). The prompt includes an example of disambiguation text (e.g., _the answerer mentions a list of weird things he or she is really good at_). GPT3 always provides answers following the template _The given answer can be interpret as [_label_], because the answerer [_disambiguation text_]_. We use the disambiguation text as an additional input for the RoBERTa-based classifier.  Question |  Answer |  G. |  Disambiguation (from GPT3) |  Valid?
---|---|---|---|---
My name regularly gets autocorrected to "Satanism". Get a surprising number of messages addressed to it. Do you have an autocorrect hell? |  An old phone of mine used to change yorkshire to workshy. outrageous |  y |  Does not mention having an autocorrect hell |  False
Do you still trust anything the media says? |  Modern media can make a collective lie to become the truth |  n |  Mentions how media can manipulate the truth, they do not trust anything the media says |  True
Do you still super follow me? |  I had to chose between spotify and superfollowing |  uk |  Chose spotify over superfollowing the asker |  False
Do you have the courage to be disliked? |  I am not able to answer this question. |  pn |  Admits they cannot answer the question, indicating they do not have the courage to be disliked |  True
Do you bring your phone while using the bathroom? |  Depends how long I think I will be there! |  py |  Implies that they do not bring their phone while using the bathroom |  False
Table 17:  Disambiguation texts produced by GPT3. ‘G’ indicates the gold
label. ‘Valid?’ indicates whether the disambiguation is valid based on the
gold label. In the first example, the answer mentions an “auto correct hell,”
however, the disambiguation disregards this critical term. In the second and
fourth examples, the answer is correctly interpreted by GPT3 by specifying
media manipulation and capturing the fact that not answering the question
shows lack of courage respectively. In the third and fifth examples, GPT3
incorrectly interprets that the author of the answer prefers Spotify and does
not bring a phone in the bathroom respectively. These examples show that GPT3
sometimes produces invalid disambiguation texts. As Table 5 shows, the
disambiguation texts can be both beneficial and detrimental.

#### Prompt-Derived Knowledge.

Liu et al. (2022) have shown that integrating knowledge derived from GPT3 in
question-answering models is beneficial. We follow a similar approach by
prompting GPT3 to generate a _disambiguation text_ given a question-answer
pair from Twitter-YN. Then we complement the input to the RoBERTa-based
classifier with the disambiguation text. Figure 8 illustrated this prompt, and
Table 17 provides examples of disambiguation texts. As the table shows,
certain disambiguation texts are invalid.

###  G.1 Results

| All | yes | no | pyes | pno | unk
---|---|---|---|---|---|---
| Accuracy | P | R | F1 |  | F1 | F1 | F1 | F1 | F1
GPT3, Zero-shot | 0.44 | 0.55 | 0.44 | 0.46 |  | 0.56 | 0.51 | 0.16 | 0.13 | 0.43
\+ annotation guidelines | 0.42 | 0.55 | 0.42 | 0.41 |  | 0.51 | 0.48 | 0.12 | 0.00 | 0.40
GPT3, Few-shot | 0.54 | 0.55 | 0.54 | 0.53 |  | 0.67 | 0.55 | 0.14 | 0.30 | 0.49
\+ annotation guidelines | 0.52 | 0.57 | 0.52 | 0.50 |  | 0.66 | 0.50 | 0.15 | 0.16 | 0.48
RoBERTa blending Twitter-YN (question and answer) and other yes-no questions corpora |
Circa (Table 5) | 0.60 | 0.59 | 0.60 | 0.58 | † | 0.72 | 0.65 | 0.37 | 0.25 | 0.44
\+ disambiguation text (GPT3) | 0.62 | 0.62 | 0.62 | 0.61 | † | 0.75 | 0.67 | 0.35 | 0.22 | 0.57
Friends-QIA (Table 5) | 0.60 | 0.59 | 0.60 | 0.58 | † | 0.71 | 0.63 | 0.38 | 0.20 | 0.45
\+ disambiguation text (GPT3) | 0.56 | 0.56 | 0.56 | 0.56 |  | 0.70 | 0.63 | 0.36 | 0.24 | 0.37
Table 18: Results obtained with GPT3 in the _unmatched question and unmatched
temporal span_ scenario. These results are moderately worse than the ones
obtained with RoBERTa-based classifiers (Table 5), although the zero-shot
prompts obtain results better than the baselines without requiring any
annotations. Statistical significance with respect to few-shot GPT3 (first
block) is indicated with an dagger (†) (McNemar’s Test, $p<0.05$).

Table 18 presents the results using GPT3. The best RoBERTa-based system
obtains an overall F1 of 0.58 (Table 5). All the systems using GPT3 obtain F1s
under 0.58. Including the disambiguation text to RoBERTa is detrimental in
case of Friends-QIA (F1: 0.56) but beneficial in case of Circa (F1: 0.61).
Unsurprisingly, few-shot approaches obtain better results than zero-shot
prompts (0.53, 0.50 vs. 0.46, 0.41). Perhaps surprisingly, including the
annotation guidelines is detrimental (zero-shot: 0.46 vs. 0.41, few-shot: 0.53
vs. 0.50).

We acknowledge that prompt engineering may lead to better results with GPT3.
We refer the reader to the Limitations section for a discussion.