# A fine-grained comparison of pragmatic language understanding in humans and
language models

Jennifer Hu1, Sammy Floyd1, Olessia Jouravlev2, Evelina Fedorenko1,3, Edward
Gibson1
1Department of Brain and Cognitive Sciences, Massachusetts Institute of
Technology
2Department of Cognitive Science, Carleton University
3McGovern Institute for Brain Research, Massachusetts Institute of Technology

###### Abstract

Pragmatics is an essential part of communication, but it remains unclear what
mechanisms underlie human pragmatic communication and whether NLP systems
capture pragmatic language understanding. To investigate both these questions,
we perform a fine-grained comparison of language models and humans on seven
pragmatic phenomena, using zero-shot prompting on an expert-curated set of
English materials. We ask whether models (1) select pragmatic interpretations
of speaker utterances, (2) make similar error patterns as humans, and (3) use
similar linguistic cues as humans to solve the tasks. We find that the largest
models achieve high accuracy and match human error patterns: within incorrect
responses, models favor the literal interpretation of an utterance over
heuristic-based distractors. We also find evidence that models and humans are
sensitive to similar linguistic cues. Our results suggest that even
paradigmatic pragmatic phenomena may be solved without explicit
representations of other agents’ mental states, and that artificial models can
be used to gain mechanistic insights into human pragmatic processing.Code and
data: https://github.com/jennhu/lm-pragmatics

##  1 Introduction

Non-literal language understanding is an essential part of communication. For
example, in everyday conversations, humans readily comprehend the non-literal
meanings of metaphors (_My new coworker is a block of ice_), polite deceits
(_I love the gift_), indirect requests (_It’s a bit cold in this room_), and
irony (_Classy pajamas, dude!_). These phenomena fall under the broad label of
pragmatics, which encompasses the aspects of meaning that go beyond the
literal semantics of what is said (Horn, 1972; Grice, 1975; Yule, 1996;
Levinson, 2000).

The breadth of pragmatics raises challenges from both engineering and
scientific perspectives. From an NLP perspective, a major question is whether
artificial language models capture pragmatic meaning. This has been
investigated from a philosophical view – given that language models are only
exposed to linguistic forms in static documents (Bisk et al., 2020; Bender and
Koller, 2020; Sap et al., 2022) – as well as through empirical evaluations on
non-literal language understanding benchmarks (Jeretic et al., 2020; Zheng et
al., 2021; Tong et al., 2021; Liu et al., 2022; Ruis et al., 2022; Stowe et
al., 2022). However, these empirical studies are primarily based on
categorical distinctions between pragmatic/non-pragmatic responses, which do
not reveal the _strategies_ that models use.

From a cognitive perspective, the mechanisms underlying pragmatic
communication remain debated. In a human behavioral study, Floyd et al. (In
prep) find that while certain abilities such as irony comprehension,
conversational implicatures, and indirect requests can be explained by a
common latent variable, these tasks dissociate from other abilities such as
intonation comprehension or world knowledge-based inferences. This suggests
that there may be no single “pragmatics” mechanism, but instead that these
tasks draw upon different types of knowledge and mental processes. Indeed,
pragmatic language comprehension has been hypothesized to involve a variety of
mechanisms, such as basic language processing, knowledge of cultural and
social norms (Trosborg, 2010), and reasoning about speakers’ mental states
(e.g., Frank and Goodman, 2012; Enrici et al., 2019; Rubio-Fernandez, 2021).
It remains an open question _how_ humans perform different pragmatic tasks,
and the contributions of mentalizing versus linguistic patterns (Andrés-
Roqueta and Katsos, 2017).

In this paper, we perform a fine-grained comparison of language models (LMs)
and humans on pragmatic language understanding tasks. Similar to the approach
of targeted syntactic evaluation (e.g., Linzen et al., 2016; Futrell et al.,
2019; Hu et al., 2020), our analysis serves two goals: assessing the pragmatic
capabilities of modern LMs, and testing what aspects of pragmatics may be
“solved” using only linguistic forms without explicit symbolic reasoning or
representations of other agents’ mental states. We use a set of English-
language test items curated by expert researchers (Floyd et al., In prep),
covering seven pragmatic phenomena. Through model analyses and human
experiments, we investigate the following questions: (1) Do models recover the
hypothesized pragmatic interpretation of speaker utterances? (2) When models
do not select the target response, what errors do they make – and how do these
error patterns compare to those of humans? (3) Do models and humans use
similar cues to arrive at pragmatic interpretations?

We use zero-shot prompting to evaluate models of multiple sizes with various
training objectives: GPT-2 (Radford et al., 2019), T$k$-Instruct (Wang et al.,
2022), Flan-T5 (Chung et al., 2022), and InstructGPT (Ouyang et al., 2022). We
find that Flan-T5 (XL) and OpenAI’s text-davinci-002 achieve high accuracy and
mirror the distribution of responses selected by humans. When these models are
incorrect, they tend to select the incorrect literal (or straightforward)
answer instead of distractors based on low-level heuristics. We also find
suggestive evidence that models and humans are sensitive to similar linguistic
cues. Our results suggest that even paradigmatic pragmatic phenomena (such as
polite deceits) may be solved without explicit representations of other
agents’ mental states, and that artificial models can be used to gain
mechanistic insights into human pragmatic processing.

##  2 Related work

Prior work has evaluated LMs’ ability to recognize non-literal interpretations
of linguistic input, such as scalar implicature (Jeretic et al., 2020;
Schuster et al., 2020; Li et al., 2021) or figurative language (Tong et al.,
2021; Liu et al., 2022; Gu et al., 2022; Stowe et al., 2022). In a broad-scale
evaluation, Zheng et al. (2021) test five types of implicatures arising from
Grice’s (1975) conversational maxims, and evaluate their models after training
on the task. In our work, we consider Gricean implicatures as one of seven
phenomena, and we evaluate pre-trained LMs without fine-tuning on our tasks.

Similar to our work, Ruis et al. (2022) also use prompting to evaluate LMs on
pragmatic interpretation tasks. They formulate implicature tests as sentences
ending with “yes” or “no” (e.g., “Esther asked “Can you come to my party on
Friday?” and Juan responded “I have to work”, which means no.”). A model is
considered pragmatic if it assigns higher probability to the token that makes
the sentence consistent with an implicature. In our work, models must select
from multiple interpretations that reflect different strategies, enabling a
detailed error analysis and comparison to humans.

Since pragmatic language understanding often draws upon knowledge of social
relations, our tasks are conceptually related to benchmarks for evaluating
social commonsense (e.g., Sap et al., 2019; Zadeh et al., 2019). These
evaluations focus on the interpretation of actions and events, whereas we
focus on the interpretation of speaker utterances. Another hypothesized
component of pragmatics is Theory of Mind (Leslie et al., 2004; Apperly,
2011), or the ability to reason about others’ mental states. Benchmarks for
evaluating Theory of Mind in models (e.g., Nematzadeh et al., 2018; Le et al.,
2019; Sap et al., 2022) have primarily focused on false-belief tasks (Baron-
Cohen et al., 1985).

##  3 Tasks & materials

Our evaluation materials are taken from Floyd et al.’s (In prep)
experiments,111Materials can be found at
https://osf.io/6abgk/?view_only=42d448e3d0b14ecf8b87908b3a618672. covering
seven pragmatic phenomena: deceits, indirect speech, irony, Gricean maxims,
metaphor, humor, and coherence inferences. Each item is a multiple choice
question, with answer options representing different strategies that a
listener might use to solve the task. For most of the tasks, the query has
three parts: a short story context (1-3 sentences), an utterance spoken by one
of the characters, and a question about what the character intended to
convey.222The exceptions are Humor and Coherence. Each task has 20-40 items,
which were manually curated by expert researchers. The items were constructed
to elicit individual differences among humans, so some are more difficult by
design. As such, these evaluation materials may not be suitable as a
traditional NLP benchmark, but provide a resource for tightly controlled,
fine-grained evaluation of LMs.

Task |  Example query |  Example answer options
---|---|---
Deceits |  Henry is sitting at his desk and watching TV, and reluctantly switches off the TV with the remote control and picks up a textbook. Shortly after, his mother comes in the room and asks, "What have you been doing up here?" Henry responds: "Reading." Why has Henry responded in such a way? |  1. CorrectNonLiteral He does not want to get into trouble for not studying. 2. IncorrectLiteral He has been reading for some time 3. IncorrectLexicalOverlap He does not want to offend his mom by not reading the books that she gave him 4. IncorrectSocialConvention He wants his mom to believe that he has been watching TV
Indirect speech |  Nate is about to leave the house. His wife points at a full bag of garbage and asks: "Are you going out?" What might she be trying to convey? |  1. CorrectNonLiteral She wants Nate to take the garbage out. 2. IncorrectLiteral She wants to know Nate’s plans. 3. IncorrectAssociate She wants Nate to bring his friends over. 4. IncorrectLexicalOverlap She wants Nate to spend more time with the family.
Irony |  It is a holiday. Stefan and Kim are sitting in the backseat of the car. They are fighting all the time. Their father says: "Oh, it is so pleasant here." What did the father want to convey? |  1. CorrectNonLiteral He does not want to listen to his kids’ arguments. 2. IncorrectLiteral He enjoys listening to his kids fighting. 3. IncorrectAssociate AC gives them some needed cool. 4. IncorrectNonSequitur He remembers about his wife’s birthday.
Maxims |  Leslie and Jane are chatting at a coffee shop. Leslie asks, "Who was that man that I saw you with last night?" Jane responds, "The latte is unbelievable here." Why has Jane responded like this? |  1. CorrectNonLiteral She does not want to discuss the topic that Leslie has raised. 2. IncorrectLiteral She thinks that it is the best latte in the town. 3. IncorrectAssociate The man who Leslie saw makes unbelievable lattes. 4. IncorrectNonLiteral A coffee break is not a good time to discuss men.
Metaphor |  Andrew and Bob were discussing the investment company where Andrew works. Bob said: “The investors are squirrels collecting nuts.” What does Bob mean? |  1. CorrectNonLiteral They buy stocks hoping for future profit. 2. IncorrectLiteral Squirrels were hired to work in the company. 3. IncorrectNonLiteral The investors dress and eat well. 4. IncorrectNonsequitur Bob is allergic to nuts. 5. IncorrectPlausibleLiteral The investors enjoy picking nuts as much as squirrels do.
Humor |  Martha walked into a pastry shop. After surveying all the pastries, she decided on a chocolate pie. "I’ll take that one," Martha said to the attendant, "the whole thing." "Shall I cut it into four or eight pieces?" the attendant asked. |  1. CorrectNonLiteral Martha said, "Four pieces, please; I’m on a diet." 2. IncorrectStraightforward Martha said: "Well, there are five people for dessert tonight, so eight pieces will be about right." 3. AssociativeNonSequitur Martha said, "You make the most delicious sweet rolls in town." 4. FunnyNonSequitur Then the attendant squirted whipped cream in Martha’s face. 5. NeutralNonSequitur Martha said, "My leg is hurting so much."
Coherence |  Cleo brushed against a table with a vase on it. She decided to study harder to catch up. |  1. Correct Incoherent 2. Incorrect Coherent
Table 1: Sample item from each task in our evaluation. All items are
originally curated by Floyd et al. (In prep).

Table 1 shows an example item for each task, with annotated answer options.
Green labels indicate the target pragmatic non-literal interpretation.  Blue
labels indicate the literal interpretation.  Red labels indicate incorrect
non-literal interpretations, which are based on heuristics such as lexical
similarity to the story. We summarize the tasks below, and refer readers to
Floyd et al. (In prep) for details.

##### Deceits.

Humans produce polite deceits (“white lies”) in many social situations (e.g.,
Camden et al., 1984). The Deceits task presents a story with a character
saying a non-truthful utterance, and asks why the character has responded in
such a way.

The underlying intentions behind polite deceits are rarely explicitly
explained in text. As a result, it is unlikely that LMs learn a direct
connection between the utterance and the speaker’s intention during training
on static texts. However, instances of polite deceits in text corpora may be
accompanied by descriptions of characters’ emotional states, which may
indicate that speakers’ intentions differ from what is literally conveyed by
their utterance. This highlights the importance of context in interpreting
deceits, which we return to in Section 5.3.2.

##### Indirect speech.

Humans often use language in a performative sense, such as indirectly
requesting an action from other individuals (e.g., Austin, 1975; Searle,
1975). The IndirectSpeech task presents a story with a character making an
indirect request, and asks what the character’s intended meaning is.

Like deceits, it’s unlikely that indirect speech acts are explicitly explained
in text data. However, indirect requests may be followed by a description of
the completion of the implied request – for example, that someone closed a
window after hearing the utterance “It’s cold in here”. Therefore, it’s
plausible that models may be able to learn relationships between the
utterances and desired outcomes, purely based on exposure to linguistic forms.

##### Irony.

Humans use irony to convey the opposite of the semantic content of their
utterance (Booth, 1974; Wilson and Sperber, 1992; Attardo, 2000; Wilson and
Sperber, 2012). As such, irony has been considered a pragmatic task that
relies heavily on social reasoning and perspective-taking (Andrés-Roqueta and
Katsos, 2017). The Irony task presents a story with a character saying an
ironic utterance, and asks what the character intends to convey.

While ironic statements are also rarely explained in text, models could
leverage accompanying cues such as descriptions of characters’ emotional
states or a mismatch in sentiment. We return to the importance of such
contextual cues in Section 5.3.2.

##### Maxims of conversation.

Grice (1975) proposes that interlocutors abide by a set of _maxims_ : be
truthful; be relevant; be clear, brief, and orderly; and say as much as
needed, and no more. These maxims are so ingrained in everyday conversations,
Grice argues, that failing to fulfill one of them gives rise to a
conversational implicature. The Maxims task presents a story with a character
flouting one of Grice’s maxims, and asks why the character has responded in
such a way.

Based on linguistic input, it may be easy for LMs to recognize when a speaker
is flouting a maxim – for example, if an utterance is particularly long,
features an uncommon syntactic construction, or diverges semantically from the
context. However, it is unclear whether LMs will be able to recover the
speaker’s underlying intentions.

##### Metaphor.

Metaphors (Lakoff and Johnson, 1980) are used to draw comparisons between
entities in a non-literal sense. The Metaphor task presents a story with a
character using a metaphor, and asks what the intended interpretation is.

The challenges of metaphor comprehension include drawing upon world knowledge
and forming abstract relationships between domains. On the other hand, models
may also be relatively well-suited for metaphor comprehension, as the relevant
properties of the entities in the metaphor could emerge from exposure to
linguistic forms.

##### Humor.

Humor is one of the most distinctive – and mysterious – aspects of human
conversation (Veatch, 1998; Martin and Ford, 2018). The Humor task presents a
joke setup, and asks which punchline makes the joke the funniest. This task
differs from the others in that there is no speaker utterance.

Some theories argue that humor is triggered by linguistic incongruency effects
(e.g., Deckers and Kizer, 1975), which might be easy for LMs to detect. Recent
work has also shown that LMs can explain certain jokes (Chowdhery et al.,
2022). However, some of Floyd et al.’s Humor items require complex world
knowledge – for example, that slicing a pie into four versus eight pieces does
not change the total amount (see Table 1). As such, selecting the funniest
punchline is a nontrivial task.

##### Coherence inferences.

Humans also make pragmatic inferences beyond the sentence level – for example,
by assuming that consecutive sentences form a logical or sequential
relationship. The Coherence task presents a pair of sentences, and asks
whether the pair forms a coherent story. This task differs from the others in
that there is no speaker utterance, and the answer options are identical
across items (“Coherent” or “Incoherent”).

We assume that LMs’ training data, which consists of naturalistic text, is
primarily coherent. Therefore, we expect LMs to be able to distinguish between
coherent and incoherent sentence pairs (for an in-depth study, see Beyer et
al., 2021).

##  4 Experiments

###  4.1 Evaluation paradigm

Having described the tasks and materials, we now turn to the design of our
model evaluation. We use _zero-shot prompting_ , as it can be adapted to all
of our seven tasks. This is crucial because we would like to compare
performance across tasks _within_ individual models. Furthermore, this allows
us to present models with inputs that are nearly identical to the stimuli seen
by humans in Floyd et al.’s experiments, whereas other methods would require
converting the original multiple-choice questions into model-compatible
formats. For example, prior work has evaluated models’ understanding of
conversational maxims using question-answering (Zheng et al., 2021), metaphors
using a Winograd schema (Liu et al., 2022), and coherence inferences using
probability comparisons (Beyer et al., 2021).

##### Prompt structure.

Each prompt consisted of two parts: task instructions, and a query. The
instructions were nearly identical to the instructions presented to humans in
Floyd et al.’s experiments, prepended with the keyword “Task:”. The only other
modification was that the original instructions had a final sentence of
“Please answer as quickly as possible”, which we replaced with a sentence like
“The answer options are 1, 2, 3, or 4”.333The exact answer options changed
according to the task.

For all tasks except Humor, the query consisted of the scenario (prepended
with keyword “Scenario:”) and question, and then the numbered answer options
(prepended with “Options:”).444For the Humor task, the joke was prepended with
“Joke:”, and the answer options were prepended with “Punchlines:”. The prompt
concluded with the keyword “Answer:”. Full example prompts are given in
Appendix A.

##### Evaluation.

To evaluate a model on a given item, we feed the prompt to the model, and
measure the model’s probability distribution over tokens conditioned on the
prompt. We compare the probabilities of each answer token (e.g., “1”, “2”,
“3”, or “4”) under this distribution. The model is considered correct on a
given item if it assigns highest probability to the correct answer token,
among all the possible answer tokens for that item. We do not evaluate any
generated text from the model.

We generated 5 versions of each item by randomizing the order of answer
options. This was done to control for the base probabilities of the answer
tokens “1”, “2”, “3”, “4”, and “5”. Since we do not analyze model generations,
the model results (i.e., answer token rankings) are deterministic.

Model | # parameters | Training
---|---|---
GPT-2 | 117M | Autoregressive LM
Tk-Instruct (3B) | 3B | Multitask
Tk-Instruct (11B) | 11B | Multitask
Flan-T5 (base) | 250M | Multitask
Flan-T5 (XL) | 3B | Multitask
InstructGPT-3 (ada) | 350M (est.) | Multitask, human feedback
text-davinci-002 | Unknown | FeedME
Table 2: Models tested in our experiments.

###  4.2 Models

We test seven models across four model families, summarized in Table 2.555All
models except the OpenAI models were accessed via Huggingface (Wolf et al.,
2020). As a baseline, we first test a base GPT-2 model (117M parameters;
Radford et al., 2019), which is trained on an autoregressive language modeling
objective. Second, we test a set of models which are based on T5 (Raffel et
al., 2020) and instruction-finetuned on a diverse collection of tasks (Wei et
al., 2022). This set of models consists of two T$k$-Instruct models (3B and
11B; Wang et al., 2022) and two Flan-T5 models base (base: 250M parameters;
XL: 3B parameters; Chung et al., 2022). Finally, we test two InstructGPT-based
models (Ouyang et al., 2022) via the OpenAI API: text-ada-001 (350M
parameters), which we refer to as InstructGPT-3 (ada); and text-davinci-002,
which comes from the GPT-3.5 family of models.666Parameter count estimates
come from https://blog.eleuther.ai/gpt3-model-sizes/. Although the exact size
of text-davinci-002 is unknown, we assume that it is larger than InstructGPT-3
(ada).,777The OpenAI models are not fully reproducible, but timestamps of
model runs can be found in Appendix B. These models are fine-tuned to follow
instructions and align with human feedback.888Fore more details, see
https://beta.openai.com/docs/model-index-for-researchers.

We compare models to a baseline from 374 humans, collected by Floyd et al. (In
prep). Their experiments presented multiple choice questions to humans in
nearly identical format to our prompts.

Figure 1: Accuracy for each task. Error bars denote 95% CI. Dashed line
indicates task-specific random baseline. Figure 2: Mean accuracy vs. millions
of parameters. Vertical dashed line indicates 1 billion parameters.

##  5 Results

We now return to the three questions posed in the Introduction, in each of the
following subsections.

Figure 3: Response distributions across models and humans. For models, y-axis
denotes probability assigned to each answer option. For humans, y-axis denotes
empirical frequency of each answer option being selected. Ground-truth answer
is marked in boldface. Error bars denote 95% CI. Dashed line indicates random
baseline.

###  5.1 Do models choose the target pragmatic interpretation?

Figure 1 shows the proportion of items where models and humans select the
target pragmatic answer. We find that the smallest models (GPT-2, Flan-T5
(base), InstructGPT-3 (ada)) fail to perform above chance. The largest models
(T$k$-Instruct (11B), Flan-T5 (XL), text-davinci-002) perform significantly
above chance on all tasks (with the exception of T$k$-Instruct (11B) on
Maxims), and in some cases near human-level. These successes suggest that
linguistic forms may provide sufficient signal for solving certain pragmatic
tasks, without explicitly representing other agents’ mental states.

Overall, models perform worst at the Humor, Irony, and Maxims tasks.
Interestingly, these phenomena involve speakers violating listeners’
expectations in some way: producing a funny punchline to a mundane story
(Humor), stating the direct opposite of the speaker’s belief (Irony), or
disobeying one of the assumed principles of conversation (Maxims). It may be
the case that models fail to represent certain social-convention-based
expectations that are maintained by human listeners.

Next, we investigated the relationship between model size and accuracy. Figure
2 shows the mean accuracy achieved by each model (averaged across tasks) vs.
millions of parameters.999text-davinci-002 was excluded from this analysis, as
the number of parameters is unknown. The line and error bars denote the mean
and 95% CIs, while points represent individual models. We find a coarse effect
of model size: there is a stark jump in accuracy after 1B parameters (dashed
line). However, model size does not fully explain variance in accuracy: all
models with $<$1B parameters achieve similar accuracy, and Flan-T5 (XL)
outperforms T$k$-Instruct (3B), despite both having 3B parameters.

###  5.2 Do models and humans make similar types of errors?

Recall from Section 3 that each item has a set of answer options that
correspond to strategies a listener might use to solve the task (see Table
1).101010The exception is Coherence, which is excluded here. In addition to
the target non-literal interpretation (CorrectNonLiteral), each item also has
a plausible but unlikely literal interpretation
(IncorrectLiteral/IncorrectStraightforward), as well as distractor answer
options that correspond to non-literal interpretations based on lexical
overlap or semantic associations. For each item, we computed a human
distribution over the answer choices by taking the empirical frequencies.
Then, we took the model’s distribution over the tokens corresponding to those
answers (e.g., tokens “1”, “2”, “3”, and “4”).

Figure 3 shows the answer distributions across models and humans for each
task. First, we note that humans (brown bars) primarily select the
CorrectNonLiteral option, occasionally select the IncorrectLiteral or
IncorrectStraightforward option, and rarely select the heuristic-based
options. We find a similar qualitative pattern for text-davinci-002 (dark blue
bars), although the model is more likely to select the IncorrectLiteral option
in general. The other large models (T$k$-Instruct (11B) and Flan-T5 (XL)) also
generally assign highest probability to the CorrectNonLiteral or
IncorrectLiteral options, although the shape of the distribution looks less
human-like. The next-largest models (T$k$-Instruct (3B) and Flan-T5 (base))
prefer the IncorrectLiteral options, and the remaining models (GPT-2 and
InstructGPT-3 (ada)) are at chance.

These results suggest that, for our test items, models satisfy the basic
language processing component of pragmatic comprehension: when models fail,
they overwhelmingly select the literal interpretation instead of non-sensical
but semantically related distractors. However, even highly performing models
occasionally do select the distractor answers. This highlights the importance
of carefully constructed distractor items when evaluating models’ language
abilities (e.g., McCoy et al., 2019).

###  5.3 Are models and humans sensitive to similar linguistic cues?

Finally, we investigate whether models and humans use similar cues to perform
the tasks. We first ask whether model and human performance is correlated at
the item level – i.e., if models and humans exhibit similar sensitivity to the
properties that make certain items more difficult than others (Section 5.3.1).
Next, we ask to what extent models and humans depend on the linguistic context
to select the pragmatic interpretation (Section 5.3.2).

####  5.3.1 Do models and humans find the same items difficult?

Recall that the items were designed to vary in difficulty for humans. We
leverage this variability to investigate whether models and humans find the
same items difficult. For each task and model, we compute the Pearson
correlation between by-item mean accuracy achieved by humans and by-item mean
probability that models assigned to the correct answer. A strong positive
correlation would suggest that models and humans are sensitive to similar cues
(e.g., specific phrases, or frequency effects) that modulate the difficulty of
an item.

Figure 4: Correlation between by-item human accuracy and model probability
assigned to the pragmatic answer. Cells are marked with significance codes.

Figure 4 shows the Pearson correlation coefficient (darker $=$ higher) between
by-item human accuracy scores and model probabilities assigned to the correct
answer. Columns in the heatmap correspond to tasks, and rows correspond to
models. In general, the larger models (T$k$-Instruct (11B), Flan-T5 (XL),
text-davinci-002) are better aligned with humans (i.e., achieve stronger
correlations). Furthermore, when there is a significant correlation, it occurs
for the IndirectSpeech, Irony, Maxims, and Metaphor tasks (with one exception:
T$k$-Instruct (11B) and Coherence). This suggests that for those tasks, models
and humans are similarly sensitive to cues that make items more or less
challenging.

####  5.3.2 What is the role of context?

Next, we investigate the role of context for performing each task. We do this
by removing the context story from the prompt, leaving only the speaker
utterance and final query (e.g., _Dan says, “The dog knocked it over.” Why has
Dan responded in such a way?_).111111This manipulation is not compatible with
the Humor and Coherence tasks, so they are excluded from this analysis. We
then measure the difference in performance on these no-context materials
versus the original materials. This analysis could reveal which pragmatic
tasks are “solveable” without context, and which tasks require specific
descriptions of characters’ mental states, actions, and desires.

To measure human performance on the no-context modified materials, we re-ran
the human experiment on 30 participants, following the protocols of Floyd et
al. (In prep)’s original experiment.121212Details can be found in Appendix C.
For model analysis, we only re-ran the best-performing models: T$k$-Instruct
(11B), Flan-T5 (XL), and text-davinci-002. We did not consider the models that
performed poorly on the original items, as the performance gap between the
original and no-context items would be less interpretable.

Figure 5 shows the mean accuracy difference on the original versus no-story
versions of each item. We find that models and humans exhibit a similar
qualitative pattern: removing the story leads to the largest degradation for
Irony, followed by Deceits and Maxims. This aligns with our intuitions,
because in these cases, speakers’ utterances can be interpreted either
literally or as the complete opposite, based on the specific social situation
(e.g., “It is so pleasant here”). In contrast, there are smaller degradations
for IndirectSpeech and Metaphor. This suggests that some indirect requests are
conventionalized (e.g., “I am getting cold”), although their interpretations
may be facilitated by context (e.g., Gibbs, 1979). Similarly, this suggests
that metaphors are somewhat self-contained – their interpretations may draw
more upon global world knowledge than local context.

Figure 5: Mean by-item difference in accuracy once story context was removed.

##  6 Discussion

We used an expert-curated set of materials (Floyd et al., In prep) to compare
language models to humans on seven pragmatic phenomena. We found that Flan-T5
(XL) and text-davinci-002 achieve high accuracy and match human error
patterns: within incorrect responses, these models tend to select the literal
interpretation of an utterance over distractors based on low-level heuristics.
We also found suggestive evidence that models and humans are sensitive to
similar linguistic cues. For the IndirectSpeech, Irony, Maxims, and Metaphor
tasks, humans and models tend to find similar items difficult. Furthermore, we
conducted a behavioral experiment using a modified set of items without
linguistic context, and found that models and humans degraded across tasks in
similar ways.

Our results suggest that models only exposed to linguistic forms can
consistently select the pragmatic interpretation of a speaker’s utterance over
literal or semantically similar interpretations. This provides an existence
proof that even paradigmatic pragmatic phenomena – such as polite deceits –
could potentially be solved without explicit representations of other agents’
mental states. Andreas (2022) offers a potential explanation: whereas language
models have no explicit representation of communicative intents, they can
infer approximate representations of the mental states of the agents that
produce a given linguistic context.

Importantly, we do not claim that models are “understanding” language or
“performing pragmatic reasoning” in a strong sense (see, e.g., Bender and
Koller, 2020). Rather, we take model successes to suggest that explicit
mentalizing is not _necessary_ to mimic pragmatic behaviors – instead,
experience with linguistic forms may be sufficient for deriving many human-
like behavioral patterns. More broadly, our analyses contribute to a growing
body of research using large language models to test cognitive hypotheses
(e.g., Wilcox et al., 2022a, b; Baroni, 2022; Warstadt and Bowman, 2022; Shain
et al., 2022), enabling a new generation of theories of human language
acquisition and processing.

## Limitations

We note several methodological limitations with our experiments. First, since
the evaluation materials were manually crafted, there is a rather small number
of items (compared to the size of automatically generated NLP benchmarks).
Small evaluation sets can introduce issues of statistical power (Card et al.,
2020) and introduce bias based on lexical items. We feel this is not a major
concern, because (1) our materials are validated by expert researchers; (2)
models can be directly compared to humans in Floyd et al.’s experiments; and
(3) in practice, there is enough signal to distinguish between the tested
models.

Second, we only evaluate models on English-language items, and some of the
tasks were designed based on norms of communication and social interaction in
Western cultures. As pragmatics can vary widely across language and cultures
(Li, 2012; Rubio-Fernandez and Jara-Ettinger, 2020; Floyd, 2021; Brown et al.,
2021; Dideriksen et al., 2022), an important direction for future work is to
evaluate pragmatics beyond English (Ameka and Terkourafi, 2019; Blasi et al.,
2022).

Third, aside from the OpenAI API models, we were only able to test models with
$\leq$11B parameters due to limited computational resources. It could be the
case that models with parameter sizes between 11B and the size of text-
davinci-002 could exhibit qualitatively different behaviors.

Finally, we emphasize that it is impossible to predict how models will respond
to an arbitrary input. Therefore, we caution against extrapolating from our
results and expecting that models will behave “pragmatically” in downstream
applications. This is especially true for models behind the OpenAI API, and
text-davinci-002 in particular, for which very little is publicly known about
the training protocol.

## Ethics statement

Language technologies have the potential to cause harm at the individual and
societal levels. Large language models (LLMs), which are typically trained on
vast amounts of internet text, have been shown to perpetuate stereotypes based
on gender, race, and sexual orientation. Applications using LLMs could
reinforce systematic discrimination and amplify existing socioeconomic
inequities. For example, LLMs could perpetuate social biases by assisting with
hiring decisions or legal rulings.

The remarkable fluency of LLM-generated text also poses risks for the general
public. LLMs have long been used to generate text that is difficult to
distinguish from human-written text, raising concerns about detecting fake
news and misinformation. Recently, LLMs have been used to synthesize knowledge
– for example, by answering scientific questions (Taylor et al., 2022) or
acting as search engines (Shah and Bender, 2022). Using LLMs as knowledge-
providers could tremendously impact the nature of human collaboration and
work, raising the need for model transparency and explainability.

## Acknowledgements

We thank Quinn Langford for help with coding details of the stimuli. J.H. was
supported by an NSF Graduate Research Fellowship (#1745302) and NSF Doctoral
Dissertation Research Improvement grant (BCS-2116918). S.F. is funded by the
NSF SPRF (#2105136). This work was in part supported by a grant from the
Simons Foundation to the Simons Center for the Social Brain at MIT. E.F. was
additionally supported by NIH award R01-DC016950 and by research funds from
the McGovern Institute for Brain Research and the Department of Brain and
Cognitive Sciences.

## References

  * Ameka and Terkourafi (2019) Felix K. Ameka and Marina Terkourafi. 2019.  What if…? Imagining non-Western perspectives on pragmatic theory and practice.  _Journal of Pragmatics_ , 145:72–82.
  * Andreas (2022) Jacob Andreas. 2022.  Language Models as Agent Models.
  * Andrés-Roqueta and Katsos (2017) Clara Andrés-Roqueta and Napoleon Katsos. 2017.  The Contribution of Grammar, Vocabulary and Theory of Mind in Pragmatic Language Competence in Children with Autistic Spectrum Disorders.  _Frontiers in Psychology_ , 8.
  * Apperly (2011) Ian Apperly. 2011.  _Mindreaders: The cognitive basis of "Theory of Mind"_.  Psychology Press, New York.
  * Attardo (2000) Salvatore Attardo. 2000.  Irony as relevant inappropriateness.  _Journal of Pragmatics_ , 32(6):793–826.
  * Austin (1975) John L. Austin. 1975.  _How to do things with words_.
  * Baron-Cohen et al. (1985) Simon Baron-Cohen, Alan M. Leslie, and Uta Frith. 1985.  Does the autistic child have a “theory of mind”? _Cognition_ , 21(1):37–46.
  * Baroni (2022) Marco Baroni. 2022.  On the proper role of linguistically-oriented deep net analysis in linguistic theorizing.  In Shalom Lappin and Jean-Philippe Bernardy, editors, _Algebraic Systems and the Representation of Linguistic Knowledge_. Taylor & Francis.  To appear.
  * Bender and Koller (2020) Emily M. Bender and Alexander Koller. 2020.  Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5185–5198, Online. Association for Computational Linguistics.
  * Beyer et al. (2021) Anne Beyer, Sharid Loáiciga, and David Schlangen. 2021.  Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4164–4173, Online. Association for Computational Linguistics.
  * Bisk et al. (2020) Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020.  Experience Grounds Language.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8718–8735, Online. Association for Computational Linguistics.
  * Blasi et al. (2022) Damián E. Blasi, Joseph Henrich, Evangelia Adamou, David Kemmerer, and Asifa Majid. 2022.  Over-reliance on English hinders cognitive science.  _Trends in Cognitive Sciences_ , 26(12):1153–1170.  Publisher: Elsevier.
  * Booth (1974) W.C. Booth. 1974.  _A Rhetoric of Irony_.  Literature/Criticism - The University of Chicago Press. University of Chicago Press.
  * Brown et al. (2021) Penelope Brown, Mark A. Sicoli, and Olivier Le Guen. 2021.  Cross-speaker repetition and epistemic stance in Tzeltal, Yucatec, and Zapotec conversations.  _Journal of Pragmatics_ , 183:256–272.
  * Camden et al. (1984) Carl Camden, Michael T. Motley, and Ann Wilson. 1984.  White lies in interpersonal communication: A taxonomy and preliminary investigation of social motivations.  _Western Journal of Speech Communication_ , 48(4):309–325.  Publisher: Routledge.
  * Card et al. (2020) Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020.  With Little Power Comes Great Responsibility.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9263–9274, Online. Association for Computational Linguistics.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022\.  PaLM: Scaling Language Modeling with Pathways.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling Instruction-Finetuned Language Models.
  * Deckers and Kizer (1975) Lambert Deckers and Philip Kizer. 1975.  Humor and the Incongruity Hypothesis.  _The Journal of Psychology_ , 90(2):215–218.
  * Dideriksen et al. (2022) Christina Dideriksen, Morten H Christiansen, Mark Dingemanse, Malte Højmark-Bertelsen, Christer Johansson, Kristian Tylén, and Riccardo Fusaroli. 2022.  Language specific constraints on conversation: Evidence from Danish and Norwegian.
  * Enrici et al. (2019) Ivan Enrici, Bruno G. Bara, and Mauro Adenzato. 2019.  Theory of Mind, pragmatics and the brain: Converging evidence for the role of intention processing as a core feature of human communication.  _Pragmatics & Cognition_, 26(1):5–38.
  * Floyd et al. (In prep) Sammy Floyd, Olessia Jouravlev, Zachary Mineroff, Leon Bergen, Evelina Fedorenko, and Edward Gibson. In prep.  Deciphering the structure of pragmatics: A large-scale individual differences investigation.
  * Floyd (2021) Simeon Floyd. 2021.  Conversation and Culture.  _Annual Review of Anthropology_ , 50(1):219–240.  Publisher: Annual Reviews.
  * Frank and Goodman (2012) Michael C. Frank and Noah D. Goodman. 2012.  Predicting Pragmatic Reasoning in Language Games.  _Science_ , 336(6084):998–998.
  * Futrell et al. (2019) Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019.  Neural language models as psycholinguistic subjects: Representations of syntactic state.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 32–42, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gibbs (1979) Raymond W. Gibbs. 1979.  Contextual effects in understanding indirect requests.  _Discourse Processes_ , 2(1):1–10.  Publisher: Routledge.
  * Grice (1975) Herbert P. Grice. 1975.  Logic and Conversation.  In Peter Cole and Jerry L. Morgan, editors, _Syntax and Semantics: Speech Acts_ , volume 3, pages 41–58. Academic Press.
  * Gu et al. (2022) Yuling Gu, Yao Fu, Valentina Pyatkin, Ian Magnusson, Bhavana Dalvi Mishra, and Peter Clark. 2022.  Just-DREAM-about-it: Figurative Language Understanding with DREAM-FLUTE.
  * Horn (1972) Laurence R. Horn. 1972.  _On the semantic properties of logical operators in English_.  PhD Thesis, University of California Los Angeles.
  * Hu et al. (2020) Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020.  A Systematic Assessment of Syntactic Generalization in Neural Language Models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1725–1744, Online. Association for Computational Linguistics.
  * Jeretic et al. (2020) Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020.  Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8690–8705, Online. Association for Computational Linguistics.
  * Lakoff and Johnson (1980) G. Lakoff and M. Johnson. 1980.  _Metaphors We Live By_.  University of Chicago Press.
  * Le et al. (2019) Matthew Le, Y-Lan Boureau, and Maximilian Nickel. 2019.  Revisiting the Evaluation of Theory of Mind through Question Answering.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5872–5877, Hong Kong, China. Association for Computational Linguistics.
  * Leslie et al. (2004) Alan M. Leslie, Ori Friedman, and Tim P. German. 2004.  Core mechanisms in ‘theory of mind’.  _Trends in Cognitive Sciences_ , 8(12):528–533.
  * Levinson (2000) Stephen Levinson. 2000.  _Presumptive meaning: The theory of generalized conversational implicature_.  MIT Press.
  * Li et al. (2021) Elissa Li, Sebastian Schuster, and Judith Degen. 2021.  Predicting Scalar Inferences From "Or" to "Not Both" Using Neural Sentence Encoders.  In _Proceedings of the Society for Computation in Linguistics_ , volume 4.
  * Li (2012) Jin Li. 2012.  _Cultural Foundations of Learning: East and West_.  Cambridge University Press, Cambridge.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016.  Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies.  _Transactions of the Association for Computational Linguistics_ , 4:521–535.  Place: Cambridge, MA Publisher: MIT Press.
  * Liu et al. (2022) Emmy Liu, Chenxuan Cui, Kenneth Zheng, and Graham Neubig. 2022.  Testing the Ability of Language Models to Interpret Figurative Language.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4437–4452, Seattle, United States. Association for Computational Linguistics.
  * Martin and Ford (2018) R.A. Martin and T. Ford. 2018.  _The Psychology of Humor: An Integrative Approach_.  Academic Press.
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448, Florence, Italy. Association for Computational Linguistics.
  * Nematzadeh et al. (2018) Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Tom Griffiths. 2018\.  Evaluating Theory of Mind in Question Answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2392–2400, Brussels, Belgium. Association for Computational Linguistics.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language Models are Unsupervised Multitask Learners.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rubio-Fernandez (2021) Paula Rubio-Fernandez. 2021.  Pragmatic markers: the missing link between language and Theory of Mind.  _Synthese_ , 199(1):1125–1158.
  * Rubio-Fernandez and Jara-Ettinger (2020) Paula Rubio-Fernandez and Julian Jara-Ettinger. 2020.  Incrementality and efficiency shape pragmatics across languages.  _Proceedings of the National Academy of Sciences_ , 117(24):13399–13404.
  * Ruis et al. (2022) Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. 2022.  Large language models are not zero-shot communicators.
  * Sap et al. (2022) Maarten Sap, Ronan LeBras, Daniel Fried, and Yejin Choi. 2022.  Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs.  To appear at EMNLP.
  * Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019.  Social IQa: Commonsense Reasoning about Social Interactions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4463–4473, Hong Kong, China. Association for Computational Linguistics.
  * Schuster et al. (2020) Sebastian Schuster, Yuxing Chen, and Judith Degen. 2020.  Harnessing the linguistic signal to predict scalar inferences.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5387–5403, Online. Association for Computational Linguistics.
  * Searle (1975) John R. Searle. 1975.  Indirect Speech Acts.  In _Speech Acts_ , pages 59–82. Brill, Leiden, The Netherlands.
  * Shah and Bender (2022) Chirag Shah and Emily M. Bender. 2022.  Situating Search.  In _ACM SIGIR Conference on Human Information Interaction and Retrieval_ , CHIIR ’22, pages 221–232, New York, NY, USA. Association for Computing Machinery.  Event-place: Regensburg, Germany.
  * Shain et al. (2022) Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger P Levy. 2022\.  Large-Scale Evidence for Logarithmic Effects of Word Predictability on Reading Time.
  * Stowe et al. (2022) Kevin Stowe, Prasetya Utama, and Iryna Gurevych. 2022.  IMPLI: Investigating NLI Models’ Performance on Figurative Language.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5375–5388, Dublin, Ireland. Association for Computational Linguistics.
  * Taylor et al. (2022) Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022\.  Galactica: A Large Language Model for Science.
  * Tong et al. (2021) Xiaoyu Tong, Ekaterina Shutova, and Martha Lewis. 2021.  Recent advances in neural metaphor processing: A linguistic, cognitive and social perspective.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4673–4686, Online. Association for Computational Linguistics.
  * Trosborg (2010) Anna Trosborg, editor. 2010.  _Pragmatics across Languages and Cultures_.  De Gruyter Mouton.
  * Veatch (1998) Thomas C. Veatch. 1998.  A theory of humor.  _Humor_ , 11(2):161–216.
  * Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddhartha Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi, and Daniel Khashabi. 2022.  Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks.  To appear at EMNLP.
  * Warstadt and Bowman (2022) Alex Warstadt and Samuel R. Bowman. 2022.  What Artificial Neural Networks Can Tell Us About Human Language Acquisition.  In Shalom Lappin and Jean-Philippe Bernardy, editors, _Algebraic Structures in Natural Language_. Taylor & Francis.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022.  Finetuned Language Models are Zero-Shot Learners.  In _International Conference on Learning Representations_.
  * Wilcox et al. (2022a) Ethan Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2022a.  Learning syntactic structures from string input.  In Shalom Lappin and Jean-Philippe Bernardy, editors, _Algebraic Structures in Natural Language_. Taylor & Francis.
  * Wilcox et al. (2022b) Ethan Gotlieb Wilcox, Richard Futrell, and Roger Levy. 2022b.  Using Computational Models to Test Syntactic Learnability.  _Linguistic Inquiry_ , pages 1–88.
  * Wilson and Sperber (2012) D. Wilson and D. Sperber. 2012.  _Meaning and Relevance_.  Cambridge University Press.
  * Wilson and Sperber (1992) Deirdre Wilson and Dan Sperber. 1992.  On verbal irony.  _Lingua_ , 87(1):53–76.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-Art Natural Language Processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yule (1996) George Yule. 1996.  _Pragmatics_ , 1 edition.  Oxford Introduction to Language Study. Oxford University Press.
  * Zadeh et al. (2019) Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. 2019.  Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8799–8809.
  * Zheng et al. (2021) Zilong Zheng, Shuwen Qiu, Lifeng Fan, Yixin Zhu, and Song-Chun Zhu. 2021.  GRICE: A Grammar-based Dataset for Recovering Implicature and Conversational rEasoning.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 2074–2085, Online. Association for Computational Linguistics.

##  Appendix A Example prompts

This section contains example prompts for each task in our experiments. See
Section 3 and Table 1 for details on the materials, and Section 4.1 for
discussion of how prompts were constructed.

###  A.1 Deceits

Task: You will read short stories that describe two characters interacting
with each other. Each story will be followed by a multiple-choice question.
Read each story and choose the best answer to each question. Your task is to
decide why the character in the story responds in a certain way. The answer
options are 1, 2, 3, or 4.

Scenario: Henry is sitting at his desk and watching TV, and reluctantly
switches off the TV with the remote control and picks up a textbook. Shortly
after, his mother comes in the room and asks, "What have you been doing up
here?" Henry responds: "Reading." Why has Henry responded in such a way?

Options:

1) He has been reading for some time.

2) He does not want to offend his mom by not reading the books that she gave
him.

3) He does not want to get into trouble for not studying.

4) He wants his mom to believe that he has been watching TV.

Answer:

###  A.2 IndirectSpeech

Task: You will read short stories that describe everyday situations. Each
story will be followed by a multiple-choice question. Read each story and
choose the best answer. Your task is to decide what the character in the story
is trying to convey. The answer options are 1, 2, 3, or 4.

Scenario: Nate is about to leave the house. His wife points at a full bag of
garbage and asks: "Are you going out?" What might she be trying to convey?

Options:

1) She wants Nate to spend more time with the family.

2) She wants to know Nate’s plans.

3) She wants Nate to take the garbage out.

4) She wants Nate to bring his friends over.

Answer:

###  A.3 Irony

Task: You will read short stories that describe everyday situations. Each
story will be followed by a multiple-choice question. Read each story and
choose the best answer. Your task is to decide what the character in the story
is trying to convey. The answer options are 1, 2, 3, or 4.

Scenario: It is a holiday. Stefan and Kim are sitting in the backseat of the
car. They are fighting all the time. Their father says: "Oh, it is so pleasant
here." What did the father want to convey?

Options:

1) He enjoys listening to his kids fighting.

2) He remembers about his wife’s birthday.

3) He does not want to listen to his kids’ arguments.

4) AC gives them some needed cool.

Answer:

###  A.4 Maxims

Task: You will read short stories that describe everyday situations. Each
story will be followed by a multiple-choice question. Read each story and
choose the best answer. Your task is to decide why the character in the story
responds in a certain way. The answer options are 1, 2, 3, or 4.

Scenario: Leslie and Jane are chatting at a coffee shop. Leslie asks, ""Who
was that man that I saw you with last night?"" Jane responds, ""The latte is
unbelievable here."" Why has Jane responded like this?

Options:

1) She does not want to discuss the topic that Leslie has raised.

2) The man who Leslie saw makes unbelievable lattes.

3) She thinks that it is the best latte in the town.

4) A coffee break is not a good time to discuss men.

Answer:

###  A.5 Metaphor

Task: You will read short stories that describe everyday situations. Each
story will be followed by a multiple-choice question. Read each story and
choose the best answer to each question. The answer options are 1, 2, 3, 4, or
5.

Scenario: Andrew and Bob were discussing the investment company where Andrew
works. Bob said: "The investors are squirrels collecting nuts." What does Bob
mean?

Options:

1) The investors dress and eat well.

2) Squirrels were hired to work in the company.

3) Bob is allergic to nuts.

4) They buy stocks hoping for future profit.

5) The investors enjoy picking nuts as much as squirrels do.

Answer:

###  A.6 Humor

Task: You will read jokes that are missing their punch lines. A punch line is
a funny line that finishes the joke. Each joke will be followed by five
possible endings. Please choose the ending that makes the joke funny. The
answer options are 1, 2, 3, 4, or 5.

Joke: Martha walked into a pastry shop. After surveying all the pastries, she
decided on a chocolate pie. "I’ll take that one," Martha said to the
attendant, "the whole thing." "Shall I cut it into four or eight pieces?" the
attendant asked.

Punchlines:

1) Martha said, "My leg is hurting so much."

2) Martha said, "Four pieces, please; I’m on a diet."

3) Martha said: "Well, there are five people for dessert tonight, so eight
pieces will be about right."

4) Then the attendant squirted whipped cream in Martha’s face.

5) Martha said, "You make the most delicious sweet rolls in town."

Answer:

###  A.7 Coherence

Task: You will read pairs of sentences. Reach each pair and decide whether
they form a coherent story. The answer options are 1 or 2.

Scenario: Cleo brushed against a table with a vase on it. She decided to study
harder to catch up.

Options:

1) Incoherent

2) Coherent

Answer:

##  Appendix B Timestamps of OpenAI model queries

Table 3 shows timestamps of requests sent to the OpenAI API.

Model | Phenomenon | Timestamp
---|---|---
text-ada-001 | Coherence | 2022-10-11 12:28 -0400
text-ada-001 | Deceits | 2022-10-11 12:28 -0400
text-ada-001 | IndirectSpeech | 2022-10-11 12:28 -0400
text-ada-001 | Irony | 2022-10-11 12:28 -0400
text-ada-001 | Humor | 2022-10-11 12:28 -0400
text-ada-001 | Maxims | 2022-10-11 12:29 -0400
text-ada-001 | Metaphor | 2022-10-11 12:29 -0400
text-davinci-002 | Coherence | 2022-10-11 11:56 -0400
text-davinci-002 | Deceits | 2022-10-11 11:55 -0400
text-davinci-002 | IndirectSpeech | 2022-10-11 11:55 -0400
text-davinci-002 | Irony | 2022-10-11 11:54 -0400
text-davinci-002 | Humor | 2022-10-11 11:53 -0400
text-davinci-002 | Maxims | 2022-10-11 11:56 -0400
text-davinci-002 | Metaphor | 2022-10-11 11:57 -0400
Table 3: Timestamps of OpenAI API model queries.

##  Appendix C Details of human experiments

Below, we discuss details of the no-context human experiments described in
Section 5.3.2.

##### Participants.

All participants were recruited through Amazon.com’s online survey platform
and were paid at a base rate of 7 USD for completing the study. We collected
data from 30 participants. Additionally, we pre-screened participants using a
qualification task in which they were asked to perform 10 simple sentence
completions, which were judged for basic levels of coherence and
grammaticality. This study was approved by the Institutional Review Board at
the home institution of the authors (protocol 2010000243).

##### Procedure.

Participants completed these tests during one individual testing session.
After giving informed consent, they were shown instructions and a training
trial, in which they were told they would be answering questions about a
character in a short interaction. They then saw 105 trials (similar to those
described in Appendix A), without the scenario context. For example:

Bob said: "The investors are squirrels collecting nuts." What does Bob mean?

1) The investors dress and eat well.

2) Squirrels were hired to work in the company.

3) Bob is allergic to nuts.

4) They buy stocks hoping for future profit.

5) The investors enjoy picking nuts as much as squirrels do.

Items were presented within blocks according to their phenomenon, as in Floyd
et al.’s (In prep) original experiments. Blocks and items were presented in a
random order.