# An information-theoretic model of shallow and deep language comprehension

Jiaxuan Li<EMAIL_ADDRESS>
Department of Language Science, University of California Irvine
Irvine, CA 92617 USA Richard Futrell<EMAIL_ADDRESS>
Department of Language Science, University of California Irvine
Irvine, CA 92617 USA

###### Abstract

A large body of work in psycholinguistics has focused on the idea that online
language comprehension can be shallow or ‘good enough’: given constraints on
time or available computation, comprehenders may form interpretations of their
input that are plausible but inaccurate. However, this idea has not yet been
linked with formal theories of computation under resource constraints. Here we
use information theory to formulate a model of language comprehension as an
optimal trade-off between accuracy and processing depth, formalized as bits of
information extracted from the input, which increases with processing time.
The model provides a measure of processing effort as the change in processing
depth, which we link to EEG signals and reading times. We validate our theory
against a large-scale dataset of garden path sentence reading times, and EEG
experiments featuring N400, P600 and biphasic ERP effects. By quantifying the
timecourse of language processing as it proceeds from shallow to deep, our
model provides a unified framework to explain behavioral and neural signatures
of language comprehension.

Keywords: Good-enough comprehension; Information theory; Rate–Distortion
Theory; N400; P600; Reading Time; Syntactic Ambiguity

## Introduction

Language comprehension can be inaccurate in the sense that a comprehender’s
interpretations of their input may not reflect the input veridically. For
example, given input such as

  * (1)

The storyteller could turn any incident into an amusing _antidote_ …

a comprehender may initially interpret the anomalous word _antidote_ as the
more plausible _anecdote_ , and the misinterpretation may persist [83, 84, 78,
69, 117, 80, 79]. Similarly, processing of syntactic structure can be
inaccurate, with comprehenders interpreting a sentence such as

  * (2)

When the little girl attacked the lamb remained calm.

as meaning that the little girl attacked the lamb, while the ‘correct’
interpretation derived through a full parse of the sentence is that the girl
was attacked by the lamb [76, 81].

Such observations have motivated the idea that language processing can be
‘good enough’ or heuristic: comprehenders are willing to tolerate some level
of inaccuracy in their interpretations in return for not having to expend too
much computational effort [80, 82]. More generally, language processing has
been held to proceed through stages reflecting different levels of depth, with
an initial shallow, heuristic, and local stage of processing, which may be
followed by deeper, global, and more effortful processing when necessary [123,
88, 110, 91, 96, 125, 70, 87, 122].

While the idea of shallow processing explains a wide variety of phenomena and
has intuitive appeal, the idea of ‘depth of processing’ has yet to be made
precise in a way that links it formally to theories of computation under
resource constraints [106, 109]. We propose a computational model in which
language comprehension instantiates optimal tradeoffs between accuracy and
processing depth, where processing depth is measured information-theoretically
in terms of bits of information extracted from perceptual input. Our model
joins recent explanations of human and animal perception and behavior based on
rate–distortion theory [118, 119, 133, 132, 112, 75, 71, 68, 89]. Furthermore,
our model provides a link between notions of depth of processing, processing
effort, and processing time: we hold that processing depth increases with
time, and that an increase in processing depth is reflected in processing
effort, which in turn is reflected in EEG signals and reading times.

Below, we present our model using comprehension of Example (1) as a running
example. Then we validate the model by simulating N400 and P600 effects
elicited by a variety of complex semantic and syntactic manipulations from two
experiments, and by simulating reading times for garden-path constructions in
the Syntactic Ambiguity Processing dataset [97], successfully predicting the
relative magnitudes of garden path slowdowns in reading time.

## Model

Figure 1: Tradeoff between distortion and processing depth (KL divergence) in
optimal interpretation policies for the given input. Each location in the
white part of the plane represents a possible interpretation policy for the
input; the tradeoffs in the gray region are unachievable. The black line shows
the _efficient frontier_ of policies that achieve the minimal distortion for a
given level of processing depth. We hold that interpretation policies move
down this frontier with increasing processing time.

We characterize online language comprehension using an , which represents a
comprehender’s probability distribution on interpretations $w$ given
perceptual inputs $x$. We hold that comprehenders’ interpretation policies
minimize a distortion metric—a measure $d(w,x)$ of how bad it is to form an
interpretation $w$ given input $x$—subject to a constraint on bits of
information extracted from the sensory input. An example of interpretation
policies and their tradeoffs is shown in Figure 1. We furthermore posit that
the tradeoff of distortion and processing depth depends on time: while a
comprehender is perceiving input $x$, more information becomes available to
them, and so their interpretation policy moves along the efficient frontier in
Figure 1, evolving from a shallow, inaccurate policy to a a deep, accurate
policy while maintaining an optimal tradeoff.

### Formal model

We propose that the comprehenders’ interpretation policy $p_{t}(w|x)$ at a
given time $t$ is chosen to minimize the expected distortion subject to a
constraint on the depth of processing, $D(t)$:

$\displaystyle\mathop{\text{minimize}}_{p_{t}(w\mid
x)}\mathop{\mathbb{missing}}{E}_{p(x)p_{t}(w\mid x)}\left[d(w,x)\right]\text{
subject to }D(t)\leq C(t),$ (1)

where the processing depth $D(t)$ is the average KL divergence [77] between
the interpretations $p_{t}$ and the ‘default’ policy $p_{0}(w)$ representing a
comprehender’s prior expectations:

$D(t)=\mathop{\mathbb{missing}}{E}_{p(x)}\left[D_{\text{KL}}\left[p_{t}(w\mid
x)\|p_{0}(w)\right]\right],$ (2)

and $C(t)$ is a constraint on the processing depth achievable at time $t$
[75]. The KL divergence reflects the extent to which the interpretation policy
$p_{t}(w\mid x)$ moves away from the default interpretations $p_{0}(w)$ in
response to the information provided by the input $x$.

To summarize, we are optimizing the processing accuracy (minimizing
distortion) in Eq. 1 with a processing depth constraint as defined in Eq. 2.
Applying the method of Lagrange multipliers, the optimal interpretation
policies have the form

$p_{t}(w\mid x)=\frac{1}{Z_{\lambda}(x)}p_{0}(w)e^{-\lambda(t)d(w,x)},$ (3)

where $Z_{\lambda}(x)$ is a normalizing constant. We posit that time-varying
function $\lambda(t)$ has the form $\lambda(t)=\lambda_{0}t$ for constant
$\lambda_{0}$ (thus implicitly defining the time-varying depth constraint
$C(t)$). This stipulation is inspired by models of perception as noisy
evidence accumulation [73, 113, 72], where a prior $p_{0}(w)$ is continually
updated in response to noisy perceptual evidence. The final interpretation
policy which we use in this paper is

$p(w\mid x)\propto p_{0}(w)e^{-\lambda_{0}td(w,x)}.$ (4)

To give some intuition for model behavior: at time $t=0$, the interpretation
policy reduces to $p_{t}(w\mid x)=p_{0}(w)$, meaning that a comprehender’s
interpretations are based on prior expectations, taking into account no
information from the input. As the time $t\rightarrow\infty$, the policy
$p_{t}(w\mid x)$ concentrates all probability mass on interpretations with
minimum distortion, that is, on veridical interpretations, at which point the
processing depth is equal to the surprisal of the input, $D(t)=-\log p(x)$,
indicating that all the information in the input $x$ has been processed.

Figure 2: Top. Probabilities of four interpretations $w$ given input
$x=\text{``story''}$ in the given context as a function of processing time, as
predicted from Eq. 4 with parameters described in the text. Bottom. Measures
of processing effort over time for the same input. Figure 3: Processing
timecourses for four different inputs in the given context. “Anecdote” is the
control and the most likely completion. “Hearse” represents a semantic
anomaly, “anecdotes” represents a syntactic anomaly, and “antidotes”
represents a recoverable anomaly—semantically anomalous input that can be
easily mistaken for a more likely input.

### Application to language comprehension

We represent the inputs $x$ and interpretations $w$ as strings. We use a
distortion measure that reflects a mixture of semantic and form-based distance
between $w$ and $x$:

$d(w,x)=d_{\varphi}(w,x)+\gamma d_{\sigma}(w,x),$ (5)

where $d_{\varphi}$ is a form-based phonological or orthographic distance
metric, $d_{\sigma}$ is a semantic distance metric, and $\gamma$ is a scalar
controlling the relative importance of semantic as opposed to form-based
distance. We implement the form-based distance as orthographic edit distance
and semantic distance as cosine distance between GPT-2 word embeddings [115].
For the default policy $p_{0}(w)$, we use the GPT-2 language model [115].

Figure 2 shows model behavior using this setup for the last word in Example
(1). For this example, we use only form-based distortion (that is,
$\gamma=0$), and the set of possible interpretations $w$ ranges over the
10,000 most frequent words in SUBTLEXus [74]. Initially, when no perceptual
input is taken into account, “story” is the most likely interpretation, as it
is the most likely continuation in the context under the GPT-2 language model.
With short processing time, “anecdote” appears most likely, as it is highly
expected in context and also close in distortion to the true input. With
deeper processing, all probability mass is concentrated asymptotically on the
veridical “antidote”. The shift from “anecdote” to “antidote” happens when
processing depth reaches around 5 bits.

### Processing effort

So far our model describes the evolution of a comprehender’s interpretations
of their input with time. We require a link between this model and measures of
processing effort, such as EEG amplitude and reading time. We propose that
processing effort corresponds to _change_ in processing depth: it requires
effort to increase depth. We define the instantaneous at time $t$ as the rate
of change of processing depth $D(t)$, given by the time derivative
$D^{\prime}(t)=\frac{d}{dt}D(t)$.

Example timecourses for depth $D(t)$ and effort $D^{\prime}(t)$ are shown in
Figure 2, bottom. Here, the early concentration of probability mass on
“anecdote” is reflected in an initial pulse in the effort $D^{\prime}(t)$. The
later shift of probability mass from “anecdote” onto the veridical “antidote”
creates a second pulse. Thus, processing _appears_ to proceed in two distinct
stages: first a ‘shallow’ stage of processing, followed by a later ‘deep’
error correction. This behavior is in fact generated by a single process of
continuously increasing processing depth with time.

### Link to EEG measures

The idea of shallow-to-deep language processing has also been used to explain
EEG signals of language comprehension, with the N400 signal a function of
effort in shallow processing and the later P600 signal a signal of anomalies
detected through deep processing [104, 93, 100, 127, 126, 99, 102, 103, 128,
101, 96]. In support of this idea, anomalous words such as those in Example
(1), which are surprising but close in form to a plausible alternative, yield
a biphasic N400 and P600 effect [116].

To link our measures to EEG data, we posit the voltage $V(t)$ of EEG signal at
time $t$ is proportional to the effort $D^{\prime}(t)$ modulated by a carrier
wave with angular frequency $\omega$ and phase $\phi$, as

$V(t)\propto-D^{\prime}(t)\sin(\omega t+\phi).$ (6)

In this view, the N400 and P600 signals are manifestations of a singular
underlying process of increasing processing depth. N400 arises when there is a
large early pulse in effort, and P600 arises when there is a late pulse.

In support of this idea, Figure 3 shows processing timecourses for four
different inputs used in [116]: a control input “anecdote”, a syntactically
anomalous input “anecdotes”, a semantically anomalous input “hearse”, and a
recoverable semantic anomaly “antidote” that is phonologically close to the
control. The semantic anomaly yields a single, large, delayed pulse in
processing effort. The delay happens because early processing for this input
fails to concentrate probability mass on any particular
interpretation—“hearse” is initially so unlikely under $p_{0}(w)$ that a great
deal of evidence has to be accumulated for its probability to rise. In
contrast, the syntactic and recoverable semantic anomalies create two pulses
of processing effort. For the syntactic anomaly, the timecourse of processing
is nearly identical to the control input up to a depth of around 5 bits, at
which point there is a very late and temporally diffuse second pulse of
processing effort, corresponding to the P600 signal. Mapping the processing
effort onto predicted voltages following Eq. 6 in Figure 4, we predict (1)
early negativity for the semantic and recoverable semantic anomalies, and (2)
late positivity for the syntactic and recoverable semantic anomalies. This
pattern matches the empirical findings.

Figure 4: Simulated EEG signal corresponding to the processing timecourses in
Figure 3, using Eq. 6 with $\omega=1$ and $\phi=-\nicefrac{{2}}{{3}}$.

### Link to reading times

To capture reading times, we posit that comprehenders move to the next word
when they judge that deeper processing is no longer necessary. Concretely, we
model reading times as the minimum amount of time needed for processing effort
to fall below a threshold $\varepsilon$. It is thus possible that a reader may
move on to the next word before fully resolving the current word, as long as
the _change_ in processing depth with time is small enough. For example, in
the processing timecourse for the syntactically anomalous “anecdotes” in
Figure 3, a reader may move on during the trough between the two pulses in
processing effort, if the processing effort falls below the threshold
$\varepsilon$ at this point. In that case the reader would have a lingering
misinterpretation.

### Relation to noisy-channel models

Our shallow interpretation policy generalizes noisy channel models of language
comprehension [90, 116, 114]. The policy reproduces the predictions of noisy-
channel models in the special case when $\lambda(t)=1$ and the distortion
metric is equal to a log-likelihood under a noise model, i.e. $d(w,x)=-\ln
p_{N}(x\mid w)$ for some noise model $p_{N}$. It differs from noisy channel
models in its cognitive interpretation and in its generality: in the shallow
processing model, the distortion $d(w,x)$ need not be a log-likelihood (that
is, it need correspond to a normalized probability $p_{N}(x\mid w)$).
Increasing $t$ in Eq. 4 corresponds to accumulation of samples of evidence in
a noisy-channel model of perception [113, 72].

## Study 1: N400, P600, and biphasic EEG signals

Exp | Condition | Context | Target | ERPs
---|---|---|---|---
Ito-16 | SemRelated |  The student is going to the library to borrow a … | page | Reduced N400
FormRelated | hook | Biphasic
Unrelated | sofa | N400
Control | book | NA
Ryskin-21 | Semantic |  The storyteller could turn any incident into an amusing … | hearse | N400
Syntactic | anecdotes | P600
SemCrit | antidote | Biphasic
Control | anecdote | NA

Table 1: List of conditions, sample sentences and ERP patterns in dataset.

### Dataset

We validate our theory on two EEG experiments, featuring N400 effect, P600
effect and biphasic effect. Table 1 shows a list of conditions with sample
stimuli and empirical ERP patterns across experiments. The ERP effects in the
experimental conditions are all calculated in terms of differences to the EEG
amplitude in the control condition. The experiment _Ito-16_ includes three
different kinds of semantic violations [99] that are semantically related
(_SemRelated_), orthographically related (_FormRelated_) or unrelated
(_Unrelated_) to the predicable target. The N400 effects in _FormRelated_ and
_SemRelated_ conditions are smaller than in the _Unrelated_ conditions, and
only _FormRelated_ condition elicits a P600 effect. The experiment _Ryskin-21_
introduces semantic violations (_Semantic_ condition), syntactic violations
(_Syntactic_ condition) and semantic violations that are phonologically
similar to the predictable control target (_SemCrit_ condition). There is a
biphasic N400–P600 effect in the _SemCrit_ condition, but the effect size is
smaller than the N400 in the _Semantic_ condition and P600 in the _Syntactic_
condition.

### Implementation

We calculate interpretation policies over time for the critical word in each
sentence, with the set of alternative interpretations $w$ consisting of the
words in all conditions. We set the parameter $\gamma$ by visual inspection,
obtaining $\gamma=10$ for Ryskin-21 and $\gamma=6$ for Ito-16.

### Results

Figure 5 shows that our model simulates the observed EEG patterns in both
Ito-16 and Ryskin-21. In Ito-16, our model predicts a graded N400 effects
elicited by three kinds of semantic violations (_Unrelated_ $>$ _SemRelated_
$>$ _FormRelated_), where semantic violations that are either orthographically
nor semantically related to the predicted word trigger the largest N400
responses. Our model also predicts a greater P600 amplitude in _FormRelated_
than other conditions. In Ryskin-21, our model predicts the order the effect
size across conditions: a graded N400 effect (_Semantic_ $>$ _SemCrit_ $>$
_Syntactic_) and a graded P600 effect (_Syntactic_ $>$ _SemCrit_ $>$
_Semantic_).

The model behavior can be explained in terms of the different rates of
probabilistic distribution shift: the semantically anomalous conditions with a
large N400 effect tend to have the fastest probability update, and anomalous
conditions with a large P600 effect (e.g. _Syntactic_ condition in Ryskin-21;
_FormRelated_ condition in Ito-16) has the lowest probability update rate. The
different probability shift rate results in the increase of instantaneous
effort to appear early or delayed.

(a) Ito-16 experiment

(b) Ryskin-21 experiment

Figure 5: Simulation results for EEG experiments. Left. Instantaneous
processing effort ($D^{\prime}(t)$) Middle. Simulated N400 effect size. Right.
Simulated P600 effect size.

## Study 2: Garden path reading times

Type |  Sentence
---|---
MV/RR |  The little girl fed the lamb remained relatively calm…
The little girl who was fed the lamb remained relatively calm…
NP/S |  The little girl found the lamb remained relatively calm.
The little girl found that the lamb remained relatively calm…
NP/Z |  When the little girl attacked the lamb remained relatively calm…
When the little girl attacked, the lamb remained relatively calm…

Table 2: A list of garden path constructions and example sentences. Critical
words are marked in bold. The critical words in garden path sentences are
underlined, and the control counterparts are italic.

### Dataset

We used a set of three classic garden path constructions from large-scale
syntactic ambiguity processing benchmark, consisting of self-paced reading
time data from 2000 participants [97]. The constructions include: Main verb/
reduced relative clause garden path (MV/RR), Direct object/ sentential
complement garden path (NP/S), Transitive/ intransitive garden path (NP/Z)
(see Table 2). There are 24 lexically matched item sets for each construction.
Each set has an experimental sentence with a tempory syntactic ambiguity and
an unambiguous control sentence. These constructions generate reliable garden
path effects [86, 70, 92, 121], but differ in the magnitude of the effect
[129, 97]. In particular, NP/Z constructions have the largest effect size at
the critical region, followed by MV/RR and NP/S: see Figure 6(a).

Figure 6:  (a) Average effect size (RT difference to unambiguous control) at
critical region estimated from human experiment. (b) Average effect size
estimated from model simulation. (c) Average surprisal difference between
experimental and control sentences. (d) Average edit distance between
presented sentence and closest alternative. Error bars show standard errors.

### Implementation

We consider interpretations $w$ ranging over the entire sentence, rather than
only the critical word, with a set of alternatives consisting of all sentences
in a lexically matched set across different constructions as possible
interpretations given a single sentence. This simulates the scenario where a
comprehender is re-evaluating and updating their beliefs about the entire
sentence in parallel [131, 130]. The prior $p(w)$ is calculated as the sum of
conditional log probability for every word in the sentence from GPT-2. We set
$\gamma=0$ because all alternatives are lexically matched, and set the reading
time threshold to $\varepsilon=.001$.

Figure 7: Top. Instantaneous processing effort over time for (a) MV/RR; (b)
NP/S and (c) NP/Z constructions. The dotted red line indicates the reading
time threshold $\varepsilon=.001$. Bottom. Probabilities of three different
interpretations $w$ given an experimental sentence (_Target_). _Distractor_ is
the counterpart control sentence and _Far Distractor_ is a sentence with a
different syntactic construction.

### Results

Figure 6 shows that our simulation replicates the relative magnitude of the
three garden path effects. This result cannot be explained by the surprisal
difference between experimental and control conditions, where the MV/RR has
the largest surprisal difference followed by NP/S and NP/Z, nor by the
distortion (edit distance) alone: see Figure 6(c–d).

The model captures the reading time differences among garden paths because the
reading time is affected by the comparative strength of different candidate
interpretations, which is jointly decided by both prior expectations and
similarity to the input. In Figure 7(a–b), we see that for MV/RR and NP/S, the
veridical interpretation initially has a very low probability, but the
probability increases rapidly because the distance to the more-likely control
interpretation is large. On the other hand, for NP/Z—shown in Figure 7(c)—the
veridical interpretation is not extremely low probability initially, but the
probability for this interpretation grows very slowly because there is only
weak evidence for it: the distance between the veridical input and the closest
distractor (the unambiguous variant with a comma) is very small.

## Discussion

We have proposed a unified computational model of language comprehension as a
function of depth of processing. In the model, shallow interpretations are
generated based on a trade-off between accuracy and the amount of information
extracted from perceptual input. We quantify reading time and EEG signals as
two different indices of changes in comprehenders’ beliefs. Our model provides
a simple and transparent explanation for the order of effect size across three
different types of garden path constructions, and successfully simulates
certain ERP effects.

Our model quantifies the well-known hypothesis of a heuristic “shallow
processing” mechanism in a time-sensitive and dynamic way. Rather than
positing two discrete stages of shallow and then deep processing, or that
processing involves separate streams of heuristic and veridical input [100,
127, 124], our model posits a single interpretation process with continuously
increasing in processing depth. For certain inputs, this process creates
dynamics with two pulses of processing effort, corresponding to formation of a
shallow interpretation followed by error detection or correction.

Our model enhances theories of processing difficulty based on expectations by
introducing a new factor in the form of distortion. Our model’s indices of
processing effort are closely related to the surprisal of the input, which has
been proposed as a predictor of reading times [95, 105] and EEG amplitudes
[85, 111]. The veridical interpretation’s initial probability at time $t=0$ is
determined entirely by prior expectations, and the total cumulative effort to
process a word is equal to its surprisal. Our model differs from Surprisal
Theory in that reading time is also influenced by distances to distractor
interpretations. These distances determine the _rate_ at which the probability
of the correct interpretation increases, yielding potentially dramatic
slowdowns when that rate is small. Our model is also compatible with theories
of ERPs where the N400 reflects the surprisal of a likely but non-veridical
interpretation, P600 reflects reconciliation of interpretations, and the total
cumulative amplitude reflects surprisal [107, 108].

While we have demonstrated initial success for our model, it has some
limitations. Most serious is the representation of possible interpretations
$w$ as strings, with the concomitant use of edit distance (augmented with
semantic distance) for the distortion. In a more complete model
implementation, it is possible that interpretations $w$ may range over
structured objects such as syntactic treelets, with distortion reflecting a
structure-sensitive distance function among these objects. Future work will
determine the proper structure and distortion measure for comprehenders’
interpretations.

Our model and results provide evidence that processing constraints in language
comprehension can be captured information-theoretically in terms of rational
use of computational resources [94], joining with similar models in the
domains of language production [89], semantics [133, 120, 98], and pragmatics
[132, 134], as well as other areas of cognition [71, 68].

## References

  * [1] Dilip Arumugam, Mark K Ho, Noah D Goodman and Benjamin Van Roy  “On Rate-Distortion Theory in Capacity-Limited Cognition & Reinforcement Learning”  In _arXiv preprint arXiv:2210.16877_ , 2022
  * [2] Stephen B. Barton and Anthony J. Sanford  “A case study of anomaly detection: Shallow semantic processing and cohesion establishment”  In _Memory & cognition_ 21.4 Springer, 1993, pp. 477–487
  * [3] Thomas G Bever  “The cognitive basis for linguistic structures”  In _Cognition and the development of language_ Wiley, 1970
  * [4] Rahul Bhui, Lucy Lai and Samuel J Gershman  “Resource-rational decision making”  In _Current Opinion in Behavioral Sciences_ 41 Elsevier, 2021, pp. 15–21
  * [5] Klinton Bicknell and Roger Levy  “A rational model of eye movement control in reading”  In _Proceedings of the 48th annual meeting of the Association for Computational Linguistics_ , 2010, pp. 1168–1178
  * [6] Rafal Bogacz et al.  “The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks.”  In _Psychological review_ 113.4 American Psychological Association, 2006, pp. 700
  * [7] Marc Brysbaert and Boris New  “Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English”  In _Behavior research methods_ 41.4 Springer, 2009, pp. 977–990
  * [8] Samuel J Cheyette and Steven T Piantadosi  “A unified account of numerosity perception”  In _Nature Human Behaviour_ 4.12 Nature Publishing Group UK London, 2020, pp. 1265–1272
  * [9] Kiel Christianson, Andrew Hollingworth, John F. Halliwell and Fernanda Ferreira  “Thematic roles assigned along the garden path linger”  In _Cognitive psychology_ 42.4 Elsevier, 2001, pp. 368–407
  * [10] Thomas M Cover and Joy A Thomas  “Elements of information theory second edition solutions to problems”  In _Internet Access_ , 2006, pp. 19–20
  * [11] Thomas D Erickson and Mark E Mattson  “From words to meaning: A semantic illusion”  In _Journal of Verbal Learning and Verbal Behavior_ 20.5 Elsevier, 1981, pp. 540–551
  * [12] Fernanda Ferreira  “The misinterpretation of noncanonical sentences”  In _Cognitive psychology_ 47.2 Elsevier, 2003, pp. 164–203
  * [13] Fernanda Ferreira and Nikole D Patson  “The ‘good enough’approach to language comprehension”  In _Language and Linguistics Compass_ 1.1-2 Wiley Online Library, 2007, pp. 71–83
  * [14] Fernanda Ferreira, Kiel Christianson and Andrew Hollingworth  “Misinterpretations of garden-path sentences: Implications for models of sentence processing and reanalysis”  In _Journal of psycholinguistic research_ 30 Springer, 2001, pp. 3–20
  * [15] Fernanda Ferreira, Karl GD Bailey and Vittoria Ferraro  “Good-enough representations in language comprehension”  In _Current directions in psychological science_ 11.1 SAGE Publications Sage CA: Los Angeles, CA, 2002, pp. 11–15
  * [16] Samuel Fillenbaum  “Processing and recall of compatible and incompatible question and answer pairs”  In _Language and Speech_ 14.3 Sage Publications Sage CA: Thousand Oaks, CA, 1971, pp. 256–265
  * [17] Samuel Fillenbaum  “Pragmatic normalization: Further results for some conjunctive and disjunctive sentences.”  In _Journal of Experimental Psychology_ 102.4 American Psychological Association, 1974, pp. 574
  * [18] Stefan Frank, Leun Otten, Giulia Galli and Gabriella Vigliocco  “The ERP response to the amount of information conveyed by words in sentences”  In _Brain and language_ 140 Elsevier, 2015, pp. 1–11
  * [19] Lyn Frazier  “On comprehending sentences: Syntactic parsing strategies.”  University of Connecticut, 1979
  * [20] Lyn Frazier and Janet Dean Fodor  “The sausage machine: A new two-stage parsing model”  In _Cognition_ 6.4 Elsevier, 1978, pp. 291–325
  * [21] Lyn Frazier and Keith Rayner  “Making and correcting errors during sentence comprehension: Eye movements in the analysis of structurally ambiguous sentences”  In _Cognitive psychology_ 14.2 Elsevier, 1982, pp. 178–210
  * [22] Richard Futrell  “Information-theoretic principles in incremental language production”  In _Proceedings of the National Academy of Sciences_ 120.39 National Acad Sciences, 2023, pp. e2220593120
  * [23] Edward Gibson, Leon Bergen and Steven T. Piantadosi  “Rational integration of noisy evidence and prior semantic expectations in sentence interpretation”  In _Proceedings of the National Academy of Sciences_ 110.20 National Acad Sciences, 2013, pp. 8051–8056
  * [24] Ana C Gouvea, Colin Phillips, Nina Kazanina and David Poeppel  “The linguistic processes underlying the P600”  In _Language and cognitive processes_ 25.2 Taylor & Francis, 2010, pp. 149–188
  * [25] Daniel Grodner, Edward Gibson, Vered Argaman and Maria Babyonyshev  “Against repair-based reanalysis in sentence comprehension”  In _Journal of Psycholinguistic Research_ 32 Springer, 2003, pp. 141–166
  * [26] Peter Hagoort, Colin Brown and Jolanda Groothusen  “The syntactic positive shift (SPS) as an ERP measure of syntactic processing”  In _Language and cognitive processes_ 8.4 Taylor & Francis, 1993, pp. 439–483
  * [27] Michael Hahn, Richard Futrell, Roger Levy and Edward Gibson  “A resource-rational model of human processing of recursive linguistic structure”  In _Proceedings of the National Academy of Sciences_ 119.43 National Acad Sciences, 2022, pp. e2122602119
  * [28] John Hale  “A probabilistic Earley parser as a psycholinguistic model”  In _Second meeting of the north american chapter of the association for computational linguistics_ , 2001
  * [29] John CJ Hoeks, Laurie A. Stowe and Gina Doedens  “Seeing words in context: the interaction of lexical and sentence level information during reading”  In _Cognitive brain research_ 19.1 Elsevier, 2004, pp. 59–73
  * [30] Kuan-Jung Huang et al.  “Surprisal does not explain syntactic disambiguation difficulty: evidence from a large-scale benchmark”  PsyArXiv, 2023
  * [31] Nathaniel Imel and Shane Steinert-Threlkeld  “Modal semantic universals optimize the simplicity/informativeness trade-off”  In _Semantics and Linguistic Theory_ 1, 2022, pp. 227–248
  * [32] Aine Ito et al.  “Predicting form and meaning: Evidence from brain potentials”  In _Journal of Memory and Language_ 86 Elsevier, 2016, pp. 157–171
  * [33] Albert Kim and Lee Osterhout  “The independence of combinatory semantic processing: Evidence from event-related potentials”  In _Journal of memory and language_ 52.2 Elsevier, 2005, pp. 205–225
  * [34] Herman HJ Kolk, Dorothee J Chwilla, Marieke Van Herten and Patrick JW Oor  “Structure and limited capacity in verbal working memory: A study with event-related potentials”  In _Brain and language_ 85.1 Elsevier, 2003, pp. 1–36
  * [35] Gina R. Kuperberg  “Neural mechanisms of language comprehension: Challenges to syntax”  In _Brain research_ 1146 Elsevier, 2007, pp. 23–49
  * [36] Gina R Kuperberg, Trevor Brothers and Edward W Wlotko  “A tale of two positivities and the N400: Distinct neural signatures are evoked by confirmed and violated predictions at different levels of representation”  In _Journal of Cognitive Neuroscience_ 32.1 MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info …, 2020, pp. 12–35
  * [37] Marta Kutas and Steven A. Hillyard  “Event-related brain potentials to semantically inappropriate and surprisingly large words”  In _Biological psychology_ 11.2 Elsevier, 1980, pp. 99–116
  * [38] Roger Levy  “Expectation-based syntactic comprehension”  In _Cognition_ 106.3 Elsevier, 2008, pp. 1126–1177
  * [39] Richard L Lewis, Andrew Howes and Satinder Singh  “Computational rationality: Linking mechanism and behavior through bounded utility maximization”  In _Topics in cognitive science_ 6.2 Wiley Online Library, 2014, pp. 279–311
  * [40] Jiaxuan Li and Allyson Ettinger  “Heuristic interpretation as rational inference: A computational model of the N400 and P600 in language processing”  In _Cognition_ 233 Elsevier, 2023, pp. 105359
  * [41] Jiaxuan Li and Richard Futrell  “A decomposition of surprisal tracks the N400 and P600 brain potentials”  In _Proceedings of the Annual Meeting of the Cognitive Science Society_ 45.45, 2023
  * [42] Falk Lieder and Thomas L Griffiths  “Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources”  In _Behavioral and brain sciences_ 43 Cambridge University Press, 2020, pp. e1
  * [43] Maryellen C. MacDonald, Neal J. Pearlmutter and Mark S. Seidenberg  “The lexical nature of syntactic ambiguity resolution.”  In _Psychological review_ 101.4 American Psychological Association, 1994, pp. 676
  * [44] James A Michaelov et al.  “Strong Prediction: Language model surprisal explains multiple N400 effects”  In _Neurobiology of language_ MIT Press One Broadway, 12th Floor, Cambridge, Massachusetts 02142, USA …, 2024, pp. 1–29
  * [45] Francis Mollica et al.  “The forms and meanings of grammatical markers support efficient communication”  In _Proceedings of the National Academy of Sciences_ 118.49 National Acad Sciences, 2021, pp. e2025993118
  * [46] Dennis Norris and Sachiko Kinoshita  “Perception as evidence accumulation and Bayesian inference: insights from masked priming.”  In _Journal of Experimental Psychology: General_ 137.3 American Psychological Association, 2008, pp. 434
  * [47] Till Poppels and Roger Levy  “Structure-sensitive Noise Inference: Comprehenders Expect Exchange Errors.”  In _CogSci_ , 2016
  * [48] Alec Radford et al.  “Language models are unsupervised multitask learners”  In _OpenAI blog_ 1.8, 2019, pp. 9
  * [49] Rachel Ryskin et al.  “An ERP index of real-time error correction within a noisy-channel framework of human communication”  In _Neuropsychologia_ 158 Elsevier, 2021, pp. 107855
  * [50] Anthony J Sanford  “Context, attention and depth of processing during interpretation”  In _Mind & language_ 17.1-2 Wiley Online Library, 2002, pp. 188–206
  * [51] Chris R Sims  “Rate–distortion theory and human perception”  In _Cognition_ 152 Elsevier, 2016, pp. 181–198
  * [52] Chris R Sims  “Efficient coding explains the universal law of generalization in human perception”  In _Science_ 360.6389 American Association for the Advancement of Science, 2018, pp. 652–656
  * [53] Shane Steinert-Threlkeld  “Quantifiers in natural language optimize the simplicity/informativeness trade-off”  In _Proceedings of the 22nd Amsterdam colloquium_ , 2020, pp. 513–522  Institute for Logic, LanguageComputation at the University of Amsterdam
  * [54] Patrick Sturt, Martin J Pickering and Matthew W Crocker  “Structural change and reanalysis difficulty in language comprehension”  In _Journal of Memory and Language_ 40.1 Elsevier, 1999, pp. 136–150
  * [55] Whitney Tabor, Bruno Galantucci and Daniel Richardson  “Effects of merely local syntactic coherence on sentence processing”  In _Journal of Memory and Language_ 50.4 Elsevier, 2004, pp. 355–370
  * [56] John C Trueswell, Michael K Tanenhaus and Christopher Kello  “Verb-specific constraints in sentence processing: separating effects of lexical preference from garden-paths.”  In _Journal of Experimental psychology: Learning, memory, and Cognition_ 19.3 American Psychological Association, 1993, pp. 528
  * [57] Nan Van de Meerendonk, Peter Indefrey, Dorothee J Chwilla and Herman HJ Kolk  “Monitoring in language perception: Electrophysiological and hemodynamic responses to spelling violations”  In _Neuroimage_ 54.3 Elsevier, 2011, pp. 2350–2363
  * [58] Roger PG Van Gompel, Martin J. Pickering and Matthew J. Traxler  “Unrestricted race: A new model of syntactic ambiguity resolution”  In _Reading as a perceptual process_ Elsevier, 2000, pp. 621–648
  * [59] Marieke Van Herten, Herman HJ Kolk and Dorothee J. Chwilla  “An ERP study of P600 effects elicited by semantic anomalies”  In _Cognitive brain research_ 22.2 Elsevier, 2005, pp. 241–255
  * [60] Marieke Van Herten, Dorothee J. Chwilla and Herman HJ Kolk  “When heuristics clash with parsing routines: ERP evidence for conflict monitoring in sentence perception”  In _Journal of cognitive neuroscience_ 18.7 MIT Press, 2006, pp. 1181–1197
  * [61] Cyma Van Petten and Barbara J Luka  “Prediction during language comprehension: Benefits, costs, and ERP components”  In _International Journal of Psychophysiology_ 83.2 Elsevier, 2012, pp. 176–190
  * [62] Marten Van Schijndel and Tal Linzen  “Single-Stage Prediction Models Do Not Explain the Magnitude of Syntactic Disambiguation Difficulty”  In _Cognitive Science_ 45.6 Wiley Online Library, 2021, pp. e12988
  * [63] Yun Wen, Jonathan Mirault and Jonathan Grainger  “Fast syntax in the brain: Electrophysiological evidence from the rapid parallel visual presentation paradigm (RPVP).”  In _Journal of Experimental Psychology: Learning, Memory, and Cognition_ 47.1 American Psychological Association, 2021, pp. 99
  * [64] Yun Wen, Jonathan Mirault and Jonathan Grainger  “The transposed-word effect revisited: The role of syntax in word position coding”  In _Language, Cognition and Neuroscience_ 36.5 Taylor & Francis, 2021, pp. 668–673
  * [65] Noga Zaslavsky, Charles Kemp, Terry Regier and Naftali Tishby  “Efficient compression in color naming and its evolution”  In _Proceedings of the National Academy of Sciences_ 115.31 National Acad Sciences, 2018, pp. 7937–7942
  * [66] Noga Zaslavsky, Jennifer Hu and Roger P Levy  “A rate-distortion view of human pragmatic reasoning”  In _arXiv preprint arXiv:2005.06641_ , 2020
  * [67] Irene Zhou, Jennifer Hu, Roger Levy and Noga Zaslavsky  “Teasing apart models of pragmatics using optimal reference game design”  In _Proceedings of the Annual Meeting of the Cognitive Science Society_ 44.44, 2022

## References

  * [68] Dilip Arumugam, Mark K Ho, Noah D Goodman and Benjamin Van Roy  “On Rate-Distortion Theory in Capacity-Limited Cognition & Reinforcement Learning”  In _arXiv preprint arXiv:2210.16877_ , 2022
  * [69] Stephen B. Barton and Anthony J. Sanford  “A case study of anomaly detection: Shallow semantic processing and cohesion establishment”  In _Memory & cognition_ 21.4 Springer, 1993, pp. 477–487
  * [70] Thomas G Bever  “The cognitive basis for linguistic structures”  In _Cognition and the development of language_ Wiley, 1970
  * [71] Rahul Bhui, Lucy Lai and Samuel J Gershman  “Resource-rational decision making”  In _Current Opinion in Behavioral Sciences_ 41 Elsevier, 2021, pp. 15–21
  * [72] Klinton Bicknell and Roger Levy  “A rational model of eye movement control in reading”  In _Proceedings of the 48th annual meeting of the Association for Computational Linguistics_ , 2010, pp. 1168–1178
  * [73] Rafal Bogacz et al.  “The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks.”  In _Psychological review_ 113.4 American Psychological Association, 2006, pp. 700
  * [74] Marc Brysbaert and Boris New  “Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English”  In _Behavior research methods_ 41.4 Springer, 2009, pp. 977–990
  * [75] Samuel J Cheyette and Steven T Piantadosi  “A unified account of numerosity perception”  In _Nature Human Behaviour_ 4.12 Nature Publishing Group UK London, 2020, pp. 1265–1272
  * [76] Kiel Christianson, Andrew Hollingworth, John F. Halliwell and Fernanda Ferreira  “Thematic roles assigned along the garden path linger”  In _Cognitive psychology_ 42.4 Elsevier, 2001, pp. 368–407
  * [77] Thomas M Cover and Joy A Thomas  “Elements of information theory second edition solutions to problems”  In _Internet Access_ , 2006, pp. 19–20
  * [78] Thomas D Erickson and Mark E Mattson  “From words to meaning: A semantic illusion”  In _Journal of Verbal Learning and Verbal Behavior_ 20.5 Elsevier, 1981, pp. 540–551
  * [79] Fernanda Ferreira  “The misinterpretation of noncanonical sentences”  In _Cognitive psychology_ 47.2 Elsevier, 2003, pp. 164–203
  * [80] Fernanda Ferreira, Karl GD Bailey and Vittoria Ferraro  “Good-enough representations in language comprehension”  In _Current directions in psychological science_ 11.1 SAGE Publications Sage CA: Los Angeles, CA, 2002, pp. 11–15
  * [81] Fernanda Ferreira, Kiel Christianson and Andrew Hollingworth  “Misinterpretations of garden-path sentences: Implications for models of sentence processing and reanalysis”  In _Journal of psycholinguistic research_ 30 Springer, 2001, pp. 3–20
  * [82] Fernanda Ferreira and Nikole D Patson  “The ‘good enough’approach to language comprehension”  In _Language and Linguistics Compass_ 1.1-2 Wiley Online Library, 2007, pp. 71–83
  * [83] Samuel Fillenbaum  “Processing and recall of compatible and incompatible question and answer pairs”  In _Language and Speech_ 14.3 Sage Publications Sage CA: Thousand Oaks, CA, 1971, pp. 256–265
  * [84] Samuel Fillenbaum  “Pragmatic normalization: Further results for some conjunctive and disjunctive sentences.”  In _Journal of Experimental Psychology_ 102.4 American Psychological Association, 1974, pp. 574
  * [85] Stefan Frank, Leun Otten, Giulia Galli and Gabriella Vigliocco  “The ERP response to the amount of information conveyed by words in sentences”  In _Brain and language_ 140 Elsevier, 2015, pp. 1–11
  * [86] Lyn Frazier  “On comprehending sentences: Syntactic parsing strategies.”  University of Connecticut, 1979
  * [87] Lyn Frazier and Janet Dean Fodor  “The sausage machine: A new two-stage parsing model”  In _Cognition_ 6.4 Elsevier, 1978, pp. 291–325
  * [88] Lyn Frazier and Keith Rayner  “Making and correcting errors during sentence comprehension: Eye movements in the analysis of structurally ambiguous sentences”  In _Cognitive psychology_ 14.2 Elsevier, 1982, pp. 178–210
  * [89] Richard Futrell  “Information-theoretic principles in incremental language production”  In _Proceedings of the National Academy of Sciences_ 120.39 National Acad Sciences, 2023, pp. e2220593120
  * [90] Edward Gibson, Leon Bergen and Steven T. Piantadosi  “Rational integration of noisy evidence and prior semantic expectations in sentence interpretation”  In _Proceedings of the National Academy of Sciences_ 110.20 National Acad Sciences, 2013, pp. 8051–8056
  * [91] Ana C Gouvea, Colin Phillips, Nina Kazanina and David Poeppel  “The linguistic processes underlying the P600”  In _Language and cognitive processes_ 25.2 Taylor & Francis, 2010, pp. 149–188
  * [92] Daniel Grodner, Edward Gibson, Vered Argaman and Maria Babyonyshev  “Against repair-based reanalysis in sentence comprehension”  In _Journal of Psycholinguistic Research_ 32 Springer, 2003, pp. 141–166
  * [93] Peter Hagoort, Colin Brown and Jolanda Groothusen  “The syntactic positive shift (SPS) as an ERP measure of syntactic processing”  In _Language and cognitive processes_ 8.4 Taylor & Francis, 1993, pp. 439–483
  * [94] Michael Hahn, Richard Futrell, Roger Levy and Edward Gibson  “A resource-rational model of human processing of recursive linguistic structure”  In _Proceedings of the National Academy of Sciences_ 119.43 National Acad Sciences, 2022, pp. e2122602119
  * [95] John Hale  “A probabilistic Earley parser as a psycholinguistic model”  In _Second meeting of the north american chapter of the association for computational linguistics_ , 2001
  * [96] John CJ Hoeks, Laurie A. Stowe and Gina Doedens  “Seeing words in context: the interaction of lexical and sentence level information during reading”  In _Cognitive brain research_ 19.1 Elsevier, 2004, pp. 59–73
  * [97] Kuan-Jung Huang et al.  “Surprisal does not explain syntactic disambiguation difficulty: evidence from a large-scale benchmark”  PsyArXiv, 2023
  * [98] Nathaniel Imel and Shane Steinert-Threlkeld  “Modal semantic universals optimize the simplicity/informativeness trade-off”  In _Semantics and Linguistic Theory_ 1, 2022, pp. 227–248
  * [99] Aine Ito et al.  “Predicting form and meaning: Evidence from brain potentials”  In _Journal of Memory and Language_ 86 Elsevier, 2016, pp. 157–171
  * [100] Albert Kim and Lee Osterhout  “The independence of combinatory semantic processing: Evidence from event-related potentials”  In _Journal of memory and language_ 52.2 Elsevier, 2005, pp. 205–225
  * [101] Herman HJ Kolk, Dorothee J Chwilla, Marieke Van Herten and Patrick JW Oor  “Structure and limited capacity in verbal working memory: A study with event-related potentials”  In _Brain and language_ 85.1 Elsevier, 2003, pp. 1–36
  * [102] Gina R. Kuperberg  “Neural mechanisms of language comprehension: Challenges to syntax”  In _Brain research_ 1146 Elsevier, 2007, pp. 23–49
  * [103] Gina R Kuperberg, Trevor Brothers and Edward W Wlotko  “A tale of two positivities and the N400: Distinct neural signatures are evoked by confirmed and violated predictions at different levels of representation”  In _Journal of Cognitive Neuroscience_ 32.1 MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info …, 2020, pp. 12–35
  * [104] Marta Kutas and Steven A. Hillyard  “Event-related brain potentials to semantically inappropriate and surprisingly large words”  In _Biological psychology_ 11.2 Elsevier, 1980, pp. 99–116
  * [105] Roger Levy  “Expectation-based syntactic comprehension”  In _Cognition_ 106.3 Elsevier, 2008, pp. 1126–1177
  * [106] Richard L Lewis, Andrew Howes and Satinder Singh  “Computational rationality: Linking mechanism and behavior through bounded utility maximization”  In _Topics in cognitive science_ 6.2 Wiley Online Library, 2014, pp. 279–311
  * [107] Jiaxuan Li and Allyson Ettinger  “Heuristic interpretation as rational inference: A computational model of the N400 and P600 in language processing”  In _Cognition_ 233 Elsevier, 2023, pp. 105359
  * [108] Jiaxuan Li and Richard Futrell  “A decomposition of surprisal tracks the N400 and P600 brain potentials”  In _Proceedings of the Annual Meeting of the Cognitive Science Society_ 45.45, 2023
  * [109] Falk Lieder and Thomas L Griffiths  “Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources”  In _Behavioral and brain sciences_ 43 Cambridge University Press, 2020, pp. e1
  * [110] Maryellen C. MacDonald, Neal J. Pearlmutter and Mark S. Seidenberg  “The lexical nature of syntactic ambiguity resolution.”  In _Psychological review_ 101.4 American Psychological Association, 1994, pp. 676
  * [111] James A Michaelov et al.  “Strong Prediction: Language model surprisal explains multiple N400 effects”  In _Neurobiology of language_ MIT Press One Broadway, 12th Floor, Cambridge, Massachusetts 02142, USA …, 2024, pp. 1–29
  * [112] Francis Mollica et al.  “The forms and meanings of grammatical markers support efficient communication”  In _Proceedings of the National Academy of Sciences_ 118.49 National Acad Sciences, 2021, pp. e2025993118
  * [113] Dennis Norris and Sachiko Kinoshita  “Perception as evidence accumulation and Bayesian inference: insights from masked priming.”  In _Journal of Experimental Psychology: General_ 137.3 American Psychological Association, 2008, pp. 434
  * [114] Till Poppels and Roger Levy  “Structure-sensitive Noise Inference: Comprehenders Expect Exchange Errors.”  In _CogSci_ , 2016
  * [115] Alec Radford et al.  “Language models are unsupervised multitask learners”  In _OpenAI blog_ 1.8, 2019, pp. 9
  * [116] Rachel Ryskin et al.  “An ERP index of real-time error correction within a noisy-channel framework of human communication”  In _Neuropsychologia_ 158 Elsevier, 2021, pp. 107855
  * [117] Anthony J Sanford  “Context, attention and depth of processing during interpretation”  In _Mind & language_ 17.1-2 Wiley Online Library, 2002, pp. 188–206
  * [118] Chris R Sims  “Rate–distortion theory and human perception”  In _Cognition_ 152 Elsevier, 2016, pp. 181–198
  * [119] Chris R Sims  “Efficient coding explains the universal law of generalization in human perception”  In _Science_ 360.6389 American Association for the Advancement of Science, 2018, pp. 652–656
  * [120] Shane Steinert-Threlkeld  “Quantifiers in natural language optimize the simplicity/informativeness trade-off”  In _Proceedings of the 22nd Amsterdam colloquium_ , 2020, pp. 513–522  Institute for Logic, LanguageComputation at the University of Amsterdam
  * [121] Patrick Sturt, Martin J Pickering and Matthew W Crocker  “Structural change and reanalysis difficulty in language comprehension”  In _Journal of Memory and Language_ 40.1 Elsevier, 1999, pp. 136–150
  * [122] Whitney Tabor, Bruno Galantucci and Daniel Richardson  “Effects of merely local syntactic coherence on sentence processing”  In _Journal of Memory and Language_ 50.4 Elsevier, 2004, pp. 355–370
  * [123] John C Trueswell, Michael K Tanenhaus and Christopher Kello  “Verb-specific constraints in sentence processing: separating effects of lexical preference from garden-paths.”  In _Journal of Experimental psychology: Learning, memory, and Cognition_ 19.3 American Psychological Association, 1993, pp. 528
  * [124] Nan Van de Meerendonk, Peter Indefrey, Dorothee J Chwilla and Herman HJ Kolk  “Monitoring in language perception: Electrophysiological and hemodynamic responses to spelling violations”  In _Neuroimage_ 54.3 Elsevier, 2011, pp. 2350–2363
  * [125] Roger PG Van Gompel, Martin J. Pickering and Matthew J. Traxler  “Unrestricted race: A new model of syntactic ambiguity resolution”  In _Reading as a perceptual process_ Elsevier, 2000, pp. 621–648
  * [126] Marieke Van Herten, Dorothee J. Chwilla and Herman HJ Kolk  “When heuristics clash with parsing routines: ERP evidence for conflict monitoring in sentence perception”  In _Journal of cognitive neuroscience_ 18.7 MIT Press, 2006, pp. 1181–1197
  * [127] Marieke Van Herten, Herman HJ Kolk and Dorothee J. Chwilla  “An ERP study of P600 effects elicited by semantic anomalies”  In _Cognitive brain research_ 22.2 Elsevier, 2005, pp. 241–255
  * [128] Cyma Van Petten and Barbara J Luka  “Prediction during language comprehension: Benefits, costs, and ERP components”  In _International Journal of Psychophysiology_ 83.2 Elsevier, 2012, pp. 176–190
  * [129] Marten Van Schijndel and Tal Linzen  “Single-Stage Prediction Models Do Not Explain the Magnitude of Syntactic Disambiguation Difficulty”  In _Cognitive Science_ 45.6 Wiley Online Library, 2021, pp. e12988
  * [130] Yun Wen, Jonathan Mirault and Jonathan Grainger  “Fast syntax in the brain: Electrophysiological evidence from the rapid parallel visual presentation paradigm (RPVP).”  In _Journal of Experimental Psychology: Learning, Memory, and Cognition_ 47.1 American Psychological Association, 2021, pp. 99
  * [131] Yun Wen, Jonathan Mirault and Jonathan Grainger  “The transposed-word effect revisited: The role of syntax in word position coding”  In _Language, Cognition and Neuroscience_ 36.5 Taylor & Francis, 2021, pp. 668–673
  * [132] Noga Zaslavsky, Jennifer Hu and Roger P Levy  “A rate-distortion view of human pragmatic reasoning”  In _arXiv preprint arXiv:2005.06641_ , 2020
  * [133] Noga Zaslavsky, Charles Kemp, Terry Regier and Naftali Tishby  “Efficient compression in color naming and its evolution”  In _Proceedings of the National Academy of Sciences_ 115.31 National Acad Sciences, 2018, pp. 7937–7942
  * [134] Irene Zhou, Jennifer Hu, Roger Levy and Noga Zaslavsky  “Teasing apart models of pragmatics using optimal reference game design”  In _Proceedings of the Annual Meeting of the Cognitive Science Society_ 44.44, 2022


# GPT Struct Me: Probing GPT Models
on Narrative Entity Extraction

Hugo Sousa§ INESC TEC
University of Porto
Porto, Portugal
<EMAIL_ADDRESS>Nuno Guimarães§ INESC TEC
University of Porto
Porto, Portugal
<EMAIL_ADDRESS>Alípio Jorge  INESC TEC
University of Porto
Porto, Portugal
<EMAIL_ADDRESS>Ricardo Campos  INESC TEC
University of Beira Interior
Porto, Portugal
<EMAIL_ADDRESS>
###### Abstract

The importance of systems that can extract structured information from textual
data becomes increasingly pronounced given the ever-increasing volume of text
produced on a daily basis. Having a system that can effectively extract such
information in an interoperable manner would be an asset for several domains,
be it finance, health, or legal. Recent developments in natural language
processing led to the production of powerful language models that can, to some
degree, mimic human intelligence. Such effectiveness raises a pertinent
question: Can these models be leveraged for the extraction of structured
information? In this work, we address this question by evaluating the
capabilities of two state-of-the-art language models – GPT-3 and GPT-3.5,
commonly known as ChatGPT – in the extraction of narrative entities, namely
events, participants, and temporal expressions. This study is conducted on the
Text2Story Lusa dataset, a collection of 119 Portuguese news articles whose
annotation framework includes a set of entity structures along with several
tags and attribute values. We first select the best prompt template through an
ablation study over prompt components that provide varying degrees of
information on a subset of documents of the dataset. Subsequently, we use the
best templates to evaluate the effectiveness of the models on the remaining
documents. The results obtained indicate that GPT models are competitive with
out-of-the-box baseline systems, presenting an all-in-one alternative for
practitioners with limited resources. By studying the strengths and
limitations of these models in the context of information extraction, we offer
insights that can guide future improvements and avenues to explore in this
field.

###### Index Terms:

narrative entity extraction, GPT models, prompt-base learning

§§footnotetext: Equal contribution.

##  I Introduction

A narrative is a type of written or spoken discourse that tells a story by
describing a sequence of connected events [1]. At its core, a narrative is
composed of three main entities: participants, events, and temporal
expressions (referred as timexs henceforward). Participants are the drivers of
the actions, and can be people, animals, or abstract entities. Events refer to
what happens to these participants, forming the plot of the narrative. Lastly,
timexs help establish the sequence, duration, and frequency of these events,
providing the timeline of the narrative. Given the prevalence of narratives in
human communication, methods for the automatic extraction of narrative
entities have garnered significant attention and research efforts from the
natural language processing (NLP) community [2]. As an example, in the
narrative “The suspect was convicted last night.”, a narrative entity
extraction system must be able to identify “the suspect” as a participant,
“convicted” as an event, and “last night” as a timex.

The field of NLP has evolved significantly in recent years, mainly driven by
the introduction of the Transformer architecture [3]. This architecture has
been used in a multitude of large language models (LLMs) such as BERT [4], and
BLOOM [5]. Initially, the main paradigm was to apply these models to
downstream tasks by fine-tuning them. However, the introduction of GPT-3 [6]
led to a shift towards “prompt-based learning”, emphasizing the optimization
of the input prompt rather than the model itself [7]. This paradigm was
further reinforced with the introduction of GPT-3.5, commonly known as ChatGPT
[8]. Such advances have led several research studies to probe the capabilities
of LLMs such as reasoning [9], semantics [10], machine translation[11], and
named entity recognition [12], presenting state-of-the-art results in some of
them.

The rising capabilities of these models present an intriguing question: Can
LLMs effectively extract narrative entities from textual data? In an effort to
answer this question we probe, GPT-3 and ChatGPT on the extraction of events,
participants, and timexs on a set of news articles written in European
Portuguese. To design the prompts, we conducted experiments using various
templates on a subset of $20$ carefully sampled and representative documents
of our dataset. After identifying the best prompt templates for each entity-
model pair, we assess GPT-3 and ChatGPT effectiveness and compared them with
out-of-the-box baseline systems in a partition of the data that we allocated
to test the models.

In summary, the main contributions of this paper are:

  1. 1.

A prompt engineering methodology to evaluate LLMs using different levels of
information to leverage their capabilities in the extraction of entities in
narrative texts.

  2. 2.

A robust evaluation of GPT-3 and ChatGPT in the extraction of narrative
entities in European Portuguese news.

The remainder of the paper is structured as follows: Section II frames our
research within the context of existing studies. Section III provides a
description of the experimental setup employed for this research.
Subsequently, Section IV briefly outlines the construction of the prompts
along with the preliminary ablation study performed for the selection of the
most effective prompt-model-entity triplet. Section V presents the results of
the GPT models and compares these with the results achieved by baseline
systems. Lastly, Section VI presents the conclusions and sheds light on
potential avenues for future research.

##  II Related Work

The Generative Pre-trained Transformers (GPT) models from OpenAI[6, 8]
presented several leaps in effectiveness, achieving state-of-the-art results
in a variety of NLP tasks [13], and demonstrated impressive capabilities
across a broad spectrum [12]. For instance, in machine translation, GPT models
achieved competitive effectiveness when compared with domain-specific state-
of-the-art systems in high-resource languages [11]. Additionally, GPT-3
summaries have been preferred by human evaluators in text summarization tasks
[14].

Regarding the capabilities of LLMs in extracting structured information from
text, the research studies present mixed results. Tang et al. [15] reported
poor effectiveness from ChatGPT in the named entity recognition task in the
medical domain, achieving an $F_{1}$ score of only $23.37\%$. Conversely, in
the narrative domain [1], Stammbach et al. [16] found that GPT-3 outperformed
previous methods in extracting participants, from narrative texts using a
zero-shot prompt approach.

For the extraction of events, there have been works [17, 18] that achieved
state-of-the-art results on the ACE dataset [19] by treating event extraction
as a machine reading comprehension task. More recently, GPT models have also
been evaluated for the same task [20]. Nevertheless, the results indicated
that ChatGPT is not robust enough. Furthermore, the authors also stated that
the refinement of the prompt did not improve the effectiveness of the models.
However, the study used a different prompt template from ours. Finally,
concerning temporal expression extraction, our research fills a gap since, to
the best of our knowledge, no studies have explored prompt-based approaches in
this domain.

##  III Experimental Setup

With this study, we intend to probe the capabilities of LLMs in the extraction
of narrative entities through a prompt-based approach. We aim to explore how
well these models can generate and comprehend narrative content by leveraging
their vast language understanding and generation abilities.

To ensure the replicability of the experiments, we made the code available (as
well as more details on the experimental setup and results) in a GitHub
repository111https://github.com/hmosousa/gpt_struct_me.

###  III-A Model Selection

When selecting which models to implement in our research, several layers of
criteria had to be taken into account.

#### Effectiveness

We aim to apply models that present state-of-the-art results on the generative
task. This ensures that our research is not only up-to-date but also
contributes with new findings to the scientific community.

#### Hardware Requirements

The most effective LLMs are (typically) the ones with the highest number of
parameters. This implies a demand for specific hardware that can execute these
large models, which is far beyond the capacity of typical consumer-grade
equipment.

#### Availability

Although numerous LLMs have been developed, most of them are typically owned
and maintained by large corporations or research institutions that choose to
keep them proprietary. This includes models such as PaLM [21] and Chinchilla
[22], which despite reporting great effectiveness on several benchmarks, are
not of public access, limiting their study.

#### Domain

We sought models that have been developed for general use, rather than being
fine-tuned for a specific task. For example, CodeX [23] has been optimized for
producing code, and Galatica [24] has been optimized for scientific texts.
Such specialization was not desirable for our research.

After evaluating all these factors, GPT-3 and ChatGPT emerged as the obvious
choices. These models are not only effective and easily accessible but also
suitable for a broad range of tasks, which aligns with the objectives of our
research.

However, it is important to acknowledge certain drawbacks associated with
these models. The most prominent is the lack of reproducibility and control,
as the OpenAI API222https://openai.com/blog/openai-api parameters alone cannot
guarantee deterministic answers. To address this concern, we set the
temperature parameter of the API to zero. Although this does not ensure
complete determinism, it significantly reduces
variability333https://platform.openai.com/docs/guides/gpt/why-are-model-
outputs-inconsistent. Despite this limitation, using these models provides us
with a reliable estimate of the effectiveness of current state-of-the-art
LLMs, which is the main goal of this study.

###  III-B Dataset Description

Three main factors guided the selection of the appropriate dataset. The first
concern was to ensure that the dataset had not been employed during the
training phase of the models. This factor mitigates the risk of data leakage
which could artificially inflate the results. The second requirement was that
the dataset included annotations for all the entities, namely: events,
participants, and timexs. The final consideration was the dataset’s size and
complexity since it had to be sufficiently large and diverse to yield
trustable results.

A dataset we found to meet all the above criteria is the Text2Story Lusa
dataset [25]. This dataset comprises $119$ news articles written in European
Portuguese that were densely annotated with a multilayer semantic annotation
scheme [26], which include the annotation of $3,057$ events, $3,546$
participants, and $325$ timexs. Furthermore, this dataset was only released
after the development of this research project, therefore ensuring that the
models did not have access to it in the learning phase.

###  III-C Task Definition

Conventional methods treat narrative extraction as an extractive task. This
involves having the system pinpoint the exact position or “offset” of tokens
that refer to an entity within a given passage. However, when humans are asked
to identify entities within a text, they do not pinpoint the exact offsets of
those entities within the text. Instead, humans tend to list out the entities
they identify. For instance, if one was asked to name the characters in a
story, the answer would likely be the list of entities – “Mary”, “Gaspar”,
“the baby” – rather than providing the offsets of their mentions within the
text.

In our work, we follow this human-oriented approach, by prompting the models
to list the entities in the text rather than presenting the offsets in which
the entities appeared. Then we compared this list of predicted entities with
the list of entities on the annotation file as explained in the next section.

###  III-D Evaluation Metrics

To evaluate the effectiveness of the systems we compare the annotated and
model-generated entity lists with the traditional precision ($P$), recall
($R$), and $F_{1}$ score. These, however, only take into account exact matches
as correct, ignoring predictions that partially overlap with the true label.
Thus, to enhance the comprehensiveness of our evaluation, we introduce a
supplementary metric: the relaxed $F_{1}$ score ($F_{1_{r}}$). This score
considers the partial (but contiguous) overlap between the annotated entities
and the ones predicted by the system, thereby accounting for a more nuanced
evaluation.

##  IV Prompt Engineering

Extracting the best effectiveness of generative models relies heavily on
carefully constructing the prompt. Although several studies have been
developed [7], the process of constructing an effective prompt is iterative
and experimental, requiring multiple iterations and empirical experiments to
achieve the desired outcomes.

###  IV-A Prompt Template Definition

We crafted a prompt template that we found to be able to successfully complete
the narrative entity extraction task. This is composed of seven modules 444We
encourage interested readers to delve into our code repository, where they can
find not only the code responsible for generating the prompts but also the
generated prompts.:

  1. 1.

Task States the task that the generative model is expected to perform.

  2. 2.

Definition The description associated with the entity being extracted. The
idea is to provide the model with the same guidelines given to the annotators.

  3. 3.

Classes A list of possible classes for the entity being extracted. However,
when using this module, we also modify the task of the model to “Extract and
classify all <entity>.” instead of just “Extract all <entity>.”. The idea is
that by asking the models to perform a classification task we are encouraging
them to gain a more nuanced understanding of the text content, as they need to
not only identify the specific entity but also understand and categorize it
appropriately. This aligns, to some extent, with the chain-of-though prompting
approach, which has been shown to improve the ability of LLMs to perform
complex reasoning [27].

  4. 4.

Example This is a simple one-shot example that presents the input text along
with the expected output.

  5. 5.

Format To ensure that the generated outputs could be programmatically parsed,
we specified the required format explicitly.

  6. 6.

Input The input text from which we want the model to extract the entities.

  7. 7.

Output This is the last line of the prompt, left blank for the model to
generate the answer.

Note that some of the components in the template are not required for the
task, namely: Definition, Classes, and Example. The inclusion of these aims to
provide additional information to the prompt, potentially enhancing the
effectiveness of the model. However, since we cannot determine a priori which
of these modules is beneficial, we conduct a preliminary ablation study to
evaluate the impact of each optional module of the template on each model-
entity pair.

###  IV-B Preliminary Ablation Study

We use $20$ documents from our dataset to estimate the effectiveness of each
prompt-model-entity triplet. Due to the small size of the sample, one needs to
be careful in the sampling process, since the goal is to have a sample that
produces a good estimate of the expected effectiveness of the templates on the
full corpus. Thus, the following steps were used to select the documents to be
included in the sample: 1) Compute the $T-1$ quantiles of the number of tokens
in the documents, with $T$ being the number of documents to be sampled (in our
case $T=20$); 2) Divide the dataset into $T$ ranges, each being bounded by the
computed quantiles; and 3) Randomly sample one document from each range. Using
this procedure, we ensured that the sample was representative of the entire
corpus. Additional information regarding the sampling process is available in
our code repository.

After building the prompts for each of the $20$ documents, we prompt the
models. We briefly summarize the results achieved.

First, the effectiveness of both models significantly differs according to the
type of entity. The GPT models’ capability in extracting timexs seems to
achieve better results when compared to participants and events. When
comparing GPT-3 and ChatGPT, the first achieves a better overall effectiveness
in the extraction of timexs and participants while ChatGPT seems to be better
in the extraction of events. Second, when analyzing the results at a prompt
level, we observed that adding more information does not always translate into
better results (for example, combining the Definition and the Task results in
an overall worse effectiveness than prompting only with the Task). Third, the
inclusion of the Example improves, on average, the $F_{1}$ score in the
different extraction tasks. Lastly, one-shot prompts provide the best results
when they combine all three types of information, while in zero-shot prompts
the best results are achieved when the prompts do not include Definitions. A
more detailed description of the results is available in the paper repository.

Given the results, we select the six prompts that achieved the highest $F_{1}$
score for each entity-model pair and use them to evaluate the models’
effectiveness on the remaining documents of the dataset.

##  V Model Evaluation

For the final evaluation, the remaining $96$ documents within the dataset were
used. The GPT models’ effectiveness is compared against two established
baselines for the extraction of events (SRL [28] and TEFE [29]) and timexs
(HeidelTime [30] and TEI2GO [31]), and one baseline for the participant
extraction task (SRL [28]). The results are presented in Table I.

Table I: Results on the test set of the Baseline and GPT models. |  |  $P$ |  $R$ | $F_{1}$ |  $F_{1_{r}}$
---|---|---|---|---|---
Timexs | TEI2GO | $50.7$ |  59.4 | 54.7 | $55.7$
| HeidelTime | $50.1$ |  59.3 | 54.2 | $56.3$
| SRL | $23.6$ |  35.7 | 28.4 | $50.5$
| GPT-3 | $58.5$ |  73.5 | 65.1 | 73.0
| ChatGPT | $34.5$ |  58.6 | 43.5 | $48.0$
Participants | SRL | $27.1$ |  20.7 | 23.4 | $47.7$
| GPT-3 | $35.9$ |  35.8 | 35.8 | 51.0
| ChatGPT | $40.1$ |  46.6 | 43.1 | $47.4$
Events | SRL | $67.1$ |  40.4 | 50.4 | $83.5$
| TEFE | $95.4$ |  9.1 | 16.6 | 95.6
| GPT-3 | $41.9$ |  53.3 | 46.9 | $51.7$
| ChatGPT | $50.7$ |  44.3 | 47.2 | $61.4$

As observed in Table I, the GPT models outperform the baselines in the
extraction of timexs and participants. However, concerning events, we can see
that SRL and TEFE beat both GPT models in the $F_{1}$ strict and relaxed
metrics. Given that all the baselines are used out-of-the-box while the
prompts used in one-shot include an annotation example, the results go
according to what was expected in timexs and participants. However, it is
interesting to see that both GPT models cannot surpass the baselines in the
events extraction task. We hypothesize that, given that the concept of event
in the annotation is very specific, one annotation example is not enough for
these models. Since SRL and TEFE are more task-driven models and are already
more familiar with the concept through their learning process, they can
outperform more broader and general models without seeing an example of the
data beforehand.

In the extraction of timexs and participants, the GPT models outperform the
best baseline in $10$ and $20$ points of strict $F_{1}$, respectively. We
hypothesize that since the concept of timexs and participants in our
annotation is more aligned with the common knowledge, GPT models have a better
chance to outperform the baselines due to the large textual data on which they
were trained on. In addition, with the inclusion of an example, the model is
capable of gathering additional knowledge making it capable of surpassing the
results achieved by the baselines.

Finally, we can see that ChatGPT outperforms GPT-3 in the extraction of
participants and events, with GPT-3 having better effectiveness in timexs.
However, it is important to note that our evaluation is not focused on the
models themselves but on the combination of these with our prompt methodology.
We provide a uniform approach that enables the results presented in Table I
but, by itself, it is not guaranteed to maximize the effectiveness of GPT
models.

##  VI Conclusions and Future Work

In this work, we evaluate through prompt-based learning the capability of
GPT-3 and ChatGPT in the task of narrative entity extraction. To ensure a
systematic evaluation, we develop a prompt methodology that combines different
levels of knowledge. We conduct a preliminary ablation study to choose the
best template prompts for each model-entity pair on a subset of the dataset
and use the best templates to compare the models with a set of baselines on
the remaining documents of the dataset. The results show that the combination
of the best prompts and GPT models outperform (using strict $F_{1}$) the
baseline in the extraction of participants and timexs but failed to reach the
same results in the extraction of events. Although there is no definitive
guarantee that superior results could not be attained using alternative prompt
templates, the robustness of out-of-the-box models, applicable to a variety of
entity extraction tasks, still offer significant advantages since they do not
require exhaustive experimentation with different prompts.

In future work, we intend to refine and improve our prompt methodology and
evaluate more complex tasks in narrative extraction using LLMs, like the
extraction of temporal, aspectual, objectual, and semantic role links. In
addition, we also intend to tweak the current prompts, for instance, by
extending the number of examples used on the Example component of the prompt.
Further, we intend to access the present approach in other languages and
domains to extend the evaluation of the models. All these possibilities
combined with the release of more effective LLMs (such as GPT-4 [32]) lead to
several promising research paths on narrative extraction that are worth
exploring.

##  VII Acknowledgments

This work is financed by National Funds through the Portuguese funding agency,
FCT - Fundação para a Ciência e a Tecnologia, within project LA/P/0063/2020
and the project StorySense, with reference 2022.09312.PTDC.

## References

  * [1] M. Toolan, _Narrative: A critical linguistic introduction_. Routledge, 2012.
  * [2] B. Santana, R. Campos, E. Amorim, A. Jorge, P. Silvano, and S. Nunes, “A Survey on Narrative Extraction from Textual Data,” _Artificial Intelligence Review_ , 1 2023. [Online]. Available: https://link.springer.com/10.1007/s10462-022-10338-7
  * [3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention Is All You Need,” in _Advances in Neural Information Processing Systems_ , vol. abs/1706.03762. Long Beach, California, USA: Neural Information Processing Systems Foundation, Inc. (NeurIPS), 12 2017, pp. 5998–6008. [Online]. Available: http://arxiv.org/abs/1706.03762
  * [4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in _Proceedings of the 2019 Conference of the North_. Stroudsburg, PA, USA: Association for Computational Linguistics, 2019, pp. 4171–4186. [Online]. Available: http://aclweb.org/anthology/N19-1423
  * [5] B. Workshop and Others, “BLOOM: A 176B-Parameter Open-Access Multilingual Language Model,” _arXiv_ , 11 2022. [Online]. Available: http://arxiv.org/abs/2211.05100
  * [6] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language Models are Few-Shot Learners,” _arXiv_ , 5 2020. [Online]. Available: http://arxiv.org/abs/2005.14165
  * [7] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing,” _ACM Computing Surveys_ , vol. 55, no. 9, pp. 1–35, 9 2023.
  * [8] OpenAI, “Introducing ChatGPT.” [Online]. Available: https://openai.com/blog/chatgpt
  * [9] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “ReAct: Synergizing Reasoning and Acting in Language Models,” _arXiv_ , 10 2022\.
  * [10] L. Zhang, M. Wang, L. Chen, and W. Zhang, “Probing GPT-3’s linguistic knowledge on semantic tasks,” in _Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_. Abu Dhabi, United Arab Emirates (Hybrid): Association for Computational Linguistics, Dec. 2022, pp. 297–304. [Online]. Available: https://aclanthology.org/2022.blackboxnlp-1.24
  * [11] A. Hendy, M. Abdelrehim, A. Sharaf, V. Raunak, M. Gabr, H. Matsushita, Y. J. Kim, M. Afify, and H. H. Awadalla, “How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation,” _arXiv_ , 2 2023.
  * [12] C. Qin, A. Zhang, Z. Zhang, J. Chen, M. Yasunaga, and D. Yang, “Is ChatGPT a General-Purpose Natural Language Processing Task Solver?” 2 2023. [Online]. Available: http://arxiv.org/abs/2302.06476
  * [13] A. Srivastava and Others, “Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models,” _arXiv_ , 6 2022.
  * [14] T. Goyal, J. J. Li, and G. Durrett, “News Summarization and Evaluation in the Era of GPT-3,” _arXiv_ , 9 2022.
  * [15] R. Tang, X. Han, X. Jiang, and X. Hu, “Does Synthetic Data Generation of LLMs Help Clinical Text Mining?” _arXiv_ , 3 2023.
  * [16] D. Stammbach, M. Antoniak, and E. Ash, “Heroes, Villains, and Victims, and GPT-3: Automated Extraction of Character Roles Without Training Data,” in _Proceedings of the 4th Workshop of Narrative Understanding (WNU2022)_. Stroudsburg, PA, USA: Association for Computational Linguistics, 2022, pp. 47–56.
  * [17] X. Du and C. Cardie, “Event Extraction by Answering (Almost) Natural Questions,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Stroudsburg, PA, USA: Association for Computational Linguistics, 2020, pp. 671–683.
  * [18] J. Liu, Y. Chen, K. Liu, W. Bi, and X. Liu, “Event Extraction as Machine Reading Comprehension,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Stroudsburg, PA, USA: Association for Computational Linguistics, 2020, pp. 1641–1651.
  * [19] Linguistic Data Consortium, “ACE (Automatic Content Extraction) English Annotation Guidelines for Entities,” Linguistic Data Consortium, Tech. Rep., 2008. [Online]. Available: http://projects.ldc.upenn.edu/ace/
  * [20] J. Gao, H. Zhao, C. Yu, and R. Xu, “Exploring the Feasibility of ChatGPT for Event Extraction,” _arXiv_ , 3 2023.
  * [21] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H. Michalewski, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern, D. Eck, J. Dean, S. Petrov, and N. Fiedel, “PaLM: Scaling Language Modeling with Pathways,” _arXiv_ , 4 2022.
  * [22] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. v. d. Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre, “Training Compute-Optimal Large Language Models,” _arXiv_ , 3 2022.
  * [23] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating Large Language Models Trained on Code,” _arXiv_ , 7 2021.
  * [24] R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and R. Stojnic, “Galactica: A Large Language Model for Science,” _arXiv_ , 11 2022.
  * [25] P. Silvano, A. Jorge, A. Leal, E. Amorim, H. Sousa, I. Cantante, R. Campos, and S. Nunes, “Text2Story Lusa Annotated ,” 2023. [Online]. Available: https://rdm.inesctec.pt/dataset/cs-2023-018
  * [26] P. Silvano, A. Leal, F. Silva, I. Cantante, F. Oliveira, and A. Jorge, “Developing a multilayer semantic annotation scheme based on ISO standards for the visualization of a newswire corpus,” in _Proceedings of the 17th Joint ACL - ISO Workshop on Interoperable Semantic Annotation_ , 2021, pp. 1–13. [Online]. Available: https://aclanthology.org/2021.isa-1.1.pdf
  * [27] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” _arXiv_ , 1 2022.
  * [28] S. Oliveira, D. Loureiro, and A. Jorge, “Improving Portuguese Semantic Role Labeling with Transformers and Transfer Learning,” in _2021 IEEE 8th International Conference on Data Science and Advanced Analytics (DSAA)_. Porto, Portugal: IEEE, 10 2021, pp. 1–9. [Online]. Available: https://ieeexplore.ieee.org/document/9564238/
  * [29] A. d. S. B. Sacramento and M. Souza, “Joint event extraction with contextualized word embeddings for the portuguese language,” in _Intelligent Systems_ , A. Britto and K. Valdivia Delgado, Eds. Cham: Springer International Publishing, 2021, pp. 496–510.
  * [30] J. Strötgen and M. Gertz, “Multilingual and Cross-Domain Temporal Tagging,” _Language Resources and Evaluation_ , vol. 47, no. 2, pp. 269–298, 6 2013. [Online]. Available: http://link.springer.com/10.1007/s10579-012-9179-y
  * [31] H. Sousa, R. Campos, and A. Jorge, “TEI2GO: A Multilingual Approach for Fast Temporal Expression Identification,” in _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_ , 2023.
  * [32] OpenAI, “GPT-4 Technical Report,” _arXiv_ , 3 2023. [Online]. Available: http://arxiv.org/abs/2303.08774